次世代PCクラスタのためのシステムソフトウェア

14:20 - 14:45

システムソフトウェア技術部会
石川裕（東京大学）

謝辞：IHKおよびMcKernelの開発の一部は文部科学省「将来のHPCIシステムのあり方の調査研究」における課題名「レイテンシコアの高度化・高効率化による将来のHPCIシステムに関する調査研究」の支援を受けている

1

今後のPCクラスタ像
 HPC向けコモディティ製品の動向
 メニーコア化
 メモリ階層

キャッシュ、オンパッケージ、オンボード、SSD、HDD

Many Core
Host
Many
Core
Host
Node
CPU Unit
Many
Core
Host
Node
CPU Unit
HostNode
CPU Unit Many Core
SSD Node
CPU Unit
SSD
SSD
Interconnect
SSD
Interconnect
Interconnect
Interconnect

 インターコネクト

Network for Nodes andStorage Area Network

Many Core
Many CoreCPU
Many CoreCPU
Many CoreCPU
Many CoreSSD
CPUSSDCPUSSD
Interconnect
SSD
Interconnect
SSD
Interconnect
Interconnect
Interconnect

Hierarchical Storage
System
出展：ISC’13 Keynote by Raj Hazra, Intel

2013/12/13

2

なぜ新しいＯＳ？
A Novel OS architecture is required to support the following features:
•••





Core Capability

O(100) Core / NodeO(1M+) Node, O(100M+) Core

Reducing cache pollution •Localizing data•
Managing memory
hierarchy••

Reducing memory contention •Reducing data movement
among cores
Providing fast communication •Parallelizing OS functions
achieving less data movement

Providing scalable process
management, communication,and file I/OProviding fault resilience

Exploring such a novel operating system is huge design space New OS mechanisms and APIs are revolutionarily/evolutionally created andexamined, and selected
Needs a research vehicle, also deployable To enable kernel developers quickly to program a new kernel feature andevaluate it To share ideas and their implementations

2013/12/13

3

なぜ新しいＯＳ？


先端アーキテクチャ上のシステムソフトウェアの未成熟
我々が必要とするシステムソフトウェア環境がタイムリに提供されるか？
 要求する基本性能・スケーラビリティを満たすことができるのか？
Linuxカーネルのメニィコア対応 vs. 新OS

ユーザに対してはLinuxカーネルAPIが提供できれば良いLinuxカーネルAPI維持コストがどのくらいか？

2013/12/13

3,482
204,523
98,771
9,086
65,048300,301
311,707992,918

mckernel 28,339
ihk
23,608TOTAL 51,947.
.
.

LinuxKernel.
.
.

User process

init
kernelmmipc
fs (core)arch/x86
include
TOTAL

開発中カーネル行数
User process

Lunux 3.12.2の行数（一部）

User process

Daemon

メニィコア向けの独自拡張を維持する手間 vs. 新OSの維持コスト

DaemonLight-weight
Micro Kernel

Many Core
4

IHK and McKernel in Xeon Phi
In case of Non-Bootable Many Core
(This configuration is called “Attached”)
mcexecLinux Kernel
HelperThreads

MPI

HostPGASLinux API (glibc)

mcctrlIHK-IKCIHK-Linux driver

OpenMP

McKernel
IHK-IKC
IHK-cokernel

PCI-Express

Many Core

Infiniband
Network Card

mcexecLinux Kernel
HelperThreads

MPIPGAS• IHK (Interface for Heterogeneous Kernel)• IHK-Linux driver: provides the functions of co-processors,
such as booting, memory copy, and interrupt• IHK-cokernel: Abstracts the hardware functions of themanycore devices• IHK-IKC: providing communication between the Linuxand co-kernel• McKernel• lightweight micro kernel running on co-processors

Linux API (glibc)

mcctrl

McKernel

IHK-IKC
IHK-Linux driverHostOpenMPFeatures implemented so far in KNC• glibc and pthread• Thread and memory management• File I/O, delegated to Linux in host
• Memory map and dynamic link library• Process launcher in host
• Direct Communication with InfiniBand• MPI library (not fully) running on Xeon Phi• OpenMP environment with Intel compiler

IHK-IKC
IHK-cokernel
Application Cores

McKernel
IHK-IKC
IHK-cokernel

Many Core
Dedicated OS service Cores

Infiniband
Network Card

2013/12/13

5

IHK and McKernel
In case of Bootable Many CoreIHK and McKernel also run on Xeon andthe vmware virtual machine(This configuration is called “Built-in”)
mcexecLinux Kernel
HelperThreads

MPImcctrl

IHK-IKC
IHK-Linux driver

Cores for LinuxOpenMPPGASLinux API (glibc)

McKernel

McKernel

IHK-IKC
IHK-cokernel
Application Many
Cores CoreIHK-IKC
IHK-cokernel
Dedicated OS service Cores

Interconnect• IHK (Interface for Heterogeneous Kernel)• IHK-Linux driver: provides the functions of co-processors,
such as booting, memory copy, and interrupt• IHK-cokernel: Abstracts the hardware functions of themanycore devices• IHK-IKC: providing communication between the Linuxand co-kernel• McKernel• lightweight micro kernel running on co-processors

2013/12/13

Features implemented so far in KNC• glibc and pthread• Thread and memory management• File I/O, delegated to Linux in host
• Memory map and dynamic link library• Process launcher in host
• Direct Communication with InfiniBand• MPI library (not fully) running on Xeon Phi• OpenMP environment with Intel compiler

6

計算ノード上のOSカーネル
 Linux Kerneｌ＋Loadable LWMK (McKernelとは限らず)
 アプリケーションによってLWMKを入れ替えられるようにする
 Linuxが動いているコア以外のコア全てを使うLWMK Linuxが動いているコア以外のコアを分割していくつかのLMKが動作

 LWMKのuserlandはLinux API踏襲
Linux kernel is resident

App A, requiring LWKA, Is invoked

App C, requiring
LWK-C, Is invokedLinux
Kernel

LMK-C
LMK-C
LMK-C
Linux
KernelFinishLinux LWK-A

FinishKernelApp B, requiring
LWK-C, Is invoked

LMK-B
LMK-B
LMK-B
Linux
KernelFinish2013/12/13

7

How an application program is executed (1/2)$ mcexec ./a.out• The mcexec program is compiled with options -fPIE -pie
• The mcexec binary is loaded in the different address than the regulara.out binary• It creates an a.out process in McKernel• The a.out user virtual memory space in McKernel is shared by the same
address space in the mcexec process.

2013/12/138

How an application program is executed (2/2)
part of a.out code
01: int fd; /* bss area */02: main() {03:
int nr; char buf[512]; /* stack area */04:
fd = open(“/tmp/abc”); /* string is in data area */05:
nr = read(fd, buf, 512);06:/* …. */ }$ mcexec ./a.out
part of mcexec code
while (ioctl(fd, MCEXEC_UP_WAIT_SYSCALL, &w)== 0) {
switch (w.sysnumber) {case SYS_OPEN: /***/; break;case SYS_EXIT:/***/; break;/* …. */
default: do_generic_syscall(…); }do_syscall_return(…); }

1)
2)
3)
4)

5)
6)

Linux: ioctl is issued to receive requests from McKernel
McKernel: The open system call is issued by a.out. McKernel forwards this request toLinux with arguments without any modification of argument addresses (not copy).
Linux: the request is received and returned to the mcexec
Linux: mcexec performs the request of the system call. In case of open system call,the file name is checked whether or not it is under /proc or /sys special directory. Ifso, special handling for MIC, others regular open system call is invoked.
Linux: mcexec issues do_syscall_return to send the result back to McKernel
McKernel: When the reply is received, the result is returned to the user program.

McKernelLinux4) Issueing open syscall where thebss area of a.out, i.e., “/tmp/abc”,
is accessed1)ioctl(….)

fd = open(“/tmp/abc”);

2)5) do_syscall_return(…);3)

2013/12/13

mcctrl.kokernel module6)

9

DCFA and MPICH/DCFA
Hostmcexec
DCFA-MPICMD Server
DCFA
CMD Server

IB Verbs Library

DCFA (Host)
IBV Extension

MLX4 Library

DCFA (Host)
MLX4 Library Extension

Initialization of HCA
Memory Registration
Creation of WQE, QP, and CQ

DCFA (Host)
Modified IB core
MLX4 Driver

The OFED kernel
module is modified totake care of memoryarea of PCI device, i.e.,
MIC memory area

MPIApplication

DCFA-MPIDCFA IB IFUser / Kernel Mode

User / Kernel ModeIB uverbs

Xeon Phi

MCCTRL

DCFA-MPICMD Client

IHK-IKCIHK-IKC

IHK-Linux driver

IHK-cokernel

DCFA
CMD Client

Issuing Send/RDMA

PCI Express

InfiniBand HCA
2013/12/13

10

McKernelシステムコール群• 下記以外のLinux system callはLinuxに委譲*はシステムコールエントリとして存在するがダミー関数
実装済み

実装予定

Process
Threadclone, exit_group, exit, kill, tgkill,
rt_sigaction, rt_sigprocmask,
set_tid_address, getrlimit, getpid,
arch_prctl, futex, set_robust_list

getuid, getgid, setuid, setgid, geteuid, getegid, setpgid,getpgrp, setsid, setreuid, setregid, getgroups, setgroups,
setresuid, getresuid, setresgid, getresgid getpgid,
setfsuid, setfsgid, getsid, setrlimit, gettid,
set_thread_area, get_thread_area

Memorymanagement

mmap, munmap, mprotect, brk, madvise*mremap, remap_file_pages, mincore, shmget, shmat,shmctl, shmdt, mlock, munlock, mlockall, munlockall,
modify_ldt, swapon*, swapoff*

Schedulingsched_setaffinity*, sched_getaffinity*,sched_yieldnanosleep, getitimer, alarm, setitimer, gettimeofday,times, settimeofday, time, sched_setaffinity,sched_getaffinity

Performance
Counter

独自インターフェイスpmc_init, pmc_start,
pmc_stop, pmc_reset

PAPI対応 McKernel上へのプロセス生成はLinuxからおこなう。現在McKernel上は1プロセスNスレッド。今後複数プロセス生成
 Mmapすべてのフラグを実装していない swap機能はない /sys, /procは、Linux側でXeon Phi環境をエミュレーション•

さらにMcKernel側で実装すべき機能は検討中

2013/12/13

11

MPICHの移植状況

ネットワークバンド幅

ネットワークバンド幅(4B -- 1024B)

6.E+086.E+07

5.E+085.E+07

4.E+084.E+07

3.E+083.E+072.E+082.E+07

1.E+081.E+070.E+00
1.E+00 1.E+01 1.E+02 1.E+03 1.E+04 1.E+05 1.E+06 1.E+07 1.E+08 1.E+09

0.E+00MPICH-takagiIntel MPI

0200400600

MPICH-takagi

800

1,000

1,200

Intel MPI

122013/12/13

現状＆今後 2013年11月：1st リリース
 IHK, McKernel, DCFA, MPICH/DCFA for Xeon Phi

 IHK and McKernel for Xeon and vmware

 メーリングリスト
 mckernel-users@pccluster.org立ち上げ

 Linux API機能検証実施中
 LTP (Linux Test Project)利用

 IHK, McKernel内部ドキュメント作成中
 今後
 Linux API性能検証実施
 McKernel OS Services実装
 2014年3月：2ndリリース
 Xeon & Xeon PHI以外のアーキテクチャへの適用

2013/12/13

13関連論文発表1.

2.

3.

4.
5.
6.
7.

8.

9.

10.

Masamichi Takagi, Yuichi Nakamura, Atsushi Hori, Balazs Gerofi, and Yutaka Ishikawa, "Revisiting
Rendezvous Protocols in the Context of RDMA-capable Host Channel Adapters and Many-Core Processors,"
EuroMPI2013, 2013.
Balazs Gerofi, Akio Shimada, Atsushi Hori, Yutaka Ishikawa, “Partially Separated Page Tables for Efficient Operating System
Assisted Hierarchical Memory Management on Heterogeneous Architectures,” CCGRID 2013, pp. 360-368, 2013.
Min Si, Yutaka Ishikawa, and Masamichi Takagi, "Direct MPI Library for Intel Xeon Phi co-processors," The3rd Workshop on Communication Architecture for Scalable Systems (CASS 2013) in conjunction withIPDPS2013, 2013.
Min Si, Yutaka Ishikawa, “Design of Communication Facility on Heterogeneous Cluster ,” 情報処理学会、2012-HPC-133、2012.
Min Si, “An MPI Library Implementing Direct Communication for Many-Core Based Accelerators,” SC’12 poster, 2012
Yuki Matsuo, Taku Shimosawa, and Yutaka Ishikawa, "A File I/O System for Many-Core Based Clusters,” in conjunction
with ICS2012, 2012.
Min Si and Yutaka Ishikawa, "Design of Direct Communication Facility for Manycore-based Accelerators, “ to appear atCASS2012 in conjunction with IPDPS2012, 2012
下沢、石川、堀敦史、並木美太郎、辻田祐一、「メニーコア向けシステムソフトウェア開発のための実行環境の設計と実装」、情報処理学会、 2011-OS-118(1), 1-7, 2011.
Taku Shimosawa, Yutaka Ishikawa, "Inter-kernel Communication between Multiple Kernels on MulticoreMachines", IPSJ Transactions on Advanced Computing Systems Vol.2 No.4 (ACS 28), pp. 64-82, 2009.
Taku Shimosawa, Hiroya Matsuba, Yutaka Ishikawa, "Logical Partitioning without Architectural Supports",
32nd IEEE Intl. Computer Software and Applications Conference (COMPSAC 2008), pp. 355-364, 2008.

2013/12/13

14