Joint Center for Advanced High Performance Computing

Overview

System(Configuration

JCAHPC (Joint Center for Advanced HPC), which is a cooperative organization by the University of Tokyo and Universityof Tsukuba for joint procurement and operation of the largest scale of supercomputer in Japan, introduced a newsupercomputer system “Oakforest9PACS” with 25 PFLOPS peak performance and started its operation from December
1st, 2016. The Oakforest9PACS system is ranked at #6 in TOP500 List of November 2016 with 13.55 PFLOPS of Linpack
performance, and also recognized as Japan's fastest supercomputer. The system is installed at the Kashiwa ResearchComplex II building in the Kashiwa9no9Ha campus, the University of Tokyo.
The Oakforest9PACS system has 8,208 compute nodes, each of which consists of the latest version of Intel Xeon Phi
processor (code name: Knights Landing), and Intel Omni9Path Architecture as the high performance interconnect. The
Oakforest9PACS system is the largest cluster solution with Knights Landing processor as well as also the largest
configuration with Omni9Path Architecture in the world. The system is integrated by Fujitsu Co. Ltd, and its PRIMERGY
server is employed as each of compute node. Additionally, the system employs the Lustre shared files system (capacity:26 PB), and IME (fast file cache system, 940 TB), both of which are provided by DataDirect Network (DDN).
All the computation nodes and servers including login nodes, Lustre servers and IME servers are connected by a fullbisection bandwidth of Fat9Tree interconnection network with Intel Omni9Path Architecture to provide highly flexible job
allocation over the nodes and high performance file access.

TOP(500(#6((#1(in(Japan),(HPCG(#3((#2),(
Green(500(#6((#2)((@Nov.(2016,(IO(500(#1(@Nov.(2017ComputeZNodes:Z25 PFlops
FujitsuZPRIMERGYZCX1640ZM1Z
xZ8ZnodeZinsideZCX600ZM1Z(2U)
CPU:ZIntelZXeonZPhiZ7250Z(KNLZ68Zcore,Z1.4ZGHz)
×8,208
Mem:Z16ZGBZ(MCDRAM,Z
490ZGB/sec,Zeffective)Z+Z96ZGBZ(DDR492400,Z115.2ZGB/sec)

Research(&(Education
The Oakforest9PACS is offered to researchers in Japanand their international collaborators through varioustypes of programs operated by HPCI under MEXT, andby original supercomputer resource sharing programsby two universities.
It is expected to contribute to dramatic development ofnew frontiers of various field of studies. The Oakforest9
PACS will be also utilized for education and training ofstudents and young researchers. We will continue tomake further social contributions through operations ofthe Oakforest9PACS.

12Zof
768ZportZDirectorZSwitch(SourceZbyZIntel)

22
Uplink:Z24

Downlink:Z24
1 .Z.Z.

362Zof48ZportZEdgeZSwitch
24.Z.Z.
25
48.Z.Z.
49
72

TotalZpeakZperformance 25ZPFLOPS
TotalZnumberZofZ
8,208
compute nodes
Power consumption
4.2ZMWZ(includingZcooling)#ZofZracks
102
CoolingZ ComputeZ TypeWarm9waterZcoolingsystem NodeDirect coolingZ(CPU)RearZdoorZcoolingZZ(exceptZCPU)Z
Facility CoolingZtowerZ&ZChiller
Others TypeAirZcooling
Facility PAC

Omni9PathZArchitectureZ(100ZGbps),ZZFull9bisectionZBWZFat9treeZDDNZIME14KXZx251560(GB/sFileZCacheZSystemZ
940TB
Lustre Filesystem
DDNZES14KXZx10500(GB/s

ParallelZFileZSystem
26.2(PB

LoginZNodeZx20
LoginZ
LoginZ
LoginZ
LoginZ
LoginZ
LoginZ
nodeZ
nodeZ
nodeZ
nodeZ
nodeZ
nodeZ

U.ZTokyousers

LoginZ
LoginZ
LoginZ
LoginZ
LoginZ
LoginZ
nodeZ
nodeZ
nodeZ
nodeZ
nodeZ
nodeZ

U.ZTsukuba
users

Joint(Center(for(Advanced(HPC(|(http://jcahpc.jp/