Joint&Center&for&Advanced&HPC&|&http://jcahpc.jp/

OakforestIPACS
Overview
We decided to introduce a new supercomputer system “Oakforest5PACS” (aka. Post T2K system) with 25 PFLOPS peakperformance. We will start its full operation on December 1st, 2016. The Oakforest5PACS system will be the fastest
supercomputer system in Japan at that time. This system will be installed at the Kashiwa Research Complex II building in theKashiwa5no5Ha (Oakleaf) campus, the University of Tokyo.
The Oakforest5PACS system has 8,208 compute nodes, each of which consists of next5generation Intel Xeon Phi processor(code name: Knights Landing, and self5bootable), and Intel Omni5Path Architecture as a high5performance interconnect,
which is a brand5new fabric developed by Intel. This is the first large5scale system with such a processor in Japan. The system
is integrated by Fujitsu and its PRIMERGY server is employed as each of compute node. Additionally, the system employs theshared files system (capacity: 26 PB), and the fast file cache system (940 TB), both of which are provided by DataDirect
Network (DDN).
Peak performance of the Oakforest5PACS is 25 PFLOPS and the total memory capacity is more than 900 TB. All computenodes and servers of file systems are connected by fat5tree topology based on Intel OPA, which provides full bisection
bandwidth. Therefore, flexible and efficient utilization and operation of compute nodes and file systems is available.
Moreover, the file cache system is equipped with SSD’s and it is suitable for such applications that require higher file I/O
performance.
ComputedNodes:d25 PFlops
FujitsudPRIMERGYdCX1640dM1d
xd8dnodedinsidedCX600dM1d(2U)

Research&&&Education
The Oakforest5PACS will be offered to researchers in Japan andtheir international collaborators through various types ofprograms operated by HPCI, by MEXT’s Joint Usage/ResearchCenters, and by each of us under original supercomputerresource sharing programs. It is expected to contribute todramatic development of new frontiers of various field ofstudies. The Oakforest5PACS will be also utilized for educationand training of students and young researchers. We willcontinue to make further social contributions throughoperations of the Oakforest5PACS.

CPU:dInteldXeondPhid7250d(KNL 68dcore,d1.4dGHz)
×8,208
Mem:d16dGBd(MCDRAM,d490dGB/sec,deffective)d+d96dGBd(DDR452400,d115.2dGB/sec)

Omni5PathdArchitectured(100dGbps),ddFull5bisectiondBWdFat5treed

DDNdIME14Kdx25

1560&GB/s

FiledCachedSystemd
940TB
Lustre Filesystem 500&GB/s
DDNdSFA14KEdx10

ParalleldFiledSystem
26.2&PB

System&Configuration
12dof
768dportdDirectordSwitch(SourcedbydIntel)

22
Uplink:d24

Downlink:d24
1 .d.d.

362dof48dportdEdgedSwitch
2425 .d.d. 48

49 .d.d. 72

LogindNodedx20
LogindLogindLogindLogindLogindLogind
noded
noded
noded
noded
noded
noded

U.dTokyousersLogind
LogindLogindLogindLogindLogind
noded
noded
noded
noded
noded
noded

U.dTsukuba
users

Totaldpeakdperformance 25dPFLOPS
Totaldnumberdofd
8,208
compute nodes
Power consumption
4.2dMWd(includingdcooling)#dofdracks
102
Coolingd Computed TypeWarm5waterdcoolingsystem Node
Direct coolingd(CPU)
Rearddoordcoolingdd(exceptdCPU)d
Facility Coolingdtowerd&dChiller
Others TypeAirdcooling
Facility PAC
Joint Center for Advanced High Performance Computing