Joint Center for Advanced High Performance Computing

Overview

System Configuration

JCAHPC (Joint Center for Advanced HPC), which is a cooperative organization by the University of Tokyo and Universityof Tsukuba for joint procurement and operation of the largest scale of supercomputer in Japan, introduced a newsupercomputer system “Oakforest-PACS” with 25 PFLOPS peak performance and started its operation from December
1st, 2016. The Oakforest-PACS system is ranked at #6 in TOP500 List of November 2016 with 13.55 PFLOPS of Linpack
performance, and also recognized as Japan's fastest supercomputer. The system is installed at the Kashiwa ResearchComplex II building in the Kashiwa-no-Ha campus, the University of Tokyo.
The Oakforest-PACS system has 8,208 compute nodes, each of which consists of the latest version of Intel Xeon Phi
processor (code name: Knights Landing), and Intel Omni-Path Architecture as the high performance interconnect. The
Oakforest-PACS system is the largest cluster solution with Knights Landing processor as well as also the largest
configuration with Omni-Path Architecture in the world. The system is integrated by Fujitsu Co. Ltd, and its PRIMERGY
server is employed as each of compute node. Additionally, the system employs the Lustre shared files system (capacity:26 PB), and IME (fast file cache system, 940 TB), both of which are provided by DataDirect Network (DDN).
All the computation nodes and servers including login nodes, Lustre servers and IME servers are connected by a fullbisection bandwidth of Fat-Tree interconnection network with Intel Omni-Path Architecture to provide highly flexible job
allocation over the nodes and high performance file access.
TOP 500 #6 (#1 in Japan), HPCG #3 (#2), Green 500 #6 (#2)
@Nov. 2016
IO 500 #1 @Nov. 2017, Jun. 2018
IO-500 BW #1 @Jun. 2019

Compute Nodes: 25 PFlops
Fujitsu PRIMERGY CX1640 M1
x 8 node inside CX600 M1 (2U)
CPU: Intel Xeon Phi 7250
(KNL 68 core, 1.4 GHz)×8,208
Mem: 16 GB (MCDRAM,490 GB/sec, effective)+ 96 GB (DDR4-2400, 115.2 GB/sec)

Research & Education
The Oakforest-PACS is offered to researchers in Japanand their international collaborators through varioustypes of programs operated by HPCI under MEXT, andby original supercomputer resource sharing programsby two universities.
It is expected to contribute to dramatic development ofnew frontiers of various field of studies. The OakforestPACS will be also utilized for education and training ofstudents and young researchers. We will continue tomake further social contributions through operations ofthe Oakforest-PACS.

12 of768 port Director Switch(Source by Intel)

22
Uplink: 24

Downlink: 24
1 ...

362 of48 port Edge Switch
24.
.
.
2548.
.
.
49
72

Total peak performance 25 PFLOPS
Total number of8,208
compute nodes
Power consumption
4.2 MW (including cooling)# of racks
102
Cooling Compute Type
Warm-water coolingsystem NodeDirect cooling (CPU)
Rear door cooling (except CPU)
Facility Cooling tower & Chiller
Others TypeAir cooling
Facility PAC

Omni-Path Architecture (100 Gbps), Full-bisection BW Fat-tree

DDN IME14KX x25

1560 GB/s

File Cache System
940TB
Lustre Filesystem
DDN ES14KX x10

500 GB/s

Parallel File System
26.2 PB

Login Node x20
Login
Login
Login
Login
Login
Login
node
node
node
node
node
node

U. Tokyo
users

Login
Login
Login
Login
Login
Login
node
node
node
node
node
node

U. Tsukuba
users

Joint Center for Advanced HPC | http://jcahpc.jp/