Joint Center for Advanced High Performance Computing

Overview
JCAHPC (Joint Center for Advanced HPC), which is a cooperative organization by the University of Tokyo and University ofTsukuba for joint procurement and operation of the largest scale of supercomputer in Japan, introduced a newsupercomputer system “Oakforest-PACS” with 25 PFLOPS peak performance and started its operation from December 1st,2016. The Oakforest-PACS system is ranked at #6 in TOP500 List of November 2016 with 13.55 PFLOPS of Linpack
performance, and also recognized as Japan's fastest supercomputer. The system is installed at the Kashiwa ResearchComplex II building in the Kashiwa-no-Ha campus, the University of Tokyo.
The Oakforest-PACS system has 8,208 compute nodes, each of which consists of the latest version of Intel Xeon Phiprocessor (code name: Knights Landing), and Intel Omni-Path Architecture as the high performance interconnect. The
Oakforest-PACS system is the largest cluster solution with Knights Landing processor as well as also the largest configuration
with Omni-Path Architecture in the world. The system is integrated by Fujitsu Co. Ltd, and its PRIMERGY server is employed
as each of compute node. Additionally, the system employs the Lustre shared files system (capacity: 26 PB), and IME (fast filecache system, 940 TB), both of which are provided by DataDirect Network (DDN).
All the computation nodes and servers including login nodes, Lustre servers and IME servers are connected by a fullbisection bandwidth of Fat-Tree interconnection network with Intel Omni-Path Architecture to provide highly flexible job
allocation over the nodes and high performance file access.

Research & Education

Compute Nodes: 25 PFlops

The Oakforest-PACS is offered to researchers in Japan andtheir international collaborators through various types ofprograms operated by HPCI under MEXT, and by original
supercomputer resource sharing programs by twouniversities.
It is expected to contribute to dramatic development of new
frontiers of various field of studies. The Oakforest-PACS willbe also utilized for education and training of students andyoung researchers. We will continue to make further social
contributions through operations of the Oakforest-PACS.

Fujitsu PRIMERGY CX1640 M1
x 8 node inside CX600 M1 (2U)
CPU: Intel Xeon Phi 7250
(KNL 68 core, 1.4 GHz)×8,208
Mem: 16 GB (MCDRAM,490 GB/sec, effective)+ 96 GB (DDR4-2400, 115.2 GB/sec)

Omni-Path Architecture (100 Gbps), Full-bisection BW Fat-tree

DDN IME14KX x25

System Configuration

1560 GB/s

File Cache System
940TB

12 of
768 port Director Switch(Source by Intel)

Lustre Filesystem
DDN ES14KX x10

500 GB/s

Parallel File System
26.2 PB

Login
Login
Login
Login
Login
Login
node
node
node
node
node
node

U. Tokyo
users

Login
Login
Login
Login
Login
Login
node
node
node
node
node
node

U. Tsukuba
users

22Uplink: 24

Downlink: 24
1 ...

Login Node x20

362 of48 port Edge Switch24.
.
.
25
48.
.
.
49
72

Total peak performance 25 PFLOPS
Total number of8,208
compute nodes
Power consumption
4.2 MW (including cooling)# of racks
102
Cooling Compute Type
Warm-water coolingsystem Node
Direct cooling (CPU)
Rear door cooling (except CPU)
Facility Cooling tower & Chiller
Others TypeAir cooling
Facility PAC

Joint Center for Advanced HPC | http://jcahpc.jp/