Basic Specification of Oakforest-PACS
Joint Center for Advanced HPC (JCAHPC)
by Information Technology Center, the University of Tokyo
and Center for Computational Sciences, University of Tsukuba

Oakforest-PACS in JCAHPC• Very tight relationship andcollaboration with two universities• For primary supercomputerresources, uniform specification tosingle shared system• Each university is financially
responsible to introduce the machine
and its operation• First attempt in Japan• Unified procurement toward single system withlargest scale in Japan

Administrative Council
Chairman:Professor aHiroshi Nakamura

AdvancedComputational ScienceSupport Technology
Division

Kashiwa Campus,The University of Tokyo

2017/03/10

Advanced HPC
Facility(Supercomputer)

BudgetBudgetCenter for Computational Sciences
University of Tsukuba

Information Technology Center

The University of Tokyo

OFP specification, JCAHPCOperation
Technology
Division

Public Relations
and Planning
Division

Operation

Operation

2Specification of Oakforest-PACS system
Total peak performance

25 PFLOPS

Total number of compute nodes

8,208

Compute node ProductFujitsu PRIMERGY CX600 M1 (2U) + CX1640 M1 x
8node

Processor

MemoryIntel® Xeon Phi™ 7250
(Code name: Knights Landing),
68 cores, 1.4 GHz
High BW 16 GB, 490 GB/sec (MCDRAM, effective rate)
Low BW

Interconnect96 GB, 115.2 GB/sec (peak rate)

ProductIntel® Omni-Path Architecture

Link speed

100 Gbps

Topology

Fat-tree with (completely) full-bisection bandwidth

2017/03/10

OFP specification, JCAHPC

3

Full bisection bandwidth Fat-tree byIntel® Omni-Path Architecture
12 of768 port Director Switch(Source by Intel)

22Uplink: 24

362 of48 port Edge Switch

Downlink: 24
1...

24

25...

48

49...

72

Firstly, to reduce switches&cables, we considered :• All the nodes into subgroups are connected with FBB Fat-tree• Subgroups are connected with each other with >20% of FBB
But, HW quantity is not so different from globally FBB, and globally FBB is preferred
for flexible job management.

2017/03/10

OFP specification, JCAHPC

4

Specification of Oakforest-PACS system (Cont’d)ParallelType
File System Total Capacity
Product

Lustre File System
26.2 PB
DataDirect Networks SFA14KE

Aggregate BW 500 GB/sec
File Cache
SystemType

Burst Buffer, Infinite Memory Engine (by DDN)

Total capacity

940 TB (NVMe SSD, including parity data by erasure coding)

Product

DataDirect Networks IME14K

Aggregate BW 1,560 GB/sec
Power consumption

4.2 MW (including cooling)# of racks

102

2017/03/10

OFP specification, JCAHPC

5

Software of Oakforest-PACS system
Compute node

Login nodeOS

CentOS 7, McKernel

Red Hat Enterprise Linux 7Compilergcc, Intel compiler (C, C++, Fortran)

MPIIntel MPI, MVAPICH2LibraryIntel MKLLAPACK, FFTW, SuperLU, PETSc, METIS, Scotch, ScaLAPACK, GNU Scientific Library, NetCDF, Parallel netCDF,Xabclib, ppOpen-HPC, ppOpen-AT, MassiveThreads

Application

mpijava, XcalableMP, OpenFOAM, ABINIT-MP, PHASE system, FrontFlow/blue, FrontISTR, REVOCAP, OpenMX,xTAPP, AkaiKKR, MODYLAS, ALPS, feram, GROMACS, BLAST, R packages, Bioconductor, BioPerl, BioRuby

Distributed FS

Globus Toolkit, Gfarm

Job Scheduler

Fujitsu Technical Computing Suite

Debugger

Allinea DDTProfilerIntel VTune Amplifier, Trace Analyzer & Collector

2017/03/10

OFP specification, JCAHPC

6

Software of Oakforest-PACS• OS: Red Hat Enterprise Linux (Login nodes),CentOS or McKernel (Compute nodes, dynamically switchable)• McKernel: OS for many-core CPU developed by RIKEN AICS
• Ultra-lightweight OS compared with Linux, no background noise to user program• Will be deployed to post-K computer• Compiler: GCC, Intel Compiler, XcalableMP• XcalableMP: Parallel programming language developed by RIKEN AICS and University ofTsukuba• Easy to develop high-performance parallel application by adding directives to originalcode written by C or Fortran• Application: Open-source software• OpenFOAM, ABINIT-MP, PHASE system, FrontFlow/blue, and so on
2017/03/10

OFP specification, JCAHPC

7

Photo of computation nodeComputation node (Fujitsu PRIMERGY CX1640 M1)
with single chip Intel Xeon Phi (Knights Landing, 3+TFLOPS)
and Intel Omni-Path Architecture card (100Gbps)
2017/03/10

OFP specification, JCAHPC

Chassis with 8 nodes, 2U size
(Fujitsu PRIMERGY CX600 M1)

8

Machine location: Kashiwa Campus of U. Tokyo

U. Tsukuba

KashiwaCampusof U. Tokyo

Hongo Campus of U. Tokyo

2017/03/10

OFP specification, JCAHPC9