スーパーコンピュータ『富岳』の概要
理化学研究所 計算科学研究センター
プロジェクトリーダ
⽯川 裕

PCCCワークショップ
2020/02/21

スーパーコンピュータ「富岳」


2018年夏、富岳の試作機完成



2019年12⽉2⽇（⽉）富⼠通⼯場から出荷開始



富岳の約10ラックで「京」と同じピーク性能（「京」は864ラック）。電⼒は 1/20

10 racks =写真は2ラック分

FugakuK

A64FXCPU Architecture (Armv8.2‐A SVE+Fujitsu Extension)
Cores

Node

Peak DP performance
Main Memory
Peak Memory Bandwidth
Peak Network Performance
Nodes

Rack

Peak DP performance

Process Technology
2020/02/21

48
3.0720 TF(3.3792 TF)
32 GiB
1024 GB/s
40.8 GB/s
384
1.2/1.3 PF
7 nm FinFET

SPARC64 VIIIfx
80.128 TF16 GiB
64 GB/s
20 GB/s
102< 0.013PF
45 nm

RIKEN Center for Computational Science2CPU A64FX
Courtesy of FUJITSU LIMITED

Architecture Armv8.2‐A SVE (512 bit SIMD)
Core

48 cores for compute and 2/4 for OS activities
DP: 3.0720 TF (3.3792 TF)

Cache L1 64 KiB
Cache L2 CMG(NUMA): 8 MiB, 16way
Memory HBM2 32 GiB, 1024 GB/s
Interconnect TofuD (28 Gbps x 2 lane x 10 port)
I/O PCIe Gen3 x 16 lane
Technology 7nm FinFET

2020/02/21

RIKEN Center for Computational Science

3TofuD Interconnect

2 lanes x 10 portsTNR(Tofu Network Router)

TNI5

TNI4

TNI3

TNI2

TNI1

TNI0

40.8 GB/s(6.8 GB/s x 6)

TNI: Tofu Network Interface (RDMA engine)• 6 RDMA Engines• Hardware barrier support• Network operation offloading capability
8B Put latency
1MiB Put throughput

0.49 – 0.54 usec
6.35 GB/s

rf. Yuichiro Ajima, et al. , “The Tofu Interconnect D,” IEEE Cluster 2018, 2018.

2020/02/21

RIKEN Center for Computational Science4

スーパーコンピュータ「富岳」
FugakuK

A64FX
CPU Architecture (Armv8.2‐A SVE+Fujitsu Extension)
CoresNodePeak DP performance
Main Memory
Peak Memory Bandwidth
Peak Network Performance
Nodes

Rack

Peak DP performance

Process Technology

48
3.0720 TF(3.3792 TF)
32 GiB
1024 GB/s
40.8 GB/s
384
1.2/1.3 PF
7 nm FinFET

総ノード数︓150kノード以上



第1階層ストレージ



SPARC64 VIIIfx
80.128 TF
16 GiB
64 GB/s
20 GB/s
102< 0.013PF
45 nm

16計算ノード毎に1計算兼ストレージノード 約1.6 TB

提供サービス
第２階層ストレージキャッシュ

計算ノード向けローカルファイルシステム



ジョブ向け共有ファイルシステム

第2階層ストレージLustreベースグローバルファイルシステム 約150PB

2020/02/21

RIKEN Center for Computational Science

写真は2ラック分

Pre/Post処理


⼤容量メモリログインノード（約6TB)



可視化⽤ノード5外部ネットワーク 接続

Frontend Network
40 Gbps x 140 Gbps x 1

DC（L3）

SINET5 (100Gbps)

File Transfer Network
40 Gbps x 4
40 Gbps x 4

HPCIStorage

40 Gbps x 6

次期SINETにも対応できるような機器構成としている

2020/02/21

RIKEN Center for Computational Science6

システムソフトウェアスタック

AI Platform (DL4Fugaku)
RIKEN: Chainer, PyTorch, TensorFlow, …

Batch Job and Management
SystemMath Libraries
Fujitsu: BLAS, LAPACK, ScaLAPACK, SSL II
RIKEN: EigenEXA, KMATH_FFT3D, Batched BLAS,,,,

Hierarchical File System

Compiler and Script Languages
Fortran, C/C++, OpenMP, Java, python, …
Tuning and Debugging ToolsFujitsu: Profiler, Debugger, GUI
High-level Prog. Lang. Domain Spec. Lang.
XMPFDPS

Process/Thread
PIP

Open Source Management Tool
SpackRed Hat Enterprise Linux 8Communication
Fujitsu MPIRIKEN MPI

Low Level CommunicationuTofu, LLC

File I/O
DTF

Virtualization & Container
KVM, Singularity

File I/O for Hierarchical Storage
LLIO

Multi-Kernel System: Linux and light-weight kernel (McKernel)

2020/02/21

RIKEN Center for Computational Science7

想定プログラミングモデル
 MPI Programming Model 1 process/CMG (12 cores)1 MPI process with OpenMP parallelism or automaticparallezation runs on 12 cores.

 1 process/node

1 MPI process with OpenMP parallelism runs on 48cores

Two compute nodes are
implemented on one board

 Many processes with OpenMP parallelism / CMG

 PGAS Programming Model PGAS + OpenMP

1 PGAS process with OpenMP parallelism runs on 1node or 1 CMG

 1 PGAS process / core

 DSL (Domain Specific Language) FDPS
2020/02/21

RIKEN Center for Computational Science8

Spack Spack, developed at LLNL, is a package manager for

supercomputers enabling easy software installation

 3,000+ packages have been registered, e.g., abinit,lammps, openfoam, namd, blast, blat, …

 Build Verification Status of version: 6th September 2019

GCCClangFujitsu Compiler

x862479/3451(71.8%)

768/3574 (21.49%)*

N/A

aarch64

2387/3451(69.1%)

288/3451(60.04%)

2072/3451(60.04%)* Version 31st October 2019
 Pull Requests: Riken 66, Fujitsu 56
 Results of Verification of Compilation:https://postk-web.r-ccs.riken.jp/oss/public/
We have not tested execution of applications

2020/02/21

RIKEN Center for Computational Science9

Build Verification of Spack Recipes
 Spack version: 6th September 2019
 Verification Environment:

x86: RHEL8, Xeon E7-8867v3, Memory 768GB
aarch64: RHEL8, ThunderX2 CN9975-2000, Memory 128GB
GCCFujitsu Compiler
x862479/3451(71.8%)

N/A

aarch64

2387/3451(69.1%)

1835/2861(64.1%)(Suspended for system maintenance until mid Nov.)

2120/3242(65.3%)(previous trial with Spack ver. 18th June 2019)

 Pull Requests: Riken 47, Fujitsu 46
 Results of Verification of Compilation:https://postk-web.r-ccs.riken.jp/oss/public/
We have not tested execution of applications

2020/02/21

RIKEN Center for Computational Science10

富岳の性能 Predicted Performance of 9 Target Applications
Performance
Speedup over K

Application

Brief description

125x +

GENESISMD for proteins

8x +Genomon

Genome processing(Genome alignment)

45x +GAMERA

Earthquake simulator (FEM inunstructured & structured grid)

4. Meteorological andglobal environmental
prediction using big data

120x +NICAM+LETKF

Weather prediction system using
Big data (structured grid stencil &ensemble Kalman filter)

5. New technologies forenergy creation,
conversion / storage,
and use40x +NTChemMolecular electronic simulation(structure calculation)

6. Accelerateddevelopment ofinnovative clean energy
systems

35x +Adventure

Computational Mechanics
System for Large Scale Analysis
and Design (unstructured grid)

7. Creation of new
functional devices andhigh-performancematerials

30x +

RSDFT

Ab-initio simulation(density functional theory)

8. Development ofinnovative design andproduction processes

25x +

FFB

Large Eddy Simulation(unstructured grid)

25x +

LQCD

Lattice QCD simulation(structured grid Monte Carlo)

Area

Priority Issue

Health and longevity

1. Innovative computing

Disasterprevention andEnvironment
Energy issue
Industrial
competitivenessenhancementBasicscience

2020/02/21

As of 2019/05/14

infrastructure for drug
discovery
2. Personalized andpreventive medicine
using big data
3. Integrated simulation
systems induced byearthquake and tsunami

9. Elucidation of thefundamental laws andevolution of the universe

RIKEN Center for Computational Science11スケジュール
2021

2020
12

12

34

567891011

12

設置調整・共⽤開始準備
Early Access Program (共⽤前評価環境III)

2020/02/21

RIKEN Center for Computational Science12