Aims ofJAXA supercomputer generation 3-Purpose of introducing "TOKI " and early performance evaluation-

Japan Aerospace Exploration Agency
Manager for Supercomputer Division/Manager for Numerical Simulation Research Unit

FUJITA,Naoyuki
Feb./4th/2021

TOKI Overview

P.1

Contents
Purpose of introducing "TOKI"JSS3 overview
Early performance evaluation
Conclusion

Feb./4th/2021

TOKI Overview

P.2

■Purpose of introducing "TOKI"■Three pillars■Background••

Computational fluid dynamics(CFD), which has evolved withsupercomputers, has moved from promotion to practical use.
The field of supercomputer use is broadened not only to CFD
but also data processing such as earth observation dataprocessing and AI platform.

I.

Infrastructure of Numerical Simulations toempower our international competitiveness in theaerospace.
II. Data Center as a large-scale data analysis
platform.
III. Research and Development Platform foracceptance of emerging needs.
Feb./4th/2021

TOKI Overview

P.3

Contents
Purpose of introducing "TOKI"JSS3 overview
Early performance evaluation
Conclusion

Feb./4th/2021

TOKI Overview

P.4

■JSS3 overview

Feb./4th/2021

Start operation
December 1st, 2020

TOKI Overview

P.5

■JSS3 overview■Thoughts in the name JSS3:Jaxa Supercomputer System generation 3
 TOKI:TOkyo and ibaraKI、 ”TOKI(Time and space togetherwith SORA)”、”solution”、”Japanese bird toki(crestedibis)
 SORA: Supercomputer for earth Observation, Rockets,
and Aeronautics
JSS3
 RURI: all-RoUnd Role Infrastructure ST : STandard
TOKI
 GP : GPgpu
SORA
RURIFS XM : eXtra large Memory
STGP LM : Large Memory
LIXMLM FS : File System LI : LogIn -Txx : -Tsukuba xx
J-SPACEHigh speed-Txx

Feb./4th/2021

TOKI Overview

P.6

■JSS3 overview■TOKI-SORA

Supercomputer forearth Observation, Rockets, and Aeronautics

Chofu installation

JSS3TOKI

SORA
RURI

ST
GP

Tsukuba installation

TRURI

XM
LM

LI

J-SPACE

Feb./4th/2021

FS

TLI

TST

TGP
TLM

TFS

TOKI Overview

P.7

■JSS3 overview■TOKI-SORA

Supercomputer forearth Observation, Rockets, and Aeronautics

Configuration per node
CPU
Processor

Main memory

Interconnect※Design subject to change.

A64FX(Armv8.2-A + SVE)

Processors

1

Theoreticaloperationperformance

3.3792TFLOPS

TypeHBM2

Capacity

32GiB

Memorybandwidth

1,024GB/s

TypeTofu interconnect DTheoreticalperformanceMax 40.8GB/s × two way

Configuration per rack
Computation nodes

384 nodes

HPC system overall configuration- Theoretical peak operation performance 19.4 PFLOPS
3.3792TFLOPS/node × 384 nodes/rack × 15 racks≒ 19.464 PFLOPs- Main memory total capacity 180TiB
32GiB /node × 5,760 nodes＝ 180 TiB
Feb./4th/2021

TOKI Overview

P.8

■JSS3 overview■TOKI-RURI

all-RoUnd Role Infrastructure

Chofu installationJSS3

TOKI

SORA
RURI

ST
GP

Tsukuba installation

TRURI

XM
LM

LI

J-SPACE

Feb./4th/2021

FS

TLI

TST

TGP
TLM

TFS

TOKI Overview

P.9

■JSS3 overview■TOKI-ST
PRIMERGY RX2540 M5x375

STandard
Configuration per node
CPU
Processor

Processors
Theoretical
2.9952TFLOPS (2.6GHz × 18 core × 32 instructions
operation× 2CPU)
performanceGPU
NVIDIA Quadro P4000(1,792 CUDA core 8GiBTypememory)GPUs
1Main memory Type
16GiB DDR4 2933MHz RDIMM (ECC)
Capacity
192GiB (16GiB × 12)
Memory281.5GB/s (2933MHz × 8byte × 12)
bandwidthInternal disk Type
M.2 Flash module
Effective240GB
capacity
Interconnect TypeInfiniBand HDR100 × 1 port
Theoretical
100Gbps (two way)
performance
Network interface
1000Base-T×2 port
iRMC (CPU, Memory, HDD, Card, Fan, Power,
Enclosure monitoring
Temperature, Voltage, etc...)Height2U
PowerFeb./4th/2021

Intel Xeon Gold 6240 (2.6GHz/18 core)
2800W×2 (80PLUS PLATINUM)※Redundant、Active replacement possible

TOKI Overview

P.10

■JSS3 overview■TOKI-GP(1/2)
PRIMERGY CX2570 M5(Water cooling)

x32
PRIMERGY CX400 M4
Chassis(Water cooling)x16

Feb./4th/2021

GPgpu

Configuration per PRIMERGY CX2570 M5
CPUProcessor
Intel Xeon Gold 6240 (2.6GHz/18 core)
Processors
2
Theoretical
2.9952TFLOPS (2.6GHz×18 core×32
operationinstruction×2 CPU)
performanceGPU
SXM2 NVIDIA Tesla V100 (5,120 CUDA
Type
cores,32GiB HBM2)
GPUs
4
Double precision 31.2TFLOPS (7.8TFLOPS×4)
TheoreticalSingle precision 62.8TFLOPS
operation(15.7TFLOPS×4)
Performance
Tensor 500TFLOPS (125TFLOPS×41, half
precision and single precision mixed)
Main
Type
32GiB DDR4 2933MHz RDIMM (ECC)
memory Capacity
384GiB (32GiB×12)
Memory bandwidth 281.5GB/s (2933 MHz×8 Byte×12 channel)
Internal Type
M.2 Flash module
disk
Effective capacity
256GBInterType
InfiniBand EDR×2 port
connect Theoretical200Gbps (two way)
performance
Network interface
1000BASE-T×1port
Enclosure monitoring
iRMC (CPU, Memory, HDD, Card, Fan, Power,Temperature, Voltage etc...)Configuration per PRIMERGY CX400 M4
Installed server node
PRIMERGY CX2570 M5×2 node
Chassis height
2U
Power
Power unit (2400W)×2
TOKI Overview

P.11

■JSS3 overview■TOKI-GP(2/2)

GPgpuGPUSXM2 TrayGPU
PCI-Express3.0 (x16)GPUInterconnect network
between GPU(NVLink)
300GB/s(One way 25GB/s(twoway50GB/s×6)

PCI-Express3.0 (x16)

SwitchGPUSwitch

GPU

PCI-Express3.0 (x16)4-GPU-NVLink
Interconnect network
between core
10.4GT/s×3

DDR4
Memory32GiB

Memory32GiB

Memory32GiB

UPIMemory32GiB
Memory32GiB

DDR4CPU

Memory32GiB
Memory32GiB

CPUMemory32GiB

Memory32GiB

Memory32GiB
Memory32GiB

Memory32GiB

PCI-Express3.0 (x16)

M.2 Flash 256GB

140.7GB/s

ChipsetEthernet NIC(1Gbps x1)

Ethernet

Feb./4th/2021

iRMC(Server Monitoring

InfiniBand HCA
Card(100Gbps)

InfiniBand HCA
Card(100Gbps)

Ethernet

Interconnect
NetworkInterconnect
NetworkProcessor)

TOKI Overview

P.12

■JSS3 overview■TOKI-XM/LM(1/2)
Configuration per node
CPUProcessor
Processors

PRIMERGY RX2540 M5x2
XM

GPU

Main memory

eXtra large Memory / Large Memory
Intel Xeon Gold 6240L (2.6GHz/18 core)
2 processors

Theoreticaloperation
performance

2.9952TFLOPS (2.6GHz × 18 core × 32 instruction × 2CPU)Type

NVIDIA Quadro P4000(1,792 CUDA core, 8GiB memory)

GPUs
TypeCapacity

116GiB 64GiB DDR4 2933MHz LRDIMM (ECC)
192GiB 768GiB (64GiB × 12)

Memory bandwidth 255.9GB/s (2666MHz × 8byte × 12)

PRIMERGY RX2540 M5x7LM

Feb./4th/2021

NonvolatilememoryTypeIntel Optane DC Persistent･Memory

1.5TiB 6TiB (512GiB×12)
Capacity
InternalType
M.2 Flash moduledisk
Capacity
240GB
Interconnect TypeInfiniBand HDR(100Gbps) × 2 port
Theoretical
200Gbps (two way)
performance
Network interface
1000Base-T×2 port

TOKI Overview

P.13

■JSS3 overview■TOKI-XM/LM(2/2)

DCPMM 512GiB

CPU

UPI

Memory64GiB

DIMM×6・・・

Memory64GiBCPUDCPMM 512GiB

DCPMM×6・・・

GPU

Memory64GiBInterconnect
network
between core
10.4GT/s x 2・・・

DCPMM×6・・・

DIMM×6

Memory64GiB

eXtra large Memory / Large Memory

DCPMM 512GiB

DCPMM 512GiB

PCI-Express3.0 (x16)InfiniBand HCA
Card(100Gbps)

Interconnect Network

PCI-Express3.0 (x16)

M.2 Flash 240GB

ChipsetiRMC(Server
monitoring processer)

Ethernet 2port NIC(1Gbps x2)

Ethernet NIC(1Gbps)Ethernet

Ethernet

32GB/s(2GB/s×16)
InfiniBand HCA
Card(100Gbps)

Interconnect Network

Above is XM

Feb./4th/2021

TOKI Overview

P.14

■JSS3 overview■TOKI-LI、TOKI-FS
Chofu installation

JSS3

TOKISORARURISTGP

Tsukuba installation

TRURIXMLM

LI

J-SPACE

Feb./4th/2021

FS

TLI

TST

TGP
TLM

TFS

TOKI Overview

P.15

■JSS3 overview■TOKI-LI
PRIMERGY RX2540 M5

LogIn
Configuration per unit
CPUProcessor

Intel Xeon Gold 6240 (2.6 GHz/18 core)
2Processors
Theoretical
2.9952TFLOPS (2.6GHz × 18 core × 32 instruction ×operation2 CPU)
performanceGPU
NVIDIA Quadro P4000(1792 CUDA core, 8GiBメ
Typex14
memory ,GDDR5)
GPUs
1
Main memory Type
32GiB DDR4 2933MHz RDIMM×12 (ECC)
Capacity
384GiB (32GiB × 12)
Login system overall configuration
Memory281.5GB/s (2933MHz × 8byte × 12 / 1000)- Theoretical peal operation performance
bandwidth47.9 PFLOPSInternal disk Type
600GB SAS HDD (15krpm) × 2 (RAID1)
2.9952TFLOPS / node × 16 node
Effective600GB≒ 47.9 PFLOPs
capacity
Interconnect Type- Main memory total capacity 180TiB
InfiniBand HDR (100Gbps × 2枚 (2ポート)
384GiB / node × 16 node
Theoretical
200Gbps (two way)
＝ 6.0 TiB
performanceFeb./4th/2021

Network interface

10GBase-T×2 port, 1000BASE-T×2 port

PowerPower unit (800W)×2

TOKI Overview

P.16

■JSS3 overview■TOKI-FS
FEFS Client

TOKI-SORA

File System

TOKI-LI

TOKI-RURI

InfiniBand switch

FEFS Server・・・

System management nodeCalculation cluster management node

InfiniBand switch

OSSOSS

Metadata server (MDS)
PRIMERGY RX2540 M5 x 4台

[Per unit]
CPU: Intel Xeon Gold 5222
(4 core/3.8GHz) x 2
Memory: 384GiB (32GiB x 12)
HDD: SAS 900GB(10krpm) x 2 (RAID1)Interconnect: InfiniBand HDR100 x 2
SAN: Fibre Channel (16Gbps) x 2

Data storage (OSS/OST)Home
DDN SFA7990XE × 1台
VM: OSS × 2
Interconnect: InfiniBand HDR100 x 4
HDD: NL-SAS 8TB(7.2krpm) x 87
(43 Drive x 2 DCR Pool + 1 HS)

SSD: 1.92TB x 68
RAID1+0 configuration(4D+4M)x8+4HS

Above configuration x 1 set

OSSOSS

Metadata storage (MDT)ETERNUS AF250 S3 x 1

Above configuration x 1 set

Replication server

OSS

OSS

Data storage（OSS/OST）
Data(SSD)
DDN SFA200NVXE × 1台
VM: OSS × 2
Interconnect: InfiniBand HDR100 x 4
SSD: 15.36TB × 23
(22 Drive × 1DCR Pool + 1 HS)

Data storage (OSS/OST)
Data(HDD)DDN SFA7990XE × 1
VM: OSS × 2
DDN SS9012 × 4
HDD: NL-SAS 14TB(7.2krpm) x 340(42 Drive x 8 DCR Pool + 4 HS)

Above configuration x 12 set

Above configuration x 39 set

Protected by UPS

Feb./4th/2021

TOKI Overview

P.17

■JSS3 overview■TOKI-Txx-Tsukuba xx

Chofu installation

JSS3

TOKI

SORA
RURI

ST
GP

Tsukuba installation

TRURI

XM
LM

LI

J-SPACE

Feb./4th/2021

FS

TLI

TST

TGP
TLM

TFS

TOKI Overview

P.18

■JSS3 overview■TOKI-Txx-Tsukuba xx

Non-stop supercomputer

High speed

Feb./4th/2021

TOKI Overview

P.19

Contents
Purpose of introducing "TOKI"JSS3 overview
Early performance evaluation
Conclusion

Feb./4th/2021

TOKI Overview

P.20

■Early performance evaluation■ JAXA Applications Benchmark at the time of introduction
3.5
3.0■ Elapsed time
■ Electric power

2.5
2.0

Full marks

1.5
1.0
0.5
0.0

Feb./4th/2021

HINOCAUPACS-Lite

P-FLOW

TOKI Overview

LS-FLOWFaSTAR

P.21

■Early performance evaluation■TOP500
 Measurement condition
 Nodes used: 5,760node(All TOKI-SORA nodes)
 Total core: 276,480 (5,760node×(4 process×12 thread/node)) Theoretical operation performance: 19.4642 PFLOPS
 Node topology: 20x18x16:torus

 Runtime parameter N(Problem scale)=4,208,640、P=96、Q=240

 Measurement result
 Performance: 16.592 PFLOPS
 Execution efficiency: 85.24%
 Rank: No.19 (In Japan rank No.3)

Feb./4th/2021

TOKI Overview

P.22

■Early performance evaluation■HPCG
 Measurement condition
 Nodes used: 5,760node(All TOKI-SORA nodes)
 Total core: 276,480 (5,760node×(4process×12thread/node)) Theoretical operation performance : 19.4642 PFLOPS
 Node topology: 20x18x16:torus

 Runtime parameter Local Domain Dimensions: nx=160, ny=160, nz=160
 Processor Dimensions: npx=40, npy=36, npz=16

 Measurement result
 Performance: 0.61422 PFLOPS
 Execution efficiency : 3.15%
 Rank: No.8 (In Japan rank No.2)

Feb./4th/2021

TOKI Overview

P.23

■Early performance evaluation■Graph500
 Measurement condition
 Run node: 5,760node(All TOKI-SORA nodes)
 Total core: 276,480 (5,760node×(4process×12thread/node)) Theoretical operation performance : 19.4642 PFLOPS
 Node topology: 60x96

 Runtime parameter SCALE(Measurement scale): 36

 Measurement result
 Performance: 10,813 GTEPS Rank: No.3 (In Japan rank No.2)

Feb./4th/2021

TOKI Overview

P.24

■Early performance evaluation■IO500(Will be measured)Feb./4th/2021

TOKI Overview

P.25

Contents
Purpose of introducing "TOKI"JSS3 overview
Early performance evaluation
Conclusion

Feb./4th/2021

TOKI Overview

P.26

■Conclusion(1/2)
Purpose of introducing "TOKI"JSS3 overview
Early performance evaluation

Feb./4th/2021

TOKI Overview

P.27

■Conclusion(2/2)
 Purpose of introducing "TOKI" JSS3 overview
 Early performance evaluation

 JSS3 from now on～New utilization～
 Random IO performance of TOKI-FS
 Full-scale run of processing that requires IOPS Virtualization technology of TOKI-SORA/RURI Acceleration of cooperation with external computer environment

 Utilizing GPGPU of TOKI-GP
 JAXA’s first large GPGPU platform Operation performance enhancement→Large-scalechallenge Large-scale 3D visualization
 DNS of droplet evaporation
 LES of rocket engine combustor
Feb./4th/2021

TOKI Overview

P.28

Thank you very much.
JAXA supercomputer Website

https://www.jss.jaxa.jp/en/Feb./4th/2021

TOKI Overview

P.29