Supercomputer Fugaku
Toshiyuki Shimizu

Feb. 18th, 2020

FUJITSU LIMITED
Copyright 2020 FUJITSU LIMITEDOutline◼ Fugaku project overview◼ Co-design◼ Approach◼ Design results◼ Performance & energy consumption evaluation◼ Green500◼ OSS apps
◼ Fugaku priority issues◼ Summary1

Copyright 2020 FUJITSU LIMITEDSupercomputer “Fugaku”, formerly known as Post-K

Focus

ApproachApplication performance

Co-design w/ application developers and Fujitsu-designed CPU core w/
high memory bandwidth utilizing HBM2

Power efficiency

Leading-edge Si-technology, Fujitsu's proven low power & highperformance logic design, and power-controlling knobs

UsabilityArm®v8-A ISA with Scalable Vector Extension (“SVE”), and Arm standard
Linux2

Copyright 2020 FUJITSU LIMITEDFugaku project schedule
20112012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022

Fugaku development & deliveryApps
Feasibility study Basic
review
designSelectapps

Architecture &sizing

Detailed design &Implementation

Manufacturing,
Installation
and Tuning

Generaloperation

Co-Design w/ apps groups

3

Copyright 2020 FUJITSU LIMITEDFugaku co-design◼ Co-design goals◼ Obtain the best performance, 100x apps performance than K computer, within power budget, 30-40MW• Design applications, compilers, libraries, and hardware◼ Approach◼ Estimate perf & power using apps info, performance counts of Fujitsu FX100, and cycle base simulator• Computation time: brief & precise estimation• Communication time: bandwidth and latency for communication w/ some attributes for communication patterns• I/O time:◼ Then, optimize apps/compilers etc. and resolve bottlenecks◼ Estimation of performance and power◼ Precise performance estimation for primary kernels• Make & run Fugaku objects on the Fugaku cycle base simulator◼ Brief performance estimation for other sections• Replace performance counts of FX100 w/ Fugaku params: # of inst. commit/cycle, wait cycles of barrier, inst. fetch, branch, fp exec, dataload/store wait cycles of L1D/L2, etc.

◼ Power estimation• DGEMM execution toggles on the emulator + estimation of memory and interconnect considering utilization + loss of convertors

4

Copyright 2020 FUJITSU LIMITEDCo-design iterations around year 2015
◼ Apply each of application kernels◼ Define/refine a set of architecture parameters◼ Implement/tune the kernel under the architecture parameters◼ Evaluate execution time using the estimation tools◼ Identify hardware bottlenecks and explore design space◼ Examples of architecture parameters◼ Frequency, # of arch regs, SIMD width, cache structure & size, # of cores…
◼ Memory, interconnect parameters◼ Implementation of instructions: i.e. Combined gather…
5Copyright 2020 FUJITSU LIMITEDA64FX CPU◼ Arm SVE, high performance and high efficiency◼ DP performance 2.7+ TFLOPS, >90%@DGEMM, (CPU freq=1.8/2.0/2.2 GHz)
◼ Memory BW
12x compute cores
1x assistant core1024 GB/s,PCleController

CMGNOCCMGC

ISA (Base, extension)

Armv8.2-A, SVE

Peak DP performance

2.7+ TFLOPSSIMD width

512-bit# of cores48 + 4

Memory capacity
Memory peak bandwidth

HBM2

HBM2CC

HBM2

HBM2

C

FUGAKU
operating cond.

A64FX

Tofu
Interface

CMG

CMG

>80%@STREAM Triad

CMG：Core Memory Group NOC：Network on Chip
632 GiB (HBM2 x4)1024 GB/s

PCIe

Gen3 16 lanes

High speed interconnect

TofuD integrated
Copyright 2020 FUJITSU LIMITEDA64FX leading-edge Si-technology◼ 8.786 billion transistors◼ 594 signal pins

HBM2 Interface◼ Broadcom SerDes, HBM I/O, and SRAMs

Core

Core

Core

Core

Core

L2Cache

Core

Core

Core

Core

Core

Core

Core

Core

Core

Core

Core

Core

Core

HBM2 Interface

L2Cache

Core

Core

Core

Core

Core

Core

Core

Core

Core

L2CacheCore

RING-Bus

Core

Core

7Core

HBM2 interface

PCIe Interface

TofuD Interface

Core

Core

Core

Core

Core

Core

Core

Core

Core

Core

Core

Core

Core

Core

L2Cache

Core

Core

Core

Core

Core

Core

HBM2 Interface◼ TSMC 7nm FinFET & CoWoSCore

Copyright 2020 FUJITSU LIMITEDFujitsu-designed CPU core w/ High Memory Bandwidth◼ A64FX out-of-order controls incores, caches, and memories
achieve superior throughput

CMG 12x Compute Cores + 1x Assistant CoreCoreCoreCoreCore

512-bit wide SIMD
2x FMAs
230+
GB/s

115+GB/s

L1D 64KiB, 4way

BW and calc. perf.

A64FX

B/F

DP floating perf. (TFlops)

2.7+-L1 data cache (TB/s)

11+4

L2 cache (TB/s)

3.6+

1.3

256
GB/s

Memory BW (GB/s)

1024

0.37

HBM28GiB
8GiB
HBM2115+
GB/s

57+
GB/s

L2 Cache 8MiB, 16way

HBM28GiB
8GiB
HBM28x4
Copyright 2020 FUJITSU LIMITEDA64FX optimized load efficiency for apps performance◼ 128 bytes/cycle sustained bandwidtheven for unaligned SIMD load

Read data0
64B/cycle

L1D cache

Read port0

Read data1
64B/cycle

Read port1
128B◼ “Combined Gather” doubles gather(indirect) load’s data throughput,when target elements are within a“128-byte aligned block” for a pair of
two regs, even & odd
Suggested through Co-design work w/ app teams
9Mem.

01

flow-132

7

5 4

flow-4flow-3

6

flow-2Regs

0 1 2 3 4 5 6 7

8B

Maximizes BW to 32 bytes/cyc.
Copyright 2020 FUJITSU LIMITEDA64FX Power Knobs to reduce power consumption◼ “Power knob” limits units’ activities via user APIs

◼ Performance/Watt can be optimized by utilizing Power knobs
Fetch

L1 inst.
cache

Issue

Decode& Issue

Dispatch

RSA

Reg-read

Execute

PGPR

EAGA
EXC
EAGB
EXD

EXA
EXB

RSE0
BranchPredictor
RSE1

RSBR

PPR

Cache and Memory

CommitCSE

Fetch
PortStore
Port

L1 data
cache

Write
BufferFrequency reduction
PC

ControlRegisters

PRX

EXA pipeline only
PFPR

FLA
FLB

FLA pipeline only
L2 cacheDecode width: 4 to 2

Tofu controller

HBM2 Controller
HBM210PCI Controller

52 cores

HBM2 B/W adjust(units of 10%)
Copyright 2020 FUJITSU LIMITEDFugaku system software developed with RIKEN
Fugaku applications
Fujitsu Technical Computing Suite / RIKEN-developed system software
Management software

File system

Programming environment

System management
for high availability &power saving
operation
Job management forhigher systemutilization & power
efficiency

FEFS
Lustre-based
distributed file system

XcalableMP, FDPS
Open MPI, MPICH(PiP), DTF
OpenMP, COARRAY, Math. libs.

LLIO
NVM-based file I/O
Accelerator for Fugaku

Compilers, AI frameworks
Debug & tuning tools, Containers

Multi-Kernel System: RHEL8 and light-weight kernel (IHK/McKernel)

Fugaku system hardware
Fugaku developed w/ RIKEN

11Tensorflow
Chainer
PytorchDockerKVM
Singularity

FDPS: Framework for Developing Particle Simulator
PiP: Process-in-Process
DTF: Data Transfer Framework
IHK: Interface for Heterogeneous Kernels
Copyright 2020 FUJITSU LIMITEDOutline◼ Fugaku project overview◼ Co-design◼ Approach◼ Design results◼ Performance & energy consumption evaluation◼ Green500◼ OSS apps
◼ Fugaku priority issues◼ Summary
12

Copyright 2020 FUJITSU LIMITEDGreen500, Nov. 2019
◼ A64FX prototype –◼ Fujitsu A64FX 48C 2GHzranked#1 on the list

◼ 768x general purpose A64FX
CPU w/o accelerators• 1.9995 PFLOPS @ HPL, 84.75%• 16.876 GF/W
• Power quality level 2
https://www.top500.org/green500/lists/2019/11/
13

Copyright 2020 FUJITSU LIMITEDHow we achieved◼ Key for GF/W is {energy efficient HW} x {parallel/exec efficiency}◼ A64FX is designed for energy efficient◼ Fujitsu’s proven CPU microarchitecture & 7nm FinFET◼ SoC design: Tofu interconnect D integrated◼ CoWoS: 4x HBM2 for main memory integrated+ Concerted effortsof co-design◼ Superior parallel/exec efficiency◼ Math. libraries are tuned for application efficiency◼ Comm. libs are also tuned utilizing long experience of Tofu @ K computer◼ Performance tuning is efficiently done utilizing rich performance
analyzer/monitor
14

Copyright 2020 FUJITSU LIMITEDSC19 TOP500 calculation efficiency◼ A64FX superiorefficiency 84.75%
with small Nmax

◼ Optimizedcommunicationand math. libs
◼ Optimization of
overlapped
communication

90%

Efficiency◼ Results of:

A64FX prototype

100%

80%
70%
60%
50%

Challenge

40%
30%

w/o Accw/ Acc20%

10%
0%05,000,000

1510,000,000

Nmax

15,000,000

20,000,000

Copyright 2020 FUJITSU LIMITEDA64FX CPU performance evaluation for real apps
◼ Open source software, Realapps on an A64FX @ 2.2GHz◼ Up to 1.8x faster over thelatest x86 processor (24core,2.9GHz) x2◼ High memory BW and long
SIMD length of A64FX work
effectively with theseapplications

Relative speed up ratio
0.5

1.0

1.5

OpenFOAM
FrontlSTR
ABINITSALMON
SPECFEM3DWRF
MPAS
FUJITSU A64FX
16x86 (24core, 2.9GHz) x2
Copyright 2020 FUJITSU LIMITEDA64FX CPU power efficiency for real apps
◼ Performance / Energy
consumption on an A64FX @2.2GHzRelative power efficiency ratio
1.0

3.0

4.0

OpenFOAM
FrontlSTR◼ Up to 3.7x more efficient overthe latest x86 processor(24core, 2.9GHz) x2◼ High efficiency is achieved byenergy-conscious design andimplementation

2.0

ABINITSALMON

SPECFEM3D
WRF
MPASFUJITSU A64FX
17x86 (24core, 2.9GHz) x2
Copyright 2020 FUJITSU LIMITEDFugaku priority issues and performance prediction

https://postk-web.r-ccs.riken.jp/perf.html◼ 100x app performance andpower budget requirements
are met◼ Some apps utilize power knob
to reduce power consumption
and achieve high energy
efficiency◼ Measuring and optimizing
using real HW

18

Copyright 2020 FUJITSU LIMITEDFugaku project schedule
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022

Fugaku development & delivery

AppsFeasibility study Basicreview
designSelectapps

Architecture &sizing

Detailed design &Implementation

Manufacturing,
Installation
and TuningGeneral
operation

Co-Design w/ apps groups
Apps tuning
Previous estimation w/o HW
19w/ HW
Copyright 2020 FUJITSU LIMITEDCurrent status normalized by the estimation w/o HW

DGEMM

Faster than prev. est.

Lower energy than prev. est.

Performance
EnergyEstimation w/ HW / w/o HW◼ Performance and energy (=power*elapse) of apps

StreamA

B
20

Note: evaluation underway

CDEF
Copyright 2020 FUJITSU LIMITEDCurrent status normalized by the estimation w/o HW•

Performances are match

Computations wereGood estimation and•Lower power 8% in logic &15% in memory use;
acceptable variations

previous estimation

less than 20%

Note: evaluation underway

Performance
EnergyEstimation w/ HW / w/o HW

and energy (=power*elapse)of apps◼ Performance
reduced after theoptimization: errors aredue to precise simulation

DGEMM

StreamAB
21CDEF
Copyright 2020 FUJITSU LIMITEDSummary◼ “Fugaku” is co-designed and runs apps at highperformance w/ optimal power consumption◼ Arm HPC ecosystem and apps portfolio are growing
by efforts of communities and selections of HWFujitsu PRIMEHPC
FX1000◼ Fujitsu Supercomputer
PRIMEHPC FX1000 & FX700
based on Fugaku tech.
◼ Cray CS500 using Fujitsu
A64FX Arm CPU
FX700

https://www.riken.jp/pr/news/2019/20190827_1/
22

Copyright 2020 FUJITSU LIMITED23

Copyright 2020 FUJITSU LIMITED