System and Interconnect ofthe Supercomputer Fugaku
Yuichiro Ajima
Global Fujitsu Distinguished Engineer
Principal Architect
Future Society & Technology UnitFujitsu Limited
5/12/2022, ICEP2022

1© Fujitsu 2022

Supercomputing is the Key to the SDGs
Evolution of supercomputing technology
Simulations

Data Analytics

Fluid Dynamics
Collision Analysis
Material Scienceetc.

Artificial Intelligence
Big Dataetc.

Advanced supercomputing is expected to bring innovation in many areas
Areas

Life Science

EnergyManufacturing

Disaster Mgmt.

SDGs

Supercomputers are the key infrastructure for the SDGs
5/12/2022, ICEP2022

2© Fujitsu 2022

Supercomputer Performance Development
1E+19

Sum

1E+18

Rank1Rank500

1E+17
1E+16FLOPS

1E+15
1E+14
1E+13
1E+12
1E+11
1E+10
1E+9
1E+81993

19972001

2005

2009
Year2013

20172021⚫ Performance continues to improve at a rate that exceeds

Moore's Law, due to increases in system scale and density

5/12/2022, ICEP2022

3© Fujitsu 2022

Increased Scale and Density⚫ Modern supercomputers are distributed memory parallel

computers⚫Parallel: Interconnect connects many nodes⚫Distributed memory: OS runs per node (1-8 processor sockets)(C) JAXA

Numerical Wind
Tunnel (1993)
124GFlops
140 CPU
5/12/2022, ICEP2022(C) JAMSTEC(C) RIKEN

Earth Simulator(2002)
35.9TFlops
5,120 CPU
4K computer (2011)

Fugaku (2020)

10.5PFlops
88,128 CPU

442PFlops
158,976 CPU© Fujitsu 2022

National Large-Scale Research Facilities⚫ Fugaku is one of the government-designated sharedresearch facilities, as was its predecessor, the K computer⚫Basically, free of charge for domestic researchers⚫Examples of other facilities⚫ Radiation Facility (SPring-8), X-ray Free Electron Laser Facility (SACLA),
Proton Accelerator Research Complex (J-PARC)

http://www.mext.go.jp/a_menu/kagaku/shisetsu/index.htm
5/12/2022, ICEP2022

5© Fujitsu 2022

Organization and Location of Fugaku⚫ Organization: RIKEN R-CCS⚫Responsible for operation⚫Also, the main developer

https://www.r-ccs.riken.jp/jp/overview/⚫ Location: Kobe City
⚫Port Island⚫Opposite shore from Kobe Airport

5/12/2022, ICEP20226

https://www.mlit.go.jp/koku/koku_fr14_000081.html© Fujitsu 2022

Migration from the K Computer to Fugaku
The supercomputer Fugaku© RIKENThe K computer© RIKEN◼ The K computer shut down in August 2019 after seven years of full operation◼ Five months later, the supercomputer Fugaku started partial shared use in March 2020
5/12/2022, ICEP2022

7© Fujitsu 2022

Benchmark Results
K computerFugaku

TOP500 (Petaflops)

10.51 (#1 Jun/Nov 2011)

442.01 (#1 Jun 2020~)

HPCG (Petaflops)

0.60274 (#1 Nov 2011~Nov 2017)

1.60045 (#1 Jun 2020~)

HPL-AI (Petaflops)–

2,000 (#1 Jun 2020~)

Graph500 (GTEPS)

31,302.4 (#1 Jun 2014, Jul 2015~Jun 2019)

102,956 (#1 Jun 2020~)⚫ TOP500: Performance of solving the linear equation Ax = b with dense coefficient matrix. This benchmark has beenmeasured for 30 years and is the most popular performance indicator⚫ HPCG: Performance of solving the linear equation with sparse coefficient matrix using Conjugate Gradient method.

This performance will be dominated by memory system performance⚫ HPL-AI: Performance of solving the linear equation utilizing lower precision floating point calculation, such as fp16,

which AI applications often use⚫ Graph500: Performance of big data processing. This performance will be dominated by interconnect performance
5/12/2022, ICEP20228© Fujitsu 2022

Packaging of Fujitsu Supercomputers
K computer

Fugaku(C) RIKEN

HPC2500

FX1

FX10

FX1000
FX700

FX100

Single Socket Node
Water Cooling200320092012

3D Stacked Memory
2015

2.5D Packaging2021⚫ Fujitsu has developed single-socket node, water-cooled

supercomputers using 3D stacked memory⚫ Fugaku integrates memory into the CPU package
5/12/2022, ICEP2022

9© Fujitsu 2022

Two Types of HPC System and Interconnect⚫ Tightly-coupled parallel computer⚫Torus or multi-plane topology⚫K computer, Fugaku, Google TPU, NVIDIA DGX, etc.

⚫ Cluster and parallel storage⚫Fat-tree (Clos) or dragonfly topology⚫InfiniBand, Slingshot, etc.
TreeNetworkComputation cluster(C) RIKEN

Tightly-coupled parallel computer
5/12/2022, ICEP2022

10Parallel storage cluster© Fujitsu 2022

Interconnect Types of Supercomputers
'93

Rank

'94

'95

'96

'97

'98

'99

'00

'01

'02

'03

'04

'05

'06

'07

'08

'09

'10

'11

'12

'13

'14

'15

'16

'17

'18

'19

'20

'21

1~50~100

HPC Interconnect
Ethernet~150~200~250~300~350~400~450~500⚫ Top-level supercomputers use HPC interconnects such asHPC Interconnect

Ethernet

InfiniBand and proprietary interconnects⚫ Ethernet-only systems have increased since 2000s
5/12/2022, ICEP2022

11© Fujitsu 2022

System Architecture and Structure ofFugaku

5/12/2022, ICEP2022

12© Fujitsu 2022

Configuration and Topology of Fugaku⚫ Configuration: 158,976 nodes x 1 CPU⚫ Topology: 6D-mesh/torus (24x23x24x2x3x2)⚫ Link: 97,632 AOCs, 4X 28 Gbps

“Report on the Fujitsu Fugaku System,” Jack Dongarra (2020)

https://www.r-ccs.riken.jp/intro-hpc/hellosc-fugaku/04.html

5/12/2022, ICEP2022

13© Fujitsu 2022

Packaging Structure of Fugaku RackCPU
HBM (High Bandwidth Memory)

Si Interposer

CPU Package

CMU (CPU Memory Unit)
CPU × 2Rack
CPU × 384
CMU × 192
BoB × 24
Shelf × 8
BoB (Bunch of Blades)
CPU × 16
CMU × 8Shelf
CPU × 48
CMU × 24
BoB × 3
5/12/2022, ICEP2022

14© Fujitsu 2022

A64FX Processor Developed for Fugaku
The First

based HPC processor

A64FX⚫ A64FX integrated memory into the package
5/12/2022, ICEP2022

15© Fujitsu 2022

Configuration and Bandwidth of A64FX⚫ A64FX consists of 4 Core Memory Groups (CMGs)⚫Each CMG consists of 13 cores, shared L2 cache, and one HBM
CMG

12 compute + 1 assistant coreCore
512bit SIMD FMA x 2

256GB/s 128GB/s
L1D$ 64KiB

128GB/s 64GB/s
L2$ 8MiB

256GB/s
HBM2 8GiB

Tofu Interconnect D

128GB/s
128GB/s

PCI Express
Network on Chip

5/12/2022, ICEP2022

16© Fujitsu 2022

Tofu Interconnect D⚫ Six network interfaces per node
⚫ The network router has 10 ports with 56 Gbps links⚫Each port is directly interconnected with other CPUs

⚫Forming a 6-dimensional mesh/torus networkTofu Interconnect DCMG

CMG

Tofu Network Interface
Tofu Network Interface
Tofu Network Interface

Network on Chip

Tofu Network Interface
PCI Express

CMG

CMG

Tofu Network Interface
Tofu Network Interface

in 6.8GB/s, out 6.8GB/s
x 6 interfaces
5/12/2022, ICEP2022

Tofu Network Router

1728Gbps 2-lanes
x 10-ports© Fujitsu 2022

6D Mesh/Torus Network⚫ XYZ dimensions are scalable, and ABC have fixed size
24 x 23 x 24 x 2 x 3 x 2ZBX

YAC

⚫ The B-axis with length 3 improves the system availability⚫Virtual torus takes advantage of its redundancy5/12/2022, ICEP2022

18© Fujitsu 2022

Structure and Topology of CMU⚫ Two CPUs
⚫Connected by the C axis

CPU

⚫ Two or Three AOC ports

AOC (X)

AOC

AOC (Y)AOCAOC (Z)⚫Each AOC, 4X 28 Gbps
⚫ Shared by two CPUs
⚫ Each CPU uses 2 lanes

CPU⚫AOCs are connected in XY-axes or XYZ-axes⚫The average number of AOC ports per CMU is 2.5⚫ The average number of AOC per CMU is 1.25, and per CPU is 0.625
5/12/2022, ICEP2022

19© Fujitsu 2022

Water-Cooling of CMU⚫ Cooling water is supplied and discharged from the central

block of the rack

Primary supply pipe
Primary discharge pipe

Cooling water flow
5/12/2022, ICEP2022

20© Fujitsu 2022

Structure and Dimension of Shelf and Rack
⚫ Shelf⚫24 CMUs connected in ZAB-axes (4x2x3)⚫with backplane and electrical cables⚫ Half-rack (top or bottom)⚫Four shelves connected in XY-axes (2x2)⚫with electrical cables⚫ Rack⚫In most cases, two half-racks are connected⚫96 out of 480 AOC ports in a rack are used
5/12/2022, ICEP2022

21© Fujitsu 2022

Movable Cable Guides⚫ We developed a movable cable guide to achieve highpackaging density while avoiding interference with thereplacement work

Cable guide moved

5/12/2022, ICEP2022

22© Fujitsu 2022

Half-Mount Racks of Fugaku⚫ The network size of Fugaku is 24 x 23 x 24 x 2 x 3 x 2⚫Some half-racks contain only half of the nodes⚫The network size of a half-mounted half-rack is 2 x 1 x 4 x 2 x 3 x 2

https://www.r-ccs.riken.jp/en/fugaku/3d-models/, https://my.matterport.com/show/?m=mnpGYx1pQtx&sr=-.23,-.99&ss=176
5/12/2022, ICEP2022

23© Fujitsu 2022

Fault Isolation and Virtual Torus⚫ A failed node is isolated in a rectangular partition⚫ Virtual torus can use a partition containing a failed node
⚫Shortening the length of a virtual axis by one to exclude the failed

node

54
6

144

35
1

2

5/12/2022, ICEP2022

21133

422243© Fujitsu 2022

Discussion on Forward Error Correction (FEC)⚫ HPC Interconnects are designed for low latency⚫For example, minimal latency of TofuD is about 0.5 μsec⚫ FEC will add an additional delay of about 0.05 to 0.2 μsec
⚫ TofuD was able to transmit without FEC, so it is disabledLatency of TofuD and FEC
Minimal latency
1-hop latency
Inter-rack cable delayFEC
Low Latency FEC
05/12/2022, ICEP2022

0.1

0.2
0.3
microseconds
250.4

0.5
© Fujitsu 2022

Adjacent Communication Latency⚫ FEC increased latency by 9-40%⚫Parallel efficiencies will be reduced in some application areas⚫ Such as molecular dynamics and lattice quantum chromodynamics⚫For most apps, performance degradation can be avoided byoptimization

Estimated Adjacent Communication Latency of TofuD

Intra-rack w/o FEC
Intra-rack w/ LLFEC
Intra-rack w/ FEC
Inter-rack w/o FEC
Inter-rack w/ LLFEC
Iner-rack w/ FEC
05/12/2022, ICEP2022

0.1

0.2

0.3
26

0.4
0.5
microseconds

0.6

0.7

0.8
© Fujitsu 2022

Average Communication Latency⚫ FEC increased latency by 33% to 130%
⚫Collective communication with short message sizes will be affected⚫ Frequently used in insufficiently parallelized programs⚫ This can be a limiting factor in expanding the range of parallel applications⚫ Well-optimized applications can hide the latency by optimization⚫ Such as shared parameter updates in multi-physics simulation
Estimated Average Latency of TofuD in 24x24x24x2x3x2 system
Average latency w/ LLFEC
Average latency w/ FEC
05/12/2022, ICEP2022

12327

45
microseconds

678© Fujitsu 2022

Other Top-Level Supercomputers inRecent Years

5/12/2022, ICEP2022

28© Fujitsu 2022

World Rank One System in Recent Years⚫ Number one systems located in the U.S., China, and Japan⚫Average time for a system to stay number one is about 1.7 years

5/12/2022, ICEP202229

https://www.top500.org/resources/top-systems/© Fujitsu 2022

Lifetime Ranking of the Rank 1 Systems⚫ The rank 1 systems after the K computer have remained in

the top group for years due to their large scale
K computer Sequoia Tianhe-2A
Earth Simulator
Tianhe-1ATitan
Sunway TaihuLight
Blue Gene/LJaguar
SummitRoadrunner
1Fugaku

6Rank

11
16
21
26
31
2002
5/12/2022, ICEP2022

2004

2006

2008

2010

2012
Year30

2014

2016

2018

2020
© Fujitsu 2022

K computer (2011)⚫ Configuration: 82,944 nodes x 1 CPU⚫ Topology: 6D-mesh/torus (24x18x16x2x3x2)⚫ Link: about 200,000 electrical, 8X 6.25 Gbps

http://www.s.u-tokyo.ac.jp/ja/story/rigakuru/03/interview/info.html
5/12/2022, ICEP2022

31© Fujitsu 2022

Sequoia (2012)⚫ Configuration: 98,384 nodes x 1 CPU⚫ Topology: 5D-torus (16x12x16x16x2)⚫ Link: about 12,000 optical, 24X 10 Gbps

5/12/2022, ICEP2022

https://computing.llnl.gov/tutorials/bgq/, https://www.top500.org/featured/systems/sequoia-lawrence-livermore-national-laboratory/© Fujitsu 2022
32

Titan (2012)⚫ Configuration: 18,688 nodes x (1 CPU + 1 GPU)
⚫ Topology: 3D-torus (25x16x24)⚫ Link: about 20,000 electrical, 12X 3.125 Gbps

“Hardware concepts and terminology relevant to the programmer (Magny Cours, Gemini interconnect, architecture of XE6), Launch of parallel applications/batch system,
User Environment, Compilers of the XE6 (PGI, Pathscale, GNU, Cray),” https://www.nersc.gov/users/NUG/annual-meetings/NUG-2010/presentations/
5/12/2022, ICEP2022

33© Fujitsu 2022

Tianhe-2 (2013)⚫ Configuration: 16,000 nodes x (2 CPU + 3 Accelerator)⚫ Topology: 5-tiers tapered fat-tree⚫ Link: about 7,000 (estimated) optical, 8X 10 Gbps

Jack Dongarra, “Visit to the National University for Defense Technology Changsha, China”

5/12/2022, ICEP2022

34© Fujitsu 2022

Sunway TaihuLight (2016)⚫ Configuration: 40,960 nodes x 1 CPU⚫ Topology: 4-tiers tapered fat-tree (estimated)⚫ Link: about 6,000 (estimated) AOCs, 4X 25 Gbps

https://www.top500.org/resources/top-systems/sunway-taihulight-national-supercomputing-center-i/

5/12/2022, ICEP2022

35Haohuan Fu, "The Sunway TaihuLight Supercomputer System and Application"© Fujitsu 2022

Summit (2018)⚫ Configuration: 4,608 nodes x (2 CPU + 6 GPU)
⚫ Topology: 3-tiers full-bisectional fat-tree⚫ Link: about 9,000 (estimated) AOCs, 4X 25 Gbps

https://www.ornl.gov/news/ornl-launches-summit-supercomputer
https://www.olcf.ornl.gov/wp-content/uploads/2018/05/Intro_Summit_System_Overview.pdf
5/12/2022, ICEP2022

36© Fujitsu 2022

Frontier (2022 planned)⚫ Configuration: 9,408 nodes x (1 CPU + 4 GPU)
⚫ Topology: dragonfly (> 73+ groups x 512 terminals)⚫ Link: about 10,000 (estimated) AOCs, 8X 50 Gbps

https://science.osti.gov/-/media/ascr/ascac/pdf/meetings/202203/ASCAC_202203-Geist.pdf
5/12/2022, ICEP2022

37© Fujitsu 2022

Timeline of Fugaku Development andNGACI White Paper

5/12/2022, ICEP2022

38© Fujitsu 2022

Timeline of Fugaku Development⚫ After the SDHPC White Paper and the feasibility study,

Fugaku development started in 2014

⚫SDHPC was a community activity to discuss strategies fordeveloping HPC systems
K computer project started
K computer development

20062010

2011

NGACI White Paper released
K computer ranked #1
K computer full operation started
K computer shut down
2012

SDHPC White Paper released
Post-K feasibility study started
Post-K feasibility study report
Post-K development started
5/12/2022, ICEP2022

20132014

2018

2019

2020

2021

Fugaku development
Post-K production approved
Post-K production started
Fugaku ranked #1
Fugaku full operation started
39© Fujitsu 2022

Projection at the Time of Feasibility Study
1E+19
Sum

Rank1

Rank500

1E+18
1E+17
1E+16

K computer

FLOPS

1E+15
1E+14
1E+13
1E+12
1E+11
1E+10
1E+9
1E+8
1993
5/12/2022, ICEP202219972001

2005
402009Year201320172021© Fujitsu 2022

Prediction in SDHPC White Paper
1E+19
Sum

Rank1

Rank500

1E+18
1E+171E+16

K computer

FLOPS

1E+15
1E+14
1E+13
1E+12
1E+11
1E+10
1E+9
1E+8
1993
5/12/2022, ICEP2022

1997

2001

2005
41

2009

Year

2013

20172021© Fujitsu 2022

Target Achieved with a One-Year Delay
1E+19
Sum

Rank1

Rank500

1E+18
1E+17

Fugaku1E+16

K computer

FLOPS

1E+15
1E+14
1E+13
1E+121E+111E+101E+91E+8
1993
5/12/2022, ICEP2022

199720012005
422009Year201320172021© Fujitsu 2022

NGACI White Paper⚫ Next-Generation Advanced Computing Infrastructure⚫Overview and Objectives
In considering the sustainable development of high-performance computing in the future, we can expect furtherdevelopments such as further integration with AI and Big Data technologies and deployment in new application
fields such as Society 5.0, but it is also true that many technological challenges, such as the end of Moore's Law, lieahead. This activity (NGACI) is a forum for open exchange of ideas and opinions on the technical issues that needto be addressed for future high-performance computing environments and for shared computing infrastructure, whatkind of research and development is needed, and what kind of activities should be conducted as a community, andto summarize these ideas as White The purpose is to contribute to the development of this field by exchanging
opinions openly and summarizing them as White Papers.

⚫Activities⚫ Number of registered community members: > 100⚫ Four working groups discussed future system vision and issues⚫ White Paper 1.0.0 (164 pages) released https://sites.google.com/view/ngaci/home
5/12/2022, ICEP2022

43© Fujitsu 2022

Prediction in NGACI White Paper
1E+19Sum

Rank1

Rank500

1E+181E+17Fugaku

Prediction in NGACIWhite Paper

1E+16K computer

FLOPS

1E+151E+141E+13
1E+12
1E+11
1E+10
1E+91E+8
2001
5/12/2022, ICEP2022

2004

2007

2010

2013

2016
Year442019

2022

2025

2028
© Fujitsu 2022

Feasibility Study to Begin This Year
Ministry of Education, Culture, Sports,
Science and Technology
Development and Utilization of World-Class
Large-Scale Research Facilities
FY2022 Budget 45.7B JPY (376M USD)

Conduct necessary research and studies
on the ideal next-generation computing
infrastructure, including surveys ofdomestic and international trends intechnologies and user needs, andresearch and development of elemental
technologies
FY2022 Budget 0.4B JPY (in page 3)
https://www.mext.go.jp/content/20211223-mxt_kouhou02-000017672_1.pdf
5/12/2022, ICEP2022

45© Fujitsu 2022

Emerging Technologies for Future
System5/12/2022, ICEP2022

46© Fujitsu 2022

Increasing Cost of Processes Technologies⚫ Investment per wafer is rising exponentially⚫ No significant reduction in cost per gate after 28nm

process, even though densities are increasing

Gate Cost

https://cset.georgetown.edu/publication/ai-chips-what-they-are-and-why-they-matter/
5/12/2022, ICEP2022

47

https://www.semi.org/en/semiconductor-industry-2015-2025© Fujitsu 2022

Chiplet to Counter Increasing Die Cost
⚫ Manufacturing large die is no longer economical⚫ Increasing yield with “chiplets” becomes important

https://old.hotchips.org/hc31/Hot_Chips_2019_DrLisaSu_AMD_0819.pdf
5/12/2022, ICEP2022

48© Fujitsu 2022

Domain-Specific Architecture⚫ Specialized circuits that execute specific workload at highthroughput are becoming more important⚫Transistor density improvement has slowed down and small circuits

are required⚫ Application-specific architecture⚫e.g., Anton for molecular dynamics⚫ Ultra-low latency torus network⚫ Domain-specific architecture⚫e.g., Google TPU for Deep Learning⚫ Low-precision, high-throughput systolic array
5/12/2022, ICEP2022

49© Fujitsu 2022

High-Density, High-Speed Transmission⚫ Insertion loss on PCB is severe at >50 Gbps / lane
⚫Retimers and active electrical cables⚫Fly-over cable connection⚫ Co-packaged modules and connectors at >100 Gbps / lane
⚫CPO: Co-Packaged Optics, CPE: Co-Packaged Electronics

5/12/2022, ICEP2022

50© Fujitsu 2022

Roadmap for Future Optical Technologies
Evolution of opto-electronic fusion devices
Expand to computing
for Energy Efficient ICT

Logic and Optics
Integration

To be provided in FY2022
Compact, power-saving,high-performance opticalcommunication device
Digital Coherent
Transceiver

SwitchChip

Opto-electronic
Computingtechnologies

Inter-chip optics(2025 ~)

Optics Integration
into Logic

Optical Computing

Intra-chip optics andoptical signal processing(2030 ~)

OE co-packaging(2024 ~)
5/12/2022, ICEP2022

51(Target year for development of prototypes)© Fujitsu 2022

Summary⚫ Supercomputers are the key infrastructure for the SDGs
⚫ System scale and density continue to increase⚫ Fugaku is the #1 system in four major benchmarks⚫ System architecture and structure of Fugaku⚫ Other top-level supercomputers in recent years⚫ Timeline of Fugaku development and NGACI⚫ Emerging technologies for future systems⚫Chiplet, Domain-Specific Architecture, Co-Packaged Optics

5/12/2022, ICEP2022

52© Fujitsu 2022

Thank you

5/12/2022, ICEP2022© Fujitsu 2022