June 6, 2024
ICS 2024

From Past to Future :The Legacy and Hypothesis ofSupercomputing

Naoki Shinjo
SVP, Head of Advanced Technology Development Unit
Fujitsu Research FUJITSU LIMITED© 2024 Fujitsu Limited

Naoki Shinjo
SVP, Head of Advanced Technology Development Unit
Fujitsu Research
FUJITSU LIMITED

Naoki Shinjo
He Joined Fujitsu Limited in 1987. He belonged to the large-scalecomputer development division and was involved in supercomputer
hardware development.
In 2006, he participated in the conceptual design of K computer, andfrom 2007, he was engaged in the development of K computer. After K
computer development was completed, he participated in Fugaku
feasibility studies from 2012.
In 2014, he became Head of Next Generation Technical Computing Unitand engaged in the development of Fugaku. Current position since April
2023.
2© 2024 Fujitsu Limited

Fujitsu’s History
in Supercomputer
Development

5© 2024 Fujitsu Limited

Fujitsu's Supercomputers Released to World
Fujitsu has continuously developed and delivered world-class supercomputersSupercomputer Fugaku

K computer
VPP5000© RIKENFX1© RIKEN

VPP300/700

NWT

PRIMEHPC FX100

VP Series

SPARC
EnterpriseⒸ JAXA

PRIMEHPC FX1000/FX700
PRIMEHPC FX10PRIMEPOWER
HPC2500
PRIMEQUEST

F230-75APU
VPP500PRIMERGY CX400/600
Cluster nodeAP3000PRIMERGY BX900
Cluster nodeAP1000

HX600
Cluster node
PRIMERGY RX200
Cluster node

~1980

1985

1990

1995

2000

2005
6

2010

2020
© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

FACOM230-75 APU (1977)
Japan‘s first provision of supercomputers➢ FLOPS :22 MFLOPS (Add FP32) / 11 MFLOPS (Mul FP32)➢ Innovative Features (at that time) :• Vector Processing• APU (Array Processing Unit)➢ Activity :National Aerospace Laboratory of Japan(NAL)*NAL; a predecessor of JAXA

7

© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

VP series (1982~)
First “Giga scale” supercomputer in world *1➢ FLOPS :1.142 GFLOPS *1➢ Innovative Features (at that time) :• Static RAM• Fast data sorting and conditional operations➢ Activity :Nagoya University Plasma Research Center *2
VP-200*1 VP-400*2 VP-100
8© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

VP2000 series (1988)
Supercomputer with UNIX
➢ FLOPS :5G FLOPS *VP2600 maximum configuration➢ Innovative Features (at that time) :• Virtual Machine (Enable UNIX)• Variety of product variations➢ Activity :Rich industrial applications
VP-2100

9© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

NWT (1993)
Ranked 1st in TOP500, Nov '93 and Nov '94-'95.
➢ FLOPS :280 GFLOPS (1.68 GFLOPE/PE)➢ Innovative Features (at that time) :• GaAs LSI chips• distributed memory parallel vector
computer architecture➢ Activity :National Aerospace Laboratory of Japan (NAL)(Developed jointly by NAL and Fujitsu)Ⓒ JAXA*NAL; a predecessor of JAXA

10

© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

VPP500 (1993)
Commercial HPC developed from NWT.
➢ FLOPS :355 GFLOPS (1.68 GFLOPE/PE) *maximum configuration➢ Innovative Features (at that time) :• GaAs LSI chips• distributed memory parallel vector
computer architecture➢ Activity :National Laboratory for High Energy Physics
in Japan (KEK)*,Japan Atomic Energy Research Institute(JAERI)
and more.
11*KEK; a predecessor of High Energy Accelerator Research Organization in Japan*JAERI; a predecessor of Japan Atomic Energy Agency© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

VPP300 / VPP700 (1995~)
Successor to VPP500 using CMOS technology➢ FLOPS :* VPP700
1,126 GFLOPS (2.2 GFLOPS/PE) maximum configuration➢ Innovative Features (at that time) :• CMOS technology• Sell widely by lowering prices➢ Activity :LRZ,
Kyushu University and more

VPP-700

12© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

VPP5000 (1999)
Fujitsu's last Vector-type supercomputer➢ FLOPS :4,915 GFLOPS (9.6 GFLOPS/PE) * VPP5000
maximum configuration➢ Innovative Features (at that time) :• Fastest single CPU performance in the world• High execution efficiency(LINPACK 96.4% )➢ Activity :Nagoya University, Europe and more

13

© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

PRIMEPOWER HPC2500 (2002)
Dawn of Scalar-type supercomputer era➢ FLOPS :85.1 TFLOPS (5.2 GFLOPS/CPU)* Maximum
configuration➢ Processor
SPARC64Ⅴ: 1core, 1.30GHz, 130nm process➢ Innovative Features (at that time) :• Scalar-type Supercomputer• Fusion of HPC and UNIX server technology➢ Activity :National Aerospace Laboratory of Japan,
Nagoya University,
Kyoto University and more
14

© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

FX1 (2008)
Advanced many-core architecture➢ FLOPS :110.6 TFLOPS (40 GFLOPS/CPU) *JAXA system➢ Processor
SPARC64Ⅶ : 4cores, 2.5GHz, 65nm process➢ Innovative Features (at that time) :• Many-core architecture• highly-functional switch• High execution efficiency(LINPACK 91.19% )➢ Activity :JAXA, Nagoya university
15© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

K computer (2011)
Ranked 1st in TOP500, Jun 2011 and Nov 2011.
➢ FLOPS :10.51 PFLOPS (128 GFLOPS/CPU)➢ Processor
SPARC64Ⅷfx : 8cores, 2GHz, 45nm process➢ Innovative Features (at that time) :• Tofu Interconnect• Water cooling➢ Activity :RIKEN(Developed jointly by RIKEN and Fujitsu)

16© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

PRIMEHPC FX10 (2012)
Commercial HPC that operates seamlessly 365 days.
➢ FLOPS :23.2 PFLOPS (236.5 GFLOPS/CPU) * Maximum
configuration➢ Processor
SPARC64Ⅸfx : 16cores, 1.848GHz, 40nm process➢ Innovative Features (at that time) :• High reliability Technology• From academia to industry➢ Activity :University of Tokyo, Kyushu university,
Nagoya University and more
17

© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

PRIMEHPC FX100 (2014)
Ambitious commercial HPC developed from K.
➢ FLOPS :over 100 PFLOPS (1.126 TFLOPS/CPU) * Maximum
configuration➢ Processor
SPARC64Ⅺfx : 32+2cores, 2.2GHz, 20nm process➢ Innovative Features (at that time) :• HMC• Tofu Interconnect 2 (optical communication)➢ Activity :National Institute for Fusion Science,
JAXA, Nagoya university and more

18

© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

Supercomputer Fugaku (2020)
Ranked 1st in TOP500, Jun 2020 to Nov 2021.
➢ FLOPS :537.2 PFLOPS (3.38 TFLOPS/CPU)➢ Processor
A64FX : 48cores, 2.2GHz, 7nm process➢ Innovative Features (at that time) :• Arm SVE architecture• HBM2• Energy Efficiency➢ Activity :RIKEN(Developed jointly by RIKEN and Fujitsu)
19© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

Fugaku’s Position in TOP500 Ranking
TOP500 No.1

Fugaku442.0PFTransition of TOP500

Jun 2020 ~ Nov 2021
1E

Graph500 No.1

100P

系列1
系列3

系列2
系列4

K
10.5PF
Tianhe-2
33.8PF

Summit148.6PF
SunwayTaihuLight
93PF

FRONTIER
1.1EF

HPCG No.1
Jun 2020 ~ Jun 2024

HPL-MxP No.1
Jun 2020 ~ Nov 2021

LINPACK(Flops)

Jun 2020 ~ Jun 2024
10P
1P

Sequoia
16.3PF

Titan
17.5PF

100T
10T
1T

RoadRunner
1.0PF
BlueGene/L
70.7TF

Tianhe-1A
2.5PF

Green500 No.1
Nov 2019* Fugaku prototype20© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

Fujitsu Arm-based CPU A64FXThe A64FX CPU, independently developed by Fujitsu, is theheart of Fugaku, the world's fastest supercomputer. With
approximately 160,000 A64FX CPUs interconnected, Fugaku
achieved the highest performance at the time.
• Arm Architecture Extended for HPC• Lead partner in the development of Scalable Vector Extensions(SVE)• World's first implementation of a high-performance core
supporting SVE• Architecture for High Application Performance:• 512-bit SIMD
• High memory bandwidth with HBM2

• Enhanced AI Computing Capabilities:• Supports FP16, INT16,INT8
21© 2024 Fujitsu Limited

1997

1982

1988

1993

1995

1999

2002

2008

2011

2012

2014

2019

2020

PRIMEHPC FX1000 / FX700 (2019)
Commercial HPC capable of large-scale configuration beyond Fugaku.
➢ FLOPS :1.328 EFLOPS (3.38 TFLOPS/CPU) * FX1000
maximum configuration➢ Processor
A64FX : 48cores, 2.2GHz, 7nm process➢ Innovative Features (at that time) :• large-scale configuration (FX1000)• Fujitsu software Stacks (FX1000)• Software stacks leveraging OSS(FX700)➢ Activity :FX100022

Japan Meteorological Agency,
University of Tokyo,
Nagoya University and more

© 2024 Fujitsu Limited

Reliability

Performance

Fujitsu Processors Supporting HPC
K computer

512-bit SIMD for HPC and AI
High Bandwidth Memory
Scalable Many-core Architecture
Virtual Machine Architecture
Software on Chip
High-speed Interconnect
HPC-ACE
System on Chip
Hardware Barrier
Multi-core Multi-thread
L2$ on Die
Non-blocking $O-O-Execution
SuperO-Scalar
Single-chip CPU
Store Ahead
Branch History
Prefetch

HPC
A64FX
SPARC
64XIfxSPARC
64IXfx© RIKEN
SPARC
64VIIIfx

UNIX

SPARC
64

GS
8600

FUJITSUMONAKA

SPARC
64VII
GS21
3600
GS21
2600

SPARC
64V

GS21
1600

Supercomputer Fugaku

GS21
900
SPARC
64GP

$ ECC
Register/ALU Parity
Instruction Retry$ Dynamic Degradation
Error Checkers/History

SPARC
64VI

SPARC
64V+

SPARC
64X

SPARC
64XII

SPARC
64X+

SPARC
64GP+

GS21
600

SPARC
64 II

GS
8800

GS
8800B

Mainframe

GS
8900© RIKEN
Technology Generation

~1999

2000

2005

2010

2015

2020

© Fujitsu 2023

FUJTISU-MONAKA :Green Data Center Solution
Rooted in Supercomputer
Technology*This section is based on results obtained from a project, JPNP21029, subsidized by the New Energy and Industrial Technology Development Organization (NEDO).
24© 2024 Fujitsu Limited

Fujitsu Arm based Processor
Highperformance

Energy EfficientHigh Security &Reliability

Easy to Use

TCO reduction

AI workload

development goals
Performancex2

Compared to other companies

Power efficiency

x2

Compared to other companies

To be shipped in 2027

Create a sustainable digital society by
developing the energy-efficient processor.
25© 2024 Fujitsu Limited

Hardware overview
FUJITSU-MONAKA adopts the innovative 3D many-core architecture• 3D stacking of Core die and SRAM die via TSV enables more cores, low latency, and high throughput.
• By using different process sizes for Core die and SRAM die, the utilization rate of expensive fineprocess technology is suppressed to 30%, achieving cost-efficiency.
FUJITSU-MONAKA Specification
Armv9-A Architecture
SRAM die• SVE2 for AI and HPC• Confidential Computing for security

Core die

144 cores x 2 sockets (288 cores per node)
Ultra low voltage for energy-efficiency

DDR5

3D chiplet

IO die• Core die• SRAM die/IO die

PCIe /
Interconnect

2nm
5nm

DDR5 12 channels
PCI Express 6.0 (CXL3.0)
Air cooling26© 2024 Fujitsu Limited

Software Overview
FUJITSU-MONAKA focuses on open-source software in its software stack.

• Supports industry standard software• Working on the standard tools (Python/Java/LLVM) to provide higher performance.
• Enabling smooth transition of customer assets and continuously enhancing performance
FUJITSU-MONAKA software stack

Application

Library
Framework
Middleware
OS
FirmwareHardware

Use Case
Speech Recognition

Surrogate Model

Generative AI

Software Stack in Ecosystem
OpenBLASLinux

NumPy/SciPy

Kubernetes

scikit-learn

oneDNNOpenStackoneDALGCC/LLVM

PyTorch/TensorFlow

Confidential Containers

Arm Processor Utilization & FUJITSU-MONAKA System Development

Many Core

High-Capacity Memory

27

Low Power

Low Cost

Security© 2024 Fujitsu Limited

Leveraging Supercomputer Technology for Energy Efficient

Exceed vendor voltage, achieve next-gen efficiency.
• Operate at a lower voltage than vendor-guaranteed, achieving power efficiency one
generation ahead.
• This initiative alone is expected to achieve a 20% power reduction. Combined with othertechnologies, FUJITSU-MONAKA ultimately aims for double the power efficiency of othervendors.
Energy efficient technology example:
power development
Ultra low voltage SRAM
SRAM circuits are the initial bottleneck for low-voltage operation. Fujitsu's innovative
assist circuit design addresses this challenge, enabling stable operation at lower voltages.
WL

5nm

Memory cell
VDD

3nm

Ultra low voltage
operation on
FUJITSU-MONAKA

Post 2nm

BL

2nm

BL

Power Consumption

7nm

assist circuit

Semiconductor Technology Node28

28

© 2024 Fujitsu Limited© 2024 Fujitsu Limited

Leveraging Supercomputer Technology for AI Workload

Economical and Right-sized AI infrastructure solution• GPU clusters are costly and hinder business growth.
• FUJITSU-MONAKA, an air-cooled AI processor, offers a cost-effective and scalable solution.
• This will be achieved through AI/ML technologies cultivated by Fugaku.
AI solution example:
Surrogate Model × OpenFOAM

AI infrastructure using only CPU
HOT AI infrastructure

COOL AI infrastructure

preprocessing

post-processing

800

using only CPU

Unused capacity
CPU
usage
GPU
usage

CPU
usage

generalAI

generalFluid Simulation

OpenFOAM

AI

600

processing time

Fluid Surrogate Model

x49

400

200

100%

Interactive execution with
High-precision finalvisualization.
verification using
Acquire approximate
approximate parametersparameters.
from the surrogate model.

100%

High performance, but costly andcannot fully utilize resources.

Right-sized performance withhigh efficiency and affordability.
29
29

0
1

2

© 2024 Fujitsu Limited© 2024 Fujitsu Limited

Leveraging Supercomputer Technology for OSS

Co-development with open-source communities to build an ecosystem• Expand the scope of the software stack to datacenter systems• Building software stack using real applications• Promoting expansion of the Arm ecosystem
Key contributions to OSS Community

Latest example :First successful Arm enablement for oneDAL

Open-source collaboration, including mission-critical systems andFugaku, drives FUJITSU-MONAKA development.

Successfully replaced MKL MATH functions with optimized opensource compute kernels of OpenBLAS

2005
Linux kernel for
Mission Critical
Server2010
KVM and
Virtualization

2018

2022

OpenStack,Kubernetes

Automotive Grade Linux,
Yocto, Arm Linux on
Supercomputer Fugaku

oneAPI Interface
Machine Learning Workloads
scikit-learn Algorithms
oneDAL LibraryIn
progress• Expand Arm software ecosystem• Feature development• Quality enhancement
with

Arm

UXL

LinaroIntel MKLOpenBLAS

DFT, RNG
VSL

Open-source
Kernels

x86
CPU
30
30

Arm
CPU

Arm
CPU

Fujitsu enabled core technologyof oneDAL© 2024 Fujitsu Limited© 2024 Fujitsu Limited

Fujitsu’s Contribution to Carbon Neutrality
The development of FUJITSU-MONAKA is under NEDO program(*1).
NEDO aims energy savings of 40% or more in datacenters by 2030 to realize a carbon-neutral society.

Fujitsu will contribute to the project by developing an energy-efficient CPU,

FUJITSU-MONAKA.
Refine Fujitsu’s proven leading-edge microarchitecture.
Cooperate with other technologies such as low-power consumption accelerator,
photoelectric fusion device, wideband SSD, photonics smart NIC and disaggregation.
(details on Fujitsu press release, https://www.fujitsu.com/global/about/resources/news/press-releases/2022/0225-01.html )(*1) NEDO program“Technology Development of the Next Generation Green Data Center” for the “Green
Innovation Fund Project/Construction of Next Generation Digital Infrastructure”• NEDO is "New Energy and Industrial Technology Development Organization", a national research and development agency in Japan.
• Fujitsu has been selected for the national initiative along with NEC Corporation, AIO Core Co., Ltd., KIOXIA Corporation, Fujitsu Optical
Components Limited and KYOCERA Corporation.

31© 2024 Fujitsu Limited

Look Back at History ofSupercomputer Architecture32© 2024 Fujitsu Limited

Technology Trend Overview
From the CDC 6600, often considered the first supercomputer (1964), to today, supercomputer architecture has been
driven by the relentless pursuit of parallelization. This journey has witnessed three major turning points.
1960

1970

1980

1990

1995

2000

2005

1993 TOP500 begins

Scalarprocessing

2010

2015

2020

2024

2007 Green500 begins

CDC6600
1964

2020 HPL-MxP begins

2010 Graph500 begins

Vectorprocessing

Earth
Simulator
2002

Ceay-1
1976

Scalar Processingin Massively Parallel Systems

ASCI Red
1997

Fugaku2020

GPGPU
First human spaceflight

Apollo 11 moon landing

Fall of the Berlin WallTSUBAME
1.2
2008

Birth of Deep Learning

Windows 95 release

33

Frontier
2022

COVID-19 outbreakParis Agreement signed

ChatGPT release© 2024 Fujitsu Limited

Scalar processing

Scalar Processing in
Massively Parallel
Systems

Vectorprocessing

Heterogeneous
Computing

The CDC 6600 (1964), often considered the first supercomputer, was a groundbreaking machine that laid the foundation
for modern supercomputing. It employed a scalar processor architecture, meaning it processed instructions one at a time,rather than in parallel like later vector processors.
• First supercomputer (1964):•

Configuration diagram

Utilized a scalar processor architecture.
Peripheral Processors• Unique architecture:•

Central Processor (CP) for instruction execution andcalculations.

Central Processor•

Peripheral Processors (PPs) for I/O operations, freeing
the CP.

Operatingregister•

Main Memory shared by CP and PPs.

Main
memory• Efficient parallel processing despite scalar architecture.
RISC-like design with a simplified instruction set.

•

SMT precursor for handling multiple instruction streams
concurrently.

Exec

Exec

• Innovative features:•

Exec• Paved the way for future supercomputer advancements.
34© 2024 Fujitsu Limited

Scalarprocessing

Scalar Processing in
Massively Parallel
Systems

Vector processing

Heterogeneous
ComputingThe 1970s witnessed the emergence of vector processing, revolutionizing scientific computing. Pioneered by machines
like the Cray-1 (1976) and Fujitsu's FACOM 230-75 APU (1977), vector processors boasted specialized hardware foraccelerating computationally intensive tasks.
• Vector Registers & Units: Efficient manipulation of data
elements in arrays.

Example of configuration diagram
Vector Unit

• Bulk Data Access: Fetching entire vectors from memory
with a single instruction.

• Enhanced Parallelism: Executing multiple instructions
concurrently, boosting processing power.
• Loop Optimization: Vector units targeted loop-boundcomputations, while scalar units handled other tasks.

Image of instruction pipelineMaskregisterMaskfor(int i = 0; i < 4; i++) {c[i] = a[i] + b[i];f[i] = d[i] + e[i];}VectorIF

Vector registerMainmemory

FMA/
FEX
Load/
Store

DIV/
FQRT

IF

Scalar Unit
L2$

L1$

Scalar register
ScalarExec

DEC EXEC
EXEC
EXEC
EXEC

WB
WB
WB
WB

ScalarIF• This combination of hardware and efficient data handlingmade vector processors ideal for scientific applications
involving large datasets and repetitive operations. They
paved the way for further advancements insupercomputing architecture.

DEC EXEC WBEXEC WBEXEC WBEXEC WBDEC EXEC WB
IF DEC EXEC WB
IF DEC EXEC WB
IF DEC EXEC WB
IF DEC EXEC WB
IF DEC EXEC WB
IF DEC EXEC WB
IF DEC EXEC WB*VPP500035© 2024 Fujitsu Limited

Scalarprocessing

Scalar Processing in
Massively Parallel
Systems

Vector processing

Heterogeneous
Computing• Multi-vector-unit approach: This approach increased parallelism, boosting performance.
• memory exploration: Companies experimented with unique memory configurations for optimal performance and resource utilization.

• Fujitsu's VPP Architecture: distributed memory parallel
computer• Multiple processors with independent memory accesseliminate the central memory bottleneck, enhancing
scalability.

PE

PE
VU

• Flexible allocation of tasks and data among vector units
for optimal resource utilization.

DTU MSU• Enabled construction of powerful and efficient
supercomputers.
VPP demonstrated the effectiveness of distributed memoryarchitectures, which were a notable approach in earlysupercomputers.

SU

PE
VU

DTU MSUSUIOC

VU
DTU MSUSUIOC*VPP500036© 2024 Fujitsu Limited

Scalarprocessing

Vectorprocessing

Scalar Processing in
Massively Parallel Systems

Heterogeneous
Computing

The relentless pursuit of performance in supercomputing led to the development of specialized vector processors.
However, the increasing cost and complexity of vector architectures, coupled with the rapid advancements in scalar
processor technology, eventually shifted the landscape towards scalar-based supercomputers.
Vector processing• Cost-effectiveness: Scalar processors surpassed vector
processors in price-to-performance ratio, becoming a more
attractive option for HPC applications.

Off chipOn chip
Vector Unit

Vector register

Load/
Store

• On-chip integration: Unlike vector processors, which faced
bottlenecks due to off-chip components, scalar processors
benefited from on-chip integration of key performanceelements, aligning with Moore's Law for lower cost andhigher performance.

MainmemoryVectorExec

Scalar Unit
Scalar register
L2$

Scalar
Exec

L1$

Scalar processing• Large cache capacity: On-chip caches reduced the needfor main memory accesses, further boosting performance.

Off chip• These factors, in contrast to the limitations of vector
processing, enabled the development of more powerfuland efficient supercomputing systems based on readily
available, cost-effective, and highly scalable scalar
processors. This shift marked a turning point insupercomputing architectures.

On chip
Scalar Unit
Scalar registerMainmemory

37

L2$

L1$

ScalarExec© 2024 Fujitsu Limited

Scalarprocessing

Vectorprocessing

Scalar Processing in
Massively Parallel Systems

Heterogeneous
Computing• Around 2010, the end of Moore's Law began to be discussed, marking the slowdown of performance improvements throughfrequency scaling. This shift led to the exploration of new parallelism approaches, as exemplified by the development of Fugaku.
Large-Scale CPU Parallelization

Multi / Many Core Processors

SIMD Instructions• Interconnecting numerous general-purpose
processors through high-speed communication
networks.

• The A64FX processor itself is a prime example
of a multi-core processor, featuring 48 cores
per chip. This design allows for increased
parallelism within each node of Fugaku.

• Equipping scalar processors with vector-like
instructions to enhance parallel processing
capabilities.(Examples include ARM SVE, x86
AVX, and RISC-V RVV.)• This approach is driven by advancements inpackaging technology, cooling solutions, andthe present-day demand for cost-efficient
cloud computing, where multiple virtual
machines can be run on a single processor.

• For instance, the A64FX can perform 8 FP64
operations or 64 FP8 operations simultaneously
per core, demonstrating the potential of SIMD
for accelerating various types of calculations,
including those involving different precisionlevels.

• Fujitsu's Tofu interconnect technology, with its
6D mesh/torus structure, enabled theconnection of 393,216 nodes (1,024 racks) in
the FX1000 supercomputer.
• This approach offered high inter-nodecommunication speeds and fault tolerance.

512bit SIMD (Arm SVE)
FP64 x8
FP32 x16
FP16 x32
FP8 x64
A64fx
38

A64fx
© 2024 Fujitsu Limited

Scalarprocessing

Vectorprocessing

Scalar Processing in
Massively Parallel
Systems

Heterogeneous Computing

Heterogeneous computing, the use of multiple types of processors to achieve high performance, has become increasingly
prevalent in supercomputing. Among the various forms of heterogeneous computing, the use of Graphics Processing
Units (GPUs) has been particularly significant.
• The Rise of GPGPU:•GPGPU gained traction in the early 2000s, leveraging
GPUs for general-purpose computations.

•

Large-scale HPC systems like TSUBAME1.2 and Tianhe-1
adopted GPUs around 2010.
NVIDIA's CUDA platform (2007) made GPUs moreaccessible to HPC users.

•

Deep learning and cryptocurrency mining fueled GPU
development.

SIMT
Single Instruction, Multiple Thread
ThreadThreadThreadThread

Data1
Data2

• Driving Factors:•

SIMD
Single Instruction, Multiple Data

Exec
Resultexample of GPU architecture *NVIDIA TeslaCPU
Mainmemory• HPC adapts to GPU: GPU vendors have made HPC
advancements like FP64 support. However, specializedarchitectures like NVIDIA's SIMT may not be ideal for HPC.
Users may need to adapt their workflows.

GPU
SharedSM Memory
Devicememory• Impact on Supercomputing : GPUs have significantly
impacted supercomputing, enabling more powerful andefficient systems.

SharedSM Memory
39

SP

SP

SP

SP

SP

SP

SP

SP

SP

SP

SP

SP

SP

SP

SP

SP

© 2024 Fujitsu Limited

Scalarprocessing

Vectorprocessing

Scalar Processing in
Massively Parallel
Systems

Heterogeneous Computing

The growing demand for AI, particularly deep learning and LLMs, has driven advancements in supercomputing
architectures. This focus has spurred advancements in both specialized AI architectures and traditional HPC architectures,
exemplified by Frontier, the world's first exascale supercomputer with an AMD CPU+GPU architecture.
• AI Workloads and Heterogeneous Computing:•

AI workloads often rely on low-precision floating-point
arithmetic, which benefits from GPUs.

•This focus on AI continues to lead to improvements in
HPC workloads due to shared advancements inparallelism and memory technology.

Discrete GPU• Convergence of CPU and GPU:•

Vendors are introducing solutions that integrate CPUs
and GPUs within a single package to further optimize
performance.

• Impact on Supercomputing:
•

•

GPU

HBM

HBM

GPU

GPU

HBM

HBM

CPU
cores

GPU
cores

DDR/
HBM

HBM

CPU

AI continues to significantly influence supercomputing
architectures, leading to advancements in parallelism,
memory bandwidth, and heterogeneous computing.
These advancements enable the development of more
powerful and versatile supercomputers for bothtraditional HPC and emerging AI workloads.

GPU

Packaged GPU

DDR

40

© 2024 Fujitsu Limited

TOP10 Architectures (Jun 2024)
1.Frontier (2022~)

2.Aurora (2023~)
2.Aurora

3.Eagle (2023~)

4.Fugaku (2020~)

5.LUMI (2022~)• 1,206.0 PFLOPS (70.3%)• 22.79 MW• 9,408 nodes• 1 cpu + 4 gpu / node
• AMD EPYC
• AMD MI250X•585.31012.0PFLOPS(51.1%)PFLOPS(55.2%)•24.6938.7MWMW• around 11,000 nodes *1•10,624
2 cpunodes+ 6 gpu / node
• Intel
2 cpu+ 6Xeon
gpu /CPU
nodeMax• Intel Data Center GPU MAX• 561.2 PFLOPS (66.2%)• N/A MW• around 1800 nodes *1• 2 cpu + 8gpu / node
• Intel Xeon Platinum• NVIDIA H100

• 442.0 PFLOPS (82.2%)• 29.9 MW• 158,976 nodes• 1 cpu / node
• A64FX• 379.7 PFLOPS (71.4%)• 7.11 MW• 2,560
2,978 nodes• 1 cpu + 4 gpu /node/ node• AMD EPYC
• AMD MI250X

9.Summit (2018~)

10.Eos NVIDIA DGX SuperPOD(2023~)• 121.4 PFLOPS (64.3%)• N/A MW• 576 nodes *1• 2 cpu + 8 gpu / node
• Intel Xeon Platinum• NVIDIA H100

NVSwitch

6.Alps (2024~)

7.Leonardo (2022~)• 270.0 PFLOPS (76.3%)• 5.19 MW• 2,688 nodes• 4 x (cpu + gpu) / node
• NVIDIA GH200• 241.2 PFLOPS (78.7%)• 7.49 MW• 3,456 nodes• 1 cpu + 4 gpu / node
• Intel Xeon Platinum• NVIDIA A100

*1

8.MareNostrum 5 ACC(2023~)• 175.3 PFLOPS (70.2%)• 4.16 MW• 1,120 nodes• 2 cpu + 4 gpu /node• Intel Xeon Platinum• NVIDIA H100

*1

• 148.6 PFLOPS (74%)• 10.1 MW• 4,608 nodes• 2 cpu + 6 gpu / node
• IBM POWER9• NVIDIA GV100*1

NVSwitch*1 : The details have not been made public, so this is my speculation.

41© 2024 Fujitsu Limited

From Historical Review toFuture Speculation42© 2024 Fujitsu Limited

Unveiling the Next Era: What Will It Be Called?
Supercomputers have seen three major turning points, ushering in new eras every 10-20 years. Historically, this timeframe
has proven sufficient for significant shifts. Sixteen years have passed since GPGPUs entered large-scale configurations,indicating that we are likely on the cusp of another major transition.
1960

1970

1980

12 years
Scalarprocessing

1990

1995

2000

21 years

2005

2010

11 years

2015

2020

2025

2030

2040

16 years elapsed

CDC
6600
1964

Vectorprocessing

Earth
Simulator

Ceay-1
1976

Scalar Processingin Massively Parallel Systems

2002
ASCI
Red
1997

Fugaku2020
TSUBAME
1.2
2008

GPGPU

Frontier

2022

?????
43

???

© 2024 Fujitsu Limited

Unveiling the Next Era: What Will It Be Called?
Supercomputers have seen three major turning points, ushering in new eras every 10-20 years. Historically, this timeframe
has proven sufficient for significant shifts. Sixteen years have passed since GPGPUs entered large-scale configurations,indicating that we are likely on the cusp of another major transition.
1960

1970

1980

12 years
Scalarprocessing

1990

1995

2000

21 years

2005

2010

11 years

2015

2020

2025

2030

2040

16 years elapsed

CDC
6600
1964

Vectorprocessing

Earth
Simulator

Ceay-1
1976

Scalar Processingin Massively Parallel Systems

2002
ASCI
Red
1997

Fugaku2020
TSUBAME
1.2
2008

GPGPU

AI-Empowered
Computing44

Frontier

2022

???

© 2024 Fujitsu Limited

The focus of GPGPU is shifting towards AI.
AI is a lucrative market, driving industry focus towards low-precision computing. Double-precision computing, essential forscientific research, is neglected. Scientists must adapt to this shift by exploring alternative solutions.

1

GPU performance development

Slowdown in FP64 Performance development

FP8
Tensor

10000000

2

3

1000000

Rapid TDP Increase
Increased Complexity

FP16 Tensor

1400
1200

100000

1000

10000

800

due to Mixed Vector and Matrix Units
1000

600

FP64

100

While low-precision computing advances rapidly, FP64
stagnates, and rising TDP limits its future.
This necessitates exploring alternative solutions,
including leveraging low-precision instructions forhigh-precision scientific computing.

400

10
1
2005

45

200

TDP
2010

2015

2020

0
2025

© 2024 Fujitsu Limited

What Should HPC Users Do?
● Software● Explore the development of new libraries that use low-precision computing toachieve high-precision results.

● Develop applications like HPL-MxP that inherently use mixed-precision computing.
● Explore new AI solutions like surrogate models for high-precision computing.

● Hardware● Focus on the improved FP64 performance of CPUs and consider a return to CPUbased computing when necessary.

● Pressure GPU vendors to improve the FP64 performance of GPUs.

46© 2024 Fujitsu Limited

Practical Examples of AI-Empowered Computing
I believe that AI for HPC, which incorporates AI workloads into simulations, will become mainstream. In particular, I see
potential in combining surrogate models with simulations, where the majority of the processing is done with a rough preprocessing using deep learning, and the parameters obtained there are used to improve the accuracy with FP64 instructions.
With numerous successful precedents emerging, this approach is likely to become widely adopted in the near future.
GPGPU

AI-Empowered Computing• Conducts countless scientific simulations with high-precisioncalculations until convergence is achieved.

• Leverages low-precision calculations in a preprocessing stage torefine approximate values, followed by high-precision scientificsimulations in a post-processing stage.

• Requires high-fidelity models to be computed with high precision to
obtain promising results.

• Performs the majority of calculations with low precision using asimplified model (surrogate model), leading to enhanced
computational efficiency.

Simulation

Simulation

AI estimation

AI Inference

Simulation

High-precision
simulation

High-precision
simulation

Low-precision
create surrogate
model

Low-precision
use surrogate
model

High-precision
simulation47© 2024 Fujitsu Limited

Architectures suitable.
In pursuit of high-performance and energy-efficient architectures, the ideal form will diverge from the present. The market
will demand PURE Deep Learning Accelerators, stripped of unnecessary AI features. Additionally, reduced high-precisioncomputations will enable efficient execution on CPUs, leading to a resurgence in CPU performance requirements.
GPGPU

AI-Empowered Computing
High-precision
simulation

CPU

High-precision
simulation

Many FP64 units

GPU
GPU
GPU
GPU

Many low-precision
matrix unitGPU
DeepGPU
Learning
PURE
GPU

CPU

HBM

Low-precision
surrogate model

Accelerator

HBM

High bandwidth
Large Capacity

High bandwidth
Large Capacity

DDR

HBM
High bandwidthHigh bandwidth
Many FP64 units
48© 2024 Fujitsu Limited

Conclusion● Fujitsu's supercomputing legacy, from early machines to Fugaku, continues toinfluence data center technology like the FUJITSU-MONAKA.

● Supercomputer architecture has undergone several major transitions, driven by theneed for increased performance and efficiency.

● The market is changing, and supercomputing is once again at a major turning point.
It presents both challenges and opportunities.

● Looking ahead, advancements in AI and other technologies will drive exciting new
possibilities for supercomputing.

49© 2024 Fujitsu Limited

Thank you

50© 2024 Fujitsu Limited