New Challenge for HPC and AI
by Big Memory (Data) Supercomputer Pegasus

Taisuke Boku
Director, Center for Computational Sciences, University of Tsukuba(slides courtesy by O. Tatebe, R. Kobayashi and A. Nukada)2023/04/19

1HPC-AI Advisory Council Japan 2023
Center for Computational Sciences, Univ. of Tsukuba

CCS at University of Tsukuba
Center for Computational Sciences
n Established in 1992n

12 years as Center for Computational Physics
n Reorganized as Center for Computational Sciences in 2004nn

Daily collaborative researches with two kinds of faculty
members (45 in total)
Computational Scientists
who have NEEDS (applications)n Computer Scientists
who have SEEDS (system & solution)nn

One of national supercomputer centers under MEXT,
but we are Research Center (others are service centers)

22023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

History of PACS (PAX) series development at CCS
nnn

1978
1st gen: PACS-9

1977: research started by T. Hoshino and T. Kawai
1978: PACS-9 (with 9 nodes) completed
1996: CP-PACS, the first vendor-made supercomputer at CCS, ranked as #1 in TOP500
1980

2nd gen. PACS-32

Year
1978
1980
1983
1984
1989
1996
2006
2012~1320142019

1989
5th gen, QCDPAX

Name
PACS-9PACS-32
PAX-128
PAX-32JQCDPAX
CP-PACS
PACS-CS
HA-PACSCOMA (PACS-IX)
Cygnus (PACS-X)

1996
6th gen: CP-PACS
Ranked #1 in TOP500

Performance
7 KFLOPS
500 KFLOPS
4 MFLOPS
3 MFLOPS
14 GFLOPS
614 GFLOPS
14.3 TFLOPS
1.166 PFLOPS
1.001 PFLOPS
2.5 PFLOPS

2006

2012~2013

7th gen: PACS-CS

8th gen: GPU cluster HA-PACS

nnnn

2014

2019

9th gen: COMA

10th gen: Cygnus

co-design by computer scientists and computational
scientists toward “practically high speed computer”
Application-driven development
Sustainable development experience
Two streams of supercomputer operationn
Our own unique strategy for advanced
research → Cygnus, Pegasus
nJCAHPC: widely spreading supercomputerresource service → OFP, OFP-II (planned)

HPC-AI Advisory Council Japan 2023

32023/04/19

Center for Computational Sciences, Univ. of Tsukuba

HPC technology contributing to AI: Computationn

Neural Network Processing
early ’80s, Neural Network started to be studied for machine learning
supported by only poor processing power (CPU), no special hardware
just one middle layer, and results are not sufficient (low computation power)⇨ not a Deep Learning (deep layered CNN-base machine learning)nnnn

Accelerators, especially GPU
after GPU became attractive for numerical computing, GPU started to be introduced for NNvery regular and large capacity of computing ⇨ good for SIMD implementation
GPU is now the main player of AI (DL) and GPU vendors continue to build AI-oriented GPU as well as HPC use:
NVIDIA and AMD (followed by Intel too)nnnn

CPU instruction set
to support ML, FP16 (16-bit half precision) and BFloat16 (long mantissa) are introduced⇨ recently with FP8SIMD vector instruction is good to support them in up to 512bit

nn4

2023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of TsukubaAI contribution to HPC: efficient data analysis and reductionnefficient parameter space searchnreducing the parameter search space by machine learning
climate simulationn astrophysics
n life sciencenndata matching on large data spacenn

text base docking for creation of medicine

efficient data analysis
finding the characteristics of phenomenan machine learning base data sorting and collectionnnsurrogate modeling

HPC for AI ⇔ AI for HPC
52023/04/19

HPC-AI Advisory Council Japan 2023

⇨ GPU is important (of course!) but not enoughCenter for Computational Sciences, Univ. of Tsukuba

Why we need Big Memory
CPU/GPU Performance andMemory size per core

CPU performance 50x, butmemory size 3.8x in 8 years
Fugaku1,000
n It matters for Data-driven andSummit
514
AI-driven Science
200
CPU/GPUperformance[PFLOPS]n Memory size and Storage
100n

performance are really
importantnIntroduce Persistent Memory(PMEM)

10Memory mode for memorysize and direct mode forstorage performance

1n25K101.90

2023/04/19

HPC-AI Advisory Council Japan 2023

1.84

Memory size per core [GB]

02012
6

Oakforest-PACS/Cori

2014

2016

0.75

Fugaku0.64

Summit2018

2020

Center for Computational Sciences, Univ. of Tsukuba

Persistent Memory
One order better price/capacityn Minimum latency is ~60 ns (similar to DRAM)n ~Half of bandwidth
Bandwidthn Memory modenLarger memory space without much
performance penalty
n Possible to use DRAM as last level cachenn

App direct moden

Direct access to byte-addressable persistentmemory and high-performance storage

1 TB/sDRAMSSD

1 GB/sHDD

1 MB/s1 ns

72023/04/19

HPC-AI Advisory Council Japan 2023

PMEM

1 µs

1 mslatency

Center for Computational Sciences, Univ. of Tsukuba

Large dataset science (1): Genomic data accumulation and analysis
environmental genome analysis ⇒ specific genome analysis
n large dataset ⇒ distributed processing with queryn large data capacity on computation node enhances
performancen82023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

Large dataset science (2): Local climate simulation
Ultra high resolution of local climate simulation by multi-physics
City-LES: urban climate simulation by LES, solar effect, building structure,surface materialn

TOKYO2020 model around Tokyo Station, 1m gridsurface temperature

3D distribution of temperature

東京駅• large scale in-transit analysis on large capacity date• completely GPUized (up to 15x performance of CPU)

92023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

How to use GPU + Big Memory (PMEM) for HPC/BD/AI ?
nGPUtotally important for both HPC and AIn powerful GPU contributes HPC
AI couplingnnUltra large capacity of medium~high speed memorynn

In-Situ processingnn

astrophysics, climate, life science: requiring large capacity of working set withrelatively low FLOPS requirement
bypassing the slow in/out of large capacity of file data
⇨ PMEM technology support

High-speed distributed storage
n10even as a distributed storage, much faster solution is possible (shown later)
2023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

Cygnus (PACS-X): Extreme Computing with multi-hybrid accelerators

System integration
by NECThe world first practical supercomputer with Multi-Hybrid (GPU + FPGA)
Accelerating Architecture: 320 GPUs (V100) + 64 FPGAs (Stratix10) in 80 nodes
11

2023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

Pegasus (PACS-XI) : Big Memory SupercomputernStrategy of current Cygnus (GPU+FPGA)
accelerating traditional HPC, especially for multi-physical simulation with multiple
phenomena, by coupling of GPU + FPGAn GPU: SIMD-type of spatial parallelism
FPGA: pipelining x spatial parallelism
n GPU and FPGA compensate with each other to fill the gap for various parallelism invarious applications or even in an applicationn AI (deep learning) is mainly done on GPU, and FPGA partially support (ex. sorting ondatabase search)nn

New concept of Big Memory Supercomputer
much larger simulation on HPC applications: astrophysics, climate, biosciencen much faster distributed file system with large scale cluster computing
n122023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

Pegasus: world first combination of H100+SPR+PMEM+NDR200ntotal Performancenn

node componentsnnnn

NVIDIA H100 PCIe (gen5)
Intel Sapphire Rapids (SPR)Intel Optane ver3

Interconnection Networknn

120 nodes, 6.5 PFlops, 240 TiB (PMEM)

NDR200: NVIDIA Quantum-2 IBwith 200Gbops full bisection b/w

parallel file system (DDN)
nSPR
CPU

H100
GPU

DRAM 128 GB
PMEM 2 TB

120 nodes

7.1 PByte, 40 GB/s

HPL: 3.47 PF (54%) (as on Feb. 2023)
MPI pingpoing: 23.9GB/s (95.7%)
13

NDR200 full-bisection
interconnection

2023/04/19

HPC-AI Advisory Council Japan 2023

NVMe SSD 6.4 TB

System integration
by NEC

Center for Computational Sciences, Univ. of TsukubaCombination of SPR + Optane300 + H100+48 core, 2.1GHz••••
14+256GB PMEM + 16GB DRAMx 8 modules

26 TFLOPS (FP64)
51 TFLOPS (FP64-Tensor)
80 GB HBM3 (2TB/s)

all the brand-new parts combination (first in the world)
large capacity memory + high speed CPU and GPU for HPC
large capacity memory + ultra high speed tensor calculation for AI
MemVerge software supports straight-forward extension of current app. to Optane

2023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

Pegasus outlook

file server

Pegasus at CCS, Univ. of Tsukuba (2022/12~)

152023/04/19

HPC-AI Advisory Council Japan 2023

computation nodes (120)

IB switch+login serverCenter for Computational Sciences, Univ. of Tsukuba

Computation node

GPU (H100)
Network (IB NDR200)

CPU (SPR)

SSD
DRAM and PMEM

16

2023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

Comparison of Cygnus and Pegasus#nodesPFLOPS (DP)CPUGPUFPGA (SP)DRAM (TiB)
PMEM (TiB)
Storage (PB)

172023/04/19

Cygnus (2019)

Pegasus (2022)

81 (C: 162, G: 324)
2.30.16
2.18
0.64
10.20
2.4120 (C: 120, G: 120)
6.5 (2.8x)
0.5 (3.1x)
6.0 (2.7x)
030.7 (3.0x)
240
7.1 (3x)HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

Cygnus and Pegasus

Cygnusblue Cygnus and red Pegasus
are the sister systems

Pegasus182023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

Pegasus & Cygnus : twin system for Extreme Computing and Big Data

6.5 PFlops
GPUPmem

7.1 PBEXA Scalar192023/04/19

cross mounting (200Gbps)

HPC-AI Advisory Council Japan 2023

2.3 PFlops
GPUFPGA

2.4 PBEXA ScalarCenter for Computational Sciences, Univ. of Tsukuba

Research of ad hoc parallel file system
Temporal parallel file system using
node-local storagen Fill the performance gap between
CPU/GPU and storagen• We are developing CHFS (Consistent Hash File System) ad hoc file system toutilize persistent memory• No metadata server, no sequential processing for performance and scalability* O. Tatebe, et. al., "CHFS: Parallel Consistent Hashing File System for Node-local Persistent Memory”, HPC Asia 202220

2023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

System Architecture of CHFS
application
application

chfs libchfs lib
chfsd
chfsd

pmem

application
chfs lib
chfsd
pmem

application
chfs libchfs lib

chfsd
pmem

pmem
application
chfs lib
chfsd
pmem

21

2023/04/19

application
chfs lib
chfsd
pmem

HPC-AI Advisory Council Japan 2023

application
chfs lib
chfsd
pmem

Compute nodes

application

application
chfs lib
chfsd
pmem

chfsd
pmem
application
chfs lib
chfsd
pmem

Center for Computational Sciences, Univ. of Tsukuba

GPU performance (preliminary): V100/Cygnus vs H100/Pegasus(by R. Kobayashi@CCS)n

ARGOT: Astrophysical fundamental simulation code for early stage universeto analyze the born of first stars and galaxies

ARGOT method

ART method

1.86x faster on H100

ARGOT method: point source radiation
ART method: photon diffusion radiation

heavy computation on GPU

222023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

DO CONCURRENT preliminary evaluation (by A. Nukada@CCS)
nOpenSWPC: seismic simulation of earth quake wave propagation(original code by T. Furumura et.al.
= Fortran + OpenMP + MPI)

GPU/parallelization by
CUDA Fortran + OpenACC + MPIPegasus
CPU: Intel SPR 48core 2.1GHz
GPU: H100 PCIe
IB: NDR200 (Quantum-2 IB)

23

2023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

Utilization of PMEM
nLarge memory
MemVerge – using PMEM (2TB) in behind of DDR (128GB) to use DDR memory as“4th cache” to make balance between capacity and speedn just declaration on job script, no need for reprogrammingnn

High speed I/Omounted as in the same manner with local SSD
n will be used under CHFS ad hoc distributed file system
nn

Mixed
it is possible to provide some fraction for memory and other for storagen storage solution is “ad hoc” – life time is limited to the same job
n24

2023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba

Summarynnnnnn

AI is the latest important application and gets growing rapidly as a practical application in humanlife
HPC technologies have been contributing to AI (HPC for AI) so far, and now it’s time to use AI
technologies for efficient HPC solutions (AI for HPC)
Gap between computation performance and memory capacity is so serious
Utilizing Persisten Memory (PMEM) both for large capacity memory and high performance shared
file system (ad hoc) simultaneously, including efficient in-situ processing
GPU continues to play an important role both for HPC and AI⇨ high performance GPU system with large capacity memory is ideal
Variation of coverage is so wide to support large scale data science and simulation, based on
HPC/BD/AI and GPU/PMEM combination⇨ Pegasus (started official operation from April 1st 2023)
25

2023/04/19

HPC-AI Advisory Council Japan 2023

Center for Computational Sciences, Univ. of Tsukuba