Persistent Memory Supercomputer
Pegasus for data-driven andAI-driven ScienceOsamu Tatebe
Center for Computational Sciences, University of Tsukuba

Pegasus background
CPU/GPU Performance andMemory size per core

• CPU performance 50x, but
memory size 3.8x in 8 years• It matters for Data-driven 1,000and AI-driven Science• Memory size and Storage
performance are reallyimportant• Introduce PersistentMemory• Memory mode for memory
size and direct mode forstorage performance

100Summit

Fugaku514

200

CPU/GPU performance [PFLOPS]
2510
1010

Oakforest-PACS/Cori

1.90

1.84

KMemory size per core [GB]
2012

2014

2016

0.75

Summit2018

Fugaku0.642020

Design Goal of Pegasus• Accelerates large-scale data analysis and big data AI by utilizing
persistent memory for large memory space and high performancestorage• Fosters new fields of large-scale data analysis, new applications of big
data AI, and system software research

Persistent Memory• One order better cost performance• Minimum latency is ~60 ns (similar to DRAM)• Half of bandwidth
Bandwidth• Memory mode
• Larger memory space without much
performance penalty• App direct mode

• Direct access to byte-addressable persistentmemory and high-performance storage

1 TB/sDRAM

PMEMSSD

1 GB/sHDD

1 MB/s1 ns

1 μs

1 mslatency

Pegasus Highlights• Plans to build with Intel 4th Gen Xeon, NVIDIA H100 Tensor Core GPUwith PCIe, and 2 TiB Intel Optane PM 300 series will strongly drive Big
Data and AI• The world's first system with NVIDIA H100 PCIe GPUs connected via
PCIe Gen5• First system announced in Japan that will utilize NVIDIA Quantum-2
InfiniBand networking

NEC LX B1000E Blade Enclosure

Pegasus Specification
NEC LX 102Bk-6• Will be installed in 4Q 2022
• Total Performance

200Gbps full bisection• 120 nodes, > 6.1 PFlops, 240 TiB Pmem

• Node specification• Intel 4th Gen Xeon Scalable Processor• 51 TFlops NVIDIA H100 Tensor Core GPU• 128 GiB DDR5 DRAM
• 2 TiB Optane PM 300 series (16x DRAM)• 6 TB NVMe SSD (7 GB/s)• Interconnection Network4th GenXeon

H100Tensor
Core GPU

DDR5 128 GiB• NVIDIA Quantum-2 InfiniBand platform(200 Gbps) full bisection

Optane PM 300
2 TiB• 7.1 PByte DDN EXAScaler (40 GB/s)

NVMe SSD 6 TB

• Parallel File System120 nodes

How to use Persistent Memory
128 GiB DDR5

2 TiB Optane PM 300 series• All PM regions are configured in App direct mode
• Users can specify PM size as extended memory and as persistentmemory
128 GiB DDR5

2 TiB Optane PM 300 series
P GiBytes

Persistent Memory (devdax/fsdax)

128 + P GiBytes Memory• MemVerge Memory Machine is used for memory extension

XSBench Application Benchmark (on Optane PM 100)Proxy application of the Monte Carlo neutrontransport algorithm● Memory usage: large 5.6 GB, XL 120GB, XXL 252GB● Memory access pattern: random read
●Swap

Traditional
swap is slow

Memory Mode

FastMemVerge

MemVerge is 14%
faster than memorymodeComparison of Cygnus and Pegasus
PFLOPS (DP)CPUGPUFPGA (SP)Memory (TiB)
Pmem (TiB)
Storage (PB)

Cygnus (2019)
2.3
0.16
2.18
0.64
15.2
0
2.4

Pegasus (2022)> 6.1 (2.7x)?.?? (?.?x)
6.12 (2.8x)015.36 (1.01x)
240
7.1 (3.0x)

Research of Ad hoc parallel file system• Temporal parallel file system using nodelocal storage• Fill the performance gap betweenCPU/GPU and storage• We are developing CHFS ad hoc file system to utilize persistent memory• No metadata server, no sequential processing for performance and scalability

Design Goal of CHFS [HPC Asia 2022]• Utilize persistent memory performance• In-memory persistent key-value store (not block-based file system)• Reduce metadata overhead and achieve scalable performance
improvement• No dedicated metadata server (no additional lookup for metadata)• no sequential processing and no central data structure• Improve single-shared-file performance• File is divided into fixed-size chunks to distribute a single file among servers and toavoid lock contentions• Based on highly parallel distributed key-value store without any central
data structureDesign of File System• All data is stored in a highly parallel distributed KV store• A single Key-Value format in CHFS for a file chunk and a directory
KeyFull path and chunk number• For a directory, there is no chunknumber and no file data

Value
Metadata

File data

Metadata (64 bytes)
mode, uid, gid, size
chunk size
mtime, ctime• No file-level information in KV store, such as total chunk numbers
and total file size to avoid sequential processing

System Architecture of CHFS
application
chfs lib
chfsd
pmem
application
chfs lib
chfsd
pmem

application
chfs lib
chfsd
pmem

application
chfs lib
chfsd
pmem

application
chfs lib
chfsd
pmem

application
chfs lib
chfsd
pmem

Compute nodes

application
chfs lib
chfsd
pmem
application
chfs lib
chfsd
pmem

application
chfs lib
chfsd
pmem
application
chfs lib
chfsd
pmem

Implementation of CHFS
• Mochi-Margo [JCST 2020]• https://mochi.readthedocs.io/en/latest/• Communication library using Mercury and Argobots• Mercury [Cluster 2013]• Async RPC, RDMA communication library• libfabric, UCX, shared memory plug-ins• Argobots [IEEE TPDS 2018]• Light-weight thread library• Pmemkv – persistent key-value store in PMDKPerformance Evaluation of CHFS
• IO500 benchmark• BW: IOR easy/hard – 4 benchmarks• MD: MDtest easy/hard, Find – 8 benchmarks• Score is the geometric mean ofbenchmark results• 4-node Pmem cluster and78-node Cygnus
3,901 B
IOR hard47,008 B

MDtest hard

IO500 Bandwidth (CHFS/DAOS/GekkoFS)
1 node3 nodes
2 nodes 4 nodes
CHFS GekkoFS
DAOS

CHFS

GekkoFS
9.9x better than DAOS
6.0x better than GekkoFSDAOS

IO500 Metadata Hard (CHFS/DAOS/GekkoFS)

6.0x better than DAOS
4.4x better than GekkoFS

CHFS

GekkoFSDAOSIO500 Metadata on Cygnus (CHFS/BeeOND/GekkoFS)
1,160 K IOPS

64 nodes762 K IOPS
32 nodes𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆 =𝐸𝐸𝐸𝐸 × 𝐸𝐸𝐸𝐸 × 𝐸𝐸𝐸𝐸 ×𝐻𝐻𝐻𝐻 × 𝐻𝐻𝐻𝐻 × 𝐻𝐻𝐻𝐻 × 𝐻𝐻𝐻𝐻 × 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹

8138 K IOPS

73

1 node32 nodes

2139 K IOPS
5.8

64 nodesdoes not work
due to resourceor device busy error

#15 in 10 node list
#23 in full list

Design Goal of CHFS/Cache [ESSA 2022]• Based on CHFS ad hoc PFS [HPC Asia 2022]• Exploit node-local persistent memory• High metadata performance, high bandwidth• Scalable performance• A separate FS from the backend PFS• Design a caching FS by synchronizing with backend PFS• Not sacrifice metadata performance• Relax consistency between backend PFS in easy-to-understand semantics

IO500 Score of Oakforest-PACSBW[GiB/s]Metadata[kIOP/s]

Total

Lustre21.4

88.78*

42.18*

IME

471.2521.85

101.48

Relaxation of Consistency
Assumptions
1. Input data not changed during the execution
2. Creation of new entries succeeds
3. Before updating an existing file, the file is readonce
4. Updates not reflected until flushed(updates can be accessed from login nodes)
5. Flushing performed before the job terminatescacheflush

Backend PFSDesign of File Operation• File Open and Read• Right figure (From Assumption 1)• If exist in caching PFS,
no checking performed for backend PFS• File Creation and Write• Creates and writes in caching PFS(From Assumptions 2-4)
• No checking performed for backend PFS

Exist incaching PFS?

yes

open/readno
Exist in PFS?

noNo such entry

yesCache andopen/readMetadata performance on Cygnus P nodes (4 nodes)
700

CHFS
CHFS/Cache

600
500
400
300
200
1000

MD-E-W MD-E-S MD-E-D

FIND

MD-H-W MD-H-S MD-H-R MD-H-D

Overhead is less than 3% except FIND. It is 55% for FIND, but not problematicSummary• Pegasus will be introduced in 4Q 2022

• Big memory and high-performance storage for data-driven and AI-driven
science• Research of ad hoc parallel file system• Better and scalable performance utilizing persistent memory• #15 in 2021 June IO500 10 node list, #23 in full list

• Design CHFS/Cache caching PFS• Solve the metadata performance problem• Better bandwidth and metadata performance than the backend PFS