High-Performance and Scalable Middleware for
HPC, AI and Data Science on Heterogenous Systems
Talk at the 2023 HPC-AI Advisory Council Japan ConferencebyFollow us on
https://twitter.com/mvapich

Dhabaleswar K. (DK) Panda
The Ohio State University
E-mail: panda@cse.ohio-state.edu
http://www.cse.ohio-state.edu/~panda

Drivers of Modern HPC Cluster Architectures

Multi-/Many-core
Processors

High Performance Interconnects –InfiniBand (DPU), Slingshot<1usec latency, 200-400Gbps Bandwidth>

Accelerators
high compute density, highperformance/watt>9.7 TFlop DP on a chip•

Multi-core/many-core technologies•

Remote Direct Memory Access (RDMA)-enabled networking (InfiniBand, RoCE, Slingshot)•

Solid State Drives (SSDs), Non-Volatile Random-Access Memory (NVRAM), NVMe-SSD•

Accelerators (NVIDIA GPGPUs)•

Available on HPC Clouds, e.g., Amazon EC2, NSF Chameleon, Microsoft Azure, etc.

Frontier

Network Based Computing Laboratory

FugakuSummitJapan – April ’23

SSD, NVMe-SSD, NVRAM

Lumi

2

Broad Challenge:

How to design high-performance and scalable
middleware for HEC systems while taking advantageof heterogeneous (CPU + GPU + DPU/IPU (xPU)) HPC
and Cloud resources?

Network Based Computing Laboratory

Japan – April ’23

3Increasing Usage of HPC, AI, Big Data and Data Science

Deep/
MachineLearning

HPC(MPI, PGAS, etc.)

Convergence of HPC,
Deep/Machine Learning,and Data Science!

(TensorFlow,
PyTorch, cuML,etc.)

Big Data (Hadoop,
Spark), Data
Science(Dask)

Increasing Need to Run theseapplications on the Cloud!!

Can MPI-Driven Middleware be designed and used for all three domains?
Network Based Computing Laboratory

Japan – April ’23

4Presentation Overview•

MVAPICH MPI Library Project–••

High-Performance Support for various CPU, GPU, DPU, and Networking Technologies

HiDL Project–

High-Performance Deep Learning and Machine Learning–

Accelerating Deep Learning with DPU

HiBD Project–

Accelerating Spark with MPI–

Accelerating Data Science Applications with Dask•

Commercial Support and Value-Added Products•

Conclusions

Network Based Computing Laboratory

Japan – April ’23

5Converged Middleware for HPC, AI, Big Data and Data Science

MVAPICH2 &MVAPICH2-DPU
Libraries

Deep/
MachineLearning

HPC(MPI, PGAS, etc.)(TensorFlow,
PyTorch, cuML,etc.)

Big Data (Hadoop,
Spark), Data
Science(Dask)

Network Based Computing Laboratory

Japan – April ’23

6Overview of the MVAPICH Project•High Performance open-source MPI Library•

Support for multiple interconnects–

InfiniBand, Omni-Path, Ethernet/iWARP, RDMA over Converged Ethernet (RoCE), AWS
EFA, OPX, Broadcom RoCE, Intel Ethernet, Rockport Networks, Slingshot 10/11•

Support for multiple platforms–

x86, OpenPOWER, ARM, Xeon-Phi, GPGPUs (NVIDIA and AMD)•

Started in 2001, first open-source version demonstrated at SC ‘02•

Used by more than 3,300 organizations in 90 countries•

Supports the latest MPI-3.1 standard•

More than 1.66 Million downloads from the OSU site•

http://mvapich.cse.ohio-state.edu•

Additional optimized versions for different systems/environments:•

directly•

Empowering many TOP500 clusters (Nov ‘22 ranking)–

MVAPICH2-X (Advanced MPI + PGAS), since 2011–

7th , 10,649,600-core (Sunway TaihuLight) at NSC, Wuxi, China–

MVAPICH2-GDR with support for NVIDIA (since 2014) and AMD (since 2020) GPUs–

19th, 448, 448 cores (Frontera) at TACC–

MVAPICH2-MIC with support for Intel Xeon-Phi, since 2014–

34th, 288,288 cores (Lassen) at LLNL–MVAPICH2-Virt with virtualization support, since 2015–

MVAPICH2-EA with support for Energy-Awareness, since 2015–

46th, 570,020 cores (Nurion) in South Korea and many others–

MVAPICH2-Azure for Azure HPC IB instances, since 2019–

MVAPICH2-X-AWS for AWS HPC+EFA instances, since 2019•

Linux Distros (RedHat, SuSE, OpenHPC, and Spack)Tools:–OSU MPI Micro-Benchmarks (OMB), since 2003–

OSU InfiniBand Network Analysis and Monitoring (INAM), since 2015

Network Based Computing Laboratory

Available with software stacks of many vendors and•Partner in the 19th ranked TACC Frontera system•

Empowering Top500 systems for more than 17 years

Japan – April ’23

7

MVAPICH-3.0a

MVAPICH-Plus 3.0a

• Released on 08/15/2022• Released on 11/11/2022• Based on MPICH 3.4.3• Based on MVAPICH 3.0• Added support for the ch4:ucx and ch4:ofi devices• Advanced MPI with unified MVAPICH2-GDR and
MVAPICH2-X features• Support for MVAPICH2 enhanced collectives over OFI
and UCX• Added support for the Cray Slingshot 11 interconnect
over OFI–

Supports Cray Slingshot 11 network adapters• Added support for the Cornelis OPX library over OFI–• Support for both OFI and UCX• Support for NVIDIA and AMD GPUs
• Support for all HPC interconnects–•

InfiniBand, Omni-Path, ROCE, Slingshot

Optimized designs for HPC, DL, ML, Big Data and Data
Science applications

Supports Intel Omni-Path adapters• Added support for the Intel PSM3 library over OFI–

Supports Intel Columbiaville network adapters• Added support for IB verbs over UCX

Unify MVAPICH2-GDR and MVAPICH2-X– Supports IB and RoCE network adapters

Exploit the benefits of in-network computing capabilities in UCX and OFI transparently
Network Based Computing Laboratory

Japan – April ’23

8Presentation Overview•••

MVAPICH MPI Library Project–

RDMA-Enabled Designs–

CUDA-Awareness Designs for GPUs–

On-the-fly Compression for GPUs–

Accelerating applications with DPU

HiDL Project–

High-Performance Deep Learning and Machine Learning–

Accelerating Deep Learning with DPU

HiBD Project–

Accelerating Spark with MPI–

Accelerating Data Science Applications with Dask•

Commercial Support and Value-Added Products•

Conclusions

Network Based Computing Laboratory

Japan – April ’23

9AMD Milan + HDR 200

4M

1M

256K

64K

16K

4K

1K

4M

1M

256K

64K

Message Size (Bytes)

AMD EPYC 7V73X 64-Core Processor, Mellanox ConnectX-6 HDR HCA

Network Based Computing Laboratory

0
16K

Message Size (Bytes)

23.5GB/s

4K

0

10000
1K

2

MVAPICH2-…

20000

256

4

30000

16

MVAPICH2-2.3.7

4

6

Bandwidth1

Latency (us)Latency

Message Size (Bytes)

Bandwidth (MB/s)

Inter-Node CPU Point-to-Point

256

Message Size (Bytes)

64

23.1GB/s

640

MVAPICH2-…

16

0.5

25000
20000
15000
10000
5000
0
4

MVAPICH2-2.3.7

1

Latency (us)1Bandwidth
Bandwidth (MB/s)

Intra-Node CPU Point-to-Point

Latency

Japan – April ’23

10

MPI Level Latency on Slingshot 11
Medium/Large message Latency

3.5
3
2.5
2
1.5
1
0.5
0

200

MVAPICH2-3.0a
Latency (us)Latency (us)Small message Latency

150

MVAPICH2-3.0a

100
50
0

Message Size (Bytes)

Message Size (Bytes)• 2us inter-node point-to-point latency for small messages
Interconnect : Cray HPE Slingshot 11
Library : MVAPICH2 3.0a
CPU : AMD EPYC 7763 (milan) Processor
Network Based Computing Laboratory

Available with MVAPICH2 3.0a Release

Japan – April ’23

11

MPI Level Bandwidth on Slingshot 11
Uni-directional Bandwidth
25000

50000

MVAPICH2-3.0a

Bandwidth (MB/s)

Bandwidth (MB/s)

30000

Bi-Directional Bandwidth

20000
15000
10000
5000

40000
30000

MVAPICH2-3.0a

20000
10000
0

0

Message Size (Bytes)

Message Size (Bytes)• 23,985 MB/s uni-directional peak bandwidth• 42,034 MB/s bi-directional peak bandwidthInterconnect : Cray HPE Slingshot 11 (200 Gbps)
Library : MVAPICH2 3.0a
CPU : AMD EPYC 7763 (milan) Processor
Network Based Computing Laboratory

Japan – April ’23

Available with MVAPICH2 3.0a Release12

Concept of Non-blocking Collectives
Application
Process

Application
Process

Communication
Support Entity

Application
Process

Communication
Support Entity

Communication
Support Entity

Computation
Application
Process

Communication
Communication
Support Entity

Schedule
Operation

Schedule
Operation

Schedule
Operation

Schedule
Operation

Check if
Complete

Check if
Complete

Check if
Complete

Check if
Complete

Check if
Complete

Check if
Complete

Check if
Complete

Check if
Complete•

Application processes schedule collective operation•

Check periodically if operation is complete•

Overlap of computation and communication => Better Performance•

Catch: Who will progress communication

Network Based Computing Laboratory

Japan – April ’23

13Non-blocking Collectives Support with In-Network Computing
25

60
50

20
15

4.4x

10

30
20

5
0

40

10
4

8

16

32

0

Overall time (us)70Overlap (%)

Overall time (us)

30

100

80

80

60

60

5.9x

40
20
0

4

8

16

•

64

128

256

512 1024 2048 4096

200

Msg size (Byte)# Nodes
MVAPICH2 2.3.7

3240

Overlap (%)

Iallreduce
32 nodes 1 PPN

Ibarrier

MVAPICH2 2.3.7

MVAPICH2 2.3.6

MVAPICH2 2.3.6

With SHARP:– Flat scaling in terms of overall time
– High overlap between computation and communication
Platform: Dual-socket Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz nodes equipped with Mellanox InfiniBand, HDR-100 Interconnect

Network Based Computing Laboratory

Japan – April ’23

14

GPU-Aware (CUDA-Aware) MPI Library: MVAPICH2-GPU•

Standard MPI interfaces used for unified data movement•

Takes advantage of Unified Virtual Addressing (>= CUDA 4.0)•

Overlaps data movement from GPU with RDMA transfers

At Sender:
insideMVAPICH2

MPI_Send(s_devbuf, size, …);

At Receiver:
MPI_Recv(r_devbuf, size, …);High Performance and High Productivity
Network Based Computing Laboratory

Japan – April ’23

15

“On-the-fly” Compression Support in MVAPICH2-GDR
Weak-Scaling of HPC application AWP-ODC on Lassen cluster (V100 nodes)•

MPC-OPT achieves up to +18% GPU computing flops, -15% runtime per timestep•

ZFP-OPT achieves up to +35% GPU computing flops, -26% runtime per timestep
200

70

Baseline (No compression)
MPC-OPT
ZFP-OPT (rate:16)+18%
ZFP-OPT (rate:8)

180
160
140
120

Run Time Per Timestep (ms)GPU Computing Flops (TFLOPS)

•

100
80

+35%

60
40
20
0

8

16

32

64

GPUS

128

256

-15%

60
50

-26%

40
30
20
10

512

0

8

16

32

64

GPUS

128

256

512

Q. Zhou, C. Chu, N. Senthil Kumar, P. Kousha, M. Ghazimirsaeed, H. Subramoni, and D.K. Panda, Designing High-Performance MPI Libraries with On-the-fly Compression forModern GPU Clusters, 35th IEEE International Parallel & Distributed Processing Symposium (IPDPS), May 2021. [Best Paper Finalist]

Network Based Computing Laboratory

Japan – April ’23

16Performance of All-to-All with Online Compression
MVAPICH2-GDR-2.3.7 + Compression (zfp rate: 24)

Latency (ms)

Proposed (zfp rate:24)
100

•

29.2%

10

1

60

Latency (ms)1000

All-to-All runtime / timestep (PSDNS)

16

32

64

GPUs

128

256

40

All-to-All runtime (DeepSpeed)
MVAPICH2-GDR-2.3.7 + Compression(zfp rate:16)
MVAPICH2-GDR-2.3.7 + Compression(zfp rate:8)
Proposed (zfp rate:16)
Proposed (zfp rate:8)

35.8%

20

0

8

16

32

GPUs

64

128

Improvement compared to MVAPICH2-GDR-2.3.7 with Point-to-Point compression– 3D-FFT: Reduce All-to-All runtime by up to 29.2% with ZFP(rate: 24) on 64 GPUs
– DeepSpeed benchmark: Reduce All-to-All runtime by up to 35.8% with ZFP(rate: 16) on 32 GPUs
Q. Zhou, P. Kousha, Q. Anthony, K. Khorassani, A. Shafi, H. Subramoni, and D.K. Panda, “Accelerating MPI All-toAll Communication with Online Compression on Modern GPU Clusters”, ISC ’22.

Network Based Computing Laboratory

Japan – April ’23

Available in MVAPICH2GDR 2.3.7

17

ROCm-aware MVAPICH2-GDR - Support for AMD GPUs
Intra-Node Point-to-Point Latency

Inter-Node Point-to-Point Latency

Allreduce – 64 GPUs (8 nodes, 8 GPUs Per Node)

Bcast – 64 GPUs (8 nodes, 8 GPUs Per Node)Corona Cluster @ LLNL - ROCm-4.3.0 (mi50 AMD GPUs)
Available with MVAPICH2-GDR 2.3.5+ & OMB v5.7+
K. Khorassani, J. Hashmi, C. Chu, C. Chen, H. Subramoni, D. Panda Designing a ROCm-aware MPI Library for AMD GPUs: Early Experiences - ISC HIGH PERFORMANCE 2021, Jun 2021.

Network Based Computing Laboratory

Japan – April ’23

18

Accelerating Applications with BlueField-3 DPU• InfiniBand network adapter withup to 400Gbps speed• System-on-chip containing 16
64-bit ARMv8.2 A78 cores with
2.75 GHz each• 16 GB of memory for the ARM
cores

Network Based Computing Laboratory

Japan – April ’23

19

Staging vs. GVMI
Staged Transfer

NODE 1

Host Memory

NODE 2

Host Memory

DPU Memory

DPU

DPU

2

DPU

DPU

1
DPU Memory

RDMA Read

RDMA Write
NODE 2NODE 1

Host Memory

Host Memory

DPU Memory

DPU

1

DPU

DPU

DPU

TX/RX Unit
DPU Memory

GVMI Transfer
Network Based Computing Laboratory

Japan – April ’23

20

MVAPICH2-DPU 2023.04 Library Release•

Released on 04/02/23•

Supports all features available with the MVAPICH2 release (http://mvapich.cse.ohiostate.edu)•

Novel GVMI-based framework to offload non-blocking collectives to DPU•

Offloads non-blocking Alltoall (MPI_Ialltoall) to DPU

Available from X-ScaleSolutions, please send a note tocontactus@x-scalesolutions.com to get a trial license.
Network Based Computing Laboratory

Japan – April ’23

21

Total Execution Time with osu_Ialltoall (32 nodes), GVMI, BF-2

46%

84%

75%

84%

46%

84%
46%

46%

Benefits in Total execution time (Compute + Communication)
Network Based Computing Laboratory

Japan – April ’23

22

P3DFFT Application Execution Time (32 nodes), GVMI, BF-2
24%
32 nodes with 32 ppn(1,024 processes)

24%
23%

32x32 process grid

Benefits in application-level
execution time
Network Based Computing Laboratory

Japan – April ’23

23

MVAPICH Drives Nuclear Energy Research at Idaho National Lab(INL)

The MOOSE Multiphysics Computational Framework for Nuclear Power Applications: A Special Issue of Nuclear Technology(https://www.tandfonline.com/doi/full/10.1080/00295450.2021.1915487)
MVAPICH Integration for PBS Pro, HPC Team, Idaho National Laboratory(http://mug.mvapich.cse.ohio-state.edu/static/media/mug/presentations/21/inl.pdf)

Network Based Computing Laboratory

Japan – April ’23

24

Rapid adoption of MVAPICH2 on INL HPC systems
MPI library usage 1 Jan 2021—1 Sep 2021
1000000

100000

10000

1000

100

10

1

mvapich2/2.3.5 mvapich2/2.3.3 openmpi/4.0.2 openmpi/4.0.5

intelmpi

M. Anderson, Aggressive Asynchronous Communication in the MOOSE framework using MVAPICH2, 10th Annual MVAPICH User
Group Conference (MUG), Aug 2022
Network Based Computing Laboratory

Japan – April ’23

25

MVAPICH2 enabling life-changing NASA's DART mission•

Near-Earth asteroids (NEAs) have caused recent
and ancient global catastrophes– LLNL scientists research ways to prevent NEAs using
methods known as asteroid deflection– Joint NASA-LLNL research modelled various
asteroid deflection methods (NASA's DART mission)•

MVAPICH2 lived at the core of the (NASA-DART
mission) and enabled scalability– Underneath large-scale hydrodynamical andgravitational simulations required to compute theimpact such as Spheral models•https://twitter.com/NASA/status/1574539270987173903?s=20&t=u_4wI
V9Cui2xyn9QLj286Q•https://www.cbsnews.com/sanfrancisco/news/i-just-could-not-believeit-livermore-team-celebrates-nasas-historic-strike-on-distant-asteroid/•http://mug.mvapich.cse.ohiostate.edu/static/media/mug/presentations/18/moody-mug-18.pdf

Network Based Computing Laboratory

Japan – April ’23

26

MVAPICH enabling Nuclear Fusion Research•

LLNL’s National Ignition Facility (NIF) conducted the firstcontrolled fusion experiment in history![1]•MVAPICH, being the default MPI library on the LLNL
systems, has been enabling the thousands of simulation
jobs that have led to this amazing achievement!

•[1] https://www.llnl.gov/news/national-ignition-facility-achieves-fusion-ignition

The target chamber of LLNL’s National Ignition Facility, where 192
laser beams delivered more than 2 million joules of ultravioletenergy to a tiny fuel pellet to create fusion ignition on Dec. 5, 2022.

Network Based Computing Laboratory

The hohlraum that houses the type of cryogenic target used
to achieve ignition on Dec. 5, 2022, at LLNL’s National
Ignition Facility.

Japan – April ’23

To create fusion ignition, the National Ignition Facility’s laser energy isconverted into X-rays inside the hohlraum, which then compress a fuelcapsule until it implodes, creating a high temperature, high pressure plasma.

27

Presentation Overview•

MVAPICH MPI Library Project–••

High-Performance Support for various CPU, GPU, DPU, and Networking Technologies

HiDL Project–

High-Performance Deep Learning and Machine Learning–

Accelerating Deep Learning with DPU

HiBD Project–

Accelerating Spark with MPI–

Accelerating Data Science Applications with Dask•

Commercial Support and Value-Added Products•

Conclusions

Network Based Computing Laboratory

Japan – April ’23

28

Converged Middleware for HPC, AI, Big Data and Data Science

MVAPICH2 &MVAPICH2-DPU
Libraries

Deep/
MachineLearning

HPC(MPI, PGAS, etc.)(TensorFlow,
PyTorch, cuML,
etc.)

High-Performance DL/ML
Libraries with MVAPICH
X-Scale-AI-DPU LibraryWith MVAPICH

Big Data (Hadoop,
Spark), Data
Science(Dask)

Network Based Computing Laboratory

Japan – April ’23

29

MVAPICH2 (MPI)-driven Infrastructure for ML/DL Training
ML/DL Applications

ML/DL Applications

TensorFlow

PyTorchPyTorchMXNet

Horovod

MVAPICH2 or MVAPICH2-X
for CPU Training

Torch.distributed

MVAPICH2-GDR for
GPU Training

MVAPICH2 or MVAPICH2-X
for CPU Training

DeepSpeed

MVAPICH2-GDR for
GPU Training

More details available from: http://hidl.cse.ohio-state.edu

Network Based Computing Laboratory

Japan – April ’23

30

Sample Designs and Solutions• Exploiting Hybrid (Data and Model) Parallelism for out-of-core training• Accelerating Deep Learning with DPU• Accelerating DL-Checkpointing with DPU

Network Based Computing Laboratory

Japan – April ’23

31

HyPar-Flow with Out-of-Core Training at Scale (512 nodes on TACC Frontera)•

ResNet-1001 with variable batch
size•

Approach:

481x speedup on 512 Intel Xeon Cascade Lake nodes (TACC Frontera)– 48 model-partitions for 56 cores– 512 model-replicas for 512 nodes– Total cores: 56 x 512 = 28,672•

Speedup– 253X on 256 nodes– 481X on 512 nodes•

Scaling Efficiency– 98% up to 256 nodes– 93.9% for 512 nodes

Network Based Computing Laboratory

A. A. Awan, A. Jain, Q. Anthony, H. Subramoni, and DK Panda, “HyPar-Flow: Exploiting MPI and Keras for Hybrid
Parallel Training of TensorFlow models”, ISC ‘20, https://arxiv.org/pdf/1911.05146.pdf

Japan – April ’2332Model Parallelism (GEMS) at Scale (1,024 V100 GPUs on LLNL Lassen)•

Two Approaches:

97.32% scaling efficiency on 1024 V100 GPUs (LLNL Lassen)– Memory Aware Synchronized Training
with Enhanced Replications (MASTER)

•

Setup
– ResNet-1k on 512 X 512 images– 128 Replications on 1024 GPUs•

Images per sec(Higher is better)– Memory Aware Synchronized Training(MAST)

128
64
32
16
8
4
2
1
0.5
0.25

Scaling Efficiency– 97.32% on 1024 nodes

GEMS-HY Basic
GEMS-HY MASTGEMS-HY MASTER

8

16

32 64 128 256 512 1024
Number of GPUs

A. Jain, A. Awan, A. Aljuhani, J. Hashmi, Q. Anthony, H. Subramoni, D. Panda, R.
Machiraju, A. Parwani, “GEMS: GPU Enabled Memory Aware Model Parallelism
System for Distributed DNN”, SC ‘20

Network Based Computing Laboratory

Japan – April ’23

33

Exploiting DPUs for Deep Neural Network Training• There are several phases in Deep
Neural Network Training– Fetching Training Data
– Data Augmentation– Forward Pass
– Backward Pass
– Weight Update– Model Validation• Different phases can be offloaded to
DPUs to accelerate the training.
Network Based Computing Laboratory

Japan – April ’23

34

DPU Offloading Strategy•

Offloads data augmentation and model validation to DPUs.

•

Creates three types of processes– Training processes (on CPU)
– Data Augmentation processes (On DPU)
– Testing processes (On DPU)

A. Jain, N. Alnaasan, A. Shafi, H. Subramoni, D. K. Panda, “Accelerating CPU-based Distributed DNN Training on Modern HPC Clusters
using BlueField-2 DPUs”, Hot Interconnect ‘21

Network Based Computing Laboratory

Japan – April ’23

35

X-ScaleAI-DPU Package•

Accelerating CPU-based DNN training with DPU support•

Based on MVAPICH2 2.3.7 with Horovod 0.25.0•

Supports all features available with the MVAPICH2 2.3.7 release(http://mvapich.cse.ohio-state.edu)•

Supports PyTorch framework for Deep Learning

Available from X-ScaleSolutions, please send a note tocontactus@x-scalesolutions.com to get a trial license.
Network Based Computing Laboratory

Japan – April ’23

36

Training of ResNet-20v1 model on the CIFAR10 dataset
System Configuration

•
•
•
•
•

Two Intel(R) Xeon(R) 16-coreCPUs (32 total) E5-2697A V4 @2.60 GHzNVIDIA BlueField-2 SoC,
HDR100 100Gb/s
InfiniBand/VPI adapters
Memory: 256GB DDR4
2400MHz RDIMMs per node
1TB 7.2K RPM SSD 2.5" hard
drive per node
NVIDIA ConnectX-6
HDR/HDR100 200/100Gb/s
InfiniBand/VPI adapters with
Socket Direct

Network Based Computing Laboratory

Performance improvement using X-ScaleAI-DPU over CPU-onlytraining on the ResNet-20v1 model on the CIFAR10 dataset
Japan – April ’23

37

Training of the ShuffleNet model on the TinyImageNet Dataset

Performance improvement using X-ScaleAI-DPU over CPU-onlytraining on the ShuffleNet model on the TinyImageNet dataset
Network Based Computing Laboratory

Japan – April ’23

38

X-ScaleAI DPU Checkpointing• All DNN training runs must save snapshots of in-progress snapshots of the model
parameters called a checkpoint• Unstable HPC/Cloud clusters require frequent checkpointing• The checkpoint cost scales with the number of model parameters• Typically, the root rank saves the checkpoint while all other ranks stall (Called a root ckpt)• By offloading checkpointing to the DPU, we can overlap checkpoint I/O with compute

Default Root Checkpointing
Network Based Computing Laboratory

Checkpointing with DPU offload
Japan – April ’23

39

X-ScaleAI DPU Checkpointing• We measure the time per epoch with ResNet18 on the TinyImageNet dataset• Root and Offloaded checkpoints refer to the default and DPU checkpointing schemes, respectively• TinyImageNet is a subset of ImageNet containing 100k images downsized to 64x64 pixels• We use 4x and 10x to refer to checkpointing 4 and 10 times within an epoch, respectively• We expect users to checkpoint less frequently (4x) on stable HPC/Cloud systems, and more frequently (10x) on unstable
HPC/Cloud systems

Seconds per Epoch(Lower is better)• DPU-Offloaded checkpointing outperforms root checkpointing at all node scales• Up to 49% reduction in epoch time for ResNet18 on an unstable system where frequent checkpointing is required

300

Baseline (no ckpt)

ResNet18 on TinyImagenet

Root ckpt (4x)Root ckpt (10x)Offloaded ckpt (4x)Offloaded ckpt (10x)

200

29%

100
0

1

Network Based Computing Laboratory

2

Nodes

Japan – April ’23

41%
4

49%
8

40

X-ScaleAI DPU Checkpointing

Will be available in upcoming
X-ScaleAI-DPU Release• Similarly for larger CNN models like ResNet50 and ResNet152, DPU-Offloaded checkpointing outperforms root
checkpointing at all node scales• Up to 51% and 52% reduction in epoch time for ResNet50 and ResNet152, respectively on an unstable system
Seconds per Epoch(Lower is better)

where frequent checkpointing is required
400

Baseline (no ckpt)

ResNet50 on TinyImagenet

Root ckpt (4x)200Seconds per Epoch(Lower is better)

0500

Root ckpt (10x)Offloaded ckpt (4x)33%
1Baseline (no ckpt)

2

Nodes

41%
4

Network Based Computing Laboratory

1

51%
8

ResNet152 on TinyImagenet
Root ckpt (4x)Root ckpt (10x)33%0Offloaded ckpt (10x)

2

Nodes

Japan – April ’23

42%
4

52%
8
41

Presentation Overview•

MVAPICH MPI Library Project–••

High-Performance Support for various CPU, GPU, DPU, and Networking Technologies

HiDL Project–

High-Performance Deep Learning and Machine Learning–

Accelerating Deep Learning with DPU

HiBD Project–

Accelerating Spark with MPI–

Accelerating Data Science Applications with Dask•

Commercial Support and Value-Added Products•

Conclusions

Network Based Computing Laboratory

Japan – April ’23

42

Converged Middleware for HPC, AI, Big Data and Data Science

MVAPICH2 &MVAPICH2-DPU
Libraries

Deep/
MachineLearning

HPC(MPI, PGAS, etc.)(TensorFlow,
PyTorch, cuML,etc.)

Big Data (Hadoop,
Spark), Data
Science(Dask)

Network Based Computing Laboratory

Japan – April ’23

High-Performance DL/ML
Libraries with MVAPICH
X-Scale-AI-DPUMPI4Spark and MPI4Dask
Libraries with MVAPICH
43

The High-Performance Big Data (HiBD) Project•

Since 2013•

RDMA for Apache Spark•

RDMA for Apache Hadoop 3.x (RDMA-Hadoop-3.x)•

RDMA for Apache Hadoop 2.x (RDMA-Hadoop-2.x)–

Plugins for Apache, Hortonworks (HDP) and Cloudera (CDH) Hadoop distributions•

RDMA for Apache Kafka•

RDMA for Apache HBase•

RDMA for Memcached (RDMA-Memcached)•

RDMA for Apache Hadoop 1.x (RDMA-Hadoop)•

MPI4Dask•

MPI4Spark•

OSU HiBD-Benchmarks (OHB)–

Available for InfiniBand and RoCE
Also run on Ethernet
Available for x86 and OpenPOWER
Support for Singularity and Docker

HDFS, Memcached, HBase, and Spark Micro-benchmarks•

http://hibd.cse.ohio-state.edu•

Users Base: 355 organizations from 39 countries•

More than 47,350 downloads from the project site

Network Based Computing Laboratory

Japan – April ’23

44

MPI4Spark Interconnect Support•

The current approach is different from its predecessor design, RDMA-Spark(http://hibd.cse.ohio-state.edu)– RDMA-Spark supports only InfiniBand and RoCE
– Requires new designs for new interconnect•

MPI4Spark supports multiple interconnects/systems through a common MPI library– Such as InfiniBand (IB), Intel Omni-Path (OPA), HPE Slingshot, RoCE, and others– No need to re-design the stack for a new interconnect as long as the MPI library supports it
MPI4Spark

RDMA-Spark

MVAPICH2-J
MVAPICH2
Supported Interconnects(InfiniBand, iWARP, RoCE, Omni-Path,
EFA, Rockport, Slingshot, etc.)

Network Based Computing Laboratory

UCR

vs.

IB Verbs
Supported Interconnects

Japan – April ’23(InfiniBand, RoCE)

45

MPI4Spark: Using MVAPICH2 to Optimize Apache Spark• The main motivation of this work is to utilize thecommunication functionality provided by
MVAPICH2 in the Apache Spark framework• MPI4Spark relies on Java bindings of the
MVAPICH2 library• Spark’s default ShuffleManager relies on Netty forcommunication:– Netty is a Java New I/O (NIO) client/server
framework for event-based networking applications– The key idea is to utilize MPI-based point-to-point
communication inside Netty
Network Based Computing Laboratory

Japan – April ’23

46

MPI4Spark: Relative Speedups to Vanilla Spark and RDMASpark on Four HPC Systems
System Name

Nodes Used

Processor

Cores Used

Sockets

Cores/socket

RAM

Interconnect

TACC Frontera

34

Xeon Platinum

1792

2

28

192 GB

HDR (100G)

RI2 (OSU System)

14

Xeon Broadwell

336

2

14

128 GBEDR (100G)

MRI (OSU System)

12

AMD EPYC 7713

1280

2

64

264 GB

200 Gb/sec (4X HDR)TACC Stamepede2

10Xeon Platinum

384

2

28

192 GBIntel Omni-Path (100G)

3.65x

OHB GroupByTest

OHB SortByTest
3.52x

1.88x

Network Based Computing Laboratory

1.86x

Japan – April ’23

47Performance Evaluation with Intel HiBench Workloads
1.4x on average than RDMA-Spark1.5x on average than Vanilla Spark
1.4xIntel HiBench ML Workloads - Frontera

Intel HiBench Micro/Graph Workloads - Frontera

1.5x on average than Vanilla SparkIntel HiBench Micro/ML Workloads - Stampede2

Omni-Path
Network••••

This evaluation was done on the TACC Frontera (IB) and the TACC Stampede2 (OPA) Systems
This illustrates the portability of MPI4Spark on different interconnects
We see a speed-up for the LR machine learning workload on Stampede2 of about 2.2x
Speed-ups for the LDA machine learning workload on Frontera are 1.7x for both IPoIB and RDMA

K. Al Attar, A. Shafi, M. Abduljabbar, H. Subramoni, D. Panda, Spark Meets MPI: Towards High-Performance Communication Framework for Sparkusing MPI, IEEE Cluster '22, Sep 2022.

Network Based Computing Laboratory

Japan – April ’23

48

MPI4Dask in the Dask Architecture
Dask

Dask
DataFrame

Dask Array

Dask Bag

DelayedFutureTask Graph
Dask-MPI

Dask-CUDA

Dask-Jobqueue

WorkerClientDistributed
Scheduler
Comm Layer

tcp.pyucx.pyMPI4Dask

UCX-Pympi4pyUCXMVAPICH2(Cython wrappers)

TCP
Laptops/
Desktops
Network Based Computing Laboratory

High Performance Computing Hardware
Japan – April ’23

49

cuDF Merge Benchmark on the Cambridge Wilkes-3 System
Wilke3 GPU System:• GPU-based Operation: 𝑑𝑑𝑑𝑑𝑑𝑑𝑑. 𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚(𝑑𝑑𝑑𝑑𝑑𝑑𝑑), using persist- 80 nodes
 Merge two GPU data frames, each with length of 32*1e8- 2x AMD EPYC 7763 64-core Compute() will gather the data from all worker nodes to the client node,
Processors
and make a copy on the host memory.
- 1000 GiB RAM
 Persist() will leave the data on its current nodes without any gathering - Dual-rail Mellanox HDR200 IB- 4x NVIDIA A100 SXM4 80 GB
Aggregated Throughput

Execution Time
Averagely, MPI4Dask is:- 4.94x faster than UCX- 26.85x faster than TCP

8.4x

7.1x

MPI4Dask 0.3, Dask 2022.8.1, Distributed, 2022.8.1, MVAPICH2-GDR 2.3.7, UCX v1.13.1, UCX-py 0.27.00

Network Based Computing Laboratory

Japan – April ’23

50

NumPy Array Slicing Benchmark on TACC Frontera CPU
System
1.26x better on average

3.17x better on average

5.2x

On average, MPI4Dask is:- 1.37x faster than UCX- 1.51x faster than TCP

1.9x

2.6x

2.1x

5.3x

From 32 workers, we increase array size by 16 times
A. Shafi , J. Hashmi , H. Subramoni , and D. K. Panda, Efficient MPI-based
Communication for GPU-Accelerated Dask Applications, CCGrid ‘21
https://arxiv.org/abs/2101.08878
Network Based Computing Laboratory
Japan – April ’23

MPI4Dask 0.3 release(http://hibd.cse.ohio-state.edu)
51

Presentation Overview•

MVAPICH MPI Library Project–••

High-Performance Support for various CPU, GPU, DPU, and Networking Technologies

HiDL Project–

High-Performance Deep Learning and Machine Learning–

Accelerating Deep Learning with DPU

HiBD Project–

Accelerating Spark with MPI–

Accelerating Data Science Applications with Dask•

Commercial Support and Value-Added Products•

Conclusions

Network Based Computing Laboratory

Japan – April ’23

52Commercial Support for MVAPICH2, HiBD, and HiDL Libraries•••

Supported through X-ScaleSolutions (http://x-scalesolutions.com)
Benefits:–

Help and guidance with installation of the library–

Platform-specific optimizations and tuning–

Timely support for operational issues encountered with the library–

Web portal interface to submit issues and tracking their progress–

Advanced debugging techniques–

Application-specific optimizations and tuning–

Obtaining guidelines on best practices–

Periodic information on major fixes and updates–

Information on major releases–

Help with upgrading to the latest release–

Flexible Service Level Agreements

Support being provided to National Laboratories and International HPC centers

Network Based Computing Laboratory

Japan – April ’23

53

Value-Added Products with Support•

Multiple value-added products with commercial support– X-ScaleHPC– X-ScaleAI– MVAPICH2-DPU– X-ScaleAI-DPU•

These products have additional– Features– Performance Optimizations– Profiling and Introspection Support•

Send an e-mail to contactus@x-scalesolutions.com for free trial!!

Network Based Computing Laboratory

Japan – April ’23

54Presentation Overview•

MVAPICH MPI Library Project–••

High-Performance Support for various CPU, GPU, DPU, and Networking Technologies

HiDL Project–

High-Performance Deep Learning and Machine Learning–

Accelerating Deep Learning with DPU

HiBD Project–

Accelerating Spark with MPI–

Accelerating Data Science Applications with Dask•

Commercial Support and Value-Added Products•

Conclusions

Network Based Computing Laboratory

Japan – April ’23

55

Concluding Remarks• Upcoming Exascale systems and Cloud need to be designed with a holistic
view of HPC, Big Data, Deep/Machine Learning, Data Science, and computing
continuum• Presented an overview of opportunities and challenges in designing scalable
and high-performance middleware for such systems• Presented a set of solutions which enable these communities to takeadvantage of current and next-generation systems with latest technologies• Next-generation Zetascale systems will need continuous innovations indesigning converged software architectures …..
Network Based Computing Laboratory

Japan – April ’23

56

A New Book on High-Performance Big Data Computing• By MIT Press• Released in August 2022
• An in-depth overview of an emerging
field that brings together highperformance computing, big dataprocessing, and deep learning.
• https://mitpress.mit.edu/books/highperformance-big-data-computing

Network Based Computing Laboratory

Japan – April ’23

57

Funding Acknowledgments
Funding Support by

Equipment Support by

Network Based Computing Laboratory

Japan – April ’23

58

Acknowledgments to all the Heroes (Past/Current Students and Staffs)
Current Research Scientists

Current Students (Graduate)–

N. Alnaasan (Ph.D.)––

K. S. Khorassani (Ph.D.)

A. H. Tu (Ph.D.)–

G. Kuncham (Ph.D.)–

M. Abduljabbar

R. Vaidya (Ph.D.)–

A. Shafi

Q. Anthony (Ph.D.)–

P. Kousha (Ph.D.)–

S. Xu (Ph.D.)––

C.-C. Chun (Ph.D.)–

B. Michalowicz (Ph.D.)–

Q. Zhou (Ph.D.)–

J. Yao (Ph.D.)–

N. Contini (Ph.D.)–

B. Ramesh (Ph.D.)–

K. Al Attar (M.S.)–

M. Han (M.S.)–K. K. Suresh (Ph.D.)–

L. Xu (Ph.D.)–

A. Guptha (M.S.)–

Current Students (Undergrads)–

T. Chen

Past Students
T. Gangadharappa (M.S.)–

P. Lai (M.S.)–

A. Awan (Ph.D.)––

A. Augustine (M.S.)–

K. Gopalakrishnan (M.S.)–

J. Liu (Ph.D.)–

P. Balaji (Ph.D.)–

J. Hashmi (Ph.D.)–

M. Luo (Ph.D.)–

M. Bayatpour (Ph.D.)–

W. Huang (Ph.D.)–

A. Mamidala (Ph.D.)–

R. Biswas (M.S.)–

A. Jain (Ph.D.)–

G. Marsh (M.S.)–

S. Bhagvat (M.S.)–

W. Jiang (M.S.)––

A. Bhat (M.S.)–

J. Jose (Ph.D.)–

D. Buntinas (Ph.D.)––

L. Chai (Ph.D.)––

Current Faculty–

H. Subramoni

Current Software Engineers–

B. Seeds–

N. Pavuk–

N. Shineman–

M. Lieber

Current Research Specialist–

D. Shankar (Ph.D.)–

G. Santhanaraman (Ph.D.)–

N. Sarkauskas (B.S. and M.S)–

N. Senthil Kumar (M.S.)–

A. Singh (Ph.D.)

V. Meshram (M.S.)–

J. Sridhar (M.S.)–

A. Moody (M.S.)–

S. Srivastava (M.S.)

M. Kedia (M.S.)–

S. Naravula (Ph.D.)–

S. Sur (Ph.D.)–

S. Kini (M.S.)–

R. Noronha (Ph.D.)–

H. Subramoni (Ph.D.)

B. Chandrasekharan (M.S.)–

M. Koop (Ph.D.)–

X. Ouyang (Ph.D.)–

K. Vaidyanathan (Ph.D.)

S. Chakraborthy (Ph.D.)–

K. Kulkarni (M.S.)–

S. Pai (M.S.)–

A. Vishnu (Ph.D.)

N. Dandapanthula (M.S.)–

R. Kumar (M.S.)–

S. Potluri (Ph.D.)–

J. Wu (Ph.D.)–

A. Reifsteck–

V. Dhanraj (M.S.)–

S. Krishnamoorthy (M.S.)–

K. Raj (M.S.)–

W. Yu (Ph.D.)

K. Kandalla (Ph.D.)–

R. Rajachandrasekar (Ph.D.)

D. Bureddy

C.-H. Chu (Ph.D.)

–

–

–

–

J. Zhang (Ph.D.)–

J. Perkins–

D. Banerjee––
M. Li (Ph.D.)
H.-W. Jin–
E. Mancini–

A. Ruhela–

X. Besseron–

J. LinK. Manian–

J. Vienne–

M. Arnold–

M. S. Ghazimeersaeed–

S. Marcarelli–

H. Wang–

J. Smith–

Past Post-Docs

M. LuoNetwork Based Computing Laboratory––

Japan – April ’23–

R. Motlagh

Past Research Scientists–

K. Hamidouche–

S. Sur–X. Lu

Past Senior Research Associate–

J. Hashmi

Past Programmers

Past Research Specialist

59Thank You!

panda@cse.ohio-state.edu

Network-Based Computing Laboratory
http://nowlab.cse.ohio-state.edu/

The High-Performance MPI/PGAS Project
http://mvapich.cse.ohio-state.edu/

Network Based Computing Laboratory

The High-Performance Big Data Project
http://hibd.cse.ohio-state.edu/

Japan – April ’23

The High-Performance Deep Learning Project
http://hidl.cse.ohio-state.edu/

60