Huawei ARM HPC Software Update
The 2nd R-CCS International Symposium

Zhaohui Ding (丁肇辉)dingzhaohui@huawei.com
Director, HPC Lab,Cloud & AI Business Group, Huawei Technologies

Agenda1. Huawei HPC Overview
2. MPI & UCX
3. Compiler & Math Libraries
4. Unified Scheduler
5. Summary2

Huawei Kunpeng HPC Solution Stack……
Meteorology

Computational
Chemistry

Physics

Application
Portal/Workflow

Workload
Manager

Huawei Cloud
Compiler

Kunpeng 920

3LifeScience

Huawei MPI

Kunpeng Mainboard

EDACAE

ClusterManagement

Math Libraries

Reservoir
Simulation

Monitoring &Dashboard

Runtime

Ascend 310 & 910

Tracer, Profiler,Debugger, Porting
AdvisorFusionStorage Distributed Storage

Ethernet Switches

The Roadmap of Kunpeng

Hi161232 cores@16FF
2.1 GHz
4*64bit DDR3/4
PCIe 3.0 /SAS3.0/10GE

Kunpeng 916

Kunpeng 920

Kunpeng 930

Kunpeng 950

Up to 64 cores@7nm
Up to 3.0 GHz
4P
8 DDR4 Channels
CCIX
RoCE v2
PCIe 4.0 /100GE

32 cores@16FF+
2.4 GHz4P
4*64bit DDR3/4
PCIe 3.0 /SAS3.0/10GE

Hi1610Available 920 models in 2019 Jun：

16 cores@16FF
2.1 GHz
2*64bit DDR3/4
PCIe 3.0 /SAS3.0/10GE

20144••••

2016

Kunpeng 920-6430：64C@3.0GHz，200W
Kunpeng 920-6426：64C@2.6GHz，180W
Kunpeng 920-4826：48C@2.6GHz，150W
Kunpeng 920-3226：32C@2.6GHz，120W

2018

2020

ProductionDevelopmentPlanning2022

Agenda1. Huawei HPC Overview
2. MPI & UCX
3. Compiler & Math Libraries
4. Unified Scheduler
5. Summary5

Huawei MPI
APPOpenFOAM

……

WRF

NAMD
•

Open MPI & Open UCX•Open MPI is modular, and easily extendable MPI
implementation•
Open UCX is communication framework created byindustry, laboratories and academia.
•OpenMPI’s p2p module is using Open UCX•

Huawei MPI

OpenMPI layer
MPI P2POB1UCXMPI Collective
UCXTuned

ORTEOPAL

Open MPI
UCG

UCP
Tagmatched
send/recv

Remotememory
get/put

Active Messages

Open UCX

UCTTopologyware
algorithms

SHMbasedalgorithms

Plan&Phase

Trigger/ExecTransport: sm, rc, ud, rc/ud for
Kunpeng, dc, knem

UCS

•••

•
•

OFED•

Verbs

•

HW
6

Driverlibrdmacm

libibverbs
NIC (IB/RoCE)

Lib for kunpeng

An optimized implementation based on Open MPI
Proposed UCG(Groups) – collective operations API to UCX
Implement optimized collective operations algorithms based onOpen UCX
Support ARM and x86
Optimized topology-aware, SHM algorithms for Kunpeng CPU
Communication offloading for p2p and collComputing offload in switch

Consolidating UCX P2P Communication• Open UCX, as most P2P libraries, contains
considerable logic on how best to send data.

Open MPI

Huawei MPI• During collective operations, messages ofthe same size are often sent consecutively.
This makes most of the P2P logic redundant.

• Proposed UCG(Groups), move the collectiveoperations logic down to UCX layer• Consolidating the per-message logic, making
it per-collective, to save latency• Decoupling planning and execution, thecollective algorithms can be extended easily
by 3rd party.
7“RDMA-Based Library for Collective Operations in MPI”,
Alex Margolin and Amnon Barak, ExaMPI’19

Collective Algorithms
NUMA 0NUMA 0NUMA 1• Optimized MPI_bcast, MPI_allreduce, MPI_barrier

00

NUMA 1• NUMA aware and SHM-based intra-node collectives

11• Collective operation algorithms• Bcast: Binomial, Knomial• Allreduce: Recursive doubling, Binomial, Knomial,Ring• Barrier: Recursive doubling, Binomial, Knomial

2345

62475

NUMA-awared MPI_bcast• All the algorithms are implemented under UCG,binomial Bcast, Recursive doubling Allreduce andBarrier are open source.
SHM-based MPI_bcast

8367

8 Nodes Small Package Benchmark
40ppn, IB, Intel Skylake 6148

96ppn, IB, KunPeng920
MPI_Bcast

MPI_Bcast
2.64

2.68

2.56

2.42

2.5

2
1.5

11

1

1

1
0.5

Improvement Ratio

Improvement Ratio

304816

Bytes
HuaweiMPIOpenMPI

21.8
1.6
1.4
1.2
10.80.6
0.4
0.201.89

1432

Bytes
HuaweiMPIOpenMPI

11632

2.5
1.97

1.97

1.91
1.69

1.5

111110.5

0Improvement Ratio

Improvement Ratio

1

MPI_Allreduce

2.5

2

2.05

1.93

2.03

1.99

1.5
111

1

1
0.5
049

1.87

18

MPI_Allreduce
21.89

1.88

8

Bytes
HuaweiMPIOpenMPI

1632

48

Bytes
HuaweiMPIOpenMPI

1632

256 Nodes Benchmark

The table shows the latency (microseconds) ofAllreducewith increasing message sizes (Bytes), on
256 nodes (Intel Xeon E5-2680CPUs @2.5GHz,
Connect-X3 NIC).
10

Agenda1. Huawei HPC Overview
2. MPI & UCX
3. Compiler & Math Libraries
4. Unified Scheduler
5. Summary

11

HCC – Huawei Cloud Compiler

Huawei Compiler Lab, 300+ people, multiple

Develop various compilers for mobile, IoT, network devices and Kunpeng

HCC for HPC 1.0
 Instrument Pipeline optimization for Kunpeng 920
 Optimized mathlib A series of features and optimizations for Kunpeng

Future plan Based on latest gcc

 Optimized OpenMP runtime
 More auto parallelization

 llvm-based compiler
12andcommitters

HCC – Optimized Instrument Pipeline
Code:

No Optimization

Pipeline Optimization

13Pipeline

HCC – Feature Highlights
-fhcc-gfortran-minmax
 Uses cpu instruments to do the min/max comparison

-mcmodel
 Support tiny, small, medium and large

-DAARCH64_QUADMATH
 Support 128 bits floating- lstringlib -Wl,--wrap=memset

Open source mathlib, after optimization
0.07

0.06
0.05
0.04



 memset optimization

0.03-lmathlib

0.01

 An optimized math library

140.02
0expf

logf

powf

expOpen开源mathlib
source mathlib

pow
Optimized mathlib
自研mathlib

exp2f

log2f

exp2

MAL – Math Acceleration Library• MAL is Huawei internal project
launched from 2016, to provide highperformance math libraries for ARM• MAL includes BLAS, FFT, Lapack,Sparse BLAS

HPC

Big Data
Analysis

AIMedia

Others...

Machine learning
library

DNNL

Multimedia library(image, video, audio,etc.)

Acceleration library(Compress, Encrypt,Decrypt)

Signal processing
library (IPP, FFT)

Vector statistics
library(Random number)

Linear algebra library(BLAS, LAPACK, Sparse BLAS, Solvers)

Vector basic mathematical function library• Open source community also optimize
code for Kunpeng, e.g., OpenBLAS

15

Scalar basic mathematical function libraryAgenda1. Huawei HPC Overview
2. MPI & UCX
3. Compiler & Math Libraries
4. Unified Scheduler
5. Summary

16

The Challenges of Modern HPC Scheduler• The new challenges brought by the convergence of HPC，HPDA and Deep Learning More kinds of application frameworks run in same environments
 The traditional HPC: MPI + X
 The Bigdata: Hadoop, Spark, Flink etc.
 The Deep Learning: Tensorflow, PyTorch, paddle etc.
 How to run all the workload in one HPC cluster? Without changing user habits of the frameworks.

• More and more the task-based applications
 Tasks scheduling handled by the framework: the challenge is elastic allocation
 Task scheduling handled by the scheduler: the challenge is job throughput
 Kunpeng920 has 128 cores, 8k nodes means 1M cores• More new components than CPU and Job
 Accelerator, Container job, Resource Bursting, I/O & Storage
 Take GPU as example, it is not only a “number”, but also should be applied in the policies,
fairshare/threshold/reservation/preemption, and in the monitoring & reporting17

Donau Design Principle• Donau, a Huawei home-grown job scheduler, launched since 2018
• Target to be a unified scheduler for HPC/Bigdata/AI workload, but nointrusive modification for the application framework• Extremely high job throughput, target 1M running jobs
• Natively support• Accelerator• I/O load
• Container• Adopt the new technologies in the implementation,• Micro-service architecture• DRF (Dominate Resource Fairness)• MQ & Distributed cache18

Donau Scheduler• Donau – Huawei home-grown HPCscheduler

HPC/DLBigdata

AI/Service/
Cloud NativeScheduler CoreScheduler Core The scheduler core is an independent module,
can replaced the scheduler of Yarn andKubernetes.

• Scheduling Natively support elastic allocation
 Resource quota based + user priority based
fairshare;Donau Job Scheduler

Scheduler Core

• Features:• Unified Job/Allocation model: Array Job, MPIjob, Bigdata job/Task, TensorFlow job, ServiceJob, Workflow• Natively support container job• Accelerator(co-processor) as native resource• Resource bursting with IaaS
• Job-level I/O monitoring & control
19BLASSparse
LAPACKKunpeng

X86

GPGPU

DNN
DAAL

AscendFPGAAgenda
1. Huawei HPC Overview
2. MPI & UCX
3. Compiler & Math Libraries
4. Unified Scheduler
5. Summary20

Summary

21•

Huawei is prompting ARM HPC ecosystem from different dimensions;•Kunpeng 9x0 ARM-based CPU and Kunpeng mainboard;•Release HPC Software Suite in the middle of 2020, includes MPI, Compiler, Math Libraries,Scheduler and Management Software;•Collaborate with Community, Academic and ISV for math libraries, solvers, tool chain andapplications.

Thank you.

把数字世界带入每个人、每个家庭、
每个组织，构建万物互联的智能世界。
Bring digital to every person, home, andorganization for a fully connected,intelligent world.
Copyright©2018 Huawei Technologies Co., Ltd.
All Rights Reserved.
The information in this document may contain predictive
statements including, without limitation, statements regarding
the future financial and operating results, future productportfolio, new technology, etc. There are a number of factors that
could cause actual results and developments to differ materiallyfrom those expressed or implied in the predictive statements.
Therefore, such information is provided for reference purpose
only and constitutes neither an offer nor an acceptance. Huawei
may change the information at any time without notice.