ABCI’s Scheduling Design for
Accommodating Various AI Jobs
Shinichiro Takizawa
The National Institute of Advanced Industrial Science and Technology(AIST), Japan

8th ADAC Workshop

Outline• Introduction of AIST and ABCI
• ABCI system overview– Architecture, network topology, software stack• ABCI scheduling system– Workload characteristics of our pre-ABCI system– Detail design of scheduling system

Introduction of AIST
• A research institute as a part of the Ministry of Economy, Trade and Industry(METI) of Japan• Our mission– Integrate scientific and engineeringknowledge to address industry andsociety needs– Bridge the gap between innovativetechnological seeds andcommercialization

We develop ABCI for popularize AI technologies in Japan

4

ABCI: The Worldʼs First Large-Scale
Open AI Infrastructure
AI Bridging Cloud Infrastructure
nUniv. Tokyo / Kashiwa II Campus

OPERATED SINCE AUGUST 1ST, 2018

World Top-Level compute and data processcapability

Open, Public, and Dedicated infrastructure
for Al & Big Data Algorithms, Software, andApplicationsn Open Innovation Platform to accelerate joint
academic-industry R&D for AI
nPeak Performance:
550 PFLOPS (FP16)
37 PFLOPS (FP64)
Effective Performance: (as of Jun 2019)19.88 PFLOPS(#8 in TOP500)
14.423 GFLOPS/W (#3 in GREEN500)
508.85 TFLOPS(#5 in HPCG)
Power Usage: < 2.3 MW
Average PUE: < 1.1 (Estimated)5World’s Highest Speed in ImageNet-1k Training
ImageNet / ResNet-50 (Relative speedup & Accuracy)
1600

0.765

1400

Relative speedup

1000

0.755

800
0.75

600
400

0.745
200
00.74
MSRA Facebook Google Preferred Tencent(2015) (2017)
Brain Networks (2018)(2017) (2017)

SONY’s workhttps://arxiv.org/abs/1811.05233
Fujitsu lab’s workhttps://arxiv.org/abs/1903.12650

Sony +ABCI(2018)

Google(2018)

Relative speedup

Google(2018)

Sony +
ABCI(2019)

Accuracy

Fujitsu
Lab +ABCI(2019)NVIDIA(2019)

Google(2019)

Fujitsu
Lab +ABCI(2019)

MLPerf Training v0.6

Accuracy

0.76

1200

iFDK: CT Image Reconstruction Framework on ABCI
• A high-speed and high-resolution CT image reconstruction framework
running on GPU clusters
Micro-focus– Create 3D images from multiple 2D x-ray images– Demands for high-resolution CT image:
non-invasive inspection, reverse engineering, etc.

X-ray source• Our Achievements– High speed FDK GPU kernel whose compute cost is1/6 of the existing algorithms– Efficient FDK computation by overlapping CPU comp.,
GPU comp. and comm.
– Distributed framework for high-resolution
CBCT geometry and trajectory
image reconstruction– Good scalability up to 2K GPUs Peng Chen, Mohamed Wahib, Shinichiro Takizawa, Ryousei Takano, Satoshi Matsuoka.
iFDK: A Scalable Framework for Instant High-resolution Image Reconstruction

ABCI SYSTEM OVERVIEW

78ABCI Hardware Overview
High-Performance Computing System
550 PFlops(FP16), 37.2 PFlops(FP64)
476 TiB Memory, 1.74 PB NVMe SSD
Computing Nodes (w/ GPU) x 1088

Large-scale Storage System
1 PB Lustre (Home Directory)
DDN SFA14KX (w/ SS9012 Enclosure x 10) x1• 7.68TB SAS SSD x 185 for data
• 960GB SAS SSD x 13 for metadataGPU

NVIDIA Tesla V100 SXM2 x 4

CPUIntel Xeon Gold 6148 (2.4GHz/20cores) x 2

22 PB GPFS (Group Shared Directory, etc.)

Memory384GiB

DDN SFA14K (w/ SS8462 Enclosure x 10) x3• 12TB 7.2Krpm NL-SAS HDD x 2400
• 3.84TB SAS SSD x 216

Local Storage

Intel SSD DC P4600 (NVMe) 1.6TB x 1

Interconnect

InfiniBand EDR x 2

Multi-platform Nodes (w/o GPU) x 10
• Intel Xeon Gold 6132 (2.6GHz/14cores) x 2• 768GiB Memory, 3.8TB NVMe SSD, 1.5TB Intel Optane x2
Interactive Nodes x 4

Management and Gateway Nodes x 15

Interconnect (InfiniBand EDR)
• Mellanox CS7500 x 2• Mellanox SB7890 x 229

Service Network (10GbE)

17 PB Object Storage (Scality RING)
HPE Apollo 4510 Gen10 x 24
• 12TB SATA HDD x 1440
• 3.2TB SSD x 24

Gateway and Firewall• Nexsus 3232C x2• FortiGate 1500D x2• FortiAnalyzer 400E x1100Gbps

SINET59

ABCI Compute Node
FUJITSU PRIMERGY Server (2 servers in 2U)
CPUXeon Gold 6148 (27.5M Cache, 2.40 GHz, 20 Core) x2

GPUNVIDIA Tesla V100 (SXM2) x4

Memory384GiB DDR4 2666MHz RDIMM

Local Storage

1.6TB NVMe SSD (Intel SSD DC P4600 u.2) x1

Interconnect

InfiniBand EDR x2

Skylake
DDR4-2666
32GB x 6
IB HCA (100Gbps)

128GB/sSkylake

Xeon Gold 10.4GT/s x3 Xeon Gold6148
6148
UPI x3

DDR4-2666
32GB x 6

PCIe gen3 x16

PCIe gen3 x16
PCIe gen3 x16

128GB/s

PCIe gen3 x16x48 switch

x64 switch
NVLink2 x2Tesla V100 SXM2

Tesla V100 SXM2

Tesla V100 SXM2

Tesla V100 SXM2IB HCA (100Gbps)
NVMe10ABCI Node Rack / Interconnect
SPINE#1
CS7500SPINE#2
CS7500n Dense-packaged rack: 34 nodes, 136 Tesla V100
• Theoretical peak performance per rack︓
1.16 PFlops (FP64), 17 PFlops (FP16)
c.f. Google TPU 3.0 Pod (>100PFlops/8racks)• Power consumption per rack: 67.33 kW

1/3 Oversubscription BW
IB-EDR x 24

LEAF#1 LEAF#2 LEAF#3 LEAF#4
SB7890 SB7890 SB7890 SB7890

CX400 CX400 CX400#1#2#3CX400#17Rack #1

CX2570#3
CX2570#4

CX400#17

CX2570#3
CX2570#4

CX400 CX400 CX400#1#2#3

CX2570#1
CX2570#2

CX2570#33
CX2570#34

LEAF#1 LEAF#2 LEAF#3 LEAF#4
SB7890 SB7890 SB7890 SB7890

CX2570#5
CX2570#6

Full bisection BW
IB-EDR x 72

CX2570#1
CX2570#2

Full bisection BW
IB-EDR x 72

CX2570#33
CX2570#34

FBB#1 FBB#2 FBB#3
SB7890 SB7890 SB7890

CX2570#5
CX2570#6

FBB#1 FBB#2 FBB#3
SB7890 SB7890 SB7890

Rack #2n Interconnect• Fat-tree topology• Intra-rack: full bisection BW• Inter-rack︓1/3 over-subscription (2400/6800)• Without adoptive routing• Without Mellanox SHARP

32 Racks
InfiniBand EDR x4
InfiniBand EDR x6
InfiniBand EDR x1

11

ABCI Software Stack
Operating System

RHEL / CentOS 7.5

Job Scheduler

Univa Grid Engine 8.6.6

Container Engine

Docker 17.12.0 (Users can use only supported container images)
Singularity 2.6.1 (Users can use any container images)

MPI

Development toolsIntel MPI 2018.2.199
MVAPICH2 2.3rc2, 2.3 / MVAPICH2-GDR 2.3a, 2.3rc1, 2.3, 2.3.1, 2.3.2
OpenMPI 1.10.7, 2.1.3, 2.1.5, 2.1.6, 3.0.3, 3.1.0, 3.1.2, 3.1.3
Stack2017.8, 2018.2, 2018.3, 2019.3
Intel Parallel Studio XE Cluster EditionPGI Professional Edition 17.10, 18.5, 18.10, 19.3
NVIDIA CUDA SDK 8.0, 9.0, 9.1, 9.2, 10.0, 10.1
cuDNN 5.1, 6.0, 7.0, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6
NCCL 1.3.5, 2.1, 2.2, 2.3, 2.4
Intel MKL 2017.8, 2018.2, 2018.3, 2019.3
GCC, Python, Ruby, R, OpenJDK, Go, Perl

Deep Learning

Caffe, Caffe2, TensorFlow, Theano, Torch, PyTorch, CNTK, MXnet, Chainer, Keras, etc.
(Frameworks provided by NVIDIA GPU Cloud can also be deployed)

Big Data Processing

Hadoop, Spark

ABCIʼS JOB SCHEDULING SYSTEM

12

Objective and Preparation• Objective– Make ABCI an easy-to-use capacity computing system for AI R&D• Also need to be a capability computing system for grand challenge AIproblems• Preparation for designing the scheduling system– Operate a pre-ABCI GPU cluster for AI R&D more than two years (it’sstill in operation)– Collect job records and analyze them to understand how users use aGPU cluster for AI R&D– Use the obtained knowledges to design job scheduling system

13AAIC and its Workload Logs
• 400x NVIDIA Tesla P100s and InfiniBand EDR accelerate various AI workloads including ML (MachineLearning) and DL (Deep Learning).
• Advanced data analytics leveraged by 4PiB shared Big Data Storage and Apache Spark w/ its
ecosystem.

AAIC Workload Logs

(Jul. 14th, 2017 to Dec. 31st, 2018)https://github.com/aistairc/aaic-workload

Distribution of Number of Used GPUs/Nodes in AAIC DL Jobs
Single GPU, Single Nodeand Multi Node jobs

# Used GPUs in Single Node Jobs

# Used Nodes in Multi Node Jobs

Single GPU jobs are dominantWalltime of AAIC DL Jobs
Cumulative Distribution of Walltime• Some jobs run for two weeks– Some requested more than two weeks• Walltime of single GPU jobs isshorter than that of multi GPU jobs

Statistics of Walltime

16

Observations from AAIC AI Workload• Most jobs use only one GPU– Although severs equipped with multiple GPUs are common (e.g. DGX-2)• Small degree of parallelism are common in parallel jobs
• Walltime varies greatly depending on jobs
– Many of them are less than one hour, but some requires more than a week

• Observations not explained today– Positive correlation between degree of parallelism and walltime– Low accuracy of users’ requested walltime– Use of large array jobs
– etc.
17

Design of ABCI’s Job Scheduling System• Basic scheduling design– Define various resource types by splitting compute nodes– Interactive use, batch jobs and node reservation services– Allow long maximum walltime and reservation duration• Emulation of a rack-aware scheduling– Designed to achieve good performance for small jobs fit in arack (34 nodes)• Other parameters
18

Resource Types#CPU core
Assigned / Total#GPU
Assigned / Total

Memory (GB)Assigned / Total

Local SSD (TB)
Assigned / Total

Fee(Relative to F)

F (Full node)

40 / 404/4

360 / 384

1.4 / 1.6

1.0

G.large

20 / 404/4

240 / 384

0.7 / 1.6

0.9

G.small

5 / 401/460 / 384

0.175 / 1.6

0.3

C.large

20 / 400/4

120 / 384

0.7 / 1.6

0.6

C.small

5 / 400/430 / 384

0.175 / 1.6

0.2

Type Name

Resource types are defined by cgroups feature included in UGE
G.largeC.smallC.large
GPU0GPU0
CPU0G.small

CPU0GPU1

GPU1GPU2GPU2

CPU1

CPU1
GPU3GPU319

Job Execution Services
Normal Job Execution Services
Description

Maximum#NodesMinimum
Walltime

Maximum
Walltime

Maximum#Nodes x
Walltime

Spot

Batch job (e.g. qsub)
Charges users for nodes x time (in seconds)

512

1S

72 H

2304 NxH

On-demandInteractive job (e.g. qrsh)
Charges users for nodes x time (in seconds)

32

1S12 H12 NxH

Maximum#NodesMinimumResv. Time

MaximumResv. TimeMaximum#Nodes x
Resv. Time321D30 D

12288 NxH

Name of Service

Node Reservation Services
Name of Service
ReservedDescription
Node reservation
Charges users for nodes x reserved_time(in days)

Detail job scheduling description: https://docs.abci.ai/en/03/
20

Emulation of Rack-aware Scheduling (1/2)• Limitation– ABCI does not have full bisection-bandwidth network– UGE does not consider network topology in scheduling• Objective– Provide the maximum network performance for small jobs
• Small jobs : Use less than 34 nodes (within 1 rack)– Keep wait time for both small and large jobs as small as possible

21

Emulation of Rack-aware Scheduling (2/2)• Single Queue• Two types of host group under the queue#Nodes <= 34– RackHG: host group for each rack
– SystemHG: unique host group for all nodes

Search a rackHG in numerical order• Jobs using <=34 nodes– Find a rackHG that has enough nodes and run
the job in the rackHG

SystemHG
RckHG RckHG
01

RckHG
2

Rack0

Rack2

Rack1

Spot and On-demand

RckHG
32…

Rack32• Jobs using >34 nodes– Select nodes from systemHG• Different resource pools for different kind ofservices– (1)Spot and On-demand and (2)Others

Reservation and etc.
22

Emulation of Rack-aware Scheduling (2/2)#Nodes#Nodes<=> 3434
Select unused nodes from SystemHG in aSearcha rackHG
in numerical
dictionary
order ofnode nameorder• Single Queue• Two types of host group under the queue– RackHG: host group for each rack
– SystemHG: unique host group for all nodes• Jobs using <=34 nodes– Find a rackHG that has enough nodes and run
the job in the rackHG

SystemHG
RckHG RckHG
01

RckHG
2

Rack0

Rack2

Rack1

Spot and On-demand

RckHG
32

…

Rack32• Jobs using >34 nodes– Select nodes from systemHG• Different resource pools for different kind ofservices– (1)Spot and On-demand and (2)Others

Reservation and etc.
23

Other parameters• Enable backfill and priority option• Limitation on selecting resource types for job– Using multiple resources are allowed only when F resource type is used
• i.e. Can not use multiple G.[small | large], C.[small | large] in a job– A job can use only one resource type

• Limit the maximum number of running jobs per user: 200– Prevent resource occupation by a specific user
– No restriction on number of job submission

24Sample Job Script#! /bin/sh#$ -l rt_F=32#$ -l h_rt=1:00:00#$ -N testjob#$ -l USE_BEEOND=1#$ -v BEEOND_METADATA_SERVER=4#$ -v BEEOND_STORAGE_SERVER=16#$ -cwd

Type of resource and its amount
Walltime for the job
Create a temporal distributed filesystem
for the job using BeeGFS on DemandLocal NVMe SSD mount point

source /etc/profile.d/modules.shmodule load openmpi/3.1.3# Stage-in data to Local SSD
cp -r $HOME/something0 $SGE_LOCALDIR# Stage-in data to BeeGFS
cp -r $HOME/something1 /beeondmpirun -np 32 ...
...

BeeGFS mount point

Job Submission Command$ qsub -g gaa50004 sample_job.sh
Budget code25

Summary• Introduce ABCI, an open AI platform– Architecture, network topology and software stack– AI workload analysis results of pre-ABCI system– Detail of scheduling system design• Future work– Introduce resource types for memory intensive jobs
• Preparing several 2.8TB memory (2TB by Intel Optane) nodes– Collect job records executed on ABCI and analyze them
– Consider supporting multiple [C | G].[small | large] resource job• Combination of themhttps://abci.ai/