2024/08/06
SS研HPCフォーラム2024

HPCシステムのインタラクティブ化技術
大規模並列アプリの即時実行・効率化

富士通株式会社
コンピューティング研究所
大辻 弘貴*1 Some figures and texts are based on the SC23 Research Poster:
1 /Fine-Grained
40“ScalableGang Scheduling for HPC Systems with
Unreliable Broadcast Synchronization Mechanisms”© 2024 Fujitsu Limited自己紹介● 大辻 弘貴 (Hiroki Ohtsuji, Ph.D.)● 富士通株式会社 研究本部 コンピューティング研究所● Compute Workload Brokerコアプロジェクト● 領域● 高性能並列ファイルシステム● ジョブスケジューラ、大規模クラスタ設計・構築・運用● 研究用1056ノードFX700クラスタを新規設計・構築・運用（22年度～）● 活動● 情報処理学会（HPC研究会運営委員）● IEEE会員2 / 40

© 2024 Fujitsu Limited

はじめに● HPCシステムにおいてインタラクティブ性の要求が高まっている● 通常のHPCシステムはジョブ実行開始までの待ち時間が不可避● 原則として投入順に実行するバッチキューシステム由来の課題● AIをはじめとする新たなHPCユーザはバッチキューに慣れていない● Jupyter NotebookなどインタラクティブなUIを介した利用が主流● モデルが小さいうちはリソースを専有することができた● ところが、モデルサイズの大規模化などにより並列計算が必須になりつつある● 常にリソースを確保することは難しく、HPCベースのシステム利用が増加

大規模なジョブを効率的に即時実行する需要が生じている
3 / 40© 2024 Fujitsu Limited

HPCシステムのインタラクティブ化技術● 細粒度Gang schedulingによる並列アプリの同期並行実行● HPCアプリの時間分割実行を多ノード環境においても高効率に実現➔HPCシステムにおける高いインタラクティブ性の実現➔全系に近い大規模ジョブとそのほかのジョブを極めて少ない性能劣化で実現➔大規模ジョブにより生じる隙間を埋めることでシステム利用効率を向上

通常のバッチ実行

インタラクティブ実行
timeJob #1
Job #3
Job #4

nodesnodes

Job
#2

#1

大規模ジョブ実行に備えた隙間

待ち時間

即時実行
4 / 40

#1

#1
#2

#3
#4

#2time

#3
#4

隙間の削減
© 2024 Fujitsu Limited

AI/ML Users and the Need for Responsive
Job Execution⚫The demand for interactivity on HPC is increasing⚫Growing number of HPC users in the AI/ML fields

TimeNoded5 / 40

5/

© 2024 Fujitsu Limited

AI/ML Users and the Need for Responsive
Job Execution
JobUserTime

Noded6 / 40

6/

© 2024 Fujitsu Limited

AI/ML Users and the Need for Responsive
Job Execution

waitUserTime

JobNoded7 / 407/

© 2024 Fujitsu Limited

AI/ML Users and the Need for Responsive
Job Execution

waitUserTimeJobNodedUsers must wait for long periods even though
the submitted jobs finish quickly
8 / 408/© 2024 Fujitsu Limited

Transition from Batch Scheduling
Short Job

submission

Batch

Short Job

Pending duration of the short job

start

ShortLong Job #1

Job

Scheduling

Job#2

Short Job

start

Timesliced
Scheduling

Short Job

submission

Short job starts

Immediately !

Long Job #1

Long Job #1
Time-sliced
Instant Execution

Job#2
Time9/ 40© 2024 Fujitsu Limited

Requirements of Parallel Applications

Unsynchronized

Synchronized

Scheduling

Scheduling

Also Known as Gang Scheduling

Time
Node

#0

Running

Suspended

Running…

MPI_Send#1#0

Running

Suspended

Running…

Running

Suspended

Running…

MPI_Send

MPI_Recv

NodeTime

Node

Suspended

Running

Suspended…

MPI_Recv

Node#1

The Same Performance as usual

Poor Performance
Only one message can be sent per

Multiple messages can be sent

time-slice (the worst case)in a single time-slice

10/ 40© 2024 Fujitsu Limited

The Impact of Job Switching Jitter⚫ Impact of jitter on communication performance*1switchswitch
Time→

Jittersend

Node #0

Node #1

Wait to be
activerunning

Immediate
response

suspended…

…

Longer round-trip timecaused by jitter

Parallel applications with network communication(MPI) require precise synchronization
11/ 40© 2024 Fujitsu Limited

Scalable Synchronization with
Broadcast Control Signals<Unicast><Broadcast>Controller

12

Controller*1n…

Broadcasted Signal

Send signals SEQUENTIALLY…
Compute
Nodes

Node1

Switch job

job1

Node2

NodeN…

NodeN

Node1

Node2

Switch job

job1

job1

job1

Switch job

job2

job2

job2

job1
job1

Switch job

Time

job2
job2

Jitter caused by sequential
unicast communications

Time

job2

12/ 40Good synchronization with a single
broadcast communication© 2024 Fujitsu Limited

Unreliable Broadcast Synchronization in
Large-Scale Gang Scheduling Systems
Broadcast sync. Signal(Unreliable communication. Sent via IP over InfiniBand network)

Compute
Nodes
…
30

32

#030

3132

Job list on
Scheduler (Slurm)
JOB_ID

#1

31

30

32

#230

31

32

#N

31

Obtaining Process IDs for Each Job ID
JOB_ID

PID_0

PID_1

PID_2 …

3012345

12346

12347

312345623457

323456734568

Table on Node #013/ 40

Active Job ID© 2024 Fujitsu Limited

Unreliable Broadcast Synchronization in
Large-Scale Gang Scheduling SystemsBroadcast sync. Signal(Unreliable communication. Sent via IP over InfiniBand network)

Compute
Nodes

30

32

#0

31

32

1. Send SIG_STOP to…PIDs of Job 30(12345, 12346, 12347) #N
#2
#130
30
30
2. Send SIG_CONTto PIDs of Job
31(23456, 23457)
3132

31

32

31

Job list on
Obtaining Process IDs for Each Job ID
Scheduler (Slurm)
JOB_ID
PID_0
PID_1
PID_2 …
Unreliable broadcast provides better synchronicity than30
12345
12346
12347
JOB_ID
Active Job ID
unicast-based
communication
3123456
23457
What happens
if thesynchronization
message is dropped?
323456734568
14/ 40 doesn’t lead to critical error-> It just fails to switch
jobs#0but
Table on Node© 2024 Fujitsu Limited

Broadcast Messages:
Unreliable Yet Rarely Dropped in Practice⚫ Paper [1] reports zero packet loss during thetransfer of 50PB data on the InfiniBand Network⚫ [1] Kalia et al, FaSST: Fast, Scalable and Simple Distributed Transactions with Two-Sided(RDMA) Datagram RPCs, USENIX OSDI’16⚫ Similarly, we observed no packet loss during ourexperimental testing of this system

15/ 40© 2024 Fujitsu Limited

What Happens ifBroadcast Messages are Dropped?
⚫ Broadcast Control Message Drop:⚫ Leads to failure in job switching.
⚫ However, it does not cause any error, but rather continues thecurrent job.

⚫ Performance degradation:⚫ The degradation caused by the drop is limited.
⚫ If 0.001% of messages are dropped, it results in just 0.001% ofperformance degradation.

16/ 40© 2024 Fujitsu Limited

Demos⚫ Running two rdbench* visualization jobs
⚫Simulation programs are running on a parallel cluster⚫Demonstrating how short jobs start immediately with Gangscheduling⚫ Jupyter Notebook Interactive Demo
⚫ Running Multiple 256-node MPI Jobs Concurrently* K. Hiraga, rdbench, https://github.com/range3/rdbench
17/ 40© 2024 Fujitsu Limited

Demo #1
⚫ Running two rdbench* visualization jobs
⚫Simulation programs are running on a parallel cluster⚫Demonstrating how short jobs start immediately with Gangscheduling* K. Hiraga, rdbench, https://github.com/range3/rdbench
18/ 40© 2024 Fujitsu Limited

19/ 40© 2024 Fujitsu Limited

20/ 40© 2024 Fujitsu Limited

21/ 40© 2024 Fujitsu Limited

22/ 40© 2024 Fujitsu Limited

Demo #2
⚫ Jupyter Notebook Interactive Demo⚫Instances of IPython parallel are running on a parallel cluster⚫Comparison between batch scheduling and gang scheduling23/ 40© 2024 Fujitsu Limited

24/ 40

25/ 40© 2024 Fujitsu Limited

26/ 40© 2024 Fujitsu Limited

Providing interactive parallel programming environment
27/ 40and running large-scale real-time
applications immediately© 2024 Fujitsu Limited

Demo #3 – large scale jobs:
Running three 256-node MPI jobs
⚫ Running multiple 256-node MPI sample programs⚫ Running on ARM-based compute nodes⚫ Multiple jobs are submitted to a 256-node Slurm partition⚫ Using Allreduce operations on each iteration⚫ Time-slice duration is 100ms
Evaluation Environment (Demo 2 and evaluation)
Computing Node:
FUJITSU Supercomputer PRIMEHPC FX700
CPU: FUJITSU A64FX Arm8.2-A SVE
48cores @ 2.0GHz
RAM: 32 GiB HBM2
Network interface: InfiniBand HDR100
Interconnect: Full-bisection bandwidth fat-tree
OS: Rocky Linux 8.5
Slurm: 22.05
OpenMPI: 4.1.5

28/ 40© 2024 Fujitsu Limited

29/ 40© 2024 Fujitsu Limited

30/ 40© 2024 Fujitsu Limited

31/ 40© 2024 Fujitsu Limited

32/ 40© 2024 Fujitsu Limited

33/ 40© 2024 Fujitsu Limited

Evaluation* SC23 Research Poster, Scalable Fine-Grained Gang Scheduling for HPC Systems with Unreliable Broadcast Synchronization Mechanisms⚫ Performance of communication⚫ Comparison: Our broadcast sync andSlurm/OpenMPI signal propagation⚫ Communication performance⚫ MPI AlltoAll: Improved scalability

Scalableimprovement

and a real application(0.5sec time slice)⚫ Application performance⚫ 2.7x speedup on GROMACS (128nodes)⚫ The overhead caused by time-sliced
execution was only 6.3%

Unreliable Broadcast synchronization enables
34/ 40 large-scale clusters
scalable interactivity on© 2024 Fujitsu LimitedImproved Interactivity and Throughput● We have already deployed the mechanism on the actualsystem● Integrated with Slurm Scheduler

Ordinally batch schedulingTime● We confirmed● Shortened waiting time
● Job throughput accelerationon our on-premise clusterGaps to prepare for large jobs

With our gang schedulingNodes

gaps caused by large jobs

Nodes

● Successfully eliminated

35 / 40Time

Gaps are eliminated by
the gang scheduling© 2024 Fujitsu LimitedImproved Interactivity and Throughput● We have already deployed the mechanism on the actualsystem● Integrated with Slurm Scheduler

Ordinally batch scheduling

Timegaps caused by large jobs

● We confirmed

Nodes● Successfully eliminatedGaps to prepare for large jobs

With our gang scheduling

Nodes● Shortened waiting time
● Job throughput accelerationon our on-premisecluster
We have implemented
this mechanism on
the 1056-nodes ARM-based cluster.
More results will 36be/ 40 available soon!

TimeGaps are eliminated by
the gang scheduling© 2024 Fujitsu Limited

実システムへの適用および開発状況● 富士通の量子コンピューティングシミュレータに適用済● FX700 1056ノードのクラスタシステム● 詳細はPCCC AI/HPC OSS活用ワークショップ（2024/2/5）における発表を参照● インタラクティブHPC技術の動作環境● 現時点においてはSlurm環境に適用可能● その他のスケジューラも仕組み上は対応可能。詳細はお問合せください。

● お試しパッケージ● システム全体に適用することなく、ユーザ権限でお試しいただけるパッケージの提供を準備中です● PoC先募集中！

37 / 40© 2024 Fujitsu Limited

Conclusion

38 / 40© 2024 Fujitsu Limited

Conclusion● インタラクティブHPC技術● HPCシステムのインタラクティブ性と効率を向上するための技術● 汎用ハード・既存システムに上乗せで使用可能なGang Scheduling● 全系高精度同期技術により、並列アプリケーションの時間分割実行を高効化● 全系ジョブによる実行遅延を減らすとともに、システム利用効率を向上● 実システムへの適用実績および開発状況● 1056ノードのFX700クラスタにおいて実運用中(2024年2月～)● お試しパッケージを準備中。ご興味のある方はご連絡ください。

● ご質問・ご連絡は以下までお願いいたします● ohtsuji.hiroki at fujitsu.com

39 / 40© 2024 Fujitsu Limited

Thank you

40 / 40© 2024 Fujitsu Limited