富岳を用いた大規模言語モデルの分散並列学習
Distributed Training of Large Language Models on Fugaku

Tokyo Institute of Technolog
GSI
Rio Yokota
2022年度第2回計算科学フォーラムyC

2023年3月28日(火)GPT-Fugaku Team

Noriyuk
Kojima

Kazut
AndoCollaborators

Koj
Nishiguchi

Jung
Kasai

Keisuk
Shuka
Sakaguchi Nakamura

DL4Fugaku Team @ R-CCS

Aleksand
Drozd

Mohame
Wahib

Kent
Sato

DL4Fugaku Team @ LLNL

rideoii

ioolisl

Nikol
DrydenTa
Ben NunJen
DomkeEmi
VataiFujitsu

Koich
Shirahata

Large Language Models
Chain of Thought Prompting

Fastest growing app in history

https://nerdynav.com/chatgpt-statistics/

Will change the landscape of• Web searc• Machine translation & summarizatio• Creative writing in research & educatio• Coding & Debugging
nn…

hhttps://arxiv.org/abs/2201.11903

Multimodal Language Models

Will change the landscape of• Robotics & Autonomous drivin• Medicine & Scienc• Explainable AIg…

e

PaLM-E: An Embodied Multimodal Language Model, https://arxiv.org/abs/2303.03378

Deep Learning Scaling Law

Compute Trends Across Three Eras of Machine Learning, https://arxiv.org/abs/2202.05924

Moore’s law: 1970-2020の50年で107倍
DL Scaling law: 2010-2022の12年で1010倍

Emergent Abilities at1022-1024 FLOPs

Emergent Abilities of Large Language Models, https://arxiv.org/abs/2206.07682

How Long It Will Take to Train GPTGPT-4: 3x1025FLOPs (speculated
GPT-3.5 (ChatGPT): 3x1024FLOPs (speculated
GPT-3: 3x1023FLOPFugaku
FP32 6.76TFLOP/s x 158,976 = 1.07 EFLOP/s (theoretical peak
GPT-4: 328 days x 10
GPT-3.5: 32 days x 10 x 2 を目標に研究開発中
GPT-3: 3.3 days x 10OpenAI
BF16 312 TFLOP/s x 25,000 = 7.8 EFLOP/s (theoretical peak
GPT-4: 45 days x 2
GPT-3.5: 4.5 days x 2
GPT-3: 11 hours xActual Performance))))s2::

https://lifearchitect.ai/gpt-4/#sizeCost of GPT vs Weather Simulation

A 1024-Member Ensemble Data Assimilation with 3.5-Km Mesh Global Weather Simulations, SC 2020 (Gordon Bell Finalist)
provided by Mohamed Wahib, R-CCS

A Gordon Bell submissions will use a few hours of the whole Fugaku Training GPT-4 will take a year on the whole Fugaku even with its FP32 peakRelated ProjectsABCI GC FY2019: Second Order Optimizatio
Trained ImageNet in less than 2 minutes with 131k batch size (2048 GPUs
HPCI FY2020: Minimizing Memory Footprin
Reduced memory consumption to allow training of 14B parameter model
on a single nod
HPCI FY2021: Minimizing I/
Training directly from tar les, reduced I/O latency and inodes by 1/100
HPCI FY2022: Training Vision Transformers on Synthetic Dataset
Surpassed the accuracy of ImageNet-21k using a purely synthetic datase
HPCI FY2023: Performance Optimization of Transformers on A64FX andTheir Application to Vision & Languag)

t0rsnteOfie

INCITE FY2023: Large Vision+Language Models on Summit/Frontie
6M GPU hours allocated (PI: Irina Rish, U. Montreal)Distributed Training
Data Parallel (DP)Tensor Parallel (TPoZeRO/FSDP

Pipeline Parallel (PP)Distributed Dat

Redundant Dat

Distributed Dat

Redundant Mode

Distributed Mode

Distributed Mode

AllReduce Gradient

AllReduce ActivationoAllGather Parameter

SendRecv Activation

Large batch problem

Pipeline bubble

Frequent Communication
flssss)lllaaarr“Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis”, Ben-nun and Hoe er, ACM Computing Surveys, Article No.: 65

What is Strong Scaling in Deep Learning?
Scienti c Computing

Given a certain mesh size

Reduce solution time by partitioning

Deep Learning

Given a certain model and data size

Reduce time per step by partitioningfiReduce number of steps by partitioningPipeline ParallelfiChimera: Ef ciently Training Large-Scale Neural Networks with Bidirectional Pipelines, https://arxiv.org/abs/2107.06925

Tensor Parallel
Apply SUMMA to Attention Layer

Ef cient Large-Scale Language Model Training on GPU Clusters Using Megatron-L
https://arxiv.org/abs/2104.04473

Myfifi

Tesseract: Parallelize the Tensor Parallelism Ef cientl
https://arxiv.org/abs/2105.14500

Zero Redundancy OptimizerrsrrfidAllGathe
Parameteran
OptimizeStatesReduceScatte
Gradients
ZeRO-In nity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning, https://arxiv.org/abs/2104.07857

Megatron-DeepSpeed

https://github.com/microsoft/Megatron-DeepSpeed

Data Parallel (DP)Tensor Parallel (TPoZeRO/FSDP

Pipeline Parallel (PP)Distributed Dat

Redundant Dat

Redundant DatRedundant Mode

Distributed Mode

Distributed Mode

AllReduce Gradient

AllReduce ActivationoAllGather Parameter

SendRecv Activation

Large batch problemssss)

ll

l

aaar

r

Frequent Communication

Pipeline bubble

Strong Scalability of Data Parallel
sequence-length=1024
per-cpu-batchsize=1, global-batch-size=1024
gradient-accumulation-steps=1024/#nodes#parameters=124M

Breakdown of Data Parallel

Backward time scales
Forward time scales except for…
batch-generator and average_losses
AllReduce time is constantModel Parallel
sequence-length=1024
per-cpu-batchsize=1024, global-batch-size=1024
gradient-accumulation-steps=1#parameters=1.3B

Only scales up to 8 nodes at the momentBreakdown of Model Parallel

Backward time scales until 8 nodes
Forward time scales well
AllReduce time increases rapidly

FLOPs Achieved on 1.3B Modelsequence-length=1024
per-cpu-batchsize=1, global-batch-size=1024
gradient-accumulation-steps=1024/#DP#parameters=1.3B

We are only getting around 10% of the theoretical peak of A64fx at the moment

99% of the FLOPs is GEMMA64FX

A100Currently uses batched GEMM implementation by Daichi Mukunoki→ Achieves 2 TFLOPs (FP32) on a single A64FX
https://www.r-ccs.riken.jp/labs/lpnctrt/projects/batchedblas/index.html

Du Wu and Mohammed Wahib are also working on a faster version

Summary and OutlookFugaku
FP32 6.76TFLOP/s x 158,976 = 1.07 EFLOP/s (theoretical peak
GPT-4: 328 days x 10
GPT-3.5: 32 days x 10
GPT-3: 3.3 days x 10
Actual Performance
HPC task• Optimizing batched GEMM to scale across CMG• Develop techniques to enable FP16 trainin• Optimize non-GEMM operations on A64F• Reduce communication overhea
NLP task• Collecting, downloading, and cleaning large multilingual corp
• Discuss legal issues with lawyers https://storialaw.jp/blog/9239• New models appearing every week: Alpaca,LLaMA, RWK https://github.com/Hannibal046/Awesome-LLM• Reinforcement learning with human feedback (RLHF)

aV)

sgX 

dss

:

Reference on the NLP side : Slides by Naoaki Okazaki https://speakerdeck.com/chokkan/20230327_riken_llm