PFNにおける機械学習の取り組みとHPCについて

福田 圭祐
Preferred Networks, Inc.

Who we are?
Preferred Networks, Inc. (PFN):A Tokyo-based Deep Learning & IoT company

2

Our Strategic Partners

and Collaborators

2015: OPTIMIZATION OF BIN-PICKING FANUC ROBOTS• Picking random object is atypical task that is “easy forhuman, hard for robots”.

@CES 2016: CARS THAT DON’T CRASH• Car positions are tracked
from a ceiling camera andeach car is controlledindividually.
• White cars are autonomous• The red car is a manuallycontrolled “evil” car: trying
to disrupt other cars@ICRA 2017 VOICE RECOGNITION + OBJECT PICKING“Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions”

arXiv:1710.06280••

ICRA is a top-tier conference on robotics
Best Paper Award on Human-Robot Interaction•

Technologies:• Visual recognition• Natural language processing (NLP)•

The robot can understand ambiguous words:• ”The Teddy bear”• ”The brown fluffy stuff”

6https://projects.preferred.jp/tidying-up-robot/@CEATEC JAPAN 2018 Autonomous Tidying-up Robot Systemx27

Technologies behind the demos:
Distributed Deep Learning8

MN-1: An in-house supercomputerl MN-1a (Sep. ʻ17~)━ 1024 NVIDIA Tesla P100
━ InfiniBand FDR━ Peak 9.3 Peta FLOPS (SP)━ #227 in Top500 Nov. 2018l MN-1b (July. ʻ18~)━ 512 NVIDIA Tesla V100 32GB
━ InfiniBand EDR━ Peak 56 Peta (tensor) Flopsl Targeting

Exa FL ops by 2020

Chainer:
A Flexible Deep Learning Framework
Define-by-Run

Define-and-Run
DefineModeldefinition

Define-by-Run
Computational
graph

Ca

Gradient
functionModeldefinition
Training
data

Run
Training
data

Computational
graph

Caffe2, TensorFlow etc.

Computational
graph

Gradient
function

Gradientfunction

10

PyTorch, TensorFlow(Eager Execution) etc.

ChainerMN: Distributed Training with Chainer••

Add-on package for Chainer
Enables multi-node distributed deep learning using NVIDIA NCCL2Features• Scalable: Near-linear scaling with hundreds of GPUs
• Flexible: Even GANs, dynamic NNs, and RL are applicable

Distributed Training with ChainerMN
ForwardBackward

ForwardBackward

ForwardBackward

11Optimize
AllReduce

Optimize
OptimizeAchievement on MN-1a: ImageNet in 15 minutes
Training time of ResNet-50 (90 epochs) on ImageNet70

Time [min]605040arXiv: 1711.04325
Extremely Large Minibatch SGD: Training ResNet-50
on ImageNet in 15 Minutes

3020102017 Nov
2018 July2018 Nov 2018 Nov

012

Achievement on MN-1b: PFDet in OIC 201813

Achievement on MN-1b: PFDet in OIC 2018••Google AI Open Images - Object Detection Track– Competition using Largest-class image dataset– 12 million bounding boxes, 1.7 million images– 454 competitiors– Approx. 500GB (annotated subset)
Object detection: much harder than object recognition task

14

Object Detection
Detecting objects in an imageby predicting...
• bounding boxes that
contain them• category of the objects

Achievement on MN-1b: PFDet in OIC 2018
• We won the 2nd position (0.023% diff to the 1st)

17

Open Sourcing PFDet:

We may make theimplementation publicTechnical report is already on arXiv:
arXiv:1809.00778

Computation resources used in PFDet••

Single training process of 16 epochs takes 33 hours with 512 xV100 GPUs of MN-1b
Repeated model development & parameter tuning
June 3rd:First commit

June 29th:
FPN added

Aug. 31st:
Finish19

Autonomous Tyding-up robot
Integration of a wide range of DL:•

Object Detection(based on PFDet)

20•

Audio recognition

•

NLP

•

Picking planningTechnical topics
1. Communication & Fault tolerance
2. Storage
3. Resource Management

21Communication
Allreduce: that always matters•

Critical component for distributed deep learning•

NVIDIA NCCL is the de-facto standard communication library•

Typical usage in ChainerMN:– MPI for process coordination– NCCL for actual communication
22

Communication

Better

Open MPI 2.1.3(configured as CUDA-aware)
NCCL is the fastest productionready library
Our demonstration-purposed
prototype library

From our research blog:
https://bit.ly/2Pt2Gcg
23

Communication
NCCL is now open-sourced 🎉• Very interesting internal architecture• NCCL still has some bug in large scale execution… we know what to do to an
OSS software? J24

Communication: open questions (1)•NCCL is optimized for bandwidth-bound (i.e. large buffer)
communication•

What about short-messages?
– Inter-process Batch Normalization– Model parallelism•

Fault tolerance?
– NCCL now supports communication timeout...
– But the MPI standard does not support FT (yet?)25Communication: open questions (2)•Do we still use MPI…?
– Legacy– The software stack is huge: installation and maintenance is hardfor non-HPC users– No fault tolerance– No separation of process coordination and communication– Hard to use with Kubernetes

26

Resource Management•

We deploy Kubernetes on our computing cluster.

•Why Kubernetes?
– Advanced features from the cloud computing community• Fault tolerance, dynamic job size management, preemption, etc.
– We also deploy server-based services (such as JupyterHub, CI, etc.) onthe same cluster•

Still many challenges– Batch job scheduling,27

多種多様な実験環境-

静止画、動画、音声、テキスト
深層学習、分散深層学習、強化学習
InfiniBand

Device pluginでリソース化
k8s-host-device-plugin - https://github.com/everpeace/k8s-host-device-plugin

プライベートDockerレジストリ
Container
大規模データNode SelectorでCPU/GPUの選択
Icon pack by Icons8 - https://icons8.com
Icons by iconspng - https://www.iconspng.com

Pod Affinityでできるだけホップ数が近い物理サーバに配置

NFS

小規模データ28

効率的なスケジューリング
-毎日、数百〜数千件のジョブ
試行錯誤の実験
ハイパーパラメータチューニング

スケジュール- 分散深層学習向けのスケジューリング
- 待ち時間による優先度の重みづけ

CustomScheduler

NodePriorityClassによるプリエンプション

webhook
kube-throttlerOperator/CRD
Icon pack by Icons8 - https://icons8.com

PriorityClass毎にユーザが実行できるPod数を制御(例) 優先度毎に高：１、中：５、低：無限
kube-throttler - https://github.com/everpeace/kube-throttler

29マルチテナント-

ユーザ毎、プロジェクト毎の環境
コンフィデンシャルな案件

PSP, Admission WebhookによりLDAPのユーザIDをセット

Namespace
User A

Namespace毎のメトリックス

Network Policy
による通信制御

User A

Namespace
Project X
User B

RBACによるNamespaceへのアクセス制御

NFS
Namespace
User B

Icon pack by Icons8 - https://icons8.com

Kerberos認証,LDAPのユーザIDによるアクセス30

自由度の担保-

様々なリテラシの研究者
特定のツールでロックインしない
実験環境の自由なカスタマイズ
実行環境に直接入ってデバッグ
様々なツールからの利用Kubectl
GPUを手軽に使いたい

自社製実験ツールkubectl exec

実行中のPodに入ってデバッグ
NFS
Homeにマウントして物理サーバのように使う

Icon pack by Icons8 - https://icons8.com

31

グランドチャレンジ-

全リソースを使った実験--

ImageNet in 15 minutes (Tesla P100 x 1024)
PFDet in the Kaggle 2018 Google AI Open Images (Tesla V100 x 512)

日常のジョブはプリエンプションされる前提で使う
グランドチャレンジのジョブは最大の優先度
任意のタイミングで全GPUを使った実験が可能

Icon pack by Icons8 - https://icons8.com

32

Resource Management: Open Questions•

Preemption is a strong tool for efficient resource management• How DL framework and scheduler cooperate ?
• Job preemption/restarting & flexible resource (re-)allocation•Resource isolation• Bandwidth isolation for Infiniband?
• NUMA- / topology- aware pod placement?
NOTE: We operate an in-house cluster, so all jobs are our own.
33

HPCとAIの関係

AIのためのHPC

HPCのためのAI

HPCとAIの融合（AI技術の⼀般化）

34

AIのためのHPC•

HPCからAIへの貢献– DL/MLはHPCの1アプリ。HPCなくしてAIは無い•

今後の課題– 計算（FLOPS）＝もっと– メモリ＝もっと– ストレージ• セキュリティとアクセスコントロール／暗号化– スケジューラーと計算環境の進化・統合• コンテナ／再現性の担保／サービスとの混在（JupyterHub etc.）／遊休計算資源の有効活⽤
35

HPCのためのAI•

AIの対象としてのHPC– パラメーターサーベイ• DLのハイパーパラメーターの多さからメタパラメーターチューニングが発達• →HPCへ還流– 電⼒効率、故障検知、 ジョブスケジューリング etc.
• 特に強化学習が鍵••

Li “Transforming Cooling Optimization for Green Data Center via Deep ReinforcementLearning” arXiv:1709.05077v4
Mirhoseini et al. “Device Placement Optimization with Reinforcement Learning” ICML 201736

HPCとAIの融合（AI技術の⼀般化）••

HPCに限らず、AIは、コンピューターサイエンスのコア技術になっていく– c.f. 「演繹から帰納へ〜新しいシステム開発パラダイム〜」丸⼭宏, PPL2018 招待講演
– 特別なものではなく、実装⼿法の1つとして広く使われるようになっていくのでは？
HPC系アプリには、AIと相性の良いものが多いのでは？
AIが向いている場⾯

AIが不向きな場⾯•

データが⼤量（⽣成可能）•データが少ない•

誤差が許容される•

厳密さが必要•現象が複雑／原理が不明•

数値解析が可能•法則・原理が⼀定•

過去から未来が予測できない

•

予測が⽬的

•

メカニズムの理解が⽬的

37

Thank you!

38