Best Practices for Distributed Deep Learning on Ibex - 2021
Channel: KAUST Visualization Core Lab
Date: 2021-05-05

Advanced introduction to using Ibex effectively for Distributed Deep Learning (DL). Visualization, Computational, and Data Analytics experts from the laboratory introduce a series of best practices designed to improve performance and utilization for Distributed DL workflows. 

We discuss and demonstrate the use of Horovod, a scalable distributed training framework to train DL models on multiple GPUs on Ibex. We also discuss performance caveats when scaling-up Horovod for large mini-batch training.

Chapters:
0:00:00​​ Introduction
0:00:46​ MPI and the Mechanics of Distributed Learning 
0:28:26​ Getting Started with Horovod
1:03:27 Scaling Deep Learning
1:43:33​​ Benchmark Results and Review
1:56:50​ Concluding Q&A