Institute Seminar | Daniel Coquelin | 2022-09-20 | 1/2 Presentation
Channel: CASUS.science
Date: 13/12/2022

TITLE: Training neural networks with large datasets in parallel by Daniel Coquelin, Karlsruhe Institute of Technology (KIT), Steinbuch Centre for Computing (SCC)

Daniel is an AI Consultant in the Helmholtz AI Local Energy Consulting Team at KIT’s SCC led by Marcus Götz. He works on the optimization of computing and communication operations for AI algorithms on high-performance computers. Daniel’s focus is mainly on data and model parallel neural networks.

ABSTRACT: As datasets and neural networks grow, their training becomes more and more difficult. Current state-of-the-art networks are trained on tera- or petabytes of data and hundreds of GPUs. To achieve this, Daniel and his colleagues use a multitude of methods. In this talk he will discuss the most common form of neural network parallelism: batch parallelism. This method replicates the network on each accelerator and splits the data between them, after each batch the models are synchronized using the data that each of them found. While this remains trivial at small scales, the process becomes much more challenging when many networks are attempting to aggregate what the have found. After this background he will introduce the DASO method, a hierarchical method of merging these networks.

LICENSE:
CC BY-SA 4.0

CASUS – Center for Advanced Systems Understanding
Helmholtz-Zentrum Dresden-Rossendorf e.V. (HZDR)
Untermarkt 20
D-02826 Görlitz