Distributed Deep Learning with Horovod | by Jordi TORRES.AI | Towards Data Science

[This post will be used in the master course Supercomputers Architecture at UPC Barcelona Tech with the support of the BSC]
In the previous post we explored how we can scale the training on Multiple GPUs in one Server with TensorFlow using tf.distributed.MirroredStrategy(). Now, in this post, we will use Horovod API to scale the training on multiple servers following data parallelism strategies.
Uber Engineering introduced Michelangelo, an internal ML-as-a-service platform that makes it easy to build and deploy these systems at scale. Horovod, a component of Michelangelo, is an open-source distributed training framework for TensorFlow, PyTorch, and MXNet. Its goal is to make distributed Deep Learning fast and easy to use via ring-allreduce and requires only a few lines of modification to user code. Horovod is available under the Apache 2.0 license.
Conceptually, the data-parallel distributed training paradigm under Horovod is straightforward:
1. Run multiple copies of the training script and each copy:
reads a chunk of the data
runs it through the model
computes model updates (gradients)
2. Average gradients among those multiple copies
3. Update the model
4. Repeat (from Step 1)
Horovod applies Baidu’s algorithm for averaging gradients and communicating those gradients to all nodes (steps 2 and 3 above) that follows the ring-allreduce decentralized scheme. The algorithm was based on the approach introduced in the 2009 paper by Patarasuk and Yuan. Horovod replaced the Baidu ring-allreduce implementation with NCCL-2, which is NVIDIA’s library for collective communication that provides a highly optimized version of ring-allreduce across multiple machines.
The following figure from the paper by Sergeev and Balso shows the ring-allreduce algorithm that allows workers nodes to average gradients and disperses them to all nodes without the need for a centralized scheme with a parameter server.
A more clear and visual explanation can be obtained in this post from Medium: “Visual intuition on ring-allreduce for distributed Deep Learning”.
In this ring-allreduce algorithm, each of N nodes communicates with two of its peers 2∗(N−1) times. During this communication, a node sends and receives chunks of the data buffer. In the first N-1 iterations, received values are added to the values in the node’s buffer. In the second N-1 iterations, received values replace the values held in the node’s buffer. Patarasuk and Yuan suggest that this algorithm is bandwidth-optimal, meaning that if the buffer is large enough, it will optimally utilize the available network.
Horovod is a python package installed using pip, and in general it assumes installation of MPI for worker discovery and reduction coordination and Nvidia’s NCCL-2 libraries to support inter-GPU communication. This is because MPI is used extensively in the supercomputing community for high-performance parallel computing.
However, if MPI is not installed, Horovod includes Gloo, an open-source collective communications library developed by Facebook. You can choose to use Gloo at runtime by passing the --gloo argument to horovodrun:
$ horovodrun --gloo -np 2 python train.py
Horovod introduces an hvdobject that has to be initialized and has to wrap the optimizer (Horovod averages the gradients using allreduce or allgather). A GPU is bound to this process using its local rank, and we broadcast variables from rank 0 to all other processes during initialization.
A Horovod Python program is launched using the mpiruncommand. It takes as parameters the hostname of each server as well as the number of GPUs to be used on each server.
To use Horovod in Tensorflow, you must make the following additions to your program:
Import Horovod:
Import Horovod:
import horovod.tensorflow as hvd
2. Horovod must be initialized before starting using:
hvd.init()
3. Pin each GPU to a process.We will be employing local rank for that, such that, the first process in the server will be pinned to the first GPU; the second process to the second GPU, and so forth:
gpus = tf.config.experimental.list_physical_devices('GPU')for gpu in gpus:    tf.config.experimental.set_memory_growth(gpu, True)if gpus:    tf.config.experimental.set_visible_devices(                 gpus[hvd.local_rank()], 'GPU')
4. Scale the learning rate by the number of workers. Effective batch size in synchronous distributed training is scaled by the number of workers. An increase in learning rate compensates for the increased batch size. You can know more about this in this Facebook’s paper.
opt = tf.keras.optimizers.SGD(0.0005 * hvd.size())
5. Wrap optimizer in hvd.DistributedOptimizer. The distributed optimizer delegates gradient computation to the original optimizer, averages gradients using allreduce or allgather, and then applies those averaged gradients.
opt = hvd.DistributedOptimizer(opt)
6. Specify experimental_run_tf_function=False to ensure TensorFlow uses Horovod’s distributed optimizer to compute gradients.
7. Add hvd.callbacks.BroadcastGlobalVariablesCallback(0) to broadcast initial variable states from rank 0 to all other processes. This is necessary to ensure consistent initialization of all workers when training is started with random weights or restored from a checkpoint.
callbacks = [      hvd.callbacks.BroadcastGlobalVariablesCallback(0)]
8. If you need to save checkpoints, do it only on worker 0 to prevent other workers from corrupting them. Or if you want to run evaluation or to print information to the standard output, it is recommended to do it on worker 0. This can be accomplished with hvd.rank() = 0.
if hvd.rank() == 0:   print(model.summary())
To show how to use Horovod, we will use the same example presented in the previous post, that trains a classifier of CIFAR10 dataset based in a ResNet50, and train it on the CTE-POWER machine.
Following the steps presented in the above section on how to use Horovod API, below we present the resulting parallel code:
import tensorflow as tffrom tensorflow.keras import layersfrom tensorflow.keras import modelsimport horovod.tensorflow.keras as hvdimport numpy as npimport argparseimport timeimport syssys.path.append(‘/gpfs/projects/nct00/nct00002/cifar-utils’)from cifar import load_cifarhvd.init()parser = argparse.ArgumentParser()parser.add_argument(‘ -- epochs’, type=int, default=5)parser.add_argument(‘ -- batch_size’, type=int, default=256)args = parser.parse_args()batch_size = args.batch_sizeepochs = args.epochsmodel_name = args.model_namegpus = tf.config.experimental.list_physical_devices(‘GPU’)for gpu in gpus:    tf.config.experimental.set_memory_growth(gpu, True)if gpus:    tf.config.experimental.set_visible_devices(    gpus[hvd.local_rank()], ‘GPU’)train_ds, test_ds = load_cifar(batch_size)model = tf.keras.applications.resnet_v2.ResNet50V2(        include_top=True, weights=None,        input_shape=(128, 128, 3), classes=10)if hvd.rank() == 0:    print(model.summary())opt = tf.keras.optimizers.SGD(0.0005 * hvd.size())opt = hvd.DistributedOptimizer(opt)model.compile(     loss=’sparse_categorical_crossentropy’,     optimizer=opt,     metrics=[‘accuracy’],     experimental_run_tf_function=False)callbacks = [     hvd.callbacks.BroadcastGlobalVariablesCallback(0)]if hvd.rank() == 0:   verbose = 2else:   verbose=0model.fit(train_ds, epochs=epochs,           verbose=verbose, callbacks=callbacks)
After modifying your sequential python script with the inclusion of Horovod API calls, you must write your job script (.sh file) to submit the job to the SLURM workload manager as we did in the previous section.
For this concrete example, presented in the previous section, the SLURM script to send the jobs to run on a server with 4GPUs looks like this:
#!/bin/bash#SBATCH --job-name horovod1#SBATCH -D .#SBATCH --output hvd_1_%j.output#SBATCH --error hvd_1_%j.err#SBATCH --nodes=1#SBATCH --gres='gpu:4'#SBATCH --ntasks-per-node=4#SBATCH --cpus-per-task 40#SBATCH --time 00:25:00module purge; module load gcc/8.3.0 cuda/10.2 cudnn/7.6.4 nccl/2.4.8 tensorrt/6.0.1 openmpi/4.0.1 atlas/3.10.3 scalapack/2.0.2 fftw/3.3.8 szip/2.1.1 ffmpeg/4.2.1 opencv/4.1.1 python/3.7.4_MLhorovodrun -np $SLURM_NTASKS -H localhost:$SLURM_NTASKS --gloo \python3.7 tf2_keras_cifar_hvd.py --epochs 10 --batch_size 512
If we take a look at this SLURM script, we can find the following variables that will determine how our model will be distributed:
nodes: Amount of servers that will be employed to distribute the workload
gres: Amount of GPUs’s per server that will be used.
ntasks-per-node: Number of processes running in a server. In Horovod each GPU gets pinned to a process, therefore this should be equal to the number of GPUs employed in each server (it must be added to create SLURM_NTASKS variable that will be required for the -np flag in horovodrunthat especifies the number of processes).
cpu-per-task: In CTE-POWER each server holds 160 cores, therefore this parameter must be 160/ntasks-per-node to employ the maximum amount of cores per process.
Detailed information about the possible flags for horovodrun command can be obtained executing the command horovodrun --help.
In the example from above we are using one server with 4 GPUs. If we would like to employ 8 GPUs to do our computation we would need 2 servers, as each server in the CTE-POWER has 4GPUs. To run on more than one server (for example, on two), we need to use the following SLURM script:
#!/bin/bash#SBATCH --job-name horovod-multinode#SBATCH --D .#SBATCH --output hvd_multinode_%j.output#SBATCH --error hvd_multinode_%j.error#SBATCH --nodes=2#SBATCH --gres=’gpu:4'#SBATCH --ntasks-per-node=4#SBATCH --cpus-per-task 40#SBATCH --time 00:50:00module purge; module load gcc/8.3.0 cuda/10.2 cudnn/7.6.4 nccl/2.4.8 tensorrt/6.0.1 openmpi/4.0.1 atlas/3.10.3 scalapack/2.0.2 fftw/3.3.8 szip/2.1.1 ffmpeg/4.2.1 opencv/4.1.1 python/3.7.4_MLHOSTS_FLAG="-H "for node in $(scontrol show hostnames "$SLURM_JOB_NODELIST"); do   HOSTS_FLAG="$HOSTS_FLAG$node-ib0:$SLURM_NTASKS_PER_NODE,"doneHOSTS_FLAG=${HOSTS_FLAG%?}horovodrun --start-timeout 120 --gloo-timeout-seconds 120 \-np $SLURM_NTASKS $HOSTS_FLAG --network-interface ib0 --gloo \python3.7 tf2_keras_cifar_hvd.py --epochs 10 --batch_size 512
Where we employ 2 servers, each server using 4 GPUs, therefore we will be using 8 GPUs in total.
Remember that in Post 2 of this series, the reader can find more detailed information about all the #SBATCH commands we usually use in our supercomputer.
The previous listing shows different names, than the previous one for the #SBATCH options. It is done deliberately with the purpose of showing that SLURM has some “flexibility” with the names used.
In this section we are going to present an experiment that will help us show how the Horovod API works and, at the same time, be able to experiment with its scalability.
The reader will surely think that the canonical test that would be more suitable to evaluate the scalability of Horovod is to perform a set of tests where the same number of epochs are executed with different GPUs: 1,2,4,8,16 and 32. But this is not feasible given the availability of the resources that we have assigned in the master course.
To begin with, a larger problem than that of classifying the CIFAR10 would be required to show results that exhibit that take advantage of the computing power of 32 V100 NVIDIA GPUs.
Also, it will be required to run a minimum number of epochs per execution to be able to dampen the required initialization. Let’s assume that with the test of 32 GPUs, for example, we would perform a minimum number of 8 epochs. In the ideal canonical comparison test, this would involve running 256 epochs on 1 GPU (32x8) in order to compare. Even in this small problem of classifying the CIFAR40 dataset with a ResNet50, each epoch takes about 40 seconds. That means about 3 hours to train the model with 1 GPU — something totally unfeasible given the resources available for this master course.
Likewise, it must be taken into account that a hyperparameter tuning should be performed, which requires many tests. And finally, remember that we should carry out several executions to give statistical validity to the tests given the innate randomness in the behavior of the software stack or the initializations of the system’s initializations.
In short, a huge amount of computing resources is required, and, as we have explained, we do not have them available in this master course.
Given these constraints, we have thought of a hands-on that performs a specific test, very guaranteeing, for example with a deliberately low learning-rate so that it does not explode in tests with many GPUs.
This hands-on designed allows us to show how the throughput of the system increases (images per unit of time) and to focus on the scaling metric. This means that we must completely forget about the Accuracy metric, since to make correct tests at the Accuracy level we should do a more detailed work of the tunning hyperparameters that we cannot do.
For this reason, we cannot make any comparisons in the results of this hands-on at the Accuracy level. The literature on the subject shows, that if we do a correct hyperparameter tuning, the loss of Accuracy in the scale is reasonably when using the Horovod API in supercomputers such as CTE-POWER.
In summary, we have designed a simple synthetic test where each GPU performs the same number of epochs, and therefore with 32 GPUs, we will perform 32 times more epochs than with 1 GPU. In this scenario, the measurement of time of the training process of each test can serve as a metric of the quality of scaling at the throughput level. That is, if we had a linear speedup, the time should always be the same in all tests.
Remember that we focus our hands-on on the computational speed of the process rather than the model’s accuracy, and we will only compare the time required to execute the .fitmethod with the different tests. For this purpose, we will use the following simple code to measure the time:
start = time.time()model.fit(train_ds, epochs=epochs, verbose=verbose,           callbacks=callbacks)end = time.time()if hvd.rank() == 0:   print(‘Total Time:’, round((end — start), 2), ‘(s)’)
Once we run the previous python code, in the file .output that has stored the standard output, we find the execution time required to execute the method .fit().
There are different ways to design the experiment, but a simple way seems to be to send a job to SLURM for each test: 1,2,4,8,16 and, 32 GPUs. For example, in our case, we created:
[nct010xx@p9login2]$ lsjob_hvd_1node_1gpu.shjob_hvd_1node_2gpu.shjob_hvd_1node_4gpu.shjob_hvd_multi_node_8gpus.shjob_hvd_multi_node_16gpus.shjob_hvd_multi_node_32gpus.shjob_hvd_multi_node_64gpus.sh
And each of these SLURM jobs generated a .output file like:
[nct010xx@p9login2]$ tail hvd_1_4gpus_4721140.outputEpoch 7/1098/98–41s — loss: 1.6659 — accuracy: 0.3903Epoch 8/1098/98–41s — loss: 1.6420 — accuracy: 0.3992Epoch 9/1098/98–41s — loss: 1.6224 — accuracy: 0.4067Epoch 10/1098/98–41s — loss: 1.5977 — accuracy: 0.4162Total Time: 691.1 (s)
where the last line indicates the required to execute the method .fit().
In summary, in the .output files, we have all the times that interest us. The following graph plots the time required for executing 10 epoch using up to 64 GPUs:
We can see that the more GPUs we have, despite having the same job per GPU (epochs = 10 * num GPUs), there is more overhead time to add. It should be noted that these times come from a single execution and can vary between several executions. It is evident that if we want to do a good study, we should carry out several tests and then take the averages of the times obtained in each one of them. But given the purpose of this post (and the cost of resources, which we have to save!), it is enough only one execution.
Finally, as we already mentioned in previous posts, the speedup is a popular metric in our community. The following graph plots the speedup in blue and in white what is missing to reach the optimal linear speed up.
Each column n is obtained by, first, dividing the time required to execute 10 epoch with 1 GPU by the time required when we execute the same workload with n GPU. And finally, multiplying the resulting value by n (the number of GPUs used).
We can conclude that the training really scales with regard to images per second processed, which does not mean that in terms of Accuracy it happens the same, since in this example we have ignored the hyperparameter tunning stage and therefore we cannot draw conclusions from the Accuracy we get.
If you want to learn more and consolidate the knowledge acquired, now it’s your turn to get your hands dirty to reproduce the above results for the ResNET152V2. Explain why you think these results come out and compare the results with those obtained with ResNET50V2.
In this post, we have presented how to distribute the training of a single Deep Neural Network over many servers using Horovod. We only proposed a toy experiment due to the limitations of the resources available. However, from my point of view, It serves the purpose of bringing the reader closer to the Horovod API.
This is the last post of this series that I wrote to support my classes in my master course SA-MIRI at UPC Barcelona Tech (with the support of Barcelona Supercomputing Center). I share it at Towards Data Science just in case it can be useful for other readers.
Artificial Intelligence is a Supercomputing problemUsing Supercomputers for Deep Learning TrainingScalable Deep Learning on Parallel and Distributed InfrastructuresTrain a Neural Network on multi-GPU with TensorFlowDistributed Deep Learning with Horovod
Artificial Intelligence is a Supercomputing problem
Using Supercomputers for Deep Learning Training
Scalable Deep Learning on Parallel and Distributed Infrastructures
Train a Neural Network on multi-GPU with TensorFlow
Distributed Deep Learning with Horovod
Acknowledgment: Many thanks to Juan Luis Domínguez and Oriol Aranda from BSC-CNS, who wrote the first version of the codes that appear in this series of posts, and to Carlos Tripiana and Francisco Gonzalez for the essential support with the deployment of the software stack of POWER-CTE Supercomputer. Also many thanks to Alvaro Jover Alvarez, Miquel Escobar Castells and Raul Garcia Fuentes for their contributions to the proofreading of this document.