Distributed training on multiple GPU nodes is slower than on single GPU node
Author: YingleiZhang
Pull request: no

Our team uses pre-trained model and transformers has been a great help. 

While trying to benchmark the training speed of our computing infrastructure, we were running this example:
https://github.com/huggingface/transformers/blob/v2.3.0/examples/run_lm_finetuning.py

We found that on a single p3.16xlarge GPU instance, DDP would took about 36:33 to train wikitext-103-raw. 
However, once we move on to two p3.16xlarge GPU instance, it would take more time (1:45:49) for the same dataset. 

I would like to know what could possibly cause this? 

There are something i am suspecting:
1. The synchronization of gradiants/parameters between different process
2. The optimizer that has been used in this scripts. 

Any help is appreciated. Thanks! 

Comments:
Are those `p3.16xlarge` is the same AZ? Even in the same AZ, the throughput and latency between machines might be the bottleneck here.

i.e. you usually need infiniband between cluster machines. But @mfuntowicz knows about this stuff way better than I do...

Thanks for your reply. The two hosts are indeed in the same AZ. 

Here is how i run them:
```
# Node 1:
python3.6 -m torch.distributed.launch \
--nproc_per_node=8 \
--nnodes=2 \
--node_rank=0 \
--master_addr="10.0.0.83" \
--master_port=12345 \
run_lm_finetuning.py \
    --output_dir=output \
    --model_type=roberta \
    --model_name_or_path=roberta-base \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm

# Node 2       
python3.6 -m torch.distributed.launch \
--nproc_per_node=8 \
--nnodes=2 \
--node_rank=1 \
--master_addr="10.0.0.83" \
--master_port=12345 \
run_lm_finetuning.py \
    --output_dir=output \
    --model_type=roberta \
    --model_name_or_path=roberta-base \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm
```
I also have the nccl debug info here:
The first part is on multiple nodes, where the training is slow. The second part is on single node, and the training is fast. I can definitely see that on single node, there are many Channels, which can't be found on multiple node. 
https://gist.github.com/YingleiZhang/a8df48eb534ba20ff8f26b5309094b55

I was also suspecting that i might need high speed connections (link infiniband) between cluster machines, but in this case, would MPI help? My PyTorch did not built with MPI yet. 

I think the bandwidth between two different nodes are indeed the problem. Consider the amount of data (Our estimation is about 8G for each step) we need to move between different nodes, and the bandwidth for intra-gpu communication is much higher than that of inter-node communication. NCCL folks mentioned that this could be 120 GB/s vs 10 GB/s for all reduce operation. (See it here https://github.com/NVIDIA/nccl/issues/318)

I am closing this issue here. Thanks for the help. 

Thanks for investigating.

Also from your logs @mfuntowicz was saying that it looks like NCCL does not use [AWS EFA](https://aws.amazon.com/hpc/efa/) – maybe something to investigate there.