Tremendous slowdown in multi-node distributed training
Author: Genius1237
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert
Finetuning a bert-base model on language modeling for a particular domain

Language I am using the model on (English, Chinese ...): 

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Training is happening on Azure NC24s_v3 nodes (4 V100s each) with NCCL as the backend. I'm comparing the performance on a single node scenario vs a 2 node scenario. Note that there is no infiniband networking between the nodes, only 40Gbps ethernet.
2. Use torch.distributed.launch to launch `run_language_modeling.py` in single node (mult-gpu) and multi node (mult-gpu) scenarios

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
In the single node scenario, I'm getting about 2 iteration/sec during training. In the multi-node scenario, it drops to 4 sec/iteration.

Theorizing that the network was the issue, I reduced the model size significantly (down to 1 layer from 12 and the other hyperparameters also scaled down appropriately) and ran the test again. The same slowdown in performance was observed even then.

Am I missing something here? Is it possible to perform multi-node training of bert models without infiniband? 

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.1
- Platform: Ubuntu 18.04
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes


Comments:
Did you find an answer to what the cause is?

Well, I guess that you need infiniband between the nodes to train a model as large as bert. The ethernet interface seems to be a bottleneck during the gradient synchronization process.

To test this out, I tried out a much smaller bert model on the existing setup without infiniband. Less gradient information to exchange, so I was able to observe a speedup. I tried out a normal sized model on a setup with infiniband and I was observing some speedup (albeit not perfect scaling, only like a 50-60% improvement).

I concluded that not having infiniband was the issue. Maybe this could be updated in the readme where the instructions for `run_language_modeling.py` are given.

I also encounter this. And I found the `fairseq` code base scales linearly on the same ethernet hardware I have than `run_language_modeling.py`

Could you share some details on what model you trained on `fairseq` and that model's size?

Since `run_language_modeling.py` uses only 1 GPU per node in the code, could you share what changes need to be made to the file in order to work with Multi-GPU, Multi-Node settings like `Azure NC24s_v3 nodes`?

In the code in `examples/run_language_modeling.py`, `1` GPU per node is hard-coded (in the last line)


    
    if args.local_rank == -1 or args.no_cuda:
        device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
        args.n_gpu = 0 if args.no_cuda else torch.cuda.device_count()
    else:  # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
        torch.cuda.set_device(args.local_rank)
        device = torch.device("cuda", args.local_rank)
        torch.distributed.init_process_group(backend="nccl")
        args.n_gpu = 1


You do not need any modifications. You will have to launch the script via `torch.distributed.launch`. It'll be something like this
```
python -m torch.distributed.launch --nproc_per_node 4 --nnodes $NODE_COUNT --node_rank $RANK --master_addr $MASTER_ADDR run_lm_finetuning.py ....
```
`$NODE_COUNT` will be your number of nodes. You'll have to find a way to obtain `$RANK` and `$MASTER_ADDR` depending on your cluster configuration.

Since when you run via DistributedDataParallel, one process has only one gpu associated with it, that line does something to ensure that.

Is there any update on this? I suppose that not having infiniband interconnect was the only limiting factor? 

Yes. Ran faster on a system with infiniband.

On Thu, Nov 19, 2020 at 6:08 PM gvijqb <notifications@github.com> wrote:

> Is there any update on this? I suppose that not having infiniband
> interconnect was the only limiting factor?
>
> —
> You are receiving this because you modified the open/close state.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/3274#issuecomment-730347349>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADZB3Y3PKZBAOFOP5SSA3BLSQUGUXANCNFSM4LJHN2QA>
> .
>


-- 
Regards
Anirudh Srinivasan
Research Fellow
Microsoft Research, India