text: 
  
  TITLE
  Excessive GPU-GPU communication with GPT2 making multi-GPU training slow?
  
  COMMENTS
  26
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 5
  
  BODY
  Summary: on a multi-GPU system, training GPT2 seems to scale poorly unless a very fast GPU-GPU interconnect like NVLink is available. In particular, without NVLink using two GPUs is *slower* than using just one GPU.
  
  ## Environment info
  
  - `transformers` version: 4.1.1
  - Platform: Linux-5.8.0-rc7-custom-x86_64-with-glibc2.29
  - Python version: 3.8.5
  - PyTorch version (GPU?): 1.8.0.dev20201214+cu110 (True)
  - Tensorflow version (GPU?): not installed (NA)
  - Using GPU in script?: Yes
  - Using distributed or parallel set-up in script?: No?
  - Hardware: 2 x NVIDIA RTX 3090 w/NVLink
  
  ### Who can help
  
  Maybe @LysandreJik or @patrickvonplaten ?
  
  ## Information
  
  Model I am using (Bert, XLNet ...): GPT2
  
  The problem arises when using:
  * [ ] the official example scripts: (give details below)
  * [X] my own modified scripts: (give details below)
  
  The script is a pretty basic example of training a medium-size GPT2 from scratch. The script is here: https://panda.moyix.net/~moyix/train_csrc.py
  
  The dataset and tokenized vocab:
  * Dataset: https://panda.moyix.net/~moyix/plainsrc_all.txt.gz (718M, gzipped)
  * Vocab: https://panda.moyix.net/~moyix/csrc_vocab.tar.gz
  
  The tasks I am working on is:
  * [ ] an official GLUE/SQUaD task: (give the name)
  * [X] my own task or dataset: (give details below)
  
  Training a GPT2 language model on C source code.
  
  ## To reproduce
  
  Run with only one GPU: `CUDA_VISIBLE_DEVICES=0 python train_csrc.py`
  
  Run with two GPUs, NVLink disabled: `NCCL_P2P_DISABLE=1 python train_csrc.py`
  
  Run with two GPUs and NVLink enabled: `python train_csrc.py`
  
  Here is some benchmarking I did with my dataset on transformers 3.3.1 and 4.1.1 (note the difference in ETA is just because 3.3.1 only seems to report the ETA for the current epoch):
  
  Version|NVLINK|GPUs|ETA|Perf
  --------|--------|-----|-----|-----
  4.1.1 |  Yes  |  2GPU |  419:52:28 |  1.94it/s
  4.1.1 |  No   |  2GPU | 1025:06:27 |  1.26s/it
  4.1.1 |  N/A  |  1GPU |  599:14:57 |  2.72it/s
  3.3.1 |  Yes  |  2GPU |   83:46:51 |  1.94it/s
  3.3.1 |  No   |  2GPU |  204:54:22 |  1.26s/it
  3.3.1 |  N/A  |  1GPU |  119:02:34 |  2.73it/s
  
  You can see that using two GPUs is actually slower than using a single GPU, unless NVLink is available (599 hours for 1 GPU vs 1025 hours for two GPUs). So presumably there is a large amount of GPU-GPU communication going on?
  
  ## Expected behavior
  
  Scaling should be roughly linear with the number of GPUs. Unfortunately I am not very familiar with the implementation details of GPT2 in Huggingface, but others report roughly linear scaling with Transformer models like BERT so it should work here as well: https://towardsdatascience.com/training-bert-at-a-university-eedcf940c754
  
  Although I have  a system with NVLink at home, this issue is still affecting me because I would like to be able to run this on the university HPC cluster, where most nodes do not have NVLink.