GPU memory usage
Author: 008karan
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I am training albert from scratch. I am using 8 V100. Issue is gpu 0 is almost completely used but others have around 50% ram unused. I am getting only 85 batch size on this system  and above this OOM. 
Using transformers from source 2.10.0
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:16.0 Off |                    0 |
| N/A   77C    P0   291W / 300W |  30931MiB / 32510MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:00:17.0 Off |                    0 |
| N/A   71C    P0   255W / 300W |  18963MiB / 32510MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:00:18.0 Off |                    0 |
| N/A   71C    P0    95W / 300W |  18963MiB / 32510MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000000:00:19.0 Off |                    0 |
| N/A   68C    P0    89W / 300W |  18963MiB / 32510MiB |     72%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  On   | 00000000:00:1A.0 Off |                    0 |
| N/A   68C    P0    78W / 300W |  18963MiB / 32510MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  On   | 00000000:00:1B.0 Off |                    0 |
| N/A   69C    P0    96W / 300W |  18963MiB / 32510MiB |     65%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  On   | 00000000:00:1C.0 Off |                    0 |
| N/A   69C    P0    79W / 300W |  18963MiB / 32510MiB |     95%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  On   | 00000000:00:1D.0 Off |                    0 |
| N/A   74C    P0    80W / 300W |  18963MiB / 32510MiB |     12%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     28066      C   python                                     30917MiB |
|    1     28066      C   python                                     18949MiB |
|    2     28066      C   python                                     18949MiB |
|    3     28066      C   python                                     18949MiB |
|    4     28066      C   python                                     18949MiB |
|    5     28066      C   python                                     18949MiB |
|    6     28066      C   python                                     18949MiB |
|    7     28066      C   python                                     18949MiB |
+-----------------------------------------------------------------------------+
 ```

Comments:
How do you launch your training? Can you paste your command?

I am using the trainer.
```

training_args = TrainingArguments(
    output_dir="albert_model",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=85,
    learning_rate=5e-5,
    save_steps=50,
    save_total_limit=20,
    do_train=True,
    logging_steps=100
   )

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
)

trainer.train()
```

But how do you launch the actual script? 

To efficiently harness your 8 V100s you should probably use torch.distributed, not nn.DataParallel.

So you would need to launch your script with e.g. 
```
python -m torch.distributed.launch \
    --nproc_per_node 8 your_script.py
```



Just tried this. this is loading pretraining data in each GPU independently. Shouldn't data be read once?

It should, yes.

 Previously this was not the case and all transformer models utilize all GPUs automatically.
.
getting this : reading for each gpu using ` python -m torch.distributed.launch  --nproc_per_node 8 test_lm.py 
`
```*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated
Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated
/language_model/lm/lib/python3.6/site-packages/transformers/tokenization_utils.py:830: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  category=FutureWarning,
/language_model/lm/lib/python3.6/site-packages/transformers/tokenization_utils.py:830: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  category=FutureWarning,
Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated
/language_model/lm/lib/python3.6/site-packages/transformers/tokenization_utils.py:830: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  category=FutureWarning,
Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated
/language_model/lm/lib/python3.6/site-packages/transformers/tokenization_utils.py:830: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  category=FutureWarning,
Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated
/language_model/lm/lib/python3.6/site-packages/transformers/tokenization_utils.py:830: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  category=FutureWarning,
Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated
/language_model/lm/lib/python3.6/site-packages/transformers/tokenization_utils.py:830: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  category=FutureWarning,
Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated
/language_model/lm/lib/python3.6/site-packages/transformers/tokenization_utils.py:830: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  category=FutureWarning,
Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated
/language_model/lm/lib/python3.6/site-packages/transformers/tokenization_utils.py:830: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  category=FutureWarning,
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I face a similar problem. I wonder if this problem is ever resolved? 

I try to fine-tune a bert model and found the gpu memory usage behaves exactly like what the original post described "gpu 0 is almost completely used but others have around 50% ram unused"

```
python run_mlm.py \
--model_name_or_path bert-base-uncased \
--dataset_name wikitext \
--dataset_config_name wikitext-2-raw-v1 \
--per_device_train_batch_size 10 \
--max_seq_length=256 \
--do_train \
--output_dir /tmp/test \
```

If I run the example training script above, Here's my gpu usage:
![image](https://user-images.githubusercontent.com/43428393/99751851-f09a3b80-2a97-11eb-90d0-206f9063a895.png)

After some research, I found people has the same issue when training different model using pytorch. like [this](https://forums.fast.ai/t/training-language-model-with-nn-dataparallel-has-unbalanced-gpu-memory-usage/42494)

Then I found this [post ](https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255) written by Thomas Wolf suggesting using the [PyTorch-Encoding](https://github.com/zhanghang1989/PyTorch-Encoding) library. However, this post is 2 years old. So I was wondering if there's a more recent solution for it.

@thomwolf  @sgugger 

Thanks!