All evaluation processes overload one GPU, when other 7 are available. While Training process fine and is distributed across all 8 cards
Author: TatProg
Pull request: no

## Environment info
- `transformers` version: 4.6.1
- Platform: Linux-4.15.0-135-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, 8 Tesla V100
- Using distributed or parallel set-up in script?: training_args.parallel_mode = ParallelMode.NOT_DISTRIBUTED

### Who can help
@sgugger @LysandreJik
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using BERT - 'bert-base-multilingual-cased'

The problem arises when using:
* [ ] the official example scripts.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Masked Language Modelling 

## To reproduce
0. Enter 
`nvidia-smi`
`export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7`
<img width="557" alt="Снимок экрана 2021-06-24 в 18 35 25" src="https://user-images.githubusercontent.com/43710369/123291660-f9b22600-d51a-11eb-8972-0c2cbdfc1503.png">

1. Initialize training_args
```python
training_args = TrainingArguments(
    output_dir='My_lovely_mlm_model',
    overwrite_output_dir=True,
    do_train=True,
    do_eval=True,

    per_device_train_batch_size=100,
    per_device_eval_batch_size=50,

    evaluation_strategy='steps',
    logging_steps=10_000,
    eval_steps=None,
    prediction_loss_only=True,
    learning_rate=5e-5,
    weight_decay=0,
    adam_epsilon=1e-8,
    max_grad_norm=1.0,
    num_train_epochs=50,
    save_steps=10_000,
    save_total_limit=10
)
```

2. Catch RuntimeError: CUDA out of memory
``` 
RuntimeError: CUDA out of memory. 
Tried to allocate 17.81 GiB (GPU 0; 31.72 GiB total capacity; 
10.49 GiB already allocated; 3.57 GiB free; 
26.52 GiB reserved in total by PyTorch)
```

3. Change batch size and catch another Error
`per_device_train_batch_size = 80`
`per_device_eval_batch_size = 4`
```
RuntimeError: CUDA out of memory. 
Tried to allocate 14.25 GiB (GPU 0; 31.72 GiB total capacity; 
8.96 GiB already allocated; 8.80 GiB free; 
21.29 GiB reserved in total by PyTorch)
```
4. Change batch size and only then trainin was started
`per_device_train_batch_size = 64`
`per_device_eval_batch_size = 4`
<img width="560" alt="Снимок экрана 2021-06-25 в 1 24 39" src="https://user-images.githubusercontent.com/43710369/123340021-42d29c00-d554-11eb-9849-9aa682ef97e3.png">



**As you see, first GPU in list is loaded with 30.95 GB and other seven with only 10.5 GB. I have no idea how to fix this with default Transformers functions (or PyTorch lib).**



Before each run i deleted cache with `torch.cuda.empty_cache()`
Also, This might be helpful:
```python
print(training_args.n_gpu)
print(training_args.parallel_mode)
print(training_args.train_batch_size)
print(training_args.eval_batch_size)
print(training_args.device)
```
Got
```
8
ParallelMode.NOT_DISTRIBUTED
512
32
cuda:0
```

## Expected behavior

Equal distribution to all 8 GPUs. Normal training and evaluation process.



P.S. Also, continued training  from checkpoint ( `AutoModelWithLMHead.from_pretrained(path_to_checkpoint_model)`) cause exception. Should I write another issue?

Comments:
Hey, I think you are using `nn.DataParallel` and not `nn.DistributedDataParallel` and hence 1 GPU is taking more memory. In case of `torch.nn.DistributedDataParallel`, `training_args.parallel_mode` will be `ParallelModel.Distributed`.

In order to use `nn.DistributedDataParallel`, launch with this CMD: `python3 -m torch.distributed.launch --nproc_per_node=8 <your-script>.py`

@vasudevgupta7 Thank you for interesting idea. Sadly, it doesn't work well. May I ask you to inspect my error messages on pastebin? 

First one for  `python3 -m torch.distributed.launch --nproc_per_node=8 run_mlm_copy.py > log_BERTtugan_pretrained_5.txt &`
https://pastebin.com/MLkR8MQp

Second one for `python3 -m torch.distributed.launch --nproc_per_node=8 run_mlm_copy.py &`
https://pastebin.com/UeVg0YpV

### UPD:
First of all, I remove all arguments for evaluation:
```python
training_args = TrainingArguments(
    output_dir='My_lovely_mlm_model',
    overwrite_output_dir=True,
    do_train=True,
    per_device_train_batch_size=150,
    logging_steps=10_000,
    prediction_loss_only=True,
    num_train_epochs=50,
    save_steps=10_000,
    save_total_limit=10
)

```
And all worked fine, but this **not a solution**

Secondly, when I run script on all 8 GPUs, I caught an Error (error message below). But when I choose only 7 with `export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6`, everything works fine.


How can I solve these problems? 

Code in "for_github" folder in https://github.com/TatProg/bertugan_sample/tree/master/mlm/for_github

```bash
RuntimeError: Timed out initializing process group in store based barrier on rank: 1, for key: store_based_barrier_key:1 (world_size=8, worker_count=16, timeout=0:30:00)
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 5977) of binary: /usr/bin/python3
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 2/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvousing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
  restart_count=2
  master_addr=127.0.0.1
  master_port=29500
  group_rank=0
  group_world_size=1
  local_ranks=[0, 1, 2, 3, 4, 5, 6, 7]
  role_ranks=[0, 1, 2, 3, 4, 5, 6, 7]
  global_ranks=[0, 1, 2, 3, 4, 5, 6, 7]
  role_world_sizes=[8, 8, 8, 8, 8, 8, 8, 8]
  global_world_sizes=[8, 8, 8, 8, 8, 8, 8, 8]
```

Hey, not sure why that's happening. May be someone else can help.

Your first pastebin shows that the 8 processes are not properly initialized and joined, so this is more of a PyTorch error. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.