Facing NCCL error on Multi-GPU training(on single machine) using run_glue.py script
Author: aditya-malte
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.2
- Platform: Linux-4.19.0-14-cloud-amd64-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: 4xTesla T4 (GCP)
- Using distributed or parallel set-up in script?: torch.distributed.launch

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): DistilRoberta

The problem arises when using:
* [*] the official example scripts: (give details below)
* [  ] my own modified scripts: (give details below)


The tasks I am working on is:
* [  ] an official GLUE/SQUaD task: (give the name)
* [*] my own task or dataset: (give details below)

Regression task with a single output, using BertForSequenceClassification

## To reproduce

Steps to reproduce the behavior:

1.python -m torch.distributed.launch --nproc_per_node 4 /home/run_glue.py   --train_file /home/data/train.csv   --validation_file /home/data/dev.csv   --test_file /home/data/test.csv   --model_name_or_path distilroberta-base   --output_dir /home/model   --num_train_epochs 5   --per_device_train_batch_size 1   --per_device_eval_batch_size 16   --do_train   --do_eval   --fp16   --gradient_accumulation_steps 2   --do_predict   --logging_steps 100   --evaluation_strategy steps   --save_steps 100   --overwrite_output_dir

File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 442, in init_process_group
732793de051f:1895:1925 [1] NCCL INFO transport/shm.cc:101 -> 2
732793de051f:1895:1925 [1] NCCL INFO transport.cc:30 -> 2
732793de051f:1895:1925 [1] NCCL INFO transport.cc:49 -> 2
732793de051f:1895:1925 [1] NCCL INFO init.cc:766 -> 2
732793de051f:1895:1925 [1] NCCL INFO init.cc:840 -> 2
732793de051f:1895:1925 [1] NCCL INFO group.cc:73 -> 2 [Async thread]
    barrier()
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1947, in barrier
Traceback (most recent call last):
  File "/home/run_text_classification.py", line 480, in <module>
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1603729138878/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
    main()
  File "/home/run_text_classification.py", line 163, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/opt/conda/lib/python3.7/site-packages/transformers/hf_argparser.py", line 180, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 60, in __init__
  File "/opt/conda/lib/python3.7/site-packages/transformers/training_args.py", line 478, in __post_init__
    if is_torch_available() and self.device.type != "cuda" and self.fp16:
  File "/opt/conda/lib/python3.7/site-packages/transformers/file_utils.py", line 1346, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/training_args.py", line 583, in device
    return self._setup_devices

732793de051f:1897:1927 [3] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
  File "/opt/conda/lib/python3.7/site-packages/transformers/file_utils.py", line 1336, in __get__
732793de051f:1897:1927 [3] NCCL INFO include/shm.h:41 -> 2

732793de051f:1897:1927 [3] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-b3d54cebe4167a34-0-2-3 (size 9637888)
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Expected model training to proceed smoothly using 4xGPU. When I run the said script with nproc_per_node=1(or even 2), it runs smoothly but setting it as 4 gives strange errors. 

After updating to 1.9.0 I face a different error:

RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:832, unhandled system error, NCCL version 2.7.8 ncclSystemError: System call (socket, malloc, munmap, etc) failed.
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
cc @sgugger 

This seems like a problem in your environment install for NCCL: if your script can run on two GPUs there is nothing in the code to change to make it run on four GPUs so this is not a bug in the training script or transformers. I have never seen that particular NCCL error so I'm afraid I can't really help debugging it.

Thanks for the quick reply. 
Yeah, it’s strange that it works on 2 GPUs but not on 4. Will check again and let you know.

@sgugger just to clarify:
The system has 4 GPUs. It’s only the nproc_per_node argument I’m changing (from 1 to 2,4,etc.). 
Just want to ensure I’ve not misunderstood the cause of the error. Right?

Yes I understood that. The PyTorch launcher is going to spawn a different number of processes depending on the number your pass, which in turn will use the number of GPUs specified (and the others are idle).

Thanks. Just wanted to confirm that. Will try reinstalling the environment and update if I find the solution.


Hi @sgugger,
Good news, the issue seems to have been an environment issue.
Thanks for the instant help

I still meet the same problem， could you please tell me how to solve it？

I have the same problem on colab