DDP training hangs with `run_glue.py` and `run_seq2seq.py`
Author: Shivanshu-Gupta
Pull request: no

## Environment info

- `transformers` version: 4.16.2
- Platform: Linux-5.4.0-80-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyTorch version (GPU?): 1.10.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
@sgugger @patrickvonplaten @stas00 

## Information

Model I am using (Bert, XLNet ...): Bert for `run_glue.py`, UnifiedQA for `run_seq2seq_qa.py`

The problem arises when using:
* [x] the official example scripts:
  [run_glue.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue.py), [run_seq2seq_qa.py](https://github.com/huggingface/transformers/commits/master/examples/pytorch/question-answering/run_seq2seq_qa.py) (I may have fixed some bugs in the latter).
* [ ] my own modified scripts: 

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: GLUE, SQUAD
* [ ] my own task or dataset: 

## To reproduce

The issue is that training proceeds as expected on a single GPU but when run on multiple GPU, it simply hangs. 

Trying to debug using `py-spy` and `faulthandler`, the first issue I observed was in blocks using `main_process_first` context manager introduced by [#12367](https://github.com/huggingface/transformers/pull/12367). Eg. [run_glue.py:L406](https://github.com/huggingface/transformers/blob/db7d6a80e82d66127b2a44b6e3382969fdc8b207/examples/pytorch/text-classification/run_glue.py#L406). 

Removing this context manager allowed the script to proceed but now it gets stuck at [torch.nn.parallel.distributed:578](https://github.com/pytorch/pytorch/blob/302ee7bfb604ebef384602c56e3853efed262030/torch/nn/parallel/distributed.py#L578) in `DistributedDataParallel`'s `__init__()`. 

1. DDP on GLUE:

Command:
```
CUDA_VISIBLE_DEVICES=1,2 python -m torch.distributed.launch \
    --nproc_per_node 2 run_glue.py \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --task_name mnli \
    --do_train \
    --do_eval \
    --max_seq_length 128 \
    --per_device_train_batch_size 8 \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 --report_to none \
    --output_dir mnli_output/ --log_level debug --log_level_replica warning
```
Trace from faulthandler: 
```
Thread 0x00007f592d2d7340 (most recent call first):
  File "/home/shivag5/miniconda3/envs/transformers/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 578 in __init__
  File "/home/shivag5/miniconda3/envs/transformers/lib/python3.9/site-packages/transformers/trainer.py", line 1051 in _wrap_model
  File "/home/shivag5/miniconda3/envs/transformers/lib/python3.9/site-packages/transformers/trainer.py", line 1223 in train
  File "/home/shivag5/Documents/research/qa/unifiedqa/pytorch/run_glue.py", line 493 in main
  File "/home/shivag5/Documents/research/qa/unifiedqa/pytorch/run_glue.py", line 575 in <module>
```

2. DDP on Squad:

Command:
```
CUDA_VISIBLE_DEVICES=1,2 python -m torch.distributed.launch \
  --nproc_per_node 2 run_seq2seq_qa.py \
  --model_name_or_path t5-small \
  --dataset_name squad_v2 \
  --context_column context \
  --question_column question \
  --answer_column answers \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --report_to none \
  --output_dir runs/squad-multigpu-test \
  --overwrite_output_dir
  --log_level debug --log_level_replica warning
```
Similar trace.

<details><summary>After a while it times out and ends with:</summary>

```
[E ProcessGroupNCCL.cpp:587] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1802784 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:587] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1802688 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:341] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1802784 milliseconds before timing out.
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1323928 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 1 (pid: 1323929) of binary: /home/shivag5/miniconda3/envs/transformers/bin/python
Traceback (most recent call last):
  File "/home/shivag5/miniconda3/envs/transformers/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/shivag5/miniconda3/envs/transformers/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/shivag5/miniconda3/envs/transformers/lib/python3.9/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/home/shivag5/miniconda3/envs/transformers/lib/python3.9/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/home/shivag5/miniconda3/envs/transformers/lib/python3.9/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/home/shivag5/miniconda3/envs/transformers/lib/python3.9/site-packages/torch/distributed/run.py", line 710, in run
    elastic_launch(
  File "/home/shivag5/miniconda3/envs/transformers/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/shivag5/miniconda3/envs/transformers/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
========================================================
run_glue.py FAILED
--------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
--------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-02-10_19:37:42
  rank      : 1 (local_rank: 1)
  exitcode  : -6 (pid: 1323929)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 1323929
========================================================
```
</details>

I have also tried using running without `torch.distributed.launch` as e.g. `CUDA_VISIBLE_DEVICES=1,2 python run_seq2seq_qa.py [args]`. This produces the error from #14128 and also gets stuck right at the start of training (at a different point presumably as it does manage to produce the progress bar unlike DDP).

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The training with multiple GPUs hangs even with example scripts run with the suggested commands.

Comments:
I'm testing with transformers@master.

You're using `CUDA_VISIBLE_DEVICES=1,2` - is it intentionally because you don't want to use the gpu 0? I trust that you wanted to use gpus 1 and 2 and skip gpu 0.

If you don't want to select specific gpus you don't need to use `CUDA_VISIBLE_DEVICES`. See: https://huggingface.co/docs/transformers/master/en/main_classes/trainer#specific-gpus-selection

Now to your report:

1. the first one works just fine for me - no hanging - try `transformers@master` version?

I did:
```
CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node 2 \
examples/pytorch/text-classification/run_glue.py --model_name_or_path \
bert-large-uncased-whole-word-masking --task_name mnli --do_train --do_eval \
--max_seq_length 128 --per_device_train_batch_size 2 --learning_rate 2e-5 \
--num_train_epochs 3.0 --report_to none --output_dir mnli_output/ --log_level \
debug --log_level_replica warning
```


2. the 2nd one doesn't work even with 1 gpu, it fails:

```
CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node 1 \
examples/pytorch/question-answering/run_seq2seq_qa.py --model_name_or_path \
t5-small --dataset_name squad_v2 --context_column context --question_column \
question --answer_column answers --do_train --do_eval \
--per_device_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 2 \
--max_seq_length 384 --doc_stride 128 --report_to none --output_dir \
runs/squad-multigpu-test --overwrite_output_dir --log_level debug \
--log_level_replica warning
[...]
  File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 678, in <module>
    main()
  File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 516, in main
    eval_dataset = eval_examples.map(
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2102, in map
    return self._map_single(
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 518, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 485, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/datasets/fingerprint.py", line 413, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2498, in _map_single
    writer.write_batch(batch)
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/datasets/arrow_writer.py", line 498, in write_batch
    pa_table = pa.Table.from_arrays(arrays, schema=schema)
  File "pyarrow/table.pxi", line 1613, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 1232, in pyarrow.lib.Table.validate
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 4 named labels expected length 1007 but got length 1000
```

I think a similar issue is addressed here: https://github.com/huggingface/datasets/issues/1817 

As I'm not familiar with this script I will let its creator address this second issue. Then we can check the hanging issue once it can be run. @karthikrangasai, could you please have a look at why this script breaks with `transformers@master`? Thank you!

The relevant part of my setup:
```
datasets                          1.18.3
transformers                      4.17.0.dev0  
arrow                             1.1.1
pyarrow                           5.0.0
```



> You're using CUDA_VISIBLE_DEVICES=1,2 - is it intentionally because you don't want to use the gpu 0? I trust that you wanted to use gpus 1 and 2 and skip gpu 0.

Yeah, I didn't want to use GPU 0 as it was being used by another job.

</br>

> the first one works just fine for me - no hanging - try transformers@master version?

I haven't tried the master version - tried 4.16.2 and I think one of 4.15's. I'll try the master too. Did it work for you even with the `main_process_first` context manager?

</br>

> the 2nd one doesn't work even with 1 gpu, it fails:

Yeah, to get past this (bug?) I commented-out lines [456-457](https://github.com/huggingface/transformers/blob/db7d6a80e82d66127b2a44b6e3382969fdc8b207/examples/pytorch/question-answering/run_seq2seq_qa.py#L456) and lines [465-474](https://github.com/huggingface/transformers/blob/db7d6a80e82d66127b2a44b6e3382969fdc8b207/examples/pytorch/question-answering/run_seq2seq_qa.py#L465).

> > You're using CUDA_VISIBLE_DEVICES=1,2 - is it intentionally because you don't want to use the gpu 0? I trust that you wanted to use gpus 1 and 2 and skip gpu 0.
> 
> Yeah, I didn't want to use GPU 0 as it was being used by another job.

Thank you for confirming that

> > the first one works just fine for me - no hanging - try transformers@master version?
> 
> I haven't tried the master version - tried 4.16.2 and I think one of 4.15's. I'll try the master too. Did it work for you even with the `main_process_first` context manager?

Why won't it? It's just a set of `barrier` calls that syncs several processes.

I wonder - is it somehow possible that your gpus can't talk to each other? So any time you run into a `barrier` things fail?

Please try this diagnostics tool:
https://github.com/stas00/toolbox/blob/master/pytorch/torch-distributed-gpu-test.py

> > the 2nd one doesn't work even with 1 gpu, it fails:
> 
> Yeah, to get past this (bug?) I commented-out lines [456-457](https://github.com/huggingface/transformers/blob/db7d6a80e82d66127b2a44b6e3382969fdc8b207/examples/pytorch/question-answering/run_seq2seq_qa.py#L456) and lines [465-474](https://github.com/huggingface/transformers/blob/db7d6a80e82d66127b2a44b6e3382969fdc8b207/examples/pytorch/question-answering/run_seq2seq_qa.py#L465).

Oh, cool. I followed your instructions and there is no hanging. 

Since you have 2 unrelated programs hang, I'd say you have a hardware issue there.

Thanks for sharing the diagnostics script. I ran it on two nodes on our cluster. First, on the one that I have been using for  all the above experiments, I ran it like this:

```
CUDA_VISIBLE_DEVICES=4,5,6,7 python -m torch.distributed.run --nproc_per_node 4 --nnodes 1 torch-distributed-gpu-test.py
```

It got stuck at the first barrier (L65) as indicated by `faulthandler`. Since I couldn't remove the CUDA_VISIBLE_DEVICES setting on this node (some gpus were being used), to ablate, I ran the script on another node with all gpus idle - same command minus the env var. This time it hung at the second barrier (L77).

This does indicate communication issues b/w the GPUs but I also know people in my group who have been running distributed training on these nodes, albeit without Huggingface Trainer. Lemme check with them and the sysads to see if there is actually an issue. Thanks for the help till now!

This is very interesting. 

Were those people in your group running pytorch or tensorflow and if pytorch which version (specifically if they were using a different version than you).

I'd say your next call is debugging NCCL, so you can set 
```
NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node=4
```

and see if you catch any diagnostics in there. You may have some information that you need in those logs.

also `NCCL_DEBUG_SUBSYS=ALL`, see: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-debug-subsys

If the NCCL debug won't give you enough hints to understand the problem, the best next plan of action is to file an Issue at 
https://github.com/pytorch/pytorch as this Issue is clearly not related to the HF Trainer or Transformers.  Use my script in your report and not the example code as the devs would want a simple reproducible code.

Once you find a resolution please do share it with us. I have only encountered this kind of issues when some nodes on HPC were getting stuck and needed rebooting, so this looks as potentially new issue that I haven't encountered yet.

I hope it's something very simple, like an odd firewall rule or something. Perhaps https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-socket-ifname IF "eth" sometimes doesn't work due to firewall rules, then you can try to set this env var: `NCCL_SOCKET_IFNAME=lo` (`ifconfig` to see what interfaces are visible).




Thanks, @stas00 for the suggestions. I ran the script with the envvars you suggested. Here are the output logs:

```NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 8 --nnodes 1 torch-distributed-gpu-test.py```: [log0.txt](https://github.com/huggingface/transformers/files/8055103/log0.txt)

```NCCL_DEBUG_SUBSYS=NET NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 8 --nnodes 1 torch-distributed-gpu-test.py```: [log1.txt](https://github.com/huggingface/transformers/files/8055099/log1.txt)

```NCCL_DEBUG_SUBSYS=ALL NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 8 --nnodes 1 torch-distributed-gpu-test.py```: [log2.txt](https://github.com/huggingface/transformers/files/8055097/log2.txt) 

Since I can't interpret much of it do you think these suggest communication issues between the GPUs?  I did notice the messages like `Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)` but then it seems to be able to create all communication channels eventually..

It's pretty clear from the logs that "Could not enable P2P between dev A and dev B" is your problem - I don't have this if I try it on my setup - I'm not sure how you derived that:

>  but then it seems to be able to create all communication channels eventually.

which part of the log points to that success? `Init COMPLETE` refers to the set up of each "process" but doesn't say anything about its ability to inter-connect.

Googling it it appears to be a very rarely reported issue. I see some older pytorch report and one report had success with switching to cuda-11 - I'm not sure if yours is 10 or 11. But I'm pretty sure this is not cuda related. Most likely it's some kind of firewall rule that causes this issue.

Also are you using the gpus directly or via some kind of a virtual machine or docker?

In your first log file one rank reports OK, in the 3rd one - 2 ranks. which means that some processes can connect to all, but not all - and since some succeeded that means all can send but some cannot receive. So again potentially this is some firewall receive deny rule may be? But also it appears to be inconsistent - and not the same ranks succeed in different runs.

What happens if you try to use the loopback device `lo` instead of the `eno0` as I suggested above.

```
NCCL_SOCKET_IFNAME=lo NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 8 --nnodes 1 torch-distributed-gpu-test.py
```

I assume you have it.

Do you use a firewall? and if so check its logs - could be in `/var/log/syslog`, or for ufw `/var/log/ufw.log`. I'd check  `/var/log/syslog` anyway for any error reports.

I'd also try subsets instead of all 8 ranks and perhaps you will see which ranks have an issue? e.g. does `--nproc_per_node 2` succeed?


Thanks for the clarification. I didn't fully understand the logs and was hoping you'd have a better idea. I thought what I did because of the `Connected all trees` and `Connected all rings` messages and because there were multiple `Could not enable P2P` messages for every pair of devices. So the hypothesis was that maybe it does multiple attempts to create the p2p connections and printing an output on each failure but eventually succeeds. 

Here are the logs for the other settings you asked for (not much difference):

<details><summary>

`NCCL_SOCKET_IFNAME=lo NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 8 --nnodes 1 torch-distributed-gpu-test.py`
</summary>

```
WARNING:__main__:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
ava-s0:537553:537553 [0] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537553:537553 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537553:537553 [0] NCCL INFO NET/IB : No device found.
ava-s0:537553:537553 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537553:537553 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda11.3
ava-s0:537559:537559 [6] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537559:537559 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537559:537559 [6] NCCL INFO NET/IB : No device found.
ava-s0:537559:537559 [6] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537559:537559 [6] NCCL INFO Using network Socket
ava-s0:537556:537556 [3] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537556:537556 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537556:537556 [3] NCCL INFO NET/IB : No device found.
ava-s0:537556:537556 [3] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537556:537556 [3] NCCL INFO Using network Socket
ava-s0:537557:537557 [4] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537557:537557 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537557:537557 [4] NCCL INFO NET/IB : No device found.
ava-s0:537557:537557 [4] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537557:537557 [4] NCCL INFO Using network Socket
ava-s0:537555:537555 [2] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537555:537555 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537555:537555 [2] NCCL INFO NET/IB : No device found.
ava-s0:537555:537555 [2] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537555:537555 [2] NCCL INFO Using network Socket
ava-s0:537554:537554 [1] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537554:537554 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537554:537554 [1] NCCL INFO NET/IB : No device found.
ava-s0:537554:537554 [1] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537554:537554 [1] NCCL INFO Using network Socket
ava-s0:537560:537560 [7] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537560:537560 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537560:537560 [7] NCCL INFO NET/IB : No device found.
ava-s0:537560:537560 [7] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537560:537560 [7] NCCL INFO Using network Socket
ava-s0:537558:537558 [5] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:537558:537558 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:537558:537558 [5] NCCL INFO NET/IB : No device found.
ava-s0:537558:537558 [5] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:537558:537558 [5] NCCL INFO Using network Socket
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537558:537635 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4
ava-s0:537558:537635 [5] NCCL INFO Setting affinity for GPU 5 to ffff0000,ffff0000
ava-s0:537559:537623 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5
ava-s0:537559:537623 [6] NCCL INFO Setting affinity for GPU 6 to ffff0000,ffff0000
ava-s0:537560:537628 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6
ava-s0:537560:537628 [7] NCCL INFO Setting affinity for GPU 7 to ffff0000,ffff0000
ava-s0:537553:537622 [0] NCCL INFO Channel 00/02 :    0   2   3   4   5   7   6   1
ava-s0:537554:537627 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0
ava-s0:537555:537626 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1
ava-s0:537553:537622 [0] NCCL INFO Channel 01/02 :    0   2   3   4   5   7   6   1
ava-s0:537554:537627 [1] NCCL INFO Setting affinity for GPU 1 to ffff,0000ffff
ava-s0:537556:537624 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2
ava-s0:537555:537626 [2] NCCL INFO Setting affinity for GPU 2 to ffff,0000ffff
ava-s0:537557:537625 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3
ava-s0:537553:537622 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
ava-s0:537556:537624 [3] NCCL INFO Setting affinity for GPU 3 to ffff,0000ffff
ava-s0:537557:537625 [4] NCCL INFO Setting affinity for GPU 4 to ffff0000,ffff0000
ava-s0:537553:537622 [0] NCCL INFO Setting affinity for GPU 0 to ffff,0000ffff
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537559:537623 [6] NCCL INFO Channel 00 : 6[b1000] -> 1[1b000] via direct shared memory
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537553:537622 [0] NCCL INFO Channel 00 : 0[1a000] -> 2[3d000] via P2P/IPC
ava-s0:537555:537626 [2] NCCL INFO Channel 00 : 2[3d000] -> 3[3e000] via direct shared memory
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537559:537623 [6] NCCL INFO Channel 01 : 6[b1000] -> 1[1b000] via direct shared memory
ava-s0:537553:537622 [0] NCCL INFO Channel 01 : 0[1a000] -> 2[3d000] via P2P/IPC
ava-s0:537555:537626 [2] NCCL INFO Channel 01 : 2[3d000] -> 3[3e000] via direct shared memory
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537557:537625 [4] NCCL INFO Channel 00 : 4[88000] -> 5[89000] via direct shared memory
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537557:537625 [4] NCCL INFO Channel 01 : 4[88000] -> 5[89000] via direct shared memory
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537558:537635 [5] NCCL INFO Channel 00 : 5[89000] -> 7[b2000] via P2P/IPC
ava-s0:537554:537627 [1] NCCL INFO Channel 00 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537558:537635 [5] NCCL INFO Channel 01 : 5[89000] -> 7[b2000] via P2P/IPC
ava-s0:537554:537627 [1] NCCL INFO Channel 01 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:537556:537624 [3] NCCL INFO Channel 00 : 3[3e000] -> 4[88000] via direct shared memory
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537556:537624 [3] NCCL INFO Channel 01 : 3[3e000] -> 4[88000] via direct shared memory
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537558:537635 [5] NCCL INFO Connected all rings
ava-s0:537560:537628 [7] NCCL INFO Channel 00 : 7[b2000] -> 6[b1000] via direct shared memory
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537558:537635 [5] NCCL INFO Channel 00 : 5[89000] -> 6[b1000] via direct shared memory
ava-s0:537560:537628 [7] NCCL INFO Channel 01 : 7[b2000] -> 6[b1000] via direct shared memory
ava-s0:537558:537635 [5] NCCL INFO Channel 01 : 5[89000] -> 6[b1000] via direct shared memory
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537557:537625 [4] NCCL INFO Connected all rings
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537555:537626 [2] NCCL INFO Connected all rings
ava-s0:537559:537623 [6] NCCL INFO Connected all rings
ava-s0:537557:537625 [4] NCCL INFO Could not enable P2P between dev 4(=88000) and dev 5(=89000)
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537560:537628 [7] NCCL INFO Connected all rings
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537560:537628 [7] NCCL INFO Could not enable P2P between dev 7(=b2000) and dev 6(=b1000)
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537557:537625 [4] NCCL INFO Channel 00 : 4[88000] -> 3[3e000] via direct shared memory
ava-s0:537559:537623 [6] NCCL INFO Channel 00 : 6[b1000] -> 7[b2000] via direct shared memory
ava-s0:537559:537623 [6] NCCL INFO Could not enable P2P between dev 6(=b1000) and dev 7(=b2000)
ava-s0:537554:537627 [1] NCCL INFO Connected all rings
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537557:537625 [4] NCCL INFO Channel 01 : 4[88000] -> 3[3e000] via direct shared memory
ava-s0:537556:537624 [3] NCCL INFO Connected all rings
ava-s0:537559:537623 [6] NCCL INFO Channel 01 : 6[b1000] -> 7[b2000] via direct shared memory
ava-s0:537560:537628 [7] NCCL INFO Connected all trees
ava-s0:537560:537628 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537560:537628 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537553:537622 [0] NCCL INFO Connected all rings
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537559:537623 [6] NCCL INFO Channel 00 : 6[b1000] -> 5[89000] via direct shared memory
ava-s0:537553:537622 [0] NCCL INFO Channel 00 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:537553:537622 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:537559:537623 [6] NCCL INFO Channel 01 : 6[b1000] -> 5[89000] via direct shared memory
ava-s0:537553:537622 [0] NCCL INFO Channel 01 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:537554:537627 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537558:537635 [5] NCCL INFO Channel 00 : 5[89000] -> 4[88000] via direct shared memory
ava-s0:537558:537635 [5] NCCL INFO Could not enable P2P between dev 5(=89000) and dev 4(=88000)
ava-s0:537558:537635 [5] NCCL INFO Channel 01 : 5[89000] -> 4[88000] via direct shared memory
ava-s0:537558:537635 [5] NCCL INFO Connected all trees
ava-s0:537558:537635 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537558:537635 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537559:537623 [6] NCCL INFO Connected all trees
ava-s0:537559:537623 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537559:537623 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537556:537624 [3] NCCL INFO Channel 00 : 3[3e000] -> 2[3d000] via direct shared memory
ava-s0:537556:537624 [3] NCCL INFO Could not enable P2P between dev 3(=3e000) and dev 2(=3d000)
ava-s0:537556:537624 [3] NCCL INFO Channel 01 : 3[3e000] -> 2[3d000] via direct shared memory
ava-s0:537554:537627 [1] NCCL INFO Channel 00 : 1[1b000] -> 2[3d000] via direct shared memory
ava-s0:537554:537627 [1] NCCL INFO Channel 01 : 1[1b000] -> 2[3d000] via direct shared memory
ava-s0:537557:537625 [4] NCCL INFO Connected all trees
ava-s0:537557:537625 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537557:537625 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537553:537622 [0] NCCL INFO Connected all trees
ava-s0:537553:537622 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537553:537622 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537555:537626 [2] NCCL INFO Could not enable P2P between dev 2(=3d000) and dev 3(=3e000)
ava-s0:537555:537626 [2] NCCL INFO Channel 00 : 2[3d000] -> 1[1b000] via direct shared memory
ava-s0:537555:537626 [2] NCCL INFO Channel 01 : 2[3d000] -> 1[1b000] via direct shared memory
ava-s0:537556:537624 [3] NCCL INFO Connected all trees
ava-s0:537556:537624 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537556:537624 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537554:537627 [1] NCCL INFO Connected all trees
ava-s0:537554:537627 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537554:537627 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537555:537626 [2] NCCL INFO Connected all trees
ava-s0:537555:537626 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
ava-s0:537555:537626 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:537553:537622 [0] NCCL INFO comm 0x7f3450002fb0 rank 0 nranks 8 cudaDev 0 busId 1a000 - Init COMPLETE
ava-s0:537557:537625 [4] NCCL INFO comm 0x7f2bc0002fb0 rank 4 nranks 8 cudaDev 4 busId 88000 - Init COMPLETE
ava-s0:537559:537623 [6] NCCL INFO comm 0x7fbe4c002fb0 rank 6 nranks 8 cudaDev 6 busId b1000 - Init COMPLETE
ava-s0:537553:537553 [0] NCCL INFO Launch mode Parallel
ava-s0:537556:537624 [3] NCCL INFO comm 0x7f75d0002fb0 rank 3 nranks 8 cudaDev 3 busId 3e000 - Init COMPLETE
ava-s0:537554:537627 [1] NCCL INFO comm 0x7f40ec002fb0 rank 1 nranks 8 cudaDev 1 busId 1b000 - Init COMPLETE
ava-s0:537558:537635 [5] NCCL INFO comm 0x7f5688002fb0 rank 5 nranks 8 cudaDev 5 busId 89000 - Init COMPLETE
ava-s0:537555:537626 [2] NCCL INFO comm 0x7faf08002fb0 rank 2 nranks 8 cudaDev 2 busId 3d000 - Init COMPLETE
ava-s0:537560:537628 [7] NCCL INFO comm 0x7f9f28002fb0 rank 7 nranks 8 cudaDev 7 busId b2000 - Init COMPLETE
[ava-s0.ics.uci.edu-0] is OK (global rank: 0/8)
```
</details>

<details><summary>

`NCCL_SOCKET_IFNAME=lo NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 2 --nnodes 1 torch-distributed-gpu-test.py`

</summary>

```
WARNING:__main__:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
ava-s0:538447:538447 [0] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:538447:538447 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:538447:538447 [0] NCCL INFO NET/IB : No device found.
ava-s0:538447:538447 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:538447:538447 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda11.3
ava-s0:538448:538448 [1] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
ava-s0:538448:538448 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:538448:538448 [1] NCCL INFO NET/IB : No device found.
ava-s0:538448:538448 [1] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
ava-s0:538448:538448 [1] NCCL INFO Using network Socket
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538448:538500 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
ava-s0:538448:538500 [1] NCCL INFO Setting affinity for GPU 1 to ffff,0000ffff
ava-s0:538447:538499 [0] NCCL INFO Channel 00/02 :    0   1
ava-s0:538447:538499 [0] NCCL INFO Channel 01/02 :    0   1
ava-s0:538447:538499 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
ava-s0:538447:538499 [0] NCCL INFO Setting affinity for GPU 0 to ffff,0000ffff
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538447:538499 [0] NCCL INFO Channel 00 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:538447:538499 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538447:538499 [0] NCCL INFO Channel 01 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538448:538500 [1] NCCL INFO Channel 00 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:538448:538500 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538448:538500 [1] NCCL INFO Channel 01 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:538448:538500 [1] NCCL INFO Connected all rings
ava-s0:538448:538500 [1] NCCL INFO Connected all trees
ava-s0:538448:538500 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
ava-s0:538448:538500 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:538447:538499 [0] NCCL INFO Connected all rings
ava-s0:538447:538499 [0] NCCL INFO Connected all trees
ava-s0:538447:538499 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
ava-s0:538447:538499 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:538447:538499 [0] NCCL INFO comm 0x7f4afc002fb0 rank 0 nranks 2 cudaDev 0 busId 1a000 - Init COMPLETE
ava-s0:538447:538447 [0] NCCL INFO Launch mode Parallel
ava-s0:538448:538500 [1] NCCL INFO comm 0x7f16ac002fb0 rank 1 nranks 2 cudaDev 1 busId 1b000 - Init COMPLETE
[ava-s0.ics.uci.edu-1] is OK (global rank: 1/2)
```
</details>


<details><summary>

`NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 2 --nnodes 1 torch-distributed-gpu-test.py`
</summary>

```
WARNING:__main__:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
ava-s0:538740:538740 [0] NCCL INFO Bootstrap : Using eno0:128.195.10.163<0>
ava-s0:538740:538740 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:538740:538740 [0] NCCL INFO NET/IB : No device found.
ava-s0:538740:538740 [0] NCCL INFO NET/Socket : Using [0]eno0:128.195.10.163<0>
ava-s0:538740:538740 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda11.3
ava-s0:538741:538741 [1] NCCL INFO Bootstrap : Using eno0:128.195.10.163<0>
ava-s0:538741:538741 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ava-s0:538741:538741 [1] NCCL INFO NET/IB : No device found.
ava-s0:538741:538741 [1] NCCL INFO NET/Socket : Using [0]eno0:128.195.10.163<0>
ava-s0:538741:538741 [1] NCCL INFO Using network Socket
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538740:538760 [0] NCCL INFO Channel 00/02 :    0   1
ava-s0:538740:538760 [0] NCCL INFO Channel 01/02 :    0   1
ava-s0:538741:538761 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
ava-s0:538740:538760 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
ava-s0:538741:538761 [1] NCCL INFO Setting affinity for GPU 1 to ffff,0000ffff
ava-s0:538740:538760 [0] NCCL INFO Setting affinity for GPU 0 to ffff,0000ffff
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538741:538761 [1] NCCL INFO Channel 00 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:538741:538761 [1] NCCL INFO Could not enable P2P between dev 1(=1b000) and dev 0(=1a000)
ava-s0:538741:538761 [1] NCCL INFO Channel 01 : 1[1b000] -> 0[1a000] via direct shared memory
ava-s0:538740:538760 [0] NCCL INFO Channel 00 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:538740:538760 [0] NCCL INFO Could not enable P2P between dev 0(=1a000) and dev 1(=1b000)
ava-s0:538740:538760 [0] NCCL INFO Channel 01 : 0[1a000] -> 1[1b000] via direct shared memory
ava-s0:538741:538761 [1] NCCL INFO Connected all rings
ava-s0:538741:538761 [1] NCCL INFO Connected all trees
ava-s0:538741:538761 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
ava-s0:538741:538761 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:538740:538760 [0] NCCL INFO Connected all rings
ava-s0:538740:538760 [0] NCCL INFO Connected all trees
ava-s0:538740:538760 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
ava-s0:538740:538760 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ava-s0:538741:538761 [1] NCCL INFO comm 0x7f5690002fb0 rank 1 nranks 2 cudaDev 1 busId 1b000 - Init COMPLETE
ava-s0:538740:538760 [0] NCCL INFO comm 0x7fc454002fb0 rank 0 nranks 2 cudaDev 0 busId 1a000 - Init COMPLETE
ava-s0:538740:538740 [0] NCCL INFO Launch mode Parallel
[ava-s0.ics.uci.edu-0] is OK (global rank: 0/2)
```
</details>

> Also are you using the gpus directly or via some kind of a virtual machine or docker?

I am using them directly. As for cudatoolkit, it seems my conda environment has 11.3 but nvidia-smi shows 11.2. SO seems to [indicate](https://stackoverflow.com/a/53504578/4953139) that they are separate API versions (runtime and driver respectively) and don't need to be the same but could this be the issue? I also checked that the runtime version was compatible with driver [here](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html). Output from nvidia-smi:
`NVIDIA-SMI 460.91.03    Driver Version: 460.91.03    CUDA Version: 11.2`



I don't have sudo access so cannot open `/var/log/syslog` and there doesn't seem to be `/var/log/ufw.log`. I am not sure if there is a firewall. Given the limited access, if it's clear to you that there is communication/hardware issue, I'll might have to reach out to the sysads now.

Perhaps you're correct. And given that your previous reports gave different nodes that succeeded 1st log, rank 0, 2nd log rank 5,6 or something like that, I wonder if this is some sort of intermittent failure.

So even 2 gpus fail.

And `lo` didn't help either. 

The story with cuda is this:
1. pytorch comes with its own cuda version which is only the runtime
2. then you have a system-wide cuda version that is used by `nvidia-smi` or if you were to build a CUDA extension (like deepspeed)
3. I think - the NVIDIA driver doesn't depend on CUDA - you can run the driver just fine w/o the CUDA toolkit installed. 

therefore CUDA version they usually don't have to match as long as the major version is the same. So I don't think there is anything to worry about.

Your next step is to file an Issue with https://github.com/pytorch/pytorch/issues where there are multiple NCCL experts and who will hopefully guide you to a resolution of this very unusual problem. It'd help for you to summarize what you have tried and share the logs.

As mentioned earlier do go for the short script as the reproduction script in your report - as nobody is going to try to debug your issue with a transformers example.

I'm looking forward to learning from your journey, @Shivanshu-Gupta 

This might be completely unrelated to this issue, but I was closely monitoring this thread as I was having very similar issues and I suspected it was an nccl-related error. After days of digging I fixed my issue by turning off ACS ([1](https://forums.developer.nvidia.com/t/multi-gpu-peer-to-peer-access-failing-on-tesla-k80/39748/17) [2](https://github.com/NVIDIA/nccl/issues/19#issuecomment-213223260)).

The links filtered lspci with "plx" but my machine didn't have this, so I just I manually checked the output of `lscpi -vvv` for "ACSCtl". I took note of the sources whenever ACSCtl was NOT in this form (there should be - after each part):
`ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-`

For each source that had ACSCtl on, I disabled it using a command like this
`sudo setpci -s cb:08.0 f2a.w=0000`

I did this a total of 21 times (I'm not sure if all of it was necessary). This way I made it so that every line in `sudo lspci -vvv | grep -i ACSCtl` would be `ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-`

And after that, I fixed my weeks-long nightmare of trying to get DDP to work.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@Shivanshu-Gupta Do you by chance have an AMD CPU? If so, check [this](https://github.com/stas00/toolbox/issues/1) out.