Training hangs in the end while calling dist.barrier()
Author: hasansalimkanmaz
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.4.0-1073-azure-x86_64-with-glibc2.27
- Python version: 3.8.0
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.10.1+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: DDP
```


### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am working on a custom TokenClassificationTask. For a specific model type, I am experiencing a hanging process at the end of the training. After I set `TORCH_DISTRIBUTED_DEBUG=DETAIL` and also added rank numbers to logs (to do this, I overwrote `train` method of `Trainer` class with additional loggings), the training failed and I received the below stack trace. 

```
Training completed for rank 6. Do not forget to share your model on huggingface.co/models =)

2022-05-30 15:54:58 INFO     nlp_ner_layoutlm.layoutlm.trainers.re_trainer Before barrier for rank 6
2022-05-30 15:54:58 INFO     nlp_ner_layoutlm.layoutlm.trainers.re_trainer Entering into barrier for rank 6
2022-05-30 15:54:59 INFO     transformers.modeling_utils Model weights saved in ./data/tmpm8wxl12l/checkpoint-590/pytorch_model.bin
2022-05-30 15:55:01 INFO     transformers.trainer Deleting older checkpoint [data/tmpm8wxl12l/checkpoint-585] due to args.save_total_limit
2022-05-30 15:55:01 ERROR    __main__   Detected mismatch between collectives on ranks. Rank 6 is running inconsistent collective: CollectiveFingerPrint(OpType=BARRIER
Traceback (most recent call last):
  File "nlp_ner_layoutlm/train_pipeline/training_step/training_script.py", line 53, in <module>
    train_model(
  File "/app/nlp_ner_layoutlm/layoutlm/utils/training_utils.py", line 160, in train_model
    raise e
  File "/app/nlp_ner_layoutlm/layoutlm/utils/training_utils.py", line 158, in train_model
    trainer.train(resume_from_checkpoint=get_last_checkpoint(checkpoint_dir))
  File "/app/nlp_ner_layoutlm/layoutlm/trainers/re_trainer.py", line 698, in train
    dist.barrier()
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 2709, in barrier
    work = default_pg.barrier(opts=opts)
RuntimeError: Detected mismatch between collectives on ranks. Rank 6 is running inconsistent collective: CollectiveFingerPrint(OpType=BARRIER
2022-05-30 15:55:01 ERROR    __main__   Detected mismatch between collectives on ranks. Rank 3 is running inconsistent collective: CollectiveFingerPrint(OpType=BROADCAST, TensorShape=[514], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))
Traceback (most recent call last):
  File "nlp_ner_layoutlm/train_pipeline/training_step/training_script.py", line 53, in <module>
    train_model(
  File "/app/nlp_ner_layoutlm/layoutlm/utils/training_utils.py", line 160, in train_model
    raise e
  File "/app/nlp_ner_layoutlm/layoutlm/utils/training_utils.py", line 158, in train_model
    trainer.train(resume_from_checkpoint=get_last_checkpoint(checkpoint_dir))
  File "/app/nlp_ner_layoutlm/layoutlm/trainers/re_trainer.py", line 603, in train
    tr_loss_step = self.training_step(model, inputs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2011, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2043, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 878, in forward
    self._sync_params()
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 1379, in _sync_params
    self._distributed_broadcast_coalesced(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 1334, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(
RuntimeError: Detected mismatch between collectives on ranks. Rank 3 is running inconsistent collective: CollectiveFingerPrint(OpType=BROADCAST, TensorShape=[514], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))
2022-05-30 15:55:01 ERROR    __main__   Detected mismatch between collectives on ranks. Rank 1 is running inconsistent collective: CollectiveFingerPrint(OpType=BROADCAST, TensorShape=[514], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))
Traceback (most recent call last):
... (Same error for other processes)
```

According to the trace, while the process with rank 6 is running `dist.barrier()` from `trainer.py line 1536`, the other processes are running a `forward_call`. I think this is the issue and due to this mis-communication the training hangs. When I checked similar issues on the web, I came across [this issue from speechbrain](https://github.com/speechbrain/speechbrain/issues/1166). It is exactly the same issue and they fixed it with a PR. Currently, I can't understand why the processes are ending up in different places in the code and can't figure out how to fix this issue.


### Expected behavior

```shell
As far as I understand, the processes should meet at `dist.barrier()` and training should succeed. Could you help me or point me to a fix that I can work on it?
```


Comments:
I do not see a `dist.barrier()` at line 1536 of the `trainer.py` file and I can't really help you without knowing which training script you are launching and how you launched it.

Maybe the line corresponds to the HF version that I use. The new line number is 1679 according to main branch. You can check out from [here](https://github.com/huggingface/transformers/blob/567d9c061d04513c24872b5709adc7a1384b8129/src/transformers/trainer.py#L1679)

I think I have found the issue, my custom model has outputs with variable lengths and I wasn't gathering all outputs with `distributed_concat` function as they are not torch tensors. This results in different metrics in each process due to different outputs without gathering. In addition, I am using `EarlyStoppingCallback` during my training. As the metrics are different for each process, one process can stop the training and enter `dist.barrier` while the others go on training. This results in hanging training. 

Until now, I haven't implemented the fix yet. After the implementation,  I will confirm here and close the issue. Thanks for your time anyway. 

I have just tested my fix and concluded that it is related to what I mentioned above. Thanks for your time @sgugger. I am closing the issue.

Thanks for confirming the fix worked!

> I have just tested my fix and concluded that it is related to what I mentioned above. Thanks for your time @sgugger. I am closing the issue.

Hello,
Could you please elaborate the solution?

I have the same issue, but i ve fixed: there is a code block in main.py: `try: model.evaluate() except save_checkpoint()`. But there is something wrong with the evaluation function, which is in another evaluate.py. AND there is also `dist.barrier()` in evaluate.py. So, the first process meet the evaluation error and skip to the `dist.barrier()` in the main.py, whilst the second process is waiting at the `dist.barrier()` at evaluate.py. Check errors in your evaluation logic should make you happy again.

hi @hasansalimkanmaz 
how did u fix the problem? could u share the solution?

@zimenglan-sysu-512 as the issue was related to different metrics in different workers, I fixed the issue having uniform metrics by an aggregation. However, as it was quite a long time ago and I don't have access the repo, I couldn't verify exactly how I solved it.