Trainer Stuck at 0% Progress during Training on Multi-GPU Setup
Author: ZYM66
Pull request: no
Labels: name: solved

### System Info

- `transformers` version: 4.33.3
- Platform: Linux-4.15.0-213-generic-x86_64-with-glibc2.27
- Python version: 3.10.13
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Machine: 8 x A800 GPUs

### Who can help?

@ArthurZucker
@younesbelkada
@pacman100
@muellerzr 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am running the following script:
```
python ContinuePretrainLlama.py \
--data_path ./PretrainData/kyoto-train.txt \
--output_dir ./llama2  \
--num_train_epochs 1 \
--per_device_train_batch_size 8 \
--save_strategy no \
```
The core parts of the code that may be related to the issue are:

```python
def preprocess(data, tokenizer):
    input_encoding = tokenizer(data["text"], truncation=True, max_length=2048, padding="max_length",
                               return_tensors="np", return_special_tokens_mask=True, return_attention_mask=False)

    # Create the final dictionary
    result = {
        "input_ids": input_encoding["input_ids"],
        "special_tokens_mask": input_encoding["special_tokens_mask"],
    }

    return result


def make_pretrain_data_module(tokenizer: transformers.PreTrainedTokenizer, data_path, model) -> Dict:
    """Make dataset and collator for supervised fine-tuning."""
    # train_dataset = PretrainedDataset(tokenizer=tokenizer, data_path=data_path)
    dataset = load_dataset('text', data_files=data_path)
    train_dataset = \
        dataset.map(lambda data: preprocess(data, tokenizer), batched=True, desc="Processing", remove_columns=["text"])[
            "train"]
    data_collator = transformers.DataCollatorForLanguageModeling(tokenizer=tokenizer,
                                                                 mlm=False,
                                                                 return_tensors="pt")
    return dict(train_dataset=train_dataset, eval_dataset=None, data_collator=data_collator)


def smart_tokenizer_and_embedding_resize(
        special_tokens_dict: Dict,
        tokenizer: transformers.PreTrainedTokenizer,
        model: transformers.PreTrainedModel,
):
    """Resize tokenizer and embedding.

    Note: This is the unoptimized version that may make your embedding size not be divisible by 64.
    """
    num_new_tokens = tokenizer.add_special_tokens(special_tokens_dict)
    model.resize_token_embeddings(len(tokenizer), pad_to_multiple_of=8)

    if num_new_tokens > 0:
        input_embeddings = model.get_input_embeddings().weight.data
        output_embeddings = model.get_output_embeddings().weight.data

        input_embeddings_avg = input_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)
        output_embeddings_avg = output_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)

        input_embeddings[-num_new_tokens:] = input_embeddings_avg
        output_embeddings[-num_new_tokens:] = output_embeddings_avg


def train():
    parser = transformers.HfArgumentParser((ModelArguments, DataArguments, TrainingArguments))
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    if model_args.tokenizer_path is None:
        model_args.tokenizer_path = model_args.model_name_or_path

    tokenizer = LlamaTokenizer.from_pretrained(model_args.tokenizer_path,
                                               model_max_length=training_args.model_max_length, padding_side="right",
                                               use_fast=False, cache_dir=training_args.cache_dir)
    special_tokens_dict = dict()
    if tokenizer.pad_token is None:
        special_tokens_dict["pad_token"] = DEFAULT_PAD_TOKEN
    if tokenizer.eos_token is None:
        special_tokens_dict["eos_token"] = DEFAULT_EOS_TOKEN
    if tokenizer.bos_token is None:
        special_tokens_dict["bos_token"] = DEFAULT_BOS_TOKEN
    if tokenizer.unk_token is None:
        special_tokens_dict["unk_token"] = DEFAULT_UNK_TOKEN

    model = LlamaForCausalLM.from_pretrained(model_args.model_name_or_path).cuda(0)
    smart_tokenizer_and_embedding_resize(
        special_tokens_dict=special_tokens_dict,
        tokenizer=tokenizer,
        model=model,
    )
    if model_args.peft_lora:
        if model_args.lora_config is None:
            raise ValueError("Please specify the path to the PEFT config.")
        lora_config = LoraConfig(**LoraConfig.from_json_file(model_args.lora_config))
        model = get_peft_model(model, lora_config)
        print("You are using lora model!\n")
        model.print_trainable_parameters()

    data_module = make_pretrain_data_module(tokenizer=tokenizer, data_path=data_args.data_path, model=model)
    trainer = Trainer(model=model, tokenizer=tokenizer, args=training_args, **data_module)
    trainer.train()
    trainer.save_state()
    trainer.save_model(output_dir=training_args.output_dir)


if __name__ == "__main__":
    train()

```
As we can see, the GPU memory are successfully allocated
<img width="706" alt="image" src="https://github.com/huggingface/transformers/assets/61892155/297cf1a7-c157-4f16-8ac7-505d118b20cc">
 
It stuck at 0% more than 20 hours
<img width="1088" alt="image" src="https://github.com/huggingface/transformers/assets/61892155/a4d11920-f2ab-4335-9859-001bc9bb7ff6">

And I can't stop it with Control + C
<img width="1091" alt="image" src="https://github.com/huggingface/transformers/assets/61892155/3951a217-c46f-4baa-8949-3fb43a4e9388">

This is the dashboard link in wandb
https://wandb.ai/innovation_club/huggingface/runs/brca7vz5?workspace=user-

Additional Information:
Upon testing, the code runs perfectly on the CPU. However, when I shift to a multi-GPU setup, the training process doesn't proceed. It's essential to note that the memory allocation on the GPUs does take place, indicating that the process has initiated, but no forward or backward computations are observed.

### Expected behavior

Would appreciate any insights or suggestions on resolving this. Thank you!



Comments:
Add：I can stop it sometime and raise this exception
  File "/home/dl/zym/llamaJP/TestUseContinuePretrainLlama.py", line 221, in <module>
    train()
  File "/home/dl/zym/llamaJP/TestUseContinuePretrainLlama.py", line 215, in train
    trainer.train()
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1506, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1801, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/dl/zym/llamaJP/TestUseContinuePretrainLlama.py", line 104, in training_step
    loss = self.compute_loss(model, inputs)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 2673, in compute_loss
    outputs = model(**inputs)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 185, in forward
    outputs = self.parallel_apply(replicas, inputs, module_kwargs)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 200, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 102, in parallel_apply
    thread.join()
  File "/root/miniconda3/envs/LLM/lib/python3.10/threading.py", line 1096, in join
    self._wait_for_tstate_lock()
  File "/root/miniconda3/envs/LLM/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
KeyboardInterrupt


Most probably something related to GPU communication. I think we had something similar here: #24735. 

@ZYM66 does the training run fine on a single GPU setup? just run `CUDA_VISIBLE_DEVICES=0 yourscript.py`

> @ZYM66 does the training run fine on a single GPU setup? just run `CUDA_VISIBLE_DEVICES=0 yourscript.py`

It works well when I am using my CPU
<img width="1134" alt="image" src="https://github.com/huggingface/transformers/assets/61892155/d01c0502-c909-4c5b-a1d9-a6371c7bf55f">

But  I can't run it on a single GPU, because it will run out of memory


> @ZYM66 does the training run fine on a single GPU setup? just run `CUDA_VISIBLE_DEVICES=0 yourscript.py`

@younesbelkada 
I tried my script on GPT2 with a single GPU  It looks well!
<img width="1108" alt="image" src="https://github.com/huggingface/transformers/assets/61892155/2876aa34-a513-4e90-98e7-b642e66de334">

but
when I am using Mult-GPUs,  the trainer still stuck at 0%
<img width="1100" alt="image" src="https://github.com/huggingface/transformers/assets/61892155/44564692-7a6c-4e12-9726-8db0ddb75430">


@ZYM66 what happens when you use `accelerate launch --multi_gpu --num_processes {number_of_gpus} my script` instead? (basically, are you wanting model parallelism or data parallelism here?)

> @ZYM66 what happens when you use `accelerate launch --multi_gpu --num_processes {number_of_gpus} my script` instead? (basically, are you wanting model parallelism or data parallelism here?)
 
@muellerzr 
I tried this and found it still stuck here with no errors:
<img width="819" alt="image" src="https://github.com/huggingface/transformers/assets/61892155/4ab6c3a5-ba20-40c4-b065-d57c1f1f7eaf">
I want to use my deepspeed config for Data Parallelism, but I don't use my config for debugging.


To pinpoint the location of the program blockage, we modified the source code of `~/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py`. In the train() function call, we added the following debug information.

```python
        if delay_optimizer_creation:
            if use_accelerator_prepare:
                self.model = self.accelerator.prepare(self.model)
            self.create_optimizer_and_scheduler(num_training_steps=max_steps)

        # prepare using `accelerator` prepare
        if use_accelerator_prepare:
            self.model.train()
            if hasattr(self.lr_scheduler, "step"):
                print("4.0")
                if self.use_apex:
                    print("4.1")
                    model = self.accelerator.prepare(self.model)
                    print("4.2")
                else:
                    print("4.3")
                    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
                    print("4.4")
            else:
                print("4.5")
                # to handle cases wherein we pass "DummyScheduler" such as when it is specified in DeepSpeed config.
                model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
                    self.model, self.optimizer, self.lr_scheduler
                )
                print("4.6")
        print("5")

        if self.is_fsdp_enabled:
            self.model = self.model_wrapped = model
        print("6")
```
It is found that the program will only print 4.0 and 4.3, indicating that the following line is blocked.
```python
model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
```
We inquired about the following information:
https://github.com/pytorch/pytorch/issues/29482
https://blog.csdn.net/m0_37426155/article/details/108129952
https://espnet.github.io/espnet/espnet2_distributed.html#troubleshooting-for-nccl-with-ethernet-case

Then some modifications were made to the startup command.

① 
```bash
NCCL_DEBUG=INFO NCCL_IB_DISABLE=1 NCCL_SOCKET_IFNAME=eno accelerate launch ContinuePretrainLlama.py \
--data_path ./PretrainData/kyoto-train-dev.txt \
--output_dir ./llama2  \
--num_train_epochs 1 \
--bf16 True \
--peft True \
--lora_config lora_config.json \
--logging_steps 10 \
--model_name_or_path meta-llama/Llama-2-7b-hf
```
It encountered a new error after starting:
```
ncclInternalError: Internal check failed. Last error: Bootstrap : no socket interface found
```

②
```bash
NCCL_DEBUG=INFO NCCL_IB_DISABLE=1 NCCL_SOCKET_IFNAME=^lo,docker,virbr,vmnet,vboxnet,wl,ww,ppp accelerate launch ContinuePretrainLlama.py \
--data_path ./PretrainData/kyoto-train-dev.txt \
--output_dir ./llama2  \
--num_train_epochs 1 \
--bf16 True \
--peft True \
--lora_config lora_config.json \
--logging_steps 10 \
--model_name_or_path meta-llama/Llama-2-7b-hf
```
No error was reported when running at this time, but it eventually timed out.
```
before trainer.train()
4.0
4.3
aaa:55300:55300 [3] NCCL INFO cudaDriverVersion 12020
aaa:55300:55300 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ^lo,docker,virbr,vmnet,vboxnet,wl,ww,ppp
aaa:55300:55300 [3] NCCL INFO Bootstrap : Using br0:10.4.11.18<0>
aaa:55300:55300 [3] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
aaa:55300:55300 [3] NCCL INFO NET/Plugin : No plugin found, using internal implementation
aaa:55300:56600 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
aaa:55300:56600 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ^lo,docker,virbr,vmnet,vboxnet,wl,ww,ppp
aaa:55300:56600 [3] NCCL INFO NET/Socket : Using [0]br0:10.4.11.18<0> [1]vnet0:fe80::fc54:ff:fe3f:6ac%vnet0<0>
aaa:55300:56600 [3] NCCL INFO Using network Socket
aaa:55300:56600 [3] NCCL INFO Setting affinity for GPU 3 to 0fffff,ff000000,0fffffff
aaa:55300:56600 [3] NCCL INFO NVLS multicast support is not available on dev 3
aaa:55302:56568 [5] NCCL INFO Setting affinity for GPU 5 to ffff,fff00000,00ffffff,f0000000
aaa:55302:56568 [5] NCCL INFO NVLS multicast support is not available on dev 5
aaa:55301:56583 [4] NCCL INFO Setting affinity for GPU 4 to ffff,fff00000,00ffffff,f0000000
aaa:55301:56583 [4] NCCL INFO NVLS multicast support is not available on dev 4
aaa:55304:56549 [7] NCCL INFO Setting affinity for GPU 7 to ffff,fff00000,00ffffff,f0000000
aaa:55304:56549 [7] NCCL INFO NVLS multicast support is not available on dev 7
aaa:55303:56586 [6] NCCL INFO Setting affinity for GPU 6 to ffff,fff00000,00ffffff,f0000000
aaa:55303:56586 [6] NCCL INFO NVLS multicast support is not available on dev 6
aaa:55298:56548 [1] NCCL INFO Setting affinity for GPU 1 to 0fffff,ff000000,0fffffff
aaa:55298:56548 [1] NCCL INFO NVLS multicast support is not available on dev 1
aaa:55297:56547 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff,ff000000,0fffffff
aaa:55297:56547 [0] NCCL INFO NVLS multicast support is not available on dev 0
aaa:55299:56591 [2] NCCL INFO Setting affinity for GPU 2 to 0fffff,ff000000,0fffffff
aaa:55299:56591 [2] NCCL INFO NVLS multicast support is not available on dev 2
aaa:55299:56591 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1
aaa:55299:56591 [2] NCCL INFO P2P Chunksize set to 524288
aaa:55298:56548 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0
aaa:55297:56547 [0] NCCL INFO Channel 00/02 :    0   1   2   3   4   5   6   7
aaa:55298:56548 [1] NCCL INFO P2P Chunksize set to 524288
aaa:55304:56549 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6
aaa:55297:56547 [0] NCCL INFO Channel 01/02 :    0   1   2   3   4   5   6   7
aaa:55303:56586 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5
aaa:55304:56549 [7] NCCL INFO P2P Chunksize set to 524288
aaa:55297:56547 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
aaa:55303:56586 [6] NCCL INFO P2P Chunksize set to 524288
aaa:55302:56568 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4
aaa:55297:56547 [0] NCCL INFO P2P Chunksize set to 524288
aaa:55301:56583 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3
aaa:55302:56568 [5] NCCL INFO P2P Chunksize set to 524288
aaa:55300:56600 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2
aaa:55301:56583 [4] NCCL INFO P2P Chunksize set to 524288
aaa:55300:56600 [3] NCCL INFO P2P Chunksize set to 524288
aaa:55301:56583 [4] NCCL INFO Channel 00/0 : 4[ce000] -> 5[d1000] via P2P/IPC/read
aaa:55297:56547 [0] NCCL INFO Channel 00/0 : 0[4f000] -> 1[52000] via P2P/IPC/read
aaa:55301:56583 [4] NCCL INFO Channel 01/0 : 4[ce000] -> 5[d1000] via P2P/IPC/read
aaa:55297:56547 [0] NCCL INFO Channel 01/0 : 0[4f000] -> 1[52000] via P2P/IPC/read
aaa:55303:56586 [6] NCCL INFO Channel 00/0 : 6[d5000] -> 7[d6000] via P2P/IPC/read
aaa:55302:56568 [5] NCCL INFO Channel 00/0 : 5[d1000] -> 6[d5000] via P2P/IPC
aaa:55304:56549 [7] NCCL INFO Channel 00 : 7[d6000] -> 0[4f000] via SHM/direct/direct
aaa:55304:56549 [7] NCCL INFO Channel 01 : 7[d6000] -> 0[4f000] via SHM/direct/direct
aaa:55299:56591 [2] NCCL INFO Channel 00/0 : 2[56000] -> 3[57000] via P2P/IPC/read
aaa:55298:56548 [1] NCCL INFO Channel 00/0 : 1[52000] -> 2[56000] via P2P/IPC
aaa:55300:56600 [3] NCCL INFO Channel 00 : 3[57000] -> 4[ce000] via SHM/direct/direct
aaa:55300:56600 [3] NCCL INFO Channel 01 : 3[57000] -> 4[ce000] via SHM/direct/direct
aaa:55303:56586 [6] NCCL INFO Channel 01/0 : 6[d5000] -> 7[d6000] via P2P/IPC/read
aaa:55302:56568 [5] NCCL INFO Channel 01/0 : 5[d1000] -> 6[d5000] via P2P/IPC
aaa:55298:56548 [1] NCCL INFO Channel 01/0 : 1[52000] -> 2[56000] via P2P/IPC
aaa:55299:56591 [2] NCCL INFO Channel 01/0 : 2[56000] -> 3[57000] via P2P/IPC/read
aaa:55304:56549 [7] NCCL INFO Connected all rings
aaa:55304:56549 [7] NCCL INFO Channel 00/0 : 7[d6000] -> 6[d5000] via P2P/IPC/read
aaa:55301:56583 [4] NCCL INFO Connected all rings
aaa:55302:56568 [5] NCCL INFO Connected all rings
aaa:55303:56586 [6] NCCL INFO Connected all rings
aaa:55297:56547 [0] NCCL INFO Connected all rings
aaa:55299:56591 [2] NCCL INFO Connected all rings
aaa:55298:56548 [1] NCCL INFO Connected all rings
aaa:55300:56600 [3] NCCL INFO Connected all rings
aaa:55300:56600 [3] NCCL INFO Channel 00/0 : 3[57000] -> 2[56000] via P2P/IPC/read
aaa:55304:56549 [7] NCCL INFO Channel 01/0 : 7[d6000] -> 6[d5000] via P2P/IPC/read
aaa:55300:56600 [3] NCCL INFO Channel 01/0 : 3[57000] -> 2[56000] via P2P/IPC/read
aaa:55301:56583 [4] NCCL INFO Channel 00 : 4[ce000] -> 3[57000] via SHM/direct/direct
aaa:55301:56583 [4] NCCL INFO Channel 01 : 4[ce000] -> 3[57000] via SHM/direct/direct
aaa:55303:56586 [6] NCCL INFO Channel 00/0 : 6[d5000] -> 5[d1000] via P2P/IPC
aaa:55302:56568 [5] NCCL INFO Channel 00/0 : 5[d1000] -> 4[ce000] via P2P/IPC/read
aaa:55299:56591 [2] NCCL INFO Channel 00/0 : 2[56000] -> 1[52000] via P2P/IPC
aaa:55298:56548 [1] NCCL INFO Channel 00/0 : 1[52000] -> 0[4f000] via P2P/IPC/read
aaa:55299:56591 [2] NCCL INFO Channel 01/0 : 2[56000] -> 1[52000] via P2P/IPC
aaa:55302:56568 [5] NCCL INFO Channel 01/0 : 5[d1000] -> 4[ce000] via P2P/IPC/read
aaa:55303:56586 [6] NCCL INFO Channel 01/0 : 6[d5000] -> 5[d1000] via P2P/IPC
aaa:55298:56548 [1] NCCL INFO Channel 01/0 : 1[52000] -> 0[4f000] via P2P/IPC/read
aaa:55300:56600 [3] NCCL INFO Connected all trees
aaa:55300:56600 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
aaa:55300:56600 [3] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
aaa:55304:56549 [7] NCCL INFO Connected all trees
aaa:55304:56549 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
aaa:55304:56549 [7] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
aaa:55303:56586 [6] NCCL INFO Connected all trees
aaa:55303:56586 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
aaa:55303:56586 [6] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
aaa:55302:56568 [5] NCCL INFO Connected all trees
aaa:55302:56568 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
aaa:55302:56568 [5] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
aaa:55301:56583 [4] NCCL INFO Connected all trees
aaa:55301:56583 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
aaa:55301:56583 [4] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
aaa:55299:56591 [2] NCCL INFO Connected all trees
aaa:55299:56591 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
aaa:55299:56591 [2] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
aaa:55298:56548 [1] NCCL INFO Connected all trees
aaa:55298:56548 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
aaa:55298:56548 [1] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
aaa:55297:56547 [0] NCCL INFO Connected all trees
aaa:55297:56547 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
aaa:55297:56547 [0] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
aaa:55298:56548 [1] NCCL INFO comm 0xfa8d770 rank 1 nranks 8 cudaDev 1 busId 52000 commId 0xb081437536642957 - Init COMPLETE
aaa:55303:56586 [6] NCCL INFO comm 0xffbdc10 rank 6 nranks 8 cudaDev 6 busId d5000 commId 0xb081437536642957 - Init COMPLETE
aaa:55302:56568 [5] NCCL INFO comm 0xa225f30 rank 5 nranks 8 cudaDev 5 busId d1000 commId 0xb081437536642957 - Init COMPLETE
aaa:55299:56591 [2] NCCL INFO comm 0x1009c7e0 rank 2 nranks 8 cudaDev 2 busId 56000 commId 0xb081437536642957 - Init COMPLETE
aaa:55304:56549 [7] NCCL INFO comm 0x84f2140 rank 7 nranks 8 cudaDev 7 busId d6000 commId 0xb081437536642957 - Init COMPLETE
aaa:55297:56547 [0] NCCL INFO comm 0x15687100 rank 0 nranks 8 cudaDev 0 busId 4f000 commId 0xb081437536642957 - Init COMPLETE
aaa:55301:56583 [4] NCCL INFO comm 0x1170e1b0 rank 4 nranks 8 cudaDev 4 busId ce000 commId 0xb081437536642957 - Init COMPLETE
aaa:55300:56600 [3] NCCL INFO comm 0x16f8ac10 rank 3 nranks 8 cudaDev 3 busId 57000 commId 0xb081437536642957 - Init COMPLETE
[E ProcessGroupNCCL.cpp:474] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800111 milliseconds before timing out.
Traceback (most recent call last):
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 140, in <module>
Traceback (most recent call last):
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 140, in <module>
    train()
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 133, in train
    train()
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 133, in train
    trainer.train()
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1506, in train
    trainer.train()
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1506, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1639, in _inner_training_loop
    return inner_training_loop(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1639, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1284, in prepare
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1284, in prepare
    result = tuple(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1285, in <genexpr>
    result = tuple(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1285, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1090, in _prepare_one
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1090, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1429, in prepare_model
    return self.prepare_model(obj, device_placement=device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1429, in prepare_model
    model = torch.nn.parallel.DistributedDataParallel(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    model = torch.nn.parallel.DistributedDataParallel(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: DDP expects same model across all ranks, but Rank 3 has 448 params, while rank 0 has inconsistent 0 params.
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: DDP expects same model across all ranks, but Rank 4 has 448 params, while rank 0 has inconsistent 0 params.
Traceback (most recent call last):
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 140, in <module>
Traceback (most recent call last):
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 140, in <module>
    train()
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 133, in train
    train()
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 133, in train
    trainer.train()
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1506, in train
    trainer.train()
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1506, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1639, in _inner_training_loop
    return inner_training_loop(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1639, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1284, in prepare
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1284, in prepare
    result = tuple(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1285, in <genexpr>
    result = tuple(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1285, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1090, in _prepare_one
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1090, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1429, in prepare_model
    return self.prepare_model(obj, device_placement=device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1429, in prepare_model
    model = torch.nn.parallel.DistributedDataParallel(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    model = torch.nn.parallel.DistributedDataParallel(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
    _verify_param_shape_across_processes(self.process_group, parameters)
      File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: DDP expects same model across all ranks, but Rank 2 has 448 params, while rank 0 has inconsistent 0 params.
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: DDP expects same model across all ranks, but Rank 5 has 448 params, while rank 0 has inconsistent 0 params.
[E ProcessGroupNCCL.cpp:474] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800283 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:474] [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800289 milliseconds before timing out.
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 140, in <module>
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 140, in <module>
        train()train()

  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 133, in train
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 133, in train
    trainer.train()
    trainer.train()  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1506, in train

  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1506, in train
        return inner_training_loop(return inner_training_loop(

  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1639, in _inner_training_loop
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1639, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1284, in prepare
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1284, in prepare
        result = tuple(result = tuple(

  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1285, in <genexpr>
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1285, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
      File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1090, in _prepare_one
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1090, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
      File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1429, in prepare_model
return self.prepare_model(obj, device_placement=device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1429, in prepare_model
        model = torch.nn.parallel.DistributedDataParallel(model = torch.nn.parallel.DistributedDataParallel(

  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 795, in __init__
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 795, in __init__
        _verify_param_shape_across_processes(self.process_group, parameters)_verify_param_shape_across_processes(self.process_group, parameters)

  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
        return dist._verify_params_across_processes(process_group, tensors, logger)return dist._verify_params_across_processes(process_group, tensors, logger)

RuntimeError: RuntimeErrorDDP expects same model across all ranks, but Rank 7 has 448 params, while rank 0 has inconsistent 0 params.: 
DDP expects same model across all ranks, but Rank 0 has 448 params, while rank 1 has inconsistent 0 params.
Traceback (most recent call last):
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 140, in <module>
Traceback (most recent call last):
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 140, in <module>
    train()
  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 133, in train
    train()    
trainer.train()  File "/home/dl/lzf/llamaJP/ContinuePretrainLlama.py", line 133, in train

  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1506, in train
    trainer.train()
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1506, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1639, in _inner_training_loop
    return inner_training_loop(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1639, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1284, in prepare
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1284, in prepare
    result = tuple(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1285, in <genexpr>
    result = tuple(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1285, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1090, in _prepare_one
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1090, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1429, in prepare_model
    return self.prepare_model(obj, device_placement=device_placement)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/accelerator.py", line 1429, in prepare_model
    model = torch.nn.parallel.DistributedDataParallel(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    model = torch.nn.parallel.DistributedDataParallel(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
    _verify_param_shape_across_processes(self.process_group, parameters)    
return dist._verify_params_across_processes(process_group, tensors, logger)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
RuntimeError: DDP expects same model across all ranks, but Rank 6 has 448 params, while rank 0 has inconsistent 0 params.
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: DDP expects same model across all ranks, but Rank 1 has 448 params, while rank 2 has inconsistent 0 params.
[E ProcessGroupNCCL.cpp:474] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800522 milliseconds before timing out.
aaa:55297:56614 [0] NCCL INFO [Service thread] Connection closed by localRank 0
[E ProcessGroupNCCL.cpp:474] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800530 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:474] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800591 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:474] [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800619 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:474] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800651 milliseconds before timing out.
aaa:55297:55856 [0] NCCL INFO comm 0x15687100 rank 0 nranks 8 cudaDev 0 busId 4f000 - Abort COMPLETE
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800522 milliseconds before timing out.
aaa:55299:56611 [2] NCCL INFO [Service thread] Connection closed by localRank 2
aaa:55301:56617 [4] NCCL INFO [Service thread] Connection closed by localRank 4
aaa:55299:55850 [0] NCCL INFO comm 0x1009c7e0 rank 2 nranks 8 cudaDev 2 busId 56000 - Abort COMPLETE
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 2] NCCL watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800111 milliseconds before timing out.
aaa:55303:56613 [6] NCCL INFO [Service thread] Connection closed by localRank 6
aaa:55301:55852 [0] NCCL INFO comm 0x1170e1b0 rank 4 nranks 8 cudaDev 4 busId ce000 - Abort COMPLETE
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 4] NCCL watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800283 milliseconds before timing out.
aaa:55303:55853 [0] NCCL INFO comm 0xffbdc10 rank 6 nranks 8 cudaDev 6 busId d5000 - Abort COMPLETE
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 6] NCCL watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800289 milliseconds before timing out.
aaa:55298:56616 [1] NCCL INFO [Service thread] Connection closed by localRank 1
aaa:55300:56618 [3] NCCL INFO [Service thread] Connection closed by localRank 3
aaa:55298:55845 [0] NCCL INFO comm 0xfa8d770 rank 1 nranks 8 cudaDev 1 busId 52000 - Abort COMPLETE
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800651 milliseconds before timing out.
aaa:55302:56615 [5] NCCL INFO [Service thread] Connection closed by localRank 5
aaa:55300:55857 [0] NCCL INFO comm 0x16f8ac10 rank 3 nranks 8 cudaDev 3 busId 57000 - Abort COMPLETE
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 3] NCCL watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800530 milliseconds before timing out.
aaa:55304:56612 [7] NCCL INFO [Service thread] Connection closed by localRank 7
aaa:55302:55864 [0] NCCL INFO comm 0xa225f30 rank 5 nranks 8 cudaDev 5 busId d1000 - Abort COMPLETE
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 5] NCCL watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800591 milliseconds before timing out.
aaa:55304:55848 [0] NCCL INFO comm 0x84f2140 rank 7 nranks 8 cudaDev 7 busId d6000 - Abort COMPLETE
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 7] NCCL watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=1800000) ran for 1800619 milliseconds before timing out.
[2023-10-12 21:59:50,478] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 55297) of binary: /root/miniconda3/envs/LLM/bin/python
Traceback (most recent call last):
  File "/root/miniconda3/envs/LLM/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command
    multi_gpu_launcher(args)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
    distrib_run.run(args)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
======================================================
ContinuePretrainLlama.py FAILED
------------------------------------------------------
Failures:
[1]:
  time      : 2023-10-12_21:59:50
  host      : aaa
  rank      : 1 (local_rank: 1)
  exitcode  : -6 (pid: 55298)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 55298
[2]:
  time      : 2023-10-12_21:59:50
  host      : aaa
  rank      : 2 (local_rank: 2)
  exitcode  : -6 (pid: 55299)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 55299
[3]:
  time      : 2023-10-12_21:59:50
  host      : aaa
  rank      : 3 (local_rank: 3)
  exitcode  : -6 (pid: 55300)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 55300
[4]:
  time      : 2023-10-12_21:59:50
  host      : aaa
  rank      : 4 (local_rank: 4)
  exitcode  : -6 (pid: 55301)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 55301
[5]:
  time      : 2023-10-12_21:59:50
  host      : aaa
  rank      : 5 (local_rank: 5)
  exitcode  : -6 (pid: 55302)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 55302
[6]:
  time      : 2023-10-12_21:59:50
  host      : aaa
  rank      : 6 (local_rank: 6)
  exitcode  : -6 (pid: 55303)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 55303
[7]:
  time      : 2023-10-12_21:59:50
  host      : aaa
  rank      : 7 (local_rank: 7)
  exitcode  : -6 (pid: 55304)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 55304
------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-10-12_21:59:50
  host      : aaa
  rank      : 0 (local_rank: 0)
  exitcode  : -6 (pid: 55297)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 55297
=====================================================
```



What is your config when doing `accelerate launch`? (Run `accelerate env`)

Edit: Actually we do find the issue. Not every rank is getting the same prepared model




> What is your config when doing `accelerate launch`? (Run `accelerate env`)
> 
> Edit: Actually we do find the issue. Not every rank is getting the same prepared model

- `Accelerate` version: 0.23.0
- Platform: Linux-4.15.0-213-generic-x86_64-with-glibc2.27
- Python version: 3.10.13
- Numpy version: 1.24.1
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- PyTorch XPU available: False
- PyTorch NPU available: False
- System RAM: 1007.56 GB
- GPU type: NVIDIA A800 80GB PCIe
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []

I encounter the same issue. Anyone got a solution?

> I encounter the same issue. Anyone got a solution?

It seems work out there, https://github.com/NVIDIA/nccl/issues/1027
You should add NCCL_P2P_DISABLE=1 before your command.


If the NCCL flag fixes, would be the same as #26814! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

这是来自QQ邮箱的假期自动回复邮件。你好，我最近正在休假中，无法亲自回复你的邮件。我将在看见后，尽快给你回复。

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The latest accelerate will automatically disable P2P if a card doesn't support it. Marking as solved unless someone else has issues

I'm facing similar issue. The training is fine with one GPU, but will stuck with multiple GPUs. 
Setting NCCL_P2P_DISABLE=1 will raise error:
```
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [128,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [150,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "/home/ubuntu/jialin/iterative_lora/llm.py", line 309, in <module>
    trainer.train()
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/transformers/trainer.py", line 1553, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/transformers/trainer.py", line 1835, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/transformers/trainer.py", line 2679, in training_step
    loss = self.compute_loss(model, inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/transformers/trainer.py", line 2704, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/torch/nn/parallel/data_parallel.py", line 171, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/torch/nn/parallel/data_parallel.py", line 181, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/torch/nn/parallel/parallel_apply.py", line 89, in parallel_apply
    output.reraise()
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/torch/_utils.py", line 644, in reraise
    raise exception
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
    output = module(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/transformers/models/opt/modeling_opt.py", line 944, in forward
    outputs = self.model.decoder(
              ^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/transformers/models/opt/modeling_opt.py", line 710, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/transformers/models/opt/modeling_opt.py", line 330, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/jialin_pytorch/lib/python3.11/site-packages/transformers/models/opt/modeling_opt.py", line 225, in forward
    attn_weights, torch.tensor(torch.finfo(attn_weights.dtype).min, device=attn_weights.device)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```
what should I do now

这是来自QQ邮箱的假期自动回复邮件。你好，我最近正在休假中，无法亲自回复你的邮件。我将在看见后，尽快给你回复。

I finally solved this by disabling ACS in bios, ref https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs. Changing nvidia driver and cuda version doesn't help.

This test is very helpful. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#gpu-to-gpu-communication

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I am working on a cluster with SLURM - it took me forever to resolve this issue; however, when I changed the ddp backend to `--ddp_backend gloo`, it finally worked!

I'm on an 8x 3090 Supermicro system.

I was on driver version 545 and had the bug unless I set NCCL_P2P_DISABLE=1.

Downgrading to 535 makes things work without having to set NCCL_P2P_DISABLE=1

@muellerzr Hi I think this issue persists, as I rent a pod on RunPod, the training never begins without using "NCCL_P2P_DISABLE=1", the progress bar doesn't even show up if I run normally with `accelerate launch`. 


Still has this issue here when using CPO trainer with preprocessing class = tokenizer, but with SFT, which does not need tokenizer, it works fine on multi GPUs.

> Still has this issue here when using CPO trainer with preprocessing class = tokenizer, but with SFT, which does not need tokenizer, it works fine on multi GPUs.

Did you try disabling P2P? 

I'm experiencing the same problem on a server consisting of four A800s, Trainer is stuck at 0 without training before P2P is disabled, and reports an error after it's disabled

![Image](https://github.com/user-attachments/assets/d4369b24-322f-47eb-ad86-2dac9c55b844)

In addition I checked the communication between the GPUs and the following graph shows the results after disabling P2P

![Image](https://github.com/user-attachments/assets/af53628f-1bc6-47e0-9ea7-752d22b2375b)