Unable to run run_instance_segmentation_no_trainer with HF Accelerate
Author: gohjiayi
Pull request: no
Labels: name: bug

### System Info

I am trying to run the [examples/pytorch/instance-segmentation/run_instance_segmentation_no_trainer.py](https://github.com/huggingface/transformers/blob/d1b92369ca193da49f9f7ecd01b08ece45c2c9aa/examples/pytorch/instance-segmentation/run_instance_segmentation_no_trainer.py) with HF Accelerate. I was able to run the other Trainer API example successfully, but the No Trainer (Accelerate) version is facing the following bug.

This is using the `4.52.0.dev0` instance. The only change I've made was to change epochs=2. 

The following error arose, when trying to prompt for more information, ChatGPT suggests it could be the following issues but I have no idea on what could be the root cause. No other related issues found and the docs bot was not working. Would appreciate advice on how to run this example script as I hope to adopt it for my task.

| **Category**                | **Potential Issue**                                                                 | **Explanation**                                                                                             | **Recommended Fix**                                                                                   |
|----------------------------|--------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|
| **Model Config Mismatch**  | Mismatch in `num_labels` vs checkpoint (81 vs 3)                                     | Causes some layers (e.g., `class_predictor`) to be randomly initialized, might desync ranks                 | Set `config.num_labels = 3` **before** loading the model or use a matching checkpoint                 |
| **DDP Desynchronization**  | Different logic across ranks (e.g., `if rank == 0:` doing extra things)             | All ranks must call collectives in the same order and time                                                  | Ensure logic is **identical** across all ranks                                                         |
| **Evaluation in DDP**      | Evaluation logic not synchronized                                                   | Can cause hanging during collective ops like `all_gather`                                                   | Skip evaluation for non-zero ranks or use `if rank == 0:` carefully                                   |
| **GPU Communication**      | NCCL timeout or deadlock due to driver/hardware/GIL issues                          | Long-running or stuck collectives cause watchdog termination                                                | Set env vars: `NCCL_BLOCKING_WAIT=1`, `NCCL_ASYNC_ERROR_HANDLING=1`, and reduce batch size if needed  |
| **Distributed Setup**      | Improper `accelerate` or `torchrun` configuration                                   | One process might be behaving incorrectly                                                                   | Test with single GPU first: `CUDA_VISIBLE_DEVICES=0 accelerate launch --num_processes=1 ...`          |
| **Deprecated Args**        | `_max_size` passed to `Mask2FormerImageProcessor`                                   | Harmless, but messy                                                                                         | Remove `_max_size` from processor initialization                                                       |
| **Resource Overload**      | GPU memory, bandwidth, or CPU bottleneck                                            | Can indirectly cause slowdowns or crashes                                                                   | Monitor with `nvidia-smi`, lower batch size, reduce `num_workers`                                     |

Error message below:
```
loading weights file model.safetensors from cache at /home/jiayi/.cache/huggingface/hub/models--facebook--mask2former-swin-tiny-coco-instance/snapshots/22c4a2f15dc88149b8b8d9f4d42c54431fbd66f6/model.safetensors
Instantiating SwinBackbone model under default dtype torch.float32.
All model checkpoint weights were used when initializing Mask2FormerForUniversalSegmentation.

Some weights of Mask2FormerForUniversalSegmentation were not initialized from the model checkpoint at facebook/mask2former-swin-tiny-coco-instance and are newly initialized because the shapes did not match:
- class_predictor.bias: found shape torch.Size([81]) in the checkpoint and torch.Size([3]) in the model instantiated
- class_predictor.weight: found shape torch.Size([81, 256]) in the checkpoint and torch.Size([3, 256]) in the model instantiated
- criterion.empty_weight: found shape torch.Size([81]) in the checkpoint and torch.Size([3]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
/raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/transformers/utils/deprecation.py:172: UserWarning: The following named arguments are not valid for `Mask2FormerImageProcessor.__init__` and were ignored: '_max_size'
  return func(*args, **kwargs)
loading configuration file preprocessor_config.json from cache at /home/jiayi/.cache/huggingface/hub/models--facebook--mask2former-swin-tiny-coco-instance/snapshots/22c4a2f15dc88149b8b8d9f4d42c54431fbd66f6/preprocessor_config.json
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
/raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/transformers/utils/deprecation.py:172: UserWarning: The following named arguments are not valid for `Mask2FormerImageProcessor.__init__` and were ignored: '_max_size'
  return func(*args, **kwargs)
Image processor Mask2FormerImageProcessor {
  "do_normalize": true,
  "do_reduce_labels": true,
  "do_rescale": true,
  "do_resize": true,
  "ignore_index": 255,
  "image_mean": [
    0.48500001430511475,
    0.4560000002384186,
    0.4059999883174896
  ],
  "image_processor_type": "Mask2FormerImageProcessor",
  "image_std": [
    0.2290000021457672,
    0.2239999920129776,
    0.22499999403953552
  ],
  "num_labels": 80,
  "resample": 2,
  "rescale_factor": 0.00392156862745098,
  "size": {
    "height": 256,
    "width": 256
  },
  "size_divisor": 32
}

05/26/2025 17:59:13 - INFO - __main__ - ***** Running training *****
05/26/2025 17:59:13 - INFO - __main__ -   Num examples = 1600
05/26/2025 17:59:13 - INFO - __main__ -   Num Epochs = 2
05/26/2025 17:59:13 - INFO - __main__ -   Instantaneous batch size per device = 8
05/26/2025 17:59:13 - INFO - __main__ -   Total train batch size (w. parallel, distributed & accumulation) = 32
05/26/2025 17:59:13 - INFO - __main__ -   Gradient Accumulation steps = 2
05/26/2025 17:59:13 - INFO - __main__ -   Total optimization steps = 100
 50%|███████████████████████████████████                                   | 50/100 [00:50<00:44,  1.13it/s]05/26/2025 18:00:03 - INFO - __main__ - ***** Running evaluation *****
                                                                                                           [rank1]:[E526 18:10:06.330014458 ProcessGroupNCCL.cpp:632] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1516, OpType=_ALLGATHER_BASE, NumelIn=4, NumelOut=8, Timeout(ms)=600000) ran for 600075 milliseconds before timing out.
[rank1]:[E526 18:10:06.330380113 ProcessGroupNCCL.cpp:2268] [PG ID 0 PG GUID 0(default_pg) Rank 1]  failure detected by watchdog at work sequence id: 1516 PG status: last enqueued work: 1531, last completed work: 1515
[rank1]:[E526 18:10:06.330405467 ProcessGroupNCCL.cpp:670] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank1]:[E526 18:10:06.330496568 ProcessGroupNCCL.cpp:2103] [PG ID 0 PG GUID 0(default_pg) Rank 1] First PG on this rank to signal dumping.
[rank0]:[E526 18:10:06.348434228 ProcessGroupNCCL.cpp:632] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1520, OpType=_ALLGATHER_BASE, NumelIn=7, NumelOut=14, Timeout(ms)=600000) ran for 600093 milliseconds before timing out.
[rank0]:[E526 18:10:06.348758153 ProcessGroupNCCL.cpp:2268] [PG ID 0 PG GUID 0(default_pg) Rank 0]  failure detected by watchdog at work sequence id: 1520 PG status: last enqueued work: 1531, last completed work: 1519
[rank0]:[E526 18:10:06.348778609 ProcessGroupNCCL.cpp:670] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank0]:[E526 18:10:06.348837555 ProcessGroupNCCL.cpp:2103] [PG ID 0 PG GUID 0(default_pg) Rank 0] First PG on this rank to signal dumping.
[rank0]:[E526 18:10:07.235145069 ProcessGroupNCCL.cpp:1743] [PG ID 0 PG GUID 0(default_pg) Rank 0] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 1531, last completed NCCL work: 1519.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank1]:[E526 18:10:07.235182687 ProcessGroupNCCL.cpp:1743] [PG ID 0 PG GUID 0(default_pg) Rank 1] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 1531, last completed NCCL work: 1515.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank0]:[E526 18:10:07.235355691 ProcessGroupNCCL.cpp:1533] [PG ID 0 PG GUID 0(default_pg) Rank 0] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank1]:[E526 18:10:07.235369478 ProcessGroupNCCL.cpp:1533] [PG ID 0 PG GUID 0(default_pg) Rank 1] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank0]:[E526 18:10:07.496712146 ProcessGroupNCCL.cpp:684] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank0]:[E526 18:10:07.496742723 ProcessGroupNCCL.cpp:698] [Rank 0] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E526 18:10:07.499895520 ProcessGroupNCCL.cpp:1896] [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1520, OpType=_ALLGATHER_BASE, NumelIn=7, NumelOut=14, Timeout(ms)=600000) ran for 600093 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:635 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7f12fc3065e8 in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x23d (0x7f12fd621a1d in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0xc80 (0x7f12fd6237a0 in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f12fd624ead in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xd6df4 (0x7f13660c2df4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: <unknown function> + 0x8609 (0x7f1369f77609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f136a0b1353 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
[rank1]:[E526 18:10:07.500452346 ProcessGroupNCCL.cpp:684] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank1]:[E526 18:10:07.500470141 ProcessGroupNCCL.cpp:698] [Rank 1] To avoid data inconsistency, we are taking the entire process down.
  0%|                                                                                | 0/21 [10:04<?, ?it/s]
[rank1]:[E526 18:10:07.502423448 ProcessGroupNCCL.cpp:1896] [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1516, OpType=_ALLGATHER_BASE, NumelIn=4, NumelOut=8, Timeout(ms)=600000) ran for 600075 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:635 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7f3a12f885e8 in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x23d (0x7f3a142a3a1d in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0xc80 (0x7f3a142a57a0 in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f3a142a6ead in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xd6df4 (0x7f3a7cd44df4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: <unknown function> + 0x8609 (0x7f3a80bf9609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f3a80d33353 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1520, OpType=_ALLGATHER_BASE, NumelIn=7, NumelOut=14, Timeout(ms)=600000) ran for 600093 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:635 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7f12fc3065e8 in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x23d (0x7f12fd621a1d in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0xc80 (0x7f12fd6237a0 in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f12fd624ead in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xd6df4 (0x7f13660c2df4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: <unknown function> + 0x8609 (0x7f1369f77609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f136a0b1353 in /lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1902 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7f12fc3065e8 in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x11b4a6e (0x7f12fd5f3a6e in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xe07bed (0x7f12fd246bed in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xd6df4 (0x7f13660c2df4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #4: <unknown function> + 0x8609 (0x7f1369f77609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: clone + 0x43 (0x7f136a0b1353 in /lib/x86_64-linux-gnu/libc.so.6)

  what():  [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1516, OpType=_ALLGATHER_BASE, NumelIn=4, NumelOut=8, Timeout(ms)=600000) ran for 600075 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:635 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7f3a12f885e8 in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x23d (0x7f3a142a3a1d in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0xc80 (0x7f3a142a57a0 in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f3a142a6ead in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xd6df4 (0x7f3a7cd44df4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: <unknown function> + 0x8609 (0x7f3a80bf9609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f3a80d33353 in /lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1902 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7f3a12f885e8 in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x11b4a6e (0x7f3a14275a6e in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xe07bed (0x7f3a13ec8bed in /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xd6df4 (0x7f3a7cd44df4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #4: <unknown function> + 0x8609 (0x7f3a80bf9609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #5: clone + 0x43 (0x7f3a80d33353 in /lib/x86_64-linux-gnu/libc.so.6)

W0526 18:10:09.067974 3031878 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 3032005 closing signal SIGTERM
E0526 18:10:10.286100 3031878 torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: -6) local_rank: 0 (pid: 3032004) of binary: /raid/jiayi/safety_barrier_breach/mask2former_hf/venv/bin/python
Traceback (most recent call last):
  File "/raid/jiayi/safety_barrier_breach/mask2former_hf/venv/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
    args.func(args)
  File "/raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/accelerate/commands/launch.py", line 1189, in launch_command
    multi_gpu_launcher(args)
  File "/raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/accelerate/commands/launch.py", line 815, in multi_gpu_launcher
    distrib_run.run(args)
  File "/raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 883, in run
    elastic_launch(
  File "/raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/raid/jiayi/safety_barrier_breach/mask2former_hf/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
========================================================
original_run_instance_segmentation_no_trainer.py FAILED
--------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
--------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-05-26_18:10:09
  host      : DGX-Station
  rank      : 0 (local_rank: 0)
  exitcode  : -6 (pid: 3032004)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 3032004
========================================================
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the [examples/pytorch/instance-segmentation/run_instance_segmentation_no_trainer.py](https://github.com/huggingface/transformers/blob/d1b92369ca193da49f9f7ecd01b08ece45c2c9aa/examples/pytorch/instance-segmentation/run_instance_segmentation_no_trainer.py) script with the example command given in the README

### Expected behavior

The training is supposed to complete without any issue

Comments:
cc @sunmarc since the example command uses `accelerate` - is the documentation out of date for how those examples should be called?

@Rocketknight1 looks like model head needs to be resized but looks like its not handled correctly while training. I can take a look into this.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.