how do i stop this? [E ProcessGroupNCCL.cpp:474] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974891 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:474] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974889 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:474] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1445, OpType=ALLGATHER, NumelIn=12934, NumelOut=77604, Timeout(ms)=1800000) ran for 8974785 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:474] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974890 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:474] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974889 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974891 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974891 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 4] NCCL watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974890 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 4] NCCL watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974890 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 5] NCCL watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974889 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 5] NCCL watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974889 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 2] NCCL watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974889 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 2] NCCL watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1443, OpType=BROADCAST, NumelIn=4, NumelOut=4, Timeout(ms)=1800000) ran for 8974889 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:488] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:494] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:915] [Rank 3] NCCL watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1445, OpType=ALLGATHER, NumelIn=12934, NumelOut=77604, Timeout(ms)=1800000) ran for 8974785 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 3] NCCL watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1445, OpType=ALLGATHER, NumelIn=12934, NumelOut=77604, Timeout(ms)=1800000) ran for 8974785 milliseconds before timing out.
[rank: 1] Child process with PID 1228946 terminated with code -6. Forcefully terminating all other processes to avoid zombies 🧟
Killed
(py310) developer6@sp2sv-isk-3-2-h-h-2-6:~/kotoba-voice$ Exception in thread Thread-11 (_thread_body):
Traceback (most recent call last):
  File "/home/developer6/anaconda3/envs/py310/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/developer6/anaconda3/envs/py310/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/developer6/anaconda3/envs/py310/lib/python3.10/site-packages/wandb/filesync/step_upload.py", line 118, in _thread_body
    self._handle_event(event)
  File "/home/developer6/anaconda3/envs/py310/lib/python3.10/site-packages/wandb/filesync/step_upload.py", line 180, in _handle_event
    self._start_upload_job(event)
  File "/home/developer6/anaconda3/envs/py310/lib/python3.10/site-packages/wandb/filesync/step_upload.py", line 192, in _start_upload_job
    self._spawn_upload(event)
  File "/home/developer6/anaconda3/envs/py310/lib/python3.10/site-packages/wandb/filesync/step_upload.py", line 227, in _spawn_upload
    self._pool.submit(run_and_notify)
  File "/home/developer6/anaconda3/envs/py310/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown