Repository: rwth-i6/returnn

Issue:
multi-gpu training: training hangs after one sub-epoch
Multi-gpu training using 4 GPUs on one single node (`torch_distributed` set to `{"class": torch.nn.parallel.DistributedDataParallel, "options": {}})`: training hangs after one sub-epoch.

Training hanging issue, complete log file below:
```
Torch distributed initialized. Hostname gpu-12.apptek.local, pid 11642, rank 1 / size 4, local rank 1 / local size 4.
Torch distributed initialized. Hostname gpu-12.apptek.local, pid 11643, rank 2 / size 4, local rank 2 / local size 4.
RETURNN starting up, version 1.20231104.223301+git.b4dbaddd, date/time 2023-11-05-05-02-24 (UTC-0500), pid 11643, cwd /nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/returnn/training/ReturnnTrainingJob.SJfEN18jeHhc/work, Python /usr/bin/python3TURNN starting up, version 1.20231104.223301+git.b4dbaddd, date/time 2023-11-05-05-02-24 (UTC-0500), pid 11642, cwd /nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/returnn/training/ReturnnTrainingJob.SJfEN18jeHhc/work, Python /usr/bin/python3rch distributed initialized. Hostname gpu-12.apptek.local, pid 11641, rank 0 / size 4, local rank 0 / local size 4.
RETURNN starting up, version 1.20231104.223301+git.b4dbaddd, date/time 2023-11-05-05-02-24 (UTC-0500), pid 11641, cwd /nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/returnn/training/ReturnnTrainingJob.SJfEN18jeHhc/work, Python /usr/bin/python3rch distributed initialized. Hostname gpu-12.apptek.local, pid 11644, rank 3 / size 4, local rank 3 / local size 4.
RETURNN starting up, version 1.20231104.223301+git.b4dbaddd, date/time 2023-11-05-05-02-24 (UTC-0500), pid 11644, cwd /nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/returnn/training/ReturnnTrainingJob.SJfEN18jeHhc/work, Python /usr/bin/python3ExtModCompiler call: g++ -shared -O2 -std=c++11 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -I /nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/frontend/_native -I /usr/include/python3.10 -fPIC -v -D_GLIBCXX_USE_CXX11_ABI=0 -g /tmp/3289467.1.gpu.q/pzheng/returnn_py_ext_mod_cache/_returnn_frontend_native/0f5f147c83/_returnn_frontend_native.cc -o /tmp/3289467.1.gpu.q/pzheng/returnn_py_ext_mod_cache/_returnn_frontend_native/0f5f147c83/_returnn_frontend_native.so
Reading sequence list for MetaDataset 'dev' from sub-dataset 'dev_alignments'
PyExtModCompiler call: g++ -shared -O2 -std=c++11 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -I /nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/frontend/_native -I /usr/include/python3.10 -fPIC -v -D_GLIBCXX_USE_CXX11_ABI=0 -g /tmp/3289467.1.gpu.q/pzheng/returnn_py_ext_mod_cache/_returnn_frontend_native/0f5f147c83/_returnn_frontend_native.cc -o /tmp/3289467.1.gpu.q/pzheng/returnn_py_ext_mod_cache/_returnn_frontend_native/0f5f147c83/_returnn_frontend_native.so
Waiting for lock-file: /tmp/3289467.1.gpu.q/pzheng/returnn_py_ext_mod_cache/_returnn_frontend_native/0f5f147c83/lock_file
Torch: Hostname gpu-12.apptek.local, pid 11643, using GPU 2.
PyTorch: 2.0.1+cu117 (e9ebda29d87ce0916ab08c06ab26fd3766a870e5) (<not-under-git> in /usr/local/lib/python3.10/dist-packages/torch)
Reading sequence list for MetaDataset 'dev' from sub-dataset 'dev_alignments'
[2023-11-05 10:02:36,720] INFO: Run time: 0:00:55 CPU: 0.40% RSS: 3.01GB VMS: 88.16GB
Train data:
  input: 0 x 1
  output: {'data': [1, 2], 'classes': [9001, 1]}
  MetaDataset, sequences: 1317274, frames: unknown
Dev data:
  MetaDataset, sequences: 6579, frames: unknown
PyExtModCompiler call: g++ -shared -O2 -std=c++11 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -I /nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/frontend/_native -I /usr/include/python3.10 -fPIC -v -D_GLIBCXX_USE_CXX11_ABI=0 -g /tmp/3289467.1.gpu.q/pzheng/returnn_py_ext_mod_cache/_returnn_frontend_native/0f5f147c83/_returnn_frontend_native.cc -o /tmp/3289467.1.gpu.q/pzheng/returnn_py_ext_mod_cache/_returnn_frontend_native/0f5f147c83/_returnn_frontend_native.so
Using device: cuda ('gpu' in config)
Start running torch distributed training on local rank 3.
/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.lOKkFj0ksUUM/output/i6_models/i6_models/config.py:86: TypeHintWarning: Skipping type check against 'VGG4LayerActFrontendV1Config'; this looks like a string-form forward reference imported from another module
  typeguard.check_type(self.cfg, cfg_parameter.annotation)
Reading sequence list for MetaDataset 'train' from sub-dataset 'train_alignments'
net params #: 78693129
Train data:
  input: 0 x 1
  output: {'data': [1, 2], 'classes': [9001, 1]}
  MetaDataset, sequences: 1317274, frames: unknown
Dev data:
  MetaDataset, sequences: 6579, frames: unknown
Using device: cuda ('gpu' in config)
Start running torch distributed training on local rank 2.
/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.lOKkFj0ksUUM/output/i6_models/i6_models/config.py:86: TypeHintWarning: Skipping type check against 'VGG4LayerActFrontendV1Config'; this looks like a string-form forward reference imported from another module
  typeguard.check_type(self.cfg, cfg_parameter.annotation)
net params #: 78693129
Torch: Hostname gpu-12.apptek.local, pid 11642, using GPU 1.
PyTorch: 2.0.1+cu117 (e9ebda29d87ce0916ab08c06ab26fd3766a870e5) (<not-under-git> in /usr/local/lib/python3.10/dist-packages/torch)
Reading sequence list for MetaDataset 'dev' from sub-dataset 'dev_alignments'
PyExtModCompiler call: g++ -shared -O2 -std=c++11 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -I /nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/frontend/_native -I /usr/include/python3.10 -fPIC -v -D_GLIBCXX_USE_CXX11_ABI=0 -g /tmp/3289467.1.gpu.q/pzheng/returnn_py_ext_mod_cache/_returnn_frontend_native/0f5f147c83/_returnn_frontend_native.cc -o /tmp/3289467.1.gpu.q/pzheng/returnn_py_ext_mod_cache/_returnn_frontend_native/0f5f147c83/_returnn_frontend_native.so
[2023-11-05 10:02:41,740] INFO: Run time: 0:01:00 CPU: 0.20% RSS: 6.86GB VMS: 132.85GB
Reading sequence list for MetaDataset 'train' from sub-dataset 'train_alignments'
Train data:
  input: 0 x 1
  output: {'data': [1, 2], 'classes': [9001, 1]}
  MetaDataset, sequences: 1317274, frames: unknown
Dev data:
  MetaDataset, sequences: 6579, frames: unknown
Using device: cuda ('gpu' in config)
Start running torch distributed training on local rank 1.
/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.lOKkFj0ksUUM/output/i6_models/i6_models/config.py:86: TypeHintWarning: Skipping type check against 'VGG4LayerActFrontendV1Config'; this looks like a string-form forward reference imported from another module
  typeguard.check_type(self.cfg, cfg_parameter.annotation)
Torch: Hostname gpu-12.apptek.local, pid 11641, using GPU 0.
PyTorch: 2.0.1+cu117 (e9ebda29d87ce0916ab08c06ab26fd3766a870e5) (<not-under-git> in /usr/local/lib/python3.10/dist-packages/torch)
Reading sequence list for MetaDataset 'dev' from sub-dataset 'dev_alignments'
net params #: 78693129
Reading sequence list for MetaDataset 'train' from sub-dataset 'train_alignments'
[2023-11-05 10:02:46,756] INFO: Run time: 0:01:05 CPU: 0.40% RSS: 9.53GB VMS: 156.38GB
Train data:
  input: 0 x 1
  output: {'data': [1, 2], 'classes': [9001, 1]}
  MetaDataset, sequences: 1317274, frames: unknown
Dev data:
  MetaDataset, sequences: 6579, frames: unknown
Using device: cuda ('gpu' in config)
Start running torch distributed training on local rank 0.
/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.lOKkFj0ksUUM/output/i6_models/i6_models/config.py:86: TypeHintWarning: Skipping type check against 'VGG4LayerActFrontendV1Config'; this looks like a string-form forward reference imported from another module
  typeguard.check_type(self.cfg, cfg_parameter.annotation)
net params #: 78693129
[2023-11-05 10:02:51,777] INFO: Run time: 0:01:10 CPU: 0.20% RSS: 12.45GB VMS: 180.09GB
Optimizer: Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    capturable: False
    differentiable: False
    eps: 1e-08
    foreach: None
    fused: None
    lr: 1.0
    maximize: False
    weight_decay: 0
)
Starting training at epoch 1, global train step 0
Optimizer: Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    capturable: False
differentiable: False
    eps: 1e-08
    foreach: None
    fused: None
    lr: 1.0
    maximize: False
    weight_decay: 0
)
Starting training at epoch 1, global train step 0
Optimizer: Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    capturable: False
    differentiable: False
    eps: 1e-08
    foreach: None
    fused: None
    lr: 1.0
    maximize: False
    weight_decay: 0
)
Starting training at epoch 1, global train step 0
Optimizer: Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    capturable: False
    differentiable: False
    eps: 1e-08
    foreach: None
    fused: None
    lr: 1.0
    maximize: False
    weight_decay: 0
)
Starting training at epoch 1, global train step 0
start epoch 1 global train step 0 with effective learning rate 1e-05 ...
start epoch 1 global train step 0 with effective learning rate 1e-05 ...
start epoch 1 global train step 0 with effective learning rate 1e-05 ...
start epoch 1 global train step 0 with effective learning rate 1e-05 ...
[2023-11-05 10:02:56,909] INFO: Run time: 0:01:15 CPU: 0.40% RSS: 17.02GB VMS: 193.15GB
[2023-11-05 10:04:22,223] INFO: Run time: 0:02:40 CPU: 0.20% RSS: 18.76GB VMS: 194.88GB
[2023-11-05 10:05:12,407] INFO: Run time: 0:03:30 CPU: 0.40% RSS: 22.50GB VMS: 203.38GB
[2023-11-05 10:05:27,460] INFO: Run time: 0:03:46 CPU: 0.20% RSS: 25.71GB VMS: 222.92GB
[2023-11-05 10:11:13,808] INFO: Run time: 0:09:32 CPU: 0.40% RSS: 28.28GB VMS: 230.39GB
[2023-11-05 10:33:48,563] INFO: Run time: 0:32:07 CPU: 0.40% RSS: 31.11GB VMS: 233.33GB
Trained 13301 steps, 0:47:07 elapsed (91.5% computing time)
Trained 13301 steps, 0:47:07 elapsed (91.0% computing time)
Trained 13301 steps, 0:47:07 elapsed (91.0% computing time)
Total train loss: ce 5.298 fer 0.735
Total train loss: ce 5.303 fer 0.736
Total train loss: ce 5.323 fer 0.738
start epoch 2 global train step 13301 with effective learning rate 3.111111111111111e-05 ...
start epoch 2 global train step 13301 with effective learning rate 3.111111111111111e-05 ...
start epoch 2 global train step 13301 with effective learning rate 3.111111111111111e-05 ...
Trained 13301 steps, 0:47:07 elapsed (90.9% computing time)
Total train loss: ce 5.313 fer 0.737
Evaluating dataset 'dev'
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py:1160: UserWarning: Converting mask without torch.bool dtype to bool; this will negatively affect performance. Prefer to use a boolean mask directly. (Triggered internally at ../aten/src/ATen/native/transformers/attention.cpp:150.)
  return torch._native_multi_head_attention(
dev: score 4.095 error 0.683
Only 1 epochs stored so far and keeping last 2 epochs and best 4 epochs, thus not cleaning up any epochs yet.
start epoch 2 global train step 13301 with effective learning rate 3.111111111111111e-05 ...
Unhandled exception <class 'RuntimeError'> in thread <_MainThread(MainThread, started 22977597855552)>, proc 11644.Unhandled exception <class 'RuntimeError'> in thread <_MainThread(MainThread, started 22911415002944)>, proc 11643.



Thread current, main, <_MainThread(MainThread, started 22911415002944)>:
(Excluded thread.)

That were all threads.Thread current, main, <_MainThread(MainThread, started 22977597855552)>:

(Excluded thread.)

That were all threads.
Unhandled exception <class 'RuntimeError'> in thread <_MainThread(MainThread, started 22558761436992)>, proc 11642.

Thread current, main, <_MainThread(MainThread, started 22558761436992)>:
(Excluded thread.)

That were all threads.
EXCEPTION
Traceback (most recent call last):
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/rnn.py", line 11, in <module>
    line: main()
    locals:
      main = <local> <function main at 0x14e5df55bf40>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/__main__.py", line 634, in main
    line: execute_main_task()
    locals:
      execute_main_task = <global> <function execute_main_task at 0x14e5df55be20>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/__main__.py", line 440, in execute_main_task
    line: engine.train()
    locals:
      engine = <global> <returnn.torch.engine.Engine object at 0x14e4f5ba9b70>
      engine.train = <global> <bound method Engine.train of <returnn.torch.engine.Engine object at 0x14e4f5ba9b70>>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/torch/engine.py", line 225, in Engine.train
    line: self.train_epoch()
    locals:
      self = <local> <returnn.torch.engine.Engine object at 0x14e4f5ba9b70>
      self.train_epoch = <local> <bound method Engine.train_epoch of <returnn.torch.engine.Engine object at 0x14e4f5ba9b70>>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/torch/engine.py", line 278, in Engine.train_epoch
    line: torch.distributed.all_reduce(_has_data, op=torch.distributed.ReduceOp.MIN)
    locals:
      torch = <global> <module 'torch' from '/usr/local/lib/python3.10/dist-packages/torch/__init__.py'>
      torch.distributed = <global> <module 'torch.distributed' from '/usr/local/lib/python3.10/dist-packages/torch/distributed/__init__.py'>
      torch.distributed.all_reduce = <global> <function all_reduce at 0x14e5bd5cc820>
      _has_data = <local> tensor([1], dtype=torch.int8)
      op = <not found>
      torch.distributed.ReduceOp = <global> <class 'torch.distributed.distributed_c10d.ReduceOp'>
      torch.distributed.ReduceOp.MIN = <global> <RedOpType.MIN: 3>
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1451, in broadcast_multigpu
    line: return func(*args, **kwargs)
    locals:
      func = <local> <function all_reduce at 0x14e5bd5cc790>
      args = <local> (tensor([1], dtype=torch.int8),)
      kwargs = <local> {'op': <RedOpType.MIN: 3>}
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1707, in all_reduce
    line: work.wait()
    locals:
      work = <local> <torch.distributed.distributed_c10d.Work object at 0x14e5defc5a70>
      work.wait = <local> <bound method PyCapsule.wait of <torch.distributed.distributed_c10d.Work object at 0x14e5defc5a70>>
RuntimeError: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 1800000ms for recv operation to complete
EXCEPTION
Traceback (most recent call last):
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/rnn.py", line 11, in <module>
    line: main()
    locals:
      main = <local> <function main at 0x14845a7040d0>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/__main__.py", line 634, in main
    line: execute_main_task()
    locals:
      execute_main_task = <global> <function execute_main_task at 0x14845ad5bf40>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/__main__.py", line 440, in execute_main_task
    line: engine.train()
    locals:
      engine = <global> <returnn.torch.engine.Engine object at 0x14843a112680>
      engine.train = <global> <bound method Engine.train of <returnn.torch.engine.Engine object at 0x14843a112680>>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/torch/engine.py", line 225, in Engine.train
    line: self.train_epoch()
    locals:
      self = <local> <returnn.torch.engine.Engine object at 0x14843a112680>
      self.train_epoch = <local> <bound method Engine.train_epoch of <returnn.torch.engine.Engine object at 0x14843a112680>>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/torch/engine.py", line 278, in Engine.train_epoch
    line: torch.distributed.all_reduce(_has_data, op=torch.distributed.ReduceOp.MIN)
    locals:
      torch = <global> <module 'torch' from '/usr/local/lib/python3.10/dist-packages/torch/__init__.py'>
      torch.distributed = <global> <module 'torch.distributed' from '/usr/local/lib/python3.10/dist-packages/torch/distributed/__init__.py'>
      torch.distributed.all_reduce = <global> <function all_reduce at 0x1484389c8940>
      _has_data = <local> tensor([1], dtype=torch.int8)
      op = <not found>
      torch.distributed.ReduceOp = <global> <class 'torch.distributed.distributed_c10d.ReduceOp'>
      torch.distributed.ReduceOp.MIN = <global> <RedOpType.MIN: 3>
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1451, in broadcast_multigpu
    line: return func(*args, **kwargs)
    locals:
      func = <local> <function all_reduce at 0x1484389c88b0>
      args = <local> (tensor([1], dtype=torch.int8),)
      kwargs = <local> {'op': <RedOpType.MIN: 3>}
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1707, in all_reduce
    line: work.wait()
    locals:
      work = <local> <torch.distributed.distributed_c10d.Work object at 0x14826f4276b0>
      work.wait = <local> <bound method PyCapsule.wait of <torch.distributed.distributed_c10d.Work object at 0x14826f4276b0>>
RuntimeError: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] Timed out waiting 1800000ms for send operation to complete
XCEPTION
Traceback (most recent call last):
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/rnn.py", line 11, in <module>
    line: main()
    locals:
      main = <local> <function main at 0x14d67694beb0>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/__main__.py", line 634, in main
    line: execute_main_task()
    locals:
      execute_main_task = <global> <function execute_main_task at 0x14d67694bd90>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/__main__.py", line 440, in execute_main_task
    line: engine.train()
    locals:
      engine = <global> <returnn.torch.engine.Engine object at 0x14d67911c700>
      engine.train = <global> <bound method Engine.train of <returnn.torch.engine.Engine object at 0x14d67911c700>>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/torch/engine.py", line 225, in Engine.train
    line: self.train_epoch()
    locals:
      self = <local> <returnn.torch.engine.Engine object at 0x14d67911c700>
      self.train_epoch = <local> <bound method Engine.train_epoch of <returnn.torch.engine.Engine object at 0x14d67911c700>>
  File "/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/returnn/torch/engine.py", line 278, in Engine.train_epoch
    line: torch.distributed.all_reduce(_has_data, op=torch.distributed.ReduceOp.MIN)
    locals:
      torch = <global> <module 'torch' from '/usr/local/lib/python3.10/dist-packages/torch/__init__.py'>
      torch.distributed = <global> <module 'torch.distributed' from '/usr/local/lib/python3.10/dist-packages/torch/distributed/__init__.py'>
      torch.distributed.all_reduce = <global> <function all_reduce at 0x14d6545bc790>
      _has_data = <local> tensor([1], dtype=torch.int8)
      op = <not found>
      torch.distributed.ReduceOp = <global> <class 'torch.distributed.distributed_c10d.ReduceOp'>
      torch.distributed.ReduceOp.MIN = <global> <RedOpType.MIN: 3>
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1451, in broadcast_multigpu
    line: return func(*args, **kwargs)
    locals:
      func = <local> <function all_reduce at 0x14d6545bc700>
      args = <local> (tensor([1], dtype=torch.int8),)
      kwargs = <local> {'op': <RedOpType.MIN: 3>}
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1707, in all_reduce
    line: work.wait()
    locals:
      work = <local> <torch.distributed.distributed_c10d.Work object at 0x14d490130870>
      work.wait = <local> <bound method PyCapsule.wait of <torch.distributed.distributed_c10d.Work object at 0x14d490130870>>
RuntimeError: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 1800000ms for recv operation to complete
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 11641 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 11642) of binary: /usr/bin/python3
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/nas/models/asr/pzheng/returnn_pytorch_backend/work/i6_core/tools/git/CloneGitRepositoryJob.LClNkVyrEDKK/output/repository/rnn.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2023-11-05_06:21:10
  host      : gpu-12.apptek.local
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 11643)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2023-11-05_06:21:10
  host      : gpu-12.apptek.local
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 11644)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-11-05_06:21:10
  host      : gpu-12.apptek.local
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 11642)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

```

_Originally posted by @kuacakuaca in https://github.com/rwth-i6/returnn/issues/1452#issuecomment-1798098777_
            


Discussion:
@kuacakuaca can you give a bit more details? Can you post the full log? What are the `torch_distributed` settings? How many GPUs? 
Does this happen right at the beginning? Or only after some training? After second epoch?
 (Ah you said after one sub-epoch in the title.) So right at the beginning of the next subepoch. Or did the first sub-epoch never finish?

Is it possible that some worker died meanwhile? 1800000ms, i.e. 1800 secs, i.e. 30min, that's a quite long timeout, so I don't think this is the problem, all workers should be ready and have responded, if they are still alive. But maybe they wait at some other place and we have some deadlock? It would be very helpful if you could login to the node, and during that wait time of 30min, where it waits, check all the workers, and post the stacktrace of each (e.g. use `py-spy` to dump the stacktrace of each worker).

@Judyxujj can you help? Do you have an idea?


the allreduce op at https://github.com/rwth-i6/returnn/blob/76ab73c2480c59d96d28634f963556d5abd736e9/returnn/torch/engine.py#L285 and the data loader re-initialization at https://github.com/rwth-i6/returnn/blob/76ab73c2480c59d96d28634f963556d5abd736e9/returnn/torch/engine.py#L266 tracing back to 
https://github.com/pytorch/pytorch/blob/1d56e7b5af5bd27a1788f33536d23b60d966113e/torch/utils/data/dataloader.py#L384 and https://github.com/pytorch/pytorch/blob/1d56e7b5af5bd27a1788f33536d23b60d966113e/torch/utils/data/dataloader.py#L571
caused the deadlock. The `dist.new_group(backend="gloo")` call for the dev dataset of the rank 0 process was synchronized with the `dist.new_group(backend="gloo")` calls for the next training dataset of the other training processes.
Very nice that you found that. I'm curious, how did you debug this?

But also, this means, multi GPU training is broken currently? Or any workaround? But I thought @Judyxujj and maybe other people are using it? How can that be?

Any suggestion on how to fix this?

Btw, do you use `torch_dataloader_opts = {"num_workers": 1}`? Or what do you set there? If you don't set this, can you try with that?

I used py-spy as you suggested and speedscope to visualize the processes. As far as I know, @Judyxujj used an earlier returnn version from July or August for her multi-GPU training. Was there any change to data loading since then? I didn't set `torch_dataloader_opts`, but I can try that. 
Did you speak with @Judyxujj? Did you compare your config to her? Are there maybe other differences? What PyTorch version does she use?

If you think all other things are the same (PyTorch version, also HDF dataset, same dataloader opts, etc) or should not matter, so only the RETURNN version is different, then can you check the RETURNN version which was used by @Judyxujj, and if that works, can you git bisect to find the commit where it breaks?

it's the switch from `torchdata.dataloader2.DataLoader2` to `torch.utils.data.DataLoader` broke the multi-gpu training. If I'm not wrong, before the dataloader was just initialized once, but now every epoch.
Ah ok, good to know!

Did you try `torch_dataloader_opts = {"num_workers": 1}`?

Do you have a suggestion on how to fix this? Or @Judyxujj?

I still don't exactly understand the deadlock though. How can it be that some instances are in `torch.distributed.all_reduce(_has_data, op=torch.distributed.ReduceOp.MIN)` while others already create the new data loader iterator?

If all instances would create the data loader iterator correctly synchronized, there should be no problem, right?

@albertz If I do not understand it wrong, the deadlock happens because we only do evaluation when rank == 0, but `the dist.new_group(backend="gloo") `call for the dev dataset will wait for all the other ranks. Maybe just doing evaluations on all ranks would solve the problem? I need to test that
Ah I see. But then it should be enough if we just create the data loader iterators on all ranks. So, call `self.eval_model()` in all cases, but in there, have this logic:
```python
...

for dataset_name, dataset in self.eval_datasets.items():
    if skip_already_evaluated and self._is_dataset_evaluated(name=dataset_name):
        continue

    data_loader = self._eval_dataloaders[dataset_name]

    if self._use_torch_distributed and torch.distributed.get_rank() != 0:
        # We need to make sure the data loader iterator was created for proper synchronization.
        # However, now we only want to do evaluation on rank 0 for simplicity.
        iter(data_loader)
        continue

   print(f"Evaluating dataset {dataset_name!r}", file=log.v3)

   ...

if not self._use_torch_distributed or torch.distributed.get_rank() == 0:
    print(" ".join(eval_dump_str) if eval_dump_str else "(No evaluations.)", file=log.v1)
```
> Ah ok, good to know!
> 
> Did you try `torch_dataloader_opts = {"num_workers": 1}`?
> 
> Do you have a suggestion on how to fix this? Or @Judyxujj?
> 
> I still don't exactly understand the deadlock though. How can it be that some instances are in `torch.distributed.all_reduce(_has_data, op=torch.distributed.ReduceOp.MIN)` while others already create the new data loader iterator?
> 
> If all instances would create the data loader iterator correctly synchronized, there should be no problem, right?

the problem is that data loader iterator initialization requires synchronization of all instances (each instance should run into `dist.new_group(backend="gloo")` and wait for others there. Sadly next epoch training data loader iterator initialization of those which didn't need to run evaluations waited and synchronized with the eval data loader initialization of the rank 0 instance. As a consequence, the training data loader initialization of the rank 0 instance can't be synchronized anymore, while the other instances have been waiting for the `all_reduce` op. I think your code should solve this problem, I can test that. But I will try `torch_dataloader_opts = {"num_workers": 1}` first.
Even if `torch_dataloader_opts` also works, I think we should anyway use the code, to have that fixed in any case.
> Even if `torch_dataloader_opts` also works, I think we should anyway use the code, to have that fixed in any case.

setting `torch_dataloader_opts` alone didn't solve the hanging problem as `eval_model` still asked for the other processes. But now it worked with the fix you proposed, in both cases where `torch_dataloaser_opts` was set or not . 
Good to know. Can you open a PR with those changes?

Resolving patch:
diff --git a/returnn/torch/engine.py b/returnn/torch/engine.py
--- a/returnn/torch/engine.py
+++ b/returnn/torch/engine.py
@@ -11,7 +11,7 @@
 import os
 import torch
 import time
-from torch.distributed import init_process_group
+import torch.distributed
 from torch.nn.parallel import DistributedDataParallel
 from torch.utils.data import DataLoader
 from torch import autocast
@@ -396,8 +396,7 @@ def train_epoch(self):
             else:
                 print("Not saving model, `model` not specified.", file=log.v3)
 
-        if not self._use_torch_distributed or torch.distributed.get_rank() == 0:
-            self.eval_model()
+        self.eval_model()
         if self.config.bool_or_other("cleanup_old_models", None):
             self.cleanup_old_models()
 
@@ -422,10 +421,16 @@ def eval_model(self, *, skip_already_evaluated: bool = False):
         for dataset_name, dataset in self.eval_datasets.items():
             if skip_already_evaluated and self._is_dataset_evaluated(name=dataset_name):
                 continue
-            print(f"Evaluating dataset {dataset_name!r}", file=log.v3)
 
             data_loader = self._eval_dataloaders[dataset_name]
 
+            if self._use_torch_distributed and torch.distributed.get_rank() != 0:
+                # We need to make sure the data loader iterator was created for proper synchronization.
+                # However, now we only want to do evaluation on rank 0 for simplicity.
+                iter(data_loader)
+                continue
+            print(f"Evaluating dataset {dataset_name!r}", file=log.v3)
+
             accumulated_losses_dict = NumbersDict()
             accumulated_inv_norm_factors_dict = NumbersDict()
             step_idx = 0
@@ -484,7 +489,8 @@ def eval_model(self, *, skip_already_evaluated: bool = False):
                 )
             ]
 
-        print(" ".join(eval_dump_str) if eval_dump_str else "(No evaluations.)", file=log.v1)
+        if not self._use_torch_distributed or torch.distributed.get_rank() == 0:
+            print(" ".join(eval_dump_str) if eval_dump_str else "(No evaluations.)", file=log.v1)
 
         self._maybe_report_dev_memory_stats()
 


Test patch: