Rendevous timeout when using torchrun

username: sofia22
cooked: <p>Hello,<br>
I am new to training on multiple GPUs, and I have looked at the relevant documentation and forums, but I am still struggling with the following issue:</p>
<p>I am trying to use torchrun in the hopes of speeding up the training process. I created a .sh (<code>distributed.sh</code>) file that looks like this:</p>
<pre><code class="lang-auto">export CUDA_VISIBLE_DEVICES=0,1

NPROC_PER_NODE=1
export MASTER_PORT=29502

# Set a smaller batch size for debugging
per_gpu_bs=4
acc_step=1
bs=$per_gpu_bs

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &amp;&amp; pwd )"
# Get the root directory of the project (two levels up from script)
PROJECT_ROOT="$( cd "$SCRIPT_DIR/../.." &amp;&amp; pwd )"
export PYTHONPATH="$PROJECT_ROOT:$PYTHONPATH"

echo "PER_DEVICE_TRAIN_BATCH_SIZE="$bs


# unfortunately, rdvx-timeout does not exist

torchrun --standalone --nproc_per_node=$NPROC_PER_NODE --nnodes=2  \
    --master_port=29502 \
    --rdzv-backend=c10d \
    -m vila_u.train.train_mem_rc \
    [train_mem_rc args...]
</code></pre>
<p>I then run <code>./distributed.sh</code>. After a while, I recieve the following traceback:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/conda/envs/new_pipe/bin/torchrun", line 8, in &lt;module&gt;
    sys.exit(main())
             ^^^^^^
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 260, in launch_agent
    result = agent.run()
             ^^^^^^^^^^^
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 137, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 696, in run
    result = self._invoke_run(role)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 849, in _invoke_run
    self._initialize_workers(self._worker_group)
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 137, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 668, in _initialize_workers
    self._rendezvous(worker_group)
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 137, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 500, in _rendezvous
    rdzv_info = spec.rdzv_handler.next_rendezvous()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1157, in next_rendezvous
    self._op_executor.run(join_op, deadline, self._get_deadline)
  File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 679, in run
    raise RendezvousTimeoutError
torch.distributed.elastic.rendezvous.api.RendezvousTimeoutError
</code></pre>
<p>Furthermore, when I run <code>nvidia-smi</code>, I see there are no processes running on my server. In case it may be helpful, the output is:<br>
±--------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |<br>
|-----------------------------------------±---------------------±---------------------+<br>
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                      |               MIG M. |<br>
|=========================================+======================+======================|<br>
|   0  NVIDIA H100 80GB HBM3          Off | 00000000:1A:00.0 Off |                    0 |<br>
| N/A   31C    P0              69W / 700W |      0MiB / 81559MiB |      0%      Default |<br>
|                                         |                      |             Disabled |<br>
±----------------------------------------±---------------------±---------------------+<br>
|   1  NVIDIA H100 80GB HBM3          Off | 00000000:40:00.0 Off |                    0 |<br>
| N/A   28C    P0              70W / 700W |      0MiB / 81559MiB |      0%      Default |<br>
|                                         |                      |             Disabled |<br>
±----------------------------------------±---------------------±---------------------+<br>
|   2  NVIDIA H100 80GB HBM3          Off | 00000000:53:00.0 Off |                    0 |<br>
| N/A   28C    P0              72W / 700W |      0MiB / 81559MiB |      0%      Default |<br>
|                                         |                      |             Disabled |<br>
±----------------------------------------±---------------------±---------------------+<br>
|   3  NVIDIA H100 80GB HBM3          Off | 00000000:66:00.0 Off |                    0 |<br>
| N/A   31C    P0              70W / 700W |      0MiB / 81559MiB |      0%      Default |<br>
|                                         |                      |             Disabled |<br>
±----------------------------------------±---------------------±---------------------+<br>
|   4  NVIDIA H100 80GB HBM3          Off | 00000000:9C:00.0 Off |                    0 |<br>
| N/A   32C    P0              69W / 700W |      0MiB / 81559MiB |      0%      Default |<br>
|                                         |                      |             Disabled |<br>
±----------------------------------------±---------------------±---------------------+<br>
|   5  NVIDIA H100 80GB HBM3          Off | 00000000:C0:00.0 Off |                    0 |<br>
| N/A   27C    P0              69W / 700W |      0MiB / 81559MiB |      0%      Default |<br>
|                                         |                      |             Disabled |<br>
±----------------------------------------±---------------------±---------------------+<br>
|   6  NVIDIA H100 80GB HBM3          Off | 00000000:D2:00.0 Off |                    0 |<br>
| N/A   33C    P0              71W / 700W |      0MiB / 81559MiB |      0%      Default |<br>
|                                         |                      |             Disabled |<br>
±----------------------------------------±---------------------±---------------------+<br>
|   7  NVIDIA H100 80GB HBM3          Off | 00000000:E4:00.0 Off |                    0 |<br>
| N/A   27C    P0              72W / 700W |      0MiB / 81559MiB |      0%      Default |<br>
|                                         |                      |             Disabled |<br>
±----------------------------------------±---------------------±---------------------+</p>
<p>±--------------------------------------------------------------------------------------+<br>
| Processes:                                                                            |<br>
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |<br>
|        ID   ID                                                             Usage      |<br>
|=======================================================================================|<br>
|  No running processes found                                                           |<br>
±--------------------------------------------------------------------------------------+</p>
<p>If I set export <code>CUDA_VISIBLE_DEVICES=0, NPROC_PER_NODE=1, and nnodes=1</code>, then the script runs, but this defeats the purpose of using torchrun.</p>
<p>If I try doing `CUDA_VISIBLE_DEVICES=1 (or CUDA_VISIBLE_DEVICES=0), NPROC_PER_NODE=2, nnodes=1’, then I get:</p>
<pre><code class="lang-auto">[rank1]: Traceback (most recent call last):
[rank1]:   File "&lt;frozen runpy&gt;", line 198, in _run_module_as_main
[rank1]:   File "&lt;frozen runpy&gt;", line 88, in _run_code
[rank1]:   File "/train/train_mem_rc.py", line 19, in &lt;module&gt;
[rank1]:     train()
[rank1]:   File "/train/train_rc.py", line 307, in train
[rank1]:     trainer.train(resume_from_checkpoint=resume_from_checkpoint)
[rank1]:   File "/conda/envs/new_pipe/lib/python3.12/site-packages/transformers/trainer.py", line 2247, in train
[rank1]:     return inner_training_loop(
[rank1]:            ^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/conda/envs/new_pipe/lib/python3.12/site-packages/transformers/trainer.py", line 2376, in _inner_training_loop
[rank1]:     model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
[rank1]:                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/conda/envs/new_pipe/lib/python3.12/site-packages/accelerate/accelerator.py", line 1318, in prepare
[rank1]:     result = self._prepare_deepspeed(*args)
[rank1]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/conda/envs/new_pipe/lib/python3.12/site-packages/accelerate/accelerator.py", line 1815, in _prepare_deepspeed
[rank1]:     engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
[rank1]:                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/conda/envs/new_pipe/lib/python3.12/site-packages/deepspeed/__init__.py", line 193, in initialize
[rank1]:     engine = DeepSpeedEngine(args=args,
[rank1]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/conda/envs/new_pipe/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 264, in __init__
[rank1]:     self._set_distributed_vars(args)
[rank1]:   File "/conda/envs/new_pipe/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 1121, in _set_distributed_vars
[rank1]:     get_accelerator().set_device(device_rank)
[rank1]:   File "/conda/envs/new_pipe/lib/python3.12/site-packages/deepspeed/accelerator/cuda_accelerator.py", line 67, in set_device
[rank1]:     torch.cuda.set_device(device_index)
[rank1]:   File "/conda/envs/new_pipe/lib/python3.12/site-packages/torch/cuda/__init__.py", line 478, in set_device
[rank1]:     torch._C._cuda_setDevice(device)
[rank1]: RuntimeError: CUDA error: invalid device ordinal
[rank1]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank1]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank1]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p>I am not sure what to do. Have I done something wrong or is it that my server is just running too slow?</p>
<p>Thank you very much advance for your help!</p>