Multi gpu training on server

username: Marcelo_Sena
cooked: <p>Hi, I am trying to get some minimal multiple gpu training to work. I have managed to successully get it running with Pytorch 1.x, but not on Pytorch 2.x. Below is the GPU information and the logs I get when running it on Pytorch 2.</p>
<p>The file I run is taken from the Pytorch tutorial:<br>
<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a></p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
import torch.nn as nn
import torch.optim as optim

from torch.nn.parallel import DistributedDataParallel as DDP

class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 10)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(10, 5)

    def forward(self, x):
        return self.net2(self.relu(self.net1(x)))


def demo_basic():
    torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
    dist.init_process_group("nccl")
    rank = dist.get_rank()
    print(f"Start running basic DDP example on rank {rank}.")
    # create model and move it to GPU with id rank
    device_id = rank % torch.cuda.device_count()
    model = ToyModel().to(device_id)
    ddp_model = DDP(model, device_ids=[device_id])
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(device_id)
    loss_fn(outputs, labels).backward()
    optimizer.step()
    dist.destroy_process_group()
    print(f"Finished running basic DDP example on rank {rank}.")

if __name__ == "__main__":
    demo_basic()
</code></pre>
<p>torchrun --nnodes=1 --nproc_per_node=2 --standalone elastic_ddp.py</p>
<p>±----------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |<br>
|-----------------------------------------±-----------------------±---------------------+<br>
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                        |               MIG M. |<br>
|=========================================+========================+======================|<br>
|   0  NVIDIA GeForce RTX 2080 Ti     On  |   00000000:05:00.0 Off |                  N/A |<br>
|  0%   26C    P8             22W /  252W |       1MiB /  11264MiB |      0%   E. Process |<br>
|                                         |                        |                  N/A |<br>
±----------------------------------------±-----------------------±---------------------+<br>
|   1  NVIDIA GeForce RTX 2080 Ti     On  |   00000000:44:00.0 Off |                  N/A |<br>
|  0%   28C    P8             17W /  252W |       1MiB /  11264MiB |      0%   E. Process |<br>
|                                         |                        |                  N/A |<br>
±----------------------------------------±-----------------------±---------------------+</p>
<p>master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.<br>
[W socket.cpp:426] [c10d] The server socket cannot be initialized on [::]:29400 (errno: 97 - Address family not supported by protocol).<br>
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost]:29400 (errno: 97 - Address family not supported by protocol).<br>
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost]:29400 (errno: 97 - Address family not supported by protocol).<br>
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost]:29400 (errno: 97 - Address family not supported by protocol).<br>
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [localhost]:29400 (errno: 97 - Address family not supported by protocol).<br>
[W socket.cpp:426] [c10d] The server socket cannot be initialized on [::]:37139 (errno: 97 - Address family not supported by protocol).<br>
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [sh03-12n12.int]:37139 (errno: 97 - Address family not supported by protocol).<br>
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [sh03-12n12.int]:37139 (errno: 97 - Address family not supported by protocol).<br>
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [sh03-12n12.int]:37139 (errno: 97 - Address family not supported by protocol).<br>
[W socket.cpp:601] [c10d] The client socket cannot be initialized to connect to [sh03-12n12.int]:37139 (errno: 97 - Address family not supported by protocol).<br>
Start running basic DDP example on rank 0.<br>
Start running basic DDP example on rank 1.<br>
sh03-12n12:1768:1768 [0] enqueue.cc:100 NCCL WARN Cuda failure ‘invalid device function’<br>
sh03-12n12:1768:1768 [0] NCCL INFO Bootstrap : Using ib0:10.51.12.12&lt;0&gt;<br>
sh03-12n12:1768:1768 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
sh03-12n12:1769:1769 [1] NCCL INFO cudaDriverVersion 12040<br>
sh03-12n12:1768:1768 [0] NCCL INFO cudaDriverVersion 12040<br>
NCCL version 2.14.3+cuda11.5<br>
sh03-12n12:1769:1769 [1] enqueue.cc:100 NCCL WARN Cuda failure ‘invalid device function’<br>
sh03-12n12:1769:1769 [1] NCCL INFO Bootstrap : Using ib0:10.51.12.12&lt;0&gt;<br>
sh03-12n12:1769:1769 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
sh03-12n12:1769:1807 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/RoCE [RO]; OOB ib0:10.51.12.12&lt;0&gt;<br>
sh03-12n12:1769:1807 [1] NCCL INFO Using network IB<br>
sh03-12n12:1768:1808 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/RoCE [RO]; OOB ib0:10.51.12.12&lt;0&gt;<br>
sh03-12n12:1768:1808 [0] NCCL INFO Using network IB<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
time      : 2025-03-20_11:51:07<br>
sh03-12n12:1768:1808 [0] NCCL INFO Could not enable P2P between dev 1(=44000) and dev 0(=5000)<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1768:1808 [0] NCCL INFO Could not enable P2P between dev 0(=5000) and dev 1(=44000)<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1768:1808 [0] NCCL INFO Could not enable P2P between dev 1(=44000) and dev 0(=5000)<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1768:1808 [0] NCCL INFO Could not enable P2P between dev 0(=5000) and dev 1(=44000)<br>
sh03-12n12:1768:1808 [0] NCCL INFO Setting affinity for GPU 0 to 010000<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO Could not enable P2P between dev 1(=44000) and dev 0(=5000)<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO Could not enable P2P between dev 0(=5000) and dev 1(=44000)<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO Could not enable P2P between dev 1(=44000) and dev 0(=5000)<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO Could not enable P2P between dev 0(=5000) and dev 1(=44000)<br>
sh03-12n12:1769:1807 [1] NCCL INFO Setting affinity for GPU 1 to 010000<br>
sh03-12n12:1769:1807 [1] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0 [1] -1/-1/-1-&gt;1-&gt;0<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO Could not enable P2P between dev 1(=44000) and dev 0(=5000)<br>
sh03-12n12:1768:1808 [0] NCCL INFO Channel 00/02 :    0   1<br>
sh03-12n12:1768:1808 [0] NCCL INFO Channel 01/02 :    0   1<br>
sh03-12n12:1768:1808 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] 1/-1/-1-&gt;0-&gt;-1<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1768:1808 [0] NCCL INFO Could not enable P2P between dev 0(=5000) and dev 1(=44000)<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO Could not enable P2P between dev 1(=44000) and dev 0(=5000)<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1768:1808 [0] NCCL INFO Could not enable P2P between dev 0(=5000) and dev 1(=44000)<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO Could not enable P2P between dev 1(=44000) and dev 0(=5000)<br>
sh03-12n12:1769:1807 [1] NCCL INFO Channel 00 : 1[44000] → 0[5000] via SHM/direct/direct<br>
sh03-12n12:1769:1807 [1] NCCL INFO P2P is disabled between connected GPUs 1 and 0. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1769:1807 [1] NCCL INFO Could not enable P2P between dev 1(=44000) and dev 0(=5000)<br>
sh03-12n12:1769:1807 [1] NCCL INFO Channel 01 : 1[44000] → 0[5000] via SHM/direct/direct<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1768:1808 [0] NCCL INFO Could not enable P2P between dev 0(=5000) and dev 1(=44000)<br>
sh03-12n12:1768:1808 [0] NCCL INFO Channel 00 : 0[5000] → 1[44000] via SHM/direct/direct<br>
sh03-12n12:1768:1808 [0] NCCL INFO P2P is disabled between connected GPUs 0 and 1. You can repress this message with NCCL_IGNORE_DISABLED_P2P=1.<br>
sh03-12n12:1768:1808 [0] NCCL INFO Could not enable P2P between dev 0(=5000) and dev 1(=44000)<br>
sh03-12n12:1768:1808 [0] NCCL INFO Channel 01 : 0[5000] → 1[44000] via SHM/direct/direct<br>
sh03-12n12:1769:1807 [1] NCCL INFO Connected all rings<br>
sh03-12n12:1769:1807 [1] NCCL INFO Connected all trees<br>
sh03-12n12:1769:1807 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512<br>
sh03-12n12:1769:1807 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
sh03-12n12:1768:1808 [0] NCCL INFO Connected all rings<br>
sh03-12n12:1768:1808 [0] NCCL INFO Connected all trees<br>
sh03-12n12:1768:1808 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512<br>
sh03-12n12:1768:1808 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
sh03-12n12:1768:1808 [0] NCCL INFO comm 0x53da9150 rank 0 nranks 2 cudaDev 0 busId 5000 - Init COMPLETE<br>
sh03-12n12:1769:1807 [1] NCCL INFO comm 0x54abacb0 rank 1 nranks 2 cudaDev 1 busId 44000 - Init COMPLETE</p>
<p>sh03-12n12:1768:1768 [0] misc/strongstream.cc:166 NCCL WARN Cuda failure ‘invalid device function’<br>
sh03-12n12:1768:1768 [0] NCCL INFO enqueue.cc:992 → 1<br>
sh03-12n12:1768:1768 [0] NCCL INFO group.cc:173 → 1<br>
sh03-12n12:1768:1768 [0] NCCL INFO group.cc:340 → 1<br>
sh03-12n12:1768:1768 [0] NCCL INFO group.cc:421 → 1<br>
sh03-12n12:1768:1768 [0] NCCL INFO group.cc:106 → 1<br>
Traceback (most recent call last):<br>
File “/home/users/msena/deqn_env/mgpu_training/distributed_tutorial.py”, line 41, in </p>
<h1><a name="p-467691-sh03-12n1217691769-1-miscstrongstreamcc166-nccl-warn-cuda-failure-invalid-device-function-sh03-12n1217691769-1-nccl-info-enqueuecc992-1-sh03-12n1217691769-1-nccl-info-groupcc173-1-sh03-12n1217691769-1-nccl-info-groupcc340-1-sh03-12n1217691769-1-nccl-info-groupcc421-1-sh03-12n1217691769-1-nccl-info-groupcc106-1-traceback-most-recent-call-last-file-homeusersmsenadeqn_envmgpu_trainingdistributed_tutorialpy-line-41-in-module-demo_basic-file-homeusersmsenadeqn_envmgpu_trainingdistributed_tutorialpy-line-28-in-demo_basic-demo_basic-file-homeusersmsenadeqn_envmgpu_trainingdistributed_tutorialpy-line-28-in-demo_basic-ddp_model-ddpmodel-device_idsdevice_id-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchnnparalleldistributedpy-line-674-in-__init__-ddp_model-ddpmodel-device_idsdevice_id-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchnnparalleldistributedpy-line-674-in-__init__-_verify_param_shape_across_processesselfprocess_group-parameters-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedutilspy-line-118-in-_verify_param_shape_across_processes-_verify_param_shape_across_processesselfprocess_group-parameters-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedutilspy-line-118-in-_verify_param_shape_across_processes-return-dist_verify_params_across_processesprocess_group-tensors-logger-runtimeerror-nccl-error-1-unhandled-cuda-error-return-dist_verify_params_across_processesprocess_group-tensors-logger-runtimeerror-nccl-error-1-unhandled-cuda-error-sh03-12n1217681813-0-nccl-info-service-thread-connection-closed-by-localrank-0-sh03-12n1217681768-0-nccl-info-comm-0x53da9150-rank-0-nranks-2-cudadev-0-busid-5000-abort-complete-sh03-12n1217691814-1-nccl-info-service-thread-connection-closed-by-localrank-1-sh03-12n1217691769-1-nccl-info-comm-0x54abacb0-rank-1-nranks-2-cudadev-1-busid-44000-abort-complete-errortorchdistributedelasticmultiprocessingapifailed-exitcode-1-local_rank-0-pid-1768-of-binary-sharesoftwareuseropenpython390binpython39-traceback-most-recent-call-last-file-sharesoftwareuseropenpy-pytorch200_py39bintorchrun-line-33-in-module-sysexitload_entry_pointtorch200gitunknown-console_scripts-torchrun-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-346-in-wrapper-return-fargs-kwargs-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedrunpy-line-794-in-main-runargs-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedrunpy-line-785-in-run-elastic_launch-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedlauncherapipy-line-134-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedlauncherapipy-line-250-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1" class="anchor" href="#p-467691-sh03-12n1217691769-1-miscstrongstreamcc166-nccl-warn-cuda-failure-invalid-device-function-sh03-12n1217691769-1-nccl-info-enqueuecc992-1-sh03-12n1217691769-1-nccl-info-groupcc173-1-sh03-12n1217691769-1-nccl-info-groupcc340-1-sh03-12n1217691769-1-nccl-info-groupcc421-1-sh03-12n1217691769-1-nccl-info-groupcc106-1-traceback-most-recent-call-last-file-homeusersmsenadeqn_envmgpu_trainingdistributed_tutorialpy-line-41-in-module-demo_basic-file-homeusersmsenadeqn_envmgpu_trainingdistributed_tutorialpy-line-28-in-demo_basic-demo_basic-file-homeusersmsenadeqn_envmgpu_trainingdistributed_tutorialpy-line-28-in-demo_basic-ddp_model-ddpmodel-device_idsdevice_id-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchnnparalleldistributedpy-line-674-in-__init__-ddp_model-ddpmodel-device_idsdevice_id-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchnnparalleldistributedpy-line-674-in-__init__-_verify_param_shape_across_processesselfprocess_group-parameters-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedutilspy-line-118-in-_verify_param_shape_across_processes-_verify_param_shape_across_processesselfprocess_group-parameters-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedutilspy-line-118-in-_verify_param_shape_across_processes-return-dist_verify_params_across_processesprocess_group-tensors-logger-runtimeerror-nccl-error-1-unhandled-cuda-error-return-dist_verify_params_across_processesprocess_group-tensors-logger-runtimeerror-nccl-error-1-unhandled-cuda-error-sh03-12n1217681813-0-nccl-info-service-thread-connection-closed-by-localrank-0-sh03-12n1217681768-0-nccl-info-comm-0x53da9150-rank-0-nranks-2-cudadev-0-busid-5000-abort-complete-sh03-12n1217691814-1-nccl-info-service-thread-connection-closed-by-localrank-1-sh03-12n1217691769-1-nccl-info-comm-0x54abacb0-rank-1-nranks-2-cudadev-1-busid-44000-abort-complete-errortorchdistributedelasticmultiprocessingapifailed-exitcode-1-local_rank-0-pid-1768-of-binary-sharesoftwareuseropenpython390binpython39-traceback-most-recent-call-last-file-sharesoftwareuseropenpy-pytorch200_py39bintorchrun-line-33-in-module-sysexitload_entry_pointtorch200gitunknown-console_scripts-torchrun-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-346-in-wrapper-return-fargs-kwargs-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedrunpy-line-794-in-main-runargs-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedrunpy-line-785-in-run-elastic_launch-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedlauncherapipy-line-134-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-sharesoftwareuseropenpy-pytorch200_py39libpython39site-packagestorchdistributedlauncherapipy-line-250-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1"></a>sh03-12n12:1769:1769 [1] misc/strongstream.cc:166 NCCL WARN Cuda failure ‘invalid device function’<br>
sh03-12n12:1769:1769 [1] NCCL INFO enqueue.cc:992 → 1<br>
sh03-12n12:1769:1769 [1] NCCL INFO group.cc:173 → 1<br>
sh03-12n12:1769:1769 [1] NCCL INFO group.cc:340 → 1<br>
sh03-12n12:1769:1769 [1] NCCL INFO group.cc:421 → 1<br>
sh03-12n12:1769:1769 [1] NCCL INFO group.cc:106 → 1<br>
Traceback (most recent call last):<br>
File “/home/users/msena/deqn_env/mgpu_training/distributed_tutorial.py”, line 41, in <br>
demo_basic()<br>
File “/home/users/msena/deqn_env/mgpu_training/distributed_tutorial.py”, line 28, in demo_basic<br>
demo_basic()<br>
File “/home/users/msena/deqn_env/mgpu_training/distributed_tutorial.py”, line 28, in demo_basic<br>
ddp_model = DDP(model, device_ids=[device_id])<br>
File “/share/software/user/open/py-pytorch/2.0.0_py39/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 674, in <strong>init</strong><br>
ddp_model = DDP(model, device_ids=[device_id])<br>
File “/share/software/user/open/py-pytorch/2.0.0_py39/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 674, in <strong>init</strong><br>
_verify_param_shape_across_processes(self.process_group, parameters)<br>
File “/share/software/user/open/py-pytorch/2.0.0_py39/lib/python3.9/site-packages/torch/distributed/utils.py”, line 118, in _verify_param_shape_across_processes<br>
_verify_param_shape_across_processes(self.process_group, parameters)<br>
File “/share/software/user/open/py-pytorch/2.0.0_py39/lib/python3.9/site-packages/torch/distributed/utils.py”, line 118, in _verify_param_shape_across_processes<br>
return dist._verify_params_across_processes(process_group, tensors, logger)<br>
RuntimeError: NCCL Error 1: unhandled cuda error<br>
return dist._verify_params_across_processes(process_group, tensors, logger)<br>
RuntimeError: NCCL Error 1: unhandled cuda error<br>
sh03-12n12:1768:1813 [0] NCCL INFO [Service thread] Connection closed by localRank 0<br>
sh03-12n12:1768:1768 [0] NCCL INFO comm 0x53da9150 rank 0 nranks 2 cudaDev 0 busId 5000 - Abort COMPLETE<br>
sh03-12n12:1769:1814 [1] NCCL INFO [Service thread] Connection closed by localRank 1<br>
sh03-12n12:1769:1769 [1] NCCL INFO comm 0x54abacb0 rank 1 nranks 2 cudaDev 1 busId 44000 - Abort COMPLETE<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1768) of binary: /share/software/user/open/python/3.9.0/bin/python3.9<br>
Traceback (most recent call last):<br>
File “/share/software/user/open/py-pytorch/2.0.0_py39/bin/torchrun”, line 33, in <br>
sys.exit(load_entry_point(‘torch==2.0.0+gitunknown’, ‘console_scripts’, ‘torchrun’)())<br>
File “/share/software/user/open/py-pytorch/2.0.0_py39/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 346, in wrapper<br>
return f(*args, **kwargs)<br>
File “/share/software/user/open/py-pytorch/2.0.0_py39/lib/python3.9/site-packages/torch/distributed/run.py”, line 794, in main<br>
run(args)<br>
File “/share/software/user/open/py-pytorch/2.0.0_py39/lib/python3.9/site-packages/torch/distributed/run.py”, line 785, in run<br>
elastic_launch(<br>
File “/share/software/user/open/py-pytorch/2.0.0_py39/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 134, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/share/software/user/open/py-pytorch/2.0.0_py39/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 250, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2><a name="p-467691-distributed_tutorialpy-failed-2" class="anchor" href="#p-467691-distributed_tutorialpy-failed-2"></a>distributed_tutorial.py FAILED</h2>
<h2><a name="p-467691-failures-1-time-2025-03-20_115107-host-sh03-12n12int-rank-1-local_rank-1-exitcode-1-pid-1769-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-3" class="anchor" href="#p-467691-failures-1-time-2025-03-20_115107-host-sh03-12n12int-rank-1-local_rank-1-exitcode-1-pid-1769-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-3"></a>Failures:<br>
[1]:<br>
time      : 2025-03-20_11:51:07<br>
host      : sh03-12n12.int<br>
rank      : 1 (local_rank: 1)<br>
exitcode  : 1 (pid: 1769)<br>
error_file: &lt;N/A&gt;<br>
traceback : To enable traceback see: <a href="https://pytorch.org/docs/stable/elastic/errors.html" class="inline-onebox" rel="noopener nofollow ugc">Error Propagation — PyTorch 2.6 documentation</a></h2>
<p>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2025-03-20_11:51:07<br>
host      : sh03-12n12.int<br>
rank      : 0 (local_rank: 0)<br>
exitcode  : 1 (pid: 1768)<br>
error_file: &lt;N/A&gt;</p>
<p>Here are the modules I currenly load:<br>
module load cuda/12.4.0<br>
module load python/3.9.0<br>
module load py-pytorch/2.0.0_py39</p>
<p>I suspect the error is caused by some incorrect version of software loading? Any help is much appreciated!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Marcelo_Sena" data-post="1" data-topic="218088">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marcelo_sena/48/76267_2.png" class="avatar"> Marcelo_Sena:</div>
<blockquote>
<p>module load py-pytorch/2.0.0_py39</p>
</blockquote>
</aside>
<p>Are you seeing these issues with any newer PyTorch release as 2.0.0 is already quite old?</p>

username: Marcelo_Sena
cooked: <p>When i run it on pytorch 2.2.1 with python 3.12 i get the following error</p>
<p>torchrun --nnodes=1 --nproc_per_node=2 --standalone distributed_training.py<br>
[W socket.cpp:464] [c10d] The server socket cannot be initialized on [::]:0 (errno: 97 - Address family not supported by protocol).<br>
[W socket.cpp:697] [c10d] The client socket cannot be initialized to connect to [localhost]:38726 (errno: 97 - Address family not supported<br>
by protocol).<br>
[W socket.cpp:697] [c10d] The client socket cannot be initialized to connect to [localhost]:38726 (errno: 97 - Address family not supported<br>
by protocol).<br>
Fatal Python error: Segmentation fault<br>
Current thread 0x00007f857c2d9740 (most recent call first):<br>
File “/share/software/user/open/py-pytorch/2.2.1_py312/lib/python3.12/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_<br>
backend.py”, line 113 in <em>call_store<br>
File "/share/software/user/open/py-pytorch/2.2.1_py312/lib/python3.12/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous</em><br>
backend.py", line 64 in <strong>init</strong><br>
File “/share/software/user/open/py-pytorch/2.2.1_py312/lib/python3.12/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_<br>
backend.py”, line 253 in create_backend<br>
File “/share/software/user/open/py-pytorch/2.2.1_py312/lib/python3.12/site-packages/torch/distributed/elastic/rendezvous/registry.py”, li<br>
ne 36 in _create_c10d_handler<br>
File “/share/software/user/open/py-pytorch/2.2.1_py312/lib/python3.12/site-packages/torch/distributed/elastic/rendezvous/api.py”, line 25<br>
8 in create_handler<br>
File “/share/software/user/open/py-pytorch/2.2.1_py312/lib/python3.12/site-packages/torch/distributed/elastic/rendezvous/registry.py”, li<br>
ne 66 in get_rendezvous_handler<br>
File “/share/software/user/open/py-pytorch/2.2.1_py312/lib/python3.12/site-packages/torch/distributed/launcher/api.py”, line 238 in launc<br>
h_agent<br>
File “/share/software/user/open/py-pytorch/2.2.1_py312/lib/python3.12/site-packages/torch/distributed/launcher/api.py”, line 135 in <strong>cal<br>
l</strong><br>
File “/share/software/user/open/py-pytorch/2.2.1_py312/lib/python3.12/site-packages/torch/distributed/run.py”, line 803 in run<br>
File “/share/software/user/open/py-pytorch/2.2.1_py312/lib/python3.12/site-packages/torch/distributed/run.py”, line 812 in main<br>
File “/share/software/user/open/py-pytorch/2.2.1_py312/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>in<br>
it</strong>.py”, line 347 in wrapper<br>
File “/share/software/user/open/py-pytorch/2.2.1_py312/bin/torchrun”, line 33 in <br>
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, n<br>
umpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random.<em>mt19937, numpy.random.mtrand, numpy.random.</em><br>
philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, tor<br>
ch._C._nn, torch._C._sparse, torch._C._special (total: 20)<br>
Segmentation fault</p>
<p>Any idea? Thanks again!</p>