Pytorch 2.0.1 failed to run a simple code which works on pytorch 1.13.0

ipe-zhangyz: <pre><code class="lang-auto">import torch
import torch.distributed as dist
import time

# Code runs on each rank.
dist.init_process_group("nccl")
rank = dist.get_rank()
device = torch.device(rank)

world_size = dist.get_world_size()
datashape = world_size

input_data = torch.ones(datashape).to(device)
input_data *= rank
print(f"RAnk {rank} data: {input_data}")

#broadcast
if rank == 0:
    for i in range(1, world_size):
        dist.send(input_data, i)
        print(f"RAnk {rank} send data to rank {i}")
else:
    dist.recv(input_data, 0)
    print(f"RAnk {rank} recv data from rank 0")
print(f"after broadcast on RAnk {rank} input data: {input_data}")

</code></pre>
<p>run the code above using this command:<br>
torchrun --nnodes 1 --nproc_per_node 2 --standalone  broadcast.py<br>
On pytorch 1.13.0, it finishes with the correct result. However, on pytorch 2.0.1, it fails with this error:</p>
<p>master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.<br>
WARNING:torch.distributed.run:</p>
<hr>
<p>Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.</p>
<hr>
<h1><a name="rank-0-data-tensor0-0-devicecuda0-rank-1-data-tensor1-1-devicecuda1-errortorchdistributedelasticmultiprocessingapifailed-exitcode-7-local_rank-0-pid-530-of-binary-rootminiconda3binpython-traceback-most-recent-call-last-file-rootminiconda3bintorchrun-line-33-in-module-sysexitload_entry_pointtorch201-console_scripts-torchrun-file-rootminiconda3libpython39site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-346-in-wrapper-return-fargs-kwargs-file-rootminiconda3libpython39site-packagestorchdistributedrunpy-line-794-in-main-runargs-file-rootminiconda3libpython39site-packagestorchdistributedrunpy-line-785-in-run-elastic_launch-file-rootminiconda3libpython39site-packagestorchdistributedlauncherapipy-line-134-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-rootminiconda3libpython39site-packagestorchdistributedlauncherapipy-line-250-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1" class="anchor" href="#rank-0-data-tensor0-0-devicecuda0-rank-1-data-tensor1-1-devicecuda1-errortorchdistributedelasticmultiprocessingapifailed-exitcode-7-local_rank-0-pid-530-of-binary-rootminiconda3binpython-traceback-most-recent-call-last-file-rootminiconda3bintorchrun-line-33-in-module-sysexitload_entry_pointtorch201-console_scripts-torchrun-file-rootminiconda3libpython39site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-346-in-wrapper-return-fargs-kwargs-file-rootminiconda3libpython39site-packagestorchdistributedrunpy-line-794-in-main-runargs-file-rootminiconda3libpython39site-packagestorchdistributedrunpy-line-785-in-run-elastic_launch-file-rootminiconda3libpython39site-packagestorchdistributedlauncherapipy-line-134-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-rootminiconda3libpython39site-packagestorchdistributedlauncherapipy-line-250-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1"></a>RAnk 0 data: tensor([0., 0.], device=‘cuda:0’)<br>
RAnk 1 data: tensor([1., 1.], device=‘cuda:1’)<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 530) of binary: /root/miniconda3/bin/python<br>
Traceback (most recent call last):<br>
File “/root/miniconda3/bin/torchrun”, line 33, in <br>
sys.exit(load_entry_point(‘torch==2.0.1’, ‘console_scripts’, ‘torchrun’)())<br>
File “/root/miniconda3/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 346, in wrapper<br>
return f(*args, **kwargs)<br>
File “/root/miniconda3/lib/python3.9/site-packages/torch/distributed/run.py”, line 794, in main<br>
run(args)<br>
File “/root/miniconda3/lib/python3.9/site-packages/torch/distributed/run.py”, line 785, in run<br>
elastic_launch(<br>
File “/root/miniconda3/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 134, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/root/miniconda3/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 250, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<p>broadcast.py FAILED</p>