DDP device_ids in and world size 2 host 1 GPU per host

username: spyroot
cooked: <p>Hi Folks,</p>
<p>Does anyone know the logic of how the device id should be passed to DDP? In all examples it is multi-GPU, in my case, it is two hosts,s, and each host with a single GPU.</p>
<p>I keep hitting some sort of bug<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/_functions.py”, line 123, in _get_stream<br>
if _streams[device] is None:</p>
<p>Here is how master and work are initialized.</p>
<p>Master<br>
<strong>world_size 2 since its two hosts and host 1 GPU</strong></p>
<p>torch.distributed.init_process_group(<br>
backend=self.trainer_spec.get_backend(),<br>
init_method=self.trainer_spec.dist_url(),<br>
world_size=2<br>
rank=0)</p>
<p>Worker<br>
torch.distributed.init_process_group(<br>
backend=self.trainer_spec.get_backend(),<br>
init_method=self.trainer_spec.dist_url(),<br>
world_size=2<br>
rank=<strong>1</strong>)</p>
<p>master<br>
dev = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)<br>
model = DDP(model, device_ids=[0], output_device=0).to(dev)</p>
<p>worker<br>
dev = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)<br>
model = DDP(model, device_ids=[0], output_device=0).to(dev)</p>
<p>model on GPU.</p>
<p>I enabled NCLL debug.<br>
gpu10:5174:5174 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
gpu10:5174:5174 [0] NCCL INFO NET/Socket : Using [0]eth0:172.16.80.231&lt;0&gt;<br>
gpu10:5174:5174 [0] NCCL INFO Using network Socket<br>
gpu10:5174:5204 [0] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0 [1] 0/-1/-1-&gt;1-&gt;-1<br>
gpu10:5174:5204 [0] NCCL INFO Channel 00 : 0[b000] → 1[6000] [receive] via NET/Socket/0<br>
gpu10:5174:5204 [0] NCCL INFO Channel 01 : 0[b000] → 1[6000] [receive] via NET/Socket/0<br>
gpu10:5174:5204 [0] NCCL INFO Channel 00 : 1[6000] → 0[b000] [send] via NET/Socket/0<br>
gpu10:5174:5204 [0] NCCL INFO Channel 01 : 1[6000] → 0[b000] [send] via NET/Socket/0<br>
gpu10:5174:5204 [0] NCCL INFO Connected all rings<br>
gpu10:5174:5204 [0] NCCL INFO Connected all trees<br>
gpu10:5174:5204 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512<br>
gpu10:5174:5204 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer<br>
gpu10:5174:5204 [0] NCCL INFO comm 0x7f1330002fc0 rank 1 nranks 2 cudaDev 0 busId 6000 - Init COMPLETE</p>
<p>It looks to me that distribute.py track id by internal torch ID while DDP accepts deves_id that must rank.  So the logic is somewhere broken.</p>
<p>Spent a day so far no result.</p>

username: pritamdamania87
cooked: <aside class="quote no-group" data-username="spyroot" data-post="1" data-topic="152279">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spyroot/48/55231_2.png" class="avatar"> spyroot:</div>
<blockquote>
<p>I keep hitting some sort of bug<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/_functions.py”, line 123, in _get_stream<br>
if _streams[device] is None:</p>
</blockquote>
</aside>
<p>Could you share the full stack trace and the error message you are running into? It is not clear from this stack what was the actual error that caused your script to fail.</p>

username: spyroot
cooked: <p>Hi,</p>
<p>It looks like some of the variables read only for os.env down the line.  So somewhere in torch or maybe NCCL read local rank variable.</p>

username: spyroot
cooked: <aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/78047">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/78047" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/78047" target="_blank" rel="noopener nofollow ugc">DDP multi host with single GPU each. </a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-05-21" data-time="20:07:36" data-timezone="UTC">08:07PM - 21 May 22 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/spyroot" target="_blank" rel="noopener nofollow ugc">
          <img alt="spyroot" src="https://avatars.githubusercontent.com/u/11797329?v=4" class="onebox-avatar-inline" width="20" height="20">
          spyroot
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          oncall: distributed
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🐛 Describe the bug

Folks,

I have two hosts, and each host has a single<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden"> GPU, I'm using an example.
https://github.com/sudomaze/ttorch/blob/main/examples/ddp/run.py

if I use the master node 
rank 0 , world_size 2
worker
rank 1, world_size 2

if I use (master start training loop but worker never connected) 
rank 0 , world_size 1
rank 1 , world_size 1

Stack trace for case one. Note that master goes and waits only if world_size 2.

```gpu10:17709:17709 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
gpu10:17709:17709 [0] NCCL INFO NET/Socket : Using [0]eth0:172.16.80.231&lt;0&gt;
gpu10:17709:17709 [0] NCCL INFO Using network Socket
gpu10:17709:17729 [0] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0 [1] 0/-1/-1-&gt;1-&gt;-1
gpu10:17709:17729 [0] NCCL INFO Channel 00 : 0[b000] -&gt; 1[6000] [receive] via NET/Socket/0
gpu10:17709:17729 [0] NCCL INFO Channel 01 : 0[b000] -&gt; 1[6000] [receive] via NET/Socket/0
gpu10:17709:17729 [0] NCCL INFO Channel 00 : 1[6000] -&gt; 0[b000] [send] via NET/Socket/0
gpu10:17709:17729 [0] NCCL INFO Channel 01 : 1[6000] -&gt; 0[b000] [send] via NET/Socket/0
gpu10:17709:17729 [0] NCCL INFO Connected all rings
gpu10:17709:17729 [0] NCCL INFO Connected all trees
gpu10:17709:17729 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
gpu10:17709:17729 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
gpu10:17709:17729 [0] NCCL INFO comm 0x7f7f2c002fc0 rank 1 nranks 2 cudaDev 0 busId 6000 - Init COMPLETE
barrier released
Traceback (most recent call last):
  File "/root/git/dtc_latest/dtc/ddp_test/ddp_sample.worker.py", line 235, in &lt;module&gt;
    init_process(1, world_size, run)
  File "/root/git/dtc_latest/dtc/ddp_test/ddp_sample.worker.py", line 227, in init_process
    fn(rank, world_size)
  File "/root/git/dtc_latest/dtc/ddp_test/ddp_sample.worker.py", line 158, in run
    torch.cuda.set_device(rank)
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 313, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stack trace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```

Master node.

```
win11:15538:15538 [0] NCCL INFO Bootstrap : Using eth0:192.168.254.205&lt;0&gt;
win11:15538:15538 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation

win11:15538:15538 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
win11:15538:15538 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.254.205&lt;0&gt;
win11:15538:15538 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda11.5
win11:15538:15568 [0] NCCL INFO Channel 00/02 :    0   1
win11:15538:15568 [0] NCCL INFO Channel 01/02 :    0   1
win11:15538:15568 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] -1/-1/-1-&gt;0-&gt;1
win11:15538:15568 [0] NCCL INFO Channel 00 : 1[6000] -&gt; 0[b000] [receive] via NET/Socket/0
win11:15538:15568 [0] NCCL INFO Channel 01 : 1[6000] -&gt; 0[b000] [receive] via NET/Socket/0
win11:15538:15568 [0] NCCL INFO Channel 00 : 0[b000] -&gt; 1[6000] [send] via NET/Socket/0
win11:15538:15568 [0] NCCL INFO Channel 01 : 0[b000] -&gt; 1[6000] [send] via NET/Socket/0
win11:15538:15568 [0] NCCL INFO Connected all rings
win11:15538:15568 [0] NCCL INFO Connected all trees
win11:15538:15568 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
win11:15538:15568 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer
win11:15538:15568 [0] NCCL INFO comm 0x7fa13c002fb0 rank 0 nranks 2 cudaDev 0 busId b000 - Init COMPLETE
win11:15538:15538 [0] NCCL INFO Launch mode Parallel
releasing
2 64

win11:15538:15570 [0] include/socket.h:423 NCCL WARN Net : Connection closed by remote peer 172.16.80.231&lt;36406&gt;
win11:15538:15570 [0] NCCL INFO transport/net_socket.cc:414 -&gt; 2
win11:15538:15570 [0] NCCL INFO include/net.h:28 -&gt; 2
win11:15538:15570 [0] NCCL INFO transport/net.cc:459 -&gt; 2
win11:15538:15570 [0] NCCL INFO proxy.cc:351 -&gt; 2
win11:15538:15570 [0] NCCL INFO proxy.cc:452 -&gt; 2 [Proxy Thread]
```

I manage a bit narrow it down.

This a case 

```
master  
rank 0 local rank set 0 world size 2 
device = "cuda:0"
model.to(device)
DDP(model, device_ids=[0], output_device=[0)

worker
device = "cuda:0"
model.to(device)
DDP(model, device_ids=[0], output_device=[0)
```

```
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stack trace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```

### Versions

Collecting environment information...
PyTorch version: 1.11.0+cu115
Is debug build: False
CUDA used to build PyTorch: 11.5
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04 LTS (x86_64)
GCC version: (Ubuntu 11.2.0-19ubuntu1) 11.2.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.4 (main, Apr  2 2022, 09:04:19) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 512.15
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.11.0+cu115
[pip3] torchaudio==0.11.0+cu115
[pip3] torchvision==0.12.0+cu115
[conda] Could not collect

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>