Discovering GPUs in multinode environment

AKT_TARAFDER: <p>We have 2 nodes, each with 4 A100 GPU.<br>
We are trying to use multiprocessing to create 8 processes for each GPU.<br>
The issue is torch.distributed only shows 4 in the device count because it shows that for one node.<br>
How can we get access to all 8 GPU in 2 nodes?<br>
The goal would be to have all 8 of them in process group or a certain selection of them so that we can perform all_gather/all_reduce and also assign each process to one GPU, that could be from different nodes.<br>
We are dividing the tensors ourselves, so we are not using some of the things like DDP, FSDP. We only need some construct that allows each process to handle on of 8 GPUs each and have a process group of all 8 GPUS.<br>
we want the ranks for all 8 GPUs to be like 0, 1, 2, 3, 4, 5, 6, 7<br>
then we compute on them, handling them through different torch processes<br>
What should be the approach? From the pytorch documentation, it is not clear how we can achieve this.</p>

Tony-Y: <p><a href="https://pytorch.org/docs/stable/elastic/run.html#environment-variables" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/elastic/run.html#environment-variables</a></p>
<blockquote>
<ol start="5">
<li><code>RANK</code> - The rank of the worker within a worker group.</li>
</ol>
</blockquote>
<p>Cannot you utilize the environment variable <code>RANK</code>?</p>

AKT_TARAFDER: <p>Hello,<br>
I am not using torchrun.<br>
We are discovering the number of GPUs in the environment. Then ask users to decide how much GPUS they want, and we manually launch the distributed run based on user inputs.<br>
we did it fine for the single-node system. This easily discovers the number of GPUs.</p>
<p>Problem is torch.distributed.get_device_count() only gives single node information(I could be wrong).<br>
I want something that discovers the environment itself. or at least assign ranks through different nodes. Like node1( ranks:0,1,2,3) and node2 (4,5,6,7)</p>
<p>Is there a way to discover the whole multimode structure, like it would show node1, node2<br>
GPU0, GPU1, GPU2, GPU3 for node 1, and similarly for node 2, we would discover GPU4, GPU5, GPU6, GPU7<br>
This should be done from inside a python function so that I can launch multiple processes.</p>
<p>Thank you, the suggestions would be really helpful.</p>

Tony-Y: <p>If you don’t know the standard usage of <code>torch.distributed</code> in multi node, the following web page might be helpful:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://lambdalabs.com/blog/multi-node-pytorch-distributed-training-guide">
  <header class="source">
      <img src="https://lambdalabs.com/hubfs/1.%20Branding%20(Lambda)/Lambda%20Logos/lambda_favicon.png" class="site-icon" width="120" height="120">

      <a href="https://lambdalabs.com/blog/multi-node-pytorch-distributed-training-guide" target="_blank" rel="noopener nofollow ugc">lambdalabs.com</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/360;"><img src="https://lambdalabs.com/hubfs/multi-node-distributed-training.png" class="thumbnail" width="690" height="360"></div>

<h3><a href="https://lambdalabs.com/blog/multi-node-pytorch-distributed-training-guide" target="_blank" rel="noopener nofollow ugc">Multi node PyTorch Distributed Training Guide For People In A Hurry</a></h3>

  <p>This tutorial summarizes how to write and launch PyTorch distributed data parallel jobs across multiple nodes, with working examples with the torch.distributed.launch, torchrun and mpirun APIs.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Tony-Y: <aside class="quote no-group" data-username="AKT_TARAFDER" data-post="3" data-topic="211416">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/akt_tarafder/48/35960_2.png" class="avatar"> AKT_TARAFDER:</div>
<blockquote>
<p>Problem is torch.distributed.get_device_count() only gives single node information(I could be wrong).</p>
</blockquote>
</aside>
<p>Do you really have <code>torch.distributed.get_device_count()</code>. I didn’t find that function in the official PyTorch package. Maybe, <code>torch.cuda.device_count()</code>.</p>

AKT_TARAFDER: <p>Yes, sorry for the miss.<br>
but you are right.<br>
what I want is some mechanism to get all the node and related GPU<br>
that means if I have 2 node, I get GPU 0,1,2,34,5,6,7<br>
and can launch multiprocessing processes in each of them, then all_reduce from all of them.</p>

Tony-Y: <p>In a multi node scenario, you have to specify nodes to be used. PyTorch doesn’t detect available nodes automatically.</p>

AKT_TARAFDER: <p>Thank you.<br>
But what are the probable ways of using these nodes, GPUs apart from pytorch?</p>

Tony-Y: <p><a href="https://mpi4py.readthedocs.io/en/stable/index.html" rel="noopener nofollow ugc">MPI for Python</a> + <a href="https://cupy.dev" rel="noopener nofollow ugc">CuPy</a> is  a possible solution. Please see <a href="https://docs.cupy.dev/en/latest/user_guide/interoperability.html#mpi4py" rel="noopener nofollow ugc">Example Code</a>.</p>