In pytorch DistributedDataParallel 1 process using 2 gpu's (i.e. Processes GPU from nvida-smi command) though using torch.cuda.set_device(local_rank)

Arun_Kumar2: <p>I am using the following code snippet to ensure each task/process(local_rank) generated by slurm (using <code>#SBATCH -n 4</code>) is assigned to a specific local gpu:</p>
<pre><code>device = torch.device("cuda",local_rank)
torch.cuda.set_device(device)
</code></pre>
<p>Still, when I look at <code>nvidia-smi</code> on the “Process GPU” column I see two different GPU’s (say <code>0</code>, and <code>1</code>), assigned to only a single PID. If I tie a slurm task to a GPU using <code>torch.cuda.set_device(device)</code> (where <code>device</code> here is the <code>local_rank</code>) I should not have multiple GPU allocations right?</p>
<p>Not sure if this is important but I am using two workers on my DataLoader (see definition below), but then that should allocate the process to same GPU (say 0) right?</p>
<pre><code>    trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, num_workers=2,sampler = torch.utils.data.distributed.DistributedSampler(dataset=trainset,num_replicas=world_size,rank=world_rank))
</code></pre>
<h3>
<a name="details-1" class="anchor" href="#details-1"></a>Details:</h3>
<p>Slurm config:</p>
<pre><code>#SBATCH --nodes 2
#SBATCH -n 4
#SBATCH --gres:gpus=4
</code></pre>
<h3>
<a name="expectation-2" class="anchor" href="#expectation-2"></a>Expectation:</h3>
<p>Four tasks to be launched across two nodes. And each task to have one GPU allocated (because inside program launched by each task/process I have set <code>torch.cuda.set_device(device)</code>)</p>
<h3>
<a name="observed-3" class="anchor" href="#observed-3"></a>Observed:</h3>
<p>PID <code>6173</code> for example is allocated to two gpus (<code>0</code> and <code>1</code>). The 4th task moved to the other node; so please ignore the same.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>GPU</th>
<th>PID</th>
<th>Type</th>
<th>Process name</th>
<th>Usage</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>6172</td>
<td>C</td>
<td>…bin/python</td>
<td>2657MiB</td>
</tr>
<tr>
<td>0</td>
<td>6173</td>
<td>C</td>
<td>…bin/python</td>
<td>673MiB</td>
</tr>
<tr>
<td>0</td>
<td>6174</td>
<td>C</td>
<td>…bin/python</td>
<td>673MiB</td>
</tr>
<tr>
<td>1</td>
<td>6173</td>
<td>C</td>
<td>…bin/python</td>
<td>2657MiB</td>
</tr>
<tr>
<td>2</td>
<td>6174</td>
<td>C</td>
<td>…bin/python</td>
<td>2657MiB</td>
</tr>
</tbody>
</table>
</div>

pritamdamania87: <aside class="quote no-group" data-username="Arun_Kumar2" data-post="1" data-topic="131521">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arun_kumar2/48/23554_2.png" class="avatar"> Arun_Kumar2:</div>
<blockquote>
<p>Not sure if this is important but I am using two workers on my DataLoader (see definition below), but then that should allocate the process to same GPU (say 0) right?</p>
</blockquote>
</aside>
<p>Are the dataloaders doing any GPU work? If not, that should not be relevant here.</p>
<blockquote>
<p>Still, when I look at <code>nvidia-smi</code> on the “Process GPU” column I see two different GPU’s (say <code>0</code> , and <code>1</code> ), assigned to only a single PID.</p>
</blockquote>
<p>My guess is that this might just be some book keeping allocations happening on GPU 0 from all the processes (ex: initializing some cuda contexts etc.). Could you share a minimal script to repro this and we can figure out what might be happening here.</p>

Arun_Kumar2: <p>Thanks for this. Putting together a minimial script to repro and will get back soon.</p>