akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch using both GPUs even when after setting explictly

Malitha96: <p>Hi everyone,</p>
<p>I’m facing a strange issue with GPU allocation in PyTorch.</p>
<p><strong>Environment:</strong></p>
<ul>
<li>PyTorch version: <code>2.1.2+cu118</code></li>
<li>CUDA Version: 11.8</li>
<li>NVIDIA Driver: (recently updated to latest version)</li>
<li>2 GPUs available on the cluster</li>
</ul>
<p><strong>Issue:</strong><br>
In my training script, I explicitly specify which GPU to use by setting the environment variable:</p>
<pre data-code-wrap="bash"><code class="lang-bash">CUDA_VISIBLE_DEVICES=1
</code></pre>
<p>This should restrict PyTorch to only GPU1. However:</p>
<ul>
<li>When I run <code>lsof /dev/nvidia0</code>, I can see that my process is also holding <strong>730MB memory</strong> and showing <strong>3% utilization</strong> on <strong>GPU0</strong>.</li>
<li><code>nvidia-smi</code> and <code>gpustat</code> do <strong>not</strong> show any active process on GPU0.</li>
<li>If I reverse and set <code>CUDA_VISIBLE_DEVICES=0</code>, I observe the same issue — now <strong>GPU1</strong> shows minor utilization according to <code>lsof</code>, even though I specified to use only GPU0.</li>
</ul>
<p>I have also updated my NVIDIA drivers and tested this behavior again — it still persists.</p>
<p><strong>Summary:</strong></p>
<div class="md-table">
<table>
<thead>
<tr>
<th style="text-align:right"><code>CUDA_VISIBLE_DEVICES</code></th>
<th style="text-align:center">lsof shows usage on</th>
<th style="text-align:center">nvidia-smi / gpustat shows usage on</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:right">1</td>
<td style="text-align:center">GPU0 (minor usage)</td>
<td style="text-align:center">Only GPU1</td>
</tr>
<tr>
<td style="text-align:right">0</td>
<td style="text-align:center">GPU1 (minor usage)</td>
<td style="text-align:center">Only GPU0</td>
</tr>
</tbody>
</table>
</div><p><strong>Question:</strong><br>
Why is PyTorch touching the other GPU even when I explicitly set <code>CUDA_VISIBLE_DEVICES</code>?<br>
Is this just a harmless driver-level initialization, or can it affect performance in multi-user environments?<br>
Is there a way to prevent any interaction with the unselected GPU?</p>
<p>Any help or explanation would be greatly appreciated!</p>

H-Huang: <p>I’m not able to reproduce this but I am using cuda 12.4</p>
<p>Maybe one potential hypothesis is that you are setting CUDA_VISIBLE_DEVICES after cuda has already been initialized? Cuda is lazily initialized, but if there is some usage of cuda APIs (e.g. <code>torch.cuda.set_device()</code>) in the libraries you are using, then that will create a context of the device.</p>
<p>A simple reproduction script would help in tracking down the issue.</p>

Malitha96: <p>I am training the official nnUNet without using DDP. Hence I am not sure how it is happening.</p>
How does PyTorch c++ code do that?

cokespace2: <p>I saw some codes here: <a href="https://github.com/pytorch/pytorch/blob/428234bc285412f5d379831462b982599d1d2f6f/aten/src/ATen/native/LossNLL.cpp#L511" rel="noopener nofollow ugc">code</a>, and got a little bit confusing about this <code>target_ * (1 - label_smoothing)</code>.</p>
<p>How PyTorch c++ code now which <code>*</code> is been called and then compiled? If there’s a override of <code>Tensor operator*()</code>, where can I find it?</p>
`nodes.Batcher` on multiple outputs

pascal: <p>I am using <code>nodes.Batcher</code> on a node with multiple outputs.</p>
<pre><code class="lang-auto">from torchdata.nodes import Batcher, IterableWrapper, Loader, Mapper

node = IterableWrapper(range(16))
node = Mapper(node, map_fn=lambda x: (x, x**2))
node = Batcher(node, batch_size=4)
loader = Loader(node)
print(list(loader))
</code></pre>
<p>This results in batches of tuples (<code>[(0, 0), (1, 1), (2, 4), (3, 9)]</code>) instead of tuples of batches. What am I doing wrong here?</p>

pascal: <p>I am missing a call to <code>default_collate</code> after batching.</p>
Problems in realizing checkpoint in libtorch

fangniu: <p>checkpoint can reduced the memory usage in pytorch. However, in libtorch there is no such function yet. Here I am trying to realize such a function with torch::autograd::backward. My code can be successfully compiled. When I run it, I meet some problem. The code is attached below.</p>
<p><span class="hashtag-raw">#include</span> &lt;torch/torch.h&gt;<br>
<span class="hashtag-raw">#include</span> </p>
<p>using namespace torch::autograd;</p>
<p>class TestFunction : public Function<br>
{<br>
public:<br>
inline torch::Tensor Testf(torch::Tensor x, torch::Tensor y, const double &amp;coeff)<br>
{<br>
torch::Tensor z = torch::zeros(x.sizes());<br>
z = (3 * x + y) * coeff;<br>
return z;<br>
}</p>
<pre><code>    static variable_list forward(AutogradContext *ctx, Variable x, Variable y, const double &amp;coeff) 
    {
        std::cout &lt;&lt; " forward" &lt;&lt; std::endl;

        ctx-&gt;saved_data["coeff"] = coeff;
    ctx-&gt;save_for_backward({x, y});

    torch::NoGradGuard no_guard;
    TestFunction ts;
    auto z = ts.Testf(x, y, coeff);

        return {z};
    }

    static variable_list backward(AutogradContext *ctx, variable_list grad_output) 
{
        std::cout &lt;&lt; " backward" &lt;&lt; std::endl;

    double coeff = ctx-&gt;saved_data["coeff"].toDouble();
    auto v = ctx-&gt;get_saved_variables();
    Variable x = v[0];
    Variable y = v[1];

    std::cout &lt;&lt; grad_output[0] &lt;&lt; std::endl;

        Variable x1 = x.detach();
    Variable y1 = y.detach();

        x1.set_requires_grad(x.requires_grad());
    y1.set_requires_grad(y.requires_grad());

    torch::Tensor z;
        TestFunction ts;
        z = ts.Testf(x1, y1, coeff);

    torch::AutoGradMode enable_grad(true);
    torch::autograd::backward({z}, grad_output); 

    return {x1.grad(), y1.grad(), torch::Tensor()};
    }
</code></pre>
<p>};</p>
<p>std::vector<a>at::Tensor</a> TT(torch::Tensor x, torch::Tensor y, const double &amp;coeff)<br>
{<br>
return TestFunction::apply(x, y, coeff);<br>
}</p>
<p>auto main() → int<br>
{<br>
torch::manual_seed(3);</p>
<pre><code>Variable x = Variable(torch::rand({2, 2})).set_requires_grad(true);
std::cout &lt;&lt; "x=" &lt;&lt; x &lt;&lt; std::endl;

Variable y = Variable(torch::rand({2, 2})).set_requires_grad(true);
std::cout &lt;&lt; "y=" &lt;&lt; y &lt;&lt; std::endl;

Variable v = torch::tensordot(x, y, {1}, {0});

Variable u = torch::exp(y); 

std::vector&lt;at::Tensor&gt; z = TT(u, v, 5.0); 
z[0].sum().backward();

std::cout &lt;&lt; "x.grad=" &lt;&lt; x.grad() &lt;&lt; std::endl;
std::cout &lt;&lt; "y.grad=" &lt;&lt; y.grad() &lt;&lt; std::endl;

return 0;
</code></pre>
<p>}</p>
Torchrun launches each process on the same CPUs/GPUs

asterix_obelix: <p>Hello, I am trying to run the multinode ecample using SLURM, on a cluster where every node has 8 GPUs, and 56 CPUs - <a href="https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multinode.py" class="inline-onebox" rel="noopener nofollow ugc">examples/distributed/ddp-tutorial-series/multinode.py at main · pytorch/examples · GitHub</a></p>
<p>This is how I launch it -</p>
<pre><code class="lang-auto">srun -n2 -N2 -c7 --tasks-per-node=1 --threads-per-core=1 bash -c "torchrun  --nproc_per_node=8 --nnodes=2  --master_addr=$master_ip_addr --master_port=3442 --node_rank \$SLURM_PROCID ./multinode.py 2000 10"
</code></pre>
<p>I notice that torchrun is launching every process using the same set of CPUs/GPUs, as determined by the numa library (<a href="https://github.com/numactl/numactl" class="inline-onebox" rel="noopener nofollow ugc">GitHub - numactl/numactl: NUMA support for Linux</a>)</p>
<pre><code class="lang-auto">CPU affinity for PID 3452774: 0 1 2 3 4 5 6
GPU affinity for PID 3452774: 4
CPU affinity for PID 3452775: 0 1 2 3 4 5 6
GPU affinity for PID 3452775: 4
CPU affinity for PID 3452771: 0 1 2 3 4 5 6
GPU affinity for PID 3452771: 4
CPU affinity for PID 1004587: 0 1 2 3 4 5 6
GPU affinity for PID 1004587: 4
CPU affinity for PID 3452772: 0 1 2 3 4 5 6
GPU affinity for PID 3452772: 4
CPU affinity for PID 1004583: 0 1 2 3 4 5 6
GPU affinity for PID 1004583: 4
CPU affinity for PID 1004580: 0 1 2 3 4 5 6
GPU affinity for PID 1004580: 4
CPU affinity for PID 3452773: 0 1 2 3 4 5 6
GPU affinity for PID 3452773: 4
CPU affinity for PID 1004584: 0 1 2 3 4 5 6
GPU affinity for PID 1004584: 4
CPU affinity for PID 1004582: 0 1 2 3 4 5 6
GPU affinity for PID 1004582: 4
CPU affinity for PID 3452770: 0 1 2 3 4 5 6
GPU affinity for PID 3452770: 4
CPU affinity for PID 1004586: 0 1 2 3 4 5 6
GPU affinity for PID 1004586: 4
CPU affinity for PID 3452776: 0 1 2 3 4 5 6
GPU affinity for PID 3452776: 4
CPU affinity for PID 1004585: 0 1 2 3 4 5 6
GPU affinity for PID 1004585: 4
CPU affinity for PID 1004581: 0 1 2 3 4 5 6
GPU affinity for PID 1004581: 4
CPU affinity for PID 3452777: 0 1 2 3 4 5 6
GPU affinity for PID 3452777: 4
</code></pre>
<p>Am I launching torchrun in a sensible manner using SLURM? Is launching all processes on the same CPUs expected behavior by torchrun? Is there a workaround where I can enforce torchrun to not use CPUs already assigned to a process?</p>

asterix_obelix: <p><a class="mention" href="/u/h-huang">@H-Huang</a> Hi, not sure if you are the go to person for distributed PyTorch, but I see your responses on a lot of questions relating to distributed learning with PyTorch. So I figured might as well tag you on this.</p>
A RuntimeError during distributed training

howyeer: <p>During single-GPU training, the code runs successfully and can train multiple epochs (more than 4, with no further testing downward) without any errors. However, when using multiple GPUs for training, an error occurs, and it always happens <strong>when the first epoch is about to complete</strong>. The specific error is as follows:<br>
<em>RuntimeError: Detected mismatch between collectives on ranks. Rank 0 is running collective: CollectiveFingerPrint(OpType=BROADCAST, TensorShape=[29836], TensorDtypes=Long, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))), but Rank 1 is running collective: CollectiveFingerPrint(OpType=ALLREDUCE_COALESCED).</em><br>
<em>ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 2598132) of binary:</em><br>
And the error occured at <strong>torch.distributed.all_reduce()(or torch.distributed.barrier())</strong>.When i tried commented out this part of the code, it can complete the training of 1 epoch, but an error occurs when training for 2 or more epochs,like this:<br>
<em>RuntimeError: Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. Original exception: [/opt/conda/condabld/pytorch_1670525541702/work/third_party/gloo/gloo/transport/tcp/pair.cc:598] Connection closed by peer [192.168.1.104]:37535 ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 930361) of binary:</em></p>

howyeer: <p>And the env is torch                     2.1.1+cu121.If you need more details, I can provide them at any time. Thank you very much for helping me resolve the issue.</p>

H-Huang: <p>I’m guessing that only some ranks are calling into <code>barrier()</code>. A reproduction script would be helpful to debug this.</p>