akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Do i change dimentions correctly

username: dato_nefaridze
cooked: <p>in net class input picture dimention is 3x100x100.</p>
<pre><code class="lang-auto"> batch_x=(X[x:x+batch_size]).permute(0,3,1,2)
 batch_y=Y[x:x+batch_size].long()
</code></pre>
<p>do i permute dimentions correctly? (it doesn’t output error, but in test set it predicts same value over and over again)<br>
above batch_size=64.<br>
i am trying to input 64x3x100x100 picture.</p>

username: ptrblck
cooked: <p>If <code>X</code> contains <code>[N, H, W, C]</code> data and you would like to provide it as <code>[N, C, H, W]</code>, then your permutation is correct.</p>
<p>The constant output might come from a bad training run (overfitting to the majority class or mean value in a regression?).</p>

username: dato_nefaridze
cooked: <p>ok,but my data is almost perfect, i mean it’s distribution is  48% 52%.</p>

username: ptrblck
cooked: <p>In that case, try to overfit a small data sample (e.g. just 10 samples) and play around with the hyperparameters.<br>
Once your model is able to overfit the small data, you could try to scale up the experiment and use more data samples.<br>
If the model isn’t able to overfit, there might be some other bugs in the code, e.g. forgetting to zero out the gradients.</p>
Runtime error while multiprocessing

username: damaggu
cooked: <p>Hi,</p>
<p>I’m having trouble with multiple processes working on the same GPU. I wrote minimal error-reproducing example.<br>
I ran the example code successfully on my local machine, using CUDA 10.2 and pytorch 1.2.0.<br>
While this works just fine, it fails to run on a cluster with CUDA 10.1 and pytorch 1.2.0.</p>
<p>Does anybody know why or how to overcome this? Thanks a ton.</p>
<p><strong>CODE EXAMPLE</strong></p>
<pre><code class="lang-auto">import torch.multiprocessing as _mp
import torch
import os
import time
import numpy as np

mp = _mp.get_context('spawn')

class Process(mp.Process):
    def __init__(self, id):
        super().__init__()
        print("Init Process")
        self.id = id
        return

    def run(self):
        os.environ['CUDA_VISIBLE_DEVICES'] = '0'
        for i in range(3):
            with torch.cuda.device(0):
                x = torch.Tensor(10).to(0)
                x.to('cpu')
                del x
            time.sleep(np.random.random())

if __name__ == "__main__":
    num_processes = 2
    os.environ['CUDA_VISIBLE_DEVICES'] = '0'
    processes = [Process(i) for i in range(num_processes)]
    [p.start() for p in processes]
    [p.join() for p in processes]
</code></pre>
<p><strong>ERROR</strong></p>
<pre><code class="lang-auto">Process Process-2:
Traceback (most recent call last):
  File "/cluster/home/marksm/software/anaconda/envs/test/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/cluster/home/marksm/mp_demonstration.py", line 20, in run
    x = torch.Tensor(10).to(0)
RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable
</code></pre>
How to create a mask tensor from another mask tensor

username: fbrah
cooked: <p>lets say I have tensor of shape  (2, 10) .<br>
I want to create a mask tensor with same shape that is ‘1’ only immediately  after occurrence of a specific value and ‘0’ elsewhere.<br>
For example considering I want to mask indices after 3:</p>
<pre><code class="lang-auto">x = tensor([[1, 3, 4, 5, 3, 6],
        [1, 2, 5, 3, 7, 9]])
mask = tensor([[False,  False, True, False,  False, True],
        [False, False, False,  False, True, False]])
</code></pre>
<p>x.eq(3) will mask indices of 3 but not the next index.</p>
<p>Thanks for ur help</p>

username: Sunshine352
cooked: <p>Just x== value, you can get the result you want</p>

username: fbrah
cooked: <p>I want the next element of the index with that specific value be ‘True’ not the index of value</p>

username: Sunshine352
cooked: <p>You can return the index of value by torch.nonzero(x==3), then manipulate the index, to generate the mask.</p>

username: fbrah
cooked: <p>Yeah actually I tried that but I could not generate what I want since each row may have multiple occurrence of that value. I don’t know how to manipulate the result of torch.nonzero(x==3).</p>
<pre><code class="lang-auto">ind = (x==3).nonzero()
tensor([[0, 1],
        [0, 4],
        [1, 3]])
</code></pre>
<p>Should I do it in a for loop? Or is there a nice vectorized way of doing it?</p>

username: Eta_C
cooked: <p>===== 1</p>
<pre><code class="lang-python">mask = x == 3
mask = torch.cat([torch.zeros(mask.shape[0])[..., None].bool(), mask[:, :-1]], dim=1)
</code></pre>
<p>===== 2</p>
<pre><code class="lang-python">row, col = torch.where(x == 3)
col += 1
mask = torch.zeros_like(x).bool()
mask[row, col] = True
</code></pre>
<p>===== 3</p>
<pre><code class="lang-python">mask = torch.zeros_like(x).bool()
mask[:, 1:] = (x[..., :-1] == 3)
</code></pre>

username: fbrah
cooked: <aside class="quote no-group quote-modified" data-username="Eta_C" data-post="6" data-topic="63433">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eta_c/48/17667_2.png" class="avatar"> Eta_C:</div>
<blockquote>
<p>mask[:, 1:] = (x[…, :-1] == 3)</p>
</blockquote>
</aside>
<p>Great, thanks that worked.</p>
Do I need separate scripts for distributed training and sinlge GPU?

username: maxmatical
cooked: <p>I’ve been trying to create a training script that can use both a single GPU or multiple GPUs for distributed training by setting <code>nproc_per_node</code> to be equal to the number of GPUs being used. However, if I try <code>nproc_per_node=1</code>, I get a runtime error <code>RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.</code> So do I need separate scripts for either scenarios, or is there a way to set up the script in such a way that it can do both?</p>
Convolution among specified channels

username: yclin
cooked: <p>Description:  I want to aplply the same convolution over the concatenation of every channel in tensor A (mxhxw) and  the whole tensor B(nxhxw).<br>
Let’s say, we have two tensors A (mxhxw) and B (nxhxw). Now I want to do the following operations (convolutions, batch dimension removed):</p>
<p><em>for k in range (0, m):</em><br>
<em>Concat kth channel in tensor A with the whole tensor B, then we have a new tensor C_k of size (1+n, , h, w), then apply a convolution layer on C_k (in_channel=n+1, out_channel=p,  kernel =3, stride=1). Ideally we have an output of size (p, h, w).</em><br>
<strong>Note: apply the same concolution for every C_k (sharing weights).</strong></p>
<p>The final output would be a new tensor of size (mxp, ,h, w).</p>
<p>Is there a smart way to do this? Thanks a lot !</p>