akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: No function is registered for schema aten::thnn_conv2d_forward(Tensor self, Tensor weight, int[2] kernel_size, Tensor?

username: Raghav_Gurbaxani
cooked: <p>After static . quantization, I receive the following error-</p>
<p>RuntimeError: No function is registered for schema aten::thnn_conv2d_forward(Tensor self, Tensor weight, int[2] kernel_size, Tensor? bias, int[2] stride, int[2] padding) -&gt; (Tensor output, Tensor finput, Tensor fgrad_input) on tensor type QuantizedCPUTensorId; available functions are CPUTensorId, VariableTensorId</p>
<p>raised by traced_script_module = torch.jit.trace(quantized, q_example, check_trace=False)</p>
<p>full code here - <a href="https://github.com/raghavgurbaxani/experiments/blob/master/try_static_quant.py" rel="nofollow noopener">https://github.com/raghavgurbaxani/experiments/blob/master/try_static_quant.py</a></p>

username: jerryzh168
cooked: <p>This is saying conv2d_forward is given a quantized tensor, which means some float Conv is not swapped as quantized Conv, can you print the quantized model and see where this happens?</p>

username: liwei46
cooked: <p>Hello,</p>
<p>I am not sure this is the same problem or not.<br>
But for my example, when I forget to add “model.eval()”, there is such error.<br>
When I add “model.eval()”, it seems fine.</p>

username: banikr
cooked: <p>I only get this error when I load the model with GPU:</p>
<pre><code class="lang-auto">use_cuda = torch.cuda.is_available()
device = torch.device("cuda:0" if use_cuda else "cpu")

model = ResNet3D().to(device)
model.apply(weights_init)

input = torch.rand([2,1,86,110,78])
output = model(input) 
print(input.shape)
print(output.shape)
</code></pre>
<p>Then comes the error: <code>RuntimeError: No function is registered for schema aten::thnn_conv3d_forward(Tensor self, Tensor weight, int[3] kernel_size, Tensor? bias, int[3] stride, int[3] padding) -&gt; (Tensor output, Tensor finput, Tensor fgrad_input) on tensor type CUDATensorId; available functions are CPUTensorId, VariableTensorId</code></p>
<p>if I load the model in cpu:</p>
<pre><code class="lang-auto">model = ResNet3D()
model.apply(weights_init)
output = model(input)
print(input.shape)
print(output.shape)
</code></pre>
<p>gives output nicely:</p>
<pre><code class="lang-auto">torch.Size([2, 1, 86, 110, 78])
torch.Size([2, 256])
</code></pre>

username: banikr
cooked: <p>solution: I will correct myself. I loaded the input as following:</p>
<pre><code class="lang-auto">input = torch.rand([2,1,86,110,78]).float().to(device)
</code></pre>
<p>which solved the issue of loading model with GPUs.<br>
Thanks</p>
Why is PyTorch Maximizing the Loss?

username: Tom_Ginsberg
cooked: <p>I have a simple training loop that looks like this.</p>
<pre><code class="lang-auto">optim = torch.optim.Adam(verify_net.parameters())
# The training is designed to continue untill timeout or goal
while True:
        optim.zero_grad()
        # Forward pass
        outputs = verify_net(inputs)

        # Compute how much bigger every label is be 
        # from the true label in the worst case
        losses = torch.stack(
            [upper_bound(outputs[i] - outputs[true_label]) \
                for i in range(len(outputs)) if i != true_label])

        if (losses &lt; 0).all():
            return True

        loss = torch.sum(losses)
        print(loss)
        loss.backward()
        optim.step()
</code></pre>
<p>The optimization goal is to make <code>outputs[true_label]</code> greater then all other outputs in a <em>worst case</em> that is computed by the <code>upper_bound</code> function</p>
<p><code>upper_bound = lambda x: x[0] + torch.sum(torch.abs(x[1:]))</code></p>
<p>If after any step all upper bounds <code>losses[i] &lt; 0</code> then the loop may break and we are happy. But till then the optimizer is to do its task. The output I get by printing the loss after each step is this</p>
<pre><code class="lang-auto">tensor(632.8606, grad_fn=&lt;SumBackward0&gt;)
tensor(668.1876, grad_fn=&lt;SumBackward0&gt;)
tensor(698.2267, grad_fn=&lt;SumBackward0&gt;)
tensor(733.7394, grad_fn=&lt;SumBackward0&gt;)
tensor(764.9390, grad_fn=&lt;SumBackward0&gt;)
tensor(799.8583, grad_fn=&lt;SumBackward0&gt;)
tensor(834.7762, grad_fn=&lt;SumBackward0&gt;)
tensor(861.3510, grad_fn=&lt;SumBackward0&gt;)
tensor(895.9908, grad_fn=&lt;SumBackward0&gt;)
tensor(930.6293, grad_fn=&lt;SumBackward0&gt;)
tensor(965.2657, grad_fn=&lt;SumBackward0&gt;)
tensor(1000.8009, grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<p>It seems like <code>Adam</code> is doing a fantastic job at maximizing the loss however this really isn’t the behavior I’d expect or want. My first attempt was to change <code>loss</code> into <code>-loss</code> but I had no luck because <code>Adam</code> starting to minimize the now negated loss <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Any suggestions would be much appreciated. Thanks</p>

username: albanD
cooked: <p>Hi</p>
<p>Have you tried reducing the learning rate? Or switch to a simple SGD?<br>
High learning rates can cause the network to diverge sometimes.</p>

username: Tom_Ginsberg
cooked: <p>Even with <code> lr=.000001</code> I get the same behaviour for both <code>Adam</code> and <code>SGD</code></p>

username: albanD
cooked: <p>Could you try and simplify the problem to see if you still see the same behavior?</p>
<ul>
<li>Change the model to a very basic thing (like one single Linear layer)</li>
<li>Change the loss to something simpler? (remove the absolute values)</li>
</ul>
How to load data with the same number of samples per class?

username: noeagles
cooked: <p>Certainly there are advantages in random sampling; however, there are circumstances where the numbers of samples must be the same or pre-determined. So, I wonder how to implement this in pytorch? Should I rewrite every datasets or this can be achieved by a customized sampler used in dataloader? Thanks!\</p>
<p>To make myself clear:<br>
what i need now is, for example, a batch of 10 samples from class A, 10 from class B, 10 from class C, ETC…</p>

username: Deeply
cooked: <aside class="quote no-group" data-username="noeagles" data-post="1" data-topic="25482">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/noeagles/48/2420_2.png" class="avatar"> noeagles:</div>
<blockquote>
<p>however, there are circumstances where the numbers of samples must be the same or pre-determined. So, I wonder how to impleme</p>
</blockquote>
</aside>
<p>If I understand you correctly, you need to use a sampler but keep the data balance. If your data is large enough, then even after random sampling it will be (almost) balanced.<br>
If not, the easiest way would be to generate a weight vector denoting your data balance, then to pass this vector to the loss function.<br>
See for example,</p>
<pre><code class="lang-auto">class CrossEntropyLoss(_WeightedLoss) 
...
Args:
        weight (Tensor, optional): a manual rescaling weight given to each class.
            If given, has to be a Tensor of size `C`
</code></pre>
<p>in:<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html</a></p>

username: noeagles
cooked: <p>This is a good solution to the data imbalance problem! However, what i need now is, for example, a batch of 10 samples from class A, 10 from class B, 10 from class C, ETC… Thanks anyway!</p>

username: noeagles
cooked: <p>This is a good solution to the data imbalance problem! However, what i need now is, for example, a batch of 10 samples from class A, 10 from class B, 10 from class C, ETC… Thanks anyway!</p>

username: Deeply
cooked: <p>In this case, maybe you can try the BatchSampler</p>
<pre><code class="lang-auto">class BatchSampler(Sampler):
</code></pre>
<p><a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/sampler.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/utils/data/sampler.html</a></p>

username: noeagles
cooked: <p>Sorry, but i do not find this link useful. Thanks anyway!</p>

username: Robbie
cooked: <p>Just wondering if you got a solution for this?</p>
PyTorch gradient differs from JAX

username: jelo
cooked: <p>Hey,</p>
<p>i was given an optimizer that takes a function as an input that computes the gradient of a loss function.<br>
I implemented three versions of the gradient function:</p>
<ol>
<li>computing the gradient by hand and implementing it in Numpy</li>
<li>computing the gradient with JAX</li>
<li>computing the gradient with Torch</li>
</ol>
<p>The program for Torch looks a bit like this snippet here:</p>
<pre><code class="lang-auto">import torch
import numpy as np

X = np.random.rand(5, 5)
Y = np.random.rand(4, 5)
K = np.random.rand(4, 5)

alpha = 0.1

def loss_grad(X, Y, K):
     
    X = torch.tensor(X, requires_grad=True)
    Y = torch.tensor(Y)
    K = torch.tensor(K)

    loss_val = 1/2 * torch.pow(torch.norm(Y-torch.matmul(K,X)),2)
    print(loss_val)

    loss_val.backward()

    return X.grad.numpy()

# the function i am giving to the optimizer
dfX = lambda x : loss_grad(x, Y, K)
# this would be inside the optimizer
for _ in range(100000):
    X = X - alpha * dfX(X)
</code></pre>
<p>My problem is now, that Torch and JAX compute exactly the same value for the loss function, but after three iterations i get NaN values in my gradient with Torch. This does not happen with JAX or the other implementation. The example above works fine for Torch, so i do not really know why i get those NaN values.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You question is a bit confusing between which implementations you have and which match and which don’t.</p>
<p>The example you gave works fine for me.<br>
What most likely happens is that you hit a point of non-differentiability of your function and that torch and jax behave differently at this point.<br>
Can you check what is the point where your function becomes nan? Also you can try to enable the anomaly mode in pytorch to see which op created the nans.</p>
Multi-GPU on windows

username: jmillo
cooked: <p>I can use simple PyTorch statements on Windows 10 and they work OK. But fancier applications, such as OpenNMT require some muti-procressing functions that are available only in Linux (see trace below). Is there any plan to port these to Windows in the near future?</p>
<p>c:\work\OpenNMT-py&gt;python train.py -data data/demo -save_model demo-model -world_size 2 -gpu_ranks 0 1<br>
Traceback (most recent call last):<br>
File “train.py”, line 217, in <br>
main(opt)<br>
File “train.py”, line 93, in main<br>
raise Exception(msg)<br>
Exception:</p>
<p>– Tracebacks above this line can probably<br>
be ignored –</p>
<p>Traceback (most recent call last):<br>
File “c:\work\OpenNMT-py\train.py”, line 148, in run<br>
gpu_rank = onmt.utils.distributed.multi_init(opt, device_id)<br>
File “c:\work\OpenNMT-py\onmt\utils\distributed.py”, line 25, in multi_init<br>
torch.distributed.init_process_group(<br>
AttributeError: module ‘torch.distributed’ has no attribute ‘init_process_group’</p>
Siamese Network training

username: sharjeel
cooked: <p>I am looking for a way to train a Correlation based Siamese Network, while training i get an error saying that the target size must be the same as the input size. I wanted to ask that is there a way to set up my labels for the training data in a way that it would match the input size.</p>

username: ptrblck
cooked: <p>Could you post some information regarding your model architecture, used loss function, etc.<br>
I’m currently not sure, what your use case exactly is and where the error is being thrown.</p>

username: sharjeel
cooked: <p>model1.train()<br>
model2.train()</p>
<p>batch_size = 5<br>
epochs = 2</p>
<p>for epoch in range(epochs):<br>
for i in range(0, len(X1), batch_size):<br>
batch_X1 = X1[i:i+batch_size].view(-1, 1, 300, 300)<br>
batch_X2 = X2[i:i+batch_size].view(-1, 1, 150, 150)<br>
batch_y = y[i:i+batch_size]<br>
<span class="hashtag">#print</span>(“check”)<br>
optimizer1.zero_grad()<br>
optimizer2.zero_grad()<br>
<span class="hashtag">#print</span>(“check”)</p>
<pre><code>    feature1 = model1(batch_X1)
    feature2 = model2(batch_X2)
    
    b, c, h, w = feature1.shape
    print("cc:",feature1.view(1,b*c,h,w).shape)
    
    corr_map = F.conv2d(feature1.view(1,b*c,h,w), feature2, groups=b)
    #print("check")
    #loss1 = loss_function(feature1,feature2)
    loss = loss_function(corr_map, batch_y)
    
    #loss = loss1 + loss2
    #print("check")
    loss.backward()
    optimizer1.step()
    optimizer2.step()
    
ep = epoch+1
print(f"Epoch: {ep}. Loss: {loss}")
</code></pre>
<p>this is the training code. In this code the shape of corr_map is [1,5,38,38] and the size of batch_y is [1] and i am not able to calculate the loss between them because the size is not same. I wanted to know if there is a way i could overcome this problem. If there is a way through which the size of the target equals the size of the input plz enlighten me.</p>

username: ptrblck
cooked: <p>What loss function are you using and how would you like to calculate the loss between a 4-dimensional tensor and a scalar?</p>
This is the version of the problem, I see from other people's source code inside

username: kangwei_Xu
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/2/12ac99eb96763b4f47de1df75311064d104774f5.png" data-download-href="https://discuss.pytorch.org/uploads/default/12ac99eb96763b4f47de1df75311064d104774f5" title="Z%60%40NP_GBH%7B00V(I7A_)%5DJ%601"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12ac99eb96763b4f47de1df75311064d104774f5_2_690x288.png" alt="Z%60%40NP_GBH%7B00V(I7A_)%5DJ%601" data-base62-sha1="2FcoeMEeks9HNLOQCaRJUpxmTXf" width="690" height="288" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12ac99eb96763b4f47de1df75311064d104774f5_2_690x288.png, https://discuss.pytorch.org/uploads/default/original/3X/1/2/12ac99eb96763b4f47de1df75311064d104774f5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/2/12ac99eb96763b4f47de1df75311064d104774f5.png 2x" data-dominant-color="353535"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Z%60%40NP_GBH%7B00V(I7A_)%5DJ%601</span><span class="informations">806×337 129 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Hi,</p>
<p>Could you explain in more details the problem you’re seeing please?</p>
Why there exists no grad after tensor is reshaped?

username: kiradiso
cooked: <pre><code class="lang-auto">x = torch.tensor([1., 2., 3., 5., 6., 7.], requires_grad=True)
xx = x.reshape((2, -1))
y = xx*xx
yy = torch.mean(y)
yy.backward()
print(x.grad, xx.grad, y.grad, yy.grad)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f7a9df8e723a36840ec9551c9826ee2e55e2966.png" alt="image" data-base62-sha1="4utvsH0t1A2Aoc9cKPaUkDvfr5I" width="588" height="30"></p>

username: SimonW
cooked: <p><code>x</code>: leaf of the graph<br>
<code>xx</code>: intermediate result that does not automatically collect gradient at <code>.grad</code></p>
Batch 2d convolution

username: razvanc92
cooked: <p>I am trying to create a Conv2d layer where each batch from the input will be multiply with it’s corresponding kernels. Right now the Conv2d takes as input (N, C_in, W, H) and it has weights of shape (K, K, W, H). I would like to extend the weights to (N, K, K, W, H) so each input will have it’s own set of weights. I tried to manually set the Conv2d.weight.data to the corresponding size but it does not work. Does anyone know how can I achieve this behavior?</p>

username: JuanFMontesinos
cooked: <p>Assuming that each input belongs to a different family of data, why don’t you just instantiate several convolutions. What you propose is nothing but stacking convolution outputs, right?</p>

username: razvanc92
cooked: <p>Interesting idea, I did not thought about that. I’m trying to manually set the weights. Isn’t this approach going to be slow? I will need a loop to compute all the results.</p>

username: JuanFMontesinos
cooked: <p>Well, for sure the code is very optimized to run as it is. To create such a operation you should rewrite backward function which will be slower.</p>
<p>Anyway, if you want to do that you can just use 3D convolutions with kernel=1 padding= 0 and stride =1 (in the 3rd dimension). That would be exactly what you are asking for.</p>

username: razvanc92
cooked: <p>I am not quite sure I understand your answer. So you would change the input to have the shape (B=1, C_in, Samples. W, H) ? Where Samples is my old batch number and apply 3d conv instead of 2d?</p>

username: JuanFMontesinos
cooked: <p>Oh sorry I was wrong, will a kernel = 1 you would be applying the same convolution to each sample and with kernel = B convolutions are not independent. I would go forward stacking 2d convolution outputs</p>
How to set a good learning rate when using sync bn?

username: wangyilan
cooked: <p>i’m using mmdetection and detectron. when i use mmdet and my own face data to do face detection task, I find the learning rate is hard to set. once the batch size is bigger than 4 (2 img per gpu and 2 gpus), it’s  hard to learn a good result. So do you have any learning rate tricks or related suggestions to do it? thanks!</p>
How to get the arithmetic workload of GPU?

username: flyerjia
cooked: <p>In my code, i need to analyze my code and get the arithmetic workload of  FT32, FT64 and others, how to get the arithmetic workload of GPU? What software or toolkit can i use? Like this picture:<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c8c873a4469fcd3c49c91fccdd7cc121dcc76b2.png" alt="image" data-base62-sha1="1N0OdCEg2rtnjEvjeoQoxkQeGmC" width="579" height="395"></p>