akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Stack tensors keeping their automatic gradient information

takafusui: <p>Hi,</p>
<p>I want to stack some tensors keeping their automatic gradient information but screw up.<br>
I have the following script where I want to compute the derivative with respect to <code>x</code>:</p>
<pre><code class="lang-python">_k_plusplus = []
for aplus_idx, aplus_val in enumerate(aplus):
    state_plus = torch.tensor([x[0], aplus_val])[None, :]
    pred_mean = likelihood(gp_kplus(state_plus)).mean
    _k_plusplus.append(pred_mean)
    k_plusplus = torch.stack(_k_plusplus).flatten()
</code></pre>
<p>where <code>print(k_plusplus)</code> returns:</p>
<pre><code class="lang-python">tensor([0.3481, 0.3481, 0.3481, 0.3481, 0.3481], dtype=torch.float64,
       grad_fn=&lt;ViewBackward&gt;)
</code></pre>
<p>It seems to be OK for me, but with this syntax, <code>torch.autograd.gradcheck</code> retunrs <code>False</code>.<br>
Here note that <code>pred_mean = likelihood(gp_kplus(state_plus)).mean</code> is coming from <code>GPyTorch</code>.<br>
<code>print(pred_mean)</code> retuns:</p>
<pre><code class="lang-python">tensor([0.2027], dtype=torch.float64, grad_fn=&lt;ViewBackward&gt;)
</code></pre>
<p>again it seems to be reasonable for me.</p>
<p>My guess is when I stack the tensors by <code>torch.stack</code> or maybe at some points, I lose the automatic gradient information.<br>
When I simplify the script:</p>
<pre><code class="lang-python">k_plusplus = torch.empty(aplus.shape)
for aplus_idx, aplus_val in enumerate(aplus):
   k_plusplus[aplus_idx] = torch.tensor(0.1)
</code></pre>
<p>It passes the gradient check without changing the rest of my code.</p>
<p>Thank you in advance for your kind help.</p>

takafusui: <p>Sorry this is my bad.<br>
In the above snippet, <code>type(x)</code> is <code>torch.tensor</code> and I overwrite the tensor object by <code>torch.tensor(x[0]...)</code> where I lost the <code>grad_fn</code> information, I guess.<br>
When I modify</p>
<pre><code class="lang-python">state_plus = torch.stack([x[0], aplus_val])[None, :]
</code></pre>
<p>everything works well.</p>
Should copy.deepcopy(mdl) == mdl return true?

pinocchio: <p>I ran:</p>
<pre><code class="lang-auto">mdl0 = copy.deepcopy(mdl)
print(mdl == mdl0)
</code></pre>
<p>and it prints</p>
<pre><code class="lang-auto">False
</code></pre>
<p>I would have expected that the <code>is</code> operator returned false since they aren’t the same object but they are copies, so equality should return true.</p>
<p>Why is this not working as I expected? Is that a bug or do I have a misunderstanding?</p>
Create a kernel-wise operator

mojzaar: <p>Hi,</p>
<p>I’m wondering whether we can use the PyTorch capacity to create a kernel-wise operator or not. By kernel-wise operator, I mean an operator like max pool which applies to a window (kennel size) but instead of getting the max value over that window applies another function over that window. Is there any way to have such an operator in Pytorch?</p>

ptrblck: <p>You could try out your method by creating each window patch via <code>unfold</code>, apply your method, and reshape it to the output shape.<br>
<a href="https://discuss.pytorch.org/t/patch-making-does-pytorch-have-anything-to-offer/33850/11">This post</a> gives you an example of this approach.</p>
<p>Note that this approach will most likely be slow, but might be good enough to experiment with your method. If you need a more performance you might want to write a custom extension.</p>

mojzaar: <p>Thanks for your reply. I actually wanted to implement 2d fft on a window. I noticed that there exist a fft operator in pytorch but it seems slow on the implementation.</p>
<p>Thanks again for your reply.</p>
Wrong value when accessing tensor with pointer

deyan: <p>I’m working on some cuda extension for pytorch. When I received the grad_output, printing it yields correct values.</p>
<pre><code class="lang-auto">std::cout &lt;&lt; upstreamGrad &lt;&lt; std::endl;

output 
1 1 1 1 1 1
[ CUDAFloatType{1,6} ]
</code></pre>
<p>However, accessing it with a pointer returns wrong values.</p>
<pre><code class="lang-auto">std::vector&lt;float&gt; tmp(6);
cudaMemcpy(tmp.data(), upstreamGrad.data_ptr(), 6 * sizeof(float), cudaMemcpyDeviceToHost);
for (int i = 0; i &lt; 6; i++) {
  std::cout &lt;&lt; tmp[i] &lt;&lt; std::endl;
}

output
1
20
12
0
27
22
</code></pre>
<p>I was wondering if I’m missing something here.</p>

albanD: <p>You are not checking the <code>storage_offset</code> and stride informations.<br>
Given that the first value is correct but not the other one, I would guess that this is a non-contiguous Tensor and you should use the stride to read it properly.<br>
Doing <code>upstreamGrad.contiguous().data_ptr()</code> should give you the right result.</p>
Support torchvision.ops.nms on mobile CPU?

KevinZ1992: <p>I try to use torchvision.ops.nms in my model, and convert mymodel.pth to mymodel.pt, it all sucess.But when load mymodel.pt  on mobile,came across error.</p>
<pre><code class="lang-auto">    Serialized   File "code/__torch__/torch/nn/modules/module/___torch_mangle_324.py", line 63
        _27 = torch.slice(_26, 1, 1, 9223372036854775807, 2)
        _28 = torch.copy_(_27, torch.view(_25, [90, 2]), False)
        keep = ops.torchvision.nms(boxes, scores0, 0.5)
               ~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
        _29 = torch.slice(boxes, 1, 0, 9223372036854775807, 1)
        keep0 = torch.to(keep, dtype=4, layout=0, device=torch.device("cpu"), pin_memory=False, non_blocking=False, copy=False, memory_format=None)
    
        at org.pytorch.NativePeer.initHybrid(Native Method)
        at org.pytorch.NativePeer.&lt;init&gt;(NativePeer.java:18)
        at org.pytorch.Module.load(Module.java:23)
        at org.pytorch.helloworld.MainActivity.onCreate(MainActivity.java:39)
        at android.app.Activity.performCreate(Activity.java:7458)
        at android.app.Activity.performCreate(Activity.java:7448)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1286)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3409)
        	... 11 more
</code></pre>
<p>Someone can help,thanks.</p>

xta0: <p>We don’t currently build and ship torchvision for mobile. We’re looking into it.</p>
How can I create a mask based on comparing numerical values?

ameliatqy: <p>Hey guys,</p>
<p>How can I create a mask based on comparing numerical values? For example, I want the mask to be 1 when the first value in the second dimension is bigger than the second value in the second dimension.</p>
<p>Input: [[1, 2], [4, 1], [3, 1], [0, 1]]<br>
Output Mask: [0, 1, 1, 0]</p>
<p>Input: [[22, 11], [90,80], [15, 19], [6, 8]]<br>
Output Mask: [1, 1, 0, 0]</p>
<p>Is this possible? Thank you for your help in advance!</p>
<p>EDIT: Corrected second example</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">x = torch.tensor([[1, 2], [4, 1], [3, 1], [0, 1]])
res = (x[:, 0] &gt; x[:, 1]).long()

x = torch.tensor([[22, 11], [90,80], [15, 19], [6, 8]])
res = (x[:, 0] &gt; x[:, 1]).long()
</code></pre>
<p>The second output should probably be <code>[1, 1, 0, 0]</code>?</p>

ameliatqy: <p>It does work <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> And you are right about the second example, I will correct it. Thank you for your help!</p>
Spectrogram by torch inconsistent with scipy.signal

dnnagy: <p>I’m trying to replace <code>scipy.signal</code> with <code>torch</code> for audio preprocessing (and I don’t want to use <code>torchaudio</code> because I need to export the whole pipeline to C++).<br>
So, I need log-spectrogram, which is S = log(|STFT(x)|^2), but I get unexpected shapes from pytorch.</p>
<p>Minimal code to reproduce:</p>
<pre data-code-wrap="python"><code class="lang-python">from scipy import signal
import torch

data = np.random.uniform(-1, 1, 22500)
n_fft = 256

s, t, z = signal.spectrogram(data, nperseg=n_fft)
ss = torch.stft(torch.Tensor(data), n_fft)

print(z.shape)  # (129, 100)
print(ss.shape) # torch.Size([129, 352, 2])
</code></pre>
<p>You can see that the shapes do not match, and the output of <code>torch</code> has a lot of infs/nans:<br>
<code>plt.spy(z) </code>:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/9/39bf8680c8b856bb35ad82028c308cfdb221a89a.png" alt="z" data-base62-sha1="8eRyxu5JcXDPRulEcRXVFBwk9oC" width="208" height="252"></p>
<p><code>plt.spy(np.log(ss[:,:, 0]*ss[:, :, 1]))</code>:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66b90701ad76f04511acad1d0a2c1fb9dc2db5e3.png" alt="ss" data-base62-sha1="eEJ7wgjQLAsXLj0UYa917iQ7v1x" width="383" height="157"></p>
<p>How can I properly compute the spectrogram using torch only?</p>

JuanFMontesinos: <p>Just read the documentation…<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23f4fe5678733068b4cb0c9a1100019558de078a.png" alt="image" data-base62-sha1="585BFK5d496Psx9GVhVAOVtQJSO" width="677" height="70"><br>
Pytorch is twice as big by default. Be sure you are applying same padding, same centering same overlap etcetera.</p>
Weights in weighted loss (nn.CrossEntropyLoss)

banikr: <p>Hello Altruists,<br>
I am working on a multiclass classification with image data. The training set has 9015 images of 7 different classes.<br>
Target labeling looks like  <code>0,1,0,0,0,0,0</code><br>
But the dataset is very much skewed to one class having 68% images and lowest amount is 1.1% belongs to another class. Please take a look at the figure below:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7f1eb1b088b3eef8df65e6d16abc55711374e49.png" alt="image" data-base62-sha1="uOkXLbukdamXnaTPs3xMcqhyvNL" width="164" height="219"></p>
<p>How can I use weighted <code>nn.CrossEntropyLoss</code> ?<br>
Do I normalize the weights in order as it is or in reverse order?</p>
<pre><code class="lang-auto">weights = [9.8, 68.0, 5.3, 3.5, 10.8, 1.1, 1.4] #as class distribution
class_weights = torch.FloatTensor(weights).cuda()
Criterion = nn.CrossEntropyLoss(weight=class_weights)
</code></pre>

beaupreda: <p>Hello,</p>
<p>I do not know what you mean by reverser order, but I think it is better if you normalize the weights proportionnally to the reverse of the initial weights (so the more examples you have in the training data, the smaller the weight you have in the loss). Here is what I would do:</p>
<pre><code class="lang-auto">    weights = torch.tensor([9.8, 68.0, 5.3, 3.5, 10.8, 1.1, 1.4], dtype=torch.float32)
    weights = weights / weights.sum()
    print(weights)
    weights = 1.0 / weights
    weights = weights / weights.sum()
    print(weights)
</code></pre>

banikr: <p>Hey thanks!<br>
I did the following weighing which gave me pretty good results:</p>
<pre><code class="lang-auto">nSamples = [887, 6130, 480, 317, 972, 101, 128]
normedWeights = [1 - (x / sum(nSamples)) for x in nSamples]
normedWeights = torch.FloatTensor(normedWeights).to(device)
</code></pre>
<p>Yes you are right</p>
<blockquote>
<p>the more instance the less weight of a class</p>
</blockquote>
<p>I will also try the way you’ve mentioned.</p>
<p>And are there ways to optimize weights?<br>
Thanks again!</p>

beaupreda: <aside class="quote no-group" data-username="banikr" data-post="3" data-topic="69514">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/banikr/48/54353_2.png" class="avatar"> banikr:</div>
<blockquote>
<p>And are there ways to optimize weights?</p>
</blockquote>
</aside>
<p>Yes, it seems to be possible. I found this thread which explains how you can learn the weights for the cross-entropy loss: <a href="https://discuss.pytorch.org/t/is-that-possible-to-train-the-weights-in-crossentropyloss/19944">Is that possible to train the weights in CrossEntropyLoss?</a></p>
<p>Good luck!</p>
Possible reason for NN regressions giving inferior results to ridge regression

blade: <p>I am trying to use a simple 3 layer neural net to predict a scaler output given an input of dimension 430. For my network, I use 2 layers of dimensions 600 and 80 and I use leakyReLU non-linearities. I also tried to do the regression using ridge regression. My solution for regression is better than my pytorch implementation. I was wondering what. I have been playing around with layers, dimensions, non-linearity, learning rate but non improved the results. I was wondering if this is something to be expected and if not, what else would you suggest analyzing.</p>

ptrblck: <p>If you have a working model, e.g. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html">sklearn.linear_model.Ridge</a> make sure to dig a bit into the model and then you could try to reimplement it in PyTorch.<br>
A lot of sklearn models use some regularization, which proved to work good, while these techniques are often forgotten in the custom PyTorch implementation.</p>

blade: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>! I fixed the regularization but the error did not get much better. I was in fact using RidgeCV from sklearn, but the idea is same. I looked at the source code (thanks, that is always a good research strategy advice) and I think that the only difference now is that the code uses SVD for finding the coefficients: we find the derivative with respect to weights and find the solution in terms of SVD decomposition matrices.</p>
<p>As far as I understand, backprop does that for me in a network anyways and I don’t care about how to update weights other than my loss function. I can use pytorch to implement this but if I use SVD then it won’t be a network anymore.</p>
<p>For interested future reader, as can be seen in Bishop’s PRML book:</p>
<blockquote>
<p>The particular case of a quadratic regularizer is called ridge<br>
regression (Hoerl and Kennard, 1970). In the context of neural<br>
networks, this approach is known as weight decay.</p>
</blockquote>
<p>So we can either set <code>weight_decay</code> parameter in optimizer to a non-zero value or we can regularize the loss function with L2 as follows:</p>
<pre><code>criterion = torch.nn.MSELoss()
lmbd = 1e-8  # for custom L2 regularization
loss = criterion(y_pred, y_train)

reg_loss = None
for param in model.parameters():
    if reg_loss is None:
        reg_loss = torch.sum(param ** 2)
    else:
        reg_loss = reg_loss + param.norm(2) ** 2

loss += lmbd * reg_loss
</code></pre>
How to modify the loss function of models in torchvison?

Michael-J98: <p>Excuse me if this question is a little stupid, for I just recently got access to this extraordinary field and cannot find the answer after some researching.<br>
I invoked the pretrained mrcnn model in torchvison however its output wasn’t so ideal. So I wonder if I can modify the loss function to improve its performance without rewriting the whole framework?<br>
Thanks a lot for any advice.</p>
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long

waleed_hamid: <p>when running following notebook on google Colab it works fine<br>
<a href="https://colab.research.google.com/drive/1nZ5JCnModfGBtANK3TJZEcykhEcLPN_q?authuser=1#scrollTo=-5jscIM8R4Gv" rel="nofollow noopener">https://colab.research.google.com/drive/1nZ5JCnModfGBtANK3TJZEcykhEcLPN_q?authuser=1#scrollTo=-5jscIM8R4Gv</a></p>
<p>but on my local pc but I am getting this error when I am running the training cell<br>
RuntimeError: Expected tensor for argument <span class="hashtag">#1</span> ‘indices’ to have scalar type Long; but got torch.cuda.IntTensor instead (while checking arguments for embedding)</p>
<p>Note: I have the same TensorFlow and PyTorch Version as in Colab</p>

ra_schoonhoven: <p>I cannot access your code. Can you post relevant sections of the code here and the code line where you get the error.</p>

waleed_hamid: <p>Thanks <a class="mention" href="/u/ra_schoonhoven">@ra_schoonhoven</a> I found the solution. basically I was passing (input,labels,embedding) as IntTensor instead of LongTensor but when I passed (input,labels,embedding) as LongTensor it worked for me</p>