akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: element 0 of tensors does not require grad

EbtehalAli: <p>I got thiserror “RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn” when I run this simple code. I don’t know what the problem is?<br>
def cost(params, Y):<br>
res = [myNetwork(params)]<br>
return square_loss(res, Y)<br>
def square_loss(res, Y):<br>
loss = 0<br>
for l, p in zip(res, Y):<br>
loss += (l - p) ** 2<br>
loss = loss / len(Y)<br>
loss = torch.from_numpy(loss)<br>
return torch.mean(loss)</p>
<p>var1 = Variable(torch.tensor([0.2]), requires_grad=True)<br>
opt = torch.optim.Adam([var1], lr=0.1)<br>
print(cost(var1, Y))<br>
for i in range(100):<br>
opt.zero_grad()<br>
loss = cost(var1, Y)<br>
loss.backward()<br>
opt.step()<br>
print(“Cost:”, loss)</p>

ptrblck: <p>You are detaching the loss from the computation graph by creating a new tensor:</p>
<pre><code class="lang-python">loss = torch.from_numpy(loss)
</code></pre>
<p>If looks like you are passing PyTorch tensors to the function, so I think you don’t need this operation.</p>
<p>PS: You can add code snippets by wrapping them in three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Also, <code>Variables</code> are deprecated, so you can now directly use tensors.</p>

EbtehalAli: <p>Ok, I changed the code but got the same error?<br>
def cost(var1, Y):<br>
res = myNetowrkt(var1)<br>
loss = torch.sqrt(torch.mean((res-Y)**2))<br>
return loss<br>
X    = torch.FloatTensor([[0, 0, 0, 0]])<br>
Y    = torch.FloatTensor([[0, 1, 1]])<br>
var1 = torch.tensor(3.14159, requires_grad=True)<br>
opt  = torch.optim.Adam([var1], lr=0.5)<br>
loss = cost(var1, Y)<br>
print(loss)<br>
opt.zero_grad()<br>
loss.backward(retain_graph=True)</p>

ptrblck: <p>Could you post an executable code snippet to reproduce this issue?</p>
<p>PS: you can add code snippets by wrapping them in three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

EbtehalAli: <p>Actually, I am using a quantum library to implement the network.</p>
Why the cuda() function not take effect?

freeneuro: <p>the code is</p>
<pre><code class="lang-auto">        inputs.cuda()
        targets.cuda()
        print(inputs.device)
        print(targets.device)
</code></pre>
<p>but the output is :<br>
cpu<br>
cpu</p>

Diego: <p>The <code>.cuda()</code> operation is not in place so you need to store the returned GPU tensor. Like so:</p>
<pre><code class="lang-auto">inputs = inputs.cuda()
targets = targets.cuda()
</code></pre>
<p>Notice that this operation <strong>is in place</strong> for models: e.g., <code>model.cuda()</code> will work as expected, you don’t have to do <code>model = model.cuda()</code></p>
Error in example with gpytorch

Andrew_Czeizler: <p>model.train()<br>
likelihood.train()</p>
<h1>Use the adam optimizer</h1>
<p>optimizer = torch.optim.Adam([<br>
{‘params’: model.parameters()},  # Includes GaussianLikelihood parameters<br>
], lr=0.1)</p>
<h1>“Loss” for GPs - the marginal log likelihood</h1>
<p>mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)</p>
<p>n_iter = 50<br>
for i in range(n_iter):<br>
optimizer.zero_grad()<br>
output = model(train_x)<br>
loss = -mll(output, train_y)<br>
loss.backward()<br>
print(‘Iter %d/%d - Loss: %.3f’ % (i + 1, n_iter, loss.item()))<br>
optimizer.step()</p>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
16     optimizer.zero_grad()<br>
17     output = model(train_x)<br>
—&gt; 18     loss = -mll(output, train_y)<br>
19     loss.backward()<br>
20     print(‘Iter %d/%d - Loss: %.3f’ % (i + 1, n_iter, loss.item()))</p>
<p>3 frames<br>
/usr/local/lib/python3.6/dist-packages/gpytorch/distributions/multivariate_normal.py in log_prob(self, value)<br>
112<br>
113         mean, covar = self.loc, self.lazy_covariance_matrix<br>
–&gt; 114         diff = value - mean<br>
115<br>
116         # Repeat the covar to match the batch shape of diff</p>
<p>RuntimeError: expected backend CPU and dtype Double but got backend CPU and dtype Float</p>
<p>I am getting a runtime error i have two tensors one x and two y values and I am trying the multitask example.</p>
Calculate Jensen Shannon divergence

ashwin.raju93: <p>I would like to calculate JSD across N probabilities. This is a correct way to implement JSD ?</p>
<pre><code class="lang-auto">def jsd_loss(logits1, logits2, logits3, logits4):
    softmax1 = torch.softmax(logits1 + 1e-10, 1)
    softmax2 = torch.softmax(logits2 + 1e-10, 1)
    softmax3 = torch.softmax(logits3 + 1e-10, 1)
    softmax4 = torch.softmax(logits4 + 1e-10, 1)

    M = 0.25 * (softmax1 + softmax2 + softmax3 + softmax4)

    return 0.25 * (F.kl_div(M.log(), softmax1) +
                   F.kl_div(M.log(), softmax2) +
                   F.kl_div(M.log(),softmax3) +
                   F.kl_div(M.log(), softmax4))
</code></pre>
<p>I have added epsilon to logits to make sure that it does not go to <code>inf</code></p>
Confusion about ONNX and JIT

Diego: <p>Hello I have a few questions regarding the workings of ONNX.</p>
<ol>
<li>
<p>Let’s say I have a DNN that uses an activation function not implemented in PyTorch (a.k.a maxout). If this operation is implemented by using operators that ONNX supports (max, view, etc), will this cause any sort of problem?</p>
</li>
<li>
<p>Is there a way to implement a function inside the DNN that has an flow-control statement (if, else) and export it to ONNX?</p>
</li>
<li>
<p>Is JIT connected to ONNX in any way? Adding flow-control statements and custom operators in JIT seems very easy, is there a way to connect the 2 so that I can use operators created in JIT exporting them to ONNX?</p>
</li>
</ol>
<p>Thanks in advance!!</p>

dashesy: <p>I know a simple <code>if</code> inside a script works in the latest ONNX</p>
<pre><code class="lang-python">@torch.jit.script
def yolo_resize_helper(net_h, net_w, im_h, im_w):
    if net_w / im_w &lt; net_h / im_h:
        new_w = net_w
        new_h = ((im_h * net_w) / im_w).floor()
    else:
        new_h = net_h
        new_w = ((im_w * net_h) / im_h).floor()
    return new_h, new_w
</code></pre>
<p>But something more complicated, e.g. recursive scripts, I am trying to figure out myself.</p>
Tensor slicing using another tensor

centau: <p>I have a 2D tensor and want to select elements from it according to another Tensor with a list of cell ids.<br>
I am using <code>Tensor_A[Tensor_B]</code> but it is behaving not as I expected.</p>
<p>Why slicing a tensor using another tensor behaves differently than using a numpy array? See below for example:</p>
<pre><code class="lang-auto">In [1]: import torch
          import numpy as np

In [3]: A = torch.randint(0,10, size=(3,4))
        tensor([[6, 6, 5, 3],
               [1, 7, 7, 3],
               [5, 8, 5, 4]])

In [5]: A[np.array([[0,0],[1,1]])]
Out[5]: tensor([6, 6])

In [6]: A[torch.tensor([[0,0],[1,1]])]
Out[6]:
tensor([[[6, 6, 5, 3],
         [6, 6, 5, 3]],

        [[1, 7, 7, 3],
         [1, 7, 7, 3]]])
</code></pre>
<p>What is the difference between slicing using numpy and a tensor?</p>

ptrblck: <p>If you are using the numpy array as the index, you will get the same result as using a nested <code>list</code>:</p>
<pre><code class="lang-python">A[[[0,0],[1,1]]]    
&gt; tensor([6, 6])
</code></pre>
<p>To reproduce this behavior, you would need to split the index tensor:</p>
<pre><code class="lang-python">A[torch.tensor([[0,0],[1,1]]).split(1)]
&gt; tensor([6, 6])
</code></pre>