akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Evaluating a tracking result from MOTS

username: Samiha
cooked: <p>I am trying to see evaluation results using  MOTS dataset and  tools. I am following this link –<a href="https://github.com/VisualComputingInstitute/mots_tools" class="inline-onebox" rel="noopener nofollow ugc">GitHub - VisualComputingInstitute/mots_tools: Tools for evaluating and visualizing results for the Multi Object Tracking and Segmentation (MOTS) task</a> .  I am in the Evaluating a tracking result section. When tried to execute the command<br>
<code>python mots_eval/eval.py tracking_results gt_folder seqmap</code> I am facing an error. Here is the error message I am getting.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b359ad4a1bebe5754ebcc79a4085f4713b0f183e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b359ad4a1bebe5754ebcc79a4085f4713b0f183e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b359ad4a1bebe5754ebcc79a4085f4713b0f183e_2_690x357.jpeg" alt="image" data-base62-sha1="pABB8YiFjBvb0HGwwAdtvwnEXwy" width="690" height="357" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b359ad4a1bebe5754ebcc79a4085f4713b0f183e_2_690x357.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b359ad4a1bebe5754ebcc79a4085f4713b0f183e_2_1035x535.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b359ad4a1bebe5754ebcc79a4085f4713b0f183e_2_1380x714.jpeg 2x" data-dominant-color="3C1730"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1760×913 439 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here seqmap file is exactly copied from fulltrain.seqmap , gt_folder has Annotations in txt format and tracking_results is copied from TrackR-CNN tracking result. I have tried with Annotations in png format but getting the same error. Please let me know what I am doing wrong</p>
Masking gradient updates to weights using register_hook

username: Arvind_Subramaniam
cooked: <p>I am multiplying a mask to my weight matrix during the forward prop to sparsify my network. However, during backprop, I want the gradient updates to be zero for the masked weights (<code>mask[i] = 0</code>).</p>
<p>Is there any way to do that using register_hook?</p>
<p>Here’s a toy code I have used, but am getting an error.</p>
<pre><code class="lang-auto">mask = torch.tensor([0,1,0,1,0])

a.requires_grad = True

b = 3*a

b.retain_grad()

b.register_hook(mask)  

b.mean().backward() 

print(a.grad, b.grad)
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">---&gt;b.mean().backward()
    TypeError: 'Tensor' object is not callable
</code></pre>
<p>I have tried:</p>
<pre><code class="lang-auto">b.register_hook(lambda grad:grad*mask)
</code></pre>
<p>which works. Is there any other better way?</p>

username: albanD
cooked: <p>Hi,</p>
<p>A hook has to be a callable function, not a Tensor. So the lambda is the way to go.</p>

username: Arvind_Subramaniam
cooked: <p>Thanks a lot for your response. I have another question. Lets say I am introducing a parameter c like this:</p>
<pre><code class="lang-auto">a = torch.ones(5)
a.requires_grad = True
mask = torch.tensor([1,0,0,1,0])
b = 3*a
c = 2*b

b.retain_grad()
c.retain_grad()

b.register_hook(lambda grad: grad*mask)  

c.mean().backward() 
print(a.grad,b.grad,c.grad)
</code></pre>
<p>Here, only a.grad gets masked. Both a.grad and b.grad get masked only when I use</p>
<pre><code class="lang-auto">c.register_hook(lambda grad: grad*mask)  
</code></pre>
<p>Why is that? Shouldnt b.grad get masked even in the first scenario?</p>

username: albanD
cooked: <p>The thing is that the <code>retain_grad()</code> is also a hook <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> And hooks are executed in the order they are added. So if you add your masking after the retain_grad, the gradient seen by retain_grad won’t be masked.</p>

username: Arvind_Subramaniam
cooked: <p>Thanks a lot! Its clear now!</p>
Multiply one tensor of channel 1 with a tensors with multiple channel across all channel

username: jpainam
cooked: <p>Hi. I have a tensor <code>y</code> of shape <code>torch.Size([32, 1, 4, 2])</code> and <code>x</code> of shape <code>torch.Size([32, 2048, 4, 2])</code><br>
I want to multiply <code>x*y</code> across all <code>x</code> channel. I have tried</p>
<pre><code class="lang-python"># case 1
xy = x*y  # shape torch.Size([32, 2048, 4, 2])

# case 2
b, c, h, w = x.size()   # shape  torch.Size([32, 2048, 4, 2])
b_att, c_att, h_att, w_att = y.size()   # shape torch.Size([32, 1, 4, 2])
x = x.view(b, c, h_att, h // h_att, w_att, w // w_att)
xy= x * y.view(b, 1, h_att, 1, w_att, 1)  # shape torch.Size([32, 2048, 4, 1, 2, 1])
# Bring back the initial shape
xy = xy.view(b, c, h, w)  # shape  torch.Size([32, 2048, 4, 2])
</code></pre>
<p>The two methods don’t give any error. Which method is correct?<br>
I just want to make sure the multiplication is really done across all the channel of <code>x</code> because <code>y</code> is one channel<br>
Thank you</p>

username: ptrblck
cooked: <p>As both methods return the same result, I would prefer the first one.</p>
Elementwise_kernel kernel

username: mahmoodn
cooked: <p>I have profiled a gan network and see that the following kernel is the most important one:</p>
<pre><code class="lang-auto">_ZN2at6native18elementwise_kernelILi512ELi1EZNS0_15gpu_kernel_implIZZZNS0_15add_kernel_cudaERNS_14TensorIteratorEN3c106ScalarEENKUlvE_clEvENKUlvE2_clEvEUlffE_EEvS4_RKT_EUliE0_EEviT1_
</code></pre>
<p>Using <a href="http://demangler.com" rel="nofollow noopener">demangler.com</a>, the real name is</p>
<pre><code class="lang-auto">void at::native::elementwise_kernel&lt;512, 1, at::native::gpu_kernel_impl&lt;at::native::add_kernel_cuda(at::TensorIterator&amp;, c10::Scalar)::{lambda()#1}::operator()() const::{lambda()#4}::operator()() const::{lambda(float, float)#1}&gt;(at::TensorIterator&amp;, at::native::add_kernel_cuda(at::TensorIterator&amp;, c10::Scalar)::{lambda()#1}::operator()() const::{lambda()#4}::operator()() const::{lambda(float, float)#1} const&amp;)::{lambda(int)#2}&gt;(int, at::native::gpu_kernel_impl&lt;at::native::add_kernel_cuda(at::TensorIterator&amp;, c10::Scalar)::{lambda()#1}::operator()() const::{lambda()#4}::operator()() const::{lambda(float, float)#1}&gt;(at::TensorIterator&amp;, at::native::add_kernel_cuda(at::TensorIterator&amp;, c10::Scalar)::{lambda()#1}::operator()() const::{lambda()#4}::operator()() const::{lambda(float, float)#1} const&amp;)::{lambda(int)#2})
</code></pre>
<p>When I grep <code>elementwise_kernel</code>, the implementation is in <code>./aten/src/ATen/native/cuda/Loops.cuh</code> as below</p>
<pre><code class="lang-auto">template&lt;int nt, int vt, typename func_t&gt;
C10_LAUNCH_BOUNDS_2(nt, launch_bound2)
__global__ void elementwise_kernel(int N, func_t f) {
  int tid = threadIdx.x;
  int nv = nt * vt;
  int idx = nv * blockIdx.x + tid;
  #pragma unroll
  for (int i = 0; i &lt; vt; i++) {
    if (idx &lt; N) {
      f(idx);
      idx += nt;
    }
  }
}
</code></pre>
<p>So, the important thing is <code>func_t f</code>. From the demangler, I am confused with the function name. which one is correct?<br>
gpu_kernel_impl<br>
or<br>
add_kernel_cuda<br>
?</p>
<p>Any thought?</p>
Online Triplet Hard Mining

username: mmubeen
cooked: <p>Hello, I am trying to implement online hard/semi-hard triplet mining in pytorch. Below is my code: -</p>
<pre><code class="lang-auto">def mineHard(model, anchor, positive, negative, semiHard=False):
    cnn = model
    cnn.eval()
    margin = 0.3

    anchor, positive, negative = Variable(anchor).cuda(), Variable(positive).cuda(), Variable(negative).cuda()
    output1, output2, output3 = cnn(anchor, positive, negative)
    
    d_pos = F.pairwise_distance(output1, output2)
    d_neg = F.pairwise_distance(output1, output3)
    if semiHard:
        pred1 = (d_pos - d_neg).cpu().data
        pred2 = (d_pos + margin - d_neg).cpu().data
        indices = numpy.logical_and((pred1 &lt; 0), (pred2 &gt; 0))
    else:
        pred = (d_pos - d_neg).cpu().data
        indices = pred &gt; 0
    
    if indices.sum() == 0:
        return None, None, None, False

    x = torch.arange(0, d_pos.size()[0]).view(d_pos.size()[0], 1)
    indices = x.type(torch.cuda.FloatTensor) * indices.type(torch.cuda.FloatTensor)
    
    nonzero_indices = torch.nonzero(indices)
    indices = indices[nonzero_indices[:, 0], :].view(nonzero_indices.size()[0]).type(torch.cuda.LongTensor)
    
    anchor = torch.index_select(anchor.data, 0, indices)
    positive = torch.index_select(positive.data, 0, indices)
    negative = torch.index_select(negative.data, 0, indices)

    return anchor, positive, negative, True
</code></pre>
<p>Now, the problem is that whenever I try to train my network on these extracted examples, in the first epoch the performance starts to deteriorate, and then after the second epoch none of my triplets in validation set is correctly classified.<br>
Any help would be appreciated.</p>

username: chinmay5
cooked: <aside class="quote no-group" data-username="mmubeen" data-post="1" data-topic="16271">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmubeen/48/3987_2.png" class="avatar"> mmubeen:</div>
<blockquote>
<p>d_pos</p>
</blockquote>
</aside>
<p>Shouldn’t you do a L2 distance or similar distance based measure rather than a per element subtraction for selecting the indices?</p>
How to get cell states for each timestep from nn.LSTM?

username: Griffin_Adams
cooked: <p>I need both hidden and cell state vectors for each timestep.  The nn.LSTM function looks like it only returns hidden states for each timestep.  Thanks - this would be much appreciated <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Guohai93
cooked: <p>maybe you can step through the sequence one element at a time:</p>
<blockquote>
<p>lstm = nn.LSTM(3, 3)  # Input dim is 3, output dim is 3<br>
inputs = [autograd.Variable(torch.randn((1, 3))) for _ in range(5)]  # make a sequence of length 5<br>
hidden = (autograd.Variable(torch.randn(1, 1, 3)), autograd.Variable(torch.randn((1, 1, 3))))<br>
for i in inputs:<br>
# Step through the sequence one element at a time. after each step, hidden contains the hidden state.<br>
out, hidden = lstm(i.view(1, 1, -1), hidden)</p>
</blockquote>
<p>you can find the code from pytorch tutorial</p>

username: Jared_77
cooked: <p>This will slow down code a lot though. Is there any easier way? I know in Keras you can just do “return_sequences=True”. Is there something similar in Pytorch?</p>

username: Quella
cooked: <p>Hi, recently, I have the same problem. Do you get a better solution?</p>

username: Jared_77
cooked: <p>No, as far as I know there is no way to do this in Pytorch</p>

username: NotNANtoN
cooked: <p>I have exactly the same issue. I opened a feature request issue on Github: <a href="https://github.com/pytorch/pytorch/issues/31423" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/31423</a></p>
Parameters not updating after an indexing operation

username: v-prgmr
cooked: <p>Greetings,</p>
<p>I use a multivariate function which is a part of my forward pass of the model. I am debugging the forward pass for parameter updates. I have found that after a specific operation, i dont have gradients anymore. ( operation mentioned below).<br>
<strong>Can someone tell me why and guide me here</strong> ?</p>
<p><strong>Code in which the grads are zero</strong><br>
In the code below b is a tensor which is the output of a function which takes in the nn.parameters of nn.Module, and while debugging, i could see that b’s flag for “requires_grad” is True as it depends on the nn.Parameters.</p>
<pre><code class="lang-auto">a = torch.empty(leftImagetensor.shape, dtype=torch.float).to(device)
x = b[:, 0].long()    
y = b[:, 1].long()
a[y, x] = b[:, 3] * 256.0
return a #returning a here leads to zero grads and hence no parameter update 
</code></pre>
<p><strong>Working part of the code for which parameter update takes place:</strong></p>
<pre><code class="lang-auto">return b #returning b here and skipping the parts below yields grads and parameter updates 
a = torch.empty(leftImagetensor.shape, dtype=torch.float).to(device)
x = b[:, 0].long()    
y = b[:, 1].long()
a[y, x] = b[:, 3] * 256.0
return a 
</code></pre>
<p><strong>Train script</strong></p>
<pre><code class="lang-auto">model = Model().to(device)
print(model.state_dict())

lr = 1e-5
n_epochs = 5

optimiser = optim.sgd(model.parameters(), lr=lr)
L1 = torch.nn.L1Loss()
for epoch in range(n_epochs):
    model.train()

    yhat = model(left_imgtensor, Right_imgtensor, VeloScan, R_0, t_0, K_00, R_cam1_cam0, t_cam1_cam0, K_01)
    
    loss_fn = L1(yhat, y_target) #here y_target lives in the cuda, and yhat also lives in cuda
    print(loss_fn)
    loss_fn.backward()
    for p in model.parameters():
        if p.grad is not None:
            print(p.grad.data)
    optimiser.step()
    optimiser.zero_grad()
</code></pre>
Custom loss with Retain_graph= True slows down the code

username: Mina_soltan
cooked: <p>I have a model with two branches. Both branches share the embedding layer but then they diverge and each loss is calculated against different labels. At the end, I sum the two losses up and backward only the accumulated loss. However, without retailn_graph=True it throws an error asking for retain_graph to be set to True. However, when retaining the graph, the backward step is so slow that makes the training practically impossible. Any solution for this?</p>

username: tom
cooked: <p>Quote likely you are doing something wrong elsewhere and store a non-leaf (like <code>.to()</code> before storing).<br>
As a rule of thumb <code>retain_grad=True</code> is only for cases when you know exactly why you need it, not for when some error message tells you (maybe we should improve the message).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Mina_soltan
cooked: <p>Thanks so much for this answer! I kept focusing on the retain_graph but in fact I have a deeper issue in the way I was handling input output of my LSTM.</p>

username: nilinykh
cooked: <p>Could you please provide some more information on what was your issue and how you solved it? It might be helpful for my case.</p>
<p>I am having a similar problem (retain_graph=True causes my code to run very slow), and I assume it’s due to creating some extra non-leaf nodes as mentioned above (via .to() or cuda()) or maybe due to the fact that I do not repackage my hidden states (detach them after every batch, so that gradient computation does not go to the start of the training)…</p>
Checkpointing for a dataparallel model

username: Shubhankar
cooked: <p>Does it make a difference if you checkpoint your model for retraining after model.eval() or model.train() loop?</p>

username: ptrblck
cooked: <p>It shouldn’t make any difference, as long as you don’t update the parameters in your validation loop.</p>
<p>This question seems to be unrelated to the topic, so do you have any issues using <code>DataParallel</code>?</p>

username: Shubhankar
cooked: <p>I am trying to save a Dataparallel model but getting</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-17-e0020f404c69&gt; in &lt;module&gt;()
     69             optimizer.zero_grad() # Zero gradients
     70             loss.backward() # Calculate gradients
---&gt; 71             optimizer.step() # Update weights
     72             m.track_loss(loss)
     73             m.track_num_correct(preds, labels)

~/pytorch-1.0-p3/anaconda3/lib/python3.6/site-packages/torch/optim/adamw.py in step(self, closure)
     98 
     99                 # Decay the first and second moment running average coefficient
--&gt; 100                 exp_avg.mul_(beta1).add_(1 - beta1, grad)
    101                 exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
    102                 if amsgrad:

RuntimeError: expected device cpu but got device cuda:0
</code></pre>

username: ptrblck
cooked: <p>The stack trace points to <code>optimizer.step()</code>, which is unrelated to saving the <code>state_dict</code>.</p>
<p>How did you pass the parameters to the optimizer?</p>

username: Shubhankar
cooked: <p>So this worked for me which is little weird of a workflow:</p>
<pre><code class="lang-auto"># While saving checkpoint i.e. comment out while loading checkpoint
    if torch.cuda.device_count() &gt; 1:
        print("Let's use", torch.cuda.device_count(), "GPUs!")
        network = nn.DataParallel(network)
    
    
    network.to(device)
    optimizer = optim.AdamW(network.parameters(), lr=run.lr, weight_decay=run.weight_decay)
    
#     try:
    with open('check-point.pth', 'rb') as f:
        print('file opened')
        checkpoint = torch.load(f)
        print('file loaded')
        network.load_state_dict(checkpoint['model_state_dict'])
        print('network loaded')
        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
        print('optimizer loaded')
        epoc = checkpoint['epoch']
        print(f'blah epoc: {epoc}')    
    # While loading checkpoint
        if torch.cuda.device_count() &gt; 1:
            print("Let's use", torch.cuda.device_count(), "GPUs!")
            network = nn.DataParallel(network)
</code></pre>
How to access CrossEntropyLoss() gradient?

username: Umair_Javaid
cooked: <p>I want to modify the tensor that stores the CrossEntropyLoss() gradient, that is, P(i)-T(i). Where is it stored and how do I access it?</p>
<p><strong>code:</strong></p>
<pre><code>input = torch.randn(3, 5, requires_grad=True)
input.register_hook(lambda x: print(" \n input hook: ",x))
print(input)
target = torch.empty(3, dtype=torch.long).random_(5)
print(target)

criterion = nn.CrossEntropyLoss()
criterion.requires_grad = True
loss0 = criterion(input,target)
loss0.register_hook(lambda x: print(" \n loss0 hook: ",x))
print("before backward loss0.grad :",loss0.grad)
print("loss0 :",loss0)
loss0.backward()
print("after backward loss0.grad :",loss0.grad)
</code></pre>
<p><strong>output:</strong></p>
<pre><code>tensor([[-0.6149, -0.8179,  0.6084, -0.2837, -0.5316],
        [ 1.7246,  0.5348,  1.3646, -0.7148, -0.3421],
        [-0.3478, -0.6732, -0.7610, -1.0381, -0.5570]], requires_grad=True)
tensor([4, 1, 0])
before backward loss0.grad : None
loss0 : tensor(1.7500, grad_fn=&lt;NllLossBackward&gt;)
 
 loss0 hook:  tensor(1.)
 
 input hook:  tensor([[ 0.0433,  0.0354,  0.1472,  0.0603, -0.2862],
        [ 0.1504, -0.2876,  0.1050,  0.0131,  0.0190],
        [-0.2432,  0.0651,  0.0597,  0.0452,  0.0732]])
after backward loss0.grad : None</code></pre>
user_suspended: yes

username: JuanFMontesinos
cooked: <p>The gradient is input.grad<br>
What do you understand by loss.grad?<br>
input.grad is gradient of loss wrt input which is the cross entropy gradient.<br>
there is no loss.grad as it is not involved in further opts</p>
<pre><code class="lang-auto">import torch

input = torch.randn(3, 5, requires_grad=True)
input.register_hook(lambda x: print(" \n input hook: ",x))
print(input)
target = torch.empty(3, dtype=torch.long).random_(5)
print(target)

criterion = torch.nn.CrossEntropyLoss()
criterion.requires_grad = True
loss0 = criterion(input,target)
loss0.register_hook(lambda x: print(" \n loss0 hook: ",x))
loss1 = 1*loss0
print("before backward loss0.grad :",loss0.grad)
print("loss0 :",loss0)
loss1.backward()
print("after backward loss0.grad :",loss0.grad)
</code></pre>
<p>what you can do is to register a backward hook in criterion rather than inspecting tensors or just do the identity op multiplying by 1 to see each term grad</p>
Strange Out of memory when running image detection tutorial

username: gilles
cooked: <p>Hello,</p>
<p>PT newbie here. I’m in the process of learning PT for computer vision; I’m primarily interested in image detection use cases. I use a Quadro P600 GPU with Ubuntu 18.</p>
<p>I have a OOM problem with the following tutorial : <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>I just copy the code in a Jupyter notebook and run it.</p>
<p>It rapidly throws a OOM exception when instantiating the model at the line<br>
<code>loss_dict = model(images, targets)</code><br>
in the <code>train_one_epoch function</code> (I took it from here <a href="https://github.com/pytorch/vision/tree/master/references/detection" rel="nofollow noopener">https://github.com/pytorch/vision/tree/master/references/detection</a>)</p>
<p>If I force the execution to be done on cpu, I also get a OOM with a bit more details. It says that it tries to allocate about 22GB of memory.</p>
<p>This is a huge amount of memory. Is this really what is expected ? Am I doing something wrong ?</p>
<p>Many thanks by advance for any help you could provide.</p>

username: ptrblck
cooked: <p>If I’m not mistaken, your GPU should have only 2GB memory?<br>
If so, it might not be enough for this tutorial.<br>
Could you try to run in on Colab and check the memory usage on their GPUs?</p>

username: gilles
cooked: <p>Just tried. It works and Collab says that 2.2GB of RAM is used. I think you’re right, I need a GPU with more RAM… or maybe just use collab !</p>
<p>Many thanks for your help.</p>
Parameterizing Cholesky factors of full rank covariances

username: jbird
cooked: <p>So I’m trying to parameterize a set of full-rank covariance or precision matrices for, eg., Gaussian mixture models or mixture density networks.</p>
<p>I want to define torch.nn.Parameters for the (log) diagonals and lower triangular parts of the Cholesky factors respectively, and then package those parameters together into the Cholesky factors (and then covariance or precision matrices).</p>
<p>In tensorflow I can do this easily with tf.distributions.fill_triangular(), but as far as I can tell no such function exists in torch. What’s the easiest way to achieve something equivalent in torch?</p>

username: christianperone
cooked: <p>You can use the same trick described in tensorflow <code>fill_triangular</code>, but converted to PyTorch, here is an example for the lower triangular of a 3x3 matrix:</p>
<pre><code>x = torch.arange(6) + 1 
xc = torch.cat([x[3:], x.flip(dims=[0])])
y = xc.view(3, 3)
torch.tril(y)

tensor([[4, 0, 0],
        [6, 5, 0], 
        [3, 2, 1]])</code></pre>
Does pytorch provide ZCA matrix for CIFAR10 and 100?

username: BarCodeReader
cooked: <p>Hi,</p>
<p>as titled, recently I am looking into preprocessing and want to try some whitening methods.</p>
<p>ZCA sounds good to me but my hardware cannot support the calculation of matrix multiplication of [50000, 3072]</p>
<p>so I am wondering…does pytorch provide ZCA function? or ZCA matrix? …or is there a way to estimate the ZCA matrix for the whole dataset?</p>
<p>P.S: I have found a github repo providing CIFAR10 ZCA matrix(though not sure if it is accurate or not), i am still looking for CIFAR100 ZCA matrix and wondering how people will handle dataset like imagenet with few million images???</p>
<p>thanks in advance</p>
Pytorch:optimizer got an empty parameter list

username: lhsICT
cooked: <p>I’m practicing pytorch with an Example from an Internet blog.<br>
But, I have a problem with module.nn.parameters() version is python 3.7</p>
<p>import torch<br>
from torch.autograd import Variable</p>
<p>x_data=torch.tensor([[1.0],[2.0],[3.0]])<br>
y_data=torch.tensor([[2.0],[4.0],[6.0]])</p>
<p>class MyModel(torch.nn.Module):<br>
def <em>init</em>(self):<br>
super(MyModel,self).<em>init</em>()<br>
self.linear=torch.nn.Linear(1,1)</p>
<pre><code>def forward(self,x):
    y_pred=self.linear(x)
    return y_pred
</code></pre>
<p>model=MyModel()</p>
<p>criterion=torch.nn.MSELoss(size_average=False)<br>
optimizer=torch.optim.SGD(list(model.parameters()),lr=0.01)</p>
<p>for epoch in range(501):<br>
y_pred=model(x_data)<br>
loss=criterion(y_pred,y_data)<br>
if epoch%100==0:<br>
print(epoch,loss.data[0])<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>

username: ptrblck
cooked: <p>Your code is working fine, besides some minor issues:</p>
<ul>
<li>
<code>Variables</code> are deprecated since PyTorch <code>0.4.0</code>, so use tensors instead</li>
<li>use <code>loss.item()</code> to print our the loss value instead of <code>.data</code>
</li>
</ul>

username: lhsICT
cooked: <p>It was my mistyping. Now I fixed them(I used just a single underbar at <em>init</em>).<br>
Sorry for my mistake and thank you so much for your feedback <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Custom discrete distribution on GPU

username: Teja_Aluru
cooked: <p>I am trying to generate a random matrix that is filled with a discrete distribution with three values. In my case, my distribution looks like -1 with probability 1/4, +1 with probability -1/4, and 0 with probability 1/2. Is there an elegant way to do this in Pytorch? I am working with the C++ frontend, but would be willing to switch to the python frontend to get this working.</p>

username: beaupreda
cooked: <p>Hello,</p>
<p>I do not know if my solution is elegant, but I believe it is a way to do what you want to do!</p>
<pre><code class="lang-auto">    x = torch.tensor([-1, 0, 0, 1], dtype=torch.float32)
    print(x)
    y = torch.zeros(size=(10, 10), dtype=torch.float32)
    for i in range(y.shape[0]):
        for j in range(y.shape[1]):
            choice = torch.randint(high=x.shape[0], size=(1, ))
            y[i, j] = x[choice]
    print(y)
    print(torch.sum((y == 1.0)))
    print(torch.sum((y == 0.0)))
    print(torch.sum((y == -1.0)))
</code></pre>
<p>Good luck!</p>
Adaptive Average Pooling 2d with relative output size

username: pete7.62
cooked: <p>Is it possible to use <strong>Adaptive</strong> Average Pooling 2d not with a fixed, absolute output shape, but instead make it halve the image dimensions instead?</p>
<p>I would like the feature extraction to be fully convolutional and work on any image shape.</p>

username: pete7.62
cooked: <p>Solution:<br>
Use the adaptive average pooling in the <strong>forward pass</strong>, where you have the shape information of the input available and can set the output shape to half the input shape.</p>
<p>In the original torchvision resnet50 implementation the adaptive average pooling layer is created during class initialization and I didn’t think of the idea that it doesn’t have to be this way.</p>
What to do if your model ignores the input and learns the labels?

username: synchro
cooked: <p>Hi everyone,</p>
<p>I’m working on this time-series regression problem and I’ve already gone through the following stages:</p>
<ul>
<li>
<p>prepared different datasets by adding first only the series itself, then moving averages, then sentiment data, etc;</p>
</li>
<li>
<p>trained benchmarks: persistent models, linear regressions, ARIMA, …</p>
</li>
<li>
<p>tried a variety of different deep learning archtitectures (MLPs, resnets, wavenets, lstm, etc.)</p>
</li>
</ul>
<p>So, what happens is that no matter (i) how the dataset is built and (ii) how complex or fancy the architecture is but the model always end up ignoring the input and predicting as output at timestep t the input a timestep t-1, which is called a persistent model in literature and (it’s one of the benchmarks)</p>
<p>TL; DR:</p>
<p>Time series framing problem: DL models (of several architectures) end up totally ignoring the input and learn to give always the same prediction: ŷ(t) = x(t-1)</p>
<p>Q: How to address this issue? Is there a way to penalise this behaviour during the training?</p>
Use skimage transforms in PyTorch Transforms

username: pedro
cooked: <p>I want to apply <a href="https://scikit-image.org/docs/dev/api/skimage.feature.html#skimage.feature.local_binary_pattern" rel="nofollow noopener">skimage’s Local Binary Pattern transformation</a> on my data, and was wondering if there was any possibility of doing this inside my torch’s Transforms, which right now is the following:</p>
<pre><code class="lang-auto">data_transforms = {
    'train': transforms.Compose([
        transforms.CenterCrop(178),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ]),
    'val': transforms.Compose([
        transforms.CenterCrop(178),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ]),
}
</code></pre>
<p>If not, how would I implement it?</p>

username: ptrblck
cooked: <p>If you want to add the <code>skimage</code> transformation between other <code>Image</code> transformation, you could use the following code snippet as a base class and apply the desired transformation inside:</p>
<pre><code class="lang-python">class MySkimageTransform(object):
    def __init__(self):
        '''
        initialize your transformation here, if necessary
        '''
        

    def __call__(self, pic):
        arr = np.array(pic) # transform to np array
        # apply your transformation
        pic = Image.from_array(arr)
        return pic
</code></pre>

username: pedro
cooked: <p>Thanks for your help! It worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
CUDA Error: no kernel image is available on cuda capability==3,7

username: kl_divergence
cooked: <p>Pytorch 1.3.1 binaries only support GPUs with CUDA capabilty&gt;=3.7 as clarified by <a class="mention" href="/u/ptrblck">@ptrblck</a> <a href="https://discuss.pytorch.org/t/cuda-error-no-kernel-image-is-available-for-execution-on-the-device/64088">here</a></p>
<p>When using NMS from torchvision.ops on K80 (3,7), I’m getting this error:</p>
<pre><code class="lang-auto">CUDA error: no kernel image is available for execution on the device (nms_cuda at /tmp/pip-req-build-9d9zypi6/torchvision/csrc/cuda/nms_cuda.cu:127)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) + 0x6d (0x7f096ecbce7d in /opt/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: nms_cuda(at::Tensor const&amp;, at::Tensor const&amp;, float) + 0x8d1 (0x7f0918a32ece in /opt/anaconda3/lib/python3.7/site-packages/torchvision/_C.so)
frame #2: nms(at::Tensor const&amp;, at::Tensor const&amp;, float) + 0x183 (0x7f09189f6ed7 in /opt/anaconda3/lib/python3.7/site-packages/torchvision/_C.so)
</code></pre>
<p>Is this a bug/unintended behaviour because it should support it ? Am I missing something ?</p>
Split dataset in validation and training data

username: Hannes_De_Smet
cooked: <p>Hi All,</p>
<p>So I have a dataset with 411 classes (different persons) and a network that should try to recognise these subjects based on their iris (iris recognition system). But before I train I need to split up my images in validation and training sets. I Have around 20 images per person so I want do distribute it quite evenly thus that for each person (each class) 80% is used for training and 20% for validation.</p>
<p>How do I do this?<br>
Thanks in Advance</p>

username: kl_divergence
cooked: <p>Well after you have created the Dataset, you can use <code>sklearn.model_selection.train_test_split</code> to split the dataset for you. More details can be found <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html" rel="nofollow noopener">here</a>.</p>

username: Hannes_De_Smet
cooked: <p>So I tried using it, so I wrote the following: `<br>
dataset = torchvision.datasets.ImageFolder(DATA_PATH, transform=transform)<br>
strat = np.arange(1,classes)<br>
train_dataset, validate_dataset = train_test_split(dataset,train_size=0.8,stratify=strat)</p>
<p><span class="hashtag">#Data</span> loader<br>
train_loader = torch.utils.data.DataLoader(train_dataset,batch_size,shuffle=False)<br>
validate_loader = torch.utils.data.DataLoader(validate_dataset, batch_size, shuffle=False)`</p>
<p>But when I try to run it I get the following error: <code>Traceback (most recent call last):   File "/Users/HannesDeSmet/Documents/Unif/Thesis/Prototype/Recognition/Recognition.py", line 31, in &lt;module&gt;     train_dataset, validate_dataset = train_test_split(dataset,train_size=0.8,stratify=strat)   File "/Users/HannesDeSmet/Documents/Unif/Thesis/Prototype/Recognition/venv/lib/python3.6/site-packages/sklearn/model_selection/_split.py", line 2141, in train_test_split     train, test = next(cv.split(X=arrays[0], y=stratify))   File "/Users/HannesDeSmet/Documents/Unif/Thesis/Prototype/Recognition/venv/lib/python3.6/site-packages/sklearn/model_selection/_split.py", line 1328, in split     X, y, groups = indexable(X, y, groups)   File "/Users/HannesDeSmet/Documents/Unif/Thesis/Prototype/Recognition/venv/lib/python3.6/site-packages/sklearn/utils/validation.py", line 237, in indexable     check_consistent_length(*result)   File "/Users/HannesDeSmet/Documents/Unif/Thesis/Prototype/Recognition/venv/lib/python3.6/site-packages/sklearn/utils/validation.py", line 212, in check_consistent_length     " samples: %r" % [int(l) for l in lengths]) ValueError: Found input variables with inconsistent numbers of samples: [16082, 410]</code></p>
<p>For the record I have 16082 images with 410 different classes/ subjects</p>
Pytorch 1.1.0 gave different results comparing with pytorch 1.0.0?

username: Catherine_Fang
cooked: <p>Hello,<br>
I train a CNN.<br>
Experiment1: pytorch 1.1.0, loss is NaN<br>
Experiment2: pytorch 1.0.0, loss is around 1.0</p>
<p>What are the differents between pytorch 1.1.0 and pytorch 1.0.0?<br>
I think it is related to the initialization method.</p>
<p>Any other thoughts? Thanks!</p>

username: ptrblck
cooked: <p>It’s hard to tell without seeing any code.<br>
Could you update to the latest stable release (<code>1.3.1</code>) and check, if you still get a NaN loss?</p>

username: Catherine_Fang
cooked: <p>I tried pytorch 1.2.0, the loss is also around 1.0 .</p>

username: ptrblck
cooked: <p>You might have hit a legacy bug. If your code is working with the current version, I wouldn’t care too much about it. Is there a specific reason you need to use older PyTorch versions? I would still recommend to use the latest release.</p>
How to build a custom connections

username: Rojin
cooked: <p>I have an adjacency matrix and based on that I would like to build a costume connection between the input and the first hidden layer. I was wondering if someone can help me understand how I should go about the implementation?</p>

username: ptrblck
cooked: <p>Does your custom adjacency matrix have the same shape as the weight matrix? Would it be possible to mask the connections with a multiplication (if your matrix contains ones and zeros of course)?</p>

username: Rojin
cooked: <p>Yes, lets say that my input has 100 features, and the first hidden layer has 50 nodes. And my adjacency matrix has the following shape: 100x50  and it’s zero or one. Would it be enough to multiply the weight matrix by the adjacency to remove someone the connections and not let the any backpropagation through the nodes that have 0 adjacency?</p>

username: ptrblck
cooked: <p>The gradients will be zero at the masked points:</p>
<pre><code class="lang-python">x = torch.randn(1, 10)
weight = nn.Parameter(torch.randn(5, 10))
adj = torch.randint(0, 2, (5, 10)).float()

out = torch.matmul(x, (adj * weight).t())
out.mean().backward()

print(weight.grad)
&gt; tensor([[ 0.1776, -0.1401,  0.0000, -0.0000,  0.0000,  0.0000,  0.0000, -0.0000,
         -0.0000, -0.1063],
        [ 0.1776, -0.0000,  0.0133, -0.0294,  0.0000,  0.3073,  0.1285, -0.0000,
         -0.0000, -0.0000],
        [ 0.1776, -0.0000,  0.0000, -0.0000,  0.0000,  0.3073,  0.1285, -0.0000,
         -0.0000, -0.0000],
        [ 0.1776, -0.1401,  0.0000, -0.0000,  0.0944,  0.3073,  0.1285, -0.1295,
         -0.0000, -0.1063],
        [ 0.1776, -0.1401,  0.0133, -0.0000,  0.0944,  0.0000,  0.0000, -0.1295,
         -0.1472, -0.1063]])
print(adj)
&gt; tensor([[1., 1., 0., 0., 0., 0., 0., 0., 0., 1.],
        [1., 0., 1., 1., 0., 1., 1., 0., 0., 0.],
        [1., 0., 0., 0., 0., 1., 1., 0., 0., 0.],
        [1., 1., 0., 0., 1., 1., 1., 1., 0., 1.],
        [1., 1., 1., 0., 1., 0., 0., 1., 1., 1.]])
</code></pre>
<p>and won’t add any information to the gradient which flows to the preceding layer, if that’s what you are looking for.</p>
How to use transfer learning correctly (model zoo)?

username: Raul-Ronald_Galea
cooked: <p>I have a couple things I’d like to ask about the proper usage of the pretrained models offered by pytorch. (I’m trying to build an SSD detection model with a pretrained MobileNetV2 as backbone.)</p>
<ol>
<li>It is mentioned in the docs that pretrained models expect inputs</li>
</ol>
<blockquote>
<p>to be loaded in to a range of [0, 1] and then normalized using  <code>mean = [0.485, 0.456, 0.406]</code>  and  <code>std = [0.229, 0.224, 0.225]</code></p>
</blockquote>
<p>I believe these hardcoded values are the means and stds per channel of images in the ImageNet dataset. However, as I will be training on COCO, should these be changed to the means and stds of the COCO dataset? (or, in general, the current training set?)</p>
<ol start="2">
<li>I am currently freezing the backbone for the first few epochs, then unfreeze and train the whole model. However, as soon as I set the requires_grad flag in the params of the backbone to True, the loss rises by a significant amount, and then it can’t really recover anymore. What could cause this behaviour and how should I try to solve it?<br>
Possibilities I am currently considering are:</li>
</ol>
<ul>
<li>the learning rate is too high for the backbone (as it may have been trained with a lower one), and the large gradients ruin the weights when they get unfrozen.</li>
<li>I am not normalizing the inputs correctly</li>
</ul>
<p>Should I try to lower the learning rate before unfreezing, or perhaps unfreezing progressively, or what other solutions are there?</p>
<p>Many thanks</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Raul-Ronald_Galea" data-post="1" data-topic="64364">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raul-ronald_galea/48/12519_2.png" class="avatar"> Raul-Ronald_Galea:</div>
<blockquote>
<p>I believe these hardcoded values are the means and stds per channel of images in the ImageNet dataset. However, as I will be training on COCO, should these be changed to the means and stds of the COCO dataset? (or, in general, the current training set?)</p>
</blockquote>
</aside>
<p>Yes, these values are taken from ImageNet.<br>
If you are retraining using data from another domain, you could adapt these values (or recalculate them using your dataset). It’s hard to tell, if you’ll see any performance improvement and you would have to run some experiments.</p>
<aside class="quote no-group" data-username="Raul-Ronald_Galea" data-post="1" data-topic="64364">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raul-ronald_galea/48/12519_2.png" class="avatar"> Raul-Ronald_Galea:</div>
<blockquote>
<p>Should I try to lower the learning rate before unfreezing, or perhaps unfreezing progressively, or what other solutions are there?</p>
</blockquote>
</aside>
<p>A high learning rate sounds reasonable, so I would also recommend to lower it.</p>
Build pytorch from source error

username: Dale_Song
cooked: <p>hi,<br>
when building pytorch according to <a href="https://github.com/pytorch/pytorch#from-source" rel="nofollow noopener">https://github.com/pytorch/pytorch#from-source</a>,<br>
I always get an error</p>
<pre><code class="lang-auto">xxx/lib/libmagma.a(error.cpp.o): unrecognized relocation (0x2a) in section .text
</code></pre>
<p>What’s the reason?<br>
My building steps:</p>
<pre><code class="lang-auto">conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi
conda install -c pytorch magma-cuda100 #for cuda10.0
git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
# if you are updating an existing checkout
git submodule sync 
git submodule update --init --recursive
# my source code version is f9010d764894f3ba9bc6a129213a0faf42af9b65
export USE_NINJA=OFF
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py build
</code></pre>

username: Dale_Song
cooked: <aside class="quote no-group" data-username="Dale_Song" data-post="1" data-topic="64331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dale_song/48/13706_2.png" class="avatar"> Dale_Song:</div>
<blockquote>
<p>f9010d764894f3ba9bc6a129213a0faf42af9b65</p>
</blockquote>
</aside>
<p>solved by upgrading binutils from 2.23 to 2.27</p>
What is the difference between multiple classification binary classification in pytorch code？

username: FOXaaFOX
cooked: <p>I have already done some Text classification tasks with pytorch.</p>
<p>But when I review my  code ，I found that they are basically the same except the loss function(CrossEntropyLoss, BCEWithLogitsLoss)</p>
<p>In my opinion, at least the activation function should be different(softmax ,sigmoid)</p>
<p>So  who can tell me the difference between multiple classification and binary classification in pytorch code？</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> is used for multi-class classification use cases, i.e. each sample belongs to only one class (out of multiple classes). <code>nn.BCEWithLogitsLoss</code> is used for a binary classification use case, i.e. each sample belongs to the positive or negative class (or something in between would even be possible), or for multi-label classification use cases, i.e. each samples belongs to zero, one, or multiple classes.</p>
_register_state_dict_hook is not supported on ScriptModules

username: kl_divergence
cooked: <p>I’m quite new to torch.jit and Apex. I’m getting this error. What does it mean ?</p>
<pre><code class="lang-auto">RuntimeError: _register_state_dict_hook is not supported on ScriptModules
</code></pre>
<p>Here’s my code</p>
<pre><code class="lang-auto">model, optimizer = amp.initialize(model, optimizer,opt_level='O2',
                                          keep_batchnorm_fp32=False,
                                          loss_scale=config.TRAIN.FP16_LOSS_SCALE,
                                          min_loss_scale=128.0)
</code></pre>

username: zdevito
cooked: <p>state_dict_hook is an internal implementation that allows some modules to edit their state_dict before they get returned. torch.jit doesn’t generally support any hooks, including state_dict hooks, so it is possible that Apex is installing these hooks on a jit module and then it is failing. However, I do not know the details of how Apex works, so I am not sure that is precisely what is happening.</p>

username: ptrblck
cooked: <p>Scripting is currently not supported in <code>apex</code>, but tracing should work.<br>
We are currently working on upstreaming mixed-precision training, so that you won’t need to build <code>apex</code> soon. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Achieving different results when I use the nn.functional.upsample?

username: yunpei
cooked: <p>Recently, I utilize the nn.functional.upsample function, and I set all the seed, cudnn deterministic the same, just as follows:</p>
<p><strong>random.seed(config.seed)</strong><br>
<strong>np.random.seed(config.seed)</strong><br>
<strong>torch.manual_seed(config.seed)</strong><br>
<strong>torch.cuda.manual_seed_all(config.seed)</strong><br>
<strong>torch.cuda.manual_seed(config.seed)</strong><br>
<strong>torch.backends.cudnn.benchmark = False</strong><br>
<strong>torch.backends.cudnn.deterministic = True</strong></p>
<p>I use the same code and train the code twice, but I got two <strong>different results</strong>. And finally, I find the reason is casued by the <strong>upsample</strong> function.</p>
<p>The torch version is 0.4.0, and cuda8.0.</p>
<p>So, how should I solve the problem? Thank you very much.</p>

username: ptrblck
cooked: <p>Could you update PyTorch and then post a reproducible code snippet?<br>
I know that upsample was non-deterministic in the past, but that might have changed, as this code returns the same results:</p>
<pre><code class="lang-python">torch.manual_seed(2809)
x = torch.randn(2, 3, 24, 24).cuda()

up = torch.nn.functional.upsample(x, size=(100, 100))

for _ in range(10):
    print((up - torch.nn.functional.upsample(x, size=(100, 100))).abs().max())
</code></pre>
The Tensor.cpu() means a copy of original Tensor?

username: Weng_zhiqiang
cooked: <p>Hi everyone,<br>
There is my question, If I call a function which try to use Tensor.cpu() to make this Tensor can by change to Numpy after I have used Tensor.gpu(). And I won’t use the Tensor tackled in the function which is out of the main function.(The Tensor.gpu() is called in the main function). Will this operation change the Tensor in main function, which means the Tensor will be change to use cpu?So, the whole code may like this:</p>
<p>def change2cpu(x):<br>
a = Tensor.cpu()<br>
…</p>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
for bz_idx, x in enmerate(data):<br>
…<br>
a = Tensor.gpu()<br>
…<br>
change2cpu(a)</p>
Object detection quantization

username: amin_sabet
cooked: <p>I followed the quantization steps to qunatized faster-rcnn, but I got the following errors in the RegionProposalNetwork part of the network when calling the box_coder.decode(pred_bbox_deltas.detach(), anchors) function.</p>
<p>Could not run ‘aten::empty_strided’ with arguments from the ‘QuantizedCPUTensorId’ backend. ‘aten::empty_strided’ is only available for these backends: [CPUTensorId, VariableTensorId]</p>
<p>Could not run ‘aten::div.Tensor’ with arguments from the ‘QuantizedCPUTensorId’ backend. ‘aten::div.Tensor’ is only available for these backends: [CPUTensorId, SparseCPUTensorId, VariableTensorId].</p>

username: jerryzh168
cooked: <p>let’s track the progress here: <a href="https://github.com/pytorch/pytorch/issues/31316" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/31316</a></p>