akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch equivalent function for np.minimum.at or np.maximum.at

hofingermarkus: <p>Hi,</p>
<p>I want to prototype a function in pytorch and in order to achieve it, I need a function that behaves similar to scatter_add but for the maximum instead of add.<br>
So I do have a list of indices and according values - both large, multidimensional and on GPU.<br>
Some indices will collide – this is the crucial part! – and in such a case I want the maximum to be kept.<br>
(Of course the minimum would also work just fine).<br>
It would also be great to have this for argmax, argmin as well.</p>
<p>Here is a minimal example in numpy:</p>
<pre><code class="lang-python">import numpy as np

ids  = np.array([1,2,3,4,1,2,3,1])   # There are multiple 1s here
vals = np.array([0,0,0,1,2,3,1,1])  # 0,2,1 will written at index 1
maxs = np.zeros_like(vals)
avgs = np.zeros_like(vals)

np.maximum.at(maxs,ids,vals)
np.add.at(avgs,ids,vals)
print (maxs)
print (avgs)

#ids:0 1 2 3 4 5 6 7
&gt;&gt;&gt; [0 2 3 1 1 0 0 0]
&gt;&gt;&gt; [0 3 3 1 1 0 0 0]
</code></pre>

hofingermarkus: <p>Found a solution myself:</p>
<p>There is as library called <a href="https://github.com/rusty1s/pytorch_scatter" rel="nofollow noopener"><code>pytorch_scatter</code></a> that provides many different scatter operations (<code>add</code>, <code>div</code>, <code>max</code>, <code>mean</code>, <code>min</code>, <code>mul</code>, <code>std</code>). The <code>scatter_max</code> returns values and indices which allows to also directly use it for <code>argmax</code> operations. The operations also come with the according backwards paths.</p>
<p><a href="https://pytorch-scatter.readthedocs.io/en/latest/functions/max.html" rel="nofollow noopener"><img src="https://raw.githubusercontent.com/rusty1s/pytorch_scatter/master/docs/source/_figures/max.svg?sanitize=true" alt width="193" height="102"></a></p>
How to drop gradient information of `nn.Module`?

zeakey: <p>I’m playing some filter pruning tricks after training a model util convergence.</p>
<p>Firstly a model is trained with L1 regularization on BN weights.</p>
<p>Then I manually prune several kernels and BN parameters according to the BN weights:</p>
<pre><code class="lang-auto">conv.weight.data = conv.weight.data[pruned_index]
bn.weight.data = bn.weight.data[pruned_index]
bn.bias.data = bn.bias.data[pruned_index]
bn.running_mean.data = bn.running_mean.data[pruned_index]
bn.running_var.data = bn.running_var.data[pruned_index]
</code></pre>
<p>When I finetune the pruned model,<br>
an error occured:<br>
<code>RuntimeError: Function CudnnBatchNormBackward returned an invalid gradient at index 1 - got [3] but expected shape compatible with [512]</code></p>
<p>Since the <strong>train</strong> -&gt; <strong>prune</strong> -&gt; <strong>finetune</strong> procedure is done in a single script, there<br>
may be gradient information (shape of gradients and parameters) saved after training the model.</p>
<p>I think this is where this error comes from.</p>
<p>Currently I solve this problem by saving the model parameters to <code>checkpoint.pth</code> and then reload a new model from checkpoint, so that the gradient information is cleared:</p>
<pre><code class="lang-auto">model = resnet18()

# train model
train_model(model)

# save checkpoint
torch.save(model.state_dict(), "checkpoint.pth")

# reload
model = resnet18()
model.load_state_dict(torch.load("checkpoint"))

# prune
prune_model(model)

# finetune
finetune(model)
</code></pre>
<p>So is there any way to easily remove all gradient information?<br>
My current solution is not somehow elegant.</p>

ptrblck: <p>I haven’t tested it, but manipulating the <code>.data</code> attribute might have weird side effects and is not recommended.<br>
Could you instead warp the pruning into a <code>torch.no_grad()</code> block and reassign the pruned parameters?</p>

zeakey: <p>Wrapping the pruning into <code>torch.no_grad()</code> solved my problem.<br>
Thanks!</p>
Major Issue when doing multip gpu training (Loss gets worse)

Rafael_R: <p>Hi,</p>
<p>I had my code set for a single gpu. I used data parallel to train on multiple gpus. I am on a 4 gpu machine.</p>
<p>But now in training my losses are very large and my perplexity is just not going down from an extremely large value.</p>
<p>I want to ask:</p>
<ol>
<li>What changes do I need in batch size?</li>
</ol>
<p>I see only the zeroth gpu using large memory while the rest of the gpus have under utilized memory.</p>
<ol start="2">
<li>What changes do I need to make in learning rate?</li>
</ol>
<p>I read somewhere I should increase my learning rate, but why?</p>
<ol start="3">
<li>What is happening here in backprop? Are gradients accumulated in the zeroth gpu?</li>
</ol>
<p>Please let me know <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> AWS bills are rising <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
How to cut Tensor in Libtorch like python

NickKao: <pre><code>tensor1 = tensor[0,:,:,0]
tensor2 = tensor[0,:,:,1]</code></pre>

albanD: <p>Hi,</p>
<p>You can use the select method to get a single index on a given dimension and the narrow method to get a slice of a tensor on a given dimension</p>

NickKao: <p>Thank you.<br>
This is my way:<br>
torch::Tensor score_text = score.slice(2,0,1,1).squeeze(2);</p>
Recreating nn.BatchNorm1d() manually given it's trained parameters

abhi-iyer: <p>How does <code>nn.BatchNorm1d()</code> work in eval() mode? I looked up the source code, but this points to <code>F.batch_norm</code>, but this points to <code>torch.batch_norm</code> (which has no file I can look up).</p>
<p>Fundamentally, I want do something very simple.</p>
<p>Suppose, I have input <code>x</code>, which is passed into <code>nn.BatchNorm1d(x)</code> to get <code>y1</code> during eval() mode. I want to recreate <code>y</code> by using the parameters (<code>gamma</code>, <code>bias</code>, <code>running_mean</code>, <code>running_var</code> - where <code>gamma</code> and <code>bias</code> are stored inside the module’s <code>parameters</code> variable) that are inside nn.BatchNorm1d via the following:</p>
<p><code>y2</code> = <code>gamma</code> * (<code>x</code> - <code>running_mean</code>) / (sqrt(<code>running_var</code>)) + <code>bias</code></p>
<p>However, <code>y1</code> != <code>y2</code>. Am I missing something?</p>
<p>Thank you!</p>

ptrblck: <p>How large is the difference between <code>y1</code> and <code>y2</code>?<br>
If it’s approx. 1e-6 or smaller, you might just see the limited floating point precision of FP32 numbers.<br>
To compare these tensors, you could alternatively use <code>torch.allclose</code> and specify the absolute or relative error, if necessary.</p>
<p>I’ve created a manual approach of <code>nn.BatchNorm2d</code> some time ago <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py" rel="nofollow noopener">here</a>. Maybe it might help in debugging. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

abhi-iyer: <p>Thank you so much! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Will check the differences exactly, but I did notice one thing.</p>
<p>While the differences were very small, the total accuracy as a result of recreating batchnorm was far worse (i.e. the actual predicted answers from the network were totally different than the manual predictions I found after recreating batchnorm).</p>

ptrblck: <p>Could you rerun the code using a few seeds and check, if you get these bad results randomly using both approaches?<br>
If the error of the manual vs. PyTorch batch norm output is small, e.g. an unlucky initialization might yield these results.</p>

abhi-iyer: <p>Sorry for the delay in my response. The results were bad until I normalized the batchnorm weights.</p>
<p>After this, I actually noticed that manually forward propping through the weights of my layers/batchnorm (since it already has normalized <code>running_mean</code> and <code>running_var</code> variables) yielded a higher accuracy than auto forward propping through the whole net.</p>
<p>Thanks for your help <a class="mention" href="/u/ptrblck">@ptrblck</a>. Typically, I’ve only used batchnorm2d, and glorot initializations can be applied here, so didn’t think to apply any kind initialize on batchnorm1d.</p>
SVHN small data, model not updating

gaurav71531: <p>When I put small SVHN data (50 samples to be exact) to a CNN, the model does not update and always gets stuck at outputting same class for all validation dataset.<br>
This does not happen for CIFAR10 dataset, which works well with the same CNN architecture and data size.</p>
<p>Also, not just the small size of 50, a lot of times the update does not happen for sizes like 2000, 4000. And if I re-run the training from scratch for some times (after acknowledging the training failure), then model update after few re-run iterations.<br>
Does anybody has encountered this for SVHN?<br>
Thanks a lot</p>
How to choose the non-linear activation function for the output layer, if the most of the value of the target are negative numbers?

crissallan: <p>I try to use MLP to solve a regression problem, and most of the target numbers are negative, but there is a very small part of them are positive numbers. For now, I choose a LogSigmoid function as the non-linear function of the output layer, but the MSE loss can’t get down to a low level. Should I choose a better non-linear function?<br>
the histogram of the target value is attached below.<br>
Many thank!<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/dfa9ba8e1d2722fe129a47e88486f246bea5b217.png" data-download-href="https://discuss.pytorch.org/uploads/default/dfa9ba8e1d2722fe129a47e88486f246bea5b217" title="c_lm_hist"><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/dfa9ba8e1d2722fe129a47e88486f246bea5b217.png" alt="c_lm_hist" data-base62-sha1="vUC7jK2wIESWH5KLdDxK1gy2imH" width="690" height="345" data-dominant-color="E1ECF3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">c_lm_hist</span><span class="informations">864×432 6.97 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
why nn.ConvTranspose2d output_padding not padding 0 ?

songlongzhang: <p>m = nn.ConvTranspose2d(1, 1, (2, 2), stride=(2, 2), padding=(1, 1), output_padding=(1,1))<br>
output = m(input)<br>
print(output)<br>
tensor([[[[ 0.3960, 0.6212, 0.3960, 0.6212, 0.3960, 0.6212, 0.3960],<br>
[-0.2565, 0.0108, -0.2565, 0.0108, -0.2565, 0.0108, -0.2565],<br>
[ 0.3960, 0.6212, 0.3960, 0.6212, 0.3960, 0.6212, 0.3960],<br>
[-0.2565, 0.0108, -0.2565, 0.0108, -0.2565, 0.0108, -0.2565],<br>
[ 0.3960, 0.6212, 0.3960, 0.6212, 0.3960, 0.6212, 0.3960],<br>
[-0.2565, 0.0108, -0.2565, 0.0108, -0.2565, 0.0108, -0.2565],<br>
[ 0.3960, 0.6212, 0.3960, 0.6212, 0.3960, 0.6212, 0.3960]]]],<br>
why output_padding not padding 0 to out?What number is pad?<br>
How can I replace ConvTranspose2d（output_padding） with ConvTranspose2d(no output_padding)?I want to ConvTranspose2d with output_padding in MXNet ?</p>
Pytorch warp on GPU

dklvch: <p>Is there some equivalent to skimage.transform.warp function implemented for gpu?</p>

Sunshine352: <p>Combine<br>
<code>torch.nn.functional.</code> <code>affine_grid</code> ( <em>theta</em> ,  <em>size</em> )<br>
and<br>
<code>torch.nn.functional.</code> <code>grid_sample</code> ( <em>input</em> ,  <em>grid</em> ,  <em>mode=‘bilinear’</em> ,  <em>padding_mode=‘zeros’</em> )<br>
you can  achieve the equivalent function to skimage.transform.warp function.</p>
Backpropagation gets slower and slower

whoab: <p>I am timing my backprop loop between epochs, and every epoch it gets slower and slower for the same sized minibatch. How can I debug this? I already ran a casual sweep over the code to check for tensors that are set outside of the training loop. I assume this is happening because the graph is not getting rebuilt from scratch each backprop loop. What is the rigorous way to debug this?</p>
Using Caffe2 library from PyTorch 1.2 build

kolkir: <p>Hello, I’ve built PyTorch 1.2 from sources for Arch Linux. C++ programs written with use of libtorch.so can be compiled and linked without any errors. But I can’t link even a simple program for Caffe2 library(libcaffe2.so), because I’m getting errors:</p>
<pre><code class="lang-auto">/usr/bin/ld: /somepath/lib/libcaffe2.so: undefined reference to `c10::Device::validate()'
/usr/bin/ld: /somepath/lib/libcaffe2.so: undefined reference to `c10::ThreadPool::ThreadPool(unsigned long, int)' 
</code></pre>
<p>My compilation command line looks like this one:</p>
<pre><code class="lang-auto">/usr/bin/g++  -msse3 -fopenmp -Wall -Wextra -Wno-unused-parameter -pthread -fopenmp -fPIC -ggdb -O0  -rdynamic main.cc.o  -o caffe2-load -Wl,-rpath,/somepath/lib /somepath/lib/libcaffe2.so /somepath/lib64/libprotobuf.a 
</code></pre>
<p>I would appreciate for any help.</p>

tom: <p>These sound like c10 functions, maybe you need to link in c10?</p>
<p>Best regards</p>
<p>Thomas</p>

kolkir: <p>Thanks, for the suggestion, but linking with the c10 library doesn’t help.</p>

kolkir: <p>I solved the problem. I modified the <code>c10/core/Device.h</code> and <code>c10/core/Device.cpp</code> files by moving the implementation of <code>Device::validate</code> method to the cpp file. Also I added additional constructor <code>ThreadPool(std::size_t pool_size,int numa_node_id)</code> to the class <code>c10::ThreadPool</code>. It seems that there are some inconsistency in the linkage options for the c10 library.</p>