akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Build failure in pull requests

Lewuathe: <p>While the build status on the master branch is almost green, <a href="https://github.com/pytorch/glow/pulls" rel="noopener nofollow ugc">the pull requests</a> fail in most cases.</p>
<p>There seems to be no unit test failure but it shows a non-zero exit status code.<br>
<a href="https://app.circleci.com/pipelines/github/pytorch/glow/12589/workflows/d7b901ad-6dcf-45ee-baff-4fd85bf67f69/jobs/91634/steps" class="onebox" target="_blank" rel="noopener nofollow ugc">https://app.circleci.com/pipelines/github/pytorch/glow/12589/workflows/d7b901ad-6dcf-45ee-baff-4fd85bf67f69/jobs/91634/steps</a></p>
<p>Do you have any idea about this? I’d like to work on making all PRs back to green so that the test status can be more trustworthy.</p>

jfix: <p>If you go into the <a href="https://circleci.com/api/v1.1/project/github/pytorch/glow/91634/output/106/0?file=true&amp;allocation-id=60509d8e4523c367112e3fc0-0-build%2F6A98740C" rel="noopener nofollow ugc">full log</a> and scroll down a bit you can see that <code>OnnxExporterTest</code> failed. In general we sometimes have some flaky tests and try to clean them up, especially w/ the torch_glow PyTorch integration which occasionally gets broken. Apologies for the pain here, we try our best <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

Lewuathe: <p>That seems to be related to my change. Thank you for finding the failure!</p>
Multithread multistream

111480: <p>In a multi-threading multi-stream environment, only one device was used and the ‘setcurrentCUDAStream()’ function was used mapping streams.<br>
After one thread set the current stream to stream 1, before queuing the function ‘tensor.sum()’ to stream 1, I expected that if another thread changed the current stream to stream 2, ‘tensor.sum()’ would execute in stream 2, but this did not happen as we observed through profiling. I repeated this process several times through for loop, but sum() was only run in Stream 1.</p>
<p>How is this possible?</p>

albanD: <p>Hi,</p>
<p>The cuda streams are thread local.<br>
So each thread can have a different current stream that will be used.</p>

111480: <p>Thanks for replying.<br>
One more question.<br>
Then, What’s the difference between setCurrentCUDAStream and StreamGuard?<br>
Under what circumstances do you use StreamGuard?</p>

albanD: <p>The guard is useful to make sure that when you get outside of the current scope, the previous stream is restored (even if an error is thrown in the current scope).<br>
So you are sure that you don’t “polute” the current stream of your parent function.</p>
Check if elements of Tensor are in a List/Array/Tensor

Cottoneyejoe1: <p>Hi guys,</p>
<p>I have some problems finding the fastest way to obtain the following:</p>
<p>I have a Tensor of indices: like for example<br>
tensor1 = [2,3,4,5,6,7,8]</p>
<p>And I also have a list of indices (or array, or tensor whatever is the easiest for later), like for example:<br>
tensor2 = [3,4,5]</p>
<p>Now what I want to get is a tensor with the same shape as tensor1 with booleans such that each element is True when its in tensor2 and False otherwise. So with the above mentioned example, I would want a tensor like this:<br>
result = [False, True, True, True, False, False, False]</p>
<p>Is there a one-liner in pytorch that can reproduce this functionality? Would be really glad if someone could help me.</p>
<p>Best regards<br>
Marco</p>

Emre_Yalcinoglu: <p>A possible solution might be</p>
<pre><code class="lang-auto">f = lambda x : x in tensor2
result = list(map(f, tensor1))
</code></pre>
<p>However, I suspect that, this approach is not the best practice.</p>

dmitrysarov: <p>mb this will help (tensor1.view(-1, 1) == tensor1.view(1, -1)).sum(1)</p>
Pass custom cmake build arguments to setup.py

ShadowTemplate: <p>Hi everyone!<br>
I am trying to build <em>torchaudio</em> from source. In my environment, CUDA and its related libraries (e.g. cuDNN) are installed in non-default paths.<br>
How can I pass custom build arguments to cmake via setup.py?</p>
<p>Thanks a lot.</p>
<p>Kind regards,<br>
GT</p>

vincentqb: <p>We do not have CUDA specific code yet, and so the torchaudio build doesn’t depend the nvidia compiler. What error are you encountering? If you do run into an error with this, I’d recommend opening an issue directly on <a href="http://github.com/pytorch/audio" rel="noopener nofollow ugc">github.com/pytorch/audio</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Dataset Summary Table Request

J_Johnson: <p>I would like to request a table be added to the page <a href="https://pytorch.org/vision/stable/datasets.html" rel="noopener nofollow ugc">pytorch.org/vision/stable/datasets.html</a> that summarizes the included datasets with something like the following information:</p>
<p>Dataset Name | Total File Size | Max/Min Image Dims | Brief Summary</p>
<p>I think that this information could be helpful and informative as the uninitiated try out vision ML tasks and research.</p>
<p>Thank you.</p>
Problem with loading multiple models

creadal: <p>I am running some pytorch with Google Colab. I have 3 models with same architecture but diferrent pretrained weights that I am loading from google drive. Here’s my code for loading weights:</p>
<pre><code class="lang-auto">import torch

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
device
</code></pre>
<pre><code class="lang-auto">normal = ResNetUNet(3).to(device)
depth = ResNetUNet(1).to(device)
mask = ResNetUNet(1).to(device)
</code></pre>
<pre><code class="lang-auto">normal_save_name = 'BKP_module1_normal_full_32_epochs.pt'
path_to_normal = F"/content/gdrive/My Drive/{normal_save_name}"
normal_dict = torch.load(path_to_normal, map_location=device)
</code></pre>
<pre><code class="lang-auto">depth_save_name = 'BKP_module1_depth_full_10_epochs.pt'
path_to_depth = F"/content/gdrive/My Drive/{depth_save_name}"
depth_dict = torch.load(path_to_depth, map_location=device)
</code></pre>
<pre><code class="lang-auto">mask_save_name = 'BKP_module1_mask_full_11_epochs.pt'
path_to_mask = F"/content/gdrive/My Drive/{mask_save_name}"
mask_dict = torch.load(path_to_mask, map_location=device)
</code></pre>
<pre><code class="lang-auto">normal.load_state_dict(normal_dict)
depth.load_state_dict(depth_dict)
mask.load_state_dict(mask_dict)

normal.eval()
depth.eval()
mask.eval()
</code></pre>
<p>And then I use this code to forward and plot the result of first network:</p>
<pre><code class="lang-auto">data = json.load(open('pix3d.json'))

batch = imread(data[0]['img'])
batch = transforms.ToTensor()(batch)
batch = torch.unsqueeze(batch, 0)
batch = batch.to(device)

normal_map = normal(batch)
normal_map = normal_map.cpu().detach().numpy()

plt.imshow(normal_map[0].T)
</code></pre>
<p>So if I load only the first network the result looks pretty okay for my purposes. But if I use the code above to load all of my three networks I get garbage results:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/1/31dfc1bc13912d03c58b2b62b1580dc2c4b4be93.jpeg" alt="image" data-base62-sha1="77cN9gEE1vNMR9yTqRZW3duqVXR" width="266" height="489"><br>
Above are results of running the same code but with 1 and 3 networks loaded respectively.<br>
My first thought was that the problem is with weights. But I checked that state dict of first network is the same both times. So I have literally no I idea why this is happening.<br>
Any help will be much appreciated.</p>

ptrblck: <p>If I understand the issue correctly, then your results change significantly by just loading (and never using) two additional models?<br>
If so, could you share the Colab notebook so that we could have a look at it?</p>

creadal: <p>Yes, you understand the issue correctly. I can share the link to my notebook but it is quite a mess so far. And you won’t be able to run it as I am loading many files from my drive.<br>
Here’s the link: <a href="https://colab.research.google.com/drive/1jOtl8HxcFk3LGx9XGXzkkZiM4DotBBi8?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1jOtl8HxcFk3LGx9XGXzkkZiM4DotBBi8?usp=sharing</a></p>
How can I use pytorch==1.1.0 with cuda 10.1 and python 2.7

i_m_a_q: <p>I am running a code that requires  “pytorch==1.1.0” but I have cuda 10.1. Also, I can’t build the pytorch from source as that code also requies “python 2.7”. Is there any way I can upgrade to pytorch version that is compatible with 10.1 and my code still runs fine? Or it will simply not work with any latest pytorch. I can’t go to any previous cuda version.</p>

crcrpar: <p>How about using Docker to get an environment with CUDA 10.1 before installing Python2 and build PyTorch 1.1.0 from source?</p>
Sigmoid returns different results depending on type of CPU

brunovollmer: <p>Hey everybody,</p>
<p>I hope this is the right category for my question. I have a very strange problem and I can’t lose the feeling that it is something trivial. I have some code that post-processes some object detection code. If I run it on my main machine (Ubuntu, I5 CPU) everything works as expected. If I run the same code, with the same model and the same input I receive different results on my Coral Dev Board.</p>
<p>I narrowed it down to a sigmoid call. Is there any reason why the sigmoid function on a float32 tensor should return different results on the two different CPUs? And if yes how can I fix it so that the Coral Dev Board runs the same?</p>
<p>Thanks in Advance,</p>
<p>Bruno</p>

JuanFMontesinos: <p>It’s normal up to certain point.<br>
Each CPU computes the sigmoid with different hardware and with a different set of instructions.<br>
What’s the error between both?<br>
If it’s smaller than 1e-5 it’s kinda expected.</p>

brunovollmer: <p>The problem is that the difference is a lot higher</p>

brunovollmer: <p>Found the problem but I’m not sure if it is a bug in PyTorch or my code. After some testing I found out that the sigmoid function returns “wrong” results on large tensors on my Coral Dev Board. If for example I do the following:</p>
<pre><code class="lang-auto">a = torch.ones(1000000)
a.sigmoid()
</code></pre>
<p>I would receive a result tensor where the first half is ~ 0.7301 and then the second half is ~ 0.43. In my understanding that is not very good as the results differ quite a lot.</p>
<p>My solution was to implement the sigmoid function in numpy but I would prefer to keep it in PyTorch if there is a solution for this.</p>
What operations on sparse tensor keep it coalesced?

francois-rozet: <p>I cannot find in the documentation which operations on sparse tensors keep them coalesced.<br>
In particular, I would like to know if <code>torch.sparse.sum</code> keep them coalesced.</p>
<p>If the sum is implemented by “simply” removing one index, then it doesn’t. But I think that, using the fact the indices are sorted (when coalesced), the <code>sum</code> could be implemented efficiently while keeping the tensor coalesced.</p>
Pytorch `nn.distributed` general example

Rishav_Sapahia: <p>Hi,<br>
Is there a dmo example given for using <code>torch.nn.distributed</code> ?<br>
I am trying to test whether my University’s cluster working correctly or not?</p>

ptrblck: <p><a href="https://pytorch.org/tutorials/beginner/dist_overview.html">These docs</a> give you an overview of the different distributed implementations and link to examples. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to build a convolutional neural network with existing parameters?

jerly-hjzhou: <p>I want to build a convolutional neural network to recognize digits. And I already have the structure and the parameters of weight and bias stored in the format of “txt”. How should I use them to implement a network?</p>

ptrblck: <p>You could create the model first, load the parameters from the .txt file, transform them to floats and then tensors, and load each parameter manually via:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.layerX.weight.copy_(layerX_weight)
    model.layerX.bias.copy_(layerX_bias)
</code></pre>
<p>Note that his approach is a bit tedious, as it seems you are loading the parameters from a raw text file, which is not a supported via utility methods (such as <code>model.load_state_dict</code>).</p>
Multi GPU Training

partha: <p>Hi, I have a model that requires 14 GB to train. However, I have access to 12 GB GPU nodes. If I use two nodes (that is 24 GB) still I am getting not enough memory CUDA error. Can you help me overcome this error?</p>

edwardpwtsoi: <p>If your model cannot fit into single GPU you would need to do <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" rel="noopener nofollow ugc">model parallelism</a></p>
How to convert 3d images into libtorch tensor

WeNeedMore: <p>If it was ordinary 2d images, we could load them by opencv, and convert it to the tensor like this</p>
<pre><code class="lang-auto">auto tensor_img = torch::from_blob(buffer_img_float_.data,
                                           {1, buffer_img_float_.rows, buffer_img_float_.cols, 3}).to(get_device());
</code></pre>
<p>But what if the input tensor need to have 5 dimensions?</p>
<p>Like</p>
<p><code>torch.zeros((1,64,3,224,224)) #batch, slices, channels, height, width</code></p>
<p>I can load the nifti by simpleITK and extract the pixels, but how could I convert it to the libtorch tensor?What is the order I should feed the data into the <code>vector&lt;float&gt;</code></p>
<p>Or just</p>
<blockquote>
<p>at::Tensor foo = at::ones({1, 64, 3, 224, 224}, at::kFloat);</p>
</blockquote>
<p>Then copy the pixels to this tensor one by one?</p>
<pre><code class="lang-auto">auto foo_a = foo.accessor&lt;float,5&gt;();
 foo_a[0][slice][channel][row][col] = pixel_value[0][slice][channel][row][col]
</code></pre>
Converting objects in a 2D image to 3D meshes

Mona_Jalal: <p>I have been using Neural Mesh Renderer <a href="https://github.com/JiangWenPL/multiperson/tree/master/neural_renderer" class="inline-onebox" rel="noopener nofollow ugc">multiperson/neural_renderer at master · JiangWenPL/multiperson · GitHub</a> for converting 2D image to 3D mesh for an object (e.g. teapot as shown in the github example).</p>
<p>Is this the best package for this purpose in PyTorch or are there better and more precise candidates for this purpose? (given it’s a bit outdated by now – CVPR 2018)</p>
<p>Also, can PyTorch3D do the same thing? I was not able to find an end-to-end example that can do so. If you could link me with one would be very fantastic.</p>
<p>P.S.: I also see this <a href="https://github.com/ShichenLiu/SoftRas" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ShichenLiu/SoftRas: Project page of paper "Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning"</a> (haven’t fully set it up) but PyTorch3D first author mentions they are inspired by this paper - assuming they perform better?</p>
How to change the variables needed for gradient computation

Edward_Zhou: <p>Hi,</p>
<p>I want to quantize the values of avtivations and weights of each layer into 4bit or 8bit. So i used a method to generate the values after quantization to replace the old full precision values. The values are indeed modified because this is what i want.<br>
However, there is always<br>
“RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation”<br>
I know there is some internal mechainism to track the correctness for computing of gradient. I already tried "with torch.no_grad(): " and lowered the version of pytorch to 0.2 but such tracking still exist. Can i ask how to disable such feature and change the values in gradient computing freely?</p>
<p>Many thanks!<br>
Edward</p>

Eta_C: <p>Hello, I write an example. Is it what you want?</p>
<pre><code class="lang-python">def fake_quantize(data, scale):
    with torch.no_grad():
        data[:] = torch.round(data[:] * scale) / scale

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.body = nn.Conv2d(3, 3, 3)
    
    def forward(self, x):
        out = self.body(x)
        print("before quantize:", out)
        fake_quantize(out, torch.rand(1) * 100)
        print("after quantize:", out)
        return out 


net = Net()
y = net.forward(torch.rand(10, 3, 6, 6))
loss = y.sum()
loss.backward()
print(y)
</code></pre>

Edward_Zhou: <p>Many thanks for your reply.</p>
<p>And this example is similiar to what i have tried with torch.no_grad() method, the execution of program still warn “RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation”</p>

Eta_C: <p>Do you use <code>data[:] = new_data</code> instead of <code>data = new_data</code> ?</p>

Edward_Zhou: <p>Thanks for your reply, I used data = new_data when i got the runtime error. I have not tried data[:] = new_data yet. Can i ask why add [:] will make a difference?</p>

Eta_C: <pre><code class="lang-python">x = torch.rand(3)
print(id(x))  # 140252375853408
x = torch.rand(3)
print(id(x))  # 140252189803616

x = torch.rand(3)
print(id(x))  # 140252375853408
x[:] = torch.rand(3)
print(id(x))  # 140252375853408
</code></pre>
<p>See, <code>[:]</code> would not change the memory address.</p>

Edward_Zhou: <p>Sorry for reply late. I have tried using [:]<br>
I tried out[0][0][0][0][:] where out is a 4 dimension array. all index set to 0 just as an example here.<br>
But i got error<br>
out[0][0][0][0][:] = 1<br>
IndexError: dimension specified as 0 but tensor has no dimensions<br>
So i tried to modify the whole tensor  with out[:] = torch.rand(out.size())<br>
I found it is still<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation:</p>
Dot product along a dimension

sugh7020: <p>I have two tensors of shape [B, 3 , 240, 320] where B represents the batch size 3 represents the channels, 240 the height(H), 320 the width(W).</p>
<p>I need to find the dot product along the channels dimension(3 channels) thus the resulting tensor would be of shape [B, 1 , 240, 320].<br>
My tensors have float32 elements in gpu(cuda to backprop).</p>
<p>Can you all please suggest how I can do that?</p>
<p>Thanks!</p>
<p>More clarification:</p>
<p>Let’s say we have B=10, H=100, W=200.<br>
So from the above would be common for both the first and seconds tensors. If we keep B, H, W constant we get a 1D vector as the resultant tensor(with 3 elements). I need to take the dot product of these two vectors. Thus the resultant tensor is of dimension [B, 1, 240, 320]</p>

Eta_C: <pre><code class="lang-python">A = torch.rand(2, 3, 4, 4)
B = torch.rand(2, 4, 4)
C = (A * B.unsqueeze(1)).sum(1, keepdims=True)
</code></pre>
<pre><code class="lang-python">A = torch.rand(2, 3, 4, 4)
B = torch.rand(3)
C = (A, B[None, :, None, None]).sum(1, keepdims=True)
</code></pre>
<pre><code class="lang-python">A = torch.rand(2, 3, 4, 4)
B = torch.rand(2, 3, 4, 4)
C = (A * B).sum(1, keepdims=True)
</code></pre>
pytorch lstm output not match hidden state

naturesphere: <p>Here is the code:</p>
<pre><code class="lang-auto">import torch
from torch.nn.utils.rnn import pack_sequence, pad_packed_sequence
rnn = torch.nn.LSTM(2, 3, 1, bidirectional=True)
x1= torch.randn(1,2)
x2=torch.randn(2,2)
x=pack_sequence([x1,x2], enforce_sorted=False)
y, (hn, cn) = rnn(x)
y,lens=pad_packed_sequence(y)
</code></pre>
<p>print the hidden state result:</p>
<pre><code class="lang-auto">print(torch.cat([hn[-2], hn[-1]], dim=1))
</code></pre>
<p>get result:</p>
<pre><code class="lang-auto">tensor([[-0.0265, -0.1149, -0.0466,  0.1080,  0.0901,  0.0865],
        [ 0.0736, -0.2503, -0.0634,  0.0801,  0.1401, -0.0032]],
       grad_fn=&lt;CatBackward&gt;)
</code></pre>
<p>print the output:</p>
<pre><code class="lang-auto">for i,j in enumerate(lens):
    print(y[i][j-1])
</code></pre>
<p>get result:</p>
<pre><code class="lang-auto">tensor([-0.0265, -0.1149, -0.0466,  0.1080,  0.0901,  0.0865],
       grad_fn=&lt;SelectBackward&gt;)
tensor([ 0.0736, -0.2503, -0.0634,  0.0932,  0.0962, -0.0915],
       grad_fn=&lt;SelectBackward&gt;)
</code></pre>
<p>the second tensor is not the same with the hidden state!</p>
<p>Why?</p>

naturesphere: <p>Which one should be used as result?</p>

vdw: <p>I think I had the <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">same question</a> where I also posted my answer. Maybe you can check if it indeed matches your concerns.</p>

naturesphere: <p>I still can not understand. Why the first tensor is match and the second is not.</p>
Inference/forward passes not asynchronous?

nawinw: <p>I am currently having problems with the forward operation being seemingly synchronous when using LibTorch in C++.  I’ve taken <a href="https://github.com/yulunzhang/RCAN" rel="noopener nofollow ugc">this</a> model and converted it using a TorchScript trace and loaded it into my C++ program using LibTorch.  Before calling a number of tensor operations and forward, I set the stream using CUDAStreamGuard:</p>
<blockquote>
<p>streams.push_back(torch::cuda::getStreamFromPool());<br>
torch::cuda::CUDAStreamGuard guard(streams[i]);</p>
</blockquote>
<p>For the tensor operations, data type conversions and transfers from CPU to GPU are done using non_blocking=true.  Other operations are assumed to be asynchronous per the LibTorch documentation.</p>
<p>Without fail, all of the operations return immediately, <em>except</em> for forward.</p>
<blockquote>
<p>torch::Tensor result = rcan.forward(inputs).toTensor();</p>
</blockquote>
<p>As my GPU usage is less than 30% according to nvidia-smi, I would like to run multiple forward passes on independent data in parallel using streams.  If anyone has some insights on whether or not forward can be called synchronously, or if I am approaching this incorrectly, I would greatly appreciate it.  Thank you.</p>

ptrblck: <p>Could you try to use some random data and check the GPU utilization again please?<br>
This could give you a signal, if the data loading pipeline is the bottleneck.</p>

Shakiba_Kh: <p><a class="mention" href="/u/nawinw">@nawinw</a> Were you able to figure this out? I want to do the same and forward is not behaving asynchronous.</p>
Help me: AttributeError: 'tuple' object has no attribute 'cpu'

Nay: <p>Hi, can someone help me please:</p>
<p>I’m try to run the code below, but I got this issues:</p>
<p>rend = rend.cpu().data.numpy().transpose((0, 2, 3, 1))<br>
AttributeError: ‘tuple’ object has no attribute ‘cpu’</p>
<p>I also printed the shape of this turple:</p>
<p>torch.Size([1, 3, 244, 244])<br>
torch.Size([1, 244, 244])<br>
torch.Size([1, 244, 244])</p>
<p>I’ve tried a lot of ways to solve this, but I got other issues… <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=9" title=":expressionless:" class="emoji" alt=":expressionless:"></p>

ptrblck: <p>You cannot call <code>cpu()</code> on a Python <code>tuple</code>, as this is a method of PyTorch’s tensors.<br>
If you want to move all internal tuples to the CPU, you would have to call it on each of them:</p>
<pre><code class="lang-python">x = (torch.randn([1, 3, 244, 244]).cuda(),
     torch.randn([1, 244, 244]).cuda(),
     torch.randn([1, 244, 244]).cuda())

x.cpu() # error

y = tuple(t.cpu() for t in x)
print(y)
</code></pre>

Nay: <p>Works. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Correct way of using KL Loss

Carsten_Ditzel: <p>I am trying to use the KL loss for a VAE. The latents, i.e. output of the encoder is of dimension</p>
<p><code>z =[Batchsize,posterior,height,width], e.g. z =[2,64,4,4].</code><br>
and I want to determine the KL with a uniform prior.</p>
<p>Now I am a little confused about what to sum and what to average. The <a href="https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html" rel="noopener nofollow ugc">docs</a> state that one should use ‘batchmean’ for a mathematically consistent calculation across samples, i.e. batches. However, in my case, I also have spatial dimensions height and width and for each pixel I want to evaluate the KL divergence separately and only than average. So is it correct to use</p>
<pre><code class="lang-auto">log_p = F.log_softmax(z, dim = 1) # turn latents into probabilities
log_q = torch.log(torch.ones_like(z) / z.shape[1])  # uniform prior
kl_loss = F.kl_div(log_p, log_q, reduction='batchmean', log_target = True)
</code></pre>
<p>or the manual</p>
<pre><code class="lang-auto">p = F.softmax(z, dim=1) # turn latents into probabilities
q = torch.ones_like(z) / z.shape[1])  # uniform prior
log_ratio = torch.log(p / q)
kl_loss = torch.sum(p * log_ratio, dim=1).mean() # sum each kl over probability dim
</code></pre>
<p>which does give a different result but intuitively feels correct. First a summation for the KL over the channel/probability dimension followed by averaging over both the batch and the spatial extend dimensions.</p>
<p>Can someone help me out here please?</p>
Padding zeros along a dimension

sugh7020: <p>Hi!</p>
<p>I have a tensor X: <em>torch.float32, torch.Size([64, 3, 240, 320])</em>.</p>
<p>I need to pad zeros and add an extra column(at the <em>beginning</em>) such that the resultant shape is <em>torch.Size([64, 3, 240, 321])</em>, i.e. all the extra elements are zeros(so an added column of zeros in the first column).</p>
<p>Is there a generic way to add zeros along a dimension? Later I would need to add a row(first row) of zeros as well.</p>
<p>Thanks!</p>

AbdulsalamBande: <p>Hy man, you can try the below</p>
<pre><code class="lang-auto">import torch.nn.functional as F
data = torch.ones((64, 3, 240, 320))
# pad(left, right, top, bottom)
new_data = F.pad(input=data, pad=(0, 1, 0, 0), mode='constant', value=0)
print(new_data)
</code></pre>
<p>Reference <a href="https://stackoverflow.com/questions/48686945/reshaping-a-tensor-with-padding-in-pytorch" rel="noopener nofollow ugc">reshaping a tensor with padding in pytorch</a></p>

sugh7020: <p>Thanks <a class="mention" href="/u/abdulsalambande">@AbdulsalamBande</a></p>
Dataloader spawn re-imports scripts and crashes

enterthevoidf22: <p>a problem that i’m facing for a while, when running an experiment and simultaneously modifying the scripts that run the experiment, it happens that the interpreter will go through the script again while i’m rewriting it and crash.</p>
<p>i thought i solved it when i upgraded pytorch and started using persistent workers = True but i’m still experiencing this error.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "D:\ProgramData\Anaconda3\envs\env_zoo\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "D:\ProgramData\Anaconda3\envs\env_zoo\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "D:\ProgramData\Anaconda3\envs\env_zoo\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "D:\ProgramData\Anaconda3\envs\env_zoo\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "D:\ProgramData\Anaconda3\envs\env_zoo\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "D:\ProgramData\Anaconda3\envs\env_zoo\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\users\Gony\FMRI_inpainting1\examples\bert_fine_tune.py", line 5, in &lt;module&gt;
    import lib.train as train
  File "D:\users\Gony\FMRI_inpainting1\lib\train\__init__.py", line 4, in &lt;module&gt;
    from .trainer_gony import Trainer_gony
  File "D:\users\Gony\FMRI_inpainting1\lib\train\trainer_gony.py", line 76
    self.writer.update_tensorboard(grads=None,initial_val_loss)
                                             ^
SyntaxError: positional argument follows keyword argument
 19%|█▉        | 999/5160 [1:16:06&lt;5:16:58,  4.57s/it]
Traceback (most recent call last):
  File "D:\ProgramData\Anaconda3\envs\env_zoo\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "D:\ProgramData\Anaconda3\envs\env_zoo\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe
</code></pre>

ptrblck: <p>I’m not sure, if this is a PyTorch-specific issue or rather a Python multiprocessing limitation or if it depends on the used Python interpreter/IDE.<br>
Based on the description it seems that your Python interpreter reloads the source files after each change or at a specific interval.<br>
Based on <a href="https://stackoverflow.com/a/60910365">this post</a> it also seems to be a Windows-specific behavior.</p>
Specify which GPUs to use with torch.distributed.launch

cmplx96: <p>Hi all,</p>
<p>is there a way to specify a list of GPUs that should be used from a node?</p>
<p>The <a href="https://pytorch.org/docs/master/distributed.html#launch-utility" rel="noopener nofollow ugc">documentation</a> only shows how to specify the number of GPUs to use:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE ...
</code></pre>
<p>This was already asked in this <a href="https://discuss.pytorch.org/t/no-option-to-change-gpus-when-using-torch-distributed-init-process-group/106555/2">thread</a> but not answered.</p>
<p>Cheers!</p>

ptrblck: <p>You can make certain GPUs visible via <code>CUDA_VISIBLE_DEVICES=1,3,7 python -m ...</code>, which would map GPU1, GPU3, and GPU7 to <code>cuda:0</code>, <code>cuda:1</code>, <code>cuda:2</code> inside the script and execute the workload (DDP in your case) only on these devices.</p>
Segfault using data_ptr() after torch::nn::functional::interpolate()

Stephane_Charette: <p>New to PyTorch. Getting a segfault on Jetson NX after call to <code>torch::nn:functional::interpolate()</code> and could use some hints.  Code is similar to this:</p>
<pre><code class="lang-auto">auto options1 = torch::TensorOptions().dtype(torch::kFloat32).device(torch::kCUDA).requires_grad(false);
auto t = torch::from_blob(ptr, {4, 34, 160, 160}, options1);

// code to populate tensor here ...

// then when it comes time to interpolate, this is what I'm trying to run:
const auto options2 = torch::nn::functional::InterpolateFuncOptions().size(std::vector&lt;int64_t&gt;({320,320})).mode(torch::kBilinear).align_corners(false);
auto resized_output = torch::nn::functional::interpolate(tensors[1], options2);

// now how do we get access to the data without segfaulting?
float * new_tensor_ptr = reinterpret_cast&lt;float*&gt;(resized_output.data_ptr());
std::cout &lt;&lt; "first byte of new tensor: " &lt;&lt; new_tensor_ptr[0] &lt;&lt; std::endl; // &lt;-- this segfaults
</code></pre>

ptrblck: <p>Could you post the backtrace via:</p>
<pre><code class="lang-auto">gdb --args ./my_app
...
run
...
bt
</code></pre>
<p>Also, how did you install PyTorch and which version are you using?</p>

Stephane_Charette: <p>Dug a bit into this code again.  The tensor is created like this:</p>
<pre><code>const auto options = torch::TensorOptions().dtype(torch::kFloat32).device(torch::kCUDA).requires_grad(false);
auto t = torch::from_blob(ptr, {4, 34, 160, 160}, options);
</code></pre>
<p>This will eventually segfault when we attempt to dereference <code>data_ptr()</code>.  But if instead we use these options:</p>
<pre><code>const auto options = torch::TensorOptions().dtype(torch::kFloat32).requires_grad(false);
</code></pre>
<p>…then everything works fine.  Note the removal of <code>.device(torch::kCUDA)</code>.</p>
<p>I’ve looked through the github issues looking for segfault related to data_ptr() and torch::kCUDA, and found some things such as <a href="https://github.com/pytorch/pytorch/issues/19969" class="inline-onebox" rel="noopener nofollow ugc">libtorch Segmentation Fault: RHEL 7 - easy to reproduce · Issue #19969 · pytorch/pytorch · GitHub</a>.  I did attempt to call <code>cudaDeviceSynchronize()</code> just prior to getting <code>data_ptr()</code> but that did not help.</p>
<p>PyTorch is v1.7, cloned from github in early 2021-02, and built manually on a Jetson NX.  It does have the NVIDIA pytorch patch <code>pytorch-1.7-jetpack-4.4.1.patch</code> obtained from the NVIDIA Jetson site along with the instructions on how to build.</p>

ptrblck: <p>Thanks for the update.<br>
Where is <code>ptr</code> coming  from and did you make sure that the allocated memory is still alive and on the GPU?<br>
<a href="https://github.com/pytorch/pytorch/issues/19786#issuecomment-486929710">This post</a> shows how to use <code>cv::cuda::GpuMat</code> as the input to <code>torch::from_blob</code>.</p>
Calculate loss per sample in a batch + backward pass

saitam757: <p>Hi,</p>
<p>I am working on my first little more complex deep learning based image segmentation task where the loss function needs to be calculated per sample in a batch. The loss values are summed up to get a loss per batch. The loss function is not in a way that the batch input tensor [batch, channel, height, width]  is reduced by “simple” tensor operations to a loss output tensor of size [batch], therefore this single sample calculation approach.<br>
On github I found the implementation of the loss function in tensorflow and I converted it already to pytorch. The lines I don’t know how to convert are:</p>
<p>loss = tf.TensorArray(dtype=tf.float32, size=0, dynamic_size=True)<br>
loss, _, _, _, _ = tf.while_loop(cond, body, [loss, embedding, label_map, neighbor, 0])</p>
<p>loss = loss.stack()<br>
loss = tf.reduce_mean(loss)</p>
<p>Its actually a while loop over the samples in the batch, calling the loss function “body” for each sample. I don’t know how to do this in pytorch with regard to the backward pass and gradient calculation.</p>
<p>Thanks a lot in advance.</p>

ptrblck: <p>I’m not deeply familiar with TF, but based on <code>loss.stack()</code> I assume the corresponding PyTorch code snippet might be:</p>
<pre><code class="lang-python">losses = []
while condition:
    loss = calculate_loss(...)
    losses.append(loss)
losses = torch.stack(losses).mean()
losses.backward()
</code></pre>

saitam757: <p>So easy (-: Thank you. I will try that out.</p>
Is there a way to release GPU memory in libtorch?

cyanM: <p>I encapsulate model loading and forward calculating into a class using libtorch,and want to release the gpu memory(including model) while destroy the class.</p>
<p>I have tried c10::cuda::CUDACachingAllocator::emptyCache(), but it doesn’t seem to be working.</p>

Carol_Ye_Liu: <p>I have the same question. <a class="mention" href="/u/cyanm">@cyanM</a> did you find any solution?<br>
c10::cuda::CUDACachingAllocator::emptyCache() released some GPU memories for me, but not all of them.</p>

lbdalmendrayCaseguar: <p>I have the same problem .Any solution ?<br>
Thanks .</p>

copythatpasta: <p>check this <a href="https://discuss.pytorch.org/t/how-to-effectively-release-a-tensor-in-pytorch/114519/2" class="inline-onebox">How to effectively release a Tensor in Pytorch? - #2 by ptrblck</a></p>
Is my validation loss code correct? How to early stop and prevent overfitting?

sandeep1: <p>Hi,<br>
I am a new to pytorch. It will be helpful if you can comment on the below code, if I have done a mistake or if there is any better way to do it.<br>
Also how to stop and save the best model?</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
  batch_idx=batch_idx+1
  train_loss = 0.
  full_val_loss=0.
  for x,y,y2 in loader:
    optimizer.zero_grad()
    x = embedding(x).to(device)
    input_size= x.shape[2]
    output1 = model(x)
    loss = criterion1(output1,y)
    loss.backward()
    
    optimizer.step()
    train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

  model.eval()
  with torch.no_grad():
    for x,y in val_loader:
      x = embedding(x).to(device)
      output1 = model(x)
      val_loss = criterion1(output1,y)
      full_val_loss = full_val_loss + ((1 / (batch_idx + 1)) * (val_loss.data - full_val_loss))
  
  model.train()
  print('Epoch [{}/{}], Loss: {:.4f} , val_loss: {:.4f} '.format(epoch+1, num_epochs, train_loss,full_val_loss))
</code></pre>

Dwight_Foster: <p>What is y2 for in the train loop?</p>

patrickwilliams3: <p>Why no torch.no_grad()? I thought it was useful to have in evaluation mode since it prevents the autograd engine from creating the graph for the backwards pass, so you save memory when you do not need it.</p>

patrickwilliams3: <p>I thought that the graph is destroyed after you call .backward() and then created again next time you call forward().</p>

Dwight_Foster: <p>Yes sorry I said that wrong. The graph is destroyed when you call .backward() using torch no grad may lower your gpu usage slightly but not a lot. I have tried both ways and for some reason my gpu memory has never really been different. You can keep it in however if it impacts your performance. I am still confused what y2 does though.</p>
Pytorch W-net architecture

randinoo: <p>how can i make this architecture :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b32c76454d9c350e3a35eb59042096dda6790ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/4b32c76454d9c350e3a35eb59042096dda6790ce" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b32c76454d9c350e3a35eb59042096dda6790ce.png" alt="image" data-base62-sha1="aJeBpVWh0XE2ZTHJg2gYM4eIPVQ" width="690" height="485" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">815×573 34.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>help me plzzz <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=12" title=":pensive:" class="emoji" alt=":pensive:" loading="lazy" width="20" height="20"></p>

aguennecjacq: <p>There are many ways to do this. I would suggest that you search for the source code of the project on github. For example here: <a href="https://github.com/Andrew-booler/W-Net/blob/master/Wnet/model.py" class="inline-onebox" rel="noopener nofollow ugc">W-Net/model.py at master · Andrew-booler/W-Net · GitHub</a></p>

randinoo: <p><a class="mention" href="/u/aguennecjacq">@aguennecjacq</a> I tested this code</p>
<p>But this error</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5fa2300ca99df75fa054d5fac74a3de30a40916.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5fa2300ca99df75fa054d5fac74a3de30a40916" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5fa2300ca99df75fa054d5fac74a3de30a40916.png" alt="image" data-base62-sha1="z60TxcxsyHR5stnUJjkoVKKBYRo" width="690" height="324" data-dominant-color="F7F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">772×363 10.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

TPU: <p>Try to install <code>configure</code> in your environment before running the code again - the error says you don’t have it installed. More details: <code>https://pypi.org/project/config/</code>.</p>
Finding activations per layer

grid_world: <p>I am interested in finding a way to find the activated neurons per layer for a given neural network model. Say for a VGG styled CNN, if any compression algorithm is applied like “Deep Compression” by Han et al., " Learning both Weights and Connections for Efficient Neural Networks" by Han et al., “The Lottery Ticket Hypothesis” by Frankle et al. the number of connections are pruned, per trainable layer. I want to find the activations per layer, after the pruning has happened.</p>
<p>Help?</p>
Why are PyTorch “convolutions” implemented as cross-correlations?

iacob: <p>PyTorch convolutions are actually implemented as <a href="https://en.wikipedia.org/wiki/Cross-correlation" rel="noopener nofollow ugc">cross-correlations</a>. This shouldn’t produce issues in <em>training</em> a convolution layer, since one is just a flipped version of the other (and hence the learned function will be equally powerful), but it <strong>does</strong> <a href="https://stackoverflow.com/questions/42970009/performing-convolution-not-cross-correlation-in-pytorch" rel="noopener nofollow ugc">prove an issue</a> when:</p>
<ol>
<li>trying to implement an actual convolution with the <code>functional</code> library</li>
<li>trying to copy the weights of an actual convolution from another deep learning library</li>
</ol>
<p>The authors say the following in <a href="https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf" rel="noopener nofollow ugc">Deep Learning with PyTorch</a>:</p>
<blockquote>
<p>Convolution, or more precisely, <em>discrete convolution</em><sup>1</sup>…</p>
<p><sup><sup>1.</sup> There is a subtle difference between PyTorch’s convolution and mathematics’ convolution: one argument’s sign is flipped. If we were in a pedantic mood, we could call PyTorch’s convolutions <em>discrete cross-correlations.</em> </sup></p>
</blockquote>
<p><strong>But they don’t explain <em>why</em> it was implemented like this. Is there a reason?</strong></p>
<p>Maybe for eficiency reasons, something similar to how the PyTorch implementation of <code>CrossEntropyLoss</code> <a href="https://stackoverflow.com/questions/49390842/cross-entropy-in-pytorch" rel="noopener nofollow ugc">isn’t actually cross entropy</a> but an analogous function taking “logits” as inputs instead of raw probabilities (to avoid <a href="https://pytorch.org/docs/stable/nn.functional.html#softmax" rel="noopener nofollow ugc">numerical instability</a>)?</p>
<p>(<a href="https://stackoverflow.com/questions/66640802/why-are-pytorch-convolutions-implemented-as-cross-correlations" rel="noopener nofollow ugc">Stack Overflow post</a>)</p>

jbschlosser: <p>Convolutions are implemented as cross-correlations for performance / efficiency, as you brought up.</p>
<p>From <a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8721631" rel="noopener nofollow ugc">Performance Evaluation of cuDNN Convolution Algorithms on NVIDIA Volta GPUs</a>: “The advantage of cross-correlation is that it avoids the additional step of flipping the filters to perform the convolutions.”</p>