akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: size mismatch, m1: [2304 x 200], m2: [100 x 1]

vorlket: <p>I have the following code:<a href="https://gist.github.com/vorlkets/f38c91aa52cb4e68976e1f638d251a85" rel="noopener nofollow ugc">https://gist.github.com/vorlkets/f38c91aa52cb4e68976e1f638d251a85</a>. When run I get RuntimeError: size mismatch, m1: [2304 x 200], m2: [100 x 1] at C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensorMath.cpp:136 error. Can someone tell me what’s going on? I am a newbie btw.</p>

Abhilash_Srivastava: <p>Can you share the full error log?</p>

vorlket: <blockquote>
<p>python hw2main.py<br>
Using device: cpu</p>
</blockquote>
<p>Traceback (most recent call last):<br>
File “hw2main.py”, line 142, in <br>
main()<br>
File “hw2main.py”, line 72, in main<br>
ratingOutput, categoryOutput = net(inputs, length)<br>
File “C:\Users\Cliff\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “F:\Master of Information Technology at UNSW\COMP9444 Neural Networks and Deep Learning\Assignment\Assignment 2\hw2\student.py”, line 105, in forward<br>
out_rating = torch.sigmoid(self.fc_rating(out_rating))<br>
File “C:\Users\Cliff\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\Cliff\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\linear.py”, line 87, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “C:\Users\Cliff\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\functional.py”, line 1372, in linear<br>
output = input.matmul(weight.t())<br>
RuntimeError: size mismatch, m1: [60 x 200], m2: [100 x 1] at C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensorMath.cpp:136</p>

Abhilash_Srivastava: <p>I think this can be the problem:</p>
<pre><code class="lang-auto">        self.gru_rating = tnn.GRU(input_size = 50, hidden_size = 100, num_layers = 5, bidirectional = True)
        self.gru_buscat = tnn.GRU(input_size = 50, hidden_size = 100, num_layers = 5, bidirectional = True)
        self.fc_rating = tnn.Linear(100, 1)
        self.fc_buscat = tnn.Linear(100, 5)
</code></pre>
<p>Since, these are bidirectional gru layers, the output dimension would would be 200, which doesn’t match with the Linear layer input dimension. Try:</p>
<pre><code class="lang-auto">        self.fc_rating = tnn.Linear(200, 1)
        self.fc_buscat = tnn.Linear(200, 5)
</code></pre>
Kaggle CSV files

Naqibullah_Vakili: <p>Dear all,<br>
I have trained a pre-trained Resnet50 classifier on the kaggle images dataset and label them as 0 for No-Covid 1 for Thorax diseases and 2 for covid.<br>
now for the competition, I need to save these labels in the CSV files.<br>
when I run the following code for evaluation of the test images,<br>
total_imgs = 0<br>
covid_positive = 0<br>
for root, dirs, files in os.walk("/content/gdrive/My Drive/dlai3-phase3/VALIDATE/VALIDATE", topdown=False):<br>
for name in files:<br>
total_imgs += 1<br>
<span class="hashtag">#im_covid</span> = Image.open(“data/test/covid/03BF7561-A9BA-4C3C-B8A0-D3E585F73F3C.jpeg”)<br>
img = Image.open(os.path.join(root, name))<br>
img = img.convert(‘RGB’)<br>
pred, probs = predict_covid(img)<br>
print(pred)</p>
<p>print the string labels, not the number. Could anyone please tell me how I can get the numeric labels? thanks</p>

ptrblck: <p>Your <code>predict_covid</code> method seems to return a string, so you would have to look into this method and check, what it is doing.<br>
The standard PyTorch models return e.g. class logits and you could get the predicted class index via: <code>preds = torch.argmax(outputs, dim=1)</code> for a multi-class classification.</p>

Naqibullah_Vakili: <p>Thank you so much. I truly appreciate your help.</p>
How to add context information to U-Net input

stadtgiraffe: <p>Hello,</p>
<p>I have contextual information in addition to my input image with its corresponding segmentation mask (in my case the contextual information is the absolute body height in which the CT slice is positioned).</p>
<p>I want to add this information to the input, so my U-Net can learn the expected size and type of organs in this body height and use this to improve segmentation.<br>
How can I add this scalar to the input image? As a an additional channel?</p>
<p>Thanks for the help.</p>
CachingAllocator Vs CachingHostAllocator

yushinliu: <p>Hi, I am currently reading the code of cachingallocator and cachinghostallocator, and I found the difference between them is that the cachingallocator owns a chain of Blocks and it will merge the blocks when allocator frees. Why not the cachinghostallocator follow the same pattern as cachingallocator does? It looks like using more cached pinned-memory.</p>
User level LSTM/Conv1d

ygl: <p>Hi,</p>
<p>I try to infer a person’s gender using all of her/his tweets. I want to map each tweet into an embedding space and then fed them into LSTM tweet by tweet. But how do I do it in LSTM? I mean the input for a LSTM is like seq_len(num of words in a sentence), batch(batch size), input_size(embedding dimension). I need to get LSTM output for each tweet of each user and process from there. Is there a way to do this for all users at the same time? A for loop?</p>
<p>My original thought was that I use an LSTM to encode all these tweets with dimension seq_len * (batch_size<em>T) * embedding_dimension. So the batch_size can be set temporarily as (batch_size</em>T). After I get the hidden representations with dimension (batch_size<em>T) * hidden_dimension (2D matrix), you can reshape it to dimension batch_size</em> T * hidden_dimension (3D tensor). But the thing is different users have different number of tweets. The solution above doesnt sound right.</p>
Correspondence between two tensors

Saifullahi_Aminu: <p>I have a tensor  X = (B, 2) and another tensor Y= (b, 2), where Y is a subset of X (B&gt;=b), how can I return the indices of the values of Y in X.<br>
for example,<br>
X=(<br>
[1, 2],<br>
[4,5],<br>
[3,1],<br>
[7,8])</p>
<p>Y=(<br>
[1, 2]<br>
[7, 8])</p>
<p>I want return [0, 2].</p>
Semantic segmentation Assertion `t >= 0 && t < n_classes` failed on DistributedDataParallel

pytorchbearer: <p>Dear all,</p>
<p>I am running into a problem when training a segmentation net on Pascal VOC using multiple GPUs (DistributedDataParallel). Pytorch seems to expect targets to have label values only between 0 and n_classes–1 (inclusive). However, this does not consider the presence of the ignore label (whose value is often 255).</p>
<p>More specifically, I get the error:<br>
aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.</p>
<p>In my case, n_classes = 21 (Pascal VOC 2012) and my targets have label values in [0, 20] + [255] (for the ignore label). I ignore the 255 label in the CrossEntropy cost function. How do I get targets with ignore labels to pass the assertion test?</p>
<p>Interestingly, I don’t get this assertion failure when DistributedDataParallel is not used.</p>

ptrblck: <p>I cannot reproduce the issue using a single device as well as DDP.<br>
If I pass <code>ignore_index=255</code> to the criterion, the code runs fine.</p>
<p>Could you post an executable code snippet to reproduce this issue, please?</p>

pytorchbearer: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> . My MWE would be a bit convoluted as I am using DDP through pytorch lightning (I don’t think that is the source of this error, however). Could you please post the code snippet that works for you? I can modify that to illustrate the assertion failure.</p>

ptrblck: <p>I used the <a href="https://github.com/pytorch/examples/blob/a74badde33f924c2ce5391141b86c40483150d5a/imagenet/main.py">ImageNet example</a> and set one target value to <code>1001</code> using a single GPU and DDP.<br>
In both cases it crashed with the expected error message. After using <code>ignore_index=1001</code> both runs passed.</p>

pytorchbearer: <p>Thanks! I went back and checked the code very carefully. I had forgotten to ignore the same in the validation dataset and that is what failed the assertion…</p>
Data extension of hidden state in LTSM

Hasan1417: <pre><code class="lang-auto">h = tuple([each.data for each in h])
</code></pre>
<p>h stands for hidden state, my question is the data extension is built up function in LTSM?</p>

googlebot: <p>no, it is a tensor property, doing the same as each.detach()</p>

Hasan1417: <p>Thank you, very appreciated</p>
Handle bias when doing convolution with cudnn backend

Stonepia: <p>Hi there,<br>
I want to write a custom convolution_backward_overrideable function with cudnn backend.<br>
It should return a tuple with std::make_tuple(grad_input,grad_weight, grad_bias).</p>
<p>The weight and input could be handled by using <code>cudnn_convolution_backward_input</code> and  <code>cudnn_convolution_backward_weight</code>.</p>
<p>But according to <a href="https://github.com/pytorch/pytorch/pull/31524" rel="noopener nofollow ugc">31524</a>, handling bias inside convolution is deprecated.  However, I could not find examples or docs about how to handle bias in this case.</p>
<p>Is there any example on how to handle the bias? Or Is letting bias to 0 is acceptable?</p>

ptrblck: <p>You can add the <code>bias</code> directly to the <code>output</code> tensor as seen <a href="https://github.com/pytorch/pytorch/blob/2397c8d1f78cb5b48bcec9c180c1eef81c13b887/aten/src/ATen/native/Convolution.cpp#L707-L712">here</a>.</p>

Stonepia: <p>Pardon me. I don’t quite understand this.<br>
I want to return a tuple with three variables (grad_input,grad_weight, grad_bias), but adding the bias to output may means to handle bias out of this custom convolution_backward_overrideable behaviour,  this is a little difference than I expected.<br>
Could you please explain a little more about how to handle this? Thank you very much!</p>

ptrblck: <p>The <code>bias</code> is handled outside of the cudnn call, as it’s faster to let PyTorch add it explicitly and calculate its gradients.<br>
If you want to use the cudnn backward call, you can still use <a href="https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnConvolutionBackwardFilter"><code>cudnnConvolutionBackwardFilter</code></a> and <a href="https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnConvolutionBackwardData"><code>cudnnConvolutionBackwardData</code></a> manually.</p>
DistributedDataParallel() for Transformer model

gnadaf: <p>Done anyone know how to add <strong>DistributedDataParallel()</strong> for the transformer model?</p>
<p>Transformer model: <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
Purpose of `m` and `n` in coalesce?

vymao: <p>From the examples <a href="https://github.com/rusty1s/pytorch_sparse" rel="noopener nofollow ugc">https://github.com/rusty1s/pytorch_sparse</a>:</p>
<pre><code class="lang-auto">import torch
from torch_sparse import coalesce

index = torch.tensor([[1, 0, 1, 0, 2, 1],
                      [0, 1, 1, 1, 0, 0]])
value = torch.Tensor([[1, 2], [2, 3], [3, 4], [4, 5], [5, 6], [6, 7]])

index, value = coalesce(index, value, m=3, n=2)
</code></pre>
<p>What exactly is the purpose of <code>m</code> and <code>n</code>? They don’t seem to match the dimensions of the output and if I change them arbitrarily it doesn’t seem to make a difference, unless I make them small.</p>
How can we achieve this?

ecdrid: <p>Assume we have a Tensor like this <code>7927, 4, 7984, 45, 185, 55, 7876, 6, 172, 7898</code> and in your collate_fn you want to convert this to,</p>
<pre><code class="lang-auto">7927
7927, 4
7927, 4, 7984
7927, 4, 7984, 45
7927, 4, 7984, 45, 185
7927, 4, 7984, 45, 185, 55....
</code></pre>
<p>Is there something inbuilt which can do this, i don’t actually want to loop over to create it as i have around 3-4 tensors like that which have to undergo a similar thing.</p>
<p>Thanks a lot for your time.</p>
Function to set elements below learned cutoff to zero

vymao: <p>Is there a function/layer that can take in an input and set elements below some learned parameter to 0, leaving those above that learned parameter the same?</p>

Abhilash_Srivastava: <aside class="quote no-group" data-username="vymao" data-post="1" data-topic="101215">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vymao/48/30929_2.png" class="avatar"> vymao:</div>
<blockquote>
<p>set elements below some learned parameter to 0</p>
</blockquote>
</aside>
<p>What do you mean by this?<br>
Are you looking for <a href="https://pytorch.org/docs/stable/generated/torch.clamp.html" rel="noopener nofollow ugc">torch.clamp</a>?</p>

vymao: <p>Not exactly; clamp takes values lower than the minimum and brings them to the minimum. I am looking for the opposite in some sense - taking values lower than some threshold (but above 0) and bringing them to 0. In some sense I am looking for a right-shifted ReLU by some parameter, but that parameter should be learned and tunable.</p>
Suggested Environment for Developing C++/Cuda Extensions in Windows 10

JamesDickens: <p>I am trying to write a c++ extension using CUDA libraries in windows 10 following the tutorial <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">here</a></p>
<p>I have python 3.6.11, pytorch 1.8.0.dev20201021, rtx 3080 gpu, cuda 11.1.</p>
<p>I ended up getting pytorch c++ 1.7 (stable/debug) with cuda 11.0 working in Microsoft Visual Studio 2019 v 16.6.5 with the cl.exe compiler, and I am wondering what is the best way to write code with syntax completion, debugging abilities so that when I run python setup.py install, that I can be sure it will work.</p>
<p>Questions are what is the suggested environment/debugging tools for windows 10 ?<br>
Do I need to use Nvidia Nsight Compute to debug the Cuda Code?<br>
What flags will I need to pass in to the nvcc compiler (c++11 and maybe --gpu-architecture=compute_86 --gpu-code=sm_86)<br>
Should I use debug/release pytorch c++ ?</p>
<p>My Cmake file at the moment looks like:</p>
<pre><code class="lang-auto">cmake_minimum_required (VERSION 3.8)

project ("DAConvolution")

find_package(Torch REQUIRED)

add_library (DAConvolution "DAConvolution.cpp" )

# directory for python includes
include_directories("C:/Users/James/Anaconda3/envs/masters/include")

target_link_libraries(DAConvolution "${TORCH_LIBRARIES}" "C:/Users/James/Anaconda3/envs/masters/libs/python36.lib")

if (MSVC)
  file(GLOB TORCH_DLLS "${TORCH_INSTALL_PREFIX}/lib/*.dll")
  add_custom_command(TARGET DAConvolution
                     POST_BUILD
                     COMMAND ${CMAKE_COMMAND} -E copy_if_different
                     ${TORCH_DLLS}
                     $&lt;TARGET_FILE_DIR:DAConvolution&gt;)
endif (MSVC)
</code></pre>
<p>And my code looks like (so far with no compilation errors, as a library)</p>
<pre><code class="lang-auto">// DAConvolution.cpp
#ifdef _DEBUG
#undef _DEBUG
#include &lt;python.h&gt;
#define _DEBUG
#else
#include &lt;python.h&gt;
#endif

#include &lt;torch/extension.h&gt;

using namespace std;

#define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
#define CHECK_CONTIGUOUS(x) TORCH_CHECK(x.is_contiguous(), #x " must be contiguous")
#define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)

// CUDA Forward declarations

std::vector&lt;torch::Tensor&gt; DA_Conv_forward(
	torch::Tensor input_image,
	torch::Tensor depth_image,
	torch::Tensor bias,
	torch::Tensor weight,
	int stride,
	int dilation,
	int padding,
	float alpha
);

std::vector&lt;torch::Tensor&gt; DA_Conv_backward(
	torch::Tensor grad_output,
	torch::Tensor input_image,
	torch::Tensor depth_image,
	torch::Tensor bias,
	torch::Tensor weight,
	int stride,
	int dilation,
	int padding,
	float alpha
);

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
	m.def("forward", &amp;DA_Conv_forward, "DAConvolution forward");
	m.def("backward", &amp;DA_Conv_backward, "DAConvolution backward");
}
</code></pre>
<p>Any tips or insights very much appreciated, would be happy to provide more information.</p>
Pytorch equilvalent for Fourier based upsampling like scipy.signal.resample

catubc: <p>Hi.<br>
I’ve been looking for a way to upsample time series arrays using pytorch.  Currently I use scipy.signal.resample() which works great but there is a cost to transferring my data from device -&gt; cpu to run this function.</p>
<ol>
<li>
<p>Is there a native pytorch function that does this?  (note nn.functional.interpolate is linear as far as I can tell after plotting outputs).</p>
</li>
<li>
<p>If not, is there a way to use scipy without transferring my arrays from device-&gt;cpu?</p>
</li>
</ol>
<p>Thanks so much!<br>
Catubc</p>

ptrblck: <ol>
<li>
<p>I’m not sure how <code>scipy</code> implements <code>resample</code>, but maybe you can use the PyTorch <a href="https://pytorch.org/docs/stable/torch.html#spectral-ops" rel="nofollow noopener">Spectral OPs</a> to mimic the function.</p>
</li>
<li>
<p>Unfortunately not, since <code>scipy</code> uses <code>numpy</code> internally which only operates on the CPU.</p>
</li>
</ol>

catubc: <p>Ok, thanks.  I ended up calling on the expertise of one of our colleagues and he wrote a cuda function that does Bspline interpolation.   If there is time/interest, perhaps that could eventually be developed to be shared with the community.</p>

ksanjeevan: <p>Hey <a class="mention" href="/u/catubc">@catubc</a> , I’m having the exact same problem, any chance you could share the interpolation code? Thanks!</p>

catubc: <p>Hi Kiran.  I’m not sure you will find the interpolation code as useful as it has become quite complex and has been over-adapted to our specific problem which is not requires precomputation of bspline coefficients using scipy.  In other words, this will be very slow for most problems.  It’s also proprietary by the code-developer, but I can ask him to see if he’ll share it.</p>

soumickmj: <p>May be its too late, but I wrote a code for the same<br>
The code can be found at <a href="https://github.com/soumickmj/CodeSnippets/blob/main/PyTorch/signaltools.py" rel="noopener nofollow ugc">My GitHub Code</a> and also I wrote a small article for the same on <a href="https://discuss.pytorch.org/t/fourier-based-resampling-similar-to-scipy-signal-resample/101421">PyTorch Discuss</a></p>
Fourier-based resampling similar to scipy.signal.resample

soumickmj: <p>The signal resample function of SciPy (scipy.signal.resample) uses a Fourier method for up/downsampling. For many cases, it is observed that the performance of Fourier-based methods are better when compared against nearest, linear etc. interpolation (which are present under <code>torch.nn.functional.</code> <code>interpolate</code>).</p>
<p>My implimentation of the same can be found at:<br>
<a href="https://github.com/soumickmj/CodeSnippets/blob/main/PyTorch/signaltools.py" rel="noopener nofollow ugc">My GitHub code</a></p>
<p>This resample has one extra functionality than the scipy version. This can work with multiple axes for interpolation. The scipy version can only work with one axis. With this, one can just send multiple dimension sizes as a list to “num” and the axes list to “axis”.</p>
Calculating gradients for previous layers with only intermediate gradient input?

harryhotspur: <p>I’m trying to solve a problem, and it’s maybe a bit strange so I’ve had a lot of difficulty reaching a solution. I have a model which consists of two components: the first is constituted of convolutional layers which produces a 3D block of output, and the second uses this output as weights for linear layers. The loss is then calculated based on the final output of the whole thing. I realize how strange this is, but it would be long-winded to explain why I believe I need the model to be set up like this.</p>
<p>On the forward pass, things work alright and I can obtain a sensible output. On the backward pass I can calculate the gradients of the linear layers, but I need to re-package these (essentially just re-shaping/concatenation) so that they can be propagated back to the convolutional layers. So what I believe I need is a way to calculate the gradients of the convolutional layers with only these intermediate, re-packaged gradients.</p>
<p>I suppose my first question is whether this is even possible? I’ve been going back and forth on it, but I don’t see why it would not be possible mathematically. And if so, my second question is whether there’s a concise way to do it in PyTorch already, or if I would need to make everything myself?</p>
<p>Thank you.</p>
Batch calls to different networks with same input

neighthan: <p>The standard batching method is to provide multiple inputs to a single network. Does PyTorch have support for batching calls with the same input to multiple networks, provided those networks have the same architecture, just different weights?</p>

Abhilash_Srivastava: <p>If you want to feed the same input (or batch of inputs) to multiple networks, you can create a list of modules and optimizers. Just use these inside the training, like you would use a single network. Something like this should work.</p>
<pre><code class="lang-auto">models = list([mode1, model2,...., model10])
optimizers = list([optimizer1, optimizer2,...., optimizer10])

for model in models:
     model.train()
            
for batch in train_loader:
     inputs, targets = batch['input'].to(device), batch['target'].to(device)
     for c in range(classnum):
          optimizers[c].zero_grad()
                
                             :
                             :
                             :
           outputs = models[c](each_input)
           loss = criterion(outputs, targets)        
           loss.backward()
           optimizers[c].step()

           running_loss += loss.item()
</code></pre>

neighthan: <p>This runs each model on the input sequentially. What I’m looking for is a way to run multiple models on the same input in parallel on the GPU (the same way that one model can run on multiple inputs in parallel instead of sequentially).</p>

Abhilash_Srivastava: <p>You can read the data just once (with the <code>DataLoader</code>). Then, create multiple models. Next, push each one to the desired device via <code>to('cuda:id')</code> and simply pass the data to each model.<br>
Now, the training happens on different devices, so it should be executed in parallel.<br>
If you want to train a model using multiple devices, set <code>CUDA_VISIBLE_DEVICES</code>.</p>
Complete beginner - where to start

chiron: <p>Hi everyone,</p>
<p>New guy here. I’m a complete beginner to programming but I’m excited and eager to start learning. I was referred to Pytorch and Keras when I expressed an interest in AI programming but I’m curious if this is where I’d start as a complete beginner?</p>
<p>Can I just jump into Pytorch or should I work on getting an understanding of Python first?</p>

albanD: <p>Hi,</p>
<p>You should definitely find an introduction tutorial on python online first.<br>
Then for pytorch, the 60min blitz is the best place to start I think: <a href="https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html">https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html</a></p>
How to warp STL file using flow from STN network?

ru6928: <p>I have a STN algorithm that generates flow for two 3D image, say the inputs are (1x1x96x96x21), the output flow of STN is (1x3x96x96x21). How do I use to this flow output to warp a decimated .stl mesh file with completely different dimensions using grid_sample?</p>
Fourier transform and complex dtype restrictions

juliusbk: <p>Hi,</p>
<p>Many operations do not seem to be implemented for the dtype=complex yet. I am currently facing this problem:</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torch.fft

x = torch.tensor(np.random.random((30, 20, 10)), requires_grad=True)
t = torch.fft.rfft(x, dim=2)
t = torch.prod(t, dim=1)  # or torch.exp(torch.sum(torch.log(t), dim=1))
xn = torch.fft.irfft(t, dim=1)
</code></pre>
<p>which gives <code>RuntimeError: prod does not support automatic differentiation for outputs with complex dtype</code>. Using <code>torch.log</code> results in similar problems.</p>
<p>Do you guys have any idea how to get around this problem for the above situation?</p>
<p>Thank you in advance.</p>

albanD: <p>Hi,</p>
<p>The issue here is not with the fft but the <code>torch.prod</code> operation that does not support autograd for complex inputs at the moment.<br>
Complex autograd is work in progress but you can open a feature request on github if you need it now.</p>

juliusbk: <p>Thank you for your reply. I should have been more clear in formulating the problem.</p>
<p>I’m well aware that the problem is with <code>prod</code>, I only included the fft part to show the context – perhaps someone smart had an idea how to get around the problem knowing the context. I have a hunch that the complex dtype could be avoided entirely. Perhaps.</p>
<p>Anyway, thank you. Will open the issue.</p>

albanD: <p>You might want to try to do the multiplication by hand as I think the regular multiplication is already supported:</p>
<pre><code class="lang-auto"># replace t = torch.prod(t, dim=1) by
dim = 1
res = t.select(dim, 0)
for i in range(1, t.size(dim)):
  res = res * t.select(dim, i)
t = res
</code></pre>

juliusbk: <p>Great idea! Didn’t think about that these would be separate implementations.</p>
<p>Will try tomorrow. Thank you.</p>

juliusbk: <p>Works like a charm! Thank you again.</p>
Embedding initialization and layer initialization

dhyey: <p>i was making a nn with just categorical features , hence used nn.Embedding , after which i applied linear layer!<br>
and i found out that the output distribution does not nave 1 as standard deviation, seems to me because embedding are initalized with normal(0,1) distribution and layer with uniform distribution!<br>
hence if the std is not 1 with increasing depth of network std of output must tend towards 0 , hece vanishing gradients!!<br>
so should i change the type of initialization or it will work fine , with no problems??<br>
becuz to me seems to be a bit odd!!!</p>
<p>i tried a deep network of 8 layers and found that output std is 0.09 and same is the gradients std , hence it seems to me a vanishing gradients!</p>
How to set QNNPack parallelism?

dbalchev: <p>For our usecase we need to do the job on one thread (preferable the main one). For all other workflows we just call <code>torch.set_num_threads(1)</code> and this gets the job done <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:">.<br>
But QNNPack (more specifically the quantized Convolution) doesn’t respect this setting,  since it uses <a href="https://github.com/pytorch/pytorch/blob/v1.6.0/aten/src/ATen/native/quantized/cpu/qconv.cpp#L606" rel="noopener nofollow ugc"><code>caffe2::pthreadpool_()</code></a>.<br>
In our tests we get good results by making that function return <code>null</code>.</p>
<p>Is there a way to set the <code>num_threads</code> of that pool without recompiling pytorch?<br>
If not, what’s the minimal acceptable refactoring to make that configurable or respect the <code>torch.set_num_threads</code>?</p>

jerryzh168: <p>This is related to mobile, can you add a ‘mobile’ tag?</p>

dbalchev: <p>Our use-case is not for mobile. We plan to use QNNPACK if the machine we’re running on doesn’t support AVX2.</p>
<p>Should I add the mobile tag despite that?</p>

dbalchev: <p>I’m not sure how to add the mobile tag?</p>

ptrblck: <p>Changed the category for you. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

dbalchev: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks (:<br>
<a class="mention" href="/u/jerryzh168">@jerryzh168</a> any updates on this?</p>

kimishpatel: <p><a class="mention" href="/u/dbalchev">@dbalchev</a>, you can try <code>caffe2::pthreadpool()-&gt;set_thread_count(1)</code>.</p>

dbalchev: <p><a class="mention" href="/u/kimishpatel">@kimishpatel</a> We’re using the python wheel for inference. I’ve tried recompiling pytorch, so that <code>torch.set_num_threads</code> calls <code>pthreadpool()-&gt;set_thread_count</code> as well (<a href="https://github.com/pytorch/pytorch/pull/45566" rel="noopener nofollow ugc">PR with the rebased commit</a>). It works, but we have to ship a custom pytorch wheel. I didn’t find a way to do that through the python API or even with a C++ extension.</p>

kimishpatel: <aside class="quote no-group" data-username="kimishpatel" data-post="7" data-topic="97359">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kimishpatel/48/39320_2.png" class="avatar"> kimishpatel:</div>
<blockquote>
<p>caffe2::pthreadpool()-&gt;set_thread_count(1)</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/dbalchev">@dbalchev</a> sorry for late response. For some reason I did not get any notification. Unfortunately at the moment we dont have python API for this. Possibly in next release, or we can thin of introducing it in master if that may work for you.</p>

dbalchev: <p><a class="mention" href="/u/kimishpatel">@kimishpatel</a> Thanks for the reply! In my PR I set the size of that pool in <code>torch.set_num_threads</code> and it’s approved (:</p>
Pruning: How to know - How much and which layers to prune?

bing: <p>Hi everyone,<br>
I am building an image classification model.<br>
I want to compress the trained model. I came across this tutorial by  - <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html#extending-torch-nn-utils-pruning-with-custom-pruning-functions" rel="noopener nofollow ugc">pytorch</a><br>
Is there a defined way or checklist on how to proceed to pruning, like how to decide which pruning method to choose, which modules to prune and how much to prune?</p>
How to refresh an iterator on DataLoader

jj0mst: <p>I would like to use an iter on DataLoader instead of a for loop. At the beginning of each training step I use the following code:</p>
<pre><code class="lang-auto">train_loader = DataLoader(dataset=train_dataset, batch_size=8, shuffle=True, num_workers=4)
train_loader_iter = iter(train_loader)
while(True):
    try:
        data_dict = next(train_loader_iter)
    except StopIteration:
        print('Refreshing iterator...')
        train_loader_iter = iter(train_loader)
        data_dict = next(train_loader_iter)
        print('Iterator refreshed...')
</code></pre>
<p>However when my iterator should refresh (i.e. I read ‘Refreshing iterator…’) the process hangs and gets stuck infinitely. Am I doing something wrong or this is just not possible? I’m on Pytorch 1.6.0 and my dataset is a simple subclass of the Dataset class.</p>

suraj.pt: <p>Your code is looping infinitely over the data (that’s why it looks like your process is hung). You can limit the number of times you ‘refresh’ your iterator  (eg: 100) by replacing <code>while True</code> with <code>for _ in range(100)</code>.</p>

jj0mst: <p>That is not the problem I’m having, but exactly what I want. The problem is that the first time i refresh the iterator my process gets stuck. It would happen also if I swap the “while(True)” with “for _ in range(2)”</p>

suraj.pt: <p>Hmm, I can’t reproduce this issue on an example dataset. Can you check if you have the same problem with another dataset?</p>
Best Annotation tool with Pytorch model (Active Learning)

tcapelle: <p>I am looking on feedback on Image annotation tools that support Pytorch model in the loop.<br>
My task is segmenting images with polygons masks. I am currently using CVAT but have not been able to connect my UNET to CVAT.<br>
What tools are people using to annotate images?<br>
Any best practices I should know?<br>
Thanks!</p>
Suppress warning in CSRC in source code

ArtistBanda: <p>Is there a way to suppress a warning generated by a particular TORCH_WARN_ONCE and remove that suppression after a function is called?</p>

albanD: <p>Hi,</p>
<p>You could try to use something similar to the utility class <a href="https://github.com/pytorch/pytorch/blob/b64fc3c4b5d927928770f9b343eb845123367084/test/cpp/api/support.h#L39-L64">here</a> that we use to detect warnings during our c++ tests. Would that work for you?</p>
Implement a linear transformation on one of the model weights

NimB: <p>Hey,<br>
I want to implement a certain improvement for a Vanilla RNN.<br>
The regular format of the network is</p>
<pre><code class="lang-auto">h_t = tanh( W_ih x_t + b_ih + W_hh h_t-1 + b_hh)
</code></pre>
<p>I want to add the next change:</p>
<pre><code class="lang-auto">h_t = tanh( W_ih x_t + b_ih + C(W_hh) h_t-1 + b_hh)
</code></pre>
<p>Where C is a linear function and C(W_hh) is a linear transformation of W_hh</p>
<p>How would you recommend implementing this change?<br>
Thanks</p>
<p>​</p>
Tensorflow to pytorch format

Rabeeh_Karimi: <p>Hi,<br>
I have a tf.data.Dataset format data which I get it through a map function as below:</p>
<pre><code class="lang-auto"> dataset = source_dataset.map(encode_tf, 
        num_parallel_calls=tf.data.experimental.AUTOTUNE)
 
def encode_tf(inputs):
      inputs_plaintext = inputs['inputs_plaintext']
      targets_plaintext = inputs['targets_plaintext']
      encoded = tf.py_function(self.encode, [inputs_plaintext, targets_plaintext], [tf.int32, tf.int32, tf.int32, tf.int32])
      input_ids, input_attention, target_ids,target_attention = encoded
      input_ids.set_shape([None])
      target_ids.set_shape([None])
      input_attention.set_shape([None])
      target_attention.set_shape([None])
      data =  {'input_ids': input_ids,
            'labels':target_ids, 
            'attention_mask': input_attention,
            'decoder_attention_mask': target_attention}
      return data 
</code></pre>
<p>I need to convert each element of data which is a dictionary to pytorch tensor values, is there a way I could convert the format ? Is there a function like map in tensorflow to do this reformatting fast?<br>
thanks.</p>
How can I fix torch.cuda.is_available()

hell0w0rld: <p>I was trying to set up torch on my pc. I’m using Windows 10 with python 3.8.6.</p>
<p>I installed torch with this command<br>
<code>pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html</code>.</p>
<p>Because I didn’t install CUDA I got <code>False</code> as output at the command <code>torch.cuda.is_available()</code><br>
But now I don’t know which is the right driver and CUDA-Version for my NVIDIA GeForce 920M.</p>
<p>Can someone please tell me which versions I have to install?</p>

ptrblck: <p>You’ve installed the CPU pip wheel, so the CUDA runtime isn’t shipped.<br>
That being said, your GPU uses a compute capability of 3.5, which isn’t supported in the pre-built binaries anymore (min. architecture is 3.7), so you would need to build from source using <a href="https://github.com/pytorch/pytorch#from-source">these instructions</a>.</p>
Freezing inner layers of VAEs

Manuel_Kurz: <p>Hi,<br>
I’m working with VAEs, 3 individual ones: vae1, vae2, vae3. In vae3 i’m loading the pre-trained encoder of vae2 and the pre-trained decoder of vae1. After the decoder i add some layer aswell.</p>
<p>My problem is now: I want to freeze the decoder. Therefore, just the encoder and the last layer should be trained.</p>
<p>I’ve read a lot about freezing layers but i’m still not sure if it is enough to run through the decoder (existing of Conv2d, ConvdTrans, BatchNorm, LeakyRelu) and just set require_grad=false… The optimizer is initialized with model.parameters(). Is this the proper way? Will the encoder and the last layer be correctly optimized?<br>
Thank you!</p>
<p>EDIT: for clarification: the architecture is basically a variational autoencoder, which consists of a trainable encoder, a froze decoder and some trainable layer after the decoder.</p>

tlsshh: <p>Sorry for I can not completely understand the structure of your network, but I think your main question is how to freeze part of the network with conv layers and BN layers.</p>
<p>You can achieve it by:</p>
<ol>
<li>Set the decoder to eval mode:<pre><code class="lang-auto">decoder.eval()
</code></pre>
This is used to freeze BN layers (and dropout). In BN layers, besides parameters, there are buffers which are not optimized by the optimizer but updated automatically during forwarding in training mode. Please see explanation at <a href="https://discuss.pytorch.org/t/how-to-properly-fix-batchnorm-layers/27935">How to properly fix batchnorm layers</a>.</li>
<li>Exclude decoder parameters from the optimizer.</li>
<li>(Optional)  set:  <code>requires_grad=false</code>. I think this is mainly to speed up training and save memory. If not, please tell me.</li>
</ol>

Manuel_Kurz: <p>Thanks for your response.<br>
I edited the post, maybe the architecture is now more clear.</p>
<p>Setting the whole decoder into evaluation mode with decoder.eval(); are gradients properly calculated for this inner layers and is backpropagation done correctly from the last layer until the encoder?</p>
<p>What is the benefit of telling the optimizer not to optimize the decoder anymore? I thought that the decodee wouldn’t be optimized anyways due to freezing weights/biases.</p>

tlsshh: <p>The only way to “freeze weights and biases” is to exclude parameters from the optimizer. eval() has no effect on the convolutional layers. It is used to freeze BN and dropout. Besides, excluding parameters can save GPU memory.</p>
<p>.eval() does not affect the gradient calculation, but I am not sure if the gradient can be properly calculated when the inner layers are set <code>requires_grad=false</code></p>
How to use SyncBn in pytorch 1.0?

xwjBupt: <p>when will pytorch to support Synchronized Batch Normalization?and how to use it in pytorch 1.0?</p>

ptrblck: <p><a href="https://pytorch.org/docs/master/nn.html#torch.nn.SyncBatchNorm" rel="nofollow noopener">nn.SynchBatchNorm</a> was added into the PyTorch Master branch 9 days ago via <a href="https://github.com/pytorch/pytorch/pull/14267" rel="nofollow noopener">this PR</a>. Alternatively, you could use the NVIDIA/apex implementation from <a href="https://github.com/NVIDIA/apex/tree/master/apex/parallel#synchronized-batch-normalization" rel="nofollow noopener">here</a>, if you don’t want to build PyTorch from source.</p>

xwjBupt: <p>thanks for your reply!! can i update my pytorch version through</p>
<pre><code class="lang-auto">conda update pytorch torchvision
</code></pre>
<p>to update my pytorch to get syncbn?</p>

ptrblck: <p>You could try to install the nightly builds and check if the module is already there.<br>
Alternatively you could build from source.</p>

xwjBupt: <p>thanks for for your reply!!</p>

Alpha: <p>hi, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>What’s the major difference between <code>nn.SynchBacthNorm</code> and <code>NVIDIA/apex</code>? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>
<p>Or which one tends to achieve a better performance? <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

ptrblck: <p>We recommend to use <code>nn.SyncBatchNorm</code>, as you shouldn’t see a large difference performance-wise.</p>

Peter_Pack: <p>How to freeze the parameters of SyncBatchNorm?</p>
Minimum Loss Over A Column

Jack_Rolph: <p>Suppose I have a tensor <code>output</code>, which is the result of my network and a tensor, ‘target’, which is my desired output:</p>
<pre><code class="lang-auto">output.shape = torch.Size([B, C, N])
target.shape = torch.Size([B, C, N])
</code></pre>
<p>I am interested that the network predicts a given <code>N</code> correctly, and not the particular permutation of   <code>C</code> that is given by the network as output, as this cannot be ordered.</p>
<p>For this reason, I would like to calculate the loss for each possible permutation of <code>C</code> in <code>output</code> and <code>input</code> respectively, taking the minimum possible overall loss.</p>
<p>To demonstrate in normal code what I want to do, it would be written in normal Python script as follows:</p>
<pre><code class="lang-auto">import torch

def Loss(target, output):
    loss = 0
    min_loss=0
   
   #Calculate minimum MSE and add to loss value
    for b in range(target.shape[0]):
        for c_i in range(target.shape[1]):
             for c_ii in range(target.shape[1]):
                  loss_temp = torch.sum(target[b, c_i] - output[b,c_ii])**2)
                  if(c_ii == 0 or loss_temp &lt; min_loss):
                     min_loss = loss_temp
              loss = loss + min_loss

   #Calculate mean over batches
   loss = loss/target.shape[0]

   return loss
</code></pre>
<p>Is there a more elegant, PyTorch-oriented way of perform this operation?</p>

Jack_Rolph: <p>I have made an attempt, with the following method:</p>
<ul>
<li>
<p>utilize itertools.permutations to index one of the tensors;</p>
</li>
<li>
<p>repeat the other tensor along a new axis;</p>
</li>
<li>
<p>calculate sum of square difference along last axes</p>
</li>
<li>
<p>find minimum along the new, permutation axis</p>
</li>
<li>
<p>take average over batch.</p>
</li>
</ul>
<pre><code class="lang-auto">def Loss (target, input):
        #calculate indices
        idx = torch.from_numpy(np.array(list(itertools.permutations(range(output.shape[-2])))))
        #index tensor with indices
        input_perms = input[:, idx, :]

        #repeat other tensor to same length
        target_perms = target.unsqueeze(1).repeat(1,len(idx),1,1)

        #calculate sum of squares
        losses = (input_perms - target_perms)**2
        losses = losses.flatten(start_dim = -2)
        loss_len = losses.shape[-1]
        losses = torch.sum(losses, dim=-1)
        
        #calculate minimum along permutation axis and then mean along batch axis. Remember, we need still to divide through by the number of entries in each of the parts onf the tensor we summed!
        min_loss = torch.mean(losses.min(dim=-1, keepdim=True)[0])/loss_len
        return min_loss
</code></pre>
Changed to 64bit Python, Still Can't Install Torch

Rob1: <p>I know this is a common issue but when I try to run the command <code>pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html</code>, I get the error:</p>
<pre><code class="lang-auto">C:\Users\robsa\AppData\Local\Programs\Python\Python39\Scripts&gt;pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.6.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.6.0+cpu
</code></pre>
<p>I was initially using 32bit Python but uninstalled that, and installed Python 3.9 64 bit. I still get the same error.<br>
I have been using MSYS2 to compile C++ code. I think this comes with a version of Python as when I run ‘python -v’ in cmd it shows a lot of file paths pointing to a Python installation inside the MSYS2 folder.</p>
<p>Here is what happens when I run <code>pip -V</code>:</p>
<pre><code class="lang-auto">C:\Users\robsa\AppData\Local\Programs\Python\Python39\Scripts&gt;pip -V
pip 20.2.4 from c:\users\robsa\appdata\local\programs\python\python39\lib\site-packages\pip (python 3.9)
</code></pre>
<p>Running <code>pip -V</code> from a folder other than scripts:</p>
<pre><code class="lang-auto">c:\&gt;pip -V
pip 20.2.4 from C:\msys64\mingw32\lib\python3.8\site-packages\pip (python 3.8)
</code></pre>
<p>Cheers for any help in advance!</p>

Ackeys: <p>Did you find anything? I’m facing the same problem.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/69f51636336538a79a65ec44c9bc78c29cf5974e.png" data-download-href="https://discuss.pytorch.org/uploads/default/69f51636336538a79a65ec44c9bc78c29cf5974e" title="problems"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/9/69f51636336538a79a65ec44c9bc78c29cf5974e.png" alt="problems" data-base62-sha1="f7lehxfIyOpKqyzuIwMlkwCpe8e" width="690" height="362" data-dominant-color="211F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">problems</span><span class="informations">972×510 17.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Element wise binary classification

Luca_Viano: <p>Hi!</p>
<p>I was wondering which is the correct way to perform element wise binary classification using the BCEloss.</p>
<p>My model outputs a tensor of shape (depth, width, length) and its last activation is an element wise Sigmoid.</p>
<p>My target contains either 0 or 1 and has also shape (depth, width, length).</p>
<p>I am currently computing the loss with the default reduction “mean” and I backpropagate this.</p>
<p>The mean loss decreases as expected however my model predicts for all pixels probabilities that are lower than 0.5 so all the elements equal to 1 are wrongly classified.</p>
<p>I think that this is mainly due to the fact that the 1s are rare in the tensor.</p>
<p>Should I change loss? In particular I was wondering if I should use reduction=none? If yes how can I backpropagate this elementwise loss?</p>
<p>Thanks in advance,<br>
Luca</p>

KFrank: <p>Hi Luca!</p>
<aside class="quote no-group quote-modified" data-username="Luca_Viano" data-post="1" data-topic="101260" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luca_viano/48/30501_2.png" class="avatar"> Luca_Viano:</div>
<blockquote>
<p>element wise binary classification using the BCEloss.</p>
<p>its last activation is an element wise Sigmoid.</p>
</blockquote>
</aside>
<p>As an aside, you will have better numerical stability if you use<br>
<code>BCEWithLogitsLoss</code> and remove the final <code>Sigmoid</code> layer.</p>
<aside class="quote no-group">
<blockquote>
<p>my model predicts for all pixels probabilities that are lower than 0.5 so all the elements equal to 1 are wrongly classified.</p>
<p>I think that this is mainly due to the fact that the 1s are rare in the tensor.</p>
</blockquote>
</aside>
<p>Consider using the <code>pos_weight</code> argument passed to the constructor<br>
of <code>BCEWithLogitsLoss</code> to compensate for the rarity of “1” pixels.</p>
<p>A typical value for <code>pos_weight</code> to reweight the rare values would be<br>
<code>approx_number_of_1_pixels / approx_number_of_0_pixels</code>.</p>
<p>(The value for <code>pos_weight</code> does not need to be especially precise.)</p>
<p>Note:  You don’t need to use 0.5 as the threshold to convert predicted<br>
probabilities (or 0.0 to threshold <em>logits</em>) to 0/1 values.  You can lower<br>
the threshold to predict more 1s (but using <code>pos_weight</code> is likely to be<br>
the better approach).</p>
<p>Best.</p>
<p>K. Frank</p>

Luca_Viano: <p>Hi KFrank,</p>
<p>I am grateful for your quick answer with useful suggestions.</p>
<p>I will try these changes <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>