akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Test_deep_reentrant Bus Error

igorsyl: <p>Has anyone experienced this issue?</p>
<p>python test/test_autograd.py -k test_deep_reentrant -v<br>
test_deep_reentrant (<strong>main</strong>.TestAutograd) … Bus error: 10</p>
<p>Environment:<br>
print(sys.version)<br>
3.7.3 (default, Mar 27 2019, 16:54:48)<br>
[Clang 4.0.1 (tags/RELEASE_401/final)]</p>
<p>platform.platform()<br>
‘Darwin-18.7.0-x86_64-i386-64bit’</p>
<p>commit 60c4e74e49770ca7204b812006926ac0778d30df<br>
Date:   Fri Aug 30 20:57:45 2019 -0700</p>
<p>Build:<br>
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/…/"}<br>
MACOSX_DEPLOYMENT_TARGET=10.9<br>
CC=clang<br>
CXX=clang++<br>
USE_CUDA=0<br>
USE_MKLDNN=0<br>
python setup.py develop</p>
<p>Thanks!</p>
Source code of torch.logsumexp

Kevin96: <p>Hi,</p>
<p>In my implementation, I need logsumexp to stabilize the computation. It works well in high dimension. But it seems like I still encounter numerical issues (nan) in 1-dimensional experiments.</p>
<p>However, I couldn’t find the source code for this function. How is it implemented? Thanks!</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/blob/01d98c7cfb2031fd5ab4148444a2d34a171e700c/aten/src/ATen/native/ReduceOps.cpp#L333" rel="nofollow noopener">This</a> should be the current implementation.</p>
How to add noise to inputs as a function of input

lobachevzky: <p>I am using <code>torchvision.transforms.Lambda</code> to apply noise to each input in my dataset:</p>
<pre><code class="lang-auto">torchvision.transforms.Lambda(lambda x: x + torch.rand(x.shape))
</code></pre>
<p>The problem is that each time a particular image is sampled, the noise that is added is different. I would like to apply the noise up front (not during training) so that every time I sample a particular image the noise is the same.</p>
<p>Thanks!</p>

ptrblck: <p>If you would like to apply the same random noise to the corresponding image tensor, you could initialize a noise tensor in your <code>Dataset</code>'s <code>__init__</code> function with the same length as the number of images, and add it to each tensor after the transformation.</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, image_paths, targets, transform=None):
        self.image_paths = image_paths
        self.targets = targets
        self.transform = transform
        self.noise = torch.randn(len(self.image_paths), 3, 224, 224)
        
    def __getitem__(self, index):
        x = Image.open(self.image_paths[index])
        y = self.targets[index]
        if transform is not None:
            x = self.transform(x)
        x = x + self.noise[index]
        return x, y
    
    def __len__(self):
        return len(self.image_paths)
</code></pre>

lobachevzky: <p>Thanks for the tip. Ended up going with:</p>
<pre><code class="lang-auto">class NoiseDataset(datasets.MNIST):
    def __init__(self, *args, percent_noise=1, **kwargs):
        super().__init__(*args, **kwargs)
        self.percent_noise = percent_noise
        self.noise = torch.randn(self.data.shape)

    def __getitem__(self, index):
        x, y = super().__getitem__(index)
        noise = self.noise[index]
        x = noise * self.percent_noise + x * (1 - self.percent_noise)
        return x, y
</code></pre>

ptrblck: <p>That’s really an elegant approach to derive this class from the parent dataset. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Thanks for sharing!</p>
Fastest way to modify tensor using indices

Joseph_Fisher: <p>I need to extract elements of a tensor (at present done using a boolean mask), do some operations on them, and replace in the original tensor. The basic operations look like this, where index is a boolean tensor.</p>
<pre><code>new = out[index]

# Do some operations on new here 

rt = out.clone()
rt[index] = new
</code></pre>
<p>Is there a faster way of doing this? In the test below, training time is roughly doubled (though depends on how many elements of index are True)</p>
<pre><code>class test(nn.Module):

    def __init__(self):
        super(test, self).__init__()
        self.a = nn.Parameter(torch.ones([100, 200]))
    
    def forward(self, x, index=None):
    
        out = torch.matmul(self.a, x)
    
        if index is not None:
            new = out[index]     
            rt = out.clone()   
            rt[index] = new
        else:
            rt = out
    
        return rt

model = test()
opt = torch.optim.SGD(lr=0.01, params=model.parameters())
loss = torch.nn.MSELoss()

ixs = (torch.arange(100) &gt; 50)

st = time.time()
for _ in range(1000):

    x = torch.ones([200, 400]) + 10
    target = torch.zeros([100, 400])

    out = model(x, ixs)

    l = loss(out, target)

    l.backward()
    opt.step()
    model.zero_grad()

print(time.time() - st)</code></pre>

albanD: <p>Hi,</p>
<p>I’m afraid this is expected.<br>
Your original forward is only a matmul (the most optimized op we have) and you add 3 more operations.</p>
<p>You’ll get better performances if you use indices and index_select/index_copy or gather/scatter if the number of indices is small.</p>

Joseph_Fisher: <p>Thanks! Will have a go with index_select/index_copy and make a comparison. It’s the speed of extracting and re-inserting the elements of the tensor I’m interested in! Interesting that using indices is quicker with a small number.</p>

albanD: <p>Well the complexity of the operation with a mask is of the size of the Tensor. Because for every entry, it needs to check the mask value.<br>
For the indices, it just reads the values at the given indices, so the complexity is the number of indices.</p>

Joseph_Fisher: <p>Makes sense - thanks a lot!</p>
<p>The other aspect I’m wondering about in the above e.g. is that if I change the line</p>
<pre><code>ixs = (torch.arange(100) &gt; 50)
</code></pre>
<p>to</p>
<pre><code>ixs = (torch.arange(100) &gt; 99)
</code></pre>
<p>I get back to the baseline speed. So it seems the slowdown is not the indexing directly, but due to a copy being made of the elements of the matmul output where the indices that are True, and operations on this copy (hence the difference depending on how many elements of ixs are True). Is that kind of correct?!</p>

albanD: <p>In your code sample, you actually clone the input, not only the element you index.<br>
But anyway, the smaller the indexing, the faster it’s gonna be.</p>
Importing a single jit compiled function (not a module) into cpp

Developers_Fox: <p>Let’s say I have compiled a function (from python with jit) as;</p>
<p><span class="mention">@script</span><br>
def process_sequence(sequence):<br>
for t in range(sequence.size(0)):<br>
process_timestep_traced(sequence[t])</p>
<p>save(process_sequence, ‘process_seq.torchjit’)</p>
<p>// until here everything in python, works perfect.</p>
<p>// main part for my question ; in my cpp file (where libtorch is configured &amp; built perfect) I have ;</p>
<p>std::vector<a>torch::jit::IValue</a> sequence_in;     // just creating a sequence for testing<br>
sequence_in.push_back(torch::randn({1, 2}));<br>
sequence_in.push_back(torch::randn({1, 2}));</p>
<p>// These lines below are the cause of my confusion…</p>
<p>torch::jit::script::Module process_seq = torch::jit::load(“process_seq.torchjit”);<br>
torch::Tensor sequence_out = process_seq.forward(sequence_in).toTensor();</p>
<p>My question is:<br>
How do I call a single jit function (not a module) into cpp, and how can I execute that function?</p>
<p>ps: I’m definitely not a user of cpp… I know of namespaces, but the problem is which import from torch::jit::script:: I use as my variable type for the imported function…</p>
<p>Thank you in advance</p>
<p>Edit: The error msg I get is ;</p>
<p>libc++abi.dylib: terminating with uncaught exception of type c10::Error: Expected at most 2 argument(s) for operator ‘forward’, but received 3 argument(s). Declaration: forward(ClassType&lt;process_seq&gt; self, Tensor sequence) -&gt; (None) (checkAndNormalizeInputs at …/aten/src/ATen/core/function_schema_inl.h:134)</p>
<p>Abort trap: 6</p>
Metric for Multi Label Classification

shangeth: <p>Hi,<br>
I have a multi label problem. i have coded the model, used BCEloss(with logits), can get the prediction from the model. But i am confused on what metric to use. I used accuracy score, but its obvious accuracy score will give good score</p>
<p>label = [1,0,0,0,0,0,0,0,1,0,0,0]<br>
pred = [0,0,0,0,0,0,0,0,0,0,0,0]</p>
<p>This means even a 0-model(returning 0 always) gets more accuracy.<br>
But i searched the whole internet(may be not whole), people don’t usually use metric or use accuracy score as metric.</p>
<p>Shouldn’t we use a metric which considers the positive labels and predictions like precision, recall…?</p>

RegularGuY: <p>You can use F1 score, i think F1 micro should work,I have also used kappa score in the past.Not sure if it will be useful in your usecase.</p>
Maxpool 1d shape changing

chaslie: <p>Hi,</p>
<p>I’ve just added a Maxpool1d into a CVAE, and the shape is changing from &lt;x1,y1,z1&gt; to &lt;x2,y2&gt; ie its changing from a 3D tensor to a 2D tensor, any ideas why this is?</p>
<p>Cheers,</p>
<p>Chaslie</p>

spanev: <p>Hi <a class="mention" href="/u/chaslie">@chaslie</a><br>
What is exactly the input shape and what are the arguments of <code>MaxPool1d</code>?</p>

chaslie: <p>Hi Spanev,</p>
<p>the code line is</p>
<p><code>self.pool=nn.MaxPool1d(22,stride=1,padding=10,dilation=1)</code></p>
<p>The input shape is &lt;2,128,66&gt;<br>
the output shape is &lt;128,65&gt;</p>
<p>the output shape is correct when i  calculate, except it is now a 2D tensor and not 3D which gives:</p>
<ul>
<li></li>
</ul>
<blockquote>
<p>RuntimeError: Expected 3-dimensional input for 3-dimensional weight 128 128, but got 2-dimensional input of size [128, 65] instead</p>
</blockquote>
<ul>
<li></li>
</ul>

chaslie: <p>Having adjustedf the maxpool line to:</p>
<pre><code class="lang-auto">self.pool=nn.MaxPool1d(22,stride=1,padding=10,dilation=1,return_indices=True)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">def forward(self, data):
       data,indices=self.pool(data)
       return data, indices
</code></pre>
<p>Which is fed into a decoder using:</p>
<pre><code class="lang-auto">        self.unpool=nn.MaxUnpool1d(22,stride=1,padding=10)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">    def forward(self, z2,indices):
         z2=self.unpool(z2,indices, output_size=input.size())
</code></pre>
<p>I am now getting the error:</p>
<pre><code class="lang-auto">    model(*x)

  File "d:\Users\CHARLIE\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)

TypeError: forward() missing 1 required positional argument: 'indices'
</code></pre>

spanev: <p>It might come from somewhere else in your model, since the input data shape and layer you give produces a <code>[2, 128, 65]</code> shaped output:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; pool=nn.MaxPool1d(22,stride=1,padding=10,dilation=1)
&gt;&gt;&gt; t = torch.rand(2,128,66)
&gt;&gt;&gt; pool(t).shape
torch.Size([2, 128, 65])
</code></pre>
<aside class="quote no-group" data-username="chaslie" data-post="4" data-topic="54785">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/48.png" class="avatar"> chaslie:</div>
<blockquote>
<p>model(*x)</p>
</blockquote>
</aside>
<p>What is <code>x</code> here?</p>

chaslie: <p>File “d:\xxxx\Anaconda3\lib\site-packages\torchsummary\torchsummary.py”, line 72, in summary<br>
model(*x)</p>
GPU memory allocation

Ahmad_Khan: <p>How and where does pytorch decides how much memory should be allocated for GPU?</p>

albanD: <p>Hi,</p>
<p>Only Tensors content is allocated on the GPU. So when you call <code>.cuda()</code> on a Tensor, it allocates an array of size <code>t.nelement() * t.element_size()</code> bytes.</p>
torch::data::transforms::Normalize on kCUDA?

dambo: <p>Hello,<br>
Is it possible to apply a transform on a single image as we do in PyTorch, only directly on the kCUDA tensor? My example below works only if I apply the transform on a tensor that is in the CPU:</p>
<pre><code class="lang-auto">        auto input_tensor = torch::from_blob(frame.data, {1, kIMAGE_SIZE, kIMAGE_SIZE, kCHANNELS});
        input_tensor = input_tensor.permute({0, 3, 1, 2});

        std::vector&lt;double&gt; norm_mean = {0.485, 0.456, 0.406};
        std::vector&lt;double&gt; norm_std = {0.229, 0.224, 0.225};
//        auto norm_mean = torch::tensor(
//                {0.485, 0.456, 0.406},
//                torch::requires_grad(false).dtype(torch::kFloat32));
//
//        auto norm_std = torch::tensor(
//                {0.229, 0.224, 0.225},
//                torch::requires_grad(false).dtype(torch::kFloat32));
        input_tensor = torch::data::transforms::Normalize&lt;&gt;(norm_mean, norm_std)(input_tensor);
        input_tensor = input_tensor.to(at::kCUDA);
        torch::Tensor out_tensor = module.forward({input_tensor}).toTensor();
</code></pre>
<p>Thanks,</p>
Decomposing Conv Layer

remya: <p>I am trying to decompose the first conv layer of pretrained vgg16 using the package tensorly .and trying to fine tune as follows.</p>
<p>model = models.vgg16(pretrained=True)<br>
layer = model.features._modules[‘0’]<br>
factors=parafac(layer.data, 16, init=‘svd’)<br>
layer = tl.kruskal_to_tensor(factors)<br>
model.features._modules[‘0’] = layer<br>
model = model.to(device)</p>
<p>I am getting the following error</p>
<p>AttributeError: ‘Tensor’ object has no attribute ‘_apply’</p>
<p>Can anyone help?</p>
What augmentation methods is good for dcGAN

bemorept: <p>Hi, All.</p>
<p>How can I make augmentation for dcGAN?</p>
<p>What augmentation methods is good for dcGAN (Image generation).</p>
<p>I just know a image flip method.</p>
<p>I wonder.</p>
<p>Best,<br>
<span class="mention">@bemoregt</span>.</p>

John1231983: <p>Flip, color,…depends on your problem.</p>
How do I use LSTMCell in gpu with model.to()?

urw7rs: <p>I want to use LSTMCell so I can move them to gpu when I call model.to(). But the hidden state and cell state of LSTMCell had to be moved to the gpu for LSTMCell’s weights to be in the gpu. So I googled how I’m supposed to do it and found that I needed to use buffers with register_buffer(). But I don’t know how I’m supposed to save the output of LSTMCell to the buffers I initialized.</p>
<p>Here’s what I tried.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, ...):
        ...
        self.lstm = nn.LSTMCell(256, 256)
        self.register_buffer('hidden', torch.zeros(batch, 256))
        self.register_buffer('cell', torch.zeros(batch, 256))
        ...

    def clear_lstm(self):
        # update buffers to torch.zeros(batch, 256)
        self.register_buffer('hidden', torch.zeros(batch, 256))
        self.register_buffer('cell', torch.zeros(batch, 256))
        pass

    def forward(self, input):
        ...
        self.hidden, self.cell = self.lstm(input, (self.hidden, self.cell))
        ...
</code></pre>
<p>I tried it without calling model.clear_lstm() before running and it worked, but after 3 runs, I ran out of memory(when I called .cuda() on each state, this didn’t happened). Running the model after calling model.clear_lstm() the buffers and LSTMCell wasn’t in gpu so I got an error:</p>
<pre><code class="lang-auto">RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'mat2'
</code></pre>
How to initialize the neural network as what I want?

zikun_liu: <p>Maybe a little stupid… Suppose that I have a neural network and the last layer is sigmoid. Pytorch will give the 0.5 as the initial value. So how can I initialize the output to 0.1?  Is it correct to just subtract 0.4 at the forward process? Will that have a negative effect on the following training? Thanks!</p>

ptrblck: <p>The initial output depends on the input and parameter stats.<br>
Usually you won’t get a perfect 0.5, but the mean value of your output might be approx. 0.5.<br>
You could try to change the initialization of (some) layers and try to push the initial output downwards.</p>

zikun_liu: <p>Thanks for your reply! But that is still hard to get the exact value(0.1) I want right? And if I just want the output to have the mean of 0.1,  is it correct to just subtract 0.4 in the forward process? Because that’s much easier.</p>

ptrblck: <p>Subtracting a constant from your input won’t help much, as e.g. a relu might kill it after the first layer.<br>
Maybe you could change the bias of the last linear layer to create a specific offset, but I doubt you will be able to create exactly 0.1 without zeroing the weights.</p>
<p>May I ask what your use case is, that you need a specific initial value?</p>

zikun_liu: <p>I’m doing the reinforcement learning. The output of the agent is the probability. And the reward of the environment can only have a obvious change when the output is under 0.2.</p>
LSTM for small dataset

li0218: <p>Hi,<br>
I test LSTM with a 2-category dataset with each dataset 800 data, I trained 100 epochs and try to make the model overfit and I expect to see a 100% accuracy for the training data prediction. However I failed and LSTM learn nothing, the taining accuracy is always 50% and the loss isn’t decreasing at all.<br>
(When I try with each dataset 80k data with the same model, LSTM learn something and the prediction is around 70%.)<br>
Anyone can tell me why? Thank you!</p>
Best neural network for feature extraction from images?

zimmer550: <p>Currently, I am using resnet50 to extract features from an image (image is resized to 224 by 224). I want to know whether this is the best neural network for extracting features or are there others that are even better. At the moment, I am extracting a 20 feature vector from one image and a 6 feature vector from another image (resnet50 used for both of them)</p>
How do I pass a keyword argument to the forward used by a pre-forward hook?

alvations: <p>Given a torch’s <code>nn.Module</code> with a pre-forward hook, e.g.</p>
<pre><code>import torch
import torch.nn as nn

class NeoEmbeddings(nn.Embedding):
    def __init__(self, num_embeddings:int, embedding_dim:int, padding_idx=-1):
        super().__init__(num_embeddings, embedding_dim, padding_idx)
        self.register_forward_pre_hook(self.neo_genesis)
    
    @staticmethod
    def neo_genesis(self, input, higgs_bosson=0):
        if higgs_bosson:
            input = input + higgs_bosson
        return input
</code></pre>
<p>It’s possible to let an input tensor go through some manipulation before going to the actual <code>forward()</code> function, e.g.</p>
<pre><code>&gt;&gt;&gt; x = NeoEmbeddings(10, 5, 1)
&gt;&gt;&gt; x.forward(torch.tensor([0,2,5,8]))
tensor([[-1.6449,  0.5832, -0.0165, -1.3329,  0.6878],
        [-0.3262,  0.5844,  0.6917,  0.1268,  2.1363],
        [ 1.0772,  0.1748, -0.7131,  0.7405,  1.5733],
        [ 0.7651,  0.4619,  0.4388, -0.2752, -0.3018]],
       grad_fn=&lt;EmbeddingBackward&gt;)

&gt;&gt;&gt; print(x._forward_pre_hooks)
OrderedDict([(25, &lt;function NeoEmbeddings.neo_genesis at 0x1208d10d0&gt;)])
</code></pre>
<p><strong>How could we pass the arguments (<code>*args</code> or <code>**kwargs</code>) that the pre-forward hook needs but not accepted by the default <code>forward()</code> function?</strong></p>
<p>Without modification/overriding the <code>forward()</code> function, this is not possible:</p>
<pre><code>&gt;&gt;&gt; x = NeoEmbeddings(10, 5, 1)
&gt;&gt;&gt; x.forward(torch.tensor([0,2,5,8]), higgs_bosson=2)

----------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-102-8705a40a3cc2&gt; in &lt;module&gt;
      1 x = NeoEmbeddings(10, 5, 1)
----&gt; 2 x.forward(torch.tensor([0,2,5,8]), higgs_bosson=2)

TypeError: forward() got an unexpected keyword argument 'higgs_bosson'</code></pre>

alvations: <p>Also on <a href="https://stackoverflow.com/questions/57703808/how-do-i-pass-a-keyword-argument-to-the-forward-used-by-a-pre-forward-hook" rel="nofollow noopener">https://stackoverflow.com/questions/57703808/how-do-i-pass-a-keyword-argument-to-the-forward-used-by-a-pre-forward-hook</a></p>

albanD: <p>Hi,</p>
<p>Why is the forward pre-hook necessary here? Why not include it at the beginning of the forward?<br>
Or if you want to inherit the forward from the parent class, create a new forward, do the preprocessing and then call the parent forward with <code>super().forward(args)</code>.</p>
`torch.svd()` runs on CPU while supposed to run on GPU

rossimat: <p>Hello,</p>
<p>I am running PyTorch 1.2.0 with Conda on a remote Linux machine, but I built PyTorch from source using the CUDA toolkit 9.1. I checked whether the GPU is available with <code>torch.cuda.is_available()</code> and it is.</p>
<p>Using the Conda pre-built PyTorch was not a option for me, as the GPU drivers on the remote machine are too old for the CUDA toolkit 9.2 or 10, for which PyTorch binaries are provided.</p>
<p>I have a <code>BxNxM</code> tensor <code>t</code> on GPU, with <code>B</code> around <code>100000</code>. I call <code>torch.svd(t)</code> and the GPU is idle, while the CPU starts running at 100% and more. At the end of the computation, the resulting decomposition is on GPU. In addition, trying to run on GPU is far slower than on CPU, basically unusable.</p>
<p>The weird thing is that, before to call <code>torch.svd(t)</code>, I do computations on the GPU without any issue. It seems that the SVD computation is moved to the CPU and then the results moved back to the GPU.</p>
<p>My final goal is to compute the nuclear norm of each <code>NxM</code> matrix in my <code>BxNxM</code> tensor, therefore I tried to use also <code>torch.norm(t, p='nuc', dim=(1, 2))</code>, but the same scenario described above takes place.</p>
<p>Any guess? Thanks.</p>

albanD: <p>Hi,</p>
<p>Nothing within pytorch will move the data between CPU/GPU automatically. So the computations are definitely not done on the CPU.<br>
I am not sure though that the batched version of svd exists. It may be doing a for-loop over the batch under the hood and call svd on each submatrix. If your NxM matrices are small, then you would see a very high cpu usage due to the launching of the kernels for each matrix while the GPU will stay almost idle as it has almost nothing to do.<br>
Unfortunately, this is expected behavior for the GPU: unless you have large parallelizable operations, it will be less efficient than the CPU <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Order of dimensions for NN-layer weights as per .parameters()

PersonalFinance004: <p>I’m getting started with PyTorch and following the <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html" rel="nofollow noopener">60 minute Blitz tutorial on neural networks</a>.<br>
When defining a new Conv2d layer, one passes (input_channels, output_channels, kernel_size). But when accessing the layer’s parameters by calling .parameters(), the result is [output_channels, input_channels, kernel_dimensions]. What is the rationale behind putting the number of output channels first in the latter case? I found this to be very confusing.</p>

spanev: <p>This is the shape of the Tensor that represents the weights. You can think of the Conv2d operation as matrix multiplication (simplified):</p>
<pre><code class="lang-auto">(m x n) . (n x k) =&gt; (m x k)
</code></pre>
<p>replace <code>m</code> by <code>output_channels</code>, <code>n</code> by <code>input_channels</code> and <code>k</code> by <code>n</code>, and why <code>output_channels</code> is the first dim in the params should make sense to you.</p>

PersonalFinance004: <p>Thank you for the quick response. Let’s take a look at a concrete example: I have an input tensor of shape (1,1,32,32), and a convolutional layer with weights of shape (6,1,3,3) as per .parameters(). By taking a look at the corresponding one-layer neural network I see that its output has shape (1,6,30,30). Ignoring image-dimensions and kernel-dimension, according to your heuristic one should get:<br>
(6 x 1) . (1 x 1) =&gt; (6 x 1), but by looking at the output we can see that this is not the case, and instead the dimensions are interchanged. This is very unintuitive and confuses me.</p>

spanev: <p>Let us ignore the batch size for a moment (which is not part of the <code>weight</code> shape). Let us take something else than <code>1</code> for in the <code>input_channel</code>, <code>2</code>. You have <code>(2,32,32)</code>, you want (6,30,30). Omitting the data dimensions, what happens is: <code>(6 x 2) . (2 x 1) =&gt; (6 x 1)</code> . Note that the final <code>1</code> dim has been <em>appended</em> in the input shape, so you can remove it in the output shape and you end up with <code>(6)</code>.</p>

PersonalFinance004: <p>But by that line of reasoning, couldn’t we just as well do the following:<br>
Again, I have (2,32,32) and want (6,30,30). I omit the data dimensions, append 1 in front and have (1 x 2) . (2 x 6) =&gt; (1 x 6). Again, I remove 1 and end up with (6). In this way we would’ve kept the order of input-dimension first, output-dimension second for the kernel/layer-weights, identical to when you actually define a layer. Why do it the other way around?</p>

PersonalFinance004: <p>Can someone respond to the point in my last post.</p>
How to calculate weights for multilable dataset

RegularGuY: <p>I am trying to solve a multi label classification problem, but my dataset is heavily imbalanced.My labels look like this</p>
<p><a href="https://i.stack.imgur.com/yV0pk.png" rel="nofollow noopener"><img src="https://i.stack.imgur.com/yV0pk.png" alt="enter image description here" width="171" height="51"></a></p>
<p>I want to pass weights to my loss function,So my question is How can i calculate weights for my dataset.Any suggestions would be very helpful.Thanks in advance.</p>
<p>Also General advice on multilabel problems would also help.</p>

Megh_Bhalerao: <p>Elaborate your doubt please</p>
Adding additional loss function to a pretrained network

Ska: <p>Assume we have a network-1 with a set of convolution layers (final conv  feature map size is B.512.16.16) followed by avgpool2D (B.512) and lastly a FC layer (output size B.128). I have trained this network with loss 1. Later i load this network and finetune it with either loss 2 or with loss 1 + loss 2.<br>
Here are the steps i used in finetuning:</p>
<ol>
<li>Load network 1 (lets name it model_ft)</li>
<li>Define optimizer = optim.Adam(model_ft.parameters(), lr=0.0001)</li>
<li>Compute network 1 output(lets say out1= (B.128) and out1c = (B.512.16.16)) for some input. Assume network 1 has two outputs (out1, out1c)</li>
<li>Use above output and do<br>
weights_for_maps = torch.mm(out1, model_ft.fc.weight)<br>
cam_map = weights_for_maps.dot(out1c) to get out_final of size (B.512.16.16) activation map.</li>
<li>Compute Loss 2 = pixel wise loss (out_final, some ground truth)</li>
<li>Loss 2.backward(), optimizer.step()</li>
</ol>
<p>Current Problem: Network is training but gradients are zero. Or Loss 2 is constant over epochs. However if i use (Loss 1+ Loss 2).backward(), gradients are non zero because of Loss 1.</p>

ptrblck: <p>Are the gradients just small or are they not calculated, if you call backward on loss2?</p>

Ska: <p>Gradients are small. I see small updates in parameters initially. After 2 epochs, parameters are not being updated and thus Loss is also constant. Thanks</p>

Ska: <p>Any suggestions plz?</p>

ptrblck: <p>If the parameters are not being updated after some iterations/epochs, you might want to increase the learning rate (or experiment with other optimizers).</p>
Loading a specific layer from checkpoint

maralm: <p>Is that possible to load one specific layer of the model from the pretrained checkpoint?</p>

ptrblck: <p>If you just want to load a single layer, this code should work fine:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(10, 10)
        self.fc2 = nn.Linear(20, 20)
        self.act = nn.ReLU()
        
    def forward(self, x):
        x = self.act(self.fc1(x))
        x = self.fc2(x)
        return x

model = MyModel()
torch.save(model.state_dict(), 'tmp.pth')

state_dict = torch.load('tmp.pth')
model = MyModel()
with torch.no_grad():
    model.fc1.weight.copy_(state_dict['fc1.weight'])
    model.fc1.bias.copy_(state_dict['fc1.bias'])
</code></pre>

maralm: <p>Thanks a lot! That’s exactly what I was looking for.</p>

maralm: <p>I have one other question. I am using this method to load the layers. I see two weird things.<br>
One is that loading one weight vs loading 8 weights don’t have much difference in terms of processing time. The second thing is that by changing the batch size, I see different computation time on loading parameters which should not be related. Do you have any idea on these issues?</p>

ptrblck: <p>Could you post some (pseudo) code to see, how you’ve measured the timings?</p>

maralm: <p>Here is the psudue code of what I do:<br>
Let’s say I am loading only three layers of the network and each has two sublayers called l1 and l2:<br>
for i in range (3):<br>
self.layer[i]…l1.weight.copy_(state_dict[“fc_{}.l1.weight”.format(i)], non_blocking=True)<br>
self.layer[i].l2.weight.copy_(state_dict[“fc_{}.l2.weight”.format(i)], non_blocking=True)</p>
<p>This should not be related to batch size, but with different batch sizes, I get different timing. I use time.time from python time package to calculate the timing. Also, I am copying from cpu to gpu (my state_dict is on cpu and my model is on gpu).</p>

ptrblck: <p>Where is the batch size used in your code?<br>
As you said, the batch size should be unrelated to the copying of parameters.<br>
Since you are using <code>non_blocking=True</code>, make sure to call <code>torch.cuda.synchronize()</code> before starting and stopping the timer to get valid results.</p>

maralm: <p>Sorry for late response. Problem was cuda synchronization which was showing the wrong timing. It is solved by adding torch.cuda.synchronizae()</p>
Plot a graph in pytorch using tensorboard

OBouldjedri: <p>I am trying to plot a graph using tensorboard in pytorch and I use this  after I create a model:</p>
<p>input_tensor = torch.Tensor(1,1,28,28)<br>
if args.cuda:<br>
input_tensor = input_tensor.cuda()<br>
res = model(Variable(input_tensor), requires_grad=True))<br>
tb.add_graph(model, lastVar=res)</p>
<p>but I get this error :</p>
<p>File “/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py”, line 85, in forward<br>
raise NotImplementedError</p>
<p>NotImplementedError</p>
<p>Thank you</p>

spanev: <p>Did you implement the <code>forward</code> method in your model class as:</p>
<pre><code class="lang-auto">def forward(self, x):
    # method body
</code></pre>
<p>?<br>
The error means that <code>model</code> does not have a member function (method) overriding the <code>nn.Module</code> one.</p>
LSTM backward error: Trying to backward through the graph a second time, but the buffers have already been freed

ahmadnakib: <p>I am trying to backprop the loss from an LSTM- MDN network and get the following error:<br>
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</p>
<p>I am trying to pass in 3200 vectors of size 128 into my network which has 1024 hidden units.</p>
<pre><code class="lang-auto">model = RNNMDN(128, 1024, 8 , 1).to(device)
hidden = model.initialize_hidden(32)
x = encoded_tensors.view(-1, 100, 128)
</code></pre>
<p>My input should now change from [3200, 128] to [32, 100, 128] (so 32 batches, sequence sizes of 100 and input dimensions of 128)</p>
<pre><code class="lang-auto">inputs = x[:, 0:25, :]
targets = x[:, 25:50, :]
hidden = detach(hidden)
(pi, mu, sigma), hidden = model(inputs, hidden)
</code></pre>
<pre><code class="lang-auto">loss = mdn_loss_function(pi, sigma, mu, targets)
print(loss)
</code></pre>
<p>This gives: tensor(1.4209, device=‘cuda:0’, grad_fn=)<br>
However, when i do loss.backward() it gives the runtime error, and even when i place retain_graph=True inside the backward() function, it still gives the same error.</p>
<p>My loss function is as follows:</p>
<pre><code class="lang-auto">def mdn_loss_function(y, out_pi, out_mu,out_sigma):
    result = torch.distributions.Normal(loc=out_mu, scale=out_sigma)
    y = y.view(-1, sequence, 1, 128)
    result = torch.exp(result.log_prob(y))
    result = torch.sum(result * out_pi, dim=2)
    result = -torch.log(result)
    return torch.mean(result)
</code></pre>
Creating an LSTM network to take in muiltiple inputs

ahmadnakib: <p>I am trying to build an LSTM network that will predict the next frame in a sequence based on the current frame and the action taken (there is an action for every frame). I currently encode all frames into a latent vector of size 128, and the actions are represented by an array of size 10. How would i format the input into the LSTM network?</p>
<p>For example, if I have a video consisting of 3191 frames, i will have a tensor of shpe (3191 x 128) for all encoded frames, would appending the action array associated with each frame to the latent vector work? Or is there a a way of inputting encoded frames and actions separately into the LSTM?</p>
State of charge estimation by using ANN

dreamer: <p>Hi everyone, i have several questions about ANN on pytorch.<br>
I have been using pytorch for image and video processing for a long time. Now, I want to use it on different area. I have three different types of dataset about temperature, voltage and current. I need to train them to get very close result to real result. Here is my question, can I train these 3 types of numpy array? and how?</p>

dreamer: <p>Can anyone help me please?</p>
TypeError although same shape: if not (target.size() == input.size()): 'int' object is not callable

Ran_Ran: <p>This is the error message I get. In the first line, I output the shapes of <code>predicted</code> and  <code>target</code>. From my understanding, the error arises from those shapes not being the same but here they clearly are.</p>
<pre><code class="lang-auto">torch.Size([6890, 3]) torch.Size([6890, 3])

Traceback (most recent call last):
  File "train.py", line 251, in &lt;module&gt;
    main()
  File "train.py", line 230, in main
    train(net, training_dataset, targets, device, criterion, optimizer, epoch, args.epochs)
  File "train.py", line 101, in train
    loss = criterion(predicted, target.detach().cpu().numpy())
  File "/home/hb119056/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/hb119056/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 443, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "/home/hb119056/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 2244, in mse_loss
    if not (target.size() == input.size()):
TypeError: 'int' object is not callable

</code></pre>
<p>I hope all the relevant context information is given. Thanks for any suggestions!</p>

ptrblck: <p>You are passing the <code>target</code> as a numpy array, which works differently for <code>size()</code> (it returns the number of elements as an int, thus raising this error).<br>
Remove the <code>numpy()</code> call and the code should work fine.</p>
The sequence to the encoder (decoder)

slavavs: <p>There are two required parameters for the nn.transformer module<br>
(src - the sequence to the encoder (required) and tgt - the sequence to the decoder (required))<br>
Is src (tgt) the length of a sentence or is it the length of a dictionary?<br>
I already tried different options, but transformer absolutely does not want to learn.</p>

slavavs: <p>I have vocab, sentence, word.<br>
src - the sequence to the encoder - what does this apply to?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/7422ebbc5256f1e73e98ff4ab1c8ba25e7b469fe.png" data-download-href="https://discuss.pytorch.org/uploads/default/7422ebbc5256f1e73e98ff4ab1c8ba25e7b469fe" title="src"><img src="https://discuss.pytorch.org/uploads/default/original/2X/7/7422ebbc5256f1e73e98ff4ab1c8ba25e7b469fe.png" alt="src" data-base62-sha1="gzocEGimtWSWpACMHlMT9PsktJA" width="690" height="464" data-dominant-color="D9E4E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">src</span><span class="informations">1234×830 4.33 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

slavavs: <pre><code class="lang-auto">src – the sequence to the encoder (required).
</code></pre>
<p>Does anyone know what src means?</p>
Manually start Autograd of model using gradient

mgpoirot: <p>Hi all,</p>
<p>I’m trying to back prop a model using the gradients acquired in the front and backprop of a copy of the exact same model. I acquire the gradient using a hook:</p>
<pre><code class="lang-auto"># define the models
class ANet(nn.Module): ...

class BNet(nn.Module):
    def __init__(self):
        ... layers ...
        self.grad = None

    def forward(self, x):
        x.register_hook(lambda g: self.grad = g)
        ... layers ...
        return x
</code></pre>
<p>Running this model yields the proper gradient:</p>
<pre><code class="lang-auto"># instantiate models
a1net = ANet()
a2net = deepcopy(a1net)
bnet = BNet()

# forward and back prop a1 and b
X, y = next(iter(train_loader))
yhat = bnet(a1net(X))
loss = F.nll_loss(yhat, y)
loss.backward()
</code></pre>
<p>Afterwards, I’d like to update <code>a2net</code> using the extracted gradient. I usually just call <code>.backward()</code> to initialize back propagation using Autograd, but now I’ve got to apply the gradient acquired in the previous process.</p>
<pre><code class="lang-auto"># backprop a2, this is where I'm stuck
grad = bnet.grad

a2net.backward(grad)...?
</code></pre>
<p>I’ve thought of using <code>torch.autograd.backward(tensors=a1net, grad_tensors=grad)</code>, but that didn’t work.</p>
<p>Comparing my extracted gradient with the result of <code>a1net(X)</code> shows that my grad is lacking the reference to the model in <code>grad._grad_fn</code>, also I wouldn’t know how to create a grad explicitly for a non scalar.</p>
<p>Most related questions I see around here are about retaining graphs, but always about a single model.</p>
<p>Best,</p>

ptrblck: <p>You could most likely copy the gradients to <code>a2net</code> and apply the optimizer afterwards to update the model.<br>
Here is a small example of this workflow:</p>
<pre><code class="lang-python">model1 = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 1)
)
model2 = copy.deepcopy(model1)

# Compare parameters
for param1, param2 in zip(model1.parameters(), model2.parameters()):
    if (param1!=param2).any():
        raise RuntimeError('Param mismatch')

# create optimizer for model2
optimizer = torch.optim.SGD(model2.parameters(), lr=1e-3)

# calculate gradients in model1
model1(torch.randn(1, 10)).backward()
for name, param in model1.named_parameters():
    print(name, param.grad.norm())

# Copy gradients to model2
for param1, param2 in zip(model1.parameters(), model2.parameters()):
    param2.grad = param1.grad

# Update model2
optimizer.step()

# Compare again
# Compare parameters
for param1, param2 in zip(model1.parameters(), model2.parameters()):
    print((param1 - param2).abs().sum())
</code></pre>
<p>Let me know, if that would work.</p>
Size mismatch,m1: [2 x 512], m2: [25088 x 4096]

ahhhhah: <p>Hello, everyone.</p>
<p>I want to use the pretrained model in torchvision  to finish the cats VS dog tasks. But there is something wrong with my code  : (</p>
<p><strong>RuntimeError: size mismatch, m1: [2 x 512], m2: [25088 x 4096] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:249</strong></p>
<p>The problem seem comes from the net structure. But I just load the model in the torchvision.</p>
<p>Can anyone give me some suggestions?</p>
<p>Thanks !</p>
<p>And, python == 3.6 torch == 0.4.0, torchvision == 0.2.1.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision.models as models
import torchvision.transforms as transforms

net = models.vgg16(pretrained=True)
net = net.to(device)
test = torch.randn(2,3,32,32).to(device)
y = net(test)
</code></pre>
<p>Best,</p>

Pranavan_Theivendira: <aside class="quote no-group" data-username="ahhhhah" data-post="1" data-topic="54758">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ahhhhah/48/14337_2.png" class="avatar"> ahhhhah:</div>
<blockquote>
<p>I want to use the pretrained model in torchvision to finish the cats VS dog tasks. But there is something wrong with my code : (</p>
</blockquote>
</aside>
<p>Hi Luca Xin,</p>
<p>Tried in GPU as well as CPU. In either ways, this code works perfect from my side.  My concern is that you are using a very old version of PyTorch and Torch Vision. In my case, I am using torch - 1.1.0 and torchvision- 0.3.0. If you can upgrade these two libraries to latest version, your problem may get resolved.</p>
<p>Thanks</p>

ahhhhah: <p>ok, I’ll have a try.<br>
Thanks a lot !</p>
nn.DataParallel: mismatch between batch size of input and target

swathikirans: <p>I used nn.DataParallel for increasing the batch size of a large model. While running, I am getting the error stating that: “Mismatch between the batch size of the input and target”<br>
I used <strong>nn.DataParallel(model, device_list).cuda()</strong></p>
<p>The batch size shown in the error message is 3 times that of the batch size(I am using 3 GPUs). I checked some of the examples and my understanding is that nothing else is needed to be done in order to replicate the model into multiple GPUs and share the batches among that. I would like to know if something else is needed to be done or my understanding about the whole approach is plain wrong!</p>
<p>Thanks in advance</p>

wenyuan: <p>I got the same problem. Have you solve it?</p>

swathikirans: <p>Nope. Didn’t find anything.</p>

cfanyyx: <p>I also have the same problem…</p>

swathikirans: <p>Make sure that the first dimension is always batch while inputting to your model.</p>
How to Add Unknown number heads on backbone networks

Yuanze_li: <p>Hi Everyone,<br>
I have a question about build net in nn.Module.<br>
I use self.heads to store some nets as heads of my net, and use for loop in forward, cause I don’t want to write another module when I need change the number of heads.<br>
The problem is model.parameters() do not contain heads’ parameters, just backbone network. How can I do to make model.parameters() recognized the heads as submodules?<br>
self.heads is a list whose elements is nn.Module.<br>
I guess if I manually extracts parameters as param_group, I also can use optimizer correctly and the dynamic graph design should back prop as I wish.<br>
But it is urgely, do you guys have some suggest? Or I am wrong, the technic doesn’t work.<br>
Please let me know.</p>

InnovArul: <p>You can use <code>nn.ModuleList()</code> or <code>nn.ModuleDict()</code> to store the <code>nn.Module</code>s.</p>

Yuanze_li: <p>oh, thank you! My problem solved.</p>
Regarding builtin_function_or_method code

vainaijr: <p>I want to see the code for nn.Dropout, while staying in google colab, I ctrl+click on nn.Dropout, it takes me to dropout.py, which has Dropout class, and returns</p>
<pre><code class="lang-auto">F.dropout(input, self.p, self.training, self.inplace)
</code></pre>
<p>so I ctrl+click on F.dropout, it takes me to functional.py<br>
where something like this is returned</p>
<pre><code class="lang-auto">return (_VF.dropout_(input, p, training)
            if inplace
            else _VF.dropout(input, p, training))
</code></pre>
<p>now I am stuck, because this is builtin, and no option to go beyond this.<br>
only way is to search for this in github.<br>
this happens multiple times for a lot of builtin_function_or_method, for example searching for view, permute, all are builtin_function_or_method, so cannot access their implementation from colab notebook.</p>

ptrblck: <p>I don’t think there is an easy method using a Python notebook / REPL. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
However, why would you like to see the underlying (optimized) C++ code?</p>

vainaijr: <p>to see how dropout is implemented.<br>
this happens not only for dropout, but for other builtin also, for example torch.group_norm, I want to see how this is implemented, but it is builtin.</p>
Return indexes of Top K maximum values in a matrix

Ian: <p>Given a matrix tensor with size [A,B], how to efficiently find the indexes of Top K maximum in this matrix?</p>
<p>e.g. matrix = torch.tensor[[4,2,7,1],[9,22,5,13],[6,4,8,25],[3,9,6,10],[4,1,6,14]].<br>
When K = 4, the Top K maximum numbers are 22, 13, 25, 14, then the answer should return their indexes [1,1], [1,3], [2,3], [4,3].</p>

Pranavan_Theivendira: <p>Hi Ian,</p>
<p>You can try the following solution.</p>
<pre><code class="lang-auto">import torch

x = torch.tensor([[4, 2, 7, 1], [9, 22, 5, 13], [6, 4, 8, 25], [3, 9, 6, 10], [4, 1, 6, 14]])
H, W = x.shape

x = x.view(-1)
K = 4
_, indices = x.topk(4)
two_d_indices = torch.cat(((indices // W).unsqueeze(1), (indices % W).unsqueeze(1)), dim=1)
print(two_d_indices)
</code></pre>
<p>Thanks</p>

Ian: <p>Hi, Pranavan, thanks for your kind help.</p>
Transpose axes when making CustomDataset

davidlee: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/14d69e06799ddb53e08a0c482f47d0abc5cf4c59.png" data-download-href="https://discuss.pytorch.org/uploads/default/14d69e06799ddb53e08a0c482f47d0abc5cf4c59" title="6"><img src="https://discuss.pytorch.org/uploads/default/original/2X/1/14d69e06799ddb53e08a0c482f47d0abc5cf4c59.png" alt="6" data-base62-sha1="2YlmCnPuSChtGoXcicIgd5Oyaet" width="690" height="361" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">6</span><span class="informations">1218×638 11.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hello PyTorch!</p>
<p>Im making custom dataset class and im wondering how i can make the dimension of the batch from dataloader in [10, 100, 13] not [100,10,13], with transpose((1,0,2)), where should i put transpose in my custom dataset??</p>
<p>Please help me!!</p>

mmisiur: <p>Have you read <a href="https://discuss.pytorch.org/t/swap-axes-in-pytorch/970">this thread</a>?</p>

davidlee: <p>Hello,</p>
<p>I knew how to transpose the axes with transpose((1,0,2) but ,</p>
<p>Im wondering where should i put this to Custom Dataset class.</p>
<p>Thanks,</p>

mmisiur: <p>I don’t think that putting it in <code>Dataset</code> will change anything since batch construction is the task of <code>DataLoader</code>, so your <code>i1.shape</code> (specifically <code>100</code>), comes directly form it . Moreover PyTorch normal format is NCHW, (the batch dimension is in the 0th position, then colour, height and width), so DataLoader will make it that way. I think the easiest way is to transpose it after getting batch from <code>DataLoader</code>. I don’t know if that satisfy you, or you have to explicitly put this in <code>DataLoader</code>. If you have to put it in <code>DataLoader</code> I think you can do it via <code>collate_fn</code> (<a href="https://pytorch.org/docs/stable/data.html#dataloader-collate-fn" rel="nofollow noopener">docs</a>), but I don’t have experience with that.</p>
Libtorch module forward crash

NickKao: <p>I have successfully loaded the module(CRAFT,vgg-backbone),<br>
But when I run forward, the program will be crashed, and there is no exception message to understand. How can I debug?<br>
(I have put “cout” on the front and back to confirm that the exception point is in the forward)</p>

NickKao: <p>solved.<br>
That is forward output’s type not Tensor.</p>