akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Character-level sequence classification - weird output

demonictoaster: <p>Hi all,</p>
<p>I use a character-level LSTM model to do binary classification using sequences as input.</p>
<p>The performance is quite good (both f1 score and AUC are around 99%) but I am puzzled by the output probabilities of the model. These probabilities (both in validation and test set) are either extremely close to zero or extremely close to 1 (even equal to one in many cases), meaning the model is extremely confident about its class predictions. To give you a taste of the output probability distribution (similar on both validation and test set):</p>
<p>25% percentile = 0.000054<br>
75% percentile = 0.999986</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d90420b2fb88ba5df0c12a170c3bf9a979423b18.png" alt="04-09-2019%2017-53-33" data-base62-sha1="uXOsbUemkJaPmOo5FebiTwP17xm" width="312" height="230"></p>
<p>This seems almost too good to be true - the model seems way too confident in almost all cases. I would have expected a much ‘smoother’ distribution of predicted probabilities. It also makes it tricky to set a probability threshold to control for the amount of false positives (since many examples get a predicted probability of 1).</p>
<p>The model is a straightforward implementation of LSTM for classification:</p>
<pre><code class="lang-auto">    def forward(self, X):
        embeds = self.char_embeddings(X)
        lstm_out, self.hidden = self.lstm1(embeds, self.hidden)
        lstm_out = self.dropout1(lstm_out[:, -1, :]) 
        output = self.dense1(lstm_out)
        output = self.sig(output)
        return output
</code></pre>
<p>Has any one of you experienced something similar before when it comes to binary classification?</p>
What do you think about Pytorch?

Miguel_Campos: <p>Hi there!</p>
<p>My name is Michael, I’m from spain so forgive me if i write in wrong way.<br>
One year ago i startted to learn about machine learning science and it’s an amazing and interesting world for me.</p>
<p>I began with keras, and now i would like go deeper.</p>
<p>I have pytorch in mind but is really hard to find complete courses (even more certifications) in pytorch. I know about Fastai but i prefer start with pytorch beofre. Also i have been doing the official documentation but what i am looking for is something like a scheduled guide. For instance this can be easy found in coursera if i search by tensorflow.</p>
<p>In other hand, i can find great project of OCR and other advanced techniques that make me ask, how they got that skills? and how can i to achieve that level?it’s hard find good and complete pytorch training online by facebook or others.</p>
<p>Why this is it? Do you think tensorf flow is better than pytorch? do you think pytorch will be leaved by facebook?</p>
<p>I hope someone can lead me, greetings<br>
PDT: i am really interesting in NLP<br>
PDT2: i have a strong academic background in software by the University.</p>

ptrblck: <p>Hi Michael,</p>
<p>Facebook created a Udacity course <a href="https://www.udacity.com/course/deep-learning-pytorch--ud188" rel="nofollow noopener">here</a>, which might be a good starter. I haven’t had a chance to look into it, but maybe <a class="mention" href="/u/smth">@smth</a> could give you some more information as he’s an instructor in the course (and one of the main authors of PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>
<p>My personal point of view is to chose an interesting project and just start building it using PyTorch.<br>
If you get stuck at some point, you are more than welcome to post your issues here in this discussion board and you’ll find a lot of experts helping you out.</p>

Miguel_Campos: <p>Thank you, i have been doing the free course and it’s really well explained.<br>
is Python actually been upgraded? Torch isnt and pytorch depend of torch, this worry me., I dont would like learn something whitch can dead in the future <a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>PyTorch did share some common code base with Torch7 in the past, but is now independent and under active development.<br>
If you have a look at the <a href="https://github.com/pytorch/pytorch" rel="nofollow noopener">github repo</a>, you’ll see that we have a lot of commits daily into the master branch. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Translation of new sentence with new words

Shalom_Broyer: <p>I tried the demo of “eng-fra” translation.<br>
I want to implement this algorithm on another area.<br>
My question is what should I do when I try to translate new English sentence when it has new word which doesn’t exist in the original English sentences?</p>

zhangguanheng66: <p>This is a hard problem for NLP when a new word is not in your vocabulary. Take a look at subword method (like SentencePiece). It may be helpful. Some pretrained embedding may also help.</p>

zhangguanheng66: <p>here is an example.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/bentrevett/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/bentrevett/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb" target="_blank" rel="nofollow noopener">bentrevett/pytorch-seq2seq/blob/master/3 - Neural Machine Translation by Jointly Learning to Align and Translate.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 - Neural Machine Translation by Jointly Learning to Align and Translate\n",
    "\n",
    "In this third notebook on sequence-to-sequence models using PyTorch and TorchText, we'll be implementing the model from [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473). This model achives our best perplexity yet, ~27 compared to ~34 for the previous model.\n",
    "\n",
    "## Introduction\n",
    "\n",
    "As a reminder, here is the general encoder-decoder model:\n",
    "\n",
    "![](assets/seq2seq1.png)\n",
    "\n",
    "In the previous model, our architecture was set-up in a way to reduce \"information compression\" by explicitly passing the context vector, $z$, to the decoder at every time-step and by passing both the context vector and input word, $y_t$, along with the hidden state, $s_t$, to the linear layer, $f$, to make a prediction.\n",
    "\n",
    "![](assets/seq2seq7.png)\n",
    "\n",
</code></pre>

  This file has been truncated. <a href="https://github.com/bentrevett/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Why is grad None

Hovnatan_Karapetyan: <p>In the following program, <code>b.grad</code> is <code>None</code> but <code>a.grad</code> is not <code>None</code>. Why is that?</p>
<pre><code class="lang-auto">    a = torch.tensor([[[[1.0]]]]).requires_grad_()     
    b = torch.nn.functional.interpolate(a, size=(2, 2))
    loss = torch.sum(b)           
    loss.backward()               
    b.grad                        
    a.grad
</code></pre>

spanev: <p>Hi <a class="mention" href="/u/hovnatan_karapetyan">@Hovnatan_Karapetyan</a>,</p>
<p><code>b</code> grad is None since it is a non-leaf Tensor.<br>
In your example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a.is_leaf
True
&gt;&gt;&gt; b.is_leaf
False
</code></pre>
<p>The gradient will be calculated in the during the backward phase (since it is needed by <code>a</code>) but it won’t kept in <code>grad</code>.</p>

spanev: <p><a class="mention" href="/u/hovnatan_karapetyan">@Hovnatan_Karapetyan</a><br>
Also you can use:</p>
<pre><code class="lang-auto">b.retain_grad()
</code></pre>
<p>to keep the grad.<br>
Please find more info <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.is_leaf" rel="nofollow noopener">here</a></p>

ptrblck: <p>As <a class="mention" href="/u/spanev">@spanev</a> said, the grad won’t be kept by default, so you might want to call <code>b.retain_grad()</code> before calling <code>backward</code>.</p>
Loss increases when training the whole model but decreases when some layers are frozen

notacode: <p>Hi,</p>
<p>I am solving a multiclass classification problem and I am using a pretrained resnet101 for that. I encounter a problem with the behavior during training and validation. I start by training only the fully-connected layer and then gradually ‘unfreeze’ all others. The loss on validation steadily decreases.</p>
<p>But when I ‘unfreeze’ the whole network, the losses on both training and validation start to increase after each epoch.</p>
<p>What could be a possible reason for that?</p>

Oli: <p>Hi!! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>It could be because the network is so out of tune with your problem that the gradients become so big and then suffer from gradient explosion. I don’t think it’s a big problem when you use resnets in general due to batch norm.</p>
<p>One way to combat this is to have a burn in period of some sorts. You can do what you did, where you unfreeze the network layer by layer. Another way would be to start with a really low learning rate and then increase it for a while until the network has adapted -&gt; then you continue with your normal learning rate schedule.</p>

notacode: <p>Thank you for your advice, I will check the gradients.</p>
Why do cifar10 images appear blurred?

vainaijr: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/93c46dfa4c0bb038ef76f24ba62f0f5db9f5c559.png" data-download-href="https://discuss.pytorch.org/uploads/default/93c46dfa4c0bb038ef76f24ba62f0f5db9f5c559" title="Screenshot%20(426)"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/93c46dfa4c0bb038ef76f24ba62f0f5db9f5c559_2_690x203.png" alt="Screenshot%20(426)" data-base62-sha1="l5d2gOVRUr7bmgGkIJyBuh8jnhn" width="690" height="203" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/93c46dfa4c0bb038ef76f24ba62f0f5db9f5c559_2_690x203.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/93c46dfa4c0bb038ef76f24ba62f0f5db9f5c559_2_1035x304.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/93c46dfa4c0bb038ef76f24ba62f0f5db9f5c559_2_1380x406.png 2x" data-dominant-color="5C5C59"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20(426)</span><span class="informations">1556×458 193 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>how do I see them in higher resolution?</p>

Pranavan_Theivendira: <p>Hi,</p>
<p>That is the actual size of CIFAR10. The size is 32x32 and 3 channels. That’s why you do not have enough pixels to see a clear picture.</p>
<p>Thanks</p>
Freezing some weight in the layer during forward training

Mandy: <p>Hi all,</p>
<p>I’m doing a project and I’d like to freeze some weight and only update the others during training.</p>
<p>I wish I could use <code>hook()</code> and <code>register_forward_pre_hook</code> to achieve this  since I’m learning using hook now. Below is my codes:</p>
<pre data-code-wrap="python"><code class="lang-python">    def hook(self, model, inputs):
        with torch.no_grad():
            model.weight = model.weight * self.sparse_mask[self.type[model]]

    def register_hook(self,module):
        self.handle = module.register_forward_pre_hook(self.hook)
</code></pre>
<p>However, I got this error</p>
<blockquote>
<p>TypeError: cannot assign ‘torch.FloatTensor’ as parameter ‘weight’ (torch.nn.Parameter or None expected)</p>
</blockquote>
<p>I’v lost a day fixing this problem, I was wondering if there’s someone could help me. Welcome with any ideas</p>
<p>Thanks!</p>

spanev: <p>Hi Mandy,</p>
<p>I think this should work:</p>
<pre><code class="lang-auto">def hook(module, input):
    with torch.no_grad():
        module.weight.data = module.weight.data * self.sparse_mask[self.type[module]]
</code></pre>

Mandy: <p>Thank you <a class="mention" href="/u/spanev">@spanev</a>!! It works!!</p>

Mandy: <p>Hi <a class="mention" href="/u/spanev">@spanev</a></p>
<p>I really appreciate for your help, but I got another one <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<pre data-code-wrap="python"><code class="lang-python">def hook(module, input):
    with torch.no_grad():
        module.weight.data = module.weight.data * self.sparse_mask[self.type[module]]
</code></pre>
<p>I used exactly what you show me, and I got this error</p>
<blockquote>
<p>RuntimeError: expected device cuda:0 and dtype Float but got device cpu and dtype Float</p>
</blockquote>
<p>Could you please teach me how to fix this problem?</p>
<p>Any response will be appreciated!</p>
Avoiding memory errors in *simple* autoencoder (by batches?)

MattyMatt: <p>Hey guys,</p>
<p>I have a simple, linear autoencoder of the following shapes:</p>
<p>128**3 -&gt; 512 -&gt; 128*3</p>
<p>so</p>
<p>model = [nn.Linear(128<strong>3, 512, bias=False), nn.Linear(512,128</strong>3, bias= False)]<br>
my_ae = nn.Sequenial(*model).to(‘cuda’)</p>
<p>which I would like to train with L2 loss. Problem is, I don;t have enough memory to declare my model in<br>
such a way.</p>
<p>Is there any good way to approach this problem? Train it somehow in batches?</p>

ptrblck: <p>I assume you are running out of memory, if you try to pass the whole dataset into the model?<br>
If so, have a look at the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Data loading tutorial</a> to see, how a <code>Dataset</code> and <code>DataLoader</code> are working and how batches are created.</p>
<p>The model itself is quite small and its parameters will just take approx. 1.5MB of memory.</p>

spanev: <p>I think the model is:</p>
<pre><code class="lang-auto">model = nn.Sequential(nn.Linear(128**3, 512, bias=False), nn.Linear(512,128**3, bias= False))
</code></pre>
<p>So it would actually take +8GB of memory</p>
CUDNN_STATUS_NOT_SUPPORTED when using large samples

Ben_Marinberg: <p>Hi all,<br>
I’m trying to run a deep pure-3Dconvolution on my data.<br>
I use very heavy batch, i.g. batch size 100 and my model parameters is ~ 37Gb.<br>
There are 2 high-end graphic card available to me : Quadro RTX 8000 and Quardro GV100 that has 48Gb and 32Gb accordingly.<br>
DataParallel have crashed whenever I tried it, and any batch above 10 samples(about 4Gb) crashed even on one of these (and clearly there is enough space on GPU’s RAM).<br>
I solved it after a month of trying by adding ‘torch.backends.cudnn.enabled=False’ at the beginning of the code.<br>
apparently Cudnn cant support very large data properly but using CUDA  straight on works perfectly!<br>
It allowed me to use Quadro and Titan RTX mixture , whice means it worked on different GPUs with varying RAM sizes!!</p>
<p>I would recommend any user that got this error to add<br>
torch.backends.cudnn.enabled=False<br>
to his script and re-run, I think it will solve the problem most of the times!<br>
dear PyTorch support team, please look into this.</p>

ptrblck: <p>Your issue might be related to <a href="https://github.com/pytorch/pytorch/issues/22496" rel="nofollow noopener">this one</a>.<br>
We are currently working on a fix to enable bigger input/output sizes.<br>
Sorry for the delay.</p>

Ben_Marinberg: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  for responding, and it is similar, though my batches has ~2^25 elements which is less than 2^31<br>
Is CUDA really much slower that cudnn? by how much?<br>
because if this is the difference is small enough maybe this walk-around should be posted somewhere more noticeable in the meanwhile.</p>
How to update weight in mask region of a tensor?

John1231983: <p>I have a tensor size of 2x4x64x64x64 (bxcxdxhxw)</p>
<p>I only want to compute grad ( update weight in certain region of the tensor). For example, the region from 16 to 48 in both d,h,w. So, I made a mask such as</p>
<pre><code class="lang-auto">mask=torch.zeros_like(input_tensor)
mask [:,:,16:d-16, 16:h-16,16:w-16]=1.0
input_tensor=input_tensor*mask
</code></pre>
<p>However, my updated weight always zero. What is happen in my code? Thanks</p>

Pranavan_Theivendira: <aside class="quote no-group" data-username="John1231983" data-post="1" data-topic="55098">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john1231983/48/4040_2.png" class="avatar"> John1231983:</div>
<blockquote>
<p>mask [:,:,16:d-16, 16:h-16,16:w-16]=1.0</p>
</blockquote>
</aside>
<p>Hi,</p>
<p>I used your code and checked. I initialized input_tensor with random values. Only the values in the given region is non zero. Others are zero.</p>
<pre><code class="lang-auto">import torch

d = h = w = 64
input_tensor = torch.randn((2, 4, d, h, w))
print(input_tensor[0,0, 35, 35, 35])
print(input_tensor[0, 0, 0, 0, 0])
mask=torch.zeros_like(input_tensor)
mask [:,:,16:d-16, 16:h-16,16:w-16]=1.0
print(mask[0,0, 35, 35, 35])
print(mask[0, 0, 0, 0, 0])
input_tensor=input_tensor*mask

print(input_tensor[0,0, 35, 35, 35])
print(input_tensor[0, 0, 0, 0, 0])
</code></pre>
<p>If you are getting zero for all the entries, probably you may be passing a zero input_tensor as input (or at least zeros in the given region)</p>
<p>Thanks</p>
Given groups=1, weight of size 6 1 5 5, expected input[1, 6, 14, 14] to have 1 channels, but got 6 channels instead

Wangxingyan: <p>I am new to pytorch and  had a little trouble that i  can’t solve,please help me.</p>
<p>My code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        
        self.fc1 = nn.Linear(16*5*5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
    
    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv1(x)), 2)
        x = x.view(x.size()[0], -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
input = torch.randn(1, 1, 32, 32)
out = net(input)
</code></pre>
<p>and the error:</p>
<pre><code class="lang-auto">RuntimeError: Given groups=1, weight of size 6 6 5 5, expected input[1, 1, 32, 32] to have 6 channels, but got 1 channels instead
</code></pre>
<p>Thank you very much</p>

Pranavan_Theivendira: <p>Hi,</p>
<p>In your forward function you are calling conv1 layer 2 times. The second time, it should be conv2. This will fix your issue.</p>
<p>Thanks</p>

Wangxingyan: <p>Oh!!!Thank you very much!!!I am so stupid:joy:</p>
Where is the nn.functional.to_tensor()?

YingdiZhang: <p>when I read the source code of torchvision.transforms.transforms(<a href="http://pytorch.org/docs/0.3.0/_modules/torchvision/transforms/transforms.html#Resize" rel="nofollow noopener">http://pytorch.org/docs/0.3.0/_modules/torchvision/transforms/transforms.html#Resize</a>)  I found the class ToTensor use a method i.e. F.to_tensor(pic), but I can not find the method  in torch.nn.functional,which do I miss?</p>

SimonW: <p>Look at the namespace. That functional is under torchvision namespace. It’s different with the one under torch.</p>

Tf_Hou: <p><a class="mention" href="/u/simonw">@SimonW</a> is right. I miss ths namespace first time. and I search the torchvision.transform and find in torchvison.transform’s Docs<br>
<code>Transforms are common image transformations. They can be chained together using Compose. Additionally, there is the torchvision.transforms.functional module. Functional transforms give fine-grained control over the transformations.</code></p>
<p>details in<br>
<a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.functional.to_tensor" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.functional.to_tensor</a></p>
Slow training using inception_v3

ChrisKonishi: <p>I’m trying to implement GradCAM for inception_v3, but after my modifications, it’s taking a long time, much more than what I get without the changes. Am I doing something wrong?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/1/19ff2bba5ff3541e42f40108d7818fe5651a32c2.png" alt="code" data-base62-sha1="3HYDxlaeRoNDjz4Y6gB6JRhmY3o" width="522" height="487"></p>

ptrblck: <p>Your hooks might create sync points similar to <a href="https://discuss.pytorch.org/t/analyzing-hook-drastically-decreases-performance/53922">this topic</a>, which will slow down your code.<br>
Did you check the GPU utilization before and after using the hooks?</p>

ChrisKonishi: <p>I’m not sure if the problem is the hook, for during training I set “gradCAM = False”. Also, I’m using Google Colab, so I don’t think I can check GPU usage. I did the following modification and it didn’t change the performance, which is much slower than if I just use the model from torchvision:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5f63bd51ca72b99d05167a2753f0b29273036a2c.png" alt="code2" data-base62-sha1="dBR38GYQn8zLS0q3UZ1gPbARm7G" width="412" height="434"></p>
<p>I’m considering discarding this approach and creating a separate module to get my visualization, totally apart from my training.</p>

ptrblck: <p>Thanks for the information.<br>
<code>self.feat_conv</code> won’t contain the same forward pass as the original model, as all functional calls from the <code>forward</code> method will be lost.<br>
E.g. <a href="https://github.com/pytorch/vision/blob/26c9630bc7add88685400e54457203816c06e750/torchvision/models/inception.py#L107" rel="nofollow noopener">this pooling operation</a> will not be applied (neither the following), if you just wrap the child modules in an <code>nn.Sequential</code> block.</p>
<p>A better approach would be to derive a custom class from the base model and manipulate the <code>forward</code> method as you need.</p>

ChrisKonishi: <p>Thanks, it was indeed a better approach, it is working fine now.</p>
<p>You helped a lot!</p>
Saving the model while using another class for the layers of that model

hosseinshn: <p>Hello all.<br>
I have a main model class like this:</p>
<pre><code class="lang-auto">        class KDNN(nn.Module):
            def __init__(self):
                super(KDNN, self).__init__()
                self.EnE = torch.nn.Sequential(
                    MaskedLinear(IE_dim, h_dim1, torch.LongTensor(1- Mask1.values)),
                    nn.BatchNorm1d(h_dim1),
                    nn.ReLU(),
                    MaskedLinear(h_dim1, h_dim2, torch.LongTensor(1-Mask2.values)),
                    nn.BatchNorm1d(h_dim2),
                    nn.ReLU(),
                    MaskedLinear(h_dim2, Out_dim, torch.LongTensor(1-Mask3.values)))
            def forward(self, x):
                output = self.EnE(x)
                return output
</code></pre>
<p>And with some helps from posts here a MaskedLinear class like this (to design a layer with masked weights):</p>
<pre><code class="lang-auto">        class MaskedLinear(nn.Module):
            def __init__(self, in_dim, out_dim, mask):
                super(MaskedLinear, self).__init__()

                def backward_hook(grad):
                    # Clone due to not being allowed to modify in-place gradients
                    out = grad.clone()
                    out[torch.t(self.mask)] = 0
                    return out

                self.linear = nn.Linear(in_dim, out_dim)
                self.mask = mask.byte()
                self.linear.weight.data[torch.t(self.mask)] = 0 # zero out bad weights
                self.linear.weight.register_hook(backward_hook) # hook to zero out bad gradients

            def forward(self, input):
                return self.linear(input)
</code></pre>
<p>When I want to save an object of the KDNN class, I am getting an error: "Can’t pickle local object ‘MaskedLinear.<strong>init</strong>..backward_hook’ "</p>
<p>Any suggestions?</p>
<p>Thank you!</p>
Validation dataset

slavavs: <p>Why do we use data for validation if we do not update weights for these data. Just to see how the model works on real data? So you can then see after training.</p>

ptrblck: <p>Yes, the validation accuracy gives an estimate of the model performing on unseen data.<br>
Once your training is finished, you should test it once on a test set, as the estimate of the validation dataset might be bias, e.g. if you’ve used it for early stopping, model selection etc.</p>
Convert semantic segmentation prediction to opencv Mat in python

hygxy: <p>I am looking for a way to convert prediction(tensor) of semantic segmentation network to OpenCV Mat in python so it can be processed by OpenCV, e.g.:</p>
<pre><code class="lang-auto">...
rgb = Variable(rgb).cuda()
semantic = model(rgb.unsqueeze(0))
_,pred = torch.max(semantic, dim=1)
pred = pred*255
#contours, here = cv2.findContours(pred,cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) # this line won't work
torchvision.utils.save_image(pred, 'pred.png') # this line saves prediction as binary image correctly 
</code></pre>
<p>I would like also the commented line to work since the last line tells me that ‘pred’ is already binary image and should be able to be passed to OpenCV functions like findContours(), I’ve tried like ‘np.transpose(pred.cpu().numpy(), (1,2,0))’ and it doesn’t work. I see some examples in C++ like <a href="https://discuss.pytorch.org/t/convert-torch-tensor-to-cv-mat/42751">here</a>, but how can i achieve that in python?</p>
<p>Thanks in advance!</p>

ptrblck: <p>OpenCV works with numpy arrays in Python, so you can just call <code>pred.numpy()</code> to get the underlying array.<br>
Your transpose/permute call also seems to be valid, as OpenCV works with channel-last images.</p>
<p>The issue seems to be more related to OpenCV than PyTorch, but what kind of error do you get in <code>cv2.findContours</code>, if you pass the numpy array as the input?</p>

hygxy: <p>Hey, yes you are right it’s more related to OpenCV than Pytorch. I can’t remember the error message anymore because I found the solution myself:</p>
<pre><code class="lang-auto">cv2.findContours(pred, ...) --&gt; cv2.findContours(np.uint8(pred), ...)
</code></pre>
Only copy architecture

vainaijr: <p>I have 3 neural networks, A, B, C.<br>
A and B have different architecture, but I want C to have same architecture as B, but different weights, bias initialization, and its parameters to be updated differently.<br>
If I do<br>
C = B<br>
then it would mean both are same neural network with parameters getting updated in same way.<br>
how do I ensure that both have different parameters but same architecture?</p>

ptrblck: <p>You could initialize <code>B</code> using the same model:</p>
<pre><code class="lang-python">A = MyModel()
...
B = MyModel()
</code></pre>

vainaijr: <pre><code class="lang-auto">import torchvision, torch.nn as nn, torch, torchsummary

class CombineBase(nn.Module):
  def __init__(self, ModelOne, ModelTwo):
    super().__init__()
    self.modelone = ModelOne.to('cuda')
    self.modeltwo = ModelTwo.to('cuda')
    self.lin_one = nn.Linear(2000, 1000)
    self.lin_two = nn.Linear(1000, 10)
    self.softmax = nn.Softmax(dim=-1)
  def forward(self, x):
    out = torch.cat((self.modelone(x), self.modeltwo(x)), dim=-1)
    out = self.softmax(self.lin_two(self.lin_one(out)))
    return out

class CombineMiddle(nn.Module):
  def __init__(self, ModelOne, ModelTwo):
    super().__init__()
    self.modelone = ModelOne.to('cuda')
    self.modeltwo = ModelTwo.to('cuda')
    self.lin = nn.Linear(20, 10)
    self.softmax = nn.Softmax(dim=-1)
  def forward(self, x):
    out = torch.cat((self.modelone(x), self.modeltwo(x)), dim=-1)
    out = self.softmax(self.lin(out))
    return out

level_one = [torchvision.models.alexnet(pretrained=False) for i in range(16)]

def one_level(combination_type, number, level):
    return [combination_type(level[i], level[i+1]) for i in range(number) if i%2==0]

level_two = one_level(CombineBase, 16, level_one)
level_three = one_level(CombineMiddle, 8, level_two)
level_four = one_level(CombineMiddle, 4, level_three)
top_level = CombineMiddle(level_four[0], level_four[1])

torchsummary.summary(top_level, (3, 128, 128), batch_size=100)
</code></pre>
<p>would this ensure weights of each neural network get updated differently.</p>
<p>also it gives some error when using cuda, expected backend CPU got CUDA, for <span class="hashtag">#4</span> ‘mat1’</p>

ptrblck: <p>In this code snippet you will initialize 16 different <code>alexnet</code>s, which are shared in the different levels.<br>
E.g. you could check the <code>id</code> of some parameter and check, that it’s the same:</p>
<pre><code class="lang-python">print(id(level_one[0].classifier[1].weight))
&gt; 140500209576624
print(id(level_two[0].modelone.classifier[1].weight))
&gt; 140500209576624
</code></pre>
<p>The device error might come from the usage of <code>to('cuda')</code> inside the <code>__init__</code>.<br>
Usually you could call <code>to(device)</code> on the complete model so make sure all parameters are pushed to the device.<br>
If you are not using it in this way, <code>self.lin</code> might still be on the CPU.</p>

vainaijr: <p>it gives 10% accuracy on cifar10, I resized images to (128, 128).<br>
changing it to combination of two neural networks with convnet, 4 layered gives 70% accuracy.</p>
Single model on single GPU with multi processing

Jia_Xiaosong: <p>Hi everyone</p>
<p>Now I have samples with TxNxD size and each sample may have a different T and N. Since T and N may vary a lot, I do not want to pad.  As a result, I have to use a batch_size 1.</p>
<p>Do you have any advice to accelerate the training?  (I have a server with multiple 12GB GPUs and multiple CPUs.  My model only use 1GB+ GPU memory <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>I have read the convenient torch.nn.DataParallel API but it seems that I need to have the same size for all samples to form batches. Right now, I am trying to read DistributedDataParallel documentation .</p>
<p>So could you please tell me what I should do? If you could provide some detailed documentation or even some toy codes, it would be best. (I have limited knowledge about multiprocessing and distribution.)</p>
<p>Thanks a lot</p>
Tensorflow Equivalent of Elastic deformation for pytorch

Rishav_Sapahia: <p>I am trying to implement elastic deformation by sampling control points on a regularly spaces 100*100 grid with σ = 20 with bilinear variant for the image and nearest neighbour for the masks.<br>
I found  <a href="https://github.com/deepmind/multidim-image-augmentation/blob/master/doc/elastic_deformation_colab.md" rel="nofollow noopener">this</a> implementation for the TensorFlow by DeepMind.<br>
So far, I was using the below implementation as a lambda transformation-</p>
<pre><code class="lang-auto">def elastic_transform(image, alpha=1000, sigma=20, spline_order=1, mode='nearest', random_state=np.random):
    """Elastic deformation of image as described in [Simard2003]_.
    .. [Simard2003] Simard, Steinkraus and Platt, "Best Practices for
       Convolutional Neural Networks applied to Visual Document Analysis", in
       Proc. of the International Conference on Document Analysis and
       Recognition, 2003.
    """
#     assert image.ndim == 3
    image = np.array(image)
    assert image.ndim == 3
    shape = image.shape[:2]

    dx = gaussian_filter((random_state.rand(*shape) * 2 - 1),
                         sigma, mode="constant", cval=0) * alpha
    dy = gaussian_filter((random_state.rand(*shape) * 2 - 1),
                         sigma, mode="constant", cval=0) * alpha

    x, y = np.meshgrid(np.arange(shape[0]), np.arange(shape[1]), indexing='ij')
    indices = [np.reshape(x + dx, (-1, 1)), np.reshape(y + dy, (-1, 1))]
    result = np.empty_like(image)
    for i in range(image.shape[2]):
        result[:, :, i] = map_coordinates(
            image[:, :, i], indices, order=spline_order, mode=mode).reshape(shape)
    result = Image.fromarray(result)
    return result
</code></pre>
<p>But the issue with the above implementation is it uses spline interpolation.<br>
Is there any tensorflow equivalent for pytorch or is there any other way I can implement my requirement?</p>
Computing std over each image channel

DShadowWalker: <p>Hello, I am trying to compute the std for an image, but I want a different std for each channel. Suppose we have an image with the shape  <code>[channel,x,y]</code>, in numpy we can achieve this by using multi-axis, like:  <code>img.std(axis=(1,2))</code></p>
<p>Unfortunately, the torch std implementation does not support multi-axis, so I tryed  to implement my own function:</p>
<pre><code class="lang-python">def std_for_channel(img):
   if (torch.typename(img) == 'torch.autograd.variable.Variable'):
      num_channels = img.size()[0]
   else:
      num_channels = img.shape[0]
   out = torch.zeros(num_channels)
   for i in range(num_channels):
      out[i] = img[i,:,:].std(unbiased=False)
   return out
</code></pre>
<p>while this works when passing a tensor, I keep getting error when using this on a Variable (getting ‘RuntimeError: can’t assign a Variable to a scalar value of type float’). Since I want to use this std information in my custom loss, how can I implement it correctly so backpropagation will be able to train the weights of the network?</p>

SimonW: <p><code>img.contiguous().view(img.size(0), -1).std(-1)</code></p>

DShadowWalker: <p>Thank you very much.</p>

MadeUpMasters: <p>Sorry for bumping this, but for anyone who comes across this and it seems like magic…</p>
<p><code>img.contiguous()</code> makes sure the tensor is in a contiguous chunk of memory for efficient access.<br>
<code>.view(img.size(0), -1)</code> resizes from [c, x, y, z…etc] to [c, n] where n = x<em>y</em>z…<br>
<code>.std(-1)</code> takes the standard deviation when you collapse the last channel so you end up with the channel mean.</p>
<p>If the last command is confusing, it’s because pytorch dims can be confusing, here is a nice explanation<br>
<a href="https://towardsdatascience.com/understanding-dimensions-in-pytorch-6edf9972d3be" rel="nofollow noopener">Understanding dimensions in pytorch</a></p>
How to perform weighted resampling properly on a highly imbalanced multilabel dataset?

Abhijit_Gupta1: <p>I have a highly imbalanced multilabel dataset of images. Each image can be assigned one or more labels. There are 15 labels in total. I encoded the labels as binary vectors. {1,0}^15. I tried to do weighted resampling using sample weights method, but I realised it accepts weights as a one dim vector ,i.e, [w1(y1), w2(y2),…, wn(yn)]. This is suitable for multi-class classification but not for multi label classification.<br>
Could someone guide me about how to proceed ?</p>
How to compare all entries of a tensor with a scaler?

numbpy: <p>Hello, basically I’m trying to implement ReLU from scratch without using numpy. In numpy the simple implementation is np.maximum(input,x) and the torch equivalent of maximum is max. But using torch.max as torch.max(input, 0) basically computes the maximum value along the 0th axis.</p>
<p>One way to do is to create an all-zero matrix and then use torch.max but it’s inefficient. Any other things I can use?</p>

spanev: <p>Hi <a class="mention" href="/u/numbpy">@numbpy</a>,<br>
Maybe something as this would work for you:</p>
<pre><code class="lang-auto">torch.where((input &gt; x), input, torch.zeros_like(input))
</code></pre>
<p>Please find more about <code>torch.where</code> <a href="https://pytorch.org/docs/master/torch.html#torch.where" rel="nofollow noopener">here</a>.</p>

numbpy: <p>That’s might be a better solution. I used input[input &lt; 0] = 0 (don’t know how to use inline code) for ReLU</p>
<p>And input[input &lt; 0] = 0, input[input &gt; 0] = 1 for ReLU prime. Will check which one’s faster.</p>
<p>Thanks</p>

spanev: <p>You can inline code by adding single backquotes `as this` =&gt; <code>as this</code>.</p>
<p>Actually, you should better use <a href="https://pytorch.org/docs/stable/torch.html#torch.clamp" rel="nofollow noopener">clamp</a> (better than zeros_like + where):</p>
<pre><code class="lang-auto">output = input.clamp(min=0)
</code></pre>

numbpy: <p>Thanks, clamp works for ReLU but not for prime ReLU,<br>
<code>output = input.clamp(max=1)</code> doesn’t work since most entries are between 0 and 1.<br>
torch.where works fine but all these are still slow compared to <code>output=numpy.maximum(input,0)</code> by a wide margin.<br>
I tried searching the official documentation for ReLU but the actual implementation seems to be in lua which I couldn’t understand. Also, does pytorch has an implementation for ReLU prime or is it computed internally in the autograd</p>
Sequential model in PyTorch

dreamer: <p>Hi everyone, I wonder that  can I write same nn module on Pytorch<br>
My keras codes are:</p>
<pre><code class="lang-auto">  model = Sequential()

  # model.add(LSTM(12, input_shape=train_x.shape[1:], return_sequences=True))
  # model.add(Activation('tanh'))
  # model.add(LSTM(32, input_shape=train_x.shape[1:], return_sequences=False))
  model.add(LSTM(16, input_shape=(1, 3), return_sequences=False))
  model.add(Activation('relu'))
  # model.add(Dense(12, input_shape=train_x.shape[1:]))
  # model.add(Activation('relu'))
  model.add(Dense(12))
  model.add(Activation('relu'))
  model.add(Dense(1))

  model.compile(loss='mse',
                optimizer='adam',
                metrics=['mse'])

model.fit(train_x, train_y, 
          batch_size=batch_size,
          epochs=500,
          validation_data=(val_x, val_y),
          callbacks = [tensorboard_CB])
</code></pre>
<p>I have been trying for one week to write this in PyTorch but I confront with problem which is train_x=[23k,1,3] and train_y=[23k,1] so their shapes are not same so I cannot train nn on PyTorch</p>

dreamer: <p>People, I cannot find a way to make equal their sizes <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
Freeze a model consisting of two models

Giuseppe: <p>Hy guys,I have a model consisting of two models. How can I freeze the weight of first model and training only the second model?</p>
<p>The first model consisting of a load of .pth and I don’t want to train it more.<br>
Sorry for my bad English.</p>

spanev: <p>Hi <a class="mention" href="/u/giuseppe">@Giuseppe</a></p>
<p>Let’s say <code>supermodel</code> is the model containing the two sub-models Model1 and Model2 and you only want to train Model2. You should be able to achieve it by adding this before training:</p>
<pre><code class="lang-auto">for param in supermodel.Model1.parameters():
    param.requires_grad = False
</code></pre>
Display intermediate features

vainaijr: <p>How do I display output image in the middle of a convolutional neural network, it is (64, 30, 30) in size.</p>
<p>How do I display all 64, 30x30 images that are the outputs of middle convolutional layers?</p>

spanev: <p>There are two things to do: get the intermediate features, display them.</p>
<p>First, uou will need to use <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_forward_hook" rel="nofollow noopener">register_forward_hook</a> on the concerned convolution.<br>
To create it you have to write:</p>
<pre><code class="lang-auto">def hook(module, input, output):
    # output is the [N, 64, 30, 30] Tensor containing your features

model.concerned_conv.register_forward_hook(hook)
</code></pre>
<p>Then depending on your display method, you will have. You can use torchvision and Tensorboard to do so. The hook body would be something like:</p>
<pre><code class="lang-auto">def hook(module, input, output):
    grid = torchvision.utils.make_grid(output)
    writer.add_image('conv_features', grid, 0)
</code></pre>
<p>Please find more info on using Tensorboard with PyTorch <a href="https://pytorch.org/docs/stable/tensorboard.html" rel="nofollow noopener">here</a> .</p>

vainaijr: <p>when should I be removing this hook, is it ok to train neural network, and not remove this hook?</p>

spanev: <p>Yes you can save the returned <code>torch.utils.hooks.RemovableHandle</code>:</p>
<pre><code class="lang-auto">hook = model.concerned_conv.register_forward_hook(hook)
</code></pre>
<p>and to remove it, simply:</p>
<pre><code class="lang-auto">hook.remove()
</code></pre>
<p>However, you can safely keep the hook, which is a runtime only mechanism. It won’t be saved or influence the training of your model. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Localization loss in Yolo algorithm

IAmIronMan: <p>This might not be the best forum to ask this, but if anyone has worked on Yolo algorithm. I want to ask about the Loss function, specifically about the Localization Loss.</p>
<p>You have calculated Localization loss with original coordinates or scaled (0 to 1) coordinates.<br>
As, we have to multiply with lambda_coord, it will have major impact/</p>
About KLDivLoss

Cverlpeng: <p>hi,all<br>
I have two output of conv2d: out_a  , out_b  ,  shape is (N, C, H, W)  ,<br>
I want to calculate the kl loss of these two feature maps.<br>
I am not quite sure about KLDivLosss calculation process.   out_a, out_b  sums = 0 on the channel dimension， Is my code correct?<br>
thanks</p>
<pre><code class="lang-python">sfm_out_a = F.log_softmax(out_a, dim=1)
sfm_out_b = F.softmax(out_b, dim=1)

kl_loss = nn.KLDivLoss().forward(sfm_out_a, sfm_out_b)

</code></pre>
Cross validation with pytorch overfitting

Zasexbcbc: <p>I recently coded up my own implementation of a cross validation script for pytorch and my models keep overfitting. Can someone help me out here. I am desperate. Here’s the code.</p>
<p>def predict(val_x,model=mymod,loss=loss,thresh=0.5):<br>
valxtorch=torch.tensor(val_x,dtype=torch.long).cuda()<br>
#valytorch=torch.tensor(val_y,dtype=torch.float32).cuda()<br>
vald=torch.utils.data.TensorDataset(valxtorch)<br>
testit=torch.utils.data.DataLoader(vald,batch_size=1024)<br>
sig=nn.Sigmoid()<br>
a=[]<br>
tes=[]<br>
losse=[]<br>
for i,(trainx) in enumerate(testit):<br>
xl=trainx[0]<br>
pred=model(xl)<br>
#losses=loss(pred,trainy)<br>
pred=pred.cpu().detach()<br>
<span class="hashtag">#losse</span>.append(losses.item())<br>
pred=sig(pred).numpy()<br>
a.append(list(pred))<br>
<span class="hashtag">#tes</span>.append(trainy.cpu().detach())<br>
b=[]<br>
for i in a:<br>
for j in i:<br>
b.append(j)<br>
testq=[]<br>
for i in tes:<br>
for j in i:<br>
testq.append(int(j))<br>
a=[]<br>
for i in b:<br>
if i&gt;=thresh:<br>
a.append(1)<br>
else:<br>
a.append(0)<br>
#valy=list(np.asarray(val_y,dtype=int))<br>
#valloss=sum(losse)/len(losse)<br>
return a</p>
<p>def Train(model,train_x,train_y,n_epochs=4,splits=splits,scheduler=None):<br>
it=0<br>
best=0.0<br>
for k in range(len(splits)):<br>
print(“Fold {}”.format(k+1))<br>
v=k<br>
train,val=splits[v]<br>
train12=torch.tensor(train_x[train],dtype=torch.long)<br>
trainy12=torch.tensor(train_y[train],dtype=torch.float32)<br>
traintorch=train12.cuda()<br>
trainytorch=trainy12.cuda()<br>
valxt=torch.tensor(train_x[val],dtype=torch.long)<br>
valyt=torch.tensor(train_y[val],dtype=torch.float32)<br>
valxtorch=valxt.cuda()<br>
valytorch=valyt.cuda()<br>
reqd=torch.utils.data.TensorDataset(traintorch,trainytorch)<br>
reqd1=torch.utils.data.TensorDataset(valxtorch,valytorch)<br>
load=torch.utils.data.DataLoader(reqd,batch_size=512,shuffle=True)<br>
valload=torch.utils.data.DataLoader(reqd1,batch_size=1024,shuffle=True)<br>
opti=torch.optim.Adam(model.parameters())<br>
mymod.train()<br>
for j in range(n_epochs):<br>
avg=0<br>
for i,(x,y) in enumerate(load):<br>
xl=x<br>
opti.zero_grad()<br>
pred=mymod(xl)<br>
if scheduler!=None:<br>
scheduler.batch_step()<br>
losses=loss(pred,y)<br>
avg+=losses.item()<br>
losses.backward()<br>
opti.step()<br>
it+=1<br>
avg=avg/len(load)<br>
“”“qwe,valloss=predict(val_x,val_y,mymod)<br>
if qwe&gt;best:<br>
best=qwe<br>
torch.save(mymod,’/content/gdrive/My Drive/My files/Jigsaw/quoramodbest.pt’)”""</p>
<pre><code>  print("Epoch {}/{}: Loss:{}".format(j+1,n_epochs,avg))
mymod.eval()
sig=nn.Sigmoid()
avgs=[]
for i,(x,y) in enumerate(valload):
  a=[]
  xl=x
  pred=mymod(xl)
  pred=sig(pred)
  pred=pred.detach().cpu().numpy()
  a.append(list(pred))
  b=[]
  for i in a:
    for j in i:
      b.append(j)
  c=[]
  for i in b:
    if i&gt;=0.5:
      c.append(1)
    else:
      c.append(0)
  valyq=list(np.asarray(y.detach().cpu().numpy(),dtype=int))
  avgs.append(f1_score(c,valyq))
print(sum(avgs)/len(avgs))
</code></pre>
<p>return model</p>
Multi layer LSTM thrown an error

andreaschandra: <pre><code class="lang-auto">class Classifier(nn.Module):
    
    def __init__(self):
        super(Classifier, self).__init__()
        
        self.LSTM1 = nn.LSTM(input_size = 256, hidden_size = 1024)
        self.LSTM2 = nn.LSTM(input_size = 1024, hidden_size = 1024)
        self.LSTM3 = nn.LSTM(input_size = 1024, hidden_size = 1024)
        self.LSTM4 = nn.LSTM(input_size = 1024, hidden_size = 1024)
        self.LSTM5 = nn.LSTM(input_size = 1024, hidden_size = 1024)
        self.LSTM6 = nn.LSTM(input_size = 1024, hidden_size = 256)
        
    def forward(self, x):
        
        result, (hn, cn) = self.LSTM1(x)
        result, (hn, cn) = self.LSTM2(result, (hn, cn))
        result, (hn, cn) = self.LSTM3(result, (hn, cn))
        result, (hn, cn) = self.LSTM4(result, (hn, cn))
        result, (hn, cn) = self.LSTM5(result, (hn, cn))
        result, (hn, cn) = self.LSTM6(result, (hn, cn))
         
        return result
</code></pre>
<p>Architecture that I made above thrown an error</p>
<pre><code class="lang-auto">RuntimeError: Expected hidden[0] size (1, 256, 256), got (1, 256, 1024)
</code></pre>
<p>But, if I remove (hn, cn) in self.LSTM6, it run well. why does hidden cell affect the LSTM layer?</p>