akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cpp extension error when including torch/extension.h

almeetb: <p>Hi,</p>
<p>I am creating a custom c++ extension but I get an error on compiling when I add <span class="hashtag">#include</span> &lt;torch/extension.h&gt; with the error:</p>
<pre><code class="lang-auto">pytorch/torch/include/torch/csrc/api/include/torch/ordered_dict.h:13:7: error: redefinition of 'OrderedDict'
class OrderedDict {
      ^
pytorch/torch/csrc/api/include/torch/ordered_dict.h:13:7: note: previous definition is here
class OrderedDict {
</code></pre>
<p>My setup.py looks like:</p>
<pre><code class="lang-auto">setup(
    name="pytorch_extension",
    version="0.1",
    description="PyTorch Extension",
    packages=find_packages(exclude=['build']),
    install_requires=[],
    ext_modules=[
        CppExtension(
            'EXTENSION_NAME',
            sources,
            include_dirs=include_dirs,
            extra_compile_args=extra_compile_args,
            library_dirs=[],
            extra_link_args=[]
        ),
    ],
    cmdclass={
        'build_ext': BuildExtension,
        'clean': Clean,
    }
)
</code></pre>
<pre><code class="lang-auto">// Fails
#include "torch/extension.h" 
// #include "torch/csrc/jit/python/pybind.h" (works)

PYBIND11_MODULE(EXTENSION_NAME, m) {
  ...   
}
</code></pre>

ptrblck: <p>Are you using the latest libtorch version or a previous one?<br>
Were you able to reproduce this issue with the single import or would we need something else to reproduce it?</p>

almeetb: <p>I do have the latest version of libtorch and I was seeing the problem only when I included extension.h but I noticed some extra include_dirs in my setup.py that was not needed.</p>
<p>Thanks</p>

ptrblck: <aside class="quote no-group" data-username="almeetb" data-post="3" data-topic="84110">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/34f0e0/48.png" class="avatar"> almeetb:</div>
<blockquote>
<p>but I noticed some extra include_dirs in my setup.py that was not needed.</p>
</blockquote>
</aside>
<p>Did you solve it by removing these dirs?<br>
I just tried to build the tutorial C++ extension and it works with fine, if I include <code>extension.h</code>.</p>
Understanding pytorchs deeplabV3

mayool: <p>Hey everyone,</p>
<p>i am currently working with the torchvision.models.segmentation.deeplabv3_resnet50() model.</p>
<p>It consists of:</p>
<ul>
<li>a backbone (Resnet)</li>
<li>a classifier (DeeplabHead)</li>
<li>interpolation (biliniar to make sure output_size = input_size)</li>
</ul>
<p>what really confuesed me was the interpolation part.<br>
For testing I inserted an image of size 270x512.<br>
The result of the classifier however was 34x64.</p>
<p>So the model uses bilinear interpolation to upscale from 34x64 --&gt; 270x512 which seems like a massive Jump.<br>
I always though Deeplabs Decoder would upscale the image to something that is at least close to the original size.</p>
<p>Why does this model still peform so good? would it be possible to replace the interpolation by something like conv_transpose2d to improve the result?</p>
<p>or maybe I am just not understanding the decoder part <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> ?</p>

ptrblck: <p>The <a href="https://arxiv.org/abs/1606.00915">paper</a> explains the interpolation strategy as well as the usage of transposed convolutions in a couple of sections.<br>
This section might be interesting:</p>
<blockquote>
<p>We have adopted instead a hybrid approach that strikes a good efficiency/accuracy trade-off, using atrous convolution to increase by a factor of 4 the density of computed feature maps, followed by fast bilinear interpolation by an additional factor of 8 to recover feature maps at the original image resolution. Bilinear interpolation is sufficient in this setting because the class score maps (corresponding to log-probabilities) are quite smooth, as illustrated in Fig. 5.</p>
</blockquote>
Inferences made while doing image classification

Sravani_Lekkala: <p>if batch size is high we have to use high learning rate like this if there are any things that we should know while doing image classification please post them<br>
problems you have faced and how u overcome them<br>
like if it imbalanced dataset how u overcome it<br>
momentum tuning in optimizer<br>
how did u choose the learning rate<br>
u may have made inferences while observing accuracies and validation losses<br>
please post them…</p>
Increase channels for custom residual blocks

theairbend3r: <p>I’m trying to insert residual connection in my small 3 conv-layer.</p>
<p>Here’s what it looks like -</p>
<pre><code class="lang-auto">class ResClassifier(nn.Module):
    def __init__(self, num_classes):
        super(ResClassifier( self).__init__()
        
        
        self.block1 = self.conv_block(c_in=1, c_out=16, dropout=0.1, kernel_size=5, stride=1, padding=2)
        self.block2 = self.conv_block(c_in=16, c_out=32, dropout=0.1, kernel_size=3, stride=1, padding=1)
        self.block3 = self.conv_block(c_in=32, c_out=64, dropout=0.1, kernel_size=3, stride=1, padding=1)
        
        self.lastcnn = nn.Conv2d(in_channels=64, out_channels=num_classes, kernel_size=28, stride=1, padding=0)
       
        
    def forward(self, x):
             
        x = self.block1(x)
        
        x = self.block2(x)
    
        x = self.block3(x)
        
        x = self.lastcnn(x)
        
        return x
    
    
    def conv_block(self, c_in, c_out, dropout, **kwargs):
        seq_block = nn.Sequential(
            nn.Conv2d(in_channels=c_in, out_channels=c_out, **kwargs),
            nn.BatchNorm2d(num_features=c_out),
            nn.ReLU(),
            nn.Dropout2d(p=dropout)
        )
        
        return seq_block
</code></pre>
<p>Let’s say I want to add skip connection from <code>block-1</code> to <code>block-3</code>. To do that, I can simply do -</p>
<pre><code class="lang-auto">   def forward(self, x):
        
        residual_1 = x

        x = self.block1(x)        

        x = self.block2(x)

        x += residual_1

        x = self.block3(x)

        x = self.lastcnn(x)
        
        return x

</code></pre>
<p>But the issue is, output of <code>block-3</code> has 64 channels while the input (<code>residual_1</code>) has only 1.</p>
<p>Because of this mismatch, I can’t add the residual connection. How do I increase the number of channels in <code>residual_1</code> before I can add it to the output of <code>block-3</code>.</p>

ptrblck: <p>You could use a <code>downsample</code> layer as seen in the original <a href="https://github.com/pytorch/vision/blob/11a39aaab5b55a3c116c2e8d8001bad94a96f99d/torchvision/models/resnet.py#L66-L67">ResNet implementation</a>, which could just use a <code>1x1</code> conv with the desired number of output channels to match the activation.</p>
How to add dropout in pytorch dynamically?

Shantanu_Ghosh: <p>Normally we can create a dropout layer by<br>
self.drop_out = nn.Dropout(p=0.5)</p>
<p>For my case, the dropout probability will depend on the input and will be passed in the input vector x during forward propagation. Can anyone tell me how to achieve it?</p>

ptrblck: <p>You could use the functional API in the <code>forward</code> pass via:</p>
<pre><code class="lang-python">def forward(self, x, p):
    ...
    x = F.dropout(x, p, training=self.training)
    ...
</code></pre>
<p>Note that you should pass the <code>self.training</code> attribute to this layer so that this dropout call will be disabled, if you call <code>model.eval()</code>.</p>

Shantanu_Ghosh: <p>Do i need to specify  self.drop_out = nn.Dropout(p=0.5) in the <strong>init</strong>?<br>
I don’t think its required, right?</p>

Shantanu_Ghosh: <p>Also, I have another problem.<br>
I am passing a batch of size 32. So x has dimension (32, y)<br>
The probability p also has dimension of (32, 1) (p is an array of 32 values)<br>
Now for each x, I want the corresponding indexed probability. Can you please in achieving that?</p>

ptrblck: <p>You don’t need to create the module, if you are using the functional API.<br>
<code>p</code> is a scalar values and will not have one value per sample.</p>
Can I use the pretrained models for commercial use

Hassan_Saeed: <p>I want to use a pretrained model that I downloaded from torchvision.models, train it on my own dataset, and use it for commercial purpose, is that legal??</p>

cardboardcode: <p>Hi, I am interested in this topic as well.</p>
<p>Currently, is it right to assume that these pre-trained models offered by PyTorch fall under the same BSD permissive license as stated on its GitHub <strong><a href="https://github.com/pytorch/pytorch#license" rel="nofollow noopener">readme.md</a></strong>?</p>
Behaviour of nn.utils.weight_norm during inference

mohit117: <p>Hello,</p>
<p>I know the utility of weight_norm is quite simple to use in PyTorch</p>
<pre><code class="lang-auto">from torch.nn.utils import weight_norm
weight_norm(nn.Conv2d(in_channles, out_channels))
</code></pre>
<p>However, does it do the weight re-parameterization even during inference when model is set in <code>eval()</code> mode and running inside <code>torch.no_grad()</code> or it just do during training and freezes the weights during inference? I could not find it in the documentation or by checking the source code on GITHUB.</p>
<p>At the time of inference should I explicitly inactivate weight normalization using <code>remove_weight_norm()</code> or not?</p>
<p>Thankyou</p>
Why there are commonly used 2 dense layers at the end of image captioning models

cevvalu: <p>I see common models use 2 dense layer after merging image and text features. I understand the last dense layer used for predicition with softmax. But before softmax there is a dense layer with Relu. I think<br>
it is used to get all features as an output and feed as input to softmax for final prediction. Is this true?</p>

ptrblck: <p>Yes, the penultimate layer would process the incoming features and pass it to the last classification layer.<br>
This might be beneficial, but depends on your use case and model.<br>
E.g. <code>resnet50</code> uses a single linear layer without a preceding one.</p>

cevvalu: <p>thanks for your reply, yes my accuracy is increased when i added another dense layer before softmax. But dont know the reason actually…normally i feed the features directly to softmax for prediction and now features pass through a dense layer with relu and then softmax predicts</p>

ptrblck: <p>Another additional layer adds more capacity to the model, which might help.</p>
<p>Are you using <code>F.softmax</code> of an <code>nn.Softmax</code> module as the last non-linearity?<br>
If so, this might be wrong for a multi-class classification use case, since e.g. <code>nn.CrossEntropyLoss</code> expects raw logits while <code>nn.NLLLoss</code> expects log probabilities created with <code>F.log_softmax</code>.</p>

cevvalu: <p>I don’t use softmax, i take maximum logit with argmax to get the next preducted word as a numerical value</p>
Expected 4-dimensional input for 4-dimensional weight [64, 3, 3, 3], but got 3-dimensional input of size [3, 1936, 2592] instead

BlueTurtle: <p>I am trying to use CNNs but I can only get it working on MNIST. As soon as I use a DataLoader it all goes wrong. I’ve tried multiple competitions on Kaggle and everytime I get into a circle of DataLoader errors, either size issues, dataype conflicts, etc.</p>
<p>I am aware this problem has come up a lot before and I have read those posts but cannot carry over the solution to my own situation.</p>
<p>Notebook: <a href="https://www.kaggle.com/blueturtle/siim-cnn-intro" rel="nofollow noopener">https://www.kaggle.com/blueturtle/siim-cnn-intro</a></p>
<p>My DataLoader/collate function are doing something weird too. When unpacking the train_loader whilst trying to train the model, “images” is outputing the target value and “labels” is outputing a tuple of (target, tensor of image pixels, target). Which has lead to me having to do some weird indexing a few lines after. I assume they have swapped around at some point but the <strong>getitem</strong> function returns images, labels so I am unsure how this has happened.</p>
<pre><code class="lang-auto">        for images, labels in enumerate(train_loader):
            steps += 1
            optimizer.zero_grad()
            output = model.forward(labels[0][0])
</code></pre>
<p>I have also tried <code>iter(train_loader)</code> and that makes no difference.</p>
<p>Many thanks</p>

BlueTurtle: <p>This problem was solved with:</p>
<pre><code class="lang-auto">            images = torch.FloatTensor(images[0])
            images = images.unsqueeze(1)
            images = images.permute(3, 0, 1, 2)
</code></pre>
<p>I am still having problems with this notebook for other reasons but this has been solved.</p>
RuntimeError: Given groups=1, weight of size [256, 256, 3, 3], expected input[32, 1, 256, 256] to have 256 channels, but got 1 channels instead

yanivweiss: <p>Hi,</p>
<p>Could you please help me find the mismatch between the CustomDataset and the model?</p>
<p>Thanks!</p>
<p><strong>Code below:</strong></p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader

import torch.nn as nn

class CustomDataset(Dataset):

    def __init__(self, path,start_idx, end_idx):

        

        self.data = np.load(path)

        self.data = self.data[start_idx:end_idx]

    def __len__(self):

       

        return len(self.data)

    

    def add_noise(self,y):

        newimg = random_shapes((256, 256),min_shapes=30,max_shapes=42,

                       multichannel=False, min_size=20,max_size=30,allow_overlap=True)[0]/255.0

        x = y.copy()

        x[np.where(newimg &lt; 0.9)] = x[np.where(newimg &lt; 0.9)]+1-newimg[newimg &lt; 0.9]

        return x

    def __getitem__(self, idx):

        

        y = self.data[idx]

        x = torch.FloatTensor(self.add_noise(y)).unsqueeze(0)

        

        return x, torch.FloatTensor(y).unsqueeze(0)



class Denoise(nn.Module):

    def __init__(self):

        super(Denoise, self).__init__()

        # encoder layers

        self.enc1 = nn.Conv2d(256, 256, kernel_size=3, padding=1)

        self.enc2 = nn.Conv2d(256, 32, kernel_size=3, padding=1)

        self.enc3 = nn.Conv2d(32, 16, kernel_size=3, padding=1)

        self.enc4 = nn.Conv2d(16, 8, kernel_size=3, padding=1)

        self.pool = nn.MaxPool2d(2, 2)

        

        # decoder layers

        self.dec1 = nn.ConvTranspose2d(8, 8, kernel_size=3, stride=1)  

        self.dec2 = nn.ConvTranspose2d(8, 16, kernel_size=3, stride=1)

        self.dec3 = nn.ConvTranspose2d(16, 32, kernel_size=3, stride=1)

        self.dec4 = nn.ConvTranspose2d(32, 256, kernel_size=3, stride=1)

        self.out = nn.Conv2d(256, 256, kernel_size=3, padding=1)

  

    def forward(self, x):

        # encode

        x = F.relu(self.enc1(x))

        x = self.pool(x)

        x = F.relu(self.enc2(x))

        x = self.pool(x)

        x = F.relu(self.enc3(x))

        x = self.pool(x)

        x = F.relu(self.enc4(x))

        x = self.pool(x) # the latent space representation

        

        # decode

        x = F.relu(self.dec1(x))

        x = F.relu(self.dec2(x))

        x = F.relu(self.dec3(x))

        x = F.relu(self.dec4(x))

        x = F.sigmoid(self.out(x))

 

        return x

</code></pre>

BlueTurtle: <p>You need to permute your channels.</p>
<p>Currently you have: <code>(H, W, C, N)</code> but you need it in <code>(N, C, H, W)</code></p>
<p>If it’s a tensor:<br>
<code>x = x.permute(3, 2, 1, 0)</code></p>
<p>Should do it. You can put this in your training loop before you feed the variable to your model.</p>

yanivweiss: <p>Hi,<br>
Thanks for the quick reply,<br>
however got the below error:<br>
RuntimeError: Given input size: (256x1x32). Calculated output size: (256x0x16). Output size is too small</p>
Check torch tensor has only one element of one

aghtaal: <p>I appreciate it if somebody helps me to check a tensor to be a valid indicator function.<br>
Meaning that all elements of each row are zero beside one<br>
So A is valid and B and C are not valid:</p>
<pre><code class="lang-auto">A = torch.tensor([[1, 0, 0], [0, 1, 0])
B = torch.tensor([[1, 0, 0], [0, 0, 0])
C = torch.tensor([[1, 0, 0], [1, 0, 1])
</code></pre>

Nikronic: <p>Hi,</p>
<p>I think this can help:</p>
<pre><code class="lang-auto">A = torch.tensor([[1, 0, 0], [0, 1, 0]])
B = torch.tensor([[1, 0, 0], [0, 0, 0]])
C = torch.tensor([[1, 0, 0], [1, 0, 1]])

(torch.sum(C, dim=1) &lt;= 1).all()
</code></pre>
<p>Actually, <code>torch.sum(tensor, dim)</code> reduces sum over specified dimension. So, for instance, for <code>C</code>,</p>
<pre><code class="lang-auto">torch.sum(C, dim=1)  # output: tensor([1, 2])
</code></pre>
<p>Which means we have two <code>1</code>s in second row of tensor <code>C</code>. Then by adding <code>&lt;=1</code> it will convert it to a boolean</p>
<pre><code class="lang-auto">torch.sum(C, dim=1) &lt;= 1  # output: tensor([ True, False])
</code></pre>
<p>But we are looking for results that satisfy the condition in all rows which means all values should be <code>True</code>. To do this we add <code>.all()</code> at the end which tells us that all values are True or False.</p>
<p>Bests</p>
Is model is multiprocessing really running in parallel?

xuehy: <p>Hello all,</p>
<p>I am trying to implement multiprocessing environments for parallel rollouts.</p>
<p>I call <code>model.share_memory()</code> and then use torch.multiprocessing to start several processes.</p>
<p>In each process, I use<code> model(input)</code> for feed-forward computation.</p>
<p>My question is:</p>
<p>Are the feed-forward in all the processes running in parallel (like parallel GPU computation)? Or actually the model is only used by one process at one time and all the processes share the model via some locks?</p>

iffiX: <p>Well, if you are using GPU, then you are still doing computations on the same GPU, but on CPU, you are…running some kind of multiprocessing parallel.</p>
<p>Please read multiprocessing tutorials first.</p>
<p>This question belongs to <code>distributed</code> and not <code>RL</code></p>
Backward error on kl divergence

Hung_Nguyen: <p>Hi, I’m trying to optimize a distribution using kl divergence. Here’s the code:</p>
<pre><code class="lang-auto">mu1 = torch.tensor([0.3, 0.9], requires_grad=True)
mu2 = torch.tensor([0.5, 0.5])
b1 = torch.distributions.Binomial(1,mu1)
b2 = torch.distributions.Binomial(1,mu2)
opt = torch.optim.Adam(params=[mu1])
kl = torch.distributions.kl_divergence

eps = 100
for i in range(eps):
    opt.zero_grad()
    l = kl(b1, b2).mean()
    l.backward()
    opt.step()
</code></pre>
<p>When I changed  eps to 1, everything worked as normal. However if I increased eps,  I got the following error:<br>
“Trying to backward through the graph a second time, but the buffers have already been freed…”</p>

MariosOreo: <p>Hi,</p>
<p>I think you should rebuild the computation graph in each iteration, since when calling <code>.backward()</code> it will clear up the graph built before, see more details from <a href="https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time/6795/35?u=mariosoreo">this thread.</a></p>

rasbt: <p>Yes, I think</p>
<pre><code>b1 = torch.distributions.Binomial(1,mu1)
b2 = torch.distributions.Binomial(1,mu2)
</code></pre>
<p>should be in the for-loop.</p>

Raza_Habib: <p>Hi Sebastian,</p>
<p>Can you please explain why these two distributions need to be reinstantiated on each forward pass?</p>
<p>My understanding was that the class definition doesnt execute any code that the graph depends on. The kl method does the forward pass. Why does class initiatilsation matter?</p>

Raza_Habib: <p>Oh I see its because mu_1 and mu_2 are parameters and every step depends on them. Thanks!</p>

Raza_Habib: <p>Actually no, I still dont see it. It looks like each call of the for-loop is independent of the previous call. What am I missing?</p>
I have doubt in deep q learning

gokul_adethya: <p>I have a doubt DQN say for nth state i am getting</p>
<pre><code class="lang-auto">qnthvalues = [1,2,3]
</code></pre>
<p>So here max q value is selected which 2 pos or the 3 rd value and i am doing the action 3 and getting qn+1thvalue and now should i apply bellman eq for that action or the 3rd value of qn+1th value and leave other value the same for target value</p>
<pre><code class="lang-auto">qn+1value = [2,3,4]
targetq_values = [2,3,bellmaneq(4)]
               Or
targetq_values = bellmaneq(qn+1values)
#for all q values of that state
</code></pre>
<p>(So for all q values we will be applying or will be applying for the action q value alone.</p>

iffiX: <p>target_value = reward (from env) + discount * (1 - terminal) * next_value</p>
<p>you can acquire next_value by using the q net itself(vanilla DQN), a target network(fixed target), or select an action using the online network and use this action to find the q value(double DQN)</p>
DQN: Error for loss.backward() at terminal state with fixed target

spadel: <p>I am currently trying DQN for Tic Tac Toe. I am stuck with the following error:</p>
<blockquote>
<p>element 0 of tensors does not require grad and does not have a grad_fn</p>
</blockquote>
<p>I think the error results whenever the game is played to the end and reaches a terminal state, as the target is then manually set to some value and now the gradient cannot be computed during the backward propagation. How can I fix this?</p>
<p>Here is my code for the updating of the NN:</p>
<pre><code class="lang-auto">def update_NN(state, next_state, action, player, discount, lr, loss_all):
    pred = torch.tensor(net(torch.tensor(state).float().view(-1, 9)).squeeze().detach().numpy()[action])
    reward = 0
    winner, game_status = check_result(next_state)
    if game_status == 'Done' and winner == player:
        reward = 100
    if game_status == 'Done' and winner != player:
        reward = -1
    if game_status == 'Draw':
        reward = 10

    if next_state.count(0) == 0:
        target = torch.tensor(reward).float()
    else:
        target = torch.tensor(reward).float() + discount * torch.max(net(torch.tensor(next_state).float()))
    # Evaluate loss
    loss = loss_fn(pred, target)
    print(loss)
    loss_all.append(loss)
    optimizer.zero_grad()
    # Backward pass
    loss.backward()
    # Update
    optimizer.step()
</code></pre>

iffiX: <aside class="quote no-group" data-username="spadel" data-post="1" data-topic="76661">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/8797f3/48.png" class="avatar"> spadel:</div>
<blockquote>
<p><code>net(torch.tensor(state).float().view(-1, 9)).squeeze().detach()</code></p>
</blockquote>
</aside>
<p>You have detached the output of your network.<br>
Besides, DQN is usually not implemented in this way, the network would accept a state s and output q values for all actions.</p>
<p>Maybe read this <a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" rel="noopener nofollow ugc">tutorial</a></p>
Use trained layers for new objective

gil_bc: <p>Hi,<br>
I have trained a network with good accuracy for a certain purpose. Now I believe the network contains interesting internal conclusions about my data.</p>
<p>So I wish to connect some of the layers in my trained network with a new, untrained network, that would learn a new objective.<br>
So let’s say I want to use the first few layers of the trained network, concatenate them to new layers and train the whole thing together.</p>
<p>How would I perform such thing?<br>
many thanks</p>

chetan_patil: <p>Hi <a class="mention" href="/u/gil_bc">@gil_bc</a> , can you relate your problem to this <a href="https://github.com/ChetanPatil28/Pytorch_In_and_Out/blob/master/Selective_Transfer_Learning.py" rel="nofollow noopener">https://github.com/ChetanPatil28/Pytorch_In_and_Out/blob/master/Selective_Transfer_Learning.py</a>  ?</p>
Gradient of tensor is none, while the gradient of its copy is not none?

Sudarshan_Babu: <pre><code class="lang-auto">   for batch_idx, (data, target) in enumerate(train_loader):
        meta_data = data
        meta_data.requires_grad = True
        print(data.requires_grad) // prints True, since we are not doing deep copy
        data, target = data.to(device), target.to(device) // moving tensors to GPU
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()
        print(meta_data.grad) //prints the corresponding gradient
        print(data.grad) // prints None, Why is this the case ??
        print((data - meta_data.cuda()).norm(2))

</code></pre>
<p>My question is why is meta_data.grad not none, while data.grad is none ? is it because of moving the data to the gpu ? Thanks in advance!</p>

albanD: <p>Hi,</p>
<p>It is because <code>.to()</code> is a differentiable op. And so the <code>data</code> that you get back is not a leaf anymore (if you print it, you’ll see the grad_fn attached to it).</p>

Sudarshan_Babu: <p>Thank you, this resolves it !</p>
Tuning batch normalisation

Sravani_Lekkala: <p>will we tune eps and momentum in batch norm often in practice</p>

googlebot: <p>eps is there for numerical stability, so it is unusual to tune it. momentum implicitly defines number of last samples used to estimate (population) moments, so it can be reasonable to decrease momentum, if estimates are not changing smoothly (for example due to small batch size)</p>

Sravani_Lekkala: <p>so it would be better to decrease the momentum when we are using small batch size to improove the accuracy?</p>

googlebot: <aside class="quote no-group" data-username="Sravani_Lekkala" data-post="3" data-topic="84071">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sravani_lekkala/48/25065_2.png" class="avatar"> Sravani_Lekkala:</div>
<blockquote>
<p>so it would be better to decrease the momentum when we are using small batch size</p>
</blockquote>
</aside>
<p>Well, it won’t hurt, to have say batch_size/momentum constant, when decreasing batch size.</p>
<aside class="quote no-group" data-username="Sravani_Lekkala" data-post="3" data-topic="84071">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sravani_lekkala/48/25065_2.png" class="avatar"> Sravani_Lekkala:</div>
<blockquote>
<p>to improove the accuracy?</p>
</blockquote>
</aside>
<p>I’d rather say that you can sometimes fix batchnorm problems by using smaller momentum, i.e. you can only improve subpar results and only when batchnorm is the issue.</p>
nn.Parameter() not in model.named_parameters()

Talita: <p>Hello!</p>
<p>I have an LSTM as follows (I will not copy everything for simplicity):</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self,
                 bert,
                 hidden_dim,
                 output_dim,
                 n_layers,
                 bidirectional, 
                 dropout, hidden_init):
        
        super().__init__()
        
    
        self.rnn = nn.LSTM(embedding_dim,
                          hidden_dim,
                          num_layers = n_layers,
                          bidirectional = bidirectional,
                          batch_first = True,
                          dropout = dropout ...., ) 
        
        (..)
        self.initial_hidden = self.init_hidden(self.batch_size)
        
    
    def forward(self, text):
          (..) 

          # initialize the hidden and cell state randomly 
           hidden = self.init_hidden(self.batch_size) 
           _, hidden = self.rnn(embedded, hidden) 
       
        return output
    
  
    def init_hidden(self, batch_size): 
        h_0 = nn.Parameter(torch.randn(self.n_layers*2, batch_size, self.hidden_dim))
        c_0 = nn.Parameter(torch.randn(self.n_layers*2, batch_size, self.hidden_dim))
        return (h_0, c_0)
</code></pre>
<p>However if I check the model parameters:</p>
<pre><code class="lang-auto">for name, param in model.named_parameters(): 
    if param.requires_grad: 
       print(name)
</code></pre>
<p>it prints:</p>
<pre><code class="lang-auto">rnn.weight_ih_l0
rnn.weight_hh_l0
rnn.bias_ih_l0
rnn.bias_hh_l0
rnn.weight_ih_l0_reverse
rnn.weight_hh_l0_reverse
rnn.bias_ih_l0_reverse
rnn.bias_hh_l0_reverse
rnn.weight_ih_l1
rnn.weight_hh_l1
rnn.bias_ih_l1
rnn.bias_hh_l1
rnn.weight_ih_l1_reverse
rnn.weight_hh_l1_reverse
rnn.bias_ih_l1_reverse
rnn.bias_hh_l1_reverse
out.weight
out.bias
</code></pre>
<p>Why can I not see the parameters <code>h_0</code> and <code>c_0</code> ??</p>

Jonathan_Harel: <p>I’m not an expert, but it seems to me that this is because <code>self.initial_hidden</code> is actually a <strong>tuple</strong> and not a parameter. You can try breaking it down to <code>self.initial_h, self.initial_c = self.init_hidden(self.batch_size)</code></p>

Talita: <p>Thank you!! It is now added in the list model.named_parameters().</p>
<p>However my collab notebook crashes when I train the LSTM. I thought that there might be a device issue, but both <code>self.initial_h</code> and <code>self.initial_c</code> are on cuda.</p>

albanD: <p>As <a class="mention" href="/u/jonathan_harel">@Jonathan_Harel</a> mentionned,  <code>nn.Parameter</code> that are inside python data structured cannot be explored when you look for parameters.<br>
You can either unpack them into two different attributes or store them in a <code>ParameterList()</code>.</p>
How do u decide which regularisation method is better l1 or l2 or drop out or batch normalisation

Sravani_Lekkala: <p>how do u decide which regularisation method is better l1 or l2 or drop out or batch normalisation</p>

albanD: <p>Hi,</p>
<p>I’m afraid it depends a lot on your task and you will have to try them to see how they perform in your particular case.<br>
If your task is close to a classical one, it can be good to look at other models used there and try similar things first.</p>
Does backward function call forward function?

Kalle: <p>So I noticed from the following <a href="https://github.com/leftthomas/SRGAN" rel="nofollow noopener">PyTorch code</a> that the <a href="https://github.com/leftthomas/SRGAN/blob/master/loss.py" rel="nofollow noopener">loss script</a> has no backward function.</p>
<pre><code class="lang-auto">class GeneratorLoss(nn.Module):
    def __init__(self):
        super(GeneratorLoss, self).__init__()
...

    def forward(self, out_labels, out_images, target_images):
....
</code></pre>
<p>And when the code is executed only the backward function are called:</p>
<pre><code class="lang-auto">generator_criterion = GeneratorLoss()
g_loss = generator_criterion(fake_out, fake_img, real_img)
g_loss.backward()
</code></pre>
<p>Is the backward a standard function that does forward propagation inside it? Or when will this forward function be called? Other posts related to this is welcome.</p>

Nikronic: <p>Hi,</p>
<p>For definining custom losses, you just need to define forward function. Backward function is completely computed by Autograd engine.</p>
<pre><code class="lang-auto">g_loss = generator_criterion(fake_out, fake_img, real_img)

</code></pre>
<p>This line is equal to:</p>
<pre><code class="lang-auto">g_loss = generator_criterion.forward(fake_out, fake_img, real_img)
</code></pre>
<p>So, explicitly you call forward, and autograd engine will compute backward operation when you can backward in line:</p>
<pre><code class="lang-auto">g_loss.backward()
</code></pre>
<p>And also, a neural network woks in the way that first you feedforward, then backpropagate errors which in most of the libraries, backward operation has been handled by an automatic engine for computing gradients.</p>
<p>Bests</p>

Kalle: <p>Thanks! That was exactly the explanation I needed.</p>
Zero Division Error When Using Pytorch Faster-RCNN

Wertiz: <p>I’m following the tutorial shown <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">here</a> but I’m also adapting it to work with FasterR-CNN using the pretrained model.<br>
When the <code>output = model(images, targets)</code> gets executed, I receive this error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “C:/Users/werti/PycharmProjects/testtorch/myFRCNN/debug2.py”, line 53, in <br>
output = model(images, targets)<br>
File “C:\Users\werti\anaconda3\envs\testtorch\lib\site-packages\torch\nn\modules\module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\werti\anaconda3\envs\testtorch\lib\site-packages\torchvision\models\detection\generalized_rcnn.py”, line 66, in forward<br>
images, targets = self.transform(images, targets)<br>
File “C:\Users\werti\anaconda3\envs\testtorch\lib\site-packages\torch\nn\modules\module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\werti\anaconda3\envs\testtorch\lib\site-packages\torchvision\models\detection\transform.py”, line 44, in forward<br>
image = self.normalize(image)<br>
File “C:\Users\werti\anaconda3\envs\testtorch\lib\site-packages\torchvision\models\detection\transform.py”, line 64, in normalize<br>
return (image - mean[:, None, None]) / std[:, None, None]<br>
RuntimeError: ZeroDivisionError</p>
</blockquote>
<p>As far as I understand the std equals zero but i can’t see how that is up to me :[<br>
Anyone got advices?<br>
My images are 512x512 before being rescaled to 240x240. each image has only one bounding box and one class label</p>
<p>My code (which is very basic) if it can help:</p>
<blockquote>
<p>model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)<br>
data = dld.DLDataset(csv_file, folder_path, transforms=get_transform(train=True))<br>
data_loader = torch.utils.data.DataLoader(data, batch_size=2)</p>
<p>images,targets = next(iter(data_loader))<br>
images = list(image for image in images)<br>
targets=toListofDict(targets) <span class="hashtag-raw">#This</span> function turns the dict of list in a list of dicts as this particular model wants<br>
output = model(images, targets)<br>
print(output)</p>
</blockquote>

ptrblck: <p>Could you upload a sample image please, so that we could try to reproduce it?</p>
<p>This error seems to be related to <a href="https://discuss.pytorch.org/t/zerodivisionerror-on-using-torchvisions-maskrcnn/82613/2">this post</a>.</p>
<p>CC <a class="mention" href="/u/fmassa">@fmassa</a>: it seems some custom images are creating this zero division. Have you seen this issue before?</p>

Wertiz: <p>I was able to see this through in the end.<br>
There was an error in my ToTensor() transform. I guess that the final image not being normalized between 0 and 1 but between 0 and 255 was causing issue.</p>
<p>Thanks for the reply!</p>
Tied "conv1d" and "conv_transpose1d" not geting the same result as the input

111324: <p>Here is a snippet of my test code：</p>
<pre><code class="lang-python">    data = torch.tensor([[[1.], [2.], [3.], [4.]]])  # N=1 * L=4 * C_in=1
    data = data.permute(0, 2, 1)
    conv = nn.Conv1d(1, 1, 2, bias=None)
    print(conv.weight)
    output = conv(data)
    print("output : {}".format(output))
    dweight = conv.weight.transpose(0, 1).flip(2, )   # dweight = conv.weight.transpose(0, 1)
    print("dweight : {}".format(dweight))

    dconv = F.conv_transpose1d(input=output, weight=dweight, bias=None)
    print(dconv)   # != data
</code></pre>
<p>The result I suppose it would be is: input -&gt; conv1d(input) -&gt; **conv_transpose1d(conv1d(input)) should be <strong>equal</strong> to <strong>input</strong>.</p>
<p>But they are <strong>not equal</strong>, whether I flip the temporal axis or not, but they are supposed to be the same, right?</p>
<p>I’m really confused and frustrated here, could anyone figure it out?</p>

ptrblck: <p>Please don’t tag specific people, as this might discourage others to answer.</p>

111324: <p>I think I misunderstand the “tied weight” concept.</p>
<p>I wrote the conv_transposed1d in <strong>doubly block circulant matrix</strong> form and I find that one don’t need to flip the temporal axis actually.</p>
<p>Suppose the conv1d’s matrix is <img src="https://chart.googleapis.com/chart?cht=tx&amp;chl=%5CLarge%20W" width="22" height="17"> and the corresponding conv_transpose1d’s matrix is  <img src="https://chart.googleapis.com/chart?cht=tx&amp;chl=%5CLarge%20W%5E%7B%5Cmathr%7BT%7D%7D" width="37" height="22">.</p>
<p>The square matrix <img src="https://chart.googleapis.com/chart?cht=tx&amp;chl=%5CLarge%20W%5Ctimes%20W%5E%7B%5Cmathr%7BT%7D%7D" width="80" height="22"> apprently is not always identity matrix. So the result need not to be identical to the input.</p>