akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Missing keys in state_dict

Khubaib_Siddiqui: <p>I trained a model and saved the state_dict that I am trying to load again. However, I am getting a “missing keys in state_dict” while loading the model again, even though it has the exact same architecture.</p>
<p>The “missing keys” and “unexpected keys” are also very strange. It looks like the only difference between missing keys and unexpected keys is that the missing keys have an extra ‘0’ in them, as follows:</p>
<p>missing key: “feature_extractor_trainable.0.0.conv1.weight”<br>
unexpected key: “feature_extractor_trainable.0.conv1.weight”</p>
<p>All missing keys have the same issue: there is an extra 0 in the parameter name.</p>
<p>Any idea of what might be causing this?</p>
<p>Thanks.</p>

IliasPap: <p>That extra 0 is probably from a torch.Sequential () module<br>
A code with your model and saving function would be useful</p>

Khubaib_Siddiqui: <p>This is the model architecture:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, n_classes=4, l1=5, l2=6):
        super(Net, self).__init__()
        resnet = torchvision.models.resnet152(pretrained=True)
        num_features = resnet.fc.in_features
        feature_extractor = nn.Sequential(*list(resnet.children())[:l1])
        feature_extractor_trainable = nn.Sequential(*list(resnet.children())[l1:l2])
        for param in feature_extractor.parameters():
            param.requires_grad = False
        self.feature_extractor = feature_extractor
        self.feature_extractor_trainable = feature_extractor_trainable
        self.conv = nn.Sequential(*list(resnet.children())[l2:-1])
        self.classifier = nn.Linear(num_features, n_classes)

    def forward(self, inputs):
        pre_img_features = self.feature_extractor(inputs[:,:3,:,:])
        post_img_features = self.feature_extractor(inputs[:,3:,:,:])
        pre_img_features = self.feature_extractor_trainable(pre_img_features)
        post_img_features = self.feature_extractor_trainable(post_img_features)
        out = post_img_features - pre_img_features
        del pre_img_features, post_img_features
        out = self.conv(out)
        out = torch.squeeze(out)
        out = self.classifier(out)
        return out
</code></pre>

IliasPap: <p>Your model state_dict is not saving the feature_extractor module beacuse you set param_require_grad=False</p>

ptrblck: <p>That’s not the case, and the <code>state_dict</code> will include all registered parameters and buffers to restore the model.</p>
<p><a class="mention" href="/u/khubaib_siddiqui">@Khubaib_Siddiqui</a><br>
Your model works fine using this code:</p>
<pre><code class="lang-python">model = Net()
sd = model.state_dict()

model = Net()
model.load_state_dict(sd)
</code></pre>
<p>Did you change anything regarding the <code>nn.Sequential</code> usage as <a class="mention" href="/u/iliaspap">@IliasPap</a> suggested?</p>
Basic Q: indexing a column of a tensor that could potentially be just a row

PyTorch_Nubie: <p>I’m a nubie to python and pytorch with a very basic question.</p>
<p>I have a tensor T and I want to access column i of this tensor:</p>
<p>T[:,i]</p>
<p>when T is multidimensional (e.g., torch.Size([3, 4])) this works fine, but if it is not (e.g., torch.Size([4])), I get the following error:</p>
<p>IndexError: too many indices for tensor of dimension 1</p>
<p>In the latter case I wish to just access element i of the row vector. My question is: what code should I write so that my indexing can deal with both the M x N shaped tensor and the 1 X N shaped tensor.</p>
<p>Thanks in advance.</p>

G.M: <p>You can try <a href="https://pytorch.org/docs/1.0.1/tensors.html#torch.Tensor.index_select" rel="nofollow noopener"><code>index_select</code></a>: <code>T.index_select(-1, tc.tensor(i))</code>. Also, you need to be careful about the shape of the output of <code>index_select</code>.</p>

PyTorch_Nubie: <p>Works great - thank you : )</p>
<p>Much appreciated.</p>
Developing Intuitions about parameter tuning

vishak_bharadwaj: <p>Hello People,</p>
<p>I just built a facial keypoints recognition dataset that feeds in a greyscale image into the network and outputs 72 keypoints on the face.<br>
I loaded up a pretrained resnet34 and trained the model on this data.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/VishakBharadwaj94/facial_keypoints/blob/master/facial_keypoints.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/VishakBharadwaj94/facial_keypoints/blob/master/facial_keypoints.ipynb" target="_blank" rel="nofollow noopener">VishakBharadwaj94/facial_keypoints/blob/master/facial_keypoints.ipynb</a></h4>
<pre><code class="lang-ipynb">{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "facial_keypoints.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "kxnZs6gIzknB",
</code></pre>

  This file has been truncated. <a href="https://github.com/VishakBharadwaj94/facial_keypoints/blob/master/facial_keypoints.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>The loss was all over the place. So what am I doing wrong here? How do I make the model work better?</p>

ptrblck: <p>Try to scale down the problem and try to overfit a small data sample (e.g. just 10 samples) by playing around with the hyperparameters.<br>
Once your model can overfit it, you could try to scale it up again.</p>
Getting 'tensor is not a torch image' for data type <class 'torch.Tensor'>

copperwiring: <p>As the question suggests, I’m trying to convert <code>images</code> to tensor.</p>
<pre><code>X, y = train_sequence[idx]  
        
images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
masks = Variable(torch.from_numpy(y)).to(device) 
print(type(images)) ## Output: &lt;class 'torch.Tensor'&gt;


images = transforms.Normalize((0.5, 0.5, 0.5, 0.5, 0.5), (0.5, 0.5, 0.5,0.5, 0.5))(images)
masks =  transforms.Normalize((0.5), (0.5))(masks)
</code></pre>
<p>But I get the error at<br>
<code>---&gt; 19         images = transforms.Normalize((0.5, 0.5, 0.5, 0.5, 0.5), (0.5, 0.5, 0.5,0.5, 0.5))(images)</code></p>
<p><code>TypeError: tensor is not a torch image.</code></p>

copperwiring: <p>I think I solved it. I was missing the first parameter.</p>
<pre><code> images = transforms.Normalize(images,(0.5, 0.5, 0.5, 0.5, 0.5), (0.5, 0.5, 0.5,0.5, 0.5))
 masks =  transforms.Normalize(masks, (0.5), (0.5))</code></pre>
Boolean masks and in-place operation with autograd

tlaurent: <p>I am using a boolean mask in a network that perform some attention mechanisms. The boolean mask is updated in-place in some loop which causes autograd to crash. Here is an example of the kind of stuff that I do and that does NOT work:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F 

d = 4
x = torch.rand(d, requires_grad=True)
mask = torch.zeros(d).bool()

# iteration 1
label = 1
mask[0] = True
y = x.masked_fill( mask , float('-inf') )
p = F.softmax(y,dim=0)
loss = - torch.log( p[label] )

# compute by hand grad of the loss at iteration 1
indicator = torch.zeros(d)
indicator[label] = 1
gradloss1 = p-indicator

# iteration 2
label = 1
mask[3] = True
y = x.masked_fill( mask , float('-inf') )
p = F.softmax(y,dim=0)
loss += - torch.log( p[label] )

# compute by hand grad of the loss at iteration 2
indicator = torch.zeros(d)
indicator[label] = 1
gradloss2 = p-indicator

# backprop
loss.backward()

# check we got correct gradient
print(x.grad)
print(gradloss1+gradloss2)
</code></pre>
<p>This can be fixed by replacing the above code by the following code, in which the boolean mask is not updated in-place:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F 

d = 4
x = torch.rand(d, requires_grad=True)

# iteration 1
label = 1
mask = torch.zeros(d).bool()
mask[0] = True
y = x.masked_fill( mask , float('-inf') )
p = F.softmax(y,dim=0)
loss = - torch.log( p[label] )

# compute by hand grad of the loss at iteration 1
indicator = torch.zeros(d)
indicator[label] = 1
gradloss1 = p-indicator

# iteration 2
label = 1
mask = torch.zeros(d).bool()
mask[0] = True
mask[3] = True
y = x.masked_fill( mask , float('-inf') )
p = F.softmax(y,dim=0)
loss += - torch.log( p[label] )

# compute by hand grad of the loss at iteration 2
indicator = torch.zeros(d)
indicator[label] = 1
gradloss2 = p-indicator

# backprop
loss.backward()

# check we got correct gradient
print(x.grad)
print(gradloss1+gradloss2)
</code></pre>
<p>I am a little confused because the boolean mask has always a requires_grad flag set to False. Why does it matter if the mask is updated in-place or not? Can someone provide a little more explanations on how autograd work in this case? Thanks!</p>

albanD: <p>The thing is that the backward operation of <code>masked_fill()</code> needs to know where the input was written into. So it needs the value of the mask. So if you modify it inplace, the autograd engine will fail to compute gradients.</p>

tlaurent: <p>Thanks,  that makes complete sense!</p>
Batch norm- regularization?

lycaenidae: <p>Can batch norm act like regularization?</p>

LeviViana: <p>You might find useful reading carefully the <a href="https://arxiv.org/pdf/1502.03167.pdf" rel="nofollow noopener">article</a>.</p>
<p>Batch norm enables training with larger learning rates, and <a href="https://arxiv.org/pdf/1806.02375.pdf" rel="nofollow noopener">this other article</a> argues “…<em>that the larger learning rate increases the implicit regularization of SGD, which improves generalization</em>…”.</p>
<p>I believe you could start reading these two articles and their references. As far as I know, there is still a lot of mystery on why exactly batch norm works so well.</p>
Is there a mistake in the implementation of vgg network in torchvision models?

kirk86: <p>In the <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py" rel="nofollow noopener">source</a> of vgg implementation from torchvision I see the the following call to the main VGG class</p>
<pre><code class="lang-auto">def _vgg(arch, cfg, batch_norm, pretrained, progress, **kwargs):
    if pretrained:
        kwargs['init_weights'] = False
 calling main VGG class --&gt; model = VGG(make_layers(cfgs[cfg], batch_norm=batch_norm), **kwargs)
    if pretrained:
        state_dict = load_state_dict_from_url(model_urls[arch],
                                              progress=progress)
        model.load_state_dict(state_dict)
    return model
</code></pre>
<p>the signature of VGG class doesn’t take any **kwargs as input</p>
<pre><code class="lang-auto">class VGG(nn.Module):

    def __init__(self, features, num_classes=1000, init_weights=True): &lt;--- no **kwargs inputs
        super(VGG, self).__init__()
        self.features = features
        self.avgpool = nn.AdaptiveAvgPool2d((7, 7))
        self.classifier = nn.Sequential(
            nn.Linear(512 * 7 * 7, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, num_classes),
        )
</code></pre>
<p>as consequence calling <code>_vgg('vgg13_bn', 'B', True, pretrained, progress, **kwargs)</code> where in the **kwargs we add <code>dropout_rate=0.5</code> will result in an error</p>
<pre><code class="lang-auto">    model = VGG(make_layers(cfgs[cfg], batch_norm=batch_norm), **kwargs)
TypeError: __init__() got an unexpected keyword argument 'dropout_rate'
</code></pre>

ptrblck: <p><code>kwargs</code> is unpacked to e.g. <code>init_weights = False</code> inside <code>VGG</code>'s <code>__init__</code> method.</p>

kirk86: <p>Even if that’s the case, does it seem the right thing to do?<br>
I guess the question is why silently unpack <code>**kwargs</code> into <code>init_weights</code>? Why not make explicit and easier for everyone read it.</p>
<p>Finally, I don’t see in the current implementation how could anyone use <code>**kwargs</code> to setup dropout rate in the dropout layers, so the whole point of having <code>**kwargs</code> in the first place seems to be vacuous.</p>

ptrblck: <p>Since <code>kwargs</code> is passed to the <code>VGG</code> class, you can use it to specify e.g. <code>num_classes</code>:</p>
<pre><code class="lang-python">model = models.vgg11(num_classes=11)
</code></pre>
<p>The <code>kwargs</code> argument is used in the current approach to use the specific models definitions, e.g. <code>vgg11</code>, <code>vgg16</code>, as an abstraction layer and pass additional arguments to the construction of the model.</p>
<p>If you want to be able to pass more arguments and think that changing the dropout probability is a missing feature, I would recommend to create a feature request on GitHub with your proposal.</p>
First Derivative matirx in L2 norm constraint

yu_Magic: <p>Dear all,<br>
Recently, I work on this loss function which has a special  L2 norm constraint.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f936fb5111fda2b8b8720571a1e3fedfaa2122aa.png" data-download-href="https://discuss.pytorch.org/uploads/default/f936fb5111fda2b8b8720571a1e3fedfaa2122aa" title="公式"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f936fb5111fda2b8b8720571a1e3fedfaa2122aa_2_690x59.png" alt="公式" data-base62-sha1="zyEGEiAvWGCVB8PLquGNnIvBlEu" width="690" height="59" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f936fb5111fda2b8b8720571a1e3fedfaa2122aa_2_690x59.png, https://discuss.pytorch.org/uploads/default/original/3X/f/9/f936fb5111fda2b8b8720571a1e3fedfaa2122aa.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/9/f936fb5111fda2b8b8720571a1e3fedfaa2122aa.png 2x" data-dominant-color="E4E4E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">公式</span><span class="informations">723×62 10.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The G denotes the first derivative matrix for the first layer in the neural network. I try to search for a lot of methods. However, it can not work for this constraint. How can I implement this constraint?  To write a new autograd function for the first layer in the neural network? or implement a new optimizer?<br>
Thank you for your kindest help!</p>

albanD: <p>Hi,</p>
<p>If <code>G = dce_loss / dW1</code> where <code>ce_loss</code> is the cross entropy loss. Then you can compute this as:</p>
<pre><code class="lang-python">ce_loss = criterion(output, target)
G = autograd.grad(ce_loss, W1, create_graph=True)[0]
loss = ce_loss + (G * W1).pow(2).sum()
</code></pre>
General method of retrieval of h5py files as training data

Fibbi: <p>I’m quite new to PyTorch and am a bit unsure if my method of storage and retrieval of training data is efficient. To clarify, my code works as expected and runs without errors, but slowly.</p>
<p>I’m using h5py (which I am also new at) and tried to model my functions after the suggestions in the forum post: <a href="https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/7" class="inline-onebox">DataLoader, when num_worker &gt;0, there is bug</a></p>
<p>My training data has the size <code>X = (400000,1000)</code> and <code>y = (400000,3)</code> which is saved as a hdf5 file using:</p>
<pre><code class="lang-python">with h5py.File(fileName, 'w') as f:
        for i in range(X.shape[0]):
            f.create_dataset('%s/data_X' % i,data = X[i])
            f.create_dataset('%s/data_y' % i,data = y[i])
</code></pre>
<p>Then later when I want to train my network my retrieval  method looks like:</p>
<pre><code class="lang-python">class H5Dataset(Data.Dataset):
    def __init__(self, h5_path):
        self.h5_path = h5_path
        self._h5_gen = None

    def __getitem__(self, index):
        if self._h5_gen is None:
            self._h5_gen = self._get_generator()
            next(self._h5_gen)
        return self._h5_gen.send(index)

    def _get_generator(self):
        with h5py.File( self.h5_path, 'r') as record:
            index = yield
            while True:
                X = record[str(index)]['data_X'][()]
                y = record[str(index)]['data_y'][()]
                index = yield X, y

    def __len__(self):
        with h5py.File(self.h5_path,'r') as record:
            return len(record)

BATCH_SIZE = 400
loader = Data.DataLoader(
        dataset=H5Dataset(fileName), 
        batch_size=BATCH_SIZE, 
        shuffle=True, num_workers=0)

for i, (X_batch, y_batch) in enumerate(loader):
    # Training occurs here
</code></pre>
<p>Running through one full iteration of the loop (without training) takes about 4 min, which to my untrained eye seems like it could be more efficient. If so what could I do to improve?</p>
On-line pytorch documents runs too slow!

sekigh: <p>How come showing pytorch documents on web is so slow ?</p>

albanD: <p>Hi,</p>
<p>I am not sure what you mean? Are you talking about the documentation? It opens quite smoothly for me.</p>

sekigh: <p>That 's right. It runs too slow, taking a few minutes to move from one page to another, seeming especially so slow in  expanding equation to final format. I do not see such a slow speed on other sites.</p>

albanD: <p>That is weird, it opens instantly for me.<br>
Where do you access it from? Are you behind a firewall? Which browser are you using?</p>

sekigh: <p>I am in Japan. For example, I have the difficulty in accessing on-line pytorch documents, say, torch.nn pages from both my work place,  behind firewall, and my home, in the global. I use chrome 66.0.3359.181（Official Build）.</p>

albanD: <p>Hum I can’t reproduce this behaviour… <a href="https://pytorch.org/docs/stable/nn.html">https://pytorch.org/docs/stable/nn.html</a> loads instantly for me…</p>

Anders_Lykke_Mollgaa: <p>I agree with sekigh that it is inconveniently slow. Takes a couple of seconds to load the first view and then the next 20 seconds it is laggy if I try to scroll. (I am in Denmark using chrome in windows and ubuntu)</p>

zerostar07: <p>it’s unbearably slow and runs the CPU hot. Is there a mirror with simple static HTML pages instead ? nothing slows you down as much as slow reference documentation.</p>

PersonalFinance004: <p>Hello. I also have this problem. Going from google to the PyTorch docs takes up to one minute. I am on MacOS, using Safari. I get a ‘load cursor’ and my computer starts lagging. I get this on no other webiste.</p>
How to set different LR for different layers?

fulltopic: <p>Hi,</p>
<p>In C++ version, how to set different learning rate for different layers as mentioned in <a href="https://pytorch.org/docs/0.3.0/optim.html#per-parameter-options" rel="nofollow noopener">torch.optim</a> ?</p>
<p>Thank you.</p>
Is it possible to do multi-machine single card training

coollofty: <p>I am a deep learning beginner and running PyTorch’s Demo for study. I have several computers and laptops at home, the situation is different for each machine, some machines only have CPU, and some have powerful GPU, but I hope they all come together to speed up a training process.</p>
<p>My implementation idea is the C / S structure. When the server is doing model input, if there is a request from the client, any number of samples are sent to the client through the socket, and the calculation of these sent samples is skipped. After the client completed, it sends back the results of the model forward, and the server merges the results. Finally, if the server completed an epoch, it blocks and waits for the results of all clients to return.</p>
<p>Now, what I don’t know is how to merge the results of the model foward on the client into the server. I don’t know if this is correct …</p>
<p>PS: Or, when the client’s model calls forward (), it does not make a classifier. Before the classifier is called, the data is sent back and the server model completes the classifier. Can the main calculation amount in the forward process be shared Arrived</p>

mrshenli: <p>IIUC, what you have in mind is the reverse structure of <a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html#distributed-reinforcement-learning-using-rpc-and-rref" rel="nofollow noopener">this tutorial</a>. In the tutorial, there are multiple observers sending inputs to the same agent, while in your case, you would like to have the server sending inputs to different clients and run forward on the clients?</p>
<p>The problem with the server-client structure is that, if forward is run on client, the autograd graph and activations will also be on client, meaning that the server cannot merge the output and run the backward locally.</p>
<p>One possible alternative is that, instead of sending forward output from client to server, you can let the client finish forward-backward and then send the model param gradients to server. Then the server collects gradients from all clients and sum them, use the summed grads to update parameters, and then broadcast the updated model to all clients.</p>
<p>Another alternative is to let the client finish forward-backward-optimizer, and then send the model params to the server. Then the server calculates the weighted average of all params and broadcast them back to the clients.</p>
Libtorch 100% CPU load

AntonZ: <p>I am trying to use libtorch to do some inference with CUDA.<br>
I have noticed that CPU load is almost 100% all the time. I have Intel Core i7-9700K.<br>
When I run Visual Studio Profiler it shows that functions like torch::stack load CPU massively.</p>
<p>Currently I am using official libtorch Release build.<br>
I suppose that probably I should built libtorch from sources with other options.</p>
<p>Am I correct? And if yes, which options should I enable?</p>
Function returned an invalid gradient at index 0

jwillette: <p>I copied the pytorch code for <a href="https://github.com/yaringal/ConcreteDropout/blob/master/concrete-dropout-pytorch.ipynb" rel="nofollow noopener">concrete dropout from the authors github</a> repo and I put the model on a GPU in my own code by calling <code>model.to(device)</code>.</p>
<p>I now get the error <code>RuntimeError: Function DivBackward0 returned an invalid gradient at index 0 - expected type torch.FloatTensor but got torch.cuda.FloatTensor</code> and I get that somewhere in the model there is a tensor that is not on the GPU, but I can’t find it and I’ve been printing out everything in the model trying to find out where it is.</p>
<p>Is there an easy way to figure out where this is being caused?</p>

IliasPap: <p>Is the  loss  Maybe calculated on CPU  ??</p>
<p>Try to run your code on cpu to see if you still receive the error</p>

jwillette: <p>I don’t think so, below is the loss function and the calculation</p>
<pre><code class="lang-auto">def heteroscedastic_loss(true, mean, log_var):
    precision = torch.exp(-log_var)
    return torch.mean(torch.sum(precision * (true - mean) ** 2 + log_var, 1), 0)

...

loss = heteroscedastic_loss(y, mean, log_var) + regularization
loss.backward()
</code></pre>
<p>and I tried it on the cpu and everything works fine</p>
Constraint parameter as in tensorflow/python/keras/constraints

lpuglia: <p>Hello, is there are any feature in pytorch that resembles tensorflow/python/keras/constraints, for example in keras it is possible to constraint the parameter values using:</p>
<p><code>tf.keras.constraints.non_neg(),</code></p>
<p>to avoid negative values, i know in pytorch you can apply function after the optimization step as suggested in:</p>
<aside class="quote" data-post="1" data-topic="23620">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/4da419/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/set-constraints-on-parameters-or-layers/23620">Set constraints on parameters or layers</a> 
  </div>
  <blockquote>
    Hi, are there any ways in Pytorch to set the range of parameters or values in each layer? For example, is it able to constrain the range of the linear product Y = WX to [-1, 1]? If not, how about limiting the range of the weight? 
I noticed in Karas, user can define this by setting constraints. Any equivalence in Pytorch? 
Thanks!
  </blockquote>
</aside>
<aside class="quote" data-post="2" data-topic="1933">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jekbradbury/48/14_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/restrict-range-of-variable-during-gradient-descent/1933/2">Restrict range of variable during gradient descent</a> 
  </div>
  <blockquote>
    I would copy the code for the Adam optimizer and modify it to do what you want.
  </blockquote>
</aside>

<p>but this doesn’t seem the cleanest way to do it, i mean, it would be much better if you could register the constraint inside your model class, i know that <code>torch.distributions.constraints</code> module exists but i’m not sure it applies to my case, any hint on this?</p>
Does a stacked LSTM share a weight matrix?

shamoons: <p>If I’m using an <code>LSTM</code> and set <code>num_layers</code> to some number other than 1, do all layers share the same weights?</p>

IliasPap: <p>No each layer has different weights and biases</p>

shamoons: <p>Is it possible to get a different hidden size for each layer?</p>

IliasPap: <p>The easiest way would be to create different LSTMs like in example</p>
<pre><code class="lang-auto">first = nn.LSTM(input_size=input_size1,hidden_size=hidden_size1,num_layers=num_layers1)
second= nn.LSTM(input_size=hidden_size1,hidden_size=hidden_size2,num_layers=num_layers2)

</code></pre>

shamoons: <p>Seems straightforward enough. Thank you. I wonder if there’s any advantage to doing this?</p>
Pytorch model integration with android studio

AryaKulkarni17: <p>I am making an android app that takes x-ray images as an input and gives the class of the disease as an output. A pretrained model has been made and I am integrating it with android studio. However, I am facing runtime errors with the model. After classifying the disease label, the result needs to be stored in a database and here is where the error is. Can someone help me with this problem of integration of model?</p>
Problems with splitting a large size image into small pieces

Patrice: <p>I am having nine large images of size (3001,401,5) and a corresponding label of the same size. I have been trying to split the images into small patches of 64*64. Subsequently, combine the nine small pieces to make up a sample input (which will be an input with 9 channels). My codes are as follows:</p>
<pre><code class="lang-auto">ind = 0

smallD1 = {}
smallD2 = {}
smallD3 = {}
smallD4 = {}
smallD5 = {}
smallD6 = {}
smallD7 = {}
smallD8 = {}
smallD9 = {}
smallD10 = {}

cutSize=64
data = [];label = []
for sliceI in range(0, data1.shape[2]):
    for i in range(0, data1.shape[0], cutSize):
        for j in range(0, data1.shape[1], cutSize):
           
            smallD1[ind] = data1[i:i + cutSize, j:j + cutSize, sliceI]
            smallD2[ind] = data2[i:i + cutSize, j:j + cutSize, sliceI]
            smallD3[ind] = data3[i:i + cutSize, j:j + cutSize, sliceI]
            smallD4[ind] = data4[i:i + cutSize, j:j + cutSize, sliceI]
            smallD5[ind] = data5[i:i + cutSize, j:j + cutSize, sliceI]
            smallD6[ind] = data6[i:i + cutSize, j:j + cutSize, sliceI]
            smallD7[ind] = data7[i:i + cutSize, j:j + cutSize, sliceI]
            smallD8[ind] = data8[i:i + cutSize, j:j + cutSize, sliceI]
            smallD9[ind] = data9[i:i + cutSize, j:j + cutSize, sliceI]

            sample_total= np.stack((smallD1, smallD2,smallD3,smallD4,smallD5,smallD6,smallD7,smallD8,smallD9))
            #sample_total= np.concatenate((smallD1, smallD2,smallD3,smallD4,smallD5,smallD6,smallD7,smallD8,smallD9),axis=2)
            #sample_total=np.transpose(sample_total,(1,2,0))
            #sample_total=np.expand_dims(sample_total, axis=-1)

            smallD10 [ind] = data10[i:i + cutSize, j:j + cutSize, sliceI]
            smallD10=np.expand_dims(smallD10, axis=0)

            data.append(sample_total.tolist()) 
            label.append(smallD10.tolist())

            ind = ind + 1
</code></pre>
<p>When I run the codes, the following error occurs</p>
<pre><code class="lang-auto">smallD10 [ind] = data10[i:i + cutSize, j:j + cutSize, sliceI]
IndexError: index 1 is out of bounds for axis 0 with size 1
</code></pre>
<p>Please, how can I make it run?</p>
<p>Thank you very much for your time and patience</p>

IliasPap: <p>It seems that data10 matrix is not read correctly, check data10 shape .</p>

Patrice: <p>Thank you for the reply sir. I have checked it several times and there is no problem with the matrix shape. I have tried different approaches, but all in vain. I still can not figure out what is wrong with the codes. Please, any other methods for splitting images into small pieces apart from my codes? Any suggestions or recommendations would be highly appreciated.</p>
Transfer learning for Full Dense Networks

Filos92: <p>Hey everyone,<br>
does anyone know if it is useful or even possible to train Full Dense Networks, i.e. networks without any convolutional layers using transferlearning?</p>
<p>Unfortunately I haven’t found any sources for this anywhere and I would like to know if it is worthwhile to put some work into it.</p>

IliasPap: <p>By transfer learning you mean that you have an identical network trained on ther task or dataset and you initialize your network with these weights</p>

Filos92: <p>Yes,<br>
i want to train my Network with a bunch of (at least similar) simulated Data and then retrain it with a few real measured data.<br>
right know i work with self normalizing neural networks, without any convolutionals layers, full dense only. They work pretty well but i have to start initialized weights with zero mean and a standard deviation of the squared root of 1/(size of input).<br>
So i wonder if it is possible to start with pretrained weights. Searching the web has made me believe that conv layer are a requirement for this, because on every site, they only replace/not freeze the full dense layers behind the conv. layers.<br>
I am not stuck on SNNs, if other (not conv) networks are better suited for transferlearning, I would switch, because I see a big advantage for my application in transferlearning.</p>
<p>Any Experience with this task?</p>
Class Interpolation using GANs with Pytorch

Maccido: <p>Hello All,</p>
<p>Can anyone please share a link, a paper or something that can help me interpolate between 2 classes using GANs with the aid of PyTorch. I have seen some online but not with Pytorch. Thank you</p>

ptrblck: <p>Could you post the code you’ve found so that we could have a look, if porting it to PyTorch would work?</p>
Image segmentation related question

mukulkhanna: <p>I am training an encoder-decoder network for semantic segmentation from single RGB image.</p>
<p>The network is being trained to detect 40 classes for the semantic segmentation.<br>
The <strong>ground_truth</strong> that I have has the dimensions - <code>[batch_size, 256, 256]</code> where each pixel in the 256x256 matrix corresponds to a class ID between 0-39.<br>
The network <strong>output</strong>s a <code>[batch_size, 40, 256, 256]</code> tensor.</p>
<p>I am using a CrossEntropy loss to train the same.</p>
<pre><code class="lang-auto">semantic_loss = torch.nn.CrossEntropyLoss()
loss = semantic_loss(output, ground_truth)
</code></pre>
<p>The code is working fine and as the network is training, the loss is going down.<br>
I wanted to know two things -</p>
<ol>
<li>Is the loss calculation approach correct?</li>
<li>How can I visualise the output results (which have shape [bs, 40, 256, 256]) ?</li>
</ol>

ptrblck: <ol>
<li>
<p>Yes, everything looks good as long as your model outputs logits (i.e. no non-linearity as the last layer).</p>
</li>
<li>
<p>You could get the predictions using <code>pred = torch.argmax(output, dim=1)</code> and visualize them e.g. with <code>matplotlib</code>.</p>
</li>
</ol>
RecursionError while saving a model's state dict

Jordan_Juras: <p>Hi,</p>
<p>I am having an ongoing issue with saving the ‘state_dict’ of my model/optimiser. I have followed the guidance from this forum, which encourages saving the state dictionaries of the model and optimiser separately. I unfortunately cannot do this, as the action produces a python recursion error - which states that the limit is reached. If I increase the limit with sys.setrecursionlimit, I get a segfault.</p>
<p>I have worked around this by simply saving the model and the optimiser in their entirety. However, I have updated some functionality in the model for visualisation and testing, and I need to transfer the state dict of the legacy model to the new model to use the functionality.</p>
<p>Has anyone else ran into this issue?</p>
<p>The model is a convolution vae with 4 layers separating the latent space from each of the input and output. The input dimension is [N_batch, 1, 33075], but kernel sizes do not exceed 5 samples. Latent space dimension is on the order of [1, hundreds].</p>
<p>As far as I can tell this is a standard size model (?). Training was not problematic in time.</p>
<p>Any help would be greatly appreciated.</p>

ptrblck: <p>You will get a max recursion error, if you have assigned the model to itself somewhere.<br>
This code demonstrates the issue:</p>
<pre><code class="lang-python">
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(1, 1)
        self.model = self # will create recursion error
        
    def forward(self, x):
        x = self.fc1(x)
        return x

model = MyModel()
model.state_dict()
</code></pre>
<p>If you remove the <code>self.model = self</code> assignment, the <code>state_dict</code> can be created.<br>
Could you check your model for such a loop?</p>
Make operations in for-loop parallel

ziqipang: <p>Hi! I am recently working on a module like this: the module has 8 different operations like conv3x3, conv5x5, skip_connection, etc. And I know that all the operations produces output in the same shape.</p>
<p>During forward, I want to get the result of each operation, combine them together and get the final result. As I have detected, the for-loop is the bottle-neck of the program. Therefore, I am wondering if there are any methods to make the for-loop more parallel?</p>
<p>An example demonstration of my module is as below.</p>
<pre><code class="lang-auto">class Module(nn.module):
    def __init__(self):
        super(Module, self).__init__()
        self.ops = nn.ModuleList()
        for i in range(8):
            # append an operation in self.ops, such as conv3x3
            self.ops.append(...)
    
    def forward(self, x):
        tmp = list()
        for i in range(8):
            tmp.append(self.ops[i](x))
        # func can be any post-preprocessing function, not important
        result = func(tmp)
        return result
</code></pre>
How do i combine two datasets

YinYang_Untalan: <p>I have a dataset of images and I used the ImageFolder for that. I have another dataset A with the same labels as the previous. Now I want to combine them such that each label corresponds to an img and something from A. How do I do this? Help. Thanks.</p>

ptrblck: <p>You could write a new custom <code>Dataset</code>, pass both datasets to it and use your logic inside <code>__getitem__</code> to draw the corresponding labels from your <code>ImageFolder</code> and <code>datasetA</code>.</p>

YinYang_Untalan: <p>Thank you for this. I really appreciate it.</p>
Intermediate Layers of AlexNet/VGG

Arjun_Gupta: <p>I am trying to use PyTorch to get the outputs from intermediate layers of AlexNet/VGG:</p>
<pre><code class="lang-auto">alexnet_model = models.alexnet(pretrained=True)
modules = list((alexnet_model).children())[:-1*int(depth)]
alexnet_model = nn.Sequential(*modules)
</code></pre>
<p>What is odd is that I get the same output values (i.e. the same exact model) when <code>depth=1</code> and <code>depth=2</code>, and then the same output values for <code>depth=3</code> all the way to <code>depth=10</code>. I observe this same phenomenon for VGG too. However, I don’t observe this for ResNet, which gives me different output values (i.e. different models) for all depths [1, 10].</p>
<p>Any ideas about what might be going on?</p>

ptrblck: <p><code>list((alexnet_model).children())</code> will return a list of length 3 containing the first <code>nn.Sequential</code> block for the feature extraction, the <code>nn.AdaptiveAvgPool2d</code> layer, and the last <code>nn.Sequential</code> block used as the classifier.<br>
If you use <code>depth&gt;=3</code>, <code>modules</code> will be empty and you will just get back your input tensor.</p>

Arjun_Gupta: <p>Thanks for your response! How do I get outputs of the layers within the sequential blocks? And how is this working for the ResNet architecture?</p>

ptrblck: <p>You could use forward hook as described in <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">this example</a>.</p>

Arjun_Gupta: <p>Thanks for the reference! Apparently, the layers within the sequential blocks (of AlexNet, VGG, etc.) don’t have names associated with them (e.g. ‘self.fc2’); how could I extract outputs from certain layers within the last sequential block using your function?</p>

ptrblck: <p>You can access a module inside an <code>nn.Sequential</code> block by indexing it:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.Conv2d(3, 6, 3, 1, 1),
    nn.ReLU(),
    nn.Conv2d(6, 1, 3, 1, 1)
)
# get second conv layer
c = model[2]
</code></pre>
Parallel model training - 'c10::Error'

mshuaibi: <p>I’m trying to train multiple models (ensembling) in an online framework - such that if a data point arrives that my model is uncertain about, I retrain the models (taking the average) and proceeding on. However, after the first round of training my models, when I come to retrain them I get a <code>terminate called after throwing an instance of 'c10::Error'</code>.</p>
<p>The error looks something like this:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c4a759f1b94b0c8c0676a7b885c769427b5c858.png" data-download-href="https://discuss.pytorch.org/uploads/default/6c4a759f1b94b0c8c0676a7b885c769427b5c858" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c4a759f1b94b0c8c0676a7b885c769427b5c858.png" alt="Capture" data-base62-sha1="frZ5UXGwPaBaa2tl6y5dsyQiiO4" width="690" height="242" data-dominant-color="422037"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">874×307 12.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m training my models in parallel with something like this:</p>
<pre><code class="lang-auto">def train(training_ensembles):
    # parallelize each model on it's own cpu
    pool = multiprocessing.Pool(len(training_ensembles))
    results = pool.map(train_model, training_ensembles)
return results
</code></pre>
<p>When I train the models in series I run into no issues. Something about the way I’m parallelizing these is causing issues with PyTorch. Any help is appreciated - Thanks!</p>
Cannot use torch.save(model.state_dict(), PATH) with transfer learning

Lopo358: <p>I keep getting this error:</p>
<pre><code class="lang-auto">for param_tensor in model.state_dict():
    AttributeError: 'NoneType' object has no attribute 'state_dict'
</code></pre>
<p>Here is a code snippet from the transfer learning I am trying to implement:</p>
<pre><code class="lang-auto">...
model = torchvision.models.densenet161(pretrained=True)

for param in model.parameters():
    param.requires_grad = False

num_ftrs = model.classifier.in_features
model.classifier = nn.Linear(num_ftrs, num_classes)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.classifier.parameters(), lr=learning_rate, momentum=0.9)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

model = train_model(model, criterion, optimizer, exp_lr_scheduler, num_epochs=num_epochs)

torch.save(model.state_dict(), './save.pt')
</code></pre>
<p>The model is able to get trained and output results, but it is the torch.save(model.state_dict(), PATH) part that does not work - any ideas??</p>

IliasPap: <pre><code class="lang-auto">model = train_model(model, criterion, optimizer, exp_lr_scheduler, num_epochs=num_epochs)
</code></pre>
<p>What does function train_model return ? Are you sure it returns the model ? Because your code seems right.</p>

Lopo358: <p>I am copying the example from here:<br>
<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
<p>the function was supposed to return the model, but I had somehow removed that line. Not that it is back, torch.save is working fine! Thank you for making me look there!</p>
nn.AdaptiveLogSoftmaxWithLoss gives unknown error for nn.DataParallel

Kishore2500: <p>I created a model :</p>
<p>model= nn.DataParallel(model, device_ids=[0,1,2,3])<br>
model=model.to(device)</p>
<p>I used nn.AdaptiveLogSoftmaxWithLoss  :</p>
<p>self.out= nn.AdaptiveLogSoftmaxWithLoss(300, 27624, cutoffs=[round(27624/15),3*round(27624/15)],div_value=4)</p>
<p>The error shows:</p>
<p>Traceback (most recent call last):<br>
File “RUN2.py”, line 237, in <br>
qnet.model_fit()<br>
File “RUN2.py”, line 73, in model_fit<br>
loss = self.model(input_F)<br>
File “/home/akmmrahman/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/akmmrahman/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 153, in forward<br>
return self.gather(outputs, self.output_device)<br>
File “/home/akmmrahman/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 165, in gather<br>
return gather(outputs, output_device, dim=self.dim)<br>
File “/home/akmmrahman/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py”, line 67, in gather<br>
return gather_map(outputs)<br>
File “/home/akmmrahman/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py”, line 62, in gather_map<br>
return type(out)(map(gather_map, zip(*outputs)))<br>
TypeError: <strong>new</strong>() missing 1 required positional argument: ‘loss’</p>
<p>Doesn’t nn.AdaptiveLogSoftmaxWithLoss  support multiple GPUs?  It works fine for a single GPU. It would be very helpful if someone could answer it. Thanks.</p>

zhangguanheng66: <p>The best place to report a potential bug is pytorch github <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How does PyTorch handle negative loss functions?

connorlee77: <p>Say for example I had a</p>
<pre><code class="lang-auto">loss = -nn.MSELoss(x1, x2)
</code></pre>
<p>Would PyTorch drive the loss between x1 and x2 towards -infinity or would it go towards 0?</p>

KFrank: <p>Hi Connor!</p>
<aside class="quote no-group quote-modified" data-username="connorlee77" data-post="1" data-topic="72623" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/connorlee77/48/21507_2.png" class="avatar"> connorlee77:</div>
<blockquote>
<p>Would PyTorch drive the loss … towards -infinity or would it go towards 0?</p>
</blockquote>
</aside>
<p>The standard pytorch optimizers attempt to minimize the loss, that is<br>
to make it algebraically smaller (less positive, more negative).</p>
<p>As an aside, because they use variations on gradient descent, they<br>
don’t care about the overall level of the loss – where zero is – just<br>
what direction it is going.  So <code>lossA</code> and <code>lossB = lossA - 1,000</code><br>
get optimized the same way.</p>
<p>Best.</p>
<p>K. Frank</p>