akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Load convolutional layers partially from saved model

ttbb: <p>Is there a way to load specific layers of convolutional layers (e.g. only 1-3), when I saved the model in following way:</p>
<pre><code class="lang-auto">torch.save({'state_dict': net.state_dict(),'optimizer': optimizer.state_dict()},PATH)
</code></pre>
<p>instead of</p>
<pre><code class="lang-auto">torch.save(net.state_dict(), PATH)
</code></pre>
<p>My network looks like:</p>
<pre><code class="lang-auto">        self.conv.add_module('conv1_s1',nn.Conv2d(3, 32, kernel_size=9, stride=1, padding=0))
        self.conv.add_module('relu1_s1',nn.ReLU(inplace=True))
        self.conv.add_module('pool1_s1',nn.MaxPool2d(kernel_size=3, stride=1))                           

        self.conv.add_module('conv2_s1',nn.Conv2d(32, 48, kernel_size=3, padding=2, groups=2))
        self.conv.add_module('relu2_s1',nn.ReLU(inplace=True))
        self.conv.add_module('pool2_s1',nn.MaxPool2d(kernel_size=3, stride=2))

        self.conv.add_module('conv3_s1',nn.Conv2d(48, 96, kernel_size=3, padding=2, groups=2))
        ..
        self.conv.add_module('conv4_s1',nn.Conv2d(96, 96, kernel_size=3, padding=2, groups=2))
        ..
        self.conv.add_module('conv5_s1',nn.Conv2d(96, 48, kernel_size=3, padding=2, groups=2))


</code></pre>
<p>I tried to load the layer partially from saved model like:</p>
<pre><code class="lang-auto">        checkpointL = torch.load(PATH)
        originalNetwork.load_state_dict(checkpointL['conv.conv1_s1.weight'])
        originalNetwork.load_state_dict(checkpointL['conv.conv1_s1.bias'])
</code></pre>
<p>and got error: <code>KeyError: 'conv.conv1_s1.weight'</code></p>
<p>Do I have to save the model using <code>torch.save(net.state_dict(), PATH)</code> in order to be able to load only specific layers of my convolutional layers.</p>
<p>Thanks for help!</p>

Ajay_Babu: <p>I think the way you have saved the model seems fine<br>
Can you print <code>checkpointL.keys()</code>  values?</p>

ttbb: <p><code>checkpointL.keys()</code> yields <code>dict_keys(['state_dict', 'optimizer'])</code>. I want to load the weights for only e.g. <code>'conv1_s1'</code> or <code>'conv2_s1'</code>.</p>
Making a batch of 2d list with varying length for non rnn purposes

hadi_gharibi: <p>Hi,<br>
each of my samples has this structure<br>
`</p>
<blockquote>
<p>[[1,2,3,…],[1,2,3,…],…]</p>
</blockquote>
<p>`<br>
The outer list can have up to 100 indices and each of those inner lists can have up to 140 indices and each of them is an embedding index.<br>
I’m trying to calculate a meta embedding for each inner list and merge them into 1 embedding per inner list and from there calculate my attention layer.<br>
My problem is how can I batchify this. Since I need two layers and padding and packing and it’s not supported by the standard pad and pack functions. Any hint could be useful.<br>
thanks in advance.</p>
PyCharm can find CUDA but not the python

tengerye: <p>I installed the PyTorch using docker on the server. I use the PyCharm to remotely develop by connecting it to the python environment in docker container. I can use the CUDA. However, when I go to the container and start the Python environment, CUDA is <strong>not available</strong>. The command I use is <code>torch.cuda.is_available()</code>. Could someone give me some suggestions please?</p>

shlomish3: <p>Had a similar problem as well.<br>
Make sure your cuda version and server’s driver are compatible:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/8/8809dda493d6aa18b478fd0bdde55df41fec8e3d.png" alt="image" data-base62-sha1="jps4xVVXj5Of9eYIMR2uFIGQgdn" width="458" height="442"></p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#abstract">
  <header class="source">

      <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#abstract" target="_blank" rel="noopener nofollow ugc">docs.nvidia.com</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#abstract" target="_blank" rel="noopener nofollow ugc">CUDA Compatibility  :: NVIDIA Data Center GPU Driver Documentation</a></h3>

  <p>CUDA Compatibility document describes the use of new CUDA toolkit components on systems with older base installations.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>On linux you can easily see that by typing <code>nvidia-smi</code> on terminal.</p>
<p>I had to downgrade my pytorch installation, since I couldn’t update the driver (not sudo).</p>

tengerye: <p>Hi <a class="mention" href="/u/shlomish3">@shlomish3</a>, thank you for your kind reply. But if it is the problem, why my PyCharm can access the CUDA? They are supposed to use the same environment.</p>

shlomish3: <p>Oh, you’re right.<br>
It’s beyond my experience, then.<br>
Thought I could help <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_frowning_face.png?v=9" title=":slightly_frowning_face:" class="emoji" alt=":slightly_frowning_face:"></p>

tengerye: <p>Super wired. I finally solve the problem: in the container I executed <code>sudo python3 XXXX</code>. When I type <code>whoami</code>, it told me <code>root</code>. I guess it must be the problem of docker.</p>
List of Lists with nn.Module()

jmaronas: <p>Hello.</p>
<p>I want to properly register a list of nn.Modules() that I structure as a list of list because they represent connections in a matrix. How is the best way to register a list of lists modules in pytorch?. As far as I understand from the doc, <code>nn.ModuleList()</code> requires an interable containing modules, hence not sure if it is useful for my purposes.</p>
<p>I was thinking in creating a <code>nn.Module</code> utility that contains each of the lists, and then create a <code>nn.ModuleList</code> with a list of these modules.</p>
<p>Thanks in advance.</p>

Nathan_Wood: <p><code>ModuleList</code> itself is a module, so you can just do something like:</p>
<p><code>nn.ModuleList(nn.ModuleList(... for _ in range(5)) for _ in range(5))</code> to create a 5x5 “grid” of modules. If you need a more complex structure, you can just do <code>nn.ModuleList([nn.ModuleList(...), ...])</code> in any format you need.</p>
<aside class="quote no-group" data-username="jmaronas" data-post="1" data-topic="87441">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jmaronas/48/2816_2.png" class="avatar"> jmaronas:</div>
<blockquote>
<p>they represent connections in a matrix</p>
</blockquote>
</aside>
<p>However, this makes me think there might be a better way to do what you want. Could you be a bit more explicit?</p>
How are the outputs of each layer of multi-layer bidirectional RNNs fed to the subsequent layer?

Scipio: <p>Working with multi-layer BLSTMs for voice activity detection this question occurred to me. There are two underlying considerations:<br>
First, in an unidirectional multi-layer LSTM the input to any layer after the first ones are the hidden states of the previous one.<br>
Second, the final layer of a bidirectional multilayer RNN is putting out a tensor with size (2xhidden_size) with the hidden states of the forward and backward directions combined.<br>
Thus my question: When feeding from one bi-directional layer to the next how are the outputs combined?</p>
Add costum regularization to loss

netaglazer: <p>How can I add custom regularization to my loss?<br>
I use cross-entropy with weights:</p>
<p><code>criterion = torch.nn.CrossEntropyLoss(weight=weights)</code></p>

ptrblck: <p>Your current code snippet would use a weighted loss (which might be seen as a type of regularization).<br>
The usual work flow would be to add the regularization term directly to the loss as seen <a href="https://discuss.pytorch.org/t/how-does-one-implement-weight-regularization-l1-or-l2-manually-without-optimum/7951/2">here</a>.</p>
Bigger images than model input size

Dennis_Hugle: <p>Dear Community,</p>
<p>I came across this phenomenon earlier this week:</p>
<pre><code>import torch
import torchvision
model = torchvision.models.densenet121(pretrained=True)
model.eval()

im224s = torch.zeros(size=(1, 3, 224, 224))
im360s = torch.zeros(size=(1, 3, 360, 360))
im640 = torch.zeros(size=(1, 3, 640, 360))
im1920 = torch.zeros(size=(1, 3, 1920, 1080))

print("--- TEST: testing 224x224 ---")
x = model(im224s)
print("--- TEST: OK              ---")

print("--- TEST: testing 360x360 ---")
x = model(im360s)
print("--- TEST: OK              ---")

print("--- TEST: testing 640x360 ---")
x = model(im640)
print("--- TEST: OK              ---")

print("--- TEST: testing 1920x1080 ---")
x = model(im1920)
print("--- TEST: OK              ---")
</code></pre>
<p>Returns:</p>
<pre><code>--- TEST: testing 224x224 ---
--- TEST: OK              ---
--- TEST: testing 360x360 ---
--- TEST: OK              ---
--- TEST: testing 640x360 --- 
--- TEST: OK              ---
--- TEST: testing 1920x1080 --- 
--- TEST: OK              ---
</code></pre>
<p>When I put images into my vanilla-pretrained-densenet121, that are bigger than what the actual input size should be (224x224), the model still returns an output.</p>
<p>I get that the input size doesn’t matter so much for the kernels and feature maps that can still be built, but the size of the feature map depends on the input image.<br>
So if this works, using a bigger-than-expected image must lead to some form of information throw-away at least right before the fully connected layer in the end, which has fixed-size input.</p>
<p><strong>So how are pytoch models handling over-sized images?</strong></p>
<p>Cheers</p>
<p>tl;dr: the output volume of feature maps are calcuated like this:<br>
Size=(Size_pre-Filtersize+2Padding)/(Stride+1), i.e. the size of a feature map is dependend on the size of the previous feature map and the filter-size. How is pytorch handling over-sized images? It does not throw an error.</p>

Dennis_Hugle: <p>Okay I solved the issue myself. In the forward function of the densenet one can see that an adaptive average pooling 2d is used. There, the sliding window for the pooling is calcuated in such a way that the result is (something x 1 x 1) no matter of the input size.</p>
<p>forward function of densenet:</p>
<pre><code class="lang-auto">    def forward(self, x):
        features = self.features(x)
        out = F.relu(features, inplace=True)
        out = F.adaptive_avg_pool2d(out, (1, 1))  # the magic happens here
        out = torch.flatten(out, 1)
        out = self.classifier(out)
        return out
</code></pre>
Probability at each timestep

Chiara: <p>I’m using the MNIST dataset for image classification as done here  <a href="https://medium.com/dair-ai/building-rnns-is-fun-with-pytorch-and-google-colab-3903ea9a3a79" rel="nofollow noopener">https://medium.com/dair-ai/building-rnns-is-fun-with-pytorch-and-google-colab-3903ea9a3a79</a> (in the “RNN for image classification” bit).<br>
The data has shape 28x28, which means that i have 28 timesteps (rows in the image) and 28 inputs (number of columns) . The data goes into the rnn row by row.<br>
I would like my code to print the outputs after each timestep i.e. I would like the code to give me the probabilities after each row goes into the rnn.</p>
<p>Can this be done?</p>

ptrblck: <p>You could pass the time steps one by one instead of the complete batch in these lines of code:</p>
<pre><code class="lang-python">lstm_out, self.hidden = self.basic_rnn(X, self.hidden) # apply loop here
out = self.FC(self.hidden)
</code></pre>
<p>Note that you could pass each row to the linear layer to get the logits, but this would change your model architecture, as currently only the hidden state of the last time step is used.</p>
Masking NaN and Infinities

Jack_Rolph: <p>Suppose I have a tensor with some unknown number of NaNs and Infinities. I do not know how many I expect, and therefore need to mask them as part of a model.</p>
<p>I have tried:</p>
<pre><code class="lang-auto">x = torch.randn(B,C,X, Y)
x_fix = torch.zeros_like(x)
x = torch.where( (torch.isnan(x)) | torch.isinf(x), x, x_fix) 
</code></pre>
<p>This, however, causes nans in my loss function. I assume this is because the function is non-differentiable.</p>
<p>How would I go about solving this problem?</p>

ptrblck: <p>Why do you have NaNs and Infs in your tensors in the first place?<br>
Would it be possible to avoid these values or are you expecting these “invalid” values at some point?</p>
From detectron2.utils.env import TORCH_VERSION

Himesh_Panchal: <p>hi,<br>
i am getting above error when i try am trying to convert to Caffe module, using google colab</p>
<h2><a href="https://detectron2.readthedocs.io/tutorials/deployment.html" rel="nofollow noopener">https://detectron2.readthedocs.io/tutorials/deployment.html</a></h2>
<h2>using command</h2>
<p>cd tools/deploy/ &amp;&amp; ./caffe2_converter.py --config-file …/…/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml <br>
–output ./caffe2_model --run-eval <br>
MODEL.WEIGHTS detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl <br>
MODEL.DEVICE cpu</p>
<p>any help is appreciated</p>
<p>thanks<br>
himesh</p>
Run multiple forward parallel with 1 GPU

Y_Y: <p>Hi all,<br>
If I only want to run forward() with same inputs and different weights in parallel, then merge the output after all forward() is done. How can I do this in Pytorch? It looks like I might be able to use cuda stream? <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-streams" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/cuda.html#cuda-streams</a> But I’m not sure if this is the right way to go. Is there a more efficient way to do this?<br>
Thank you all.</p>

ptrblck: <p>You could use streams, but would have to make sure to properly synchronize the code to avoid race conditions.<br>
Depending on the actual workload on the device, your speedup might not be huge, e.g. if the first model already uses the device sufficiently.</p>

Y_Y: <p>Hi ptrblck,<br>
Thank you for the reply.<br>
I tested streams, it looks like the speed up is not significant. Say I want to change weights in 10 different ways and run forward() as fast as possible 10 times with same inputs, what would you recommend to implement this?<br>
Thank you.</p>

ptrblck: <p>The simplest approach would be to run the forward pass in a loop and to avoid any synchronizations, such as printing the output etc.</p>
How to handle a memory expensive step in the model to support higher batch size?

thyr: <p>I have a step in my model which is very expensive as it does operation over the entire vocabulary. Let’s say the step is as follows</p>
<pre><code class="lang-auto">output =self.vocab_process(input)
</code></pre>
<p>input is of dimension bsz x seq_len x d<br>
output is of dimension bsz x seq_len x k</p>
<p>I found that batch size of 128 works well for this step but anything more than that would OOM. I am thinking of ways to make the model evaluation faster. Hence to support I am wondering if I can set the eval batch size to 1024 and in each iteration instead of calling <code>vocab_process</code> once, call it 8 times.</p>
<p>Something like</p>
<pre><code class="lang-auto">outputs = []
for i in range(8):
    output =self.vocab_process(input[i*128:(i+1)*128])
    outputs.append(output)

model_out = torch.stack(outputs)
</code></pre>
<p>Also one more step that im considering is inside the implementation of <code>vocab_process</code>, call <code>del tensor_name</code> to free up memory once i dont need a tensor anymore. How does calling <code>del</code> compare to making everything inplace?</p>
<p>Are these two techniques reasonable? Are there better/cleaner ways to do this?</p>
<p>Thank you.</p>

ptrblck: <p>If intermediate tensors in <code>vocab_process</code> need the device memory and will thus raise the OOM issue, your loop approach could work (assuming that <code>vocab_process</code> returns the right output for sub-tensors).</p>
<p>Since Python uses function scoping you wouldn’t have to delete the intermediate tensors, as they will be freed once you exit <code>vocab_process</code>. If you are running into the OOM before exiting the method, the <code>del</code> methods might still be useful.</p>
<p>In-place operations won’t allocate new memory, but will instead manipulate the tensor directly.</p>

thyr: <p>Thanks for the reply! Are you aware of any other places were we use a similar loop approach?</p>

ptrblck: <p><code>F.bilinear</code> used a matrix multiplication, which created large intermediate tensors and introduced a performance regression (in comparison to the previous pure Python implementation). <a class="mention" href="/u/tom">@tom</a> fixed this by using a loop approach some time ago in <a href="https://github.com/pytorch/pytorch/pull/6110">this PR</a>.</p>
Another weird bug of torchscript: it thinks my python bool is a tensor but it's not

Chenchao_Zhao: <p>TorchScript Errors are usually very informative.</p>
<pre><code class="lang-auto">EMPTY_FLOAT = torch.zeros(0).float()

class Bug(nn.Module):

    def __init__(self):
        super(Bug, self).__init__()

    def forward(self,
                my_bool: bool,
                x=EMPTY_FLOAT):

        if my_bool:
            y = x
        else:
            y = EMPTY_FLOAT
</code></pre>
<p>Then run</p>
<pre><code class="lang-auto">bug = Bug()
bug_script = torch.jit.script(bug)
</code></pre>
<p>Then comes the compile time error</p>
<pre><code class="lang-auto">RuntimeError: 
python value of type 'Tensor' cannot be used as a value:
  File "&lt;ipython-input-310-549c5631a7a3&gt;", line 15
    
        if my_bool:
        ~~~~~~~~~~~...  &lt;--- HERE
            
            y = x
</code></pre>
<p>The problem seems to be <code>EMPTY_FLOAT</code> if I use <code>torch.zeros(0).float()</code> instead it compiles correctly.</p>

Michael_Suo: <p>Very strange, do you mind filing an issue on GitHub and we can follow up there? Thanks!</p>
Get batch gradient for each individual element using hook

AltmanMe: <p>Hi,</p>
<p>I am trying to implement Grad-CAM with batch and I am referring to this<a href="https://github.com/utkuozbulak/pytorch-cnn-visualizations/blob/master/src/gradcam.py#L67" rel="nofollow noopener">Grad-CAM</a> repository. To my understanding, I just need to modify the shape of the <code>one_hot_output</code> (the first dimension) and accordingly get the gradients for each individual element.</p>
<p>Currently, the issue is that although I can get the right shape of each variable such as <code>gradients and weights</code> the final activation map seems incorrect when I was overlaying the CAM into the original images.</p>
Accessing layers of classifier from DeepLabv3 (torch vision model)

JvanderLinden: <p>Hi, I wanted to access certain layers of DeepLab V3. Like this I got the model</p>
<blockquote>
<p>Blockquote model = tv.segmentation.deeplabv3_resnet101(pretrained=False, progress=True, num_classes=1, aux_loss=None)</p>
</blockquote>
<p>After that I accessed the backbone layers with success:</p>
<blockquote>
<p>Blockquote self.bn1 = model.backbone.bn1<br>
self.relu = model.backbone.relu<br>
self.maxpool = model.backbone.maxpool<br>
self.layer1 = model.backbone.layer1<br>
self.layer2 = model.backbone.layer2</p>
</blockquote>
<p>However, don’t succeed to get the layers from the Classifier part, because they are defined with an integer:</p>
<blockquote>
<p>Blockquote DeepLabHead(<br>
(0): ASPP(<br>
(convs): ModuleList(<br>
(0): Sequential(<br>
(0): Conv2d(2048, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)<br>
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
)<br>
(1): ASPPConv(<br>
(0): Conv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(12, 12), dilation=(12, 12), bias=False)<br>
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
)<br>
(2): ASPPConv(<br>
(0): Conv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(24, 24), dilation=(24, 24), bias=False)<br>
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
)<br>
(3): ASPPConv(<br>
(0): Conv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(36, 36), dilation=(36, 36), bias=False)<br>
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()</p>
</blockquote>
<p>Has somebody a solution for this? Thanks.</p>

Nikronic: <p>Hi,</p>
<p>In cases that names of layers are defined using integers, you can access them by using indexing like lists in python.<br>
For instance, let’s say <code>model.deeplabhead</code> gives the quoted code. You can get ReLU in first sequential layer using <code>model.deeplabhead[0].convs[0][2]</code>.</p>
<p>PS: You can format codes by wrapping them using a pair of ```</p>
<pre>```
code
```
</pre>
<p>Bests</p>
Memory Leak When Replacing torch.nn.NLLLoss() with torch.nn.CrossEntropyLoss()

RylanSchaeffer: <p>I’m trying to refactor my code and now I’ve discovered that if I replace <code>torch.nn.NLLLoss()</code> with <code>torch.nn.CrossEntropyLoss()</code>, my code crashes from a memory error. Negative log likelihood by itself is fine. I’ve been debugging for hours and I have no ideas. Does anyone have any suggestions for what’s causing this?</p>
<p>I know there’s a memory leak from using the following code. I watch the number of non-garbage collected tensors climb:</p>
<pre><code class="lang-auto">    count = 0
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
                print(type(obj), obj.size())
                count += 1
</code></pre>

ptrblck: <p><code>nn.CrossEntropyLoss</code> would call into <code>F.cross_entropy</code> in <a href="https://github.com/pytorch/pytorch/blob/a4dec0674c4ce2fb42dedbe8fd2cf7bec021839b/torch/nn/modules/loss.py#L947-L948">this line of code</a>, which would then use <code>nll_loss(log_softmax())</code> in <a href="https://github.com/pytorch/pytorch/blob/a4dec0674c4ce2fb42dedbe8fd2cf7bec021839b/torch/nn/functional.py#L2422">this line of code</a>, so the calls should be equivalent.</p>
<p>Could you post a code snippet to reproduce this issue?</p>

RylanSchaeffer: <p>I found the issue. I was storing the loss from the function call in a tensor and this was never getting freed, for some reason. More interestingly, memory was consumed more quickly for cross entropy, perhaps because it consists of two operations instead of one, hence why NLL appeared to have no memory leak when it did.</p>
<p>What are the rules by which a tensor is garbage collected? Why does overwriting a tensor not free it?</p>

ptrblck: <p>Could you post a small code snippet which would reproduce the memory leak?<br>
Tensors should be freed once all references to them are deleted.<br>
If an object still points to the tensor, it cannot be freed, but I’m unsure what your exact use case is.</p>
How to combine two Mask-RCNN model and one-dimensional data using detectron2?

hsuzue: <p>I would like to make the architecture using detectron2 like attached image, but I don’t know how to do that. Could anyone tell me about that?<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e8d6b402216567c7a8aa75c615d43fd38e669de.png" alt="image" data-base62-sha1="4mhjorziTnLBy55MqpKzq5IauHI" width="480" height="282"></p>

ptrblck: <p>The image doesn’t give much information.<br>
Would you like to flatten all outputs of the models and concatenate it with the 1D data tensor before feeding it into the fully-connected layer?<br>
Also, which predictions from the MaskRCNN models would you like to use?</p>

hsuzue: <p>I would like to use date and time data, not just images data, as input data.<br>
I would like to do semantic or instance segmentation. The output  is also like that (detected or not in each pixel).<br>
Is it possible?<br>
I don’t care the network in detail.</p>

ptrblck: <aside class="quote no-group" data-username="hsuzue" data-post="3" data-topic="86975">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hsuzue/48/26002_2.png" class="avatar"> hsuzue:</div>
<blockquote>
<p>I don’t care the network in detail.</p>
</blockquote>
</aside>
<p>This seems to be the most important detail in order to give you some ideas. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>Generally, it’s possible to combine multiple inputs, but without a proper description, I cannot give you examples.<br>
E.g. your MaskRCNN models would already give you the predictions, while the image seems to feed (some) predictions with an additional input to a fully-connected layer. What should this fully-connected layer return?</p>

hsuzue: <p>I assume that the outputs of MaskRCNN models are the probability of the object on each pixel, and this fully-connected layer also return the probability on each pixel (2D).</p>
NVCuda.dll missing

harshini_sewani: <p>Hello Everyone,<br>
Can someone help me to solve the error shown in the image?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/7/27ea45fcd7439e4deebd5779fcea679b905a17d2.png" alt="image" data-base62-sha1="5H6y5joz8oGL0UqbXoa1oA9dlXc" width="679" height="394"></p>
Correct way to declare multiple optimizers for single model in DDP

Scott_Hoang: <p>Given the pseudo model below:</p>
<pre><code class="lang-auto">class model(nn.Module):
     def __init__(self):
            self.alpha_params1 = nn.Parameters(&lt;size&gt;, requires_grad=True)
            self.alpha_params2 = nn.Parameters(&lt;size&gt;, requires_grad=True)

           &lt; typical Conv2d layers&gt;....
     
    def forward(self, x):
           &lt;feed forward&gt;
            return output

   def net_parameters(self, filtered_name='alpha_', recurse=True):
          # return torch layer params      
          for name, param in self.named_parameters(recurse=recurse):   
                if filtered_name not in name:
                      yield param  
   def extra_params(self):
          # return user defined params
          return [self.alpha_params1, self.alpha_params2]         
</code></pre>
<p>So above is my pseudo model code.</p>
<pre><code class="lang-auto">net = model() # instantiate model above
optimizer_1 = Adam(model.net_parameters, lr=0.001,...)
optimizer_2 = Adam(model.extra_params, lr=0.003,...)
criterion = L1()
##
# typical Apex Distributed Data Parrallel initialization 
##
for epoch in epochs:
  for data1, data2 in dataloader:
       output = net(data1.data)
       loss = criterion(output, data1.gt)
       loss.backward()
       optimizer_1.zero_grad()
       optimizer_1.step()

       output2 = net(data2.data)
       loss2 = criterion(output2, data2.gt)
       loss2.backward()
       optimizer_2.zero_grad()
       optimizer_2.step()
  # save checkpoint
 torch.save(net.module.state_dict(), f"state_dict_{epoch}.pth")
 torch.save(net.module.extra_params(), f"extra_params_{epoch}.pth")
  

</code></pre>
<p>Above is my pseudo code for model instantiation and training.</p>
<p>At every 10 epoch intervals, I checkpoint by saving model.state_dict() as well as model’s alpha parameters separately. I then compare the value of my alpha parameters between different epochs. What I found is that both parameters from separate epochs are identical in values as well the model’s weights. It seems no update is happening.  Any help is appreciated.</p>

mrshenli: <aside class="quote no-group" data-username="Scott_Hoang" data-post="1" data-topic="87348">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/scott_hoang/48/9907_2.png" class="avatar"> Scott_Hoang:</div>
<blockquote>
<pre><code class="lang-auto">       loss2.backward()
       optimizer_2.zero_grad()
       optimizer_2.step()
</code></pre>
</blockquote>
</aside>
<p>I believe you need to call <code>zero_grad()</code> either before <code>backward()</code> or after <code>step()</code>. Otherwise, the grad is always 0 when optimizer <code>step</code> tries to use it.</p>

Scott_Hoang: <p>yep silly mistakes that cost me 4 days of gpu time <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
LSTM forecasting model converges on persistence algorithm

kraj011: <p>Hi,<br>
We are currently working with pytorch on an LSTM model for forecasting. However, we find that our model always seems to converge on the persistence algorithm whenever we train. Essentially, this means that the model uses the previous time step (t-1) to predict the expected outcome at the next time step (t). Does anyone know how we can prevent this from occurring?</p>
<p>Our current LSTM architecture is as follows:<br>
Input size: 2<br>
Hidden dim: 500<br>
Output size: 2<br>
Num layers: 1<br>
LSTM dropout: 0.05</p>
<p>And we have an FCL at the end with a 0.05 dropout. We appreciate any help! Thanks!</p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [256, 5]] is at version 3; expected version 2 instead

Mohamed_Elashry: <p>I am using a custom Linear Layer in Pytorch, whose job is to add noise to the network.<br>
I have checked also whether there are in-place operation but everything seems okay</p>
<pre><code class="lang-auto">class NoisyLinear(nn.Linear):

    def __init__(self, in_features, out_features, sigma_init=0.017, bias=True):
        super(NoisyLinear, self).__init__(in_features, out_features, bias=bias)
        self.sigma_weight = nn.Parameter(torch.full((out_features, in_features), sigma_init))
        self.register_buffer("epsilon_weight", torch.zeros(out_features, in_features))
        if bias:
            self.sigma_bias = nn.Parameter(torch.full((out_features,), sigma_init))
            self.register_buffer("epsilon_bias", torch.zeros(out_features))
        self.reset_parameters()

    def reset_parameters(self):
        std = math.sqrt(3 / self.in_features)
        self.weight.data.uniform_(-std, std)
        self.bias.data.uniform_(-std, std)

    def forward(self, input):
        self.epsilon_weight.normal_()
        bias = self.bias
        if bias is not None:
            self.epsilon_bias.normal_()
            bias = bias + self.sigma_bias * self.epsilon_bias
        return F.linear(input, self.weight + self.sigma_weight * self.epsilon_weight, bias)
</code></pre>
<p>the Input size would be 5 and the output would be 256. but it gave me that error.</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [256, 5]] is at version 3; expected version 2 instead.

</code></pre>

albanD: <p>Hi,</p>
<p>The call to <code>normal_()</code> are actually inplace.<br>
Have you tried removing those (as an experiment to see if they are the ones that are problematic?</p>

Mohamed_Elashry: <p>Hi,<br>
Yes it worked.<br>
I used torch.normal rather than the in-place operation.</p>
(libtorch) Saving multiple modules and optimizers in single file

Nathan_Wood: <p>Multiple modules can easily be solved by creating a class containing every module and registering them all. How can I put multiple optimizers in the same file?</p>

Nathan_Wood: <p>A solution for static optimizers may work something like this:</p>
<pre><code class="lang-auto">std::ostream &amp;operator&lt;&lt;(std::ostream &amp;os, const ModelImpl &amp;model) {
    for (const torch::Tensor &amp;param : model.parameters()) {
        size_t size = 1;
        for (long s : param.sizes()) size *= s;
        torch::Tensor cpu = param.to(torch::kCPU, false);

        os.write(static_cast&lt;const char *&gt;(cpu.data_ptr()), size * 4);
    }
    return os;
}

std::istream &amp;operator&gt;&gt;(const ModelImpl &amp;model, std::istream &amp;is) {
    torch::NoGradGuard g;
    for (const torch::Tensor &amp;param : model.parameters()) {
        size_t size = 1;
        for (long s : param.sizes()) size *= s;
        torch::Tensor cpu = torch::empty_like(param, torch::kCPU);

        is.read(static_cast&lt;char *&gt;(cpu.data_ptr()), size * 4);
        param.copy_(cpu);
    }
    return is;
}
</code></pre>
<p>where each value is just written directly to an output stream. This ignores endianness and datatypes other than floats, and doesn’t add the optimizers’ state. This is not at all optimal because of those reasons and also because it means the optimizer must be initalized on every value.</p>
Can sampler return other things besides indicies e.g. tensor of size M K*N C H W?

pinocchio: <p>Docs say:</p>
<blockquote>
<p><strong>sampler</strong>  (<a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler" rel="noopener nofollow ugc"> <em>Sampler</em> </a> <em>,</em>  <em>optional</em> ) – defines the strategy to draw samples from the dataset. If specified,  <code>shuffle</code>  must be  <code>False</code> .</p>
</blockquote>
<p>but I want it to work with multithreading properly. My data-set is a data-set of data sets so I want to create a sampler that simply sampled my meta set correctly.</p>
<pre><code class="lang-auto"> e.g. tensor of size M K*N C H W
</code></pre>

ptrblck: <p>The sampler will return indices, which will be passed to the <code>Dataset.__getitem__</code> method.<br>
The returned tensor shape from your dataset is thus unrelated to the sampler.</p>
<p>You can see the implementations (and the return values in <code>__iter__</code>) for the samplers <a href="https://github.com/pytorch/pytorch/blob/cf8a9b50cacb1702f5855859c657a5358976437b/torch/utils/data/sampler.py#L62">here</a>.</p>
<p>If your tensors are stored in the specified shape <code>[M, K*N, C, H, W]</code> inside your <code>Dataset</code>, you should be able to return them in this shape.</p>
<p>The <code>DataLoader</code> would add a batch dimension to these tensors. I don’t know what the specified shapes mean, but if <code>N</code> is the batch size, you could set <code>batch_size=1</code> in the <code>DataLoader</code> and <code>squeeze</code> dim0 in the <code>DataLoader</code> loop.</p>

pinocchio: <p>This is for N-way K-shot learning.</p>
<p>So (meta) batch size is <code>M</code>.</p>
ONNX is_in_onnx_export

dashesy: <p>I have already created <a href="https://github.com/pytorch/pytorch/issues/27265" rel="nofollow noopener">this</a> issue.</p>
<p>Many times we want to know if we are in the middle of exporting to ONNX to select a different path (say MemoryEfficientSwish vs Swish). Passing a boolean around all the time is not a great idea. So far I have been using <code>torch._C._get_tracing_state()</code> but it is private API.</p>
<p>I just noticed there is  <code>torch.onnx.is_in_onnx_export()</code> . Is it equivalent to  <code>torch._C._get_tracing_state()</code> ?</p>
How to do 3d data augmentation in parallel on the gpu?

tianmaliuxing: <p>I have a lot of 3d data and need to do various data augmentation. I want to do data augmentation in parallel on the gpu, but it seems that pytorch does not allow gpu operation in the dataloader. Is there any good way?</p>

JuanFMontesinos: <p>You need to set num_workers=0 to be able to use dataloader in gpu.</p>

tianmaliuxing: <p>Thanks for your reply, I know your meaning, but I want to do data augmentation in parallel on the gpu. If I set num_workers=0, I can only do data augmentation on single gpu.</p>

JuanFMontesinos: <p>There is no such a think like multi-gpu dataloader.<br>
The most you can do is to use dataloader as datareader.<br>
Then set a nn.module for preprocessing. Apply data parallel over it and do any preprocessing you need.<br>
I doubt it were efficient but you can try. You should have a very dense preprocessing to feel the advantage of spliting the data in several gpus.<br>
Another option is you to use a single gpu for preprocessing and another one for the main workload.</p>

AralRalud: <p>Hi, I am having the same issues. Did you find a way to implement GPU data preprocessing?</p>

lfolle: <p>You could also try something new:<br>
Use some pytorch layers for the data augmentation. But set them to <code>requires_grad=False</code>. Then each <code>n</code> steps you could change the properties of the augmentation randomly.<br>
This would support <code>DataParallel</code> and thus multiple GPUs.</p>
torch.utils.data.DataLoader "next" function?

Omroth: <p>I am loading from several Dataloaders at once, which means I can’t do</p>
<pre><code class="lang-auto">for batches, labels in dataloader
</code></pre>
<p>I really need something like</p>
<pre><code class="lang-auto">batches, labels = dataloader.next()
</code></pre>
<p>Anyone provide a solution?<br>
Thanks</p>

Rahul_pillai: <p>try this</p>
<pre><code class="lang-auto">batch = next(iter(dataloader))

input, target = batch
</code></pre>
<p>in this way i believe you can only access a single batch of data, this is more<br>
effecient than for loop if you want to view a single batch of data</p>

lfolle: <p>In case both datasets are of the same size you might also zip them and iterate them using a <code>for</code> loop as such:</p>
<pre><code class="lang-auto">from torch.utils.data import DataLoader

ds1 = [0, 1, 2, 3]
ds2 = [10, 11, 12, 13]

dl1 = DataLoader(ds1)
dl2 = DataLoader(ds2)
dl1_iter = iter(dl1)
dl2_iter = iter(dl2)

for s1, s2 in zip(dl1_iter, dl2_iter):
    print(s1)
    print(s2)
</code></pre>
<p>This yields:</p>
<pre><code class="lang-auto">tensor([0])
tensor([10])
tensor([1])
tensor([11])
tensor([2])
tensor([12])
tensor([3])
tensor([13])
</code></pre>
Maininting Gradient While Connecting Two Model

Ahmed_Abdelaziz: <p>I am connecting two models, one model that extracts entities out of a document and the other uses these entities to generate a text summary. Both models are originally trained disjointly in a pipeline. I am trying to train them jointly in a way that gradients can flow all the way from the end model to the first. However, I am having some problems maintaining the gradients in such an integration.</p>
<p>For example, the first model generates the candidate entity starts, ends, and their corresponding scores. The candidate starts and ends they don’t have any gradient function attached to it since they are just all possible candidate combinations up to a length. However, the candidate scores are the result of scoring every candidate using a FNN. I would like to feed the best candidate to the next model without detaching the gradients. What I have now is a normal python function that takes the scores, and returns the indices of the best candidates. I then use these indices to gather the corresponding candidate starts and ends. However, that doesn’t preserve the gradient since originally both candidate_starts and candidate_ends matrices don’t have a grad function attached to it.</p>
<pre><code class="lang-auto">###Model 1
# [num-sentences, #candidates]
candidate_starts = [
[1,1,3],
...
]

# [num-sentences, #candidates]
candidate_ends = [
[1,2,4],
...
]

# [num-sentences, #candidates]
candidate_scores = FNN(...) = [
[0.1, 0, 0.3],
...
]

# where candidate_scores.grad_fn != None

###Model 2
# best entities (entity 1 and entity 3)
entity_input = [
   [1], # entities expands for the range from entity_start to entity end, in that case 1 to 1
   [3,4] # 3 to 4
]

</code></pre>
<p>I tried gathering the candidate_starts and candidate_ends with the best indices returned from the function but that doesn’t keep gradients. Is there a way that I can gather the indices from the scores tensor instead (in a way similar to max) so that I can use it later in the backward path?</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>You will need to use the scores in a differentiable manner and give a differentiable input to the second network if you want gradients to be able to flow back.<br>
Note that if the inputs of the second network are integers, this most likely won’t be possible as you can’t get gradients for non-continuous types.</p>
Merge accounts on discuss.pytorch.org

pmeier: <p>The first time I signed up here I registered with my email. Recently I was tagged in by my GitHub handle. This results in me having two accounts, which is cumbersome to me. Is it possible to merge them together? I prefer to keep the account signed in from GitHub. If so, how should I get in touch for the details?</p>

albanD: <p>Hey,</p>
<p>Sure, send me a private message with the two accounts and the one you want to keep.</p>
Under the Mnist-Hogwild framework, how to use multi-gpus computing？

Srwzx: <p>When I execute the code example of mnist_hogwild, I find that multiple processes are running parallelly on one gpu.<br>
Question: Can multiple processes be executed in parallel on multiple GPUs?</p>

mrshenli: <p>Yes, that should be possible. Just move/copy the model to different devices. Did you encounter any error when doing that?</p>
What is the size of the in features Resnet 50

Giuseppe: <p>Hi guys, I need to know the size of the “In Features” (before Fully connected Layer) of resNet 50.<br>
Is it 7x7x2048? Is the right size?<br>
And if I want this “Image” this code:</p>
<pre><code class="lang-auto">class ResNetMultiHead(nn.Moule):
    def __init__(self):
        super(ResNetMultiHead, self).__init__()

        self.model = torchvison.models.resnet50()
        self.model.fc = nn.identity()

    def forward(self, x):

        head1 = self.model(x)
        head2 = head1
        head3 = head1

        return head1, head2, head3
</code></pre>
<p>Is it right?</p>
<p>So each head is 7x7x2048?</p>

Lausanne: <p>Channel should be 2048 if directly using the last stage in resnet-50, but height and width should be correlated with your input image’s resolution. The stride between input and the last feature map is 32, which means if your input size is 224 * 224 * 3, then the output (head1) should be 7 * 7 * 2048</p>

Giuseppe: <p>Ok , I make the exact calculation.<br>
Thanks for your reply</p>
Model is wrong on the dataset it was trained on

Gal_Co: <p>Hi,</p>
<p>I’ve build a new model using VGG16 architecture, and trained only the fully connected layer in order to classify gender of a person. During training, the accuracy over the training set of 10k photos is reaching 94% and over the validation set around 82%  .</p>
<p>The problem appear after the training is over, i’m doing a simple test by arbitrary choosing some photos from the training dataset,  and the results just seems wrong, the accuracy i’m getting is somewhere around 50%. If i got 94% accuracy on the model, and using the same dataset the model was trained on, that means it should be wrong for only 6 out of 100 photos, but that is not the case.</p>
<p>I didn’t provide any code because I didn’t know what could be relavent, so if anything is needed i’ll add it.</p>
<p>I’ll appreciate any thoughts on why and how this could happen.</p>

Lausanne: <p>Since your training set contains 10k photos,  how many photos have your sampled from it? The accuracy does not distribute uniformly, maybe some classes are difficult to classify, some are much easier. I think you can sample more photos to check it.</p>
Loading batch at certain index (dataloader) (restarting training)

mayool: <p>Hey everyone,</p>
<p>I am training my networks on my universities “grid engine network”. Unfortunately, I have to restart my script every 30-45 min… The reson for this is that the grid automatically kills jobs that run longer after a certain amount of time, so other students can also use the grid engine.</p>
<p>Since sometimes one epoch takes longer than the time available for each students, I have to save the current index from the train loader batch and restart it from that position.<br>
At the moment I am doing it this way:</p>
<pre><code class="lang-auto">found_restart=False

for epoch in range(start_epoch, config["num_epochs"]):
    for batch in train_loader:
        if restart_condition: # restart if condition is met
            save_checkpoint(checkpoint)
            restart_script()
            break

        # no restart, continue training
        idx, (images, labels) = batch

        # load batches in case of restarting
        if (
        not torch.all(torch.eq(idx, old_index.cpu()))) and not found_restart:  # skip until point of restart is found
            # Load next batch until the position of old index
            continue 
        # start training if last idx position was found
        found_restart = True
        pred = net(images, old_pred)
(...)
</code></pre>
<p>What I do is keep loading the batches until the indexes are matching up. This is inefficient since I am loading a bunch of images and don’t use them.<br>
Is there a way to tell the train_loader to start loading at a certain index?</p>
<p>Thanks for any help!</p>
Something dumb in the ordering of destruction

111254: <p>I am facing some Bug of Cuda destruction:<a href="https://github.com/pytorch/pytorch/issues/40425" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/40425</a><br>
Is there anything related with “dumb in the ordering of destruction”?The<br>
The fbcode setting is meaning what?</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/cudnn/Handle.cpp#L12" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/cudnn/Handle.cpp#L12" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/aten/src/ATen/cudnn/Handle.cpp#L12</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="2" style="counter-reset: li-counter 1 ;">
<li>#include &lt;ATen/cuda/detail/DeviceThreadHandles.h&gt;</li>
<li>#include &lt;c10/cuda/CUDAStream.h&gt;</li>
<li>
</li><li>namespace at { namespace native {</li>
<li>namespace {</li>
<li>
</li><li>void createCuDNNHandle(cudnnHandle_t *handle) {</li>
<li>  AT_CUDNN_CHECK(cudnnCreate(handle));</li>
<li>}</li>
<li>
</li><li class="selected">void destroyCuDNNHandle(cudnnHandle_t handle) {</li>
<li>// this is because of something dumb in the ordering of</li>
<li>// destruction. Sometimes atexit, the cuda context (or something)</li>
<li>// would already be destroyed by the time this gets destroyed. It</li>
<li>// happens in fbcode setting. @colesbury and I decided to not destroy</li>
<li>// the handle as a workaround.</li>
<li>//   - @soumith</li>
<li>#ifdef NO_CUDNN_DESTROY_HANDLE</li>
<li>#else</li>
<li>    cudnnDestroy(handle);</li>
<li>#endif</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Are you running into any issues regarding the order of the handle destructions?<br>
If so, this should be fixed in <a href="https://github.com/pytorch/pytorch/pull/39345">this PR</a>. Could you install the latest nightly binary and retry it?</p>
<p>If you are not seeing an issue, you could ignore the <code>ifdef</code> as it’s used only in some internal builds, if I’m not mistaken.</p>

111254: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, i am very glad to see you reply this.<br>
Have you ever see my question in github?<br>
<a href="https://github.com/pytorch/pytorch/issues/40425" rel="nofollow noopener">#40425</a><br>
Are you sure my question have something to do with <span class="hashtag">#39345</span>?<br>
I am trying <span class="hashtag">#39345</span> Now.</p>
Loop optimization

Hongzhang_Shan: <p>I am kind of new to pytorch. If I have a loop like following:<br>
steps = 10<br>
for i in range(steps):<br>
c = torch.mm(a, b)  # this line is indented<br>
results = c</p>
<p>How many torch.mm will be executed ? 10, or 1 ?<br>
Can pytorch figure out it only need to execute 1 time to get the results ?</p>

ptrblck: <p>The matrix multiplication will be executed 10 times.<br>
I don’t know, if the JIT would have the ability to remove redundant operations.</p>