akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
[HELP] Convert TensorFlow Flatten layer into PyTorch

Stefano_Setti: <p>Hi, I have this layer in TensorFlow:</p>
<pre><code class="lang-auto">Flatten()
</code></pre>
<p>and I’m trying to convert it into PyTorch, that’s my attempt:</p>
<pre><code class="lang-auto">nn.Flatten(2)
</code></pre>
<p>In TF i have this dimensionality reduction:<br>
<code>(None, 1, 1, 64) -&gt; Flatten() -&gt; (None, 64) </code><br>
but in PyTorch  what I achieve is:<br>
<code>(-1, 64, 1, 1) -&gt; Flatten() -&gt; (-1, 64, 1)</code></p>
<p>What am I doing wrong?<br>
Thanks!</p>

Stefano_Setti: <p>I just figured out how the flatten layer in PyTorch works, so basically as firs argument i need to express from where to start the flattening ad as second where to end it. Basically it should be:</p>
<pre><code class="lang-auto">nn.Flatten(1, 3)
</code></pre>
Kernel Regularizer

mash2612: <p>Hello, I wanted to understand how to implement kernel regularizer(parameter in Keras/Tensorflow layer) in a layer in PyTorch. I saw examples of how to implement regularizer for overall loss, but could not find the relevant documentation for this.</p>
<p>Could someone point me in the right direction? I saw something along the lines like what I intend to do here: “<a href="https://discuss.pytorch.org/t/how-does-one-implement-weight-regularization-l1-or-l2-manually-without-optimum/7951" class="inline-onebox">How does one implement Weight regularization (l1 or l2) manually without optimum?</a>”.</p>
<p>I have adapted it as follows:</p>
<pre><code class="lang-auto">l2_reg = None

for i in model.named_parameters():
   if "layer_name.weight" in i:
      if l2_reg is None:
         l2_reg = i.norm(2)**2
      else:
        l2_reg = l2_reg + i.norm(2)**2
batch_loss = some_loss_function + l2_reg * reg_lambda
batch_loss.backward()
</code></pre>
<p>Is this the correct way to do so?</p>
<p>Thanks in advance!</p>

ptrblck: <p>The code snippet looks generally correct.<br>
One minor issue: <code>named_parameters()</code> will return a tuple as <code>name</code> and <code>param</code>, so you might need to use these two variables in the for loop or alternatively unpack <code>i</code>.</p>

mash2612: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for the feedback, yeah I have accessed it through index in my code.<br>
I had encountered an error when I implemented it like this:</p>
<pre><code class="lang-auto">l2_reg = None

for i in model.named_parameters():
   if "layer_name.weight" in i[0]:
      if l2_reg is None:
         l2_reg = i[1].norm(2)**2
      else:
        l2_reg = l2_reg + i[1].norm(2)**2
batch_loss = some_loss_function + l2_reg * reg_lambda
batch_loss.backward()
</code></pre>
<p><a href="https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time/6795">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph = True when calling backward the first time</a>"<br>
After adding this line <code> batch_loss.backward(retain_graph=True)</code> this works.<br>
Though there exists a direct thread, may I take this opportunity to ask you if it is possible to avoid the use of  retain_graph=True in the above code?  I have read this increases the training time in each consecutive training iteration.Hence, I would like to avoid the use if possible.</p>
The F.affine_grid theta of 3D

oldriver-wang: <p>I already use ITK-snap to process an affine registration : 3D volume image A to B.<br>
Then I get a matrix:</p>
<p>0.97464 0.0364447 0.00781475 1.95549<br>
0.00375859 0.922498 0.0297265 -8.87417<br>
-0.00727863 -0.0145295 0.961844 -1.10425<br>
0 0 0 1</p>
<p>I think it’s incompatibal in F.affine_grid theta.<br>
The shape of affine_grid shape is (N, 3, 4)<br>
likes: R | T<br>
[<br>
[a, b, c, t0],<br>
[e, f, g, t1],<br>
[h, i, j, t2]<br>
]<br>
I think the anterior three rows of the matrix is theta, but it’s wrong.<br>
the R is likely equal to theta[:3, :3],(I am not sure)<br>
but the T (t0, t1, t2) must be incompatible.</p>
<p>How can I get a right theta?<br>
By the way, how can I know what does the torch.nn.functioal.affine_grid really do ?</p>
Using affine_grid and grid_sample on intermediate output

Deepak_Suresh: <p>Can the methods <code>affine_grid</code> and <code>grid_sample</code> be used inside the forward method of a network?<br>
The forward pass looks like this</p>
<pre><code class="lang-auto">def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
</code></pre>
<p>I’m interested in using the above methods on <code>x</code> where <code>affine_grid</code> should perform translation and <code>grid_sample</code> reflection padding</p>

ptrblck: <p>That sound be possible and the <a href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html#depicting-spatial-transformer-networks">Spatial Transformer Network</a> uses both methods in its <code>forward</code> pass.</p>
Implementing MatchNet reservoir with pyTorch Dataset and Dataloader

TheW00t: <p>Hi,<br>
I’m trying to implement MatchNet sampler <a href="https://imgur.com/a/jQGDr9P" rel="nofollow noopener">link</a> algorithm with pyTorch  Dataset but I’m having trouble warping my head around it.</p>
<p>Can you please help me to understand how to implement it?</p>
<p>Thank,<br>
Oria</p>

ptrblck: <p>Could you post your current code and explain a bit where you are stuck at the moment? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Multi masks segmentation U-Net

simongeek: <p>Hi,</p>
<p>I would like to train a neural network for landmarks detection as heatmaps using U-Net architecture.<br>
I have 20 points, and I have generated a binary mask for each point (20 classes + background = 21 classes).</p>
<p>For example, my input is 800x800x1 and I would like to have my output as 800x800x21.</p>
<p>Does anybody know how to deals with it?</p>

ptrblck: <p>Assuming your input has the dimensions <code>[height, width, channels]</code>, you would have to <code>permute</code> it and add a batch dimension, since PyTorch modules expect image tensors as <code>[batch_size, channels, height, width]</code>.</p>
<p>The same applies to your target tensor, i.e. it should have the shape <code>[batch_size, 21, 800, 800]</code>.</p>
<p>Based on your description it seems you are working on a mulit-label segemntation, i.e. each pixel position could belong to zero, one, or multiple classes.<br>
If that’s correct, your model output should have the same shape as the target and contain logits (no non-linearity at the end). For the criterion you could use <code>nn.BCEWithLogitsLoss</code>.</p>
<p>Let me know, if that would work for you.</p>
Implement total variation loss in pytorch

John1231983: <p>Hello all, this is the definition of total variation loss</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a4da67731975021a34c566b76b12a0f35ebb84ae.png" data-download-href="https://discuss.pytorch.org/uploads/default/a4da67731975021a34c566b76b12a0f35ebb84ae" title="Screenshot%20from%202019-09-09%2017-40-05"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a4da67731975021a34c566b76b12a0f35ebb84ae_2_690x99.png" alt="Screenshot%20from%202019-09-09%2017-40-05" data-base62-sha1="nwmeuEidNDAV0TFEjJJgvd1zTEa" width="690" height="99" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a4da67731975021a34c566b76b12a0f35ebb84ae_2_690x99.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/a4da67731975021a34c566b76b12a0f35ebb84ae_2_1035x148.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/a/a4da67731975021a34c566b76b12a0f35ebb84ae_2_1380x198.png 2x" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-09-09%2017-40-05</span><span class="informations">1959×282 37.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So, I implemented it as follows:</p>
<pre><code class="lang-auto">    def compute_total_variation_loss(img, weight):      
        tv_h = ((img[:,:,1:,:] - img[:,:,:-1,:]).pow(2)).sum()
        tv_w = ((img[:,:,:,1:] - img[:,:,:,:-1]).pow(2)).sum()    
        return weight * (tv_h + tv_w)

</code></pre>
<p>Is it correct? What does it means <code>img[:,:,1:,:] - img[:,:,:-1,:]</code>?</p>

Xinxiang7: <pre><code class="lang-auto">def total_variation_loss(img, weight):
     bs_img, c_img, h_img, w_img = img.size()
     tv_h = torch.pow(img[:,:,1:,:]-img[:,:,:-1,:], 2).sum()
     tv_w = torch.pow(img[:,:,:,1:]-img[:,:,:,:-1], 2).sum()
     return weight*(tv_h+tv_w)/(bs_img*c_img*h_img*w_img)
</code></pre>
<p>which is not equal to <code>img[:,:,1:,:] - img[:,:,:-1,:]</code>.</p>
Load pretrained weights of BERT, XLNet, GPT in torch.nn.transformers

AmbiTyga: <p>I am working on a script in which I want to use BERT’s encoding method as well as GPT’s decoding method for NMT. Since I need to pass the encoder’s hidden state value to decoders, I want to know how can I use torch.nn.Transformers module. Going through huggingface transformer I found that GPT expects key-value pairs as its input from encoder layer which has a shape of (2, batch_size, num_heads, sequence_length, embed_size_per_head) and BERT outputs the hidden states of each layer(+embedding layer), therefore I also would like to know how could I get the key-value pair from BERT.</p>
Why the input and output grad got via register_backward_hook are the same?

YichengWang: <pre><code class="lang-auto">def hook(m, input_, output_):
    input_=input_[0]
    output_=output_[0]

    print(m)
    print(id(input_))
    print(input_.shape)
    print(id(output_))
    print(output_.shape)

net.module.features[19].register_backward_hook(hook)
</code></pre>
<p>In one iteration, the printed content is as below:</p>
<p><em>Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))</em><br>
<em>140513351157512</em><br>
<em>torch.Size([1, 512, 32, 32])</em><br>
<em>140513351157512</em><br>
<em>torch.Size([1, 512, 32, 32])</em></p>
<p>It seems that the gradient of input tensor of this conv2d layer has been changed to  the gradient of output tensor of this conv2d layer.</p>
<p>Something wrong must have happened.</p>
<p>Who can tell me how to get the real gradient of input tensor?</p>
FindNCCL.cmake search for /usr/local/cuda

ChaiBapchya: <p>In this file - <a href="https://github.com/pytorch/pytorch/blob/master/cmake/Modules/FindNCCL.cmake" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/cmake/Modules/FindNCCL.cmake</a></p>
<p>Why aren’t we checking if nccl.h exists in the default location for unix - /usr/local/cuda</p>
<p>Or is it being done implicitly somewhere?</p>

powderluv: <p>I think it makes sense to set NCCL_ROOT to /usr/loca/cuda if it is not already set since Nvidia recommends install nccl in that path.</p>
How and where can i set index for class?

milad66: <p>Hi<br>
how and where can i set index for class?<br>
Thank You</p>

ptrblck: <p>You can set the class index manually, if you are creating the dataset.<br>
If you are using a dataset from <code>torchvision</code>, such as <code>MNIST</code>, the classes would be already predefined.<br>
For <code>ImageFolder</code>, the specified folders will be sorted and the class indices will be created according to the sorted folders starting at 0.</p>

milad66: <p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>

Praveen_K: <p>What sorting order is followed? if there is cat and dog folder , which one is index:0</p>

ptrblck: <p>The order is determined by the <code>sort()</code> method used in <a href="https://github.com/pytorch/vision/blob/c547e5cbbf25e46870abb2e925b42c67c3367afc/torchvision/datasets/folder.py#L137-L138">this line of code</a>, which might be OS-dependent. In your example, <code>cat</code> would most likely be class0 and <code>dog</code> class1 and you can check it via the <code>dataset.class_to_index</code> attribute.</p>
<p>However, there were differences on Windows and Linux in the sorting of names starting with numbers.<br>
If I remember correctly, you would have to prepend zeros to the name to get the same sorting:</p>
<pre><code class="lang-python">00001
00002
00003
...
00010
00011
</code></pre>
How portable are torch::save()/load() models in C++?

pianoman8: <p>How portable across architectures and pytorch versions and architectures is lorch::save()/::load()?</p>
<p>I’m a developer working on adding ARM support to MLC@Home, a distributed computing project training many simple neural networks in parallel.  Currently, we support amd64 clients (windows/linux 64-bit) and I’m trying to add ARM32/64 support.  Note this is <em>training</em> in <em>c++</em>, not inference, so as I understand it, torchscript and ONNX aren’t options.</p>
<p>The basic work flow is: volunteer connects to server, downloads the client and a dataset, trains for N epochs, saves the resulting network (torch::save) and sends it back to the server.  The server checks is the network is finished training (loss is below a threshold), and if the network isn’t done training, then that same network is re-sent to another client for another N epochs of training (loaded with torch::load).</p>
<p>This has been working great for our x86_64 clients on linux and windows, linked against pytorch 1.5.0 and 1.5.1.  I recently built new clients for ARM, and linked against libtorch v1.6.0. However, if the ARM clients receive a network to continue training that had previously been trained on the x86 clients, the resulting network ends up with NaNs.</p>
<p>I’m recompiling the ARM clients against v1.5.1 to try that… but… how portable across architectures and pytorch versions and architectures is lorch::save()/::load() expected to be?</p>
<p>Is there any other portable model option for pure libtorch code?</p>
Simple question about the backward() behavior of a custom extension

f10w: <p>Suppose that I have a CUDA extension with which I can define a custom <code>autograd</code> function like the following:</p>
<pre><code class="lang-auto">import my_cuda_extension

class MyFunction(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, y):
        ctx.y = y
        return my_cuda_extension.forward(x, y)
        
    @staticmethod
    def backward(ctx, x):
       result = my_cuda_extension.backward(x, ctx.y)
       return result, torch.zeros_like(ctx.y)
</code></pre>
<p>Now, <strong>suppose that my CUDA extension only accepts the NHWC data format</strong>. Thus, when defining my custom layer, I have to convert the NCHW inputs to NHWC, then convert back the results to NCHW:</p>
<pre><code class="lang-auto">class MyLayer(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x, y):
        # convert the inputs to NHWC
        result = MyFunction.apply(x.permute(0,2,3,1).contiguous(),
                                  y.permute(0,2,3,1).contiguous())
        # convert the result back to NCHW
        return result.permute(0,3,1,2)
</code></pre>
<p>The forward pass should clearly work. But how about the backward pass? Will I get the correct gradients? Because I don’t see how the backward pass is involved in the above definition of <code>MyLayer</code>.</p>
<p>Thank you in advance for your answer!</p>

albanD: <p>Hi,</p>
<p>All the code outside the custom Function is autodiffed. And so gradient will be computed for each op you do. So all will work fine here!</p>
<p>Note that in the custom function, you should not save inputs in the context but use <code>ctx.save_for_backward(y)</code> and <code>y, = ctx.saved_tensors</code> to avoid memory leak <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

f10w: <p>Sorry for the late reply. Thanks a lot for your answer! That totally makes sense.</p>
<p>Regarding the use of <code>ctx.save_for_backward</code>, according to <a href="https://discuss.pytorch.org/t/when-should-you-save-for-backward-vs-storing-in-ctx/6522/9">this comment</a>, memory leak doesn’t seem to occur starting from PyTorch 0.4. Is that true? I’ll use <code>ctx.save_for_backward</code> anyway because it seems to be a safe option, thanks for the tip!</p>

albanD: <blockquote>
<p>Regarding the use of <code>ctx.save_for_backward</code> , according to <a href="https://discuss.pytorch.org/t/when-should-you-save-for-backward-vs-storing-in-ctx/6522/9">this comment</a>, memory leak doesn’t seem to occur starting from PyTorch 0.4. Is that true?</p>
</blockquote>
<p>That is for intermediary results, not input/output. You have to use <code>save_for_backward()</code> for these.</p>

f10w: <p>Thanks for the confirmation!</p>
Change conv layer after model creation

Peter_Featherstone: <p>I want to modify the parameters of all nn.Conv2d layers in a model. Specifically, i want to change the kernel_size from (X,Y) to (X,3), padding from (X//2, Y//2) to (X//2,1), stride from (SX,SY) to (SX, 1), and of course the weight tensor to be modified accordingly.</p>
<p>I started doing something like:</p>
<pre><code class="lang-auto">for name, module in net.named_modules():
    if isinstance(module, nn.Conv2d):
        stride = list(module.stride)
        stride[1] = 1
        module.stride = tuple(stride)
</code></pre>
<p>Then I was about to change module.weight then i realised that there is a lot of initialisation that is taken care of in the constructor of the layer. So really, rather than modifying the internals of the layer, i should be reconstructing a new one fresh. How do i do this?</p>
<p>The reason why i want to do this is because i want to do something like:</p>
<pre><code class="lang-auto">import timm
net = timm.create_model("resnet50")
...adapt conv2d layers here...
</code></pre>

bsridatta: <p>wrap code in ``` to make it easier to read.</p>
<p>why dont you simply assign a new layer in the same for loop? It depends on the type of model. For sequential you can directly change like a list, check the code below. If you are using vision models you can use .features to accsss each layer and reassign it with a new one</p>
<pre><code class="lang-auto">import torch

model = torch.nn.Sequential()
_ = model.add_module('a', torch.nn.Linear(1,2))
_ = model.add_module('b', torch.nn.Linear(3,4))
model[0] = torch.nn.Linear(5,6)
model

output: Sequential(
  (a): Linear(in_features=5, out_features=6, bias=True)
  (b): Linear(in_features=3, out_features=4, bias=True)
)
</code></pre>

Peter_Featherstone: <p>Ideally I would like there to be no assumption about the model structure. Otherwise I have to write specific code for specific models.</p>

Peter_Featherstone: <p>Iterating through named_modules was the only way I found that could do this. Unfortunately you cannot modify modules in-place. At least not easily from the looks of it</p>

Nikronic: <p>Hi,</p>
<p>If I understood your case correctly, this <a href="https://discuss.pytorch.org/t/can-i-change-the-channel-for-a-specific-conv2d-inside-a-network/88571/2">post</a> might be able to help. The issue with the snippet is that it has not been extended to support hierarchical layers but it won’t be hard to extend.</p>
<p>Bests</p>
Layer wise quantization

snehaoladri: <p>How do we perform layer wise quantization in pytorch i.e I want to quantize only third and fourth layer, how can i do it?<br>
when we prepare model for quantization using model.prepare all the modules present in the whitelist are quantising . But i didn’t find a way to quantize a single layer. Any kind of help is appreciated?</p>

snehaoladri: <p>I am finally able to achieve it by setting different config for the layers i want to quantize, rather than using model.qconfig (which does for all the layers in the model).</p>
<p>For example:</p>
<p>I quantized the first layer by accessing the first layer and assigning it the qconfig.<br>
per_channel_quantized_model.features[0][0].qconfig=torch.quantization.get_default_qconfig(‘fbgemm’)<br>
per_channel_quantized_model.features[0][0].qconfig=torch.quantization.get_default_qconfig(‘fbgemm’)</p>
<p>Hope this helps. And if there is any other efficient approach to achieve this please do let me know.</p>

jerryzh168: <p>yeah, this is how we do it in eager mode, we have prototype graph mode that works on torchscript models: <a href="https://pytorch.org/tutorials/prototype/graph_mode_static_quantization_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/prototype/graph_mode_static_quantization_tutorial.html</a> which can configure layers with a <code>qconfig_dict</code>. Although we might move away from this soon, but this should still generally work if you need to use it now. Note to use the prototype you will need to use nightly build.</p>
[HELP] convert TensorFlow convolutional layer into PyTorch

Stefano_Setti: <p>Hi  have this layer written in python with the TensorFlow library:</p>
<pre><code class="lang-auto">Conv2D(64, 
       (3, 6), 
       strides=(1, 1), 
       input_shape= (10, 6, 1), 
       activation='relu'
)
</code></pre>
<p>and I’m trying to convert it into PyTorch, for now what i achieved is this:</p>
<pre><code class="lang-auto">self.conv2d = nn.Sequential(
                nn.Conv2d(1, 1, (3,6), (1, 1)),
                nn.ReLU()
            )
</code></pre>
<p>The problem is that in the TF implementation i have as output dimension [None, 8, 1, 64] and in PyTorch [-1, 1, 8, 1] so i think that the last 1 should be 64. This is even validated by the number of parameters that i have in TF and PyTorch respectively: 1216 and 19.<br>
(19*64 = 1216)</p>
<p>Am I missing something?<br>
Thanks.</p>

ZdsAlpha: <p>Set output channels to 64.</p>
<pre><code class="lang-auto">self.conv2d = nn.Sequential(
                nn.Conv2d(1, 64, (3,6), (1, 1)),
                nn.ReLU()
            )
</code></pre>
<p>In pytorch channel dimension is second not last</p>

Blizzard_boi2020: <p>You can check the tutorial for PyTorch.</p>

Stefano_Setti: <p>ok, I’m trying with that, now i have some problems with the input, I’ll check to solve this and I’ll update you. Thanks</p>

Stefano_Setti: <p>can you link it to me please?</p>

Blizzard_boi2020: <p><a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/pytorch_with_examples.html</a></p>
<p>Scroll here, lots of material</p>

Stefano_Setti: <aside class="quote no-group" data-username="ZdsAlpha" data-post="2" data-topic="92579">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zdsalpha/48/51282_2.png" class="avatar"> ZdsAlpha:</div>
<blockquote>
<p>In pytorch channel dimension is second not last</p>
</blockquote>
</aside>
<p>and thanks for the clarification, this is helping me a lot on understanding conditionality on pytorch.</p>
Problem loading pretrained model, fine tune model on custom dataset

shivangi: <p>Step 1 : I am loading pretrained xception model on ImageNet and fine-tuning for binary classification on my dataset.</p>
<pre><code class="lang-auto">from pretrainedmodels.models.xception import Xception
self.xception_model = Xception(num_classes=1000)
.
.
.
.
checkpoint = torch.load('/home/shivangi/Desktop/Projects/pretrained_models/checkpoints/xception-43020ad28.pth')
self.xception_model.load_state_dict(checkpoint)
self.xception_model.last_linear = nn.Linear(2048, 2)
</code></pre>
<p>Step 2 : Now after training is complete I save this trained model. Everything works fine.</p>
<pre><code class="lang-auto">torch.save(self.xception_model.state_dict(), model_path  + self.model_name + '.pt)
</code></pre>
<p>Step 3 : Now I want use this model in step 2 on a different dataset</p>
<pre><code class="lang-auto">self.xception_model = Xception(num_classes=2)
checkpoint = torch.load(model_path  + self.model_name + '.pt')
self.xception_model.load_state_dict(checkpoint)
</code></pre>
<p>Then I get the error : AttributeError: ‘Xception’ object has no attribute ‘last_linear’</p>

ptrblck: <p>It looks like the actual factory to create the model is defiend <a href="https://github.com/Cadene/pretrained-models.pytorch/blob/021d97897c9aa76ec759deff43d341c4fd45d7ba/pretrainedmodels/models/xception.py#L216" rel="nofollow noopener">here</a>, i.e. as <code>pretrainedmodels.models.xception()</code>, while you created an instance of <code>xception.Xception</code> manually.</p>
<p>Since <code>self.last_linear</code> is only redefined in the factory (<a href="https://github.com/Cadene/pretrained-models.pytorch/blob/021d97897c9aa76ec759deff43d341c4fd45d7ba/pretrainedmodels/models/xception.py#L233" rel="nofollow noopener">line of code</a>), you could try to set the <code>last_linear</code> again to your custom linear layer:</p>
<pre><code class="lang-python">self.xception_model = Xception(num_classes=2)
checkpoint = torch.load(model_path  + self.model_name + '.pt')
self.xception_model.last_linear = nn.Linear(2048, 2)
self.xception_model.load_state_dict(checkpoint)
</code></pre>
Write tensor into char buffer and be able to read into a tensor

copythatpasta: <p>Hello,<br>
I am in a position where i need to write the contents of a torch::Tensor to a char buffer, then be able to reconstruct the torch::Tensor from  a supplied buffer. How can i do this? Tensor data type is float</p>

Matej_Kompanek: <p>Would something like this help ?</p>
<pre><code class="lang-auto">torch::Tensor tensorToSave = torch::randn({ 2, 3 });
std::cout &lt;&lt; "tensorToSave: " &lt;&lt; tensorToSave &lt;&lt; std::endl;

// option #1
{
	std::vector&lt;char&gt; data = torch::pickle_save(tensorToSave);

	torch::Tensor loadedTensor = torch::pickle_load(data).toTensor();

	std::cout &lt;&lt; "loadedTensor: " &lt;&lt; loadedTensor &lt;&lt; std::endl;
}

// option #2
{
	std::string data;

	{
		std::ostringstream ostream;
		torch::save(tensorToSave, ostream);
		data = ostream.str();
	}

	torch::Tensor loadedTensor;

	{
		std::istringstream istream(data);
		torch::load(loadedTensor, istream);
	}

	std::cout &lt;&lt; "loadedTensor: " &lt;&lt; loadedTensor &lt;&lt; std::endl;
}
</code></pre>

copythatpasta: <p>Yep! i ended up doing something like option <span class="hashtag">#2</span> but with a sstream, thanks!</p>
Loss function for Multi-Label Multi-Classification

Reem2: <p>I have a Multi-Labeling Multi-Classification problem and I am wondering which loss function should I use.</p>
<p>My labels are positions and types of objects. There is 64 positions and each item could be 0,1,2</p>
<p>Example for label is: [0., 0., 0., 2., 0., 0., 0., 0., 2., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 2., 0., 0., 0., 2., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 2., 0., 0., 0., 0.]</p>
<p><strong>0</strong> means there is no object on this position<br>
<strong>1</strong> means there is an object on this position from type 1<br>
and <strong>2</strong> means there is an object on this position from type 2</p>
<p>I have checked the BCEWithLogitsLoss and this <a href="https://discuss.pytorch.org/t/what-kind-of-loss-is-better-to-use-in-multilabel-classification/32203">post</a>, and I am not sure whether it applies to my problem. I tested the BCEWithLogitsLoss function with one example and the result did not make sense, when output = target the result is bigger than 0 (it is 0.68) and when output != target the result is bigger (0.70). In the example I replaced the 2 with 0.5, since the targets <code>t[i]</code> should be numbers between 0 and 1.</p>

ZdsAlpha: <p>You can simply use CrossEntropyLoss to do this. CrossEntropyLoss now supports any dimensional input. Your final output should have shape of (N, C, X) where N is batch size C is total classes which is 3 in your  case X is total positions which is 64 in your case. During inference you can do argmax on dim=1 and get indices of classes.</p>
Parameters of the quantized model will miss in state_dict after being traced in pytorch1.6. Is it a bug or feature?

Wenlong_Shi: <pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.quantization import QuantStub, DeQuantStub

class QuantizableModel(nn.Module):
    def __init__(self, *args, **kwargs):
        super(QuantizableModel, self).__init__()
        # self.module = ConvBNReLU(3, 64)
        self.conv = nn.Conv2d(3, 2, 3, 1, 1, groups=1, bias=True)
        self.quant = QuantStub()
        self.dequant = DeQuantStub()

        # weight initialization
        nn.init.kaiming_normal_(self.conv.weight, mode='fan_out')
        nn.init.zeros_(self.conv.bias)

    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.dequant(x)
        return x

if __name__ == "__main__":
    model = QuantizableModel().eval()
    inp = torch.randn(1, 3, 224, 224)
    model.qconfig = torch.quantization.get_default_qconfig('fbgemm')
    torch.quantization.prepare(model, inplace=True)
    # # Dummy calibration
    model(inp)
    torch.quantization.convert(model, inplace=True)
    print("before traced: ", model.state_dict().keys())
    traced_model = torch.jit.trace(model, inp).eval()
    print("after traced: ", traced_model.state_dict().keys())
</code></pre>
<p>The output of running the above code in pytorch 1.5 is:</p>
<pre><code class="lang-auto">before traced:  odict_keys(['conv.weight', 'conv.scale', 'conv.zero_point', 'conv.bias', 'quant.scale', 'quant.zero_point'])
after traced:  odict_keys(['conv._packed_params', 'quant.scale', 'quant.zero_point'])
</code></pre>
<p>The output of running the same code in pytorch 1.6 is:</p>
<pre><code class="lang-auto">before traced:  odict_keys(['conv.weight', 'conv.bias', 'conv.scale', 'conv.zero_point', 'quant.scale', 'quant.zero_point'])
after traced:  odict_keys(['quant.scale', 'quant.zero_point'])
</code></pre>
<p>Parameters of the quantized model will miss in state_dict after being traced in pytorch1.6. Is it a bug or feature?</p>
Multi GPU training on single node with DistributedDataParallel

zlapp: <p>Hi,</p>
<p>I’m new to distributed training.</p>
<p>When I train with DistributedDataParallel do I get the functionality of DataParallel, meaning can I assume that on a single node if there is more than one GPU then all GPUs will be utilized on that node?</p>
<p>Thanks,<br>
Zlapp</p>

mrshenli: <p>Yep, DistributedDataParallel (DDP) can utilize multiple GPUs on the same node, but it works differently than DataParallel (DP). DDP uses multiple processes, one process per GPU, while DP is single-process multi-thread.</p>
<p>See this page for the comparison between the two: <a href="https://pytorch.org/tutorials/beginner/dist_overview.html#data-parallel-training" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dist_overview.html#data-parallel-training</a></p>
<p>and this to get started with DDP: <a href="https://pytorch.org/docs/stable/notes/ddp.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/ddp.html</a></p>

zlapp: <p>Great, thanks for the answer and references</p>
What can cause differences in training a model in Windows vs. Ubuntu?

Yuerno: <p>Hey all. I have some code which I’ve been training on my laptop, and the loss looks to go down steadily, indicating that the model is training. However, when I run the same code on a remote Ubuntu machine with a much better desktop GPU, the loss seems to stall out from the beginning. I expected around the same loss values/patterns on both machines, but for the desktop GPU one to be much faster. I’ve only trained for around 10 epochs so far, which is enough to see a major difference between the trainings on Windows vs. Ubuntu.</p>
<p>My question is, are there significant differences when transferring from one OS to the other with PyTorch code that I should be aware of? Do some packages behave differently? I currently use pathlib to ensure my filepaths are platform-agnostic, but other than that, I don’t really do anything special.</p>

SimonW: <p>This is using the same pytorch version, right?</p>
<p>Did you try windows GPU? How does that compare against ubuntu GPU? What about windows CPU vs ubuntu CPU?</p>
<p>Also, things like cudnn and mkldnn matter, as they provide a lot of the conv algorithms used.</p>

Yuerno: <p>Yep, both machines are using PyTorch 1.1.0 through Python 3.7.3 with Anaconda. However, there is a difference in CUDA, my Windows laptop is using 10, whereas the Ubuntu desktop is using version 9. Could that be a large enough difference to cause this training issue?</p>
<p>With regards to comparing GPUs, the Windows GPU trains (albeit a lot slower because it’s an older laptop GPU), but the Ubuntu GPU doesn’t seem to train at all. I haven’t tried comparing CPU performance yet.</p>

Yuerno: <p><a class="mention" href="/u/simonw">@SimonW</a> So just to do some comparison, I tried running the transfer learning tutorial code on both my laptop and the Ubuntu desktop, and the code trains fine on both, indicating that the problem lies probably somewhere specifically with my code and not either machine.</p>

Yuerno: <p>Found my issue! My code had me generating a list of file paths for the raw image files as well as the masks, and I forgot that file ordering is different on Windows vs. Linux when using something like <code>os.listdir()</code>. On Windows, the file orderings for both the raw images and the masks were the same, so I was able to match the images to masks correctly for processing, but on Linux, the raw images had a different file order than the masks, which meant that the wrong masks were being matched with the wrong raw images, causing the issue of no training.</p>

SimonW: <p>glad you figured out!</p>

Andrej_Hafner: <p>Thank you for this, had the same problem! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Mughees: <p>It took me three days and I was unable to understand the issue I am having until I finally I land on this page and got a solution. Bundle of thanks.</p>
Backpropagated gradients with a Nan Loss

jmaronas: <p>Hello.</p>
<p>I would like to know if this is something new in pytorch. As far as I remember, in the initial pytorch releases the gradients of a parameter computed from a loss whose value was Nan (due to numerical saturation) where also Nan. However, I have realized that when my loss goes to Nan, the gradients w.r.t the parameters is 0. Is that the correct pytorch behavior?</p>
<p>Thanks</p>

ptrblck: <p>I would expect the same and can still reproduce it:</p>
<pre><code class="lang-python">model = models.resnet18()
out = model(torch.randn(1, 3, 224, 224))

loss = out.mean() * torch.log(torch.tensor(-1.))
print(loss)
loss.backward()

for name, param in model.named_parameters():
    print(name, torch.isnan(param.grad).all())
</code></pre>
<p>Could you post a code snippet to reproduce it?</p>

jmaronas: <p>Hi.</p>
<p>It might be a bit difficult to post a snippet of the code as it is quite a big code to be able and reproduce that behaviour. I will try once a finish with the project and I have a stable version.</p>
<p>Thank you.</p>
Loss get stucks in one epoch

887574002: <p>Hi, I have a problem with running my code. I have a program that sometimes it gets stuck in some epochs and dose not procced anymore. How can I understand where the problem is that I can fix it. The problem is that it happens sometimes not always. Any help is really appreciated.</p>
<p>Thank you in advance.</p>

prateekgupta891: <p>Are you monitoring the loss function? my possible guess would be - loss becoming NaN.<br>
What do you mean by stuck though, the training stops or something else?<br>
Please post code and output screenshots to understand your issue.</p>

887574002: <p>Hi, if you mean by monitoring loss function, printing it. Yes, exactly. It’s something I am doing. My mean by getting stuck at some epoch is that I do not see any loss is printed anymore while it seems that the code is running. I did different experiments, and I noticed that it dose not happen in any certain epoch for example in the following experiment the loss got stuck in epoch 7 and it did not get ride of this step, until I had to restart the kernel</p>
<pre><code class="lang-auto">Epoch: 0 Train_Loss: 2.3234, Val_Loss: 1.8912
Epoch: 1 Train_Loss: 1.2648, Val_Loss: 1.7110
Epoch: 2 Train_Loss: 1.1671, Val_Loss: 1.6037
Epoch: 3 Train_Loss: 1.1116, Val_Loss: 1.5056
Epoch: 4 Train_Loss: 1.2287, Val_Loss: 1.6071
Epoch: 5 Train_Loss: 1.3418, Val_Loss: 1.6923
Epoch: 6 Train_Loss: 1.2899, Val_Loss: 1.7839
Epoch: 7 Train_Loss: 1.4315, Val_Loss: 1.7068
</code></pre>
<p>In another experiment, it happened at the last epoch. I mean epoch 39 in 40 epoch running. Here you can see the results.</p>
<pre><code class="lang-auto">Epoch: 13 Train_Loss: 1.4405, Val_Loss: 1.8422
Epoch: 14 Train_Loss: 1.5998, Val_Loss: 2.1091
Epoch: 15 Train_Loss: 1.8484, Val_Loss: 2.0408
Epoch: 16 Train_Loss: 1.7372, Val_Loss: 2.0536
Epoch: 17 Train_Loss: 1.6792, Val_Loss: 2.3075
Epoch: 18 Train_Loss: 1.6514, Val_Loss: 1.9093
Epoch: 19 Train_Loss: 1.7062, Val_Loss: 1.8253
Epoch: 20 Train_Loss: 1.8381, Val_Loss: 2.0517
Epoch: 21 Train_Loss: 1.9149, Val_Loss: 2.0542
Epoch: 22 Train_Loss: 2.0586, Val_Loss: 2.4727
Epoch: 23 Train_Loss: 2.0700, Val_Loss: 2.3520
Epoch: 24 Train_Loss: 2.1251, Val_Loss: 2.1675
Epoch: 25 Train_Loss: 2.2762, Val_Loss: 2.1867
Epoch: 26 Train_Loss: 2.2843, Val_Loss: 2.5865
Epoch: 27 Train_Loss: 2.3018, Val_Loss: 2.3153
Epoch: 28 Train_Loss: 2.3720, Val_Loss: 2.4467
Epoch: 29 Train_Loss: 2.2131, Val_Loss: 2.2533
Epoch: 30 Train_Loss: 2.2144, Val_Loss: 2.6644
Epoch: 31 Train_Loss: 2.2345, Val_Loss: 2.3036
Epoch: 32 Train_Loss: 2.0849, Val_Loss: 2.2778
Epoch: 33 Train_Loss: 2.1355, Val_Loss: 2.9235
Epoch: 34 Train_Loss: 2.4909, Val_Loss: 2.5539
Epoch: 35 Train_Loss: 2.5277, Val_Loss: 2.3989
Epoch: 36 Train_Loss: 2.6098, Val_Loss: 2.6083
Epoch: 37 Train_Loss: 2.6496, Val_Loss: 2.5968
Epoch: 38 Train_Loss: 2.5559, Val_Loss: 2.6308
</code></pre>
<p>The code is so huge that I can not send it here. Just it is an unsupervised learning and I am working with 3d medical images. So, the batch size here is as small as 2. So, my colleague told that I should have expected to see some fluctuations in loss but not actually getting stuck.</p>
Avoid 'Nan' while computing torch.log(torch.exp)

SaminYeasar: <p>If I have a loss function is the form  <code>torch.log(-B*torch.exp(X))</code>  what should be the best way to tackle the   <code>torch.exp</code>   and  <code>torch.log</code>   from getting nan.</p>

user_123454321: <p>I am assuming X is a real tensor…<br>
What is the value of B? If it is positive, you will get nan…<br>
However, if it is negative, you can do</p>
<pre><code class="lang-auto">torch.log(torch.tensor(-B)) + X # since torch.log(torch.exp(X)) =&gt; X
</code></pre>
<p>to avoid really high values in exp…</p>

SaminYeasar: <p>the actual computation is <code>log \mathbf{E} \Big[  -B*torch.exp(X)  \Big] = torch.log( torch.mean (  -B*torch.exp(X)  )) </code>.</p>
<p>And yes both <code>B</code> and <code>X</code> are tensors and output of two different neural networks. ex: <code>B = NN_1(b)+some_additional_calculation</code> and <code>X = NN_2(x)+some_additional_calculation</code></p>

googlebot: <p>logsumexp exists to tackle this case using identity:</p>
<p>log(exp(a)+exp(b)) = c + log(exp(a-c) + exp(b-c))<br>
c=max(a,b)</p>
<p>You can adapt this for scaling and mean with: K*exp(a) = exp(log(K))*exp(a) = exp(a+log(K))<br>
Or just use .clamp() on problematic tensors.</p>

SaminYeasar: <p>Thanks a lot for pointing that out!</p>
CUDA out of memory error for training model with HDF5 dataset

samra-irshad: <p>Hi I am trying to train my model with <code>HDF5</code> dataset. I pass this dataset to <code>torch.utils.data.Dataset</code> and then use DataLoader to feed samples to my model. The model works fine if I dont use HDF5 but that’s extremely slow so I modified my training pipeling to acquire samples from HDF5. Using HDF5, I face another issue of CUDA out of memory. Note that I am using 2 GPUs to train my model, the total batch size is 8 (thats divided between 2 GPUs) in training via <code>nn.DataParallel</code>. I have tried making the number of workers as zero to see if that helps but that does not help. Also, I have made two HDF5 files (one for images, other for labels). My application is image segmentation. I do not understand if the model works fine without HDF5 then why does it shows error for HDF5? I also checked with reading HDF5 files in get_item in <code>torch.utils.data.Dataset</code> to see if the error is because of opening files before get_item() but that does not help as well.<br>
Out of memory error comes after three training steps.</p>