akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
SlowFast-Architecture on FPGA - Conversion from .pkl-File to .pth-File

Isabell20: <p>Hello,</p>
<p>does anybody know how to convert a model given as .pkl-file into .pth?</p>
<p>in my master thesis I’m working on a solution to deploy a SlowFast-Network for human action recognition on an embedded platform with an FPGA. I use the open source codebase PySlowFast and have tested the inference on GPU already (just to see how it works). My embedded platform has an FPGA from Xilinx and I want to use Vitis AI from Xilinx for the quantization and compilation of the model.</p>
<p>The Vitis AI - quantizer needs a pre-trained PyTorch model, generally a .pth-file and a float model definition as input. The SlowFast-Network is saved as a .pkl-file. Does anybody know how I can get a .pth-file and a float model definition from this .pkl-file respectively from the PySlowFast codebase?</p>
<p>For example detectron offers a python skript to convert a model (.yaml and .pkl) to a model.pb. Do you maybe know a similar solution for PyTorch-Models?</p>
<p>Many thanks!</p>

roman.spb: <p>Hello,<br>
I would be very sceptical of defining format from the extension. I myself saw lot’s of different ways of saving pytorch models of the same format with different extensions (though pytorch’s documentations suggests conventinal way).</p>
<p>Not 100% sure but it may end up that .pth and .pkl are refering to the same format.</p>

Isabell20: <p>Hello Roman,</p>
<p>thanks for your reply. So you think I can maybe use the pkl file as input for the quantizer? Or do you think that I just need to re-save the file as pth? I hope I understood you correctly as I’m new to this topic. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>Thank you</p>
Stack torch vector as tensor's entry

WoodezXJ: <p>Hello, say I have</p>
<pre><code class="lang-auto">A = torch.rand(vec_len,1)
B = torch.rand(vec_len,1)
C = torch.rand(vec_len,1)
</code></pre>
<p>and I want to create a tensor <code>T</code>, the shape of which is <code>(vec_len,3,3)</code> and its <code>i_th</code> channel would be like this:</p>
<pre><code class="lang-auto">[[Ai, 0,  0],
 [0,  0, Bi],
 [0,  Ci, 0]]
</code></pre>
<p>How can I fast create such a tensor? Thanks!</p>

thecho7: <pre><code class="lang-auto">vec_len = 5

a = torch.rand(vec_len, 1)
b = torch.rand(vec_len, 1)
c = torch.rand(vec_len, 1)

aa = torch.tile(a, (1, 3))
bb = torch.tile(b, (1, 3))
cc = torch.tile(c, (1, 3))

index_tensor = torch.Tensor([[1, 0, 0], [0, 0, 1], [0, 1, 0]])
index_tensor_ = torch.tile(index_tensor.view(-1, 3, 3), (vec_len, 1, 1))

stacked = torch.stack((aa, bb, cc), dim=2).type(torch.double)
result = torch.where(index_tensor_ &gt; 0, stacked, 0.)
</code></pre>
<p>This is the best among what I know<br>
Any idea?</p>

anantguptadbl: <p><a class="mention" href="/u/woodezxj">@WoodezXJ</a> <a class="mention" href="/u/thecho7">@thecho7</a></p>
<pre><code class="lang-auto">vec_len = 5
A = torch.rand(vec_len)
B = torch.rand(vec_len)
C = torch.rand(vec_len)

final = torch.zeros(vec_len, 3, 3)
final[:, 0, 0] = A
final[:, 1, 2] = B
final[:, 2, 1] = C

final
</code></pre>

WoodezXJ: <p>Thanks, guess replacing the entry of a zero tensor might be the fastest way…</p>
Appending a tensor of high dimension to create training set

sparshgarg23: <p>Hi,<br>
I have an image of dimension 1024x1024 which after flatteining comes out to be a 1048576x1 .<br>
Apart from this I have a json file containing the coordinate information(3x1).In all there are 17,000 entries in that json file.<br>
As such,I am appending the 3x1 vector to the flattened image ,this will create one sample of dimension 1048579x1<br>
The final dataset dimension will be of 1048579x17,000.</p>
<p>Is there an efficient way to create this and load it appropriately in pytorch</p>
<p>I tried reducing the size of the image to 256x256,but my current code leads to a memory issue(even when it’s run on colab pro).</p>
<p>My current code (which was written in numpy is as follows)</p>
<pre><code class="lang-auto">import json
import cv2
import numpy as np



image=cv2.imread('blender_files/Model_0/image_0/model_0_0.jpg',0)
shape=image.shape
image_1=cv2.resize(image,(256,256))
flat_img=image_1.ravel()
print(flat_img.shape)
with open('blender_files/Model_0/image_0/vertices_0_0.json') as f:
    json1_str = f.read()
    json1_data=json.loads(json1_str)
print(len(json1_data))
local_coordinates=[]

for i in range(len(json1_data)):
    local_coord=json1_data[i]['local_coordinate']
    local_coord=np.array(local_coord)
    new_arr=np.hstack((flat_img,local_coord))
    new_arr=new_arr.tolist()
    local_coordinates.append(new_arr)
#local_coords=np.array(json1_data['local_coordinate'])
#print(local_coords.shape)
#new_arr=np.hstack((flat_img,local_coords))
print(new_arr.shape)
local_coordinates=np.array(local_coordinates)
print(local_coordinates.shape)

</code></pre>

thecho7: <p>In my experience, it is not good approach to stack a such large tensor at once.<br>
I recommend you to store (256x256 + 3, 1) size of tensor which indicates a single data.</p>
<p>Storing each data points separately has an advantage of easy managing and fast loading and so forth.<br>
Think about why such a huge datasets (well-known open datasets) manages all data points(images) separately.</p>

sparshgarg23: <p>So just to rephrase you are saying,that once I get that tensor X,I should simply save it in a npy file .And thanks for the comment,in hindsight I think that makes more sense.<br>
And hence my final dataset will contain about 17,000 npys right?</p>
<p>Also,is it better to downsample the image from 1024 to 256,because I feel that during training having a 1048579 vector might be a problem</p>

thecho7: <p>Exactly.</p>
<p>And downsampling is up to your model. However, I haven’t ever seen 1048579 vector for training.<br>
I recommend you to use other models which utilize 3x1 vector effectively. They perhaps have a name of ‘Conditional ~~’ something.</p>
UNET cant make good prediction on images with different histogram

thalia27: <p>Hello all,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/9680db7a447931edac4f68bdf97829b61822df6d.png" alt="image" data-base62-sha1="ltpH9W09B114qnj5sUrsXhzF1sp" width="613" height="139"></p>
<p>I have been training my model several times with Unet. Unfortunately, it seemed like the model can only make good prediction on images with certain histogram even after i added both type of images.<br>
I made sure I have 50:50 dataset of both histogram. I applied also dropout layer to the architecture.</p>
<p>Best regards<br>
Thalia</p>

ptrblck: <p>Unsure if this might help, but did you try to augment the images by manipulating their histograms?</p>

thalia27: <p>Hi, yes I did with cv2.convertScaleAbs(), and I also played on brightness and contrast. But it seemed that the algorithm can not work well on dark and light images. So now we have one model that can work well on dark images and another model that can work well on bright images.</p>
CUDA out of memory for a tiny network

cpp: <p>I have a tiny CNN model that gives this summary:</p>
<pre><code class="lang-auto">----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 32, 222, 222]             896
              ReLU-2         [-1, 32, 222, 222]               0
         MaxPool2d-3         [-1, 32, 111, 111]               0
            Conv2d-4         [-1, 32, 109, 109]           9,248
              ReLU-5         [-1, 32, 109, 109]               0
         MaxPool2d-6           [-1, 32, 54, 54]               0
            Conv2d-7           [-1, 32, 52, 52]           9,248
              ReLU-8           [-1, 32, 52, 52]               0
         MaxPool2d-9           [-1, 32, 26, 26]               0
          Flatten-10                [-1, 21632]               0
           Linear-11                    [-1, 1]          21,633
          Sigmoid-12                    [-1, 1]               0
================================================================
Total params: 41,025
Trainable params: 41,025
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.57
Forward/backward pass size (MB): 35.24
Params size (MB): 0.16
Estimated Total Size (MB): 35.97
----------------------------------------------------------------
</code></pre>
<p>The input images are <code>3x224x224</code> and the batch size is <code>16</code>. When I start training the model (with <code>torch.optim.SGD</code>), I get this:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 11.89 GiB (GPU 0; 8.00 GiB total capacity; 1.14 GiB already allocated; 4.83 GiB free; 1.14 GiB reserved in total by PyTorch)
</code></pre>
<p>Why does it try to allocate so much memory for such a small network? Thanks.</p>

cpp: <p>I found the bug which was trivial: I was loading all the validation images into a single batch which was too big. I would delete this post but don’t see any option to do that.</p>

thecho7: <p>Click the ‘show more’ button on the right bottom of your post. Then you can see a trash can.</p>
Modules defined in __init__() but not used in forward() still affect the results

dawnlh: <p>I have defined a <strong>channel attention</strong> block in the <strong>init</strong>, but didn’t use it in the <strong>forward</strong> function.  However, the network performance still improved a lot. I’m confused. Any suggestions? Thanks!</p>
<pre><code class="lang-auto">class ImgFeatMSNet2(nn.Module):
    '''
    image and feature space multi-scale deconvolution network: multiscale image do Wiener separately and merge in refine net
    '''

    def __init__(self, n_colors, channel_atten=True):
        super(ImgFeatMSNet2, self).__init__()
        # params
        self.channel_atten = channel_atten
        # self.n_colors = n_colors
        n_resblock = 3
        n_feats1 = 16
        n_feats = 32
        kernel_size = 5
        padding = 2

        self.FeatModule1 = FeatModule(
            n_in=n_colors, n_feats=n_feats1, kernel_size=kernel_size, padding=padding, act=True)

        self.InBlock1 = InBlock(n_in=n_feats1 + n_colors, n_feats=n_feats,
                                kernel_size=kernel_size, padding=padding, act=True)

        if channel_atten:
            self.ChannelAttention1 = ChannelAttention(
                in_planes=n_feats1+n_colors, ratio=2)
            self.ChannelAttention2 = ChannelAttention(
                in_planes=n_feats1 + n_feats+n_colors, ratio=4)

        self.RefineUnet = RefineUnet(
            n_feats=n_feats, n_resblock=n_resblock, kernel_size=kernel_size, padding=padding, act=True)

        self.OutBlock_a = OutBlock_a(
            n_feats=n_feats, n_resblock=n_resblock, kernel_size=kernel_size, padding=padding)
        self.OutBlock_b = OutBlock_b(
            n_feats=n_feats, n_out=n_colors, kernel_size=kernel_size, padding=padding)

    def forward(self, input):
        # scale1: x0.5; scale1: x1.0
        n, c, h, w = input.shape

        feat_2 = self.FeatModule1(input)  # get image feature
        wd_1 = torch.cat((feat_2, input), 1)  # cat image and feature


        # The following codes are not used
        # if self.channel_atten:
        #     wd_1 = self.ChannelAttention1(wd_1)
        
        in_1 = self.InBlock1(wd_1)
        refine_1 = self.RefineUnet(in_1, int(round(h/2)), int(round(w/2)))
        out_a1 = self.OutBlock_a(refine_1)
        out_b1 = self.OutBlock_b(out_a1)

        return out_b1

</code></pre>

ptrblck: <p>Creating an additional layer would initialize all parameters (assuming this layer contains parameters) by sampling random numbers through the pseudorandom number generator (PRNG). This would affect all future calls of the PRNG and would not yield the same results even if the additional layer is not used in the <code>forward</code>. Changing the seed would have a similar effect and if your model is sensitive to the seed it would mean that the overall training is not particularly stable.</p>

dawnlh: <p>Thanks for your answer! I will try to use a different initialization method and check the result.</p>
Design pattern for data augmentation

NicoAdamo: <p>Hi all, a bit of a basic question here - I am trying to work with a somewhat small dataset, and apply augmentation techniques to increase my training set size. My DataLoader is currently very basic, of the form:</p>
<pre><code class="lang-auto">class Dataset(torch.utils.data.Dataset):
  
  def __init__(self, images_path):
    super(Dataset, self).__init__()
    self.images_path = images_path
  
  def __len__(self):
    return len(self.images_path)

  def __getitem__(self, idx):
    img_path = self.image_paths[idx]
    img = cv2.imread(str(img_path))
    return img
</code></pre>
<p>This Dataset doesn’t really suit itself well to augmentation, as images are iterated / specified by a specific index, whereas I want one single image to give rise to multiple training examples with different augmentations. I’m wondering what the usual solution in Pytorch is for this type of augmentation.</p>
<p><code>torchvisions.transforms</code> is very well documented, but it talks about taking training samples and applying random transformations to them, rather than generating multiple samples. I always could do it manually, by applying random transformations to the batch multiple times at training time, but this seems ugly, and fails to spread out the same sample with different augmentations through different batches - is there an easy, Pytorchy solution for multiple training samples from a single image w/ augmentation?</p>

ptrblck: <p>You could transform the <code>img</code> multiple times and just return all of them. Inside the <code>DataLoader</code> loop you could then reshape the additional images to the batch dimension and reduce the <code>batch_size</code> of the <code>DataLoader</code> by the same factor.</p>
Hi. Can anyone help me with this? sm_86 is not compatible with current pytorch installation

Prabhjot_Kaur: <p>.local/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py:143: UserWarning:<br>
NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.<br>
If you want to use the NVIDIA GeForce RTX 3090 GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>

ptrblck: <p>You’ve most likely installed the pip wheel or conda binary with the CUDA 10.2 runtime, which is not compatible with your GPU. Install the CUDA 11.3 (or 11.5 nightly) binary and it should work.</p>
Can I get the CUDA tensor pointer in the python pytorch

Jakub_Mitura: <p>Hello I have device tensors defined in the pyton script - can I get the pointer to the begining of the array</p>

huahuanZ: <pre><code class="lang-auto">// get the pointer of variable anytensor (cast to float type, this is up to your data type)
auto c_pointer = anytensor.data_ptr&lt;float&gt;();
</code></pre>

Jakub_Mitura: <p>Thanks !, but can I do it from python script ?</p>

huahuanZ: <p>AFAIK, there is no pointer data type in python. What do you want it for?</p>

Jakub_Mitura: <p>Frankly I want to pass it into Julia code through python - I created Julia library that uses CUDA acceleration for medical segmentation metrics - In Julia I need data on GPU it is done by the CUDA array - which is basically list of pointers. Julia is created for interoperability with python, so I can easily call Julia from python but it will copy data on CPU and than on GPU. In order to get around it I would like to have access to a pointer in memory - and then I would probably be able to parse it into Julia Array.</p>

ptrblck: <p>I don’t know if this works with Julia, but you could try to cast the <code>x.data_ptr()</code> to a void pointer via <code>ctypes.c_void_p</code> and pass it round.</p>
NLP: How to look at sentence encoding in torchtext?

t326wang: <pre><code class="lang-python">TEXT = data.Field(tokenize = 'spacy',
                  tokenizer_language = 'en_core_web_sm',
                  include_lengths = True)

LABEL = data.LabelField(dtype = torch.float)

train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)
train_data, valid_data = train_data.split(random_state = random.seed(SEED))

MAX_VOCAB_SIZE = 25_000

TEXT.build_vocab(train_data, 
                 max_size = MAX_VOCAB_SIZE, 
                 vectors = "glove.6B.100d", 
                 unk_init = torch.Tensor.normal_)
LABEL.build_vocab(train_data)
</code></pre>
<p>Suppose I have built vocabulary from training data like above. Now I want to take a look of the one-hot encoding of the sentences in the training data, how should I do that? (I know iterator will automatically give the encoded and padded sentence, but I just want to take a look of what the encoding look like).</p>
<p>Thanks in advance!</p>
Got “Trying to backward through the graph a second time”

Ximeng: <p>Hi,</p>
<p>I’ve seen other people got this error using loop-like operations and set retain_graph=True can avoid it somehow, but I am still confused how I get this.</p>
<p>class Smoother(nn.Module):<br>
def <strong>init</strong>(self, in_dim, out_dim, k, device):<br>
super().<strong>init</strong>()<br>
assert k % 2 == 1<br>
self.base = torch.linspace(-1, 1, steps=k, device=device, requires_grad=False).float()<br>
self.alpha = torch.ones(1, device=device, requires_grad=True).float()<br>
self.beta = torch.zeros(1, device=device, requires_grad=True).float()<br>
self.theta = torch.ones(1, device=device, requires_grad=True).float()/(k/2)<br>
self.omega = torch.zeros(1, device=device, requires_grad=True).float()<br>
self.weight = self.base*self.alpha+self.beta<br>
self.weight = torch.sigmoid(self.weight)<br>
self.weight = self.weight * self.theta + self.omega<br>
self.weight = self.weight.expand(out_dim, in_dim, k)<br>
p = (k-1)//2<br>
self.padding = nn.ReplicationPad1d((2 * p, 0))<br>
def forward(self, x):<br>
# b,c,l<br>
x = self.padding(x)<br>
x = F.conv1d(x, self.weight)<br>
return x</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self, device):<br>
super(Net, self).<strong>init</strong>()<br>
self.sm = Smoother(in_dim=16, out_dim=16, k=129, device=device)<br>
def forward(self, x):<br>
x = self.sm(x)<br>
return x<br>
if <strong>name</strong> == ‘<strong>main</strong>’:<br>
device=torch.device(‘cuda:0’)<br>
m = Net(device)<br>
a = torch.rand((3, 16, 512)).float().cuda()<br>
b = m(a)<br>
loss=…</p>
<p>I thought I only initialized the function only once. Is that because of the “expand” operation？<br>
Thanks!</p>
<p>Best regards</p>
Multiple Target output architecture efficiency

pumplerod: <p>While developing a model intended for training to optimize multiple targets at once I realized I could construct the output a couple of different ways and I’m curious if they are equivalent in terms of gradient calculation or if one may be better than the other it terms of convergence…</p>
<p>Assume, for simplicity I have 2 targets.</p>
<p>Option 1:</p>
<pre><code class="lang-auto">class MyModel( torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.input_layer = torch.nn.Linear( 1024, 16)
        self.out_layer_1 = torch.nn.Linear( 16, 1)
        self.out_layer_1 = torch.nn.Linear( 16, 1)

    def forward(self, feature_data):
        X = self.input_layer( feature_data)
        Out_1 = self.out_layer_1( X)
        Out_2 = self.out_layer_2( X)
        return (Out_1, Out_2)
</code></pre>
<p>Option 2:</p>
<pre><code class="lang-auto">class MyModel( torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.input_layer = torch.nn.Linear( 1024, 16)
        self.out_layer = torch.nn.Linear( 16, 2)

    def forward(self, feature_data):
        X = self.input_layer( feature_data)
        Out = self.out_layer( X)

        return Out
</code></pre>
<p>Obviously, with option 2 I would need to slice the output for my results.  But I’m wondering if this makes a difference with regard to the optimizer and the backpropagation of loss.</p>

thecho7: <p>I think you want to use somewhat regression loss.<br>
Then, they are the same in the aspect of gradient.</p>
<p>I think the first option is much better than the second one because you can apply the different optimizer and learning rate to each targets with the first option.</p>
Best practices for model performance assessment

witl: <p>Let’s say we have two deep learning methods/models (trained in a stochastic fashion) and one dataset that we want to assess the performance of both methods to decide which performs the best.</p>
<p>Since the optimization of the models is stochastic, using the results achieved given one experiment (training just once) each model is not reliable. In this sense, each model should be trained <code>n</code> times, and then the results can be assessed statistically.</p>
<p>In this sense:</p>
<ol>
<li>What are the best practices that I should follow when performing this kind of comparison? Does the community follow any standard? For example, how many times do people train the model, i.e. what is a proper value to <code>n</code>?</li>
<li>Given multiple results of the same model on a given dataset, which metrics are commonly used? Mean, standard deviation, margin of error/confidence interval 95% (or statistical significance of 5%)?</li>
<li>I found in some papers that the results presented in the tables are followed by the <code>±</code> sign, but they don’t describe it in the text. Does it represent the confidence interval of 95%? i.e., mean ± margin of error</li>
</ol>
<p>I’m not sure if this type of question is adequate for this forum specifically, but this is the only forum I read/use outside papers. I would appreciate the recommendation if you guys are aware of any other community that this would be more adequate.</p>

ptrblck: <p>CC <a class="mention" href="/u/rwightman">@rwightman</a> who has been training lots of state-of-the-art CNNs in his <a href="https://github.com/rwightman/pytorch-image-models">timm repository</a> and might want to share his standard approach of evaluating the model accuracy.</p>
Pruning is NOT deterministic

thyeros: <p>I was running a training using DDP. When the pytorch pruning lib is used, I found that there is very little inconsistency in the prune mask density. I used global_unstructured. Is it supposed to be non-deterministic? perhaps due to the floating point operation? As long as DDP ensures that weights are always in sync, the pruning masks should be identical as well all the time?</p>

anantguptadbl: <p><a class="mention" href="/u/thyeros">@thyeros</a> Ideally pruning on the same weight and bias array, should result the same masks as the L1 norms would be the same (in case of L1Unstructured)</p>
<p>Can you check whether the weights are indeed the same on all machines before you execute the pruning call</p>

thyeros: <p>I found that the weights before pruning were indeed off sync due to the index_add op I used to massage them. Thanks for advise.</p>
Gradient backpropagation for a custom tensor

mamunm: <p>I have an NN architecture for molecular property prediction. The feature vectors for each atom go through its own MLP (based on atomic identity, i.e., atomic number). Then they are summed together to get the molecular energy. I use <code>ModuleDict</code> to keep track of those MLP. For molecular energies, I initialize an empty tensor with <code>requires_grad = True</code>, and then I populate the tensor going through each atom. My question is will it cause problems during backpropagation as pytorch doesn’t allow in-place operation (I use <code>.data</code> to bypass the problem but am not sure about the caveat of such bypassing). Below is a code snippet of the part that I am confused about,</p>
<pre><code class="lang-auto">atomic_energies = torch.zeros((self.batch_size, max_n_atoms),
            device=self.device, dtype=torch.float, requires_grad=True)
        
 for i in range(self.batch_size):
       for j in range(n_atoms):
            atomic_energies.data[i][j] = self.atomic_mlp_dict[species[i][j].item()](feature_matrix[i][j])
</code></pre>
<p>How should I verify that the gradients are being propagated correctly? Currently, the loss doesn’t change significantly for 10-20 epochs. How to debug this kind of problem in Pytorch? Any help is highly appreciated.</p>

anantguptadbl: <p><a class="mention" href="/u/mamunm">@mamunm</a></p>
<p>You can monitor the grads by applying hooks on the tensor. These hooks are triggered during the backward propagation</p>
<p>If the grad values are not changing then it can mean that the loss is not connected to the parameter that you are monitoring, or that the model is already at the maximum level of convergence</p>
<pre><code class="lang-auto">import numpy as np
import time
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable

class random_model(nn.Module):
    def __init__(self, num_layers):
        super(random_model, self).__init__()
        self.layer1 = nn.Sequential(nn.Linear(100, 20), nn.BatchNorm1d(20), nn.ReLU())
        self.layer2 = nn.Linear(20, 1)
    
    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        return x
    
model = random_model(10)
loss = torch.nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

grad_hook = model.layer2.weight.register_hook(lambda grad: print("Grad is {0}".format(grad)))

X = torch.rand(100, 100)
y = torch.rand(100)

for cur_epoch in range(100):
    optimizer.zero_grad()
    output = model(X)
    cur_loss = loss(output, y)
    cur_loss.backward()
    optimizer.step()
    print("Epoch {0} Loss is {1}".format(cur_epoch, cur_loss.item()))
</code></pre>

googlebot: <p>such use of .data doesn’t support gradient flow, normally you have to use torch.stack (or cat) for such merging</p>

mamunm: <p><a class="mention" href="/u/anantguptadbl">@anantguptadbl</a> Thanks for the suggestion. I tried it and you’re right the weight is not changing. I need to modify the code using <a class="mention" href="/u/googlebot">@googlebot</a>’s suggestion.</p>
Problems with torch.multiprocessing on CPU

Imahn: <p>Unfortunately, for quite some time now, I have encountered problems with the module <code>torch.multiprocessing</code>. As a MWE, I am trying to square a PyTorch tensor on CPU, which does not work:</p>
<pre><code class="lang-python">import torch
import numpy as np
import torch.multiprocessing as mp


def square(i, x, queue):
    print('In process {}'.format(i))
    if isinstance(x, np.ndarray):
        queue.put(np.square(x))
    else:
        queue.put(torch.square(x))


if __name__ == '__main__':
    mp.set_start_method('spawn')
    print('Number of available CPUs: {}'.format(mp.cpu_count()))

    processes = []
    queue = mp.Queue()

    # Set up some sample data:
    x = np.arange(64)

    # Start &lt;mp.cpu_count()&gt; processes with the square function as the target and an invidual piece of data to process:
    for i in range(mp.cpu_count()):
        start_index = mp.cpu_count()*i
        proc = mp.Process(target = square, args = (i, x[start_index:start_index+mp.cpu_count()], queue))
        proc.start()
        processes.append(proc)

    # Wait for each process to finish before returning to the main thread:
    for proc in processes:
        proc.join()

    # Terminate each process:
    for proc in processes:
        proc.terminate()

    # Convert the multiprocessing queue into a list:
    results = []
    while not queue.empty():
        results.append(queue.get())

    print(results)
</code></pre>
<p>This results in the following error message (Python version 3.8.9, PyTorch version 1.9.0):</p>
<pre><code class="lang-python">Traceback (most recent call last):
  File "parallelization_sampling.py", line 55, in &lt;module&gt;
    results.append(queue.get())
  File "/..../lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
  File "/..../lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 289, in rebuild_storage_fd
    fd = df.detach()
  File "/..../lib/python3.8/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/.../lib/python3.8/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/.../lib/python3.8/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/.../lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
</code></pre>
<p>However, when I import the module <code>multiprocessing</code> instead of <code>torch.multiprocessing</code> and use <code>x = np.arange(64)</code> instead of <code>x = torch.arange(64)</code>, the script works fine! I tested the code on two machines, can anybody confirm this weird behavior? And does anybody see what I am doing wrongly? Or is <code>torch.multiprocessing</code> not supposed to be used on CPU?</p>
Auxiliary loss in Fully Convolutional Network

Andrea_Rosasco: <p>Hello,<br>
I noticed that the fully convolutional network present in <code>torchvision</code> has an auxiliary head.<br>
I thought <code>torchvision.models.fcn_resnet50/101</code> used the standard resnet architecture but in the original resnet paper the authors don’t mention using any auxiliary loss.</p>
<p>Is the resnet implemented in torchvision a custom one? Or is it based on a subsequent publication? Is there a paper explaining the differences with the standard architecture?</p>
<p>Thank you very much</p>
Is torch.save() an indivisible transaction?

Sam_Lerman: <p>If I call <code>torch.save</code> and terminate mid-save, will the transaction be fully cleared, and any pre-existing save will be preserved and not over-written?</p>
<p>I’m getting an error when loading a saved <code>state_dict</code> after a training session that terminated mid-save, so it seems like the transaction occurs partially and yields a corrupted checkpoint.</p>

Sam_Lerman: <p>Is it because I’m also saving some <code>int</code> values and loading with <code>map_location</code>?</p>

anantguptadbl: <p><a class="mention" href="/u/sam_lerman">@Sam_Lerman</a> torch.save internally uses python pickling to write the objects to the file. Pickling is not a single operation but there are multiple writes of various things like : Some pickling headers, various child objects present in the object you want to pickle. And if you are overwriting on the same location, you have opened those files in write mode so data will be lost, if it is interrupted mid way</p>
<p>If you want atomic operations during write, you need to use a temp location and transfer the file only on the success of the torch.save operation</p>
How can I modify the BERT embedding Module in a specific way?

Philipp_Friebertshau: <p>Hi,</p>
<p>I want to modify the BERT embedding Module (from pytorch-transformers package) in the way that I can dynamically add a tensor ontop of the tokenembedding.  The tensor contains additionally information that can’t be stored in an embedding layer, because it isnt dependend on the tokenposition or the token_id/vocab. I think the picture attached in the appendix describes my interest quite well (in red writing it is clarified how I would like to modify it).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a0ed8b1e3e8fe7ea853588c4b9e6946066312575.png" data-download-href="https://discuss.pytorch.org/uploads/default/a0ed8b1e3e8fe7ea853588c4b9e6946066312575" title="modify"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a0ed8b1e3e8fe7ea853588c4b9e6946066312575.png" alt="modify" data-base62-sha1="mXDkrss3SpA7jPkraP6pFZ7O49T" width="690" height="453" data-dominant-color="F2F1F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">modify</span><span class="informations">888×584 17.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Li_yingting: <p>Hi, did u settle this? I also want to add some additional info.</p>
Official DDP example is broken

Olivier-CR: <p>Hi,</p>
<p>I’m trying this example code with PT1.8.1 <a href="https://pytorch.org/docs/stable/notes/ddp.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch 1.10.1 documentation</a><br>
I’m getting a <code>ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable MASTER_ADDR expected, but not set</code></p>
<p>What is wrong?</p>

rvarm1: <p>Thanks for bringing this up!</p>
<p>The issue is due to the docs, we need to set env variables MASTER_ADDR and MASTER_PORT to use the default env:// initialization correctly.</p>
<p>Fixing it in <a href="https://github.com/pytorch/pytorch/pull/71363" class="inline-onebox" rel="noopener nofollow ugc">[Docs][BE] DDP doc fix by rohan-varma · Pull Request #71363 · pytorch/pytorch · GitHub</a>.</p>
What are the main possible reasons for this error? Segmentation fault (core dumped)

gebrahimi: <p>Hi,</p>
<p>My pytorch training code stops running without any error and the command prompt shows this: Segmentation fault (core dumped).</p>
<p>I have no idea how to debug this? any suggestions? What are the main reasons for this?</p>
<p>My code is really long and related to my research and I can’t share it here, however, i can share it if someone is willing to take a look into it.</p>
<p>Thanks.</p>
Understanding the Loss Value (BCEWithLogitsLoss)

HallerPatrick: <p>Hello,</p>
<p>I am a little confused by what a Loss function produces.</p>
<p>I was looking at this post: <a href="https://discuss.pytorch.org/t/multi-label-classification-in-pytorch/905/45" class="inline-onebox">Multi Label Classification in pytorch - #45 by ptrblck</a></p>
<p>And tried to recreate it to understand the loss value calculated. So I constructed a perfect output for a given target:</p>
<pre><code class="lang-python">from torch.nn.modules.loss import BCEWithLogitsLoss

loss_function = BCEWithLogitsLoss()

# Given are 2 classes
output_tensor = Tensor([[0.0, 1.0]]) # Output of my nn
target_tensor = Tensor([[0.0, 1.0]]) # Target result which is identical to output


loss = loss_function(output_tensor, target_tensor)
# Shouldn't this yield no loss at all?
print(loss.item())

# Output: 0.5032044053077698
</code></pre>
<p>If I understand correctly, then this should yield a loss of <code>0.0</code>, because the target is identical to the output.</p>
<p>But I get: <code>0.5032044053077698</code>.</p>
<p>Did I miss something?</p>
<p>With best regards,<br>
Patrick</p>

ptrblck: <p><code>nn.BCEWithLogitsLoss</code> expects logits which are unbound and have values in <code>[-Inf, +inf]</code> and can be seen as “unnormalized” probabilities (I’m sure <a class="mention" href="/u/tom">@tom</a> or <a class="mention" href="/u/kfrank">@KFrank</a> can give you a proper mathematical definition), so your <code>output_tensor</code> won’t match the targets perfectly. Since your output contains probabilities, use <code>nn.BCELoss</code> instead or low and high values for <code>output_tensor</code> such as <code>torch.tensor([[-1e10, 1e10]])</code> with <code>nn.BCEWithLogitsLoss</code>.</p>
How to update parameters with multiple loss functions

Khah_Nguyen: <p>Hi guys, Im new in pytorch and I have met some problems in updating parameters while training model. For instance, I have 2 loss-functions: <code>loss1</code> and <code>loss2</code>, let <code>loss</code> = <code>loss1 + loss2</code>. The model has <strong>feature extractor layers</strong> and <strong>classifier layers</strong>. And I want to update the parameters in <strong>feature extractor layers</strong> using <code>loss</code> and the parameters in <strong>classifier layers</strong> using <code>loss1</code>. How can I do that in pytorch??? Thanks!</p>

anantguptadbl: <aside class="quote no-group" data-username="Khah_Nguyen" data-post="1" data-topic="141716">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/khah_nguyen/48/36676_2.png" class="avatar"> Khah_Nguyen:</div>
<blockquote>
<p>ew in pytorch and I have met some problems in updating parameters</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/khah_nguyen">@Khah_Nguyen</a> you will have to manually turn on and off requires_grad while doing the backward. This will ensure that when the optimizer step function runs, the appropriate grads are stored in the node</p>
<pre><code class="lang-auto">import numpy as np
import time
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable

class random_model(nn.Module):
    def __init__(self, num_layers):
        super(random_model, self).__init__()
        self.model1 = nn.Sequential(nn.Linear(100, 20), nn.BatchNorm1d(20), nn.ReLU())
        self.model3 = nn.Linear(20, 1)
    
    def forward(self, x):
        x = self.model1(x)
        x = self.model3(x)
        return x
    
model = random_model(10)
loss1 = torch.nn.MSELoss()
loss2 = torch.nn.SmoothL1Loss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

X = torch.rand(100, 100)
y = torch.rand(100)

for cur_epoch in range(100):
    optimizer.zero_grad()
    output = model(X)
    cur_loss1 = loss1(output, y)
    cur_loss2 = loss2(output, y)
    cur_loss = cur_loss1 + cur_loss2
    for param in model.model3.parameters():
        param.requires_grad = False
    
    cur_loss.backward(retain_graph=True)
    for param in model.model1.parameters():
        param.requires_grad = False
    for param in model.model3.parameters():
        param.requires_grad = True
    cur_loss2.backward()
    optimizer.step()
    print("Epoch {0} Loss is {1}".format(cur_epoch, cur_loss.item()))
</code></pre>

Khah_Nguyen: <p>Ok I got it, Tks sir!</p>
AdamW + ZeroRedundancyOptimizer + Weight Decay Dictionary

Cade_Gordon2: <p>I’m wondering if there’s a simple solution to translate:</p>
<pre><code class="lang-python">optimizer = optim.AdamW(
                [
                    {"params": gain_or_bias_params, "weight_decay": 0.},
                    {"params": rest_params, "weight_decay": args.wd},
                ],
                lr=args.lr,
                betas=(args.beta1, args.beta2),
                eps=args.eps,
            )
</code></pre>
<p>Into a <code>ZeroRedundancyOptimizer</code>. The class by default expects an iterable of tensors but I’m passing an iterable of dictionaries which is causing a lot of headaches. I’ve tried a lot of hacky solutions but to no avail do to assertions that occur later on.</p>
<p>The expected:</p>
<pre><code class="lang-python">optimizer = ZeroRedundancyOptimizer(
                [
                    {"params": gain_or_bias_params, "weight_decay": 0.},
                    {"params": rest_params, "weight_decay": args.wd},
                ],
                optim.AdamW,
                lr=args.lr,
                betas=(args.beta1, args.beta2),
                eps=args.eps,
            )
</code></pre>
<p>sadly doesn’t cut it.</p>
<p>Thank you for your time,<br>
Cade</p>

Cade_Gordon2: <p>The trick is parameter groups!</p>
<pre><code class="lang-python">optimizer = ZeroRedundancyOptimizer(
                rest_params,
                optim.AdamW,
                lr=args.lr,
                betas=(args.beta1, args.beta2),
                weight_decay=args.wd,
                eps=args.eps,
            )
optimizer.add_param_group({"params": gain_or_bias_params, "weight_decay": 0.})
</code></pre>

rvarm1: <p>Awesome, glad you found a solution! But it would be ideal for ZeRo to support this natively. I filed an issue as such: <a href="https://github.com/pytorch/pytorch/issues/71347" class="inline-onebox" rel="noopener nofollow ugc">[ZeRo] Parameter group support in constructor · Issue #71347 · pytorch/pytorch · GitHub</a></p>
Image normalization for UNET with resnet pretrained encoder

YASJAY: <p>Working on healthcare image datasets what should be the normalization stats used on these images, imagenet stats or stats from the new dataset being trained on?</p>
<p>Does it even matter which we choose?</p>

ptrblck: <p>I would expect the custom stats to work better especially if the actual data stats differ between the medical images and ImageNet, but would also recommend to test both approaches.</p>
Model gradients printed correctly but stored differently

shohidul: <p>I want to copy the gradients of loss, with respect to weight, for different data samples using pytorch. In the code below, I am iterating one sample each time from the data loader (batch size = 1) and collecting gradients for 1st fully connected (fc1) layer. Gradients should be different for different samples. The print function shows correct gradients, which are different for different samples. But when I store them in a list, I get the same gradients repeatedly.   Any suggestions would be much appreciated. Thanks in advance!</p>
<pre><code class="lang-auto">grad_list = [ ]

for data in test_loader:
  inputs, labels = data[0], data[1]
  inputs = torch.autograd.Variable(inputs)
  labels = torch.autograd.Variable(labels)

  # zero the parameter gradients
  optimizer.zero_grad()

  # forward + backward 
  output = target_model(inputs)
  loss = criterion(output, labels)
  loss.backward()

  grad_list.append(target_model.fc1.weight.grad.data)
  print(target_model.fc1.weight.grad.data)
</code></pre>
<p>Gradients from the print() and list, grad_list, are shown below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37f241bb7beee62a20d0f8026218a1b84c769be7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/37f241bb7beee62a20d0f8026218a1b84c769be7" title="output"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37f241bb7beee62a20d0f8026218a1b84c769be7_2_328x500.jpeg" alt="output" data-base62-sha1="7YVicC7TO5FM3yiT8pg3hxS9t79" width="328" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37f241bb7beee62a20d0f8026218a1b84c769be7_2_328x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37f241bb7beee62a20d0f8026218a1b84c769be7_2_492x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37f241bb7beee62a20d0f8026218a1b84c769be7_2_656x1000.jpeg 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">output</span><span class="informations">904×1378 284 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Don’t use the <code>.data</code> attribute, as it’s deprecated and <code>.clone()</code> the gradient before appending it to the <code>list</code> as otherwise references would be stored.</p>
Question about flow of gradients when changing values

LeanderK: <p>I have the following problem: I have a bunch of parameters that I need to project into a the valid parameter space. I can only efficiently compute the valid parameter-space during the forward-pass (it changes with the parameters).</p>
<p>So I have computations like this:</p>
<pre><code class="lang-auto">lower_bound, upper_bound = compute_bound(input)
param[param &lt; lower_bound] = lower_bound
param[param &gt; upper_bound] = upper_bound
out = f(input, param)
</code></pre>
<p>the thing is…I wonder what that means for the gradients for param? Do they get obstructed by the assignment? Because I just want to replace every value out of bounds with the nearest bound, but still optimise it.</p>

ptrblck: <p>I think your use case should work if you wrap the inplace manipulations of <code>param</code> into a <code>with torch.no_grad()</code> guard as seen here:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torch.nn.functional as F

lower_bound, upper_bound = 0., 1.
param = nn.Parameter(torch.randn(10, 10) * 10.)
optimizer = torch.optim.Adam([param], lr=1e-3)
f = F.linear

for epoch in range(10):
    print(epoch)
    input = torch.randn(10, 10)
    print(param.min(), param.max())
    
    optimizer.zero_grad()
    
    with torch.no_grad():
        param[param &lt; lower_bound] = lower_bound
        param[param &gt; upper_bound] = upper_bound
    print(param.min(), param.max())
    out = f(input, param)
    out.mean().backward()

    optimizer.step()
</code></pre>
<p>Could you check, if this code snippet does what you need?</p>