akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to freeze variable outside of the medel

WoodezXJ: <p>Hello, so I am working on this generative model, and part of the loss requires the output of <code>torch.max(torch.nn.functional.conv2d(perdict_img,filter))</code> with different filters. These filters are supposed to keep the prefixed weight during the training.</p>
<p>I set them as <code>requires_grad=False</code> and then I have this error: <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code>.</p>
<p>Is there any way to make it work without embedding this computation as some fixed layer of the model?</p>

soulitzer: <p>Are there other parameters that are part of your model that you would like to train? If none of your inputs require grad, it wouldn’t make sense to run backward, because the backward graph wasn’t create in the first place.</p>

WoodezXJ: <p>So the output of the model is <code>perdict_img</code> and it contains the parameters I would like to train.</p>

soulitzer: <p>Setting the filter to not require grad in a convolution is fine (see below):</p>
<pre><code class="lang-auto">import torch
a = torch.rand(10, 5, 5, 5, requires_grad=True)
b = torch.rand(5, 5, 2, 2, requires_grad=False)
out = torch.max(torch.nn.functional.conv2d(a, b)) 
out.requires_grad # True
</code></pre>
<p>We’d probably need some more context here. Do you have a small snippet to run that can reproduce the issue?</p>

WoodezXJ: <aside class="quote no-group" data-username="soulitzer" data-post="4" data-topic="125337">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<pre><code class="lang-auto">import torch
a = torch.rand(10, 5, 5, 5, requires_grad=True)
b = torch.rand(5, 5, 2, 2, requires_grad=False)
out = torch.max(torch.nn.functional.conv2d(a, b)) 
out.requires_grad # True
</code></pre>
</blockquote>
</aside>
<p>Thank you for the sample! I went back to check line by line and found it’s another intermediate operation that does not support backpropagation and have it fixed now.</p>
Is there a reason why residual blocks are written with high verbosity?

FeryET: <p>Hi there.</p>
<p>I was trying to implement a simple residual block, and I was looking for other people’s codes.</p>
<p>One thing that really caught my eye is that the residual block while in theory being x = f(x) + x, is implemented usually like this:</p>
<pre><code class="lang-auto">def forward(self, x):
     identity = x
     x = f(x) + identity
     return x
</code></pre>
<p>I was wondering why most are not writing this with the simpler form:</p>
<pre><code class="lang-auto">def forward(self, x):
     x += f(x)
     return x
</code></pre>
<p>The question might sound trivial or superficial, but I suspect maybe there is a deeper reason that just semantics at work here.</p>

soulitzer: <p>Both return the same value, but the difference is that the second form would modify the input in-place and return the same input tensor object. The first form returns a new tensor, which is what we want.</p>
Convert tensor defined on a non-uniform mesh grid to tensor on uniform mesh grid

DrKleiber: <p>Dear all,</p>
<p>I’m having difficulty converting a tensor defined on a non-uniform mesh grid to a tensor on a uniform grid. My problem can be illustrated using the following picture:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30730bd164df713cfb651d408788458a7b22308a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/30730bd164df713cfb651d408788458a7b22308a" title="Picture1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30730bd164df713cfb651d408788458a7b22308a_2_500x500.jpeg" alt="Picture1" data-base62-sha1="6UBp2oXt79E6FZf2zorPbCZZsvM" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30730bd164df713cfb651d408788458a7b22308a_2_500x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30730bd164df713cfb651d408788458a7b22308a_2_750x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30730bd164df713cfb651d408788458a7b22308a_2_1000x1000.jpeg 2x" data-dominant-color="EEEEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Picture1</span><span class="informations">1413×1413 187 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want to convert a tensor created from results on a very fine non-uniform mesh to a coarse uniform mesh, then take the converted tensor as input to a neural network, and get the result with the same size with the input. Then convert back the output tensor to the original fine non-uniform mesh.</p>
<p>I understand I can use <em>grid_sample</em> to convert a tensor defined on a uniform mesh to non-uniform mesh. But is there a viable way for the vice versa?</p>
<p>Any suggestions would be greatly appreciated!</p>
<p>Best,</p>
About DDP with single gpu

Kevin_Cha: <p>Is there issue or performance difference between single gpu with ddp and single gpu in single process?</p>

pritamdamania87: <p><a class="mention" href="/u/kevin_cha">@Kevin_Cha</a> There shouldn’t be significant overhead, but what is the reason to use ddp with a single gpu? Usually it makes sense to use ddp when training with multiple GPUs.</p>
RuntimeError: unexpected EOF, expected 1362420 more bytes. The file might be corrupted.`

saakshi077: <p>This is the error msg :<br>
<code>Setting up Perceptual loss... Loading model from: /content/LOHO/networks/lpips/weights/v0.1/vgg.pth ...[net-lin [vgg]] initialized ...Done Setting up Perceptual loss... Loading model from: /content/LOHO/networks/lpips/weights/v0.1/vgg.pth ...[net-lin [vgg]] initialized ...Done Constructing DeepLabv3+ model... Number of classes: 20 Output stride: 16 Number of Input Channels: 3 /content/drive/MyDrive/inference.pth Traceback (most recent call last):   File "/content/LOHO/loho.py", line 180, in &lt;module&gt;     state_dict = torch.load(graphonomy_model_path)   File "/usr/local/lib/python3.7/site-packages/torch/serialization.py", line 595, in load     return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)   File "/usr/local/lib/python3.7/site-packages/torch/serialization.py", line 781, in _legacy_load     deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly) RuntimeError: unexpected EOF, expected 1362420 more bytes. The file might be corrupted.</code><br>
`</p>
<p>``<br>
Trying to run this repo :  <a href="https://github.com/saakshi077/LOHO" class="inline-onebox" rel="noopener nofollow ugc">GitHub - saakshi077/LOHO: Demo code for "LOHO: Latent Optimization of Hairstyles via Orthogonalization".</a></p>
<p>Colab File link :  <a href="https://colab.research.google.com/drive/1ykZK0v81l20UnbFDhr1P6PH54gFGhBqV?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colaboratory</a></p>

ptrblck: <p>As given in the error message, you might want to download or create the file again, as it seems to be corrupted.</p>

saakshi077: <p>The file is not corrputed, I have contacted the people who own this repo. The problem is with , how google colab reads these checkpoints file.<br>
These files are two large to be uploaded on the colab.<br>
So ,  I mounted the drive here and uploaded these checkpoints files there and changed the path accordingly But due to some reason google colab is not reading these files and even weights are not loading.</p>

ptrblck: <p>Thanks for the update and interesting findings. In that case it seems that Colab is causing the issue, so you might want to post the question in a Colab-specific discussion board or StackOverflow.</p>

saakshi077: <p>Thanks, I have posted on stackoverflow, No one has responded yet.<br>
Are there any colab specific discussion boards/ group that you know of ?</p>

ptrblck: <p>No, unfortunately, I haven’t seen or used a Colab-specific support forum.</p>
Logistic regression with diagonal weight matrix

macazinc: <p>I’m trying to force a linear layer to only have non-zero weights along the diagonal. Is there a straightforward way to do this?</p>

pascal_notsawo: <p>Assuming that <code>x</code> is your input, and that your output <code>y</code> (and thus the resulting loss) is a function of <code>x.W + b</code>, you can force <code>W</code> to be diagonal by adding a constraint to your loss function: <code>normal_loss + sum({norm(W_ij), i!=j})</code>, (regularization on all elements of <code>W</code> except those on the diagonal).</p>

googlebot: <p>roll your own module, and change computations as you need, for example you can base them on torch.diag(F.softplus(raw_W_diag_param)) (add inverse cholesky decomposition if you want off-diagonals).</p>
Elegant way to determine whether use torch.amp

Alpha: <p>Hi,</p>
<p>Is there an elegant way to write <code>if-statement</code>, which determine whether to use the torch amp.</p>
<p>here is my naive implementation.</p>
<pre><code class="lang-auto">model = Net().cuda()
optimizer = optim.SGD(model.parameters(), ...)

# Creates a GradScaler once at the beginning of training.
scaler = GradScaler()

for epoch in epochs:
    for input, target in data:
        optimizer.zero_grad()

        # Runs the forward pass with autocasting.
        if use_amp:
            with autocast():
                output = model(input)
                loss = loss_fn(output, target)

            scaler.scale(loss).backward()
            scaler.step(optimizer)
            scaler.update()
        else:
            output = model(input)
            loss = loss_fn(output, target)
            
            loss.backward()
            optimizer.step()
             
</code></pre>

ptrblck: <p>The code looks correct and I’m not sure if there is a more elegant way.</p>

serkansulun: <p>Wouldn’t this work?</p>
<pre><code class="lang-auto">model = Net().cuda()
optimizer = optim.SGD(model.parameters(), ...)

# Creates a GradScaler once at the beginning of training.
scaler = GradScaler(enabled=use_amp)

for epoch in epochs:
    for input, target in data:
        optimizer.zero_grad()
        
        with autocast(enabled=use_amp):
            output = model(input)
            loss = loss_fn(output, target)

        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

</code></pre>

ptrblck: <p>Yes, your approach is the recommended way, so thanks for following up on this (I’ve missed to add this approach into this thread)!</p>
This initialization let me confused. Why I need weight.new two times for GPU & 2 times for CPU?

pain: <pre><code>    def init_hidden(self, batch_size):
    ''' Initializes hidden state '''
    # Create two new tensors with sizes n_layers x batch_size x n_hidden,
    # initialized to zero, for hidden state and cell state of LSTM
    weight = next(self.parameters()).data
    
        if (train_on_gpu):
            hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda(),
              weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda())
        else:
             hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_(),
                  weight.new(self.n_layers, batch_size, self.n_hidden).zero_())
    
    return hidden</code></pre>

pain: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I hope you can help, Thanks.</p>

ptrblck: <p>The code snippet shows different initializations for CPU and CUDA tensors, not only for the GPU.<br>
Alternatively to using this condition, you could also write device-agnostic code and call <code>.to(device)</code> on the newly created tensors.</p>

pain: <p>Okay that is okay.</p>
<p>But why I need to create tuples with 2 elements?</p>
<pre><code class="lang-auto">hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda(),
              weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda())
</code></pre>

ptrblck: <p>You are creating one <code>tuple</code> containing two tensors and are most likely passing them to the <code>nn.LSTM</code> module as <code>h0</code> and <code>c0</code> (hidden and cell state) as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html">docs</a></p>
Updating weights with two optimizers

AfonsoSalgadoSousa: <p>Hi. I have a system with a discriminator, encoders, decoders and classifiers where: I want to approximate two distributions in an adversarial process using the encoders and the discriminator; I want to keep the original features of the distribution using encoders and decoders; I want to keep the system discriminative to the classification task using a classifier.<br>
My problem is that I want to update the encoders’ weights using two different optimizers:</p>
<pre><code class="lang-auto">optimizer_G = torch.optim.Adam(
        itertools.chain(encoder_a.parameters(), encoder_v.parameters(), encoder_l.parameters(),
                        decoder_a.parameters(), decoder_l.parameters(), decoder_v.parameters()), weight_decay=decay, lr=lr, betas=(b1, b2))
optimizer_E = torch.optim.Adam(
        itertools.chain(encoder_a.parameters(), encoder_v.parameters(), encoder_l.parameters(),
                        classifier.parameters()), lr=lr, betas=(b1, b2), weight_decay=decay)
</code></pre>
<p>The main training process I am using is the following:</p>
<pre><code class="lang-auto">x = batch[:-1]
x_a = Variable(x[0], requires_grad=False).squeeze()
x_v = Variable(x[1], requires_grad=False).squeeze()
x_t = Variable(x[2], requires_grad=False)
y = Variable(batch[-1].view(-1, output_dim), requires_grad=False)

# encoder-decoder
optimizer_G.zero_grad()
a_en = encoder_a(x_a)
v_en = encoder_v(x_v)
l_en = encoder_l(x_t)
a_de = decoder_a(a_en)
v_de = decoder_v(v_en)
l_de = decoder_l(l_en)

rl1 = pixelwise_loss(a_de, x_a) + pixelwise_loss(v_de, x_v) + \
        pixelwise_loss(l_de, x_t)  # reconstruction loss

g_loss = alpha * (adversarial_loss(discriminator(l_en), valid) +
        adversarial_loss(discriminator(v_en), valid)) + (1 - alpha) * (rl1)
g_loss.backward(retain_graph=True)
optimizer_G.step()

# classifier
optimizer_E.zero_grad()
a = classifier(a_en)
v = classifier(v_en)
l = classifier(l_en)
c_loss = criterion(a, y) + criterion(l, y) + \
                criterion(v, y)  # classification loss
c_loss.backward(retain_graph=True)
optimizer_E.step()
</code></pre>
<p>The error I get is:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [50, 50]], which is output 0 of TBackward, is at version 2; expected version 1 instead.</p>
</blockquote>
<p>I believe the problem is that optimizer_G.step() changes the weights inplace that are otherwise needed to compute the c_loss.backward(), as noted by <a class="mention" href="/u/alband">@albanD</a> in <a href="https://discuss.pytorch.org/t/solved-pytorch1-5-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/90256">this issue</a>, but I don’t know how to circumvent this problem. Do you have any idea?</p>

AfonsoSalgadoSousa: <p>I managed to circumvent the problem by calling both <em>.backward</em> and <em>.step</em> methods at the end of the loop.</p>
Neural networks with 2 inputs of different shapes

duddal: <p>Hello,</p>
<p>I would like to discuss something here as I am going to use PyTorch to build the NN architecture. I have 2 sets of images: Grayscale (1 channel, size=(500,1200)) and Depth Images (40, 110) size. I have prepared a dataset with the following annotation format: x1, y1, x2, y2, class, depth, wherein x1, y1, x2, y2 are the coordinates of the bounding box. Now, you might be wondering, why is there depth in the annotations? Spot on!. So I wrote an algorithm that uses the depth image, point cloud to calculate the distance of the object from the sensor. This algorithm is applied in the pre-annotated bounding boxes. This distance value is then appended to the annotation of each object as explained above. I would like to build a neural network that would take the grayscale and depth image as inputs and then predict the bounding boxes, classes, and depth. I have the bounding boxes drawn inside the depth image representing the object that correlates to the boxes in the grayscale image with another algorithm.</p>
<p>My question is twofold and as follows:</p>
<ol>
<li>
<p>My idea is to have two different subnetworks, one that does the classification and bounding box regression and the other predicts depth. My doubt here is regarding the second subnetwork. This subnetwork would actually be a CNN regression Network which would regress the values lying inside each bounding box and then predict the depth value. This prediction will then be compared to the ground truth value and the RMSE loss function can be used to correct it. How do I build this Subnetwork though? Or is there a better alternative?</p>
</li>
<li>
<p>My first idea is to use an Encoder-Decoder Network which takes in the 2 inputs and then outputs what’s needed. My question, however, is that the inputs are of different sizes. Do I need to upsample or downsample either of the images before feeding to the Encoder-Decoder network? Also, how will the network decide which features to use for classification and depth prediction? How can the network focus only inside the bounding boxes of the depth images to figure out value? or there is no need?</p>
</li>
</ol>
<p>Would really appreciate your help.<br>
Thank You.</p>
Use argmax to get values from another tensor

Manuel_Alejandro_Dia: <p>Hi,</p>
<p>Currently I am trying to implement a network where a set of convolution weights are predicted. <a href="https://arxiv.org/pdf/2003.10152.pdf" rel="noopener nofollow ugc">Link to paper</a></p>
<p>First, some part of the network (category branch) predicts the class and its probability score for a set of cells which represent the image. Based on this probability score, the network decides which cells will use their predicted weights (kernel branch) to be used in the final output.</p>
            <div class="onebox imgur-album">
              <a href="https://imgur.com/a/jFOJchs" target="_blank" rel="noopener nofollow ugc">
                <span class="outer-box" style="width:600px">
                  <span class="inner-box">
                    <span class="album-title">[Album] imgur.com</span>
                  </span>
                </span>
                <img src="https://i.imgur.com/mjI3Hpo.png?fb" title="imgur.com" height="315" width="600">
              </a>
            </div>

<p>For example, let’s say I have a grid of 10x10 on the category branch, with 8 possible classes. And my kernel branch will predict the weights for a 1x1 convolution applied to a feature map with depth of 256 for each of these cells in the grid. Here I am assuming a batch size of 4.</p>
<p>Here is the code I have so far:</p>
<pre><code class="lang-python">
import torch
import torch.nn.functional as F

cat_branch = torch.rand(4,8,10,10).cuda()*0.55 
# I'm multiplying by 0.55 just to have some False
# values when applying a threshold
kernel_branch = torch.rand(4,256,10,10).cuda()
feature_map = torch.rand(4,256,128,256).cuda()

# I find the biggest values and apply my 
# threshold to see which ones I keep
values, idx = cat_pred.max(dim=1, keepdim=True)
values_thr = values &gt; 0.5
values_thr = values_thr.int()
values_thr = values_thr.nonzero()

#Then I just take the indexes of those with respect to the 10x10 grid 
#and get the weights from kernel_branch for each batch

weights_conv = [torch.FloatTensor().cuda() for x in range(values_thr[-1,0]+1)]

for v in values_thr:
    weights_conv[v[0]] = torch.cat((weights_conv[v[0]],kernel_pred[v[0],:,v[2],v[3]]),0)

#Then, using the predicted weights I perform 
#the Convolution on the feature map
for i in range(len(feature_map)):
    f = feature_map[i].unsqueeze(0)
    weights = weights_conv[i].view( len(weights_conv[i])//feature_map.shape[1], feature_map.shape[1],1,1)
    
    seg_preds = F.conv2d(f.cuda(), weights.cuda(), stride=1).sigmoid()

</code></pre>
<p>So what I want to do is use the highest scoring cells on the category branch to select the set of filters from the kernel branch and then applying them without having to use these for-loops.</p>
<p>Does anyone have any ideas on how to do this?</p>