akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.load(model) is giving inconsistent results for different iterations during re-training

copperwiring: <p>I ran a training job for some <code>n</code> sample with a given architecture and save the model using <code>torch.save()</code>.</p>
<p>Now, I want to use the saved model (it’s weights) to run a different training job with</p>
<ul>
<li>exact same architecture</li>
<li>a large number (m samples) of similar (but not same) data</li>
<li>more epocs</li>
</ul>
<p>However, when I use <code>model = torch.load(saved_model)</code> as my model for re-training the new job, I get inconsistent results i.e. I get different predictions for different run (clear variables, restart kernel) of the new training job.</p>
<p>Sample code can be seen here: <a href="https://discuss.pytorch.org/t/how-to-use-pretrained-weights-for-initializing-the-weights-in-next-iteration/83786" class="inline-onebox">How to use pretrained weights for initializing the weights in next iteration?</a></p>

Nikronic: <p>Hi,</p>
<aside class="quote no-group" data-username="copperwiring" data-post="1" data-topic="84004">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/48.png" class="avatar"> copperwiring:</div>
<blockquote>
<p>model = torch.load(saved_model)</p>
</blockquote>
</aside>
<p>Just load weights as a dicitionary where the keys are positions of the layers. You need to assign this weights to the defined model using <code>model.load_state_dict(weights)</code> where <code>weights = torch.load("my_model.pt")</code>. Have you tried that as the post you have mentioned?</p>
<p>Bests</p>
How do we implement a custom loss that backpropagates with PyTorch?

Filippo_Portera: <p>In a neural network code written in PyTorch, we have defined and used this custom loss:</p>
<pre><code class="lang-auto">def my_loss(output, target):
    global classes    
 
    v = torch.empty(batchSize)
    xi = torch.empty(batchSize)
    
    for j in range(0, batchSize):
        v[j] = 0
        for k in range(0, len(classes)):
            v[j] += torch.exp(output[j][k]) 

    for j in range(0, batchSize):
        xi[j] = -torch.log( torch.exp( output[j][target[j]] ) / v[j] )
    
    loss = torch.mean(xi)
    print(loss)
    loss.requires_grad = True
    return loss
</code></pre>
<p>but it doesn’t converge to accetable accuracies.</p>
Prediction top label

oasjd7: <pre><code class="lang-auto">def classifier(channel, pool, out):
    layers = [nn.Conv2d(channel, out, kernel_size=1), nn.AvgPool2d(pool), ViewFlatten()]
    return nn.Sequential(*layers)

out = classifier(input)
1) top_prob, top_label = torch.topk(F.softmax(out, dim=1), 1)
2) top_prob, top_label = out.max(1)
</code></pre>
<p>which one is right to get top probability and top label (predicted label) ?</p>

ptrblck: <p>Both approaches should give the same prediction.<br>
While the first one will give you the probability, the second one should return the logits instead.</p>
How to clear cuda memory?

TAF: <p>I tried this but it does not work<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/0/906de1874d93836f123125ab06c02d547c52d76b.png" alt="cuda" data-base62-sha1="kBGaPjkPYYHuJLeNELVFhyQXIEz" width="615" height="458"></p>

ptrblck: <p>You might not have deleted all references to all parameters and tensors, so these objects might still hold the memory.<br>
Also, another application might of course use the GPU memory (but I assume you are sure that PyTorch uses it).</p>
Manual 2D Convolution per channel?

soulslicer: <p>I have a tensor of size [Batch, Channels, H, W]</p>
<p>I want to manually apply a single 5x5 filter on every channel for every batch equally. How exactly do I do this?</p>
<p>I tried the following but it doesnt work:</p>
<pre><code class="lang-auto">N=5
kernel = torch.Tensor(np.ones((N, N)).astype(np.float32)) # Example 5x5 Kernel
kernel = kernel.unsqueeze(0).unsqueeze(0)
kernel = kernel.repeat((1, Channels, 1, 1))
kernel = {'weight': kernel, 'padding': N // 2}
tensor_conv = F.conv2d(tensor, **kernel)
</code></pre>
<p>I get:</p>
<p>tensor = torch.Size([2, 96, 64, 64]) #<br>
tensor_conv = torch.Size([2, 1, 64, 64])</p>
<p>Why is the channel of the conv now only 1. I want the 5x5 conv to be repeated for all channels.</p>

ptrblck: <p><code>dim1</code> in a kernel is the number of input channels, while <code>dim0</code> is the number of filters.<br>
You could use a depthwise convolution as shown here:</p>
<pre><code class="lang-python">x = torch.randn(1, 96, 64, 64)

N=5
kernel = torch.Tensor(np.ones((N, N)).astype(np.float32)) # Example 5x5 Kernel
kernel = kernel.unsqueeze(0).unsqueeze(0)
kernel = kernel.repeat((x.size(1), 1, 1, 1))
out = F.conv2d(input=x, weight=kernel, stride=1, padding=N//2, groups=x.size(1))
</code></pre>
Load dataset images with no subfolders / labels

Tom_Chung: <p>Hello I am fairly new to pytorch and I am trying to load a dataset that consist of 2016, 2017 Google Earth images of a region. I have two folders named 2016 and 2017, and inside each folder there are ~9000 images with different file names that contains the longitude/latitude numbers of the regions. My ultimate goal is to use a triplet loss to train anchor (2016 image), positive (2017 image that is same region as 2016) and negative (random different region).</p>
<p>I want to be able to make a custom class dataset that designates the anchor, positive, and negative accordingly. Any help will be appreciated. Thank you.</p>

ptrblck: <p>Have you already taken a look at <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a> to get familiar with custom datasets?<br>
If so, where are you currently stuck? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to use torch.topk() to set non-topk values of a tensor to zero?

lmnt: <p>I feel like the solution should be super simple but I can’t quite figure it out?<br>
Given a float tensor, and indices from torch.topk of the top k values, in this case 2, how could I use the long tensor indices of torch.topk to set the values of x that aren’t in the topk to zero either in-place, or as a new object?</p>
<pre><code> x = torch.arange(0,10).resize_((2,5))

 0  1  2  3  4
 5  6  7  8  9
 [torch.FloatTensor of size 2x5]
 
 topk, indices = torch.topk(x, 2)
</code></pre>
<p>The final product from all this should look like:</p>
<pre><code> 0 0 0 3 4
 0 0 0 8 9
</code></pre>
<p>Of couse this one is strcuture to always be the last two elements of each row, but in my case, it will vary randomly.</p>
<p>Thanks in advance!</p>

lmnt: <p>Not sure if this is the best way, but I am now using:</p>
<pre><code> torch.zeros(2, 5).scatter_(indices, topk)
</code></pre>
<p>However, this seems to mess up any autograd related to these variables…</p>

fmassa: <p>Your solution should work, but instead of using the inplace operation on Variables, it might be better to use them out-of-place.</p>
<pre><code class="lang-auto">res = Variable(torch.zeros(2, 5))
res = res.scatter(1, indices, topk)
</code></pre>

Ashok_Vardhan: <p>Suppose if I want to change the sign of the <code>topk</code> elements, is there a simple way to do it? The only way I am aware of is to use a for-loop but I believe it would be inefficient.</p>
Environment for AI

MV_q: <p>Hello,<br>
I would like to ask for some information regarding what environment professional deep reinforcement learning researchers use for development. I am aware of the fact that laboratories have equipment that is inaccessible for an average enthusiast in the field working from home, however, IDEs and coding languages being used are still relevant in this respect.</p>
<p>There is that infamous debate regarding Emacs and Vim, people use not only Linux, but other operating systems as well; python, C++ and other languages are being used, and so on. I have done some basic research on <a href="http://arxiv.org" rel="nofollow noopener">arxiv.org</a>, decided to ask for some additional information on the matter here as well. A combination of operating system / editor / language is what I am asking for, to be concrete.</p>
<p>I am looking forward to hearing from you.</p>
<p>Sincerely</p>

ptrblck: <p>I would recommend to start with whatever feels good and just get your hands dirty.<br>
After a while you’ll develop some preferences, which might also change in the future.</p>
TypeError: 'int' object is not iterable

CS.Enthu: <p>I have a dataloader for my training dataset.<br>
"</p>
<h1>Pytorch dataloader</h1>
<pre><code>    a_loader = torch.utils.data.DataLoader(trainingDataset, batch_size=1 , 
     shuffle=False,num_workers=0)

    print('a_loader has {} samples'.format(len(a_loader)))

    a_fake_sample = utils.Sample_from_Pool()
    b_fake_sample = utils.Sample_from_Pool()

    for epoch in range(self.start_epoch, args.epochs):

        lr = self.g_optimizer.param_groups[0]['lr']
        print('learning rate = %.7f' % lr)


     
        for i, (batchct, batchpet) in enumerate(a_loader, 0):



            # step
        
            step = epoch * min(len(a_loader)) + i + 1
</code></pre>
<p>But for the step it gives me an error  ‘int’ object is not iterable.</p>
<p>How can I solve this?</p>

ptrblck: <p>Could you create an executable code snippet to reproduce this issue with random input tensors, so that we could debug it?</p>
<p>The current code looks alright and I cannot spot the error.</p>
Single image prediction for a trained UNET model gone wrong

Aashish_P: <p>I have trained my UNet model and got a good accuracy result though for a single image prediction I am suffering more and more. I am using google colab for training purposes and saving my model into google drive. After loading the trained model prediction is not good, say only about 30/40 % though I was getting 90% accuracy during training. What might be the possible cause for this?<br>
My training loop:</p>
<pre><code class="lang-auto">for batch in train_loader:
    model.train()
    images, labels = batch
    labels = labels.squeeze(dim=1)
    labels = labels.to(torch.long)

    iteration += 1

    preds = model(images)
    loss = criterion(preds, labels)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    predicted = torch.argmax(preds.data, 1)
    total_train += labels.nelement()
    correct_train += predicted.eq(labels.data).sum().item()
    train_accuracy = 100 * correct_train / total_train
</code></pre>
<pre><code class="lang-auto">from PIL import Image
from torchvision import transforms
input_image = Image.open('drive/My Drive/BackgroundImageRemoval/Test Images/400pixel.jpg')
preprocess = transforms.Compose([
    transforms.CenterCrop(248),
    transforms.Resize(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

input_tensor = preprocess(input_image)

input_batch = input_tensor.unsqueeze(0)
with torch.no_grad():
  output2 = model(input_batch)
  output2 = output2.squeeze(0)
output2_predictions = output2.argmax(0)
r = Image.fromarray(output2_predictions.byte().cpu().numpy()).resize(input_image.size)
r
</code></pre>

ptrblck: <p>It’s hard to tell, where the accuracy drop might be coming from and I would recommend to load a specific image sample during training and validation and compare the output of these runs.<br>
If your model doesn’t give the same output (in <code>eval()</code> mode), than something might be different in the data loading or preprocessing.</p>
Conv.weight.data VS conv.weight

mohit117: <p>Hello,</p>
<p>I am confused when to use conv.weight.data VS conv.weight. For example the following <a href="https://github.com/pytorch/vision/blob/e89c4c0198fd8cbd11344564162c68771524b2d7/torchvision/models/resnet.py#L162" rel="nofollow noopener">code</a> uses,</p>
<pre><code class="lang-auto">nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
</code></pre>
<p>but I also see at many places</p>
<pre><code class="lang-auto">nn.init.kaiming_normal_(m.weight.DATA, mode='fan_out', nonlinearity='relu')
</code></pre>
<p><em><strong>Whic one to use? I am using PYTORCH 1.3.1.</strong></em></p>
<p>To investigate I checked the source code on GITHUB for v1.3.1 (<a href="https://github.com/pytorch/pytorch/blob/v1.3.1/torch/nn/init.py#L353" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/v1.3.1/torch/nn/init.py#L353</a>)</p>
<pre><code class="lang-auto">fan = _calculate_correct_fan(tensor, mode)
    gain = calculate_gain(nonlinearity, a)
    std = gain / math.sqrt(fan)
    with torch.no_grad():
        return tensor.normal_(0, std)
</code></pre>
<p>Since the function accepts a TENSOR I expect not to use .DATA and using it should throw a <em><strong>runtime error</strong></em>. But surprisingly no error comes. Hence I am confused which one to use (especially for v1.3.1)?</p>
<p>Thank you</p>

ptrblck: <p>Don’t use the <code>.data</code> attribute as it might yield unwanted side effects.<br>
While you will be able to manipulate the underlying data without raising an error, Autograd won’t be able to track these operations and you might run into a variety of issues later (we had quite a few of these issues already here in the forum <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>
<p>This attribute is also removed step by step as seen in <a href="https://github.com/pytorch/pytorch/pull/34211">this PR</a> by <a class="mention" href="/u/alband">@albanD</a>.</p>

mohit117: <p>Thankyou for the explanation,<br>
but then how to do the following operation,</p>
<pre><code class="lang-auto">conv_shuffle.weight.copy_(kernel)
</code></pre>
<p>here  <code>conv_shuffle</code> is an instance of nn.Conv2d. I want to explicitly state its weights using <code>kernel</code>.</p>
<p>However, this results in the following error</p>
<pre><code class="lang-auto">conv_shuffle.weight.copy_(kernel)
RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
</code></pre>
<p>but is rectified using the following.</p>
<pre><code class="lang-auto">conv_shuffle.weight.data.copy_(kernel)
</code></pre>
<p>So now that <code>.data</code> is discouraged, what other alternative do I have?</p>
<p>Thankyou</p>

ptrblck: <p>You could explicitly wrap this manipulation in a <code>torch.no_grad()</code> block:</p>
<pre><code class="lang-python">conv = nn.Conv2d(3, 3, 3, 1, 1)
with torch.no_grad():
    conv.weight.copy_(torch.rand_like(conv.weight))
</code></pre>
transfrom.Normalize | RuntimeError: expected device cpu but got device cuda:0

vipiet: <p>Hi,</p>
<p>when I try to normalize an image with transforms.Normalize I get this error:</p>
<p>“RuntimeError: expected device cpu but got device cuda:0”</p>
<p>I know that my model is on the GPU and I want that, how to I change the device for the normalization within my dataset class?</p>
<pre><code class="lang-auto">            norm = Normalize([0.5, 0.5, 0.5], [1, 1, 1])
            transform_list_signal += [transforms.ToTensor()]
            transform_list_signal.append(norm)
            transform_list += [transforms.ToTensor()]
            apply_transformation_signal = Compose(transform_list_signal)
            apply_transformation = Compose(transform_list)
            signal_im_transformed = apply_transformation_signal(signal_im)
            target_im_transformed = apply_transformation(target_im)
</code></pre>

WaterKnight: <p>Did you solved that error?</p>

vipiet: <p>Yes. For me it was just a updating problem. I was was using an old torchvision version. I can’t recall which, but updating torchvision to the newest version solved the problem.</p>
VGG16 torch weights trained with scale jitting

amangasaryan: <p>Torchvision provides pretrained weights for vgg16 with an error rate of 28.4. This seems to correspond to fixed 256 scale training. As authors of VGG released <a href="https://gist.github.com/ksimonyan/211839e770f7b538e2d8#file-readme-md" rel="nofollow noopener">caffe weights</a> for the model trained with scale jitting, was wondering if there are also torch weights available for vgg16 training with scale jitting?</p>

ptrblck: <p>Would you like to apply the scale jittering during evaluation as described in Section 4.2?<br>
If I’m not mistaken, <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py">this script</a> was used to pretrain the models, which uses <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomResizedCrop">RandomResizedCrop</a> during training and is thus already applying a variety of different scaled and ratios.</p>

amangasaryan: <p>I actually would like the scale jittering to be applied during training as described in Section 3.1. i.e <em>each training image should be isotropically-rescaled with shortest edge S being randomly sampled from the range [256, 512]. And only then cropped to the fixed size of 224</em>.<br>
Indeed <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomResizedCrop" rel="nofollow noopener">RandomResizedCrop</a> brings a variety of scales and aspect ratios into the training data. However, it results in a different distribution of data then random rescaling of the whole image as suggested in the paper.<br>
Giving a  context to the topic I want to reproduce generalization experiments on VOC2007 described in Appendix B. As I use torchvision pretrained weights for vgg16 I am getting 72.06 mean AP which is around 17 points less than the reported score. Thus, I was suspecting the difference might be caused by vgg weights not being trained with scale jittering.</p>

ptrblck: <p>You could apply <code>transform.Resize</code> first with a manually sampled <code>size</code> in the range <code>[256, 512]</code> first and later <code>transforms.CenterCrop</code>.<br>
Since you would sample the size in each iteration, I would suggest to use the functional API via <code>torchvision.transforms.functional.resize(img, size, interpolation=2)</code>.</p>
<aside class="quote no-group" data-username="amangasaryan" data-post="3" data-topic="83681">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amangasaryan/48/25004_2.png" class="avatar"> amangasaryan:</div>
<blockquote>
<p>However, it results in a different distribution of data then random rescaling of the whole image as suggested in the paper.</p>
</blockquote>
</aside>
<p>That makes sense, so you could follow the previous suggestion to manually reproduce the paper.</p>
MobileNet-V1 SSD Ideal Loss

Jimmy_2times: <p>I’m performing transfer learning on MbNet1 SSD for object detection and the best loss I’ve had so far is 2.09.<br>
I noticed that this number goes down when I increase the size of the dataset. But I haven’t seen it below 2.09 yet.</p>
<p>What is an ideal loss I’m looking for? How do I figure when is the loss good enough? What is the typical value of the loss we want to aim at?</p>
Checkerboard artifact free sub-pixel convolution

manozzm: <p>I have recently come across this note, which proposes better initialization scheme for sub-pixel/pixel-shuffle<br>
<a href="https://arxiv.org/abs/1707.02937" class="onebox" target="_blank" rel="nofollow noopener">https://arxiv.org/abs/1707.02937</a><br>
Here is a sample implementation I found in Lasagne (<a href="https://github.com/Lasagne/Lasagne/issues/862" rel="nofollow noopener">https://github.com/Lasagne/Lasagne/issues/862</a>)</p>
<p>How can we implement the same in pytorch ?</p>
<p>I’m new to pytorch. I have gone through forums and found that we can custom initialize weights with,</p>
<pre><code>for m in self.modules():
    if isinstance(m, nn.Conv2d):
        m.weight.data.copy_(some_weights)
</code></pre>
<p>It would be great if someone can provide a sample implementation the authors have suggested in the paper.</p>
<p>Thanks</p>

Kaixhin: <p><a class="mention" href="/u/alykhantejani">@alykhantejani</a> is this something that you can be contribute to <code>nn.init</code>?</p>

alykhantejani: <p>Sorry - just saw this (I don’t check the forums much). Is this still something that’s desired?</p>

Kaixhin: <p>If it’s not too difficult, it would be nice to have in <code>nn.init</code>. It’ll make it much easier for people to drop in and see if it helps for their problems, and we can also link <code>PixelShuffle</code> to it so that people are aware that there’s a principled weight initialisation available.</p>

mohit117: <p><a href="https://github.com/pytorch/pytorch/pull/5429/files" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/5429/files</a> has a code for it. A slightly cleaned up form if it to do the required transformation I am posting below</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

def ICNR(tensor, upscale_factor=2, inizializer=nn.init.kaiming_normal_):
    
    new_shape = [int(tensor.shape[0] / (upscale_factor ** 2))] + list(tensor.shape[1:])
    subkernel = torch.zeros(new_shape)
    subkernel = inizializer(subkernel)
    subkernel = subkernel.transpose(0, 1)

    subkernel = subkernel.contiguous().view(subkernel.shape[0],
                                            subkernel.shape[1], -1)

    kernel = subkernel.repeat(1, 1, upscale_factor ** 2)

    transposed_shape = [tensor.shape[1]] + [tensor.shape[0]] + list(tensor.shape[2:])
    kernel = kernel.contiguous().view(transposed_shape)

    kernel = kernel.transpose(0, 1)

    return kernel
    
    
upscale = 2
num_classes = 1
previous_layer_features = torch.Tensor(1, 2, 6, 6)
conv_shuffle = nn.Conv2d(2, num_classes * (upscale ** 2), 3, padding=1, bias=0)
print('initial weights 2 ip and 4 op{}'.format(conv_shuffle.weight))
ps = nn.PixelShuffle(upscale)
kernel = ICNR(conv_shuffle.weight, upscale)
conv_shuffle.weight.data.copy_(kernel)
print('\n ICNR weights {}'.format(conv_shuffle.weight))
output = ps(conv_shuffle(previous_layer_features))
print(output.shape)
</code></pre>
How to use pretrained weights for initializing the weights in next iteration?

copperwiring: <p>I have a model architecture. I have saved the entire model using <code>torch.save()</code> for some n number of iterations. I want to run another iteration of my code by using the pre-trained weights of the model I saved previously.</p>
<p><strong>Edit</strong>: I want the weight initialization for the new iteration be done from the weights of the pretrained model</p>
<p>Right now, I do something like:</p>
<pre><code># default_lr = 5
# default_weight_decay = 0.001
# model_io = the pretrained model in .pth format
model = torch.load(model_io) 
optim = torch.optim.Adam(model.parameters(),lr=default_lr, weight_decay=default_weight_decay)  
loss_new = BCELoss()  
epochs = default_epoch 
.
.
training_loop():
....
outputs = model(input)
....
.
#similarly for test loop
</code></pre>
<p>The architecture and everything of my new iteration is same right now but might want to fiddle in future.</p>
<p>Am I missing something? I have to run for a very long epoch for a huge number of sample so can not afford to wait to see the results then figure out things.</p>
<p>Thank you!</p>

chetan_patil: <p>Where is the network here ?<br>
I see only the weights stored in <code>model</code>.<br>
You need to load your network with the <code>model</code>.<br>
Typically, it goes this way,<br>
<code>#simple initialisation of your network. net = MyNet() #loading the pertained weights. weights= torch.load("*.pth*) #placing the weights inside your network. net.load_state_dict(weights) #using your pretrained network. outputs = net(input) </code></p>
<p>In your case, there’s no network defined.</p>

copperwiring: <p><a class="mention" href="/u/chetan_patil">@chetan_patil</a><br>
Oh! Makes sense. My understanding was incorrect. I thought <code>torch.load</code> would load the entire model along with the parameters, architecture and the weights.</p>
<p>It appears, it will only load the weight and not anything else. Is that correct?</p>
<p>Edit: Just to add, I don’t plan to resume training. I intend to save the model and use it for a separate training with same parameters. Think of it like using a saved model with weights etc for a larger run and more samples.</p>

chetan_patil: <p>Yes, <code>torch.load()</code> loads weights, however, it also contains information about BatchNorm’s statistics, which you might need during the testing-phase.</p>

copperwiring: <p>Thanks <a class="mention" href="/u/chetan_patil">@chetan_patil</a>. As for my use case right now, I didn’t need batch normalization. Since my architecture is absolutely same, I think I should expect  <code>torch.save()</code> to do the job i.e. using the saved model for another iteration for more number of samples.</p>

chetan_patil: <p>Yes <a class="mention" href="/u/copperwiring">@copperwiring</a>, <code>torch.save()</code> and <code>torch.load()</code> are the one.</p>
Shape of conv1d layer in PyTorch

Tehreem: <p>How we can calculate the shape of conv1d layer in PyTorch. IS there any command to calculate size and shape of these layers in PyTorch.</p>
<pre><code class="lang-auto">            nn.Conv1d(depth_1, depth_2, kernel_size=kernel_size_2, stride=stride_size),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=stride_size),
            nn.Dropout(0.25)```</code></pre>
Recommended way to replace several values in a tensor

aghtaal: <p>Is there a batch way to replace several particular values? for example,  <code>old_new_value = [[2,22],[3,33]]</code> , which means 2 should be replaced by 22, and 3 should be replaced by 33.<br>
Can I have an efficient way to achieve this?<br>
Example:<br>
old_values = torch.Tensor([1, 2, 3, 4, 5])<br>
old_new_value = [[2,22], [3,33]]<br>
end_result = torch.Tensor([1, 22, 33, 4, 5])</p>
Does conv_transpose2d do a convolution or a cross-correlation operator?

Caesar: <p>The <a href="https://pytorch.org/docs/stable/nn.html#convtranspose2d" rel="noopener nofollow ugc">documentation</a> for the nn mentions it does a cross-correlation, however, my results indicate it does a convolution operator.</p>
<blockquote>
<p>import torch<br>
import torch.nn.functional as F<br>
im = torch.Tensor([[0,1],[2,3]]).unsqueeze(0).unsqueeze(0)</p>
<p>kernel = torch.Tensor([[0,1,2],[3,4,5], [6,7,8]]).unsqueeze(0).unsqueeze(0)<br>
op = F.conv_transpose2d(im, kernel, stride=2)<br>
print(op)</p>
</blockquote>
<p>It outputs :</p>
<blockquote>
<p>tensor([[[[ 0.,  0.,  0.,  1.,  2.],<br>
[ 0.,  0.,  3.,  4.,  5.],<br>
[ 0.,  2., 10., 10., 14.],<br>
[ 6.,  8., 19., 12., 15.],<br>
[12., 14., 34., 21., 24.]]]])</p>
</blockquote>
<p>which would be the result if there was a convolution, I had expected the correlation result to be:</p>
<blockquote>
<p>tensor([[[[ 0.,  0.,  8.,  7.,  6.],<br>
[ 0.,  0.,  5.,  4.,  3.],<br>
[16., 14., 38., 22., 18.],<br>
[10.,  8., 21., 12.,  9.],<br>
[ 4.,  2.,  6.,  3.,  0.]]]])</p>
</blockquote>
<p>Did I misunderstand something or is it a bug?</p>
VAE , train encoder & decoder

NearIt: <p>trying to train a VAE network, and the use the encoder part later on for other things.</p>
<p>I have:</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self):
    ...
    def init_weights(self,m):
        #print(m)
        for name, param in m.named_parameters():
        if 'bias' in name:
          nn.init.constant_(param, 0.0)
        elif 'weight' in name:
          nn.init.normal_(param, mean=0.0, std=0.001)
    ....


class Decoder(nn.Module):
    def __init__(self):
        ...
    def init_weights(self,m):
        #print(m)
        for name, param in m.named_parameters():
        if 'bias' in name:
          nn.init.constant_(param, 0.0)
        elif 'weight' in name:
          nn.init.normal_(param, mean=0.0, std=0.001)
</code></pre>
<p>as for now they both are separate classes.</p>
<p>I do want to train them together meaning, (Encoder -&gt;Decoder) as a big network, with 1 loss function.<br>
but after the training is done, i want to use only the trained encoder part.<br>
I am not sure of how to do that.</p>
<p>if i keep the two network separated, i will need two criterion &amp; optimizer, but i want them to be the same thing. and i am not sure how can I use the same loss to train both together.<br>
and if i combined those to one class , i may be able to train, but how to save and use only the trained encoder later on?</p>
Link libtorch with make instead of cmake

yeedTorch: <p>Hey guys!</p>
<p>For a project I am forced to link LibTorch with a Makefile, I can’t use cmake. Now looking into the LibTorch<br>
folder I am a little dazzled by all the different things. Could somebody point me in the right direction on how to do this, i.e. which libraries do I have to link specifically?</p>
<p>Best<br>
yeedTorch</p>

glaringlee: <p><a class="mention" href="/u/yeedtorch">@yeedTorch</a><br>
You can do it, but you have to keep in mind that pytorch code changed very fast, symbols might changed their .so, for eg. we might want to split some of the .so to make the size of those .so smaller.<br>
We do recommend to use cmake.</p>
<p>If you want to use make, check this post:<br>
‘<a href="https://discuss.pytorch.org/t/how-do-i-to-create-a-new-project-in-c-to-run-libtorch-without-cmake/79046/5" class="inline-onebox">How do i to create a new project in C++ to run libtorch without cmake?</a>’<br>
And I believe now you need to add -ltorch_python after -lc10 as well. And if your code has symbols in other .so, you should add them as well. All the .so are in the {your_libtorch_root|/lib, and if you get and missing symbol when linking, you can try finding the missing symbol within the .so in that folder and then add them into your make line.</p>

yeedTorch: <p>Thank you!</p>
<p>I looked at the g++ commands when running the Makefile that cmake produces and was able to glue together something that works.</p>
<p>I am all for using cmake, unfortunately we have a little transition time before we switch build systems completely. My team doesn‘t want to mix systems until then.</p>
How to make BCELoss batch size invariant?

InfoMax: <p>I am training a BCE multi-label classifier on timeseries data using a common nested cross validation scheme where I train on the first t data points and predict t+1 then retrain from scratch up to t+1 and predict t+2, etc.</p>
<p>As you can see the batch size is increasing at each iteration and this is causing the training loss to grow logarithmically with the batch size. The way I understanding it is that the model becomes harder to fit by some average error log(1+N*eps). This is undesirable as it is equivalent to increasing the learning rate at every iteration and makes training losses not readily comparable.</p>
<p>Is there a way to scale the BCE loss to prevent this effect? I am using the default mean reduction.</p>
Implementation of Attention in Pytorch

Vishu_Gupta: <p>I saw the following implementation of attention in PyTorch:</p>
<pre><code class="lang-auto">class WeightedAttention(nn.Module):

    """

    Weighted softmax attention layer

    """

    def __init__(self, gate_nn, message_nn, num_heads=1):

        """

        Inputs

        ----------

        gate_nn: Variable(nn.Module)

        """

        super(WeightedAttention, self).__init__()

        self.gate_nn = gate_nn

        self.message_nn = message_nn

        self.pow = torch.nn.Parameter(torch.randn((1)))

    def forward(self, fea, index, weights):

        """ forward pass """

        gate = self.gate_nn(fea)
        
        gate = gate - scatter_max(gate, index, dim=0)[0][index]       

        gate = (weights ** self.pow) * gate.exp()

        # gate = weights * gate.exp()

        # gate = gate.exp()

        gate = gate / (scatter_add(gate, index, dim=0)[index] + 1e-13)
        
        fea = self.message_nn(fea)

        out = scatter_add(gate * fea, index, dim=0)

        return out

    def __repr__(self):

        return '{}(gate_nn={})'.format(self.__class__.__name__,

                                       self.gate_nn)
</code></pre>
<p>Is it the correct way of implementing attention in Pytorch?</p>
Change weight decay during training

Gianluigi_Silvestri: <p>While training my CNN, I need to apply weight decay only to a subset of the layers, which changes at each forward pass. Is it possible to set weight decay to 0 for some groups of weights dynamically at each forward pass?</p>

ayushm-agrawal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Any insights on this? I am looking for something like this as well <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>You should be able yo change the <code>weight_decay</code> for the current <code>param_group</code> via:</p>
<pre><code class="lang-python"># Setup
lin = nn.Linear(1, 1, bias=False)
optimizer = torch.optim.SGD(
    lin.parameters(), lr=1., weight_decay=0.1)

# Store original weight
weight_ref = lin.weight.clone()

# Set gradient to zero (otherwise the step() op will be skipped)
lin.weight.grad = torch.zeros_like(lin.weight)

# Apply weight decay
optimizer.step()

# Store weights after weight decay
weight1 = lin.weight.clone()

# Remove weight decay for this param group
optimizer.param_groups[0]['weight_decay'] = 0.

# Step again
optimizer.step()

# Store weight to compare
weight2 = lin.weight.clone()

# Compare
print(weight_ref)
&gt; tensor([[0.8899]], grad_fn=&lt;CloneBackward&gt;)
print(weight1)
&gt; tensor([[0.8009]], grad_fn=&lt;CloneBackward&gt;)
print(weight2)
&gt; tensor([[0.8009]], grad_fn=&lt;CloneBackward&gt;)
</code></pre>
<p>Note that I had to manually set the <code>grad</code> to zeros, as otherwise the <code>optimizer.step()</code> operation would just skip this parameter.</p>
Lazy alternative to torch.where

aRI0U: <p>Hi,<br>
I would like to use <code>torch.where</code> to insert a default value in my tensor when some exception occurs. Here is a minimal example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.tensor([3,1,5])
&gt;&gt;&gt; b = torch.tensor([2,1,0,2,3,1])
&gt;&gt;&gt; torch.where(b &lt; len(a), a[b], -1) # expects tensor([5,1,3,5,-1,1])
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
IndexError: index 3 is out of bounds for dimension 0 with size 3
</code></pre>
<p>Is there an alternative to that piece of code which evaluates <code>a[b]</code> only on indices i such that <code>b[i]</code> satisfies the condition?</p>

googlebot: <pre><code class="lang-auto">torch.where(b &lt; len(a), a[b.clamp_max(len(a)-1)], torch.tensor([-1]))
</code></pre>
<p>true “lazy” ops have “masked” in their names, but there is no “masked_gather” I think.</p>

aRI0U: <p>Exactly what I needed, thanks a lot !</p>
Does optimizer.zero_grad() set input.grad to zero as well?

Sudarshan_Babu: <p>Suppose we have the following setup:<br>
optimizer = Adam(model.parameters())</p>
<p>for (data,labels) in dataloader:<br>
output = model(data)<br>
loss = F.nll_loss(output,labels)<br>
loss.backward()<br>
data.grad // will this be reset to zero after every iteration by optimizer.zero_grad() ?<br>
optimizer.step()<br>
optimizer.zero_grad()</p>
<p>so my question is will optimizer.zero_grad set the gradients with respect to the data to zero as well, and more generally will it set the gradients of parameters not tracked by the optimizer to zero ?</p>

albanD: <p>Hi,</p>
<p>No it will only reset the gradient of the parameters associated with it.</p>

Sudarshan_Babu: <p>Thank you for your response.</p>
<p>But in my experiments I observe that for data.grad the value is reset to 0, and I think that is because for every iteration within the dataloader loop, data is a new tensor (since we have different datapoints for every new iteration), and consequently data.grad is set to none, and after loss.backward() data.grad gets a new unaccumulated gradient. Is this correct ?</p>

albanD: <p>If data is a different Tensor, then yes it won’t have any existing .grad field.</p>

Sudarshan_Babu: <p>Gotcha, thanks a lot !</p>
Ignoring losses that are too high

Cirets0h: <p>Due to certain data I want my network to only learn on losses that are lower than 1. For that I use the following code:</p>
<pre><code class="lang-auto">def train(img, label):
   x = self.net(img)
    loss_val = self.loss(x, label)
    self.optimizer.zero_grad()
        
    if loss_val &lt; 1:
        loss_val.backward()
        self.optimizer.step()
</code></pre>
<p>Is this correct? Because I have the feeling that the gradients are not reseting so as soon as a loss_val is under 1, all updates on the gradients are done, even the ones on which the osses were higher than 1.</p>
<p>How can I otherwise reset all the gradients from the optimiser?</p>

ptrblck: <p>The code looks alright. What do you mean by “the gradients are not resetting”?<br>
Are you seeing valid gradients at the end of the iteration even though the loss was &gt;=1?</p>

Cirets0h: <p>I don’t really know how to check that.</p>
<p>My worry that the following is happening. I do an iteration where the loss is over 1 and I don’t do a backwards on the loss, so all tensors still hold their gradient trees from the iteration. Then, on the second iteration the gradient trees are then also changed according to the information of the new iteration (while still holding the information of the first iteration) and if the second iteration loss is then lower than 1, the gradients are calculated, but there is also the information about the first iteration that was actually lower than 1.</p>

ptrblck: <p>Each forward pass will create a new computation graph.<br>
To double check your use case, you can print the gradients via:</p>
<pre><code class="lang-python">for name, param in net.named_parameters():
    print(name, param.grad)
</code></pre>
<p>You can print it at different places to see, if the gradients are valid or have been zeroed out.</p>
Combine network outputs and loss functions

Erhan: <pre><code class="lang-auto">                    outputs1 = model1(input1)
                    outputs2 = model2(input2)
                    outputs = outputs1 + outputs2 

                    loss = criterion(outputs, labels)
                    loss.bacward()
</code></pre>
<p>or</p>
<pre><code class="lang-auto">                   outputs1 = model1(input1)
                   outputs2 = model2(input2)
                   loss1 = criterion(outputs1, labels)
                   loss2 = criterion(outputs2, labels)
                   
                   loss = loss1 + loss2
                   loss.bacward()
</code></pre>
<p>Is there any difference between them?</p>

Sudarshan_Babu: <p>I think you mean concatenate outputs1 with outputs2, in which case there should be no difference.</p>
Batched eigenvector computation

Tin: <p>Hey guys, I am trying to perform a batched computation of eigenvectors from a tensor of dimension (B, M, M) at each epoch. This computation will be used to compute a term in the loss function of the overall model.<br>
B is the batch size, therefore I want to obtain B separate, individual results.<br>
I am currently achieving this via a simple for loop over the batch dimensions as follows:</p>
<pre><code class="lang-auto">batch_eig_vals, batch_eig_vecs = [], []
for i in range(S_B.shape[0]):
    eig_vals, eig_vecs = torch.symeig(S_B[i], eigenvectors=True)
    batch_eig_vals.append(eig_vals)
    batch_eig_vecs.append(eig_vecs)

batch_eig_vals = torch.stack(batch_eig_vals)
batch_eig_vecs = torch.stack(batch_eig_vecs)
</code></pre>
<p>However, this is <em>extremely</em> slow: each epoch runs about 5x slower than its counterpart without this particular term.<br>
Are there any clever tricks I can apply to speed this up?</p>

albanD: <p>Hey,</p>
<p>Note that symeig accepts batch of inputs. So you can actually give it your whole <code>S_B</code> at once.<br>
Other than that, I don’t think there is much to do. eigenvalue decomposition is quite expensive. Note that if you’re on the GPU and using small Tensors, it might be better to send them back to the CPU before doing the symeig.</p>

Tin: <p>Thank you. Having been using <code>torch.eig()</code> I had missed that.<br>
You mention sending Tensors to the CPU, but what performance advantages would I have doing so?</p>

albanD: <p>Linear algebra on the GPU is not great (more branching). So especially for small Tensors, doing these computations on the CPU might be better. But it depends a lot on your task, you’ll have to try.</p>
Text generator LSTM- Embedding layer

cevvalu: <p>Why there is an embedding layer before LSTMs. I see various reasons but every source tells something different.  Some say to ignore padded zerous or create word vectors for LSTM’s input.</p>

vdw: <p>You need to give a bit more information to get useful response. What is your learning task? What’s your input, text?</p>
<p>An embedding layer is specific requirement for a LSTM. The requirement stems from the input. Basically all models using textual content to encode words or characters.</p>

cevvalu: <p>Hi Chris, thanks for reply<br>
I have built a image captioning model based on Tanti’s paper<br>
I used 2 inputs for image and text data and concatenated them before the last dense layer. I do not use a softmax, i take the maximum logit from dense layer’s output to get the next word predicted<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3ffc04fb02724b9dac65483b9eec122ed852dbd4.png" alt="Screenshot_1" data-base62-sha1="9821QFtmVBNtAeKcisVFZCrb3Tu" width="317" height="91"><br>
I had several experiments by changing embedding layer’s output dimension to 128, 256 and 512<br>
As I have observed from my captions, As the dimension is increased, generated captions became more complex. But i am trying to understand the logic behind it</p>
How to fill Torch Tensors

Giuseppe: <p>I have some tensor of difference size(1x512, 1x2048). These tensor will be the input of autoencoder, to have a small dimension.<br>
How can fill the tensors with dimension 1x512 to have the same dimension of others? With zeros? And if yes have i to use torch.cat?<br>
Or maybe it’s better to have two autoencoder (one for 1x512 and one with 1x2048)?</p>

Nikronic: <p>Hi,</p>
<p>I do not know about your task, but usually zeros are the best way of padding in different tasks if 0 does not mean special case.<br>
Concerning padding smaller size tensor to have same size with bigger tensors, <code>torch.nn.functional.pad</code> can help. For instance in your case:</p>
<pre><code class="lang-auto">x = torch.randint(1, 2, (1, 512))
x = F.pad(x, pad=((2048-512)//2, (2048-512)//2), mode='constant', value=0)
</code></pre>
<p>Will do the trick.<br>
This code, will add 768 zeros before and after 512 values of original tensor. If you want to add different padding size in each side, you can just pass different values as a tuple to the <code>pad</code> argument of this method.<br>
Note that there are few other options for padding or constant value.</p>
<p>I do not know much about autoencoders, but I think defining two networks because of different input sizes is not convention as there are some tricks to handle this issue without hurting performance. A typical example is resizing images in CNNs using interpolation and padding.</p>
<p>Bests</p>
How can I get the same size output?

wonchulSon: <p>Using 2x2 conv, how to get the same size output as input?</p>
<pre><code class="lang-auto">nn.Conv2d(input, output, kernel_size=2, stride=1, padding=padding)
</code></pre>
<p>Like above, when I set stride=1, padding must be 0.5 but it’s impossible.</p>
<p>Thank you.</p>

Nikronic: <p>Hi,</p>
<p>I do not know the input size, but if 0.5 padding size is the answer, it is impossible, as we do not have half-pixel. That is one of the reasons that odd size kernels are preferred. Also, odd size kernels have a single pixel as the origin cell, which enables better weighting.</p>
<p>But if you like one sided padding, you can do this by explicitly padding your input then applying conv2d. For padding you can use <code>torch.nn.functiona.pad</code> function. See this post for more information:</p><aside class="quote" data-post="1" data-topic="21212">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/49beb7/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-pad-one-side-in-pytorch/21212">How to pad one side in pytorch</a> 
  </div>
  <blockquote>
    By default pytorch only support padding on both side, but for example, I have a feature of 1x512x37x56(NCHW) and I want to pad on one side to 1x512x38x57, how can I do it?
  </blockquote>
</aside>

<p>The reason I suggested one side padding is that, PyTorch by default uses two sided paddings so I think that is why you ended up with padding size of 0.5. S, other choice could be padding only one side.</p>
<p>Bests</p>