akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Error in Shared Layers

AvioGG: <p>Hi there.</p>
<p>I am trying to implement two simple models, in which their last connected layer is shared. I have seen this post  <a href="https://discuss.pytorch.org/t/share-a-layer-between-two-different-models/54611">Sharing Layers Discussion</a> and tried to run the code provided by the original poster, however it gave an error.</p>
<p>Then, I tried to implement the thing myself with the code below. (First layer for both models are exclusive and the last layer must be shared.)</p>
<pre><code class="lang-auto">import torch
from torch import nn, optim
import torch.nn.functional as F

torch.autograd.set_detect_anomaly(True)

class A(nn.Module):
   def __init__(self):
       super(A, self).__init__()
       self.base_fc_a = nn.Linear(3,3)
       self.fc = nn.Linear(3, 1)

   def forward(self, x):
       x2 = self.base_fc_a(x)       
       x3 = self.fc(x2)
       return x3

class B(nn.Module):
   def __init__(self, shared_re):
       super(B, self).__init__()
       self.base_fc_b = nn.Linear(3,3)
       self.shared_fc = shared_re
       
   def forward(self, x):
       x2 = self.base_fc_b(x)
       x3 = self.shared_fc(x2)
       return x3
       
net_A = A()
net_B = B(shared_re = net_A.fc)
optim_A = optim.Adam(net_A.parameters())
optim_B = optim.Adam(net_B.parameters())
target = torch.randn(1,1)

x_A = torch.rand(1, 3)
y_A_hat = net_A(x_A)
loss_A = F.mse_loss(y_A_hat, target)

x_B = torch.rand(1, 3)
y_B_hat = net_B(x_B)
loss_B = F.mse_loss(y_B_hat, target)

optim_A.zero_grad()
loss_A.backward()
optim_A.step()

optim_B.zero_grad()
loss_B.backward()
optim_B.step()

</code></pre>
<p>Well, this gives an error:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<p>However, when I do the loss calculation for  loss_B after loss_A.backward, there is no issue. What causes this? Any help or explanation is appreciated.</p>

Zheng_An_Zhu: <p>HI,<br>
.step() will update the model weight ‘inplace’.<br>
You can try to do .step() after second .backward(), or redo the forward process with new model weight after first .step().<br>
Hope this help.</p>
Same CNN model, but different results?

Z_Huang: <p>Hi community,</p>
<p>I have created 2 CNN models. The first model is my baseline model and the second model is an improvement on top of the first model.</p>
<p>I have observed one weird scenario. These two models always produce different results (difference between +0.05% and -0.05%). Sometimes model one produces a better result and some other times model two produces a better result.</p>
<p>How could I avoid that from happening? I know I can use “seed”, but it doesn’t mean the second model outperformed the performance of the first model in general.</p>
<p>I have done some research. It seems I need to generate a common shared initial weight for both of the models. Then I can compare apple to apple.</p>
<p>For example, if both of the models are all ResNet50 based. The logic should be:</p>
<pre><code class="lang-auto">def set_seeds(seed):
    os.environ["PL_GLOBAL_SEED"] = str(seed)
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
</code></pre>
<pre><code class="lang-auto">1. Create an empty ResNet50 model and save the state dictionary X;
2. Load the state dictionary X to ResNet50-A;
3. Load the state dictionary X to ResNet50-B;
4. Compare the results between ResNet50-A and ResNet50-B.
</code></pre>
<p>Any idea? Recommendation?</p>

ptrblck: <p>I don’t think using a single seed (or the same parameters) would give you a good signal, since this seed (or parameter set) could stil be beneficial for one of the models.<br>
To properly check the model accuracy, you could rerun each training script using different seeds and calculate the mean +/- stddev  of the achieved accuracy.</p>

Z_Huang: <p>I am thinking to take 5 or more (maybe 30 seeds to reach statistical significance?) different seeds to compare these 2 models.</p>
<p>For example, if each of my models has 1 million parameters. Should I set seeds like seed(1), seed(1million+1), seed(2million+1), seed(3million+1), seed(4million+1), and seed(5million+1) to avoid overlapped parameters?</p>

ptrblck: <p>The seed values themselves can be sequential for each run. <a href="https://www.johndcook.com/blog/2016/01/29/random-number-generator-seed-mistakes/">This article</a> doesn’t target your use case exactly, but also claims that sequential seeds should work.</p>

Z_Huang: <p>Yes. I understand seed values are sequential. A manual seed function basically tells you a fixed point to start generating the sequential, random values. The sequence lasts infinitely. So I try to get the seed values without overlapping. I may try even larger distances between different manual seeds. It’s a good conversation with you though. Thanks.</p>
Overwrite network parameters

PhysicsIsFun: <p>Hi everyone!</p>
<p>What is the current most efficient way to overwrite model parameters?</p>
<p>For example, I have a trained network with a given weight parameter W, and I have another parameter W_new (generated by external code, i.e. not part of the network), and I want to replace W with W_new.</p>
<p>W_new is already an nn.Parameter.</p>
<p>Cheers!<br>
PiF</p>

ptrblck: <p>You could <code>.copy_</code> the new parameter into the old one via:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.layer.weight.copy_(new_param)
</code></pre>
ConcatDataset and Transformation Preserving

santhosh_subramanian: <p>Hey all! Quick question. I have 5 different classes; called ImageLoader on each class with a different Transformation. Now that I have 5 different classes with 5 different transformations, my question is that, does the transformations get preserved if I call ConcatDataSet([set1, set2,…, set5]) and then call my loader on the concatenated dataset?</p>

ptrblck: <p>I don’t know how <code>ImageLoader</code> is defined, I assume it’s a custom <code>Dataset</code> using the passed transformation inside its <code>__getitem__</code>. If so, then the transformations shouldn’t be changed by <code>ConcatDataset</code>, as this class would only concatenate the passed datasets and call them sequentially as seen <a href="https://github.com/pytorch/pytorch/blob/90d31cb311e8de0b91da7891077755911b6479bb/torch/utils/data/dataset.py#L225-L235">here</a>.</p>
How to perform one backprop after two feedforward passes?

stark: <p>Hi everyone,</p>
<p>Currently, I can’t feed a batch more than <code>64 samples</code> <em>(memory constraints on single GPU)</em> for training . But my custom loss calculation requires at least <code>128 predictions</code> since I’ll be using PCA to reduce feature dimensions to 128 and it won’t work when <code>samples &lt; components</code>.</p>
<p>The routine way of getting model predictions is:</p>
<pre><code class="lang-auto">for batch_idx, data in enumerate(dataloader['train']):
    batch, lbl = data[0], data[1]
    out = model(batch)
    loss = custom_loss(out, lbl)
    loss.backward()
</code></pre>
<p>I’m trying collecting model outputs for two training batches (64×2=128) before continuing with the loss calculations i.e.,</p>
<pre><code class="lang-auto">out_, lbl_ = [], []
for batch_idx, data in enumerate(dataloader['train']):
    batch, lbl = data[0], data[1]
    out_.append(model(batch))
    lbl_.append(lbl)

    if ((batch_idx + 1) % 2 == 0):
        loss = custom_loss(torch.stack(out_), torch.stack(lbl_))
        loss.backward()
</code></pre>
<p>yet there’s still this error: “<em>RuntimeError: CUDA out of memory.</em>”  <strong>¯(°_o)/¯</strong></p>
<p>I’d be so thankful if you may help answering these questions:</p>
<ol>
<li>Is there a separate graph created (requiring more memory) each time a model is fed with a training batch?</li>
<li>How may I get multiple outputs before calculating and backpropagating the loss?</li>
</ol>

tom: <aside class="quote no-group" data-username="stark" data-post="1" data-topic="149800">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<ol>
<li>Is there a separate graph created (requiring more memory) each time a model is fed with a training batch?</li>
</ol>
</blockquote>
</aside>
<p>Basically, if things require gradients, the autograd graph is created and only released once you call .backward/torch.autograd.grad on the output (unless you use <code>retain_graph=True</code> in which case it is not released) or the references go out of scope.</p>
<p>This means that splitting the 128 into two batches won’t help you if you want to compute backwards on all of it.</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>How may I get multiple outputs before calculating and backpropagating the loss?</li>
</ol>
</blockquote>
</aside>
<p>Now, what you could do is compute 64 outputs at a time, save a detached version of them for the next batch, and then combine the undetached outputs with the saved, detached 64 outputs of the last batch for your PCA.</p>
<p>This way, you do your PCA with 128 but only need the autograd graph for 64 samples.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: There are mathematical intricacies to this, but you could try and it might work. Another option could be to compute 64 samples with <code>no_grad</code> and 64 in the normal way to preserve the independence between batches better.</p>
Change a semantic segmentation code from binary to multi-class，model predict error

CHENT_DOU: <p>I’m changing a semantic segmentation network from two-class to multi-class, and changed the last layer of the network to x=softmax(x,dim=1). Since it’s a six-class (no background class) problem, I changed num_classes to 6, and one-hot encoding is done on the targets to measure the loss. But when making predictions on the image, unclassified pixels appear on the image. Can someone help? Thanks in advance.</p>

Andrei_Cristea: <p>Could you share a small runnable snippet of code that highlights the problem? Make sure to include the loss function you used. Depending on the loss function used, it may or may not do the softmax for you already, and it may expect one-hot encoded target or an index target. Just going to guess that there’s some chance you have a mismatch between the loss function used and the rest of your model.</p>

CHENT_DOU: <p>Thanks for your hint, I changed the loss function to <strong>nn.CrossEntropy</strong> and removed the one-hot encoding of labels and the softmax function of the input in the network. After I check the <strong>colormap</strong> in <strong>plt.imsave()</strong>, the correct result showed. <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=12" title=":laughing:" class="emoji" alt=":laughing:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>

Andrei_Cristea: <p>Delighted to hear it!</p>
Calculate the IoU between a Mask and a Box

chandlerbing65nm: <p>Do you have any insights on how to calculate the Intersection-over-Union between Instance Segmentation Mask and Bounding box?.</p>

gug: <p>First, Convert Bounding Box to the Segmentation Mask, and then you could use some function like <a href="https://github.com/Ilyabasharov/instance_tracker/blob/92b2b5f602955df6acbc9fc282d2e4909bd47386/scripts/utils.py#L82" rel="noopener nofollow ugc">this</a>.</p>
<p>I hope this helps.</p>

chandlerbing65nm: <p>I get that, or the other way around. But for RCNN (for example) the bounding box size is (batch, 4 coordinates). How can I turn that into segmentation mask?</p>
<p>In mask rcnn, predicted masks come from a RoI’s of size (batch, channels, roi_width, roi_height). Totally unrelated to the bounding box</p>

Matias_Vasquez: <p>If you have coordinates, you can use something like <a href="https://github.com/aladdinpersson/Machine-Learning-Collection/blob/master/ML/Pytorch/object_detection/metrics/iou.py" rel="noopener nofollow ugc">this</a>.</p>
<p>Here you can choose between two formats for the coordinates:</p>
<ul>
<li>Midpoint: x, y, w, h</li>
<li>Corners: x1, y1, x2, y2</li>
</ul>
<p>This way you won’t have to turn it into a segmentation mask.</p>

chandlerbing65nm: <p><a class="mention" href="/u/matias_vasquez">@Matias_Vasquez</a> a What do you mean by</p>
<blockquote>
<p>This way you won’t have to turn it into a segmentation mask.</p>
</blockquote>
<p>It is already in mask. After RoI extraction, masks are predicted as well as bounding boxes but in different sizes, just like I mentioned in my previous reply to <a class="mention" href="/u/gug">@gug</a> .</p>

Matias_Vasquez: <p>Oh sorry,</p>
<p>what I meant is that you can go in the other direction (as you also said in your second post) and use this format instead of the segmentation masks to get the IoU.</p>
<aside class="quote no-group" data-username="chandlerbing65nm" data-post="3" data-topic="149813">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chandlerbing65nm/48/48832_2.png" class="avatar"> chandlerbing65nm:</div>
<blockquote>
<p>bounding box size is (batch, 4 coordinates)</p>
</blockquote>
</aside>
<p>Torchvision has a function to get the bounding boxes out of a mask. (<a href="https://pytorch.org/vision/master/auto_examples/plot_repurposing_annotations.html" rel="noopener nofollow ugc">example</a>)</p>
<pre data-code-wrap="python"><code class="lang-python">from torchvision.ops import masks_to_boxes

boxes = masks_to_boxes(masks)
</code></pre>
<p>So now if you have everything in coordinates you can use the approach in my first post.</p>
<p>But maybe I´m not understanding correctly, if so please let me know if this approach does not</p>

chandlerbing65nm: <p>I did not know about this. Thank you.</p>
Swa model: Unexpected key(s) in state_dict: "n_averaged"

John_J_Watson: <p>I have followed the swa tutorial of pytorch and saved the state dictionary of the AveragedModel as described here: <a href="https://pytorch.org/blog/pytorch-1.6-now-includes-stochastic-weight-averaging/" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 1.6 now includes Stochastic Weight Averaging | PyTorch</a></p>
<p>Now, in my standalone inference script, I load the backbone (mobilenet in this case) and try to load the weights of the swa model to it. but I get:</p>
<pre><code class="lang-auto"> Unexpected key(s) in state_dict: "n_averaged".
</code></pre>
<p>I tried finding resources to load swa model state dict but no luck <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Any pointers to docs or tips on how to do this would be helpful!</p>
How to implement moving average of parameters

JaniceXiong: <p>Hi everyone! I want to implement the model in the paper <a href="https://arxiv.org/pdf/2109.03481.pdf" rel="noopener nofollow ugc">“Sequence Level Contrastive Learning for Text Summarization”</a>.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b6fa817d2c3de5aa934ef9bc38dd750d90e10d0.png" data-download-href="https://discuss.pytorch.org/uploads/default/0b6fa817d2c3de5aa934ef9bc38dd750d90e10d0" title="1650683619(1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b6fa817d2c3de5aa934ef9bc38dd750d90e10d0_2_579x500.png" alt="1650683619(1)" data-base62-sha1="1Dattj3m0NyKXEeYYotmChTUkms" width="579" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b6fa817d2c3de5aa934ef9bc38dd750d90e10d0_2_579x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b6fa817d2c3de5aa934ef9bc38dd750d90e10d0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b6fa817d2c3de5aa934ef9bc38dd750d90e10d0.png 2x" data-dominant-color="EFF3F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1650683619(1)</span><span class="informations">755×651 81.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As the paper mentioned, the architecture of two <code>f</code> is the same, but only updates the parameters <code>θ</code> in <code>fθ</code>, and the parameters <code>ξ</code> in <code>fξ</code> is the moving average of <code>θ</code>.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2efea2fcd9eef22a8eeb391bef80a9d3770d34be.png" alt="1650683749(1)" data-base62-sha1="6HJwfCdikSzptRQKbysTfyqXSBE" width="191" height="43"></p>
<p>I want to know:</p>
<ol>
<li>whether or not parameters <code>ξ</code> should be calculated after every backward update the parameters <code>θ</code></li>
<li>how to write codes to implement the moving average update of parameters <code>ξ</code> after I set <code>require_grad=False</code> for <code>ξ</code></li>
</ol>

mxahan: <p>You may find something from <a href="https://github.com/facebookresearch/moco-v3/blob/main/moco/builder.py" rel="noopener nofollow ugc">here.</a></p>
<p>Let me know if it works.</p>
Does PyTorch have Decoder Function for CTC just like tf.nn.ctc_beam_search_decoder in TF?

DCSong: <p>I’m tring my work with CTC, but I find no decoder funtions in PyTorch for CTC. I implyment CTC_greedy_decoder and CTC_beam_search_decoder with data on Internet. The CTC_greedy_decoder works, but CTC_beam_search_decoder runs so slowly. So does PyTorch have Decoder Function for CTC just like tf.nn.ctc_beam_search_decoder in TF? Thank you very much.</p>

Sanbot-kk: <p>DId you find a decoder?</p>

DCSong: <p>No. I just implyment CTC_greedy_decoder by myself.</p>

harald_scheidl: <p>greedy decoder is really easy to implement: <a href="https://github.com/githubharald/CTCDecoder/blob/master/src/BestPath.py" rel="noopener nofollow ugc">code</a></p>
<p>you could compute argmax even using pytorch directly.</p>

Sanbot-kk: <p>Thank you. THhs helps</p>

MohamedAmine_SAIGHI: <p>Hi Sanbot-kk, I’m trying to implement a CTC decoder, could you please provide me with an exemple of code so i can understand how the decoder works.</p>
Dataloader length inconsistent

yuyanruan179: <p>Hi there, I was stuck and found something very very abnormal. I am training an image classifier</p>
<pre><code class="lang-auto">def train(self, current_epoch=0, is_init=False, fold=0):
    self.logger.info('Current fold: %d' %(fold))

    self.model.train()
    if (not is_init):
        self.scheduler.step()
    self.logger.info('Current epoch learning rate: %e' %(self.optimizer.param_groups[0]['lr']))

    running_loss = 0.0
    running_acc = 0.0
    dataloader = self.dataloader[str(fold)]
    optimizer = torch.optim.Adam(self.model.parameters(), lr=0.0001, weight_decay=0.0001)
    self.logger.info('Len of train loader: '+str(len(dataloader['train'])))
    for i, sample in enumerate(dataloader['train'], 1):
        self.logger.info("i: {}".format(i))
        sample = self.prepare(sample)
        images, labels = sample['img'], sample['label']
        optimizer.zero_grad()
        outputs = self.model(images)
        loss = self.loss_fn(outputs, labels)
        loss.backward()
        optimizer.step()
</code></pre>
<p>I found that the output log very strange.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/1055dce9baf21014f08f89c7d4e28d33c1d36944.png" data-download-href="https://discuss.pytorch.org/uploads/default/1055dce9baf21014f08f89c7d4e28d33c1d36944" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/1055dce9baf21014f08f89c7d4e28d33c1d36944.png" alt="image" data-base62-sha1="2kvBcw8MrInLkQPY8Zc3QNB5BBO" width="690" height="311" data-dominant-color="272929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1073×484 14.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
As shown in the picture above, we can see that the first epoch only gets 2 batches, while the second one gets far more than that. In fact, it gets 12 and epoch 3 gets 15 batches.<br>
Shouldn’t it be the same number of batches? Why are they not 129 as len(dataloader) is 129?</p>
<p>You may find the entire project here. <a href="https://drive.google.com/drive/folders/1jTEY8U-wkJBOy9VpDJIc0qJRZ3WTzG_e?usp=sharing" rel="noopener nofollow ugc">Project</a></p>
<p>Sincerely thank you in advance. So frustrating…</p>

ptrblck: <p>Could you try to come up with a standalone, minimal, executable code snippet showing this behavior, please (without the need to download the actual dataset and all files)?</p>
Running Custome function on given tensor dimenstions

ammar_naich: <p>Like <code>torch.sum</code>(<em>input</em>, <em>dim</em>, <em>keepdim=False</em>, ***, <em>dtype=None</em> )<code>→ [Tensor](https://pytorch.org/docs/stable/tensors.html#torch.Tensor) can we create our custom functions that can  run on  given dimension</code>dim` ?</p>

ptrblck: <p>If your custom method can be split into native PyTorch operations, you should be able to define a <code>dim</code> argument where it makes sense. On the other hand, if your custom method is not supported at all, you could try to write a custom extension.</p>

ammar_naich: <p>Many thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, i will really appreciate it if you can refer me to any resource on how to write custom extensions. Actually, my custom function includes performing histograms on tensor features. Initially, I am running a loop on each tensor Row and doing <code>torch.histogram()</code> which takes 4.5sec for a specific feature set to complete. That also results in low GPU utilization too. I hope there is any way to speed it up.</p>

ptrblck: <p><a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">This tutorial</a> might be useful as it shows you how to write custom extensions and use them in your script.</p>

ammar_naich: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="149757">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This tutorial</p>
</blockquote>
</aside>
<p>Much appreciated <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
What is the formula for cross entropy loss with label smoothing?

KaiHoo: <p>Hello, I found that the result of build-in cross entropy loss with label smoothing is different from my implementation. Not sure if my implementation has some bugs or not. Here is the script:</p>
<pre><code class="lang-auto">import torch

class label_smooth_loss(torch.nn.Module):
    def __init__(self, num_classes, smoothing=0.1):
        super(label_smooth_loss, self).__init__()
        self.negative = smoothing / (num_classes - 1)
        self.positive = (1 - smoothing)
    
    def forward(self, pred, target):
        pred = pred.log_softmax(dim=1)
        true_dist = torch.zeros_like(pred)
        true_dist.fill_(self.negative)
        true_dist.scatter_(1, target.data.unsqueeze(1), self.positive)
        return torch.sum(-true_dist * pred, dim=1).mean()


x = torch.randn(1,10)
y = torch.randint(10,size=[1])

loss1 = label_smooth_loss(num_classes=10, smoothing=0.1)
loss2 = torch.nn.CrossEntropyLoss(label_smoothing=0.1)

print(loss1(x,y), loss2(x,y))



</code></pre>

InnovArul: <p><code>label smoothing</code> concept was introduced in the paper <a href="https://arxiv.org/pdf/1512.00567.pdf">https://arxiv.org/pdf/1512.00567.pdf</a> and the same is followed in PyTorch.<br>
You need to change your function as follows:</p>
<pre><code class="lang-auto">class label_smooth_loss(torch.nn.Module):
    def __init__(self, num_classes, smoothing=0.1):
        super(label_smooth_loss, self).__init__()
        eps = smoothing / num_classes
        self.negative = eps
        self.positive = (1 - smoothing) + eps
    
    def forward(self, pred, target):
        pred = pred.log_softmax(dim=1)
        true_dist = torch.zeros_like(pred)
        true_dist.fill_(self.negative)
        true_dist.scatter_(1, target.data.unsqueeze(1), self.positive)
        return torch.sum(-true_dist * pred, dim=1).mean()

</code></pre>
Gradient becomes None, after manually updating the weight (Using Federated Learning)

codeflux: <p>Hi,<br>
what I am trying to do is speeding up the learning process using federated reinforcement learning.<br>
In this case, I want to share the average gradient to all users and update the weights using this average value.</p>
<p>I do the update of the weights manually with the following syntax:</p>
<blockquote>
<p>with torch.no_grad():<br>
data_input = model.input_layer.weight.data + learning_rate * average_gradient.data<br>
model.input_layer.weight = torch.nn.Parameter(data_input)</p>
</blockquote>
<p>Now I thought that using torch.no_grad() would ensure, that this operation has no influence on the weight update.<br>
As it seems, the gradient is set to None after using model.input_layer.weight = torch.nn.Parameter(data_input)</p>

ptrblck: <p>In your code snippet you are assigning a new <code>nn.Parameter</code> to a <code>weight</code> attribute, which would then be missing e.g. from the optimizer unless you add it to the optimizer as a new <code>param_group</code>.<br>
The proper way to update the parameters is to use an inplace operation such as <code>.sub_()</code> or <code>.copy_()</code> in the <code>no_grad()</code> context.</p>
How is torch::add is implemented?

Krill: <p>Hi! Can anyone point to the source code of tensor addition. I found many math operations in <a href="https://aten/src/ATen/native/BinaryOps.cpp" rel="noopener nofollow ugc">BinaryOps.cpp</a> but I haven’t found any add or add_ functions specified in native_functions.yaml. Also, it’s hard to say what auto generated functions point to: the sequence goes through dispatch to findOp that searches in some lookup table.</p>
<p>Thanks in advance !</p>
How does broadcasting work when the input of torch.max is two tensors of different shape?

tan_chaolei: <p>How does broadcasting work when the input of torch.max is two tensors of different shape? For example, say a.shape=(2, 3) and b.shape=(2, 3), then what’s the meaning of torch.max(a[:, None, :], b)? It 's confusing to me that element-wise maximum is applied on two tensors of different shape.</p>

Matias_Vasquez: <p>Let´s consider the following example</p>
<pre><code class="lang-python">a = torch.arange(6).reshape(2, 3)
# tensor([[0, 1, 2],
#         [3, 4, 5]])

b = torch.ones(2, 3) * 3
# tensor([[3., 3., 3.],
#         [3., 3., 3.]])
</code></pre>
<p>In your example, the shape of  <code>a</code> is modified as following</p>
<pre><code class="lang-python">print(a[:, None, :].shape)
# torch.Size([2, 1, 3])
</code></pre>
<p>And as you correctly mentioned, it will result in broadcasting, thus the new dimension will be repeated. If we print only the result of adding this dimension and repeating it to the resulting shape of your example we get ↓, where dim 1 is being repeated.</p>
<pre><code class="lang-python">print(a[:, None, :].repeat(1, 2, 1))
# tensor([[[0, 1, 2],
#          [0, 1, 2]],
#
#         [[3, 4, 5],
#          [3, 4, 5]]])
</code></pre>
<p>If we now perform the operation that you suggested of max, the tensor <code>b</code> is compared to each of the “batches”, where each “batch” has the same size as <code>b</code>. In this trivial example we can see that the first batch is all smaller than <code>b</code>, thus the maximum will be <code>3</code>. The second “batch” is either equal or bigger than <code>3</code>.</p>
<pre><code class="lang-python">c = torch.max(a[:, None, :], b)
# tensor([[[3., 3., 3.],
#         [3., 3., 3.]],
#
#        [[3., 4., 5.],
#         [3., 4., 5.]]])
</code></pre>
<p>Hope this is clear and helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
PyTorch runtime error "expected all tensors to be on the same device" when all tensors are on the same device

GraphitePearl: <p>I’m working on an inference script to quickly make predictions and get useful statistics. I decided it would be a good idea to get the predictions in batches and use CUDA to make the process much faster (inference on cpu takes more that 5 seconds so that’s not an option).</p>
<p>When I use the CPU to run my model, there is no problem, I get the output and no errors are thrown. However, when I transfer the model and its inputs to CUDA, I get the message that tensors are on different devices, which makes no sense to me.</p>
<pre><code class="lang-auto">device = torch.device('cuda:0')
sample = next(iter(dataloader))

img, _ = sample
output = model(img.unsqueeze(0))
# The line above works fine

model = model.to(device)
img_cuda = img.to(device)
output = model(img_cuda.unsqueeze(0))
# The line above causes the error
</code></pre>
<p>I have a feeling, something is not right with transfering the model to CUDA.</p>

InnovArul: <aside class="quote no-group" data-username="GraphitePearl" data-post="1" data-topic="149739">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/graphitepearl/48/48858_2.png" class="avatar"> GraphitePearl:</div>
<blockquote>
<p><code>model = model.to(device)</code></p>
</blockquote>
</aside>
<p>Are you explicitly handling movement of data inside <code>model</code> with <code>.to(), .cuda(), .cpu()</code> calls?</p>

GraphitePearl: <p>Thank you. That’s very clever idea, but there are no such statements inside the model. However, a simpler model causes no such errors and works amazingly quick on a GPU. I’ll have to carefully examine the more sophisticated model.</p>

GraphitePearl: <p>I found the fix, although I’m still not sure why the problem occurred. The model creates an instance of an ConvLSTM cell, which initializes a tensor that needed to be manually transferred to CUDA.</p>
Model always gives the same output with different inputs

Percy: <p>Hi,<br>
I am working on a project were I need to export my PyTorch model to LibTorch (to create an application). I traced the mode according to the PyTorch documentation and want to load it in C++. I have the following code:</p>
<pre><code class="lang-auto">torch::Tensor loadExample(const char *path)
{

  // a grayscale picture, 280x280
  int picture [280][280];
  // load data from file, loadData is a custom method
  picture = loadData(path);

  // array to tensor 
  torch::Tensor dataTensor = torch::from_blob(picture, {280, 280}, at::kByte);
  // add batch-dim so its size is 1, 280, 280
  dataTensor = dataTensor.to(at::kFloat).unsqueeze(0);

  return dataTensor;
}

int main(int argc, const char *argv[])
{


  // argv[1] == &lt;path-to-exported-script-module&gt;
  // argv[2] == &lt;path-to-data&gt;

  torch::jit::script::Module model;
  model = torch::jit::load(argv[1]);

  // get the input
  std::vector&lt;torch::jit::IValue&gt; input;
  input.push_back(loadExample(argv[2]));

  at::Tensor output = model.forward(input).toTensor();

  std::cout &lt;&lt; output;
}
</code></pre>
<p>It loads a serialized picture as sample input. Loading the image works 100%, but it does not matter what the input is, the model always returns the same output. Only changing the weights changes the output, which is completely different to the output of the traced PyTorch model. Any ideas what the error is? I have been trying to figure out what’s wrong for hours…</p>

eqy: <p>I would first try some sanity checks to ensure that the tensor the model sees is changing between evaluations (e.g., with something like <code>torch.sum</code> across all the values). If this is the case and the model is a pretrained one, I would check that the corresponding data augmentations are applied such as any normalizations.</p>

Percy: <p>Thanks for your advice, it was a problem with <code>torch::from_blob</code>. It finally worked when I inserted the values via for-loops</p>
Customizable classification

Sylvain_Ard: <p>Hello all !<br>
I need to classify images in supervised training, I have 500 classes in my training dataset with sometimes tens of images sometimes thousands of images. Have you a code in ResNet 101 or 152 which could do the job (so customizable) and which take account of classes of different number of images please ? (an expert told me about class_weights).<br>
Thank you<br>
Best regards</p>

Sylvain_Ard: <p>Hello !<br>
Noone has answer ?<br>
Thank you<br>
Best regards</p>

eqy: <p>I’m not sure the number of classes affects the model definition itself beyond specifying <code>num_classes</code> when declaring the model: <a href="https://pytorch.org/vision/stable/_modules/torchvision/models/resnet.html" class="inline-onebox" rel="noopener nofollow ugc">torchvision.models.resnet — Torchvision 0.12 documentation</a>.</p>
<p>If you are referring to splitting the dataset for evaluation, you might need to write a little custom code to do the stratified sampling e.g., <a href="https://discuss.pytorch.org/t/how-to-do-a-stratified-split/62290?msclkid=cdbabbe8c1bb11ecbfa1268a15c4fbd4">How to do a stratified split - PyTorch Forums</a></p>

Sylvain_Ard: <p>I don’t talk about the number of classes but the number of images per class wich vary a lot</p>
Pytroch 1.7.1 cannot load checkpoint saved by pytorch1.10

npkhgl: <p>I have a checkpoint saved by pytorch1.10, and errors occurred when I loaded it with my own environment (pytorch1.7.1),</p>
<pre><code class="lang-auto">ModuleNotFoundError: No module named 'torch.torch_version'
</code></pre>
<p>I know that updating pytorch can solve this problem, but whether there is a way to load it by pytorch1.7.1?</p>

ptrblck: <p>PyTorch does not guarantee forward compatibility, so the best approach would be to update PyTorch to the latest or same version.<br>
Even if you could solve this particular issue by manually defining the missing <code>.torch_version</code> attribute, you could still run into more errors.</p>

npkhgl: <p>Alright, I’m just puzzled about why <code>torch.load</code> will call <code>torch.torch_version</code>. I loaded this checkpoint by pytorch1.10 and found that some meta information, including <code>torch_version</code>, was saved in it. Maybe this is the reason why I will meet this strange error <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20">.<br>
Anyway, thanks for you reply.</p>
[CUDA out of memory] How to reserve memory in GPU?

C_J: <p>I get CUDA out of memory error when I have enough memory than it needs.<br>
FYI, I’m using <strong>Dataparallel</strong>. (not Distributed)</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 3.62 GiB (GPU 3; 47.99 GiB total capacity; 13.14 GiB already allocated; 31.59 GiB free; 13.53 GiB reserved in total by PyTorch)</code></p>
<p>I’ve checked hundred times to monitor the GPU memory using nvidia-smi and task manager, and the memory never goes over 33GiB/48GiB in each GPU. (I’m using x4). It tries to allocate 3.62GiB, but there are 13.53GiB free memory.</p>
<p>Now I’m suspecting that it’s because pytorch didn’t reserve enough memory when there are more available, and hoping if I could force pytorch to reserve more memory manually.<br>
This problem is really painful and I saw there were some similar cases, but found no working solution.<br>
Any advice/suggestions would be appreciated.</p>
<p>Thanks for reading!</p>

ptrblck: <p>Is this a double post from <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-out-of-memory-i-have-larger-gpu-memory-than-it-needs/149770">here</a> or how do they differ?</p>
Backward became extremely slow when batchsize increased

Ferd: <p>I changed the batchsize of input(from 1 to 2), however, loss.backward() became extremely slow(from 0.4s to 1s).<br>
I am really confused, any help would be appreciated.</p>

AlphaBetaGamma96: <p>Can you share a minimal reproducible example of this behavior?</p>

Ferd: <p>Thanks for your reply!</p>
<p>Scenario one:</p>
<pre><code class="lang-auto"># depths and poses are the output of NN
# mr_loss is the loss function

forward_depths, backward_depths = torch.split(depths, [b, b], dim=0)
forward_poses, backward_poses = torch.split(poses, [b, b], dim=0)
forward_mr_loss, forward_flows = mr_loss(forward_depths, forward_poses, img_seq, intrinsics)
backward_mr_loss, backward_flows = mr_loss(backward_depths, backward_poses, img_seq.flip(dims=[1]), intrinsics)
all_mr_loss = (forward_mr_loss + backward_mr_loss) / 2

all_mr_loss.backward()
</code></pre>
<p>Scenario two:</p>
<pre><code class="lang-auto">all_mr_loss, flows = mr_loss(depths, poses, torch.cat([img_seq, img_seq.flip(dims=[1])], dim=0), intrinsics.repeat(2, 1, 1))

all_mr_loss.backward()
</code></pre>
<p>Both scenarios have the same <strong>all_mr_loss</strong> value, actually the only different factor is the size of input. In scenario one, the input was split into two parts, and the results were added up.</p>
<p>However, the backward time is 0.4s in scenario one but 1s in scenario two.</p>

eqy: <p>Would it possible to share a snippet that is executable (e.g., with dummy/random tensors if the dataset is not available)? It would also be useful to see how the timing is being done to pinpoint the issue.</p>
[RuntimeError: CUDA out of memory] I have larger gpu memory than it needs

C_J: <p>I have made a similar post, but I couldn’t get the answer I wanted so I’m re-posting.<br>
I’m using DataParallel using 4 gpus, and I get these out-of-memory error:</p>
<p>[case 1]<br>
RuntimeError: CUDA out of memory. Tried to allocate 3.62 GiB (GPU 0; 47.99 GiB total capacity; 24.39 GiB already allocated; 20.41 GiB free; 24.71 GiB reserved in total by PyTorch)</p>
<p>[case 2]<br>
RuntimeError: CUDA out of memory. Tried to allocate 3.62 GiB (GPU 2; 47.99 GiB total capacity; 13.14 GiB already allocated; 31.59 GiB free; 13.53 GiB reserved in total by PyTorch)</p>
<p>Both cases, my gpu has enough memory but it cannot allocate much smaller memory.<br>
In some other post, I saw how “available” free memory is calculated :<br>
free memory = reserved memory - allocated memory<br>
This makes me suspecting that maybe it’s because Pytorch didn’t reserve enough memory although there are more available. But then I don’t know how to make pytorch reserve enough (more) memory since it automatically assigns.</p>
<p>Any advice or suggestions will be very appreciated.<br>
Thanks for reading!</p>

eqy: <p>Could you check the output of <code>nvidia-smi</code> in these cases? Would it be possible that another process or instance of PyTorch is also using memory?</p>

C_J: <p>I am monitoring the gpu memory using nvidia-smi, and it never reaches even close to the maximum.<br>
up to 33GiB / 48GiB. All of gpus.</p>

eqy: <p>Are there any other processes contending for GPU memory or just the single PyTorch instance here?</p>

C_J: <p>I was only running a pytorch.</p>
Compile without cmakelist.txt

complexhhs: <p>I want to compile C++ program without “CMakeList.txt” compiling method as below guide.<br>
(<a href="https://pytorch.org/cppdocs/installing.html" class="inline-onebox" rel="noopener nofollow ugc">Installing C++ Distributions of PyTorch — PyTorch master documentation</a>)</p>
<pre><code class="lang-auto">mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch ..
cmake --build . --config Release
</code></pre>
<p>Is there anyway to write Makefile to compile below simple code?</p>
<pre data-code-wrap="c++"><code class="lang-nohighlight">//torch_test.cpp
#include&lt;torch/torch.h&gt;
#include&lt;torch/script.h&gt;
#include&lt;iostream&gt;

int main()
{
    std:: cout &lt;&lt; "Torch header successfully called" &lt;&lt; std::endl;
    return 0;
}
</code></pre>
<p>I’ve tried all the way when I compile using gcc above code. But I got this error message in Ubuntu.</p>
<pre><code class="lang-auto">gcc -I&lt;libtorch path&gt; torch_test.cpp
torch_test.cpp:1:10: fatal error: torch/torch.h: No such file or directory
 #include &lt;torch/torch.h&gt;
          ^~~~~~~~~~~~~~~
compilation terminated.
</code></pre>
<p>Any ideas?</p>
Weight initialization in Pytorch?

raesol: <p>I am new to PyTorch and machine learning in general and I wanted to make sure I understand weight initialization correctly. as I understand weight initialization can be done to all layers but if we choose to do it ourselves, we use the   nn.init.any_type_of_weight_initialization(moudle.weight, *args). so I understand that the purpose is to replace the already initialized default weights(by module.weight) by the any_type_of_weight_initialization method we want. but while trying it, I found both module.weight and the one we initialized have the same weight entries. so my questions are:</p>
<ol>
<li>what is the default weight initialization method used in module.weight?</li>
<li>do we really replace the already initialized weights by other methods or I am misunderstanding the whole process? thanks in advance</li>
</ol>

ptrblck: <ol>
<li>
<p>The default initialization depends on the used <code>nn.Module</code> and can be found in its <code>reset_parameters()</code> method. E.g. <a href="https://github.com/pytorch/pytorch/blob/333da3eaef82ea9e573fd1f9c134587ee2222401/torch/nn/modules/linear.py#L92-L100">here</a> for <code>nn.Linear</code>.</p>
</li>
<li>
<p>You are understanding the workflow correctly. The model initialization via <code>model = MyModel()</code> will initialize all parameters using their default method (unless you’ve already manipulated the <code>MyModel.__init__</code> to use a custom parameter initialization). Afterwards you are usually calling into a custom <code>weight_init</code> method via <code>model.apply()</code>.</p>
</li>
</ol>
Is there way to check pretrained model features?

BongHoe_Koo: <p>Hi everyone.</p>
<p>Is there way to check features from pretrained model (torch.jit.load method) and use???<br>
i want to combine pretrained backbone + centernet detector (FPN)</p>
<p>Thanks<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d16efff2f4de022dbfe7688a122266c982208cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d16efff2f4de022dbfe7688a122266c982208cc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d16efff2f4de022dbfe7688a122266c982208cc_2_690x192.png" alt="image" data-base62-sha1="1RNlJ9xTpQiHnNkrniZNGqEBc28" width="690" height="192" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d16efff2f4de022dbfe7688a122266c982208cc_2_690x192.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d16efff2f4de022dbfe7688a122266c982208cc_2_1035x288.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d16efff2f4de022dbfe7688a122266c982208cc_2_1380x384.png 2x" data-dominant-color="2E2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1382×386 25 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75f0d175af2a8d312b8b2ff1a7e4cd8628f0a33f.png" data-download-href="https://discuss.pytorch.org/uploads/default/75f0d175af2a8d312b8b2ff1a7e4cd8628f0a33f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75f0d175af2a8d312b8b2ff1a7e4cd8628f0a33f_2_690x140.png" alt="image" data-base62-sha1="gPlOvysvK9hFjYu03hKeP39zE3d" width="690" height="140" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75f0d175af2a8d312b8b2ff1a7e4cd8628f0a33f_2_690x140.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75f0d175af2a8d312b8b2ff1a7e4cd8628f0a33f_2_1035x210.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/5/75f0d175af2a8d312b8b2ff1a7e4cd8628f0a33f.png 2x" data-dominant-color="363437"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1101×224 28.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>