akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Joining multiple neural nets in non-sequential model

username: Hadi_Nayebi
cooked: <p>Hey, I hope everyone is having a great day.<br>
I have the following setup:</p>
<p>state = netA(input)<br>
output1=netB(state)<br>
output2=netC(state)</p>
<p>I can define target values for output1 and output2. How would you approach training the netA?<br>
I am trying to combine the gradient on the input of netB and netC (state) as the loss to train netA.<br>
or should I use them separately?!!!<br>
how can I get the gradient on the input of netB and netC?</p>
<p>Best,<br>
hn</p>

username: ptrblck
cooked: <p>You could calculate the losses from using <code>output1</code> and <code>output2</code>, accumulate them, and call <code>loss.backward()</code> to calculate the gradients for all 3 models. If you don’t want to train <code>netB</code> and <code>netC</code>, you could set the <code>.requires_grad</code> attribute if their parameters to <code>False</code>.</p>
<p>In case you want to get the gradients in <code>state</code>, you could call <code>state.retrain_grad()</code>, which would then  allow you to access its <code>.grad</code> after the <code>backward</code> call.</p>

username: Hadi_Nayebi
cooked: <p>Thank you ptrblck. I will try your suggestions and get back to you if I had more questions, I am a newbie, and things are slow on my side.</p>
Left multiplying using parameter not possible?

username: ADONAI_TZEVAOT
cooked: <p>I have an input x of dimension 1x 2, batch size = 128, … Hence input gets passed in batches as 128x2. I have 2 parameters L_p(dimension = 1 x 1) and R_p (dimension = 2 x 2).  The operation <code>x @ R_p</code> works but the operation <code>L_p @ x</code> is throwing error that matrix dimensions are not satisfied since x is passed as 128 x 2. But actually x is 1x2 , so how do I make it work? Please help!</p>

username: ptrblck
cooked: <p><code>x @ R_p</code> will apply a matrix multiplication with the shapes:</p>
<pre><code class="lang-python">[128, 2] @ [2, 2] = [128, 2]
</code></pre>
<p>which are the expected shapes.<br>
However, <code>L_p @ x</code> tries to execute:</p>
<pre><code class="lang-python">[1, 1] @ [128, 2]
</code></pre>
<p>which is invalid for a matmul.<br>
I’m not sure what the expected output shape is, but assuming that you would like to broadcast <code>L_p</code> in the batch dimension, you could use:</p>
<pre><code class="lang-python">out = L_p.expand(x.size(0), 1, -1) @ x.unsqueeze(1)
</code></pre>
<p>which would create an output of <code>[128, 1, 2]</code>.</p>
RunTimeError on running transfer learning

username: Santhoshnumberone
cooked: <p>Running this file <a href="https://github.com/timesler/facenet-pytorch/blob/master/examples/finetune.ipynb" class="inline-onebox" rel="noopener nofollow ugc">facenet-pytorch/finetune.ipynb at master · timesler/facenet-pytorch · GitHub</a></p>
<p>I want to train only the final layer of</p>
<pre><code class="lang-auto">resnet = InceptionResnetV1(
    classify=True,
    pretrained='vggface2',
    num_classes=len(dataset.class_to_idx)
).to(device)
</code></pre>
<p>I did freeze all layers but unfroze last layer</p>
<pre><code class="lang-auto">#Freeze all layers
for param in resnet.parameters():
    param.requires_grad = False

#Unfreeze the last layer
for param in resnet.logits.parameters():
    param.requires_grad = True 
</code></pre>
<p>I tried running the code I get this error</p>
<pre><code class="lang-auto">Initial
----------
Valid |     2/2    | loss:    6.2345 | fps:  182.8512 | acc:    0.0000   

Epoch 1/100
----------
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-45-49d240c83dad&gt; in &lt;module&gt;()
     19         resnet, loss_fn, train_loader, optimizer, scheduler,
     20         batch_metrics=metrics, show_running=True, device=device,
---&gt; 21         writer=writer
     22     )
     23 

2 frames
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--&gt; 147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    148 
    149 

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>

username: ptrblck
cooked: <p>The <code>logits</code> layer will only be used, if <code>self.classify</code> is set to <code>True</code> as seen <a href="https://github.com/timesler/facenet-pytorch/blob/54c869c51e0e3e12f7f92f551cdd2ecd164e2443/models/inception_resnet_v1.py#L298-L299">here</a>, so I guess this might not be the case.<br>
If so, this layer won’t be used while all other parameters are frozen, which would yield this error.</p>
IOPub Problem when saving multiple PyTorch models

username: Yousra_Chahinez_HADJ
cooked: <p>I implemented the pix2pix model and saved the result of each epoch for the discriminator and the generator in the same file by following:</p>
<p><a href="https://pytorch.org/tutorials/recipes/recipes/saving_multiple_models_in_one_file.html" rel="noopener nofollow ugc">SAVING AND LOADING MULTIPLE MODELS IN ONE FILE USING PYTORCH</a></p>
<p>but when the model is training a warning arise said that:</p>
<pre><code class="lang-auto">IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)
</code></pre>
<p>The model is still training but I want to know that is training properly, and thanks in advance.</p>
<p>Note: I’m using a Kaggle notebook.</p>
Cannot import name 'VOCSegmentation' from 'torchvision.datasets'

username: 887574002
cooked: <p>Hi, I was working on Voc dataset. But when I tried to download the dataset I got the following error.</p>
<p><code>cannot import name 'VOCSegmentation' from 'torchvision.datasets</code></p>
<p>I found that the version of torchvision that I use is : <code>0.2.0</code> and Voc dataset is available in torchvision with version: 0.6.1</p>
<p>Then I removed torchvision from site-packages and installed it using the following command:</p>
<p><code>conda install torchvision==0.6.1 cudatoolkit=10.2 -c pytorch</code></p>
<p>But when I install the torchvision of version <code>0.6.1</code>, pytorch is also reinstalled automatically with version <code>1.5.1</code>, while before reinstalling torchvision I was working with updated version of pytorch.</p>
<p>Is there any way that I can install the newer version of torchvision but I do not have to downgrade to older version of pytorch?</p>

username: ptrblck
cooked: <p>Yes, you could install the latest stable <code>torchvision</code> (and PyTorch) release by removing the version specification:</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
</code></pre>

username: 887574002
cooked: <p>Ok, thank you for your answer.</p>
Is combining 2 models performing 2 tasks into 1 model accelerates inference time?

username: vauthority
cooked: <p>Hello everyone! I’m recently working on my graduate project and my task is to have the model perform 2 tasks ( tracking and depth estimation) at the same time which is very slow. Is that combining 2 models into 1 accelerate the inference time? Thanks</p>

username: ptrblck
cooked: <p>In case the two models would reuse a common internal model, you could see a speedup due to the decreased number of operations needed the model execution.<br>
However, it depends on your actual use case, if this would be a valid option.</p>

username: vauthority
cooked: <p>Thanks ptrblck for replying to me! But if two models don’t reuse a common internal model what’s the solution for this problem?</p>

username: ptrblck
cooked: <p>Assuming your “problem” is the speed: you could profile your workload, check where the bottlenecks are, and try to avoid/optimize them. Also, if you have two independent models, you could try to run them in parallel on different devices, if possible.</p>
How to unite Linear, Batchnorm, ReLU, dropout?

username: Arthur_Zakirov
cooked: <p>Hello everyone,</p>
<p>what is a valid way to construct a Hierarchical NN Architecture of subcomponents.<br>
The goal is accomplish something like this:</p>
<pre><code class="lang-auto">class FeedForwardLayer( nn.Module ):
     def __init__(self):
         super().__init__()

         self.lin = nn.Linear( ... ) 
         self.bn = nn.BatchNorm1d( ... )
         self.act = nn.ReLU( )
         self.do = nn.dropout( ... )

    def forward( self , x ):
        return self.do(   self.act(    self.bn(    self.lin(  x )  )  )  )

class MultiLayerNetwork( nn.Module  ):
    def __init__( self ):
         super().__init__()
         self.layer_1 = FeedForwardLayer()
         ...
         self.layer_n = FeedForwardLayer()

    def forward(self, x ):
         hidden_1 = self.layer_1( x )
         hidden_2 = self.layer_2( hidden_1 )
         ...
         return hidden_n

</code></pre>
<p>For the purpose of simplicity there are some pseudo code elements,<br>
no need to correct them.<br>
My question is regarding the structure.</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>Your code looks alright, so I’m unsure if you are seeing any issues with it or just would like to have some feedback?</p>
How can I compute the W and B in notron software?

username: Deep
cooked: <p>I use notron to visualize my model, but I dont know how to compute the W(weights) and B(bias) ,I use formula, but it dont work. as you can see, how to compute the second Conv structure from the first Conv stucture?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c5dfd21aa9e45bba0dbea1aecf5cc0305435acd.png" data-download-href="https://discuss.pytorch.org/uploads/default/3c5dfd21aa9e45bba0dbea1aecf5cc0305435acd" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c5dfd21aa9e45bba0dbea1aecf5cc0305435acd.png" alt="image" data-base62-sha1="8C218jmoCm39JqotuYGPKkvWFLD" width="341" height="500" data-dominant-color="E5E7EB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">359×526 9.95 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The <code>in_channels</code> of the second convolution would be defined by the <code>out_channels</code> of the previous one, which defines the size of <code>dim1</code> (32 in your example). Besides that you can pick other parameters as you wish.</p>
Torch profiler strack trace is cut off

username: agt
cooked: <p>I’m using torch 1.7.0 and when I run</p>
<pre><code class="lang-auto">prof.key_averages(group_by_stack_n=10).table(sort_by="cpu_time_total", row_limit=10)
</code></pre>
<p>It neither shows every level of the stack (it shows third party library functions, but not which functions I wrote that called them), nor does it show the full path or line number for the levels it does show, they cut off around 50 characters.</p>
<p>How can I get a full stack trace?</p>
Efficient net as backbone network in few shot learning

username: treadstone
cooked: <p>I am new to PyTorch, so not sure how to use efficientNet as backbone CNN model for feature extraction, so that embeddings of images can be generated. As most of example on GitHub use 4 layer ConvNet so  I can not understand how to use same thing for large CNN model. Like there are implementation of efficient-net for Torch, so what steps I need to use them as feature extractor?<br>
I am using this efficient net code which implemeted network in PyTorch -</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/osmr/imgclsmob/blob/master/pytorch/pytorchcv/models/efficientnet.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/osmr/imgclsmob/blob/master/pytorch/pytorchcv/models/efficientnet.py" target="_blank" rel="noopener nofollow ugc">osmr/imgclsmob/blob/master/pytorch/pytorchcv/models/efficientnet.py</a></h4>
<pre><code class="lang-py">"""
    EfficientNet for ImageNet-1K, implemented in PyTorch.
    Original papers:
    - 'EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,' https://arxiv.org/abs/1905.11946,
    - 'Adversarial Examples Improve Image Recognition,' https://arxiv.org/abs/1911.09665.
"""

__all__ = ['EfficientNet', 'calc_tf_padding', 'EffiInvResUnit', 'EffiInitBlock', 'efficientnet_b0', 'efficientnet_b1',
           'efficientnet_b2', 'efficientnet_b3', 'efficientnet_b4', 'efficientnet_b5', 'efficientnet_b6',
           'efficientnet_b7', 'efficientnet_b8', 'efficientnet_b0b', 'efficientnet_b1b', 'efficientnet_b2b',
           'efficientnet_b3b', 'efficientnet_b4b', 'efficientnet_b5b', 'efficientnet_b6b', 'efficientnet_b7b',
           'efficientnet_b0c', 'efficientnet_b1c', 'efficientnet_b2c', 'efficientnet_b3c', 'efficientnet_b4c',
           'efficientnet_b5c', 'efficientnet_b6c', 'efficientnet_b7c', 'efficientnet_b8c']

import os
import math
import torch.nn as nn
import torch.nn.functional as F
import torch.nn.init as init
from .common import round_channels, conv1x1_block, conv3x3_block, dwconv3x3_block, dwconv5x5_block, SEBlock
</code></pre>

  This file has been truncated. <a href="https://github.com/osmr/imgclsmob/blob/master/pytorch/pytorchcv/models/efficientnet.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to efficiently do cross-entropy on masked output layer?

username: EvanZ
cooked: <p>I am using <code>TransformerEncoder</code> to train a model on sequences with a single masked value. Let’s say every sequence is <code>S=10</code> and batch size is <code>N=64</code>. The number of classes  <code>C</code> is very large, but for sake of this example, let’s make it <code>C=128</code>. So the output from the encoder is <code>SxNxC=10x64x128</code>. If I understand correctly, though, the idea is to only count the loss corresponding to the output index of the masked token in the input sequence eg some position <code>S_i=0...9</code>, so that the actual shape of the output I want to feed the loss function should be <code>64x128</code>. I was going to naively use a for loop, but I am thinking there is probably a clever way to do this in a vectorized fashion. I tried playing around with creating my own mask, but so far haven’t come upon a correct solution. Any advice appreciated!</p>
Result with CNN image

username: randinoo
cooked: <p>how I get the classification result with CNN as an image?</p>

username: eduardo4jesus
cooked: <p><a class="mention" href="/u/randinoo">@randinoo</a> what CNN architecture do you have in mind as an example? And, what do you mean to get a result as an image?</p>

username: eduardo4jesus
cooked: <aside class="quote no-group quote-post-not-found" data-username="randinoo" data-post="3" data-topic="122364">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/c5a1d2/48.png" class="avatar"> randinoo:</div>
<blockquote>
<p>classification</p>
</blockquote>
</aside>
<p>If the output is an image, the network you are use is not for classification. Your case is a segmentation problem. In your case, which image would you be looking for? <code>Distance map D</code>?</p>

username: randinoo
cooked: <p><a class="mention" href="/u/eduardo4jesus">@eduardo4jesus</a>  yes in this case is the segmentation, but who I find final change map image?<br>
distance =f1 -f2 who I find diantance in image??</p>

username: eduardo4jesus
cooked: <p>Can you provide the code for the network?</p>

username: randinoo
cooked: <p>how I display each time each pixel its label?</p>
<p>help me plzz</p>
Mapping of image with class label

username: aleemsidra
cooked: <p>Hey. I am completely new to PyTorch and have no idea where to begin with. I am working on a classification problem. It’s a 4 class classification problem. I have all the images in a single folder. I have a .csv file of the one-hot encoded representation corresponding to each image<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40e081295101e44d995497ae7506d7450b03547d.png" alt="ground_truth_labels" data-base62-sha1="9fVywzocgXUZF1A8Q8fSV9ofGI5" width="472" height="210"><br>
. The column name represents the actual class and rows contain corresponding one hot encoded representation like this [0 0 0 1]. How to map this representation with images?</p>
<p>If I map these things, then how would it be ensured that the images are passed with their corresponding labels to the neural network? Can someone please give me idea about how to proceed?</p>

username: eqy
cooked: <p>If you want to train with <code>CrossEntropyLoss</code>, you can define a simple dataset like</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, mapping):
        self.mapping = {filename : mapping[filename].index(1) for filename in mapping}
        self.filenames = self.mapping.keys()
        self.len = len(mapping)

    def __len__(self):
        return self.len

    def __getitem__(self, idx):
        return Image.open(self.filenames[idx]), self.mapping[self.filenames[idx]]
</code></pre>
<p>The ImageNet example: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at master · pytorch/examples · GitHub</a> is a good reference for how to train a classification model.</p>
Image resoultion

username: hesham_gabr
cooked: <p>How can I feed images with 1024*1024 pixels into pre-trained model Vgg16?<br>
how to modify that in pytorch code?</p>

username: eqy
cooked: <p>It depends on what you mean by feed images that are 1024*1024. If you are using a pretrained model without further finetuning, you likely want to simply resize the images to something around 256x256 and take a 224x224 center crop. If you decide not to resize or crop, the model should “work” in that there should not be any shape mismatches due to the use of average pooling with input images at 1024x1024, but the model accuracy will likely not be great due to the potential scale mismatch. Choosing the right resolution is actually a tricky problem (e.g., as studied in <a href="https://arxiv.org/abs/1906.06423" class="inline-onebox" rel="noopener nofollow ugc">[1906.06423] Fixing the train-test resolution discrepancy</a>).</p>
<p>Finally, you might need to consider at at 1024x1024, the model will likely use ~(1024/224)**2 or ~9 times the memory and computation due to the quadratic scaling of convolution activations.</p>

username: hesham_gabr
cooked: <p>the images are different in sizes &gt;1000<em>1000 and I was feed 224</em>224 sizes to the vgg16_bn model, the accuracy is 53%.<br>
I think if I’ll change the image size input the accuracy will higher than that? or NOT?<br>
and in code, I can’t change the input sizes of those images.<br>
how I modify the model to receive the 1024*1024 images.<br>
Thanks man</p>

username: eqy
cooked: <p>The important thing is that the scale of objects (e.g., how many pixels a cat or dog is) is close to what was shown during training time. So if you just change the input size without finetuning or retraining, the accuracy will most likely go down.</p>
<p>You don’t need to modify the model for higher resolution, but some kind of finetuning or retraining is probably needed.</p>
Batch Normalization over which dimension?

username: Arthur_Zakirov
cooked: <p>Hello everyone,</p>
<p>over which dimension do we calculate the mean and std? Is it over the hidden dimensions of the NN Layer, or over all the samples in the batch for every hidden dimension separately?<br>
In the paper it says we normalize over the batch.<br>
In torch.nn.BatchNorm1d hower the input argument is “num_features”. Why would we calculate the mean and std over the different features instead of the different samples?</p>
<p>Thanks in advance</p>

username: eqy
cooked: <p>You are correct that <code>num_features</code> corresponds to the “hidden dimension” rather than the batch size. However, if you think about this from the perspective of what statistics batchnorm needs to track, this makes sense. For example, for a hidden dimension of size 512, batchnorm needs to keep track of mean and variance for <em>each</em> of the 512 dimensions. Here, <code>num_features</code> is really just telling the module how much storage it needs to track its stats. Note that this size doesn’t depend on the batch size as taking the mean <em>reduces</em> across the batch dimension.</p>

username: Arthur_Zakirov
cooked: <p><a class="mention" href="/u/eqy">@eqy</a><br>
Thanks for the answer. Assuming I have sequential data of the shape (bs, dim, seq_len), does BatchNorm1d calculate the mean and std of the batch seperately for every timestep, or are the timesteps somehow merged as well?</p>
SSIM output is negative

username: OuisYasser
cooked: <p>hello everyone,<br>
So, I implemented the ssim loss with pytorch. it was working one month ago. Now, when I tried to execute the code, it returns that the ssim output is negative and should be at least 0.<br>
the code is:</p>
<pre><code class="lang-auto">&gt; !pip install piqa
&gt; from piqa import ssim
&gt; 
&gt; class SSIMLoss(ssim.SSIM):
&gt;     def forward(self, x, y):
&gt;         return 1. - super().forward(x, y)
&gt; criterion_ssim = SSIMLoss().cuda() # .cuda() if you need GPU support
</code></pre>

username: eqy
cooked: <p>Can you share the explicit SSIM calculation used and an example input pair that produces a negative result?</p>
Concatenate same vector to all raws of matrix

username: lightbooster
cooked: <p>Hey, i would like to know how i can concatenate two tensors like this:</p>
<pre><code class="lang-auto">t1 = torch.rand(2, 10, 512)
t2 = torch.rand(2, 768)
</code></pre>
<p>and get tensor like this:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.Size([2, 10, 1280])
</code></pre>
<p>Let’s assume that shapes are:</p>
<pre><code class="lang-auto">t1_shape = (batch_size, sequence_len, embedding_dim)
t2_shape = (batch_size, embedding_dim)
</code></pre>
<p>I want to concatenate these tensors along the <code>embedding_dim</code>, so every different tensor in <code>sequence_len</code> dimension will be concatenated with the <strong>same</strong> t2 Tensor.</p>
<p>As a solution i see:</p>
<pre><code class="lang-auto">t2 = t2.unsqueeze(1)
t2.size()
&gt;&gt;&gt; torch.Size([2, 1, 768])
t2 = torch.cat((t2,) * 10, dim=1)
t2.size()
&gt;&gt;&gt; torch.Size([2, 10, 768])
torch.cat((t1, t2), dim=2)
</code></pre>
<p>But i’m afraid this approach costs memory when duplicating t2 Tensor 10 times (in reality it can be much more).<br>
Is there any memory efficient solution?<br>
Thank you!</p>

username: tom
cooked: <p>You could use <code>t2.unsqueeze(1).expand(-1, 10, -1)</code>. This will take no extra memory (before the cat, that is) as it is a view (print stride() and shape to see what is going on) and the 10-dimension is just stride 0 (i.e. all copies in the same location).</p>
<p>Best regards</p>
<p>Thomas</p>

username: lightbooster
cooked: <p>That’s what i was looking for!<br>
Thank you</p>
Best Practices for Storing/Accessing Preprocessed Data

username: J_Johnson
cooked: <p>Hey guys,</p>
<p>I’m looking for suggestions on how to speed up process times when storing/accessing large amounts of data for training. Presently, I am using Pandas DataFrames saved to csv. Then I load just the part of the csv I need at any given time in the CustomDataset.<br>
But this has been quite slow, in my opinion, as the csv file is over 300mb(might be bigger later, too). Loading the entire file and accessing the parts I need via .iloc is even slower. So I wanted to check what you all use. I see there are a few options:</p>
<ol>
<li>Pandas DataFrames</li>
<li>Numpy Arrays</li>
<li>Tensors saved to .pt files</li>
<li>???</li>
</ol>
<p>What have you found to work best for performance?</p>

username: eqy
cooked: <p>It likely depends on the particular bottlenecks of your system. If there is a large amount of system memory, but limited file I/O, you might want to keep the entire file in memory at all times. If there isn’t much system memory, but sufficient file I/O you could split the file. The actual file format may not matter as much as the basic loading algorithm design.</p>

username: J_Johnson
cooked: <p>What I ended up doing that worked quite well was split the data between files with training samples of 1000 each quite easily by setting the filename to <code>'data-'+str(i//1000)+'.csv'</code>. Since the data has 180 points of data per training sample, the csv file sizes are just a couple mb each. But if your training samples are larger, the same would work by changing 1000----&gt;100.</p>
<p>Just make sure to print “i” at the end of your preprocessing script just to make sure you know what to set the <code>__len__</code> to in your CustomDataset.</p>
How to train Centernet model for an instance segmentation?

username: PytorchLearning
cooked: <p>Hi all,</p>
<p>Can someone please show me how to train Centernet model for instance segmentation.</p>
<p>I have well understood the Centernet model, but I didn’t understand how to train this model for instance segmentation.</p>
<p>Can someone show me an example?</p>
<p>Thanks in advance</p>

username: PytorchLearning
cooked: <p>Hi all,</p>
<p>Can someone explain to me how to code the training part with pytorch for the Centernet model for instance segmentation.</p>
<p>Thanks in advance.</p>

username: Cohee514
cooked: <p>Hi, I’m trying to do instance segmentation on small objects like a baseball (30x30) in I’ve tried training Mask RCNN following the balloon example, but little success. CenterNet does 28.9 APs which is close to the state of the art PANet. detection, batch them and feed into a lightweight semantic segmentation model.</p>
Colab crashes after torch.cat

username: sakharok13
cooked: <p>I’m just wondering why colab notebook crashes after running this code:</p>
<pre><code class="lang-auto">torch.cat( [torch.mean(torch.tensor([1.])), torch.mean(torch.tensor([1.]))])
</code></pre>
<p>Can someone help with this ?</p>

username: eqy
cooked: <p>This is because <code>mean</code> will implicitly reduce the first dimension and produce a tensor (really a scalar) without dimensions and <code>cat</code> expects at least a single dimension.<br>
You can work around this with</p>
<pre><code class="lang-auto">torch.cat( [torch.mean(torch.tensor([1.]), dim=0, keepdim=True), torch.mean(torch.tensor([1.]), dim=0, keepdim=True)])
</code></pre>
How do I squeeze a layer's dimensions down inside of nn.Sequential?

username: torch_user_92
cooked: <p>Hi, I have this neural net:</p>
<pre><code class="lang-auto">        self.network = nn.Sequential(
                        nn.Conv2d(in_channels=30, out_channels=1, kernel_size=(1, 1), stride=1),
                        nn.ReLU(),
                        nn.Flatten(),
                        nn.Linear(400,1),
                    )
</code></pre>
<p>The input size is 100 rows x 30 columns x 1 x 1. Before passing it through the Linear layer, I need it to be a completely flattened 100 number array. Squeeze() does the trick but I can’t insert it into nn.Sequential for some reason. nn.Flatten() brings it down to 400 x 1 but I need it to be essentially 400 x 0 for it to work.</p>
<p>Thanks</p>

username: sio277
cooked: <p>Modifying <code>nn.Flatten()</code> to <code>nn.Flatten(start_dim=0)</code> should work.</p>
What is the batched version torch.diagflat?

username: ProGamerGov
cooked: <p>So, I am trying to modify some of my code so that it supports the batch dimension. One of the lines uses <code>torch.diagflat</code>, and I was wondering what would be the batched version of it?</p>
<p>I see there are the <code>torch.diag</code> and <code>torch.diagonal</code> functions, but it’s not clear if they replicate <code>torch.diagflat</code>?</p>
<pre><code class="lang-auto">import torch
x = torch.randn(2, 3) # batch of 2
print(torch.diagflat(x).shape)
# size is torch.Size([6, 6]) instead of torch.Size([2, 3, 3])
</code></pre>

username: tom
cooked: <p>You can use <a href="https://pytorch.org/docs/stable/generated/torch.diag_embed.html" rel="noopener nofollow ugc"><code>torch.diag_embed</code></a> with <code>torch.view(batch_size, -1)</code> as the input. If your tensor is not necessarily contiguous, you can use <code>torch.resize</code> instead of <code>torch.view</code>.</p>
<p>Diagonals seems to be one of the bits of the numpy api that isn’t thought out terribly well w.r.t. being flexible/intuitive for use-cases like batching (see the default dimension behaviour for <code>numpy.diagonal</code> always seemed odd to me, too)…</p>
<p>Best regards</p>
<p>Thomas</p>

username: ProGamerGov
cooked: <p>Thank you for the help!</p>
Integrate libtorch into a C program

username: lhc
cooked: <p>I have tried to integrate ffmpeg with libtorch, but  found  that libtorch can not compiled  by C. Did someone do this work sucessfully?</p>

username: tom
cooked: <p>I don’t think PyTorch supports that.<br>
You would likely need to define a wrapper library with extern “C” functions and compile that in C++. The wrapper can then be called from C.</p>
<p>Best regards</p>
<p>Thomas</p>
Save checkpoint every step instead of epoch

username: ngoquanghuy
cooked: <p>My training set is truly massive, a single sentence is absolutely long. An epoch takes so much time training so I don’t want to save checkpoint after each epoch. Instead i want to save checkpoint after certain steps.<br>
Can I just do that in normal way?</p>

username: ptrblck
cooked: <p>Yes, you can store the <code>state_dict</code>s whenever wanted.</p>

username: ngoquanghuy
cooked: <p>Thanks sir!<br>
But my goal is to resume training from the last checkpoint (checkpoint after curtain steps).<br>
With <code>epoch</code>, it’s so easy to continue training with several more epochs. But with <code>step</code>, it is a bit complex.<br>
Could you please give any snippet?</p>

username: ptrblck
cooked: <p>In case you want to continue from the same iteration, you would need to store the model, optimizer, and learning rate scheduler <code>state_dict</code>s as well as the current epoch and iteration. Assuming you want to get the same training batch, you could iterate the <code>DataLoader</code> in an “empty” loop until the appropriate iteration is reached (you could also seed the code properly so that the same “random” transformations are used, if needed).</p>

username: ngoquanghuy
cooked: <p>Thanks sir!<br>
I got it.</p>
Target 1 is out of bounds

username: WilliManFR
cooked: <p>Hello,<br>
I have a problem that I don’t understand. It concerns the CrossEntropyLoss function. I am currently working on segmentation of a certain type of pattern in medical images. For this, I have built an architecture that returns after a softmax an array of output images of the form: [N, C, H, W] where N is the size of my batch, C the number of channels in each image and H, W the height and width of my image respectively. However, when I enter something of the form [10, 1, 240, 240] for my outputs and [10, 240, 240] for my targets, it always returns:</p>
<blockquote>
<p>IndexError: Target 1 is out of bounds.</p>
</blockquote>
<p>I’ve seen many things on the forums. Each time it mentions “classes” but I don’t understand what that means. It seems to correspond to my number of channels for my output images (in my case 1 because they are 2D binary images) however it doesn’t work. I think I’m missing something.</p>
<p>Could someone please enlighten me?</p>
<p>I am attaching the few lines in question.</p>
<pre><code class="lang-auto">        # inputs =&gt; [10, 3, 240, 240]
        outputs = segnetModel(images)

        # outputs =&gt; [10, 1, 240, 240]
        # targets =&gt; [10, 240, 240]
        loss = cost(outputs, labels)
</code></pre>
<p>I also found a question on the forum that is very close to what I wanted but still talks about “classes” : <a href="https://discuss.pytorch.org/t/the-cost-function-for-semantic-segmentation/27668" class="inline-onebox">The cost function for semantic segmentation?</a></p>
<p>Thank you very much for your help in advance.</p>

username: ptrblck
cooked: <p>As you’ve already pointed out the issue is created by the missing “class dimension” in the output.<br>
In a multi-class segmentation use case <code>nn.CrossEntropyLoss</code> expects the model output to have the shape <code>[batch_size, nb_classes, height, width]</code>, where each “channel” represents an activation maps, which gives the logits (unnormalized probabilities) for each pixel location.<br>
E.g. if you are working on a binary segmentation you could treat it as a 2-class multi-class segmentation using an output in the shape <code>[batch_size, 2, height, width]</code>, where the first channel would correspond to the logits of each pixel for class0 and the second channel for class1, respectively.</p>
<p>In your current code you are using a single output channel, so that your model would only be able to predict class0, which makes it a bit useless.</p>
<p>In any case, since you are working on a binary segmentation, you could also keep the single output channel and use <code>nn.BCEWithLogitsLoss</code> alternatively, which would then map a “low” value to class0 and a “high” one to class1. In that case your target should have the same shape as the model output.</p>

username: WilliManFR
cooked: <p>Thank you for your answer!<br>
I finally used nn.BCEWithLogitsLoss and I could compile my CNN (it doesn’t learn again but it compiles at least ^^)!</p>
Torch.detect_anomaly: Function 'Atan2Backward' returned nan values in its 0th output

username: Bhawna_Paliwal
cooked: <p>I am getting nan values as my loss in the model having MSE and euclidean loss as the only loss functions. I used torch.detect_anomaly to get this error. I tried clipping gradients and adding epsilon terms to the losses, but there was no change.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10c624d08d177a6b97faee652c97f0b22e4086bb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/10c624d08d177a6b97faee652c97f0b22e4086bb" title="error2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10c624d08d177a6b97faee652c97f0b22e4086bb_2_690x189.jpeg" alt="error2" data-base62-sha1="2oo9XDFjmtfJU4YEV71CPSeiSeL" width="690" height="189" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10c624d08d177a6b97faee652c97f0b22e4086bb_2_690x189.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10c624d08d177a6b97faee652c97f0b22e4086bb_2_1035x283.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/0/10c624d08d177a6b97faee652c97f0b22e4086bb.jpeg 2x" data-dominant-color="F5F6EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error2</span><span class="informations">1136×312 67.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does anyone have any ideas on how I can solve/debug it further? Can it be a GPU related error?</p>

username: ptrblck
cooked: <p>I guess your input to <code>torch.atan2</code> might create invalid gradients as seen here:</p>
<pre><code class="lang-python">x = torch.zeros(4, requires_grad=True)
y = torch.zeros(4)
out = torch.atan2(x, y)
out.mean().backward()
print(x.grad)
&gt; tensor([nan, nan, nan, nan])
</code></pre>
<p>and as is pointed to in the error message.</p>

username: Bhawna_Paliwal
cooked: <p>Thanks a lot for the reply. Can you please point out to some loss functions/possible computations where torch.atan2 might have occurred as I haven’t used torch.atan2 anywhere directly in my implementation. As I mentioned in the query, only loss functions I use are MSE and Euclidean.</p>
<pre><code class="lang-auto">Euclidean(a,b):
squared_difference = torch.pow(a - b, 2)
ssd = torch.sum(squared_difference, axis=tuple(range(1, a.ndim)))
return torch.sqrt(ssd)
</code></pre>

username: ptrblck
cooked: <p>I don’t know, where it’s used as I cannot see the model definition.<br>
In case you cannot find the usage, you could use the “brute force” approach of printing the <code>.grad_fn</code> attribute of all intermediate tensors and check, where <code>Atan2Backward</code> is shown.</p>
Using both CPU and GPU during training to avoid CUDA out of memory

username: fn246.c
cooked: <p>Is it possible to use <strong>both CPU and GPU during training</strong> to avoid bellow error ?</p>
<blockquote>
<p>CUDA error: out of memory</p>
</blockquote>
<p>I have CPU: 32G RAM and GPU: 8G RAM.<br>
When I train on smaller network with batch size =4 , it is OK. But I can not increase batch size, because it faces CUDA out of memory. In this case, it uses just 20% of CPU and all GPU capacity.<br>
Is it possible to set data in CPU and model in GPU? How ??<br>
Thanks</p>

username: TinfoilHat0
cooked: <p>Training on a CPU would be very slow if you’re doing deep learning stuff. Perhaps, you might want to try some  other approaches such as mixed-precision training ?</p>

username: fn246.c
cooked: <p>I mean the model trains in GPU and also loss calculated in GPU, but reading data and batches are in CPU?<br>
I don’t know about mixed-precision training, What is it used for ?<br>
I just want to solve error CUDA out of memory to train with larger batch size than 4 !</p>
Can't find class when load model

username: tetelevm
cooked: <p>Hello!<br>
I created, trained and saved my model <code>MyModel</code> in Jupiter as <code>'mymodel.pt'</code>. But then I want to load it in another application. To do this, I created the same class in the file where I load it:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        ...
    def forward(self, x):
        ...
</code></pre>
<p>This file is the file with the predictor class, which is run through <code>uvicorn</code>. And when I run <code>torch.load(path_to_pt)</code>, I see an error that there is no class <code>MyModel</code>:</p>
<pre><code class="lang-auto">File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 851, in _load
    result = unpickler.load()
AttributeError: Can't get attribute 'MyModel' on &lt;module '__main__' from '/usr/local/bin/uvicorn'&gt;
</code></pre>
<p>I understand that the  <code>MyModel</code> class is called differently:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(MyModel)
# &lt;class 'classification.my_model.MyModel'&gt;
</code></pre>
<p>And I understand that this shouldn’t be a problem when working through <code>env</code>, but is there any way I can solve this problem without going to <code>env</code>?</p>

username: ptrblck
cooked: <p>It seems you’ve stored the model directly (not its <code>staste_dict</code>), so I think you would need to make sure the folder structure etc. on the inference application is the same as was used during training.<br>
The recommended way would be to store the <code>state_dict</code> of models instead, recreate the model object in another script, and load its <code>state_dict</code> later, which would avoid these pickling issues.</p>
Balance Classes and Random Split

username: Stylianos_Charalambo
cooked: <p>Hello,</p>
<p>I have an ImageFolder object with datapoints of 3 unbalanced classes and I want to randomly choose n points from each class where n is the minimum class count and then split the new dataset formed into a training and validation set (either keeping the proportions or randomly).<br>
I have this code but I am not sure if I am doing it correctly.</p>
<pre><code class="lang-auto">dataset = datasets.ImageFolder(image_dir, transform=transformations)
images_label = {image[0]: image[1] for image in dataset.imgs}

class_counts = {}
for image_id in images_label.keys():
    label = images_label[image_id]
    class_counts[label] = class_counts.get(label, 0) + 1

class_weights = list(class_counts.values())
class_weights /= np.sum(class_weights)

sampler = torch.utils.data.sampler.WeightedRandomSampler(class_weights, sum(class_counts.values()))

data_loader = DataLoader(dataset, sampler=sampler)   

train_length=int(0.8* len(data_loader))
test_length=len(data_loader)-train_length

train_dataset,test_dataset = torch.utils.data.random_split(data_loader.dataset,(train_length,test_length))

dataloader_train = torch.utils.data.DataLoader(train_dataset)
dataloader_test = torch.utils.data.DataLoader(test_dataset)

</code></pre>

username: ptrblck
cooked: <p>In your current approach it seems that you are using the <code>class_counts</code> to create a <code>WeightedRandomSampler</code>, while the each sample should get a weight as described in <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">this post</a>.</p>
<p>I’m also unsure, if each batch should contain at least <code>n</code> samples from each class or the dataset splits.<br>
In the former case, you could write a custom sampler (and remove the <code>WeightedRandomSampler</code>) such that indices are samples using your condition.</p>
Memory usage understanding

username: Moft
cooked: <p>Hello all,</p>
<p>I’m using Pytorch for awhile and I’m wondering about the memory usage.<br>
The case is simple, 1 windows machine with  1 GPU.<br>
When I’m training a model I see that the RAM and the GPU usage increase but not to their full potential. GPU doesn’t pass the 10% of usage for example.</p>
<p>To have a faster training session, would make it sense to push the hardware to the limit? And if so, how?<br>
It is just a matter of putting more batches?<br>
Are any setup settings from Pytorch controlling this?</p>
<p>This is my first post so any formatting issues regarding the question please let me know.</p>
<p>Best</p>
Function that calculates the output shape of a block

username: Marcel_Gibier
cooked: <p>Hi,</p>
<p>is there a function on pytorch that allows to get the shape of an output of a Sequential module knowing the input shape ? For example for this mini block :</p>
<pre><code class="lang-auto">self.localization = nn.Sequential(
            nn.Conv2d(in_channels, 8, kernel_size=7),
            nn.MaxPool2d(2, stride=2),
            nn.ReLU(True),
            nn.Conv2d(8, out_channels, kernel_size=5),
            nn.MaxPool2d(2, stride=2),
            nn.ReLU(True)
        )
</code></pre>
<p>We know of course the output formula of a 2D convolution or a max pooling that we can code easily :</p>
<pre><code class="lang-auto">
def conv_output_shape(h_w, kernel_size=1, stride=1, pad=0, dilation=1):
    from math import floor
    if type(kernel_size) is not tuple:
        kernel_size = (kernel_size, kernel_size)
    h = floor(((h_w[0] + (2 * pad) - (dilation * (kernel_size[0] - 1)) - 1) / stride) + 1)
    w = floor(((h_w[1] + (2 * pad) - (dilation * (kernel_size[1] - 1)) - 1) / stride) + 1)
    return h, w
</code></pre>
<p>But as I said I’m looking to do the same thing so a function I can call in a network for a group of operations so for a nn.Sequential or even a network in general.</p>

username: sio277
cooked: <p>You can attach forward hook to get the output size of certain module.<br>
Example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

seq = nn.Sequential(nn.Conv2d(1, 2, 3), nn.BatchNorm2d(2), nn.Conv2d(2, 3, 3))

def forward_hook(module, input, output):
    print(output.size())
    
seq.register_forward_hook(forward_hook)
seq(torch.rand(1, 1, 10, 10))
</code></pre>
<p>which prints out torch.Size([1, 3, 6, 6]).</p>

username: Marcel_Gibier
cooked: <p><a class="mention" href="/u/sio277">@sio277</a> wah great !! thanks <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Torch.fx and no_grad()

username: kyong
cooked: <p>I am trying to ramp up on torch.fx. I have a forward function with torch.no_grad() (example below).</p>
<p>def forward(self, x):<br>
with torch.no_grad():<br>
tmp_val = … do something …</p>
<p>out = torch.mul(x, tmp_val)<br>
return out</p>
<p>I did symbolic trace then printed out the code.</p>
<p>traced = torch.fx.symbolic_trace(model)<br>
print(traced.code)</p>
<p>Then I can’t see no_grad() in the traced code anymore. Is there a way to keep this in the traced model?</p>

username: James_Reed
cooked: <p>Hi <a class="mention" href="/u/kyong">@kyong</a>,</p>
<p>FX does not support context managers (such as <code>no_grad</code>) in traced regions. Can you refactor the original code to something like:</p>
<pre><code class="lang-auto">class MyModel(torch.nn.Module):
  def forward(self, x):
    tmp_val = … do something …
    out = torch.mul(x, tmp_val)
    return out

class NoGradWrapper(torch.nn.Module):
  def __init__(self):
    self.mm = MyModel()

  def forward(self, x):
    with torch.no_grad():
      return self.mm(x)
</code></pre>
<p>Then you can apply FX transforms to <code>NoGradWrapper.mm</code> and still preserve the no-gradient behavior</p>

username: kyong
cooked: <p>Hi James,</p>
<p>Thank you for your comments! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I actually ended up not going with torch.fx, but modified the deep net in the original source code I used. But it is great to know torch.fx does not preserve the context managers.</p>
Does the order of execution of 'net.cuda()' and 'net.load_state_dict' affect the result?

username: jokerWRN
cooked: <p>Hi, I am confused about the order of execution of ‘net.cuda()’ and ‘net.load_state_dict’.<br>
Will the following two execution orders produce the same result?</p>
<pre><code class="lang-auto"># method 1
net = model(args)
net.cuda()
net.load_state_dict(checkpoints)

# method 2
net = model(args)
net.load_state_dict(checkpoints)
net.cuda()
</code></pre>

username: devashishkprasad
cooked: <p>I never tried but I think they should produce the same results.</p>
<p>I generally load the state dict first and then send model to cuda device. As shown <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-on-gpu-load-on-gpu" rel="noopener nofollow ugc">here</a> in the documentation.</p>

username: jokerWRN
cooked: <p>Thanks for your share!</p>
Is troch.fx going to support scripting mode?

username: yzhliu
cooked: <p>So far torch.fx supports symbolic trace only, therefore inevitably has some limitation. Is it going to support “scripting” mode to directly translate python code as torch.jit.script does?</p>

username: James_Reed
cooked: <p>Hi <a class="mention" href="/u/yzhliu">@yzhliu</a>,</p>
<p>No, at this time the plan is to support tracing only. This stems from a desire to reduce the complexity of a) the front-end b) the represenatation and c) transforms and to keep FX accessible to a more general audience. True, this limits the type of models that can be transformed with FX, however we believe that the types of models folks work with (vision, NLP, etc) tend not to require sophisticated constructs</p>