akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to load torchvision.models.detection from local .pth file without internet

Oliver: <p>I tried to follow this answer on stackoverflow as well as the load/save tutorials</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/60312752/how-to-cache-pytorch-models-for-use-when-not-connected-to-the-internet">
  <header class="source">

      <a href="https://stackoverflow.com/questions/60312752/how-to-cache-pytorch-models-for-use-when-not-connected-to-the-internet" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/10598247/joseph-summerhays" target="_blank" rel="noopener nofollow ugc">
    <img alt="Joseph Summerhays" src="https://lh3.googleusercontent.com/-0eyBjNxK1Uo/AAAAAAAAAAI/AAAAAAAAAAA/ABtNlbBzmk-C3WfTzBcSVt4MFJljKiIh9A/mo/photo.jpg?sz=256" class="thumbnail onebox-avatar" width="" height="">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/60312752/how-to-cache-pytorch-models-for-use-when-not-connected-to-the-internet" target="_blank" rel="noopener nofollow ugc">How to cache Pytorch models for use when not connected to the internet?</a>
</h4>

<div class="tags">
  <strong>python, pytorch, vgg-net</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/10598247/joseph-summerhays" target="_blank" rel="noopener nofollow ugc">
    Joseph Summerhays
  </a>
  on <a href="https://stackoverflow.com/questions/60312752/how-to-cache-pytorch-models-for-use-when-not-connected-to-the-internet" target="_blank" rel="noopener nofollow ugc">04:08AM - 20 Feb 20 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>but it seems I need internet everywhere, whenever I just declare the model</p>
<p><code>model = detection.fasterrcnn_resnet50_fpn(pretrained=True)</code><br>
also tried and few variations based on the github and reasoning<br>
<code>model = detection.fasterrcnn_resnet50_fpn(pretrained=False)</code></p>
<p>no matter if in a function or not, it tries and fails to download from the internet, so I can’t even get to the loading step.</p>
<p>how do I do this correctly?</p>

ptrblck: <p>Are you also seeing the failure when <code>pretrained=False</code> is used? This sounds weird, as nothing should be downloaded (at least I would assume it).</p>

Oliver: <p>Hi, yes, I found the bug though, there is another parameter, pretrained_backbone which by default is set to True. If you use pretrained=False and pretrained_backbone=False it works.</p>
Comparing two strings in c++

TheHero_Coder: <p>I am facing an Issue while Comparing two strings in c++.</p>
<p>c pointing to “dreamy” , which I have confirmed from the printf statement. Now I want to compare this with “dreamy” string but every time it is going in else part and showing not matching.</p>
<p>cout statement of both Str1 and Str2 also printing same output “dreamy” but the length of Str1 is showing 7 and length of Str2 is 6.</p>
<p>Can anybody tell me what is the problem and how to fix it.</p>
<pre><code class="lang-auto">char *c;
        c = strtok (temp,",");   
                    // Here   printf ("%s\n",c);   prints    dreamy
        while (c != NULL)
        {
            std::string Str1 = std::string(c);
            std::string Str2 = std::string("dreamy");
            cout &lt;&lt; "Str1  " &lt;&lt; Str1 &lt;&lt; "Len" &lt;&lt; Str1.length() &lt;&lt;endl;  //  Len = 7 showing for  c = "dreamy"
            cout &lt;&lt; "Str2  " &lt;&lt; Str2 &lt;&lt; "Len" &lt;&lt; Str2.length() &lt;&lt;endl;  //  Len = 6 for "dreamy"
            if(Str1.compare(Str2) == 0)
            {
                cout &lt;&lt; "Finally Match";
                presence[1] = 1;
            }
            else
                cout &lt;&lt; " Dont Match";
            printf ("%s\n",c);
</code></pre>
<p>Any help will be appreciated.<br>
Thanks in advance!</p>

fluffycookie: <p>Hey,<br>
I believe that there’s a spurious character in the first string. It has Len = 7<br>
Maybe a newline or a space.</p>
<p>You can read <a href="https://www.scaler.com/topics/comparing-two-strings-in-cpp/" rel="noopener nofollow ugc">this</a> article, it’s informative and may help clear your concepts.</p>
Diagonal Hessian wrt parameters when using mini-batches

ccook: <p>I am interested in computing the diagonal values of the Hessian of the loss function with respect to the parameters of an <code>nn.Module</code> to use for statistical inference (standard errors depend on the Hessian). The complication is that the model is estimated using mini-batches, because the full data doesn’t fit into memory, and existing solutions I’ve come across only give you the Hessian for the loss evaluated on a single batch of data.</p>
<p>Does anyone have ideas for how to compute the Hessian evaluated on the full dataset? For a single batch, a (not safe) method like the code below works. My current thinking is that I can just compute the Hessian for a loss function with `sum’ reduction on each batch separately and then sum across all batches, but I haven’t yet convinced myself this is valid so figured I’d see if someone else had thoughts</p>
<pre><code class="lang-auto">_input=torch.randn(32,3)
layer = nn.Linear(3,4)
criterion=nn.CrossEntropyLoss()

weight = layer.weight

def func(weight):
    del layer.weight
    layer.weight=weight
    return criterion(layer(_input), torch.zeros(len(_input),dtype=torch.long))

torch.autograd.functional.hessian(func,weight)
</code></pre>
<p>Related links:</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/issues/49171" rel="noopener nofollow ugc">Open issue with some discussion</a></li>
<li><a href="https://discuss.pytorch.org/t/using-autograd-functional-jacobian-hessian-with-respect-to-nn-module-parameters/103994/3">Hackish solution for single batch</a></li>
<li><a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7" rel="noopener nofollow ugc">Slightly less hackish solution</a></li>
<li><a href="https://discuss.pytorch.org/t/efficient-computation-with-multiple-grad-outputs-in-autograd-grad/66594">Discussion on efficient computation of Jacobian</a></li>
</ul>

Ren_Pang: <p>Hi there, I see that you are using my silly hack solution haha:)<br>
Please take a look at <a href="https://github.com/pytorch/pytorch/issues/49171#issuecomment-934055267" class="inline-onebox" rel="noopener nofollow ugc">NN Module functional API · Issue #49171 · pytorch/pytorch · GitHub</a><br>
which shall provide an elegant solution after 1.11 release.</p>

ccook: <p>Neat, thanks <a class="mention" href="/u/ren_pang">@Ren_Pang</a>. Don’t suppose you have a sense of when 1.11 might drop? Can’t imagine it’s too soon given we’re on 1.9.1</p>
<p>Thanks again!</p>

Ren_Pang: <p>They just finished the rc2 build for 1.10, which means it will release days later.<br>
I assume 1.11 will take more than 4 months after 1.10 release.</p>
<p>This feature is implemented 1 or 2 weeks ago, so it missed the 1.10 branch cut. And it’s not a feature in urgent demand so it’s not included in the recent cherry picks.</p>
Help with Fourier transform and vision CNNs

AndreiTA: <p>Hello,</p>
<p>Recently I started working on some research project at school. I try to do a quick summary.<br>
Input image = I<br>
Kernel = K</p>
<p>Apply Fourier Transform on the Image I. Filter out the 50% highest frequencies. Then do conv in the F domain and pass it through a smaller CNN. Then finally do Inverse FT and get results.</p>
<p>Can somebody please help me how to implement this in PyTorch (I am a mediocre w python and pytorch). I am stuck at the step after doing the conv in the freq domain(dotp). How shall I continue with the process in the freq domain and how can I use a smaller network(I would use an input image of 224x224 lets say, and then in the Freq domain I could use a smaller resnet for Magnitude only images of around 50x50) ? To have benefits of filtering out the high frequencies.</p>
<p>Thanks in advance.</p>
How to define a model as a function of its parameters and get its Jacobian?

homocomputeris: <p>If one defines a model <code>f</code>, and <code>x</code> is training data, then we have <code>f(x)</code>.<br>
How can one concisely define a model <code>f(w)</code> as a function of its parameters <code>w</code>? Will PyTorch be able to differentiate it wrt <code>w</code>, like JAX/STAX can?</p>

ptrblck: <p>Could you explain this use case a bit or post some dummy code what should be achieved?</p>

homocomputeris: <p>Basically, I just want to linearize a model with respect to its weights to have <code>f(w) = f(w0) + jac(w0)@(w-w0)</code> when <code>x</code> is fixed.</p>
<p>JAX, for example, already has Jacobian and Hessian functions, and it’s quite low-lever, so it allows to set weights directly.</p>

avish: <p>Given a model (inherited from nn.Module), calculate Jacobian of parameters of this model with respect to output vector.<br>
This looks like a hack to do this,</p><aside class="quote quote-modified" data-post="3" data-topic="103994">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ren_pang/48/11836_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/using-autograd-functional-jacobian-hessian-with-respect-to-nn-module-parameters/103994/3">Using `autograd.functional.jacobian`/`hessian` with respect to `nn.Module` parameters</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    I’m also working on this issue when calculating the influence function. 
My current solution is to follow this workaround: 

So I write this sample code (which works as a workaround): 
import torch
import torch.nn as nn

_input=torch.randn(32,3)
layer = nn.Linear(3,4)
criterion=nn.CrossEntropyLoss()

weight = layer.weight

def func(weight):
    del layer.weight
    layer.weight=weight
    return criterion(layer(_input), torch.zeros(len(_input),dtype=torch.long))

torch.autograd.functional.hessia…
  </blockquote>
</aside>


Ren_Pang: <p>Hi, I see that you are using the unsafe hack I previously proposed.<br>
Just want to notice you that there seems to be a new elegant way to do this.<br>
Please take a look at <a href="https://github.com/pytorch/pytorch/issues/49171#issuecomment-934055267" class="inline-onebox" rel="noopener nofollow ugc">NN Module functional API · Issue #49171 · pytorch/pytorch · GitHub</a><br>
which shall provide an elegant solution after 1.11 release.</p>
Is ImageNet pre-trained model really important?

sunshine1: <p>When I train object detector, if it is trained based on ImageNet pre-trained model, the loss will drop to 0 through 50 epochs. But if it trained from scratch, the loss still remain high through 100 epochs. I want to know if the ImageNet pre-trained model is required for training?</p>

Sayed_Nadim: <p>Hi,<br>
ImageNet “pre-trained” model means the filters in the model are already “trained” to detect features. So, if you use a pre-trained model in your detector, it already can detect features, and hence, you have the rapid drop of loss within a short time.<br>
However, if you are training a new model from scratch, the filters are learning to detect features and hence, the loss stays high.<br>
So, whether to use a pre-trained model or not will be decided by which kind of training strategy you will choose.<br>
Hope this helps.<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji only-emoji" alt=":slight_smile:"></p>
Get error message when excute "loss.backward()"

cleexiang: <p>I’m learning “Neural Transfer Using Pytorch” example, I have typed the same code with the <a href="https://pytorch.org/tutorials" rel="noopener nofollow ugc">Tutorials</a>, but I get the error, anyone can help me?</p>
<h4>
<a name="error-message-1" class="anchor" href="#error-message-1"></a>Error Message</h4>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 65536]], which is output 0 of ViewBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later.</p>

Sayed_Nadim: <p>Hi,<br>
It is quite impossible to track down the issue without any code/example. Can you please share some working codes to replicate the issue?</p>
Weight sharing: two model objects using a reference to the same parameters

nscotto: <p>Hi,</p>
<p>I would like to know if it’s possible to share a reference to the same parameters between two models.</p>
<p>This is not for traditional weight sharing, the purpose of it is to overcome a limitation from torch.fx when the model have different behaviours depending on it’s training mode.</p>
<p>Basically I am a framework for automatic model surgery, however when tracing a model the resulting graph only captures the mode it was ran on and therefore loses the original model behaviour.<br>
So the idea was to trace and operate surgery in both mode, and then wrap the two into a module that will call the appropriate one depending on its training mode.<br>
The problem is that I need the two modules to share the same weight.</p>
<p>One solution would be to force transferring the weights of the training version to the evaluation one every time the mode is being changed to evaluation.<br>
However this would be a lot cleaner if I could just have the two models using a reference to the same weights.</p>
<p>Is it even possible?</p>
<p>Cheers</p>
Model's loss started at low value and converged too fast

Homura: <p>Hi everyone,<br>
I am new to pytorch and after some tutorials, I tried to mimic a source github about Flor HTR architecture (you can search this git in the internet). This source used tensorflow and I want to test the flor architecture in pytorch. However, when I train the model, my CTCloss always goes from around 10 then down to around 3 and 2. I has tried some methods like:</p>
<ul>
<li>Change zero infinity to True in the CTCloss (this was when I got negative values but now I only see positive value after some changes in the model so I think this is not the case).</li>
<li>Change the implement of Batchrenorm I found in the internet which are included  batchrenorm by ludvb, and batchrenorm by mf1024 but the situation is still the same.<br>
I am very appreciate if somebody helps me in this case. To produce the situation, this is my colab link (<a href="https://drive.google.com/file/d/1mgJ-asR6iosANUI0goL697kLMYn8LJeX/view?usp=sharing" rel="noopener nofollow ugc">https://drive.google.com/file/d/1mgJ-asR6iosANUI0goL697kLMYn8LJeX/view?usp=sharing</a>) and my file data format hdf5 transformed from IAM dataset (<a href="https://drive.google.com/file/d/1I3cEdQmFF7loqj0r0XPjpO5RPgBamR5m/view?usp=sharing" rel="noopener nofollow ugc">https://drive.google.com/file/d/1I3cEdQmFF7loqj0r0XPjpO5RPgBamR5m/view?usp=sharing</a>)</li>
</ul>
<p>Thank you,<br>
If you have any question, I am happy to answer.</p>

tom: <aside class="quote no-group" data-username="Homura" data-post="1" data-topic="134024">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/cdc98d/48.png" class="avatar"> Homura:</div>
<blockquote>
<p>Change zero infinity to True in the CTCloss (this was when I got negative values but now I only see positive value after some changes in the model so I think this is not the case).</p>
</blockquote>
</aside>
<p>One of the things to watch out for is that CTCLoss expects you to input log probabilities, so you need to use <code>log_softmax</code> on the input argument. Getting negative values may originate from a violation of this constraint. (And after I checked, it looks like you use <code>softmax</code> instead from a cursory look.)</p>
<p>Best regards</p>
<p>Thomas</p>

Homura: <p>Thanks tom for the fast reply. However, I am using Logsoftmax at the moment so the negative values have not appeared again. You can see it in my code that I assign self.softmax = nn.LogSoftmax(dim=-1) at the code I marked with model.py. So I think I use the right way for the CTCloss but my loss value still start at 10 and go down to 3. I am happy to hear if you find out anything may cause the problem.</p>
<p>Thank you</p>
Is it faster to do inference on 1 image on a CPU or GPU?

ayrts: <p>I wonder if transferring the model and data to the gpu by calling <code>.cuda()</code> produces any significant overhead.</p>
<p>For reference, I’m using a resnet-18, so I also wonder if a small model has a significant advantage of using CPU over GPU. Thanks.</p>

tom: <p>It depends on your GPU and GPU, likely. In my experience, there is a speedup to be had with the GPU even in batch 1 resnet18, but again, I am sure there are combinations of GPU and CPU where it comes out differently.</p>
<p>Best regards</p>
<p>Thomas</p>
How Can I split a Sequential Model

ankahira: <p>Say I have a sequential model as follows.</p>
<pre><code class="lang-auto">model = nn.Sequential(
    nn.Linear(16, 12),
    nn.ReLU(),
    nn.Linear(12, 10),
    nn.ReLU(),
    nn.Linear(10, 1),
    nn.Sigmoid()
)
</code></pre>
<p>How can I split it into two different models, with model containing a couple of layers?</p>

tom: <p>You can just use slicing these days: e.g. <code>model[:4]</code> and <code>model[4:]</code> is a split after the second ReLU.</p>
<p>Best regards</p>
<p>Thomas</p>
3D CNN-Pose-estimation

Mukesh1729: <p>Hi,<br>
I am trying to implement an architecture from a paper and stumbled across the below problem.</p>
<p><strong>note:</strong></p>
<ul>
<li>I am using a 3D CNN architecture for pose estimation of image data(videos split to frames)</li>
<li>I am using 17 key points therefore my target heatmap has 17 channels for this problem with</li>
<li>MSELoss() as the loss function</li>
<li>output-softmax layer</li>
<li>Heatmap dims: (1,17, 200, 200)</li>
<li>input dims:(1,3,4, 200, 200)</li>
</ul>
<p><strong>Question:</strong><br>
What does my target heatmap need to contain since, I am using a network that takes in an input image shape: (1,3,4,200,200) and outputs a target heatmap: (1,17, 200, 200)?<br>
My target heatmap can be only of one image but my input is a sequence of frames(*4) from a video?<br>
How can I go about resolving this problem to train my network?</p>
<p><strong>reference:</strong><br>
<a href="https://arxiv.org/abs/2104.08029#:~:text=16%20Apr%202021%5D-,T%2DLEAP%3A%20occlusion%2Drobust%20pose%20estimation%20of,walking%20cows%20using%20temporal%20information&amp;text=As%20herd%20size%20on%20dairy,analyzing%20the%20gait%20of%20cows" class="inline-onebox" rel="noopener nofollow ugc">[2104.08029] T-LEAP: occlusion-robust pose estimation of walking cows using temporal information</a>.</p>
ResNet as backbone/feature extractor - undesired output dimensions

Zador: <p>Hi, I have been attempting to leverage the pre-trained ResNet model as a feature extractor. I have removed the final fc and pooling stages of the network and the output shape is (1, 2048, 7, 7).<br>
This is not the type of features I want. What I want is a feature per pixel, i.e. an output of the following shape: (1, n_features, H, W) where H and W are the height and width of the input image.</p>
<p>Is there any intermediate layer which outputs this kind of shape in ResNet? Is there any other pre-trained model in pytorch which offers this?</p>

ptrblck: <p>I don’t think you could use ResNets for your use case, as the first conv layer would already reduce the spatial size (layer definition <a href="https://github.com/pytorch/vision/blob/9ad33265c08049260b481cbcc16e213d3310ae88/torchvision/models/resnet.py#L193">here</a>):</p>
<pre><code class="lang-python">conv1 = nn.Conv2d(3, 6, kernel_size=7, stride=2, padding=3, bias=False)
x = torch.randn(1, 3, 224, 224)
out = conv1(x)
out.shape
&gt; torch.Size([1, 6, 112, 112])
</code></pre>
<p>I would guess that the majority of CNNs are downsampling the spatial size in the feature extractor stage and don’t know a specific model, which would keep the spatial size throughout the model.</p>

Zador: <p>Thanks for your response! That is sad to hear. Do you perhaps know of any networks that have been used to upsample resnet (or any other pre-trained network) features?</p>

ptrblck: <p>I think you might find interesting model architectures when searching for segmentation models, as they are often returning the original input shape (e.g. UNets).</p>
datasets.ImageFolder - Expected all tensors to be on the same device pytorch 1.9

Ja_Wie: <p>When iterating over the data in the dataloader, I got the RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!</p>
<p>My code still runs on an older computer (python 3.7, cuda 10, pytorch 1.4.0), but doesn’t work with my new laptop (python 3.8, cuda 11.1, pytorch 1.9.1).<br>
When debugging, it seems to give a problem in the transforms.Normalize(means, stds) function. the mean and std are tensors on cuda (have been checking it), so it must be the data itself.<br>
I’m loading my data with datasets.ImageFolder(path, transforms=transforms).</p>
<p>Does anyone knows what I should do to fix the error in this newer version?</p>

ptrblck: <p>It’s quite unexpected to see an error in <code>transforms.Normalize</code> as internally the <code>mean</code> and <code>std</code> would be pushed to the same device as the incoming tensor as seen <a href="https://github.com/pytorch/vision/blob/85b8fe6f7952cd586f3522362cccd4ead1fe2120/torchvision/transforms/functional.py#L342-L344">here</a>. So even if you have a device mismatch (<code>mean</code> and <code>std</code> on CUDA while the incoming tensor would be on the CPU), it should still work:</p>
<pre><code class="lang-python">norm = transforms.Normalize(
    mean=torch.randn(3).cuda(), std=torch.randn(3).cuda())

x = torch.randn(1, 3, 24, 24)
out = norm(x)
</code></pre>
<p>Are you sure this operation raises the error? Could you post the entire stacktrace, please?</p>

Ja_Wie: <p>I just found out that this line of code (from the previous version…) was raising the error:<br>
torch.set_default_tensor_type(‘torch.cuda.FloatTensor’)</p>
<p>The problem was indeed not the normalize function itself, thanks!</p>
WGAN penalty problems

DSL: <p>hello together,<br>
I am trying to reverse engineer the WGAN-GP and adapt it to my Pix2Pix problem for depth estimation. Everything works fine, but my gradient penalty function gives me the following error:<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
<pre><code class="lang-auto">    def _gradient_penalty(self, rgb, real_data, generated_data):
        batch_size = real_data.size()[0]

        # Calculate interpolation
        alpha = torch.rand(batch_size, 1, 1, 1)
        alpha = alpha.expand_as(real_data)


        if self.settings["cuda"]:
            alpha = alpha.cuda()
        interpolated = alpha * real_data.data + (1 - alpha) * generated_data.data
        interpolated = Variable(interpolated, requires_grad=True)
        if self.settings["cuda"]:
            interpolated = interpolated.cuda()

        # Calculate probability of interpolated examples
        prob_interpolated = self.diskriminator(rgb, interpolated)

        concat = torch.cat((rgb, interpolated), dim=1)
        concat = Variable(concat, requires_grad=True)
        #allow_unused=True

        Variable(prob_interpolated, requires_grad=True)
        # Calculate gradients of probabilities with respect to examples
        gradients = torch_grad(outputs=prob_interpolated.detach(), inputs=concat.detach(),
                               grad_outputs=torch.ones(
                                   prob_interpolated.size()).cuda() if self.settings["cuda"] else torch.ones(
                                   prob_interpolated.size()),
                               create_graph=True, retain_graph=True)[0]
        # Gradients have shape (batch_size, num_channels, img_width, img_height),
        # so flatten to easily take norm per example in batch
        gradients = gradients.view(batch_size, -1)
        self.losses['gradient_norm'] = gradients.norm(2, dim=1).mean().data[0]

        # Derivatives of the gradient close to 0 can cause problems because of
        # the square root, so manually calculate norm and add epsilon
        gradients_norm = torch.sqrt(torch.sum(gradients ** 2, dim=1) + 1e-12)

        # Return gradient penalty
        return self.settings["gp_weight"] * ((gradients_norm - 1) ** 2).mean()
</code></pre>

ptrblck: <p>Don’t use the <code>.data</code> attribute, as it’s deprecated and could yield unwanted side effects since Autograd isn’t able to track these tensors.<br>
Also, <code>Variable</code>s are deprecated since PyTorch <code>0.4</code>, so you can use tensors now.<br>
Note that re-wrapping tensors into a new tensor (or <code>Variable</code> in your case) will also detach these tensors from the computation graph, so avoid this in case you want to keep the computation graph of the previous operations.</p>
Graph level classification using structural features in pytorch gyometric

sun1: <p>I’m very new to GNN and graphs. I want to do a graph level classification using pytorch gyometric and I’m very confused. In fact, I want only to use the structural features of the nodes to do classification and there is no attribute features for nodes. Is it possible to do graph level classification based on topological structure only? If so, what is the best way to do that?  my graphs have 24 nodes, so not large graphs. thank you!</p>
How to apply loss to subset of layers

jfkback: <p>I have a model with multiple layers/components, these layers have various losses associated with them. Right now I am summing these losses and then running <code>loss.backwards(loss1 + loss2 + ...)</code>, but if my understanding is correct, this loss is applied equally to all components even if they should not be assigned certain losses. How can I make sure only certain parts of my model have certain losses applied to them?</p>

soulitzer: <p><code>.backward</code> has an <code>inputs</code> argument where you can specify which parameters get gradients accumulated into.</p>
Libtorch 1.9.1 and 1.7.1 releases with cuda fail to detect CUDA

ccas007: <p>I am trying to load a premade Yolo torchscript, which is failing.  However, it is also failing to detect any CUDA at all.   I have seen places describing a missing function call in messages a year old, is this problem still around?  How do I rebuild under windows to fix this?</p>

JohnSidKim: <p>how about add “-INCLUDE: ? warp_size@cuda@at@<span class="mention">@YAHXZ</span>” in project property / linker / Add dependencies.</p>
QAT: AttributeError: 'tuple' object has no attribute 'dequantize'

ynjiun_wang: <p>Hi,</p>
<p>I have done QAT model seems successfully but when I run</p>
<p><code>_ = model(x)</code></p>
<p>I got the following error:</p>
<pre><code class="lang-auto">    _ = model(x)
  File "/home/paul/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/paul/rknn2/PytorchProject/ssd/r-ssd8.py", line 255, in forward
    x = self.dequant(x)
  File "/home/paul/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/paul/pytorch/lib/python3.6/site-packages/torch/nn/quantized/modules/__init__.py", line 74, in forward
    return Xq.dequantize()
AttributeError: 'tuple' object has no attribute 'dequantize'
</code></pre>
<p>The model class is defined as below:</p>
<pre><code class="lang-auto">class QuantizedRSSD(nn.Module):
    def __init__(self, model_fp32):
        super(QuantizedRSSD, self).__init__()
        # QuantStub converts tensors from floating point to quantized.
        # This will only be used for inputs.
        self.quant = torch.quantization.QuantStub()
        # DeQuantStub converts tensors from quantized to floating point.
        # This will only be used for outputs.
        self.dequant = torch.quantization.DeQuantStub()
        # FP32 model
        self.model_fp32 = model_fp32

    def forward(self, x):
        x = self.quant(x)
        x = self.model_fp32(x)
        x = self.dequant(x)
        return x
</code></pre>
<p>Can anyone help to identify why during the QAT the QuantizedRSSD seems working well but when run it then complains no dequantize attribute? Does the torch.quantization.DeQuantStub() work? or what’s missing?</p>
<p>Thanks a lot for your help.</p>

jerryzh168: <p>DeQuantStub only works for Tensors, if you have a tuple output (and both of them needs to be dequantized), you can write something like: x = map(self.dequant, x)</p>

ynjiun_wang: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>
<p>Thank you so much for your help. Your solution works!</p>
<p>Then I tried to port my resulting quantized_model to RKNN h/w and it requires to</p>
<p><code>trace_model = torch.jit.trace(quantized_model, torch.Tensor(1,3,300,300))</code></p>
<p>before it can convert the quantized_model into rknn model. But when run the torch.jit.trace, I encountered the following error:</p>
<pre><code class="lang-auto">RuntimeError: Tracer cannot infer type of &lt;map object at 0x7f308b270cc0&gt;
:Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type map.
</code></pre>
<p>I believe it is the x=map(self.dequant, x) we just introduced to solve the previous problem.</p>
<p>I wonder is there a way to get the torch.jit.trace to work for this case? Thanks a lot for your help again.</p>

ynjiun_wang: <p>the torch.jit.trace works when I manually split the map() into</p>
<p><code>x1,x2=self.dequant(x1),self.dequant(x2)</code></p>
Why classifier might fails when training on latent vectors?

Asya: <p>I want to use latent vectors from my autoencoder (which provides with good reconstruction results) for classification problem, but I have troubles classifying my data so far. I tried following:</p>
<p>method №1<br>
I have saved latent vectors as <code>.npy</code> files and was running <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#define-a-convolutional-neural-network" rel="noopener nofollow ugc">this classifier</a> on them, but my model started to overfit very fast (giving me 80-90% accuracy on a train set, but 5-10% on validation), though I have 50k latent vectors. But at least it was good on the train set</p>
<p>method №2<br>
I was trying to use saved autoencoder model with the same classifier for the same purpose (because to keep all the vectors as <code>.npy</code> takes too much storage). So i loaded encoder part (and froze all layers with <code>param.requires_grad = False</code> as I only need to produce latent vectors but not to train autoencoder) and put only classifier to optimization: <code>optim.SGD(clf.classifier.parameters(), lr=0.001, momentum=0.9)</code>, I also added  <code>clf.classifier.train()</code> and <code>clf.encoder.eval()</code>. But this model simply wouldn’t work - it just predicts mainly one class for all vectors in a batch that gives maximum 16% accuracy on train set</p>
<p>Where possibly did I do wrong? And what are the ways to make a classifier work?</p>
Use of models on devices with shared cpu/gpu memory

Jcwscience: <p>I am having some difficulties loading models on my device, it’s an Nvidia Xavier NX, and when I try to load a model to gpu there is a very large memory spike, over double of when the model finishes loading.</p>
<p>The only thing I can think of that would cause this is the model being loaded to cpu memory, then being copied to gpu memory, and this being the same memory device it temporarily has the model loaded to ram twice.</p>
<p>Is this the cause or something else? And is there a way I can work around it?</p>
Adding model parameters to a list

blade: <p>I have a model defined as:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.cl1 = nn.Linear(25, 60)
        self.cl2 = nn.Linear(60, 84)
        self.fc1 = nn.Linear(84, 10)

        self.other_params()

    def other_params(self):

        self.other_params_list = nn.ParameterList([
            nn.Parameter(torch.randn(60, 25)),
            nn.Parameter(torch.randn(84, 60)),
            nn.Parameter(torch.randn(84, 10))
        ])

    def forward(self, x):
        x = F.relu(self.cl1(x))
        x = F.relu(self.cl2(x))

        return self.fc1(x)
</code></pre>
<p>I’d like to use a meta-learning approach to learn weights of the linear layers of the network in the inner loop and learn the parameters in <code>other_params_list</code> in an outer optimization loop. How can I define a new list of params that excludes <code>other_params_list</code> from <code>model.parameters()</code>, to pass to the inner loop optimization model?</p>
<hr>
<p>EDIT: My solution is to define:</p>
<pre><code class="lang-auto">        self.model_params_list = nn.ParameterList([
            self.cl1.weight,
            self.cl1.bias,
            self.cl2.weight,
            self.cl2.bias,
            self.fc1.weight,
            self.fc1.bias
        ])
</code></pre>
<p>is there anything more glorious than this? like subtracting lists without a need to add every parameter one by one? or perhaps a loop over model params? Note that for the latter, I can’t define <code>model.parameters()</code> in the <code>MyModel</code> class.</p>
How to save and load model_final.pth detectron2

hichem_felouat: <p>I saved model_final.pth on my drive then I wrote this piece of code but it does not work. could somone check it ?</p>
<p>from detectron2.modeling import build_model<br>
cfg = get_cfg()<br>
model = build_model(cfg)<br>
from detectron2.checkpoint import DetectionCheckpointer<br>
DetectionCheckpointer(model).load("/content/gdrive/My Drive/model_final.pth")<br>
checkpointer = DetectionCheckpointer(model, save_dir=“output”)<br>
checkpointer.save(“my_model”)</p>
<p><strong>Inference :</strong></p>
<p>cfg = get_cfg()<br>
cfg.MODEL.WEIGHTS = “/content/output/my_model.pth”<br>
cfg.DATASETS.TEST = (“my_dataset_test”, )<br>
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.8<br>
predictor = DefaultPredictor(cfg)<br>
test_metadata = MetadataCatalog.get(“my_dataset_test”)</p>
<p>from detectron2.utils.visualizer import ColorMode<br>
import glob</p>
<p>for imageName in glob.glob("/content/my_dataset/test/*jpg"):<br>
im = cv2.imread(imageName)<br>
outputs = predictor(im)<br>
v = Visualizer(im[:, :, ::-1],<br>
metadata=test_metadata,<br>
scale=0.8<br>
)<br>
out = v.draw_instance_predictions(outputs[“instances”].to(“cpu”))<br>
print(“pred_classes : \n”,outputs[“instances”].pred_classes)<br>
print(“pred_boxes   : \n”,outputs[“instances”].pred_boxes)<br>
cv2_imshow(out.get_image()[:, :, ::-1])<br>
print("\n")</p>

ptrblck: <p>What is not working in the code?<br>
Could you post the error message?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Aakash_Gupta: <aside class="quote no-group" data-username="hichem_felouat" data-post="1" data-topic="95077">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hichem_felouat/48/28414_2.png" class="avatar"> hichem_felouat:</div>
<blockquote>
<p>model = build_model(cfg)</p>
</blockquote>
</aside>
<p>While loading the model during inference it needs to be built by setting the <code>eval_only</code> parameter.</p>
<pre><code class="lang-auto">model = MyTrainer.build_model(cfg, eval_only=True)
</code></pre>
GTX 1650 4 GB vs GTX 1060 6GB

sparshgarg23: <p>So,i am planning on finally buying a laptop with GPU(Yeah) ,at this point I know that deep learning models tend to be memory intensive ,so if I was to say use my system to train a deep learning model on the kitti/waymo dataset or let’s say I want to train a model for video action recognition (common datasets for video that come to my mind are Charades ,youtube etc)then would a GTX 1650 be a good choice to buy ,I am a bit skeptical as it only has 4GB memory.<br>
I know I can jump onto an RTX but for now I want to stick with GTX as it would be suitable for helping me understand how to properly work with a GPU.</p>
Pixel Time Series

sesale: <p>I have a time series of pixels (from satellite data). They are represented in an HDF5 file and I can read them using pandas such that each row corresponds to a pixel and each column to a time step. So each cell contains an array of values where each value corresponds to the value of one of the satellite bands. It is the first time, I am working with time series data in Pytorch, so I am not sure how to feed them into a (sequential model). When calling the DataLoader, what should one sample represent? One specific time step of all pixels or just a time step of one pixel?</p>
Autograd and implementing custom optimizer

blade: <p>I’m trying to implement a custom optimization rule for my neural network. I particularly don’t want to use .backward() method. Here is my optimizer:</p>
<pre><code class="lang-auto">class MyOptimizer(optim.Optimizer):

    def __init__(self, params, lr=1e-3):
        defaults = dict(lr=lr)
        super(MyOptimizer, self).__init__(params, defaults)

    def step(self, loss):

        for group in self.param_groups:

            grad = torch.autograd.grad(loss, group['params'], create_graph=True)

            for idx, p in enumerate(group['params']):

                p.grad = grad[idx]

                with torch.no_grad():

                    if p.grad is None:
                        continue
                    d_p = p.grad

                    p.add_(d_p, alpha=-group['lr'])
</code></pre>
<ol>
<li>
<p>Does the <code>step</code> function implement <code>torch.optim.SGD</code> without introducing any new parameters to the computational graph?</p>
</li>
<li>
<p>Why is there a need for <code>with torch.no_grad()</code> and is its placement correct?</p>
</li>
<li>
<p>How would using a <code>@torch.no_grad</code> decorator instead of <code>with torch.no_grad()</code> change the model’s behavior?</p>
</li>
<li>
<p>I’m assuming that this implementation does not need to call <code>self.optimizer.zero_grad()</code> and <code>loss.backward()</code> anymore. Is that correct?</p>
</li>
</ol>
How to load weights of custom dataset onto fpn for segmentation

azer: <p>Hey all,</p>
<p>I would like to use the classification weights  I obtained by running classification on ResNet50 and Mobilenetv3.   The dataset for segmentation is very similar to the classification. I would like to transfer the weights to fpn network of the Mask RCNN without calling the pre-trained backbone or model. I did as below but not sure if it is correct.</p>
<pre><code class="lang-auto">def maskrcnn_50fpn_transfer(num_classes=2, pretrained_backbone=False, path=None):
    """
    :param num_classes: num_classes + background
    :param pretrained_backbone: pretrained  backbone
    :param path: path to weights from classification
    :return: Maskrcnn model with fpn trained on classification
    """

    resnet50 = torchvision.models.resnet50(pretrained=False)

    # Resnet was orinally  pretrained on three classes
    resnet50.fc = nn.Linear(in_features=2048, out_features=3)

    print('Loading covid weights from classification')
    checkpoint = torch.load(path, map_location=torch.device('cpu'))

    resnet50.load_state_dict(checkpoint['model'], strict=False)

    backbone = resnet_fpn_backbone('resnet50', pretrained=resnet50)

    model = MaskRCNN(backbone, num_classes)

    return model
</code></pre>
<p>Any hint would be greatly appreciated<br>
Az</p>
AttributeError when importing model from Linux to Windows

talh: <p>I have a Windows 10 machine and an Ubuntu VM, both running PyTorch 1.9.1.</p>
<p>After quantizing a model on the Linux machine, I used <code>torch.save(net, output_file)</code> in order to save it, and loaded it on the Windows machine, by running <code>net = torch.load(input_file)</code>. This step completed without any error.</p>
<p>The problem starts when on Windows I try to call methods of the network, such as <code>net.eval()</code>, <code>net.to('cpu')</code>.</p>
<p>Here is the error I get:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "d:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 852, in to
    return self._apply(convert)
  File "d:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 530, in _apply
    module._apply(fn)
  File "d:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 529, in _apply
    for module in self.children():
  File "d:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1527, in children
    for name, module in self.named_children():
  File "d:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1545, in named_children
    for name, module in self._modules.items():
  File "d:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1130, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Conv2d' object has no attribute '_modules'
</code></pre>
<p>If I try to forward an image, I get another AttributeError:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "d:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "d:\Anaconda3\lib\site-packages\torchvision\models\resnet.py", line 249, in forward
    return self._forward_impl(x)
  File "d:\Anaconda3\lib\site-packages\torchvision\models\resnet.py", line 232, in _forward_impl
    x = self.conv1(x)
  File "d:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1049, in _call_impl
    if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
  File "d:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1130, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Conv2d' object has no attribute '_backward_hooks'
</code></pre>
Resize Mask RCNN transform

azer: <p>Hi there,</p>
<p>I resized the MasK RCNN transform  as follow:</p>
<pre><code class="lang-auto"> res_ = GeneralizedRCNNTransform(224, 224, [0.485, 0.456, 0.406],
                                        [0.229, 0.224, 0.225])
</code></pre>
<p>I am using weights from a model I trained for the backbone. I was wondering if this is correct or if it will impact the COCO metrics.  Any help is highly regarded.</p>
<p>Thanks</p>
Crossentropyloss with Float16

Benjamin_Blundell: <p>Hello there,<br>
I’m trying to reduce the memory used by my u-net network, in order to increase the batch size and increase the speed of convergence. It’s been very tricky so far but one of the biggest savings was to use float16 instead of float32.</p>
<p>However, I’m having trouble with the Cross Entropy Loss function - I’m getting NaNs from the first go.</p>
<pre><code class="lang-auto">def loss_func(result, target) -&gt; torch.Tensor:
    class_weights = torch.tensor(
        [0.1, 1.0, 1.0, 1.0, 1.0], dtype=torch.float16, device=result.device)
    criterion = nn.CrossEntropyLoss(weight=class_weights)
    loss = criterion(result, target)
    return loss
</code></pre>
<p>result and target have a number of 0 values but no NaNs or Infs or anything like that. The only thing I can think of is perhaps the target mask being of type <em>long</em> might not work with the input and unet type of float16.</p>
<p>Changing to float32 seems to fix things, so I’m wondering if theres something I’ve missed or is multi-class U-Net cross entropy loss just not compatible with half precision?</p>
<p>Cheers<br>
Ben</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> is considers unsafe in <code>float16</code> which is why it’s using <code>float32</code> if you are using the mixed-precision training utilities via <code>torch.cuda.amp.autocast</code> as described <a href="https://pytorch.org/docs/stable/amp.html#op-eligibility">here</a>.<br>
Since you are seeing invalid outputs, my best guess is that you are indeed hitting an overlow or other numerical issues.</p>

Benjamin_Blundell: <p>Thanks for pointing out that page - that explains it.</p>
Average Precision and Loss decrease

azer: <p>Hi,</p>
<p>I am using running Maskrcnn and using AP as a metric. After a certain number of epochs, the loss continues decreasing. However, the APcollapses. What can I do to solve this issue?<br>
Please advise.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/502f09e46695a199829fa27a79daab014d2c458e.png" alt="image" data-base62-sha1="brkYBm9ihPPDIoVg9dnNfuoTkgm" width="504" height="278"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cde945305809842c4afda479f5f6589fd8740bb9.png" alt="image" data-base62-sha1="tnzEXGvH8lBGChnG2lisRCon3qp" width="510" height="278"></p>

Preetham_R_Patlolla: <p>Hello,</p>
<p>What is the loss function you are using? Could post a snippet of your code?</p>

azer: <p>Hi <a class="mention" href="/u/preetham_r_patlolla">@Preetham_R_Patlolla</a><br>
I am using engine.py from the references/detection. So this is the sum of all losses for segmentation and detection. I am also using Adam optimizer.</p>