akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDA Out of memory when using autogrid to do multiple round of back propagation

Peter_Ham: <p>I’m trying to train a function <code>f</code>  of weights <code>w</code> that takes in some inputs <code>x</code> and outputs <code>y</code>. The loss function is <code>MSELoss()(y, target)</code>. Now I want to optimize <code>f</code> such that given a random <code>x</code> input, we can optimize <code>x</code> so as to minimize the loss function. This means that for a random initial guess, <code>f</code> should be optimized so as to most quickly find the solution <code>x</code> through gradient updates. The code looks like this:</p>
<pre><code>    initial_x = torch.Tensor()
    opt = optim.Adam([initial_x], lr=0.001)
    opt2 = optim.Adam([w], lr=0.001)
    for i in range(10):
         y = f(w, initial_x)
         loss = MSELoss()(y, target)
         initial_x.grad = torch.autograd.grad(loss, initial_x, retain_graph=True, create_graph=True)[0]
         opt.step()
    loss = MSELoss(f(w, initial_x), target)
    opt2.zero_grad()
    loss.backward()
    opt2.step()
</code></pre>
<p>however, this would make the occupied memory to grow. any fix?</p>
BatchNorm weird behavior on Eval

Marat: <p>I have simple example (1.1.0 pytorch)</p>
<pre><code class="lang-auto">import torch
print(torch.__version__)

x = torch.rand(1, 3, 2, 2)

for running_stats in [False, True]:
    for eval in [False, True]:
        print('running_stats:', running_stats, 'eval:', eval)
        bn = torch.nn.BatchNorm2d(3)
        bn.track_running_stats = running_stats
        if eval:
            bn.eval()
        print('\t', bn.running_mean)
        y1 = bn(x)
        print('\t', bn.running_mean)
        y2 = bn(x)
        print('\t', bn.running_mean)
        print('\t', (y1 - y2).abs().sum())
</code></pre>
<p>I expecting here that setting bn.eval() is enough for BN to return same results and keep same running mean and var. I also expect that different running mean and var will produce different results.</p>
<p>Output:</p>
<pre><code class="lang-auto">1.1.0
running_stats: False eval: False
	 tensor([0., 0., 0.])
	 tensor([0.0529, 0.0558, 0.0643])
	 tensor([0.1005, 0.1061, 0.1222])
	 tensor(0., grad_fn=&lt;SumBackward0&gt;)
running_stats: False eval: True
	 tensor([0., 0., 0.])
	 tensor([0.0529, 0.0558, 0.0643])
	 tensor([0.1005, 0.1061, 0.1222])
	 tensor(0., grad_fn=&lt;SumBackward0&gt;)
running_stats: True eval: False
	 tensor([0., 0., 0.])
	 tensor([0.0529, 0.0558, 0.0643])
	 tensor([0.1005, 0.1061, 0.1222])
	 tensor(0., grad_fn=&lt;SumBackward0&gt;)
running_stats: True eval: True
	 tensor([0., 0., 0.])
	 tensor([0., 0., 0.])
	 tensor([0., 0., 0.])
	 tensor(0., grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<p>Issues:</p>
<ol>
<li>Running mean and var seems to have ABSOLUTELY no effect on output in this example</li>
<li>Setting bn.eval() is not enough to keep same running mean and var</li>
</ol>

albanD: <p>Hi,</p>
<p>You cannot change <code>bn.track_running_stacts</code>. It should be given as an argument when you create the <code>BatchNorm()</code> layer.</p>

Marat: <p>But still changing running mean has no effect on the output it seems that this E and Var somehow incapsulated and immutable inside C++ part</p>

ptrblck: <p>This shouldn’t be the case as shown here:</p>
<pre><code class="lang-python">x = torch.rand(1, 3, 2, 2)
bn = nn.BatchNorm2d(3)
bn.eval()
print(bn.running_mean, bn.running_var)
out = bn(x)
print(out)

bn.running_mean = torch.tensor([100., 100., 100])
print(bn.running_mean, bn.running_var)
out = bn(x)
print(out)
</code></pre>
Between-class label

shirui-japina: <p>I want to implement a network which can predict <strong>‘between-class’</strong> label.</p>
<p>I mean, when feed the model with the data created by the data from <em><strong>class A</strong></em> and the data from <em><strong>class B</strong></em>, the label of the created data should <strong>not</strong> be [1 , 1], but [0.5, 0.5].</p>
<p>It is a bit like <strong>‘multi-label’</strong>, but it ’s actually different. Because it is no longer possible to use <strong>one-hot label</strong>.</p>
<p>What confuses me is how to define and use the <strong>loss function</strong>?</p>

ptrblck: <p>You could try to use <code>nn.BCEWithLogitsLoss</code>, but note that the loss values might be unexpected, as described <a href="https://discuss.pytorch.org/t/i-somehow-got-wrong-result-of-bceloss/43058/5">here</a>.</p>
<p>Alternatively, <code>nn.MSELoss</code> could work or have a look at <a href="https://github.com/pytorch/pytorch/issues/7455">label smoothing for NLLLoss</a>.</p>

shirui-japina: <p>Thanks for your help. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
NN architecture (weights, biases) independent of the input size?

argha004: <p>Hi everyone, I am new to Pytorch. Is there a way by which I could define the architecture of my network here in a way that it does not get affected by the number of inputs I am having? Eg, in Tensorflow, I can define a network having layers = [1,30,30,30,1]. In the first/input layer having a single unit, I can put in a vector of any size by storing it as a list and then passing it into the network just like a scalar. But, I am not able to implement the same in Pytorch. Any help would be appreciated.</p>
Feature pyramid

solsol: <p>Hi, I want to just use layer three or four of FPN. From the following code line, how may I extract third line?<br>
features = self.backbone(images.tensors)</p>
<p>features[1]? By this I face with an error for this part:<br>
x, result, detector_losses = self.roi_heads(features, proposals…</p>
<p>Any help?</p>

ptrblck: <p>To get intermediate activations, you could use <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">forward hooks</a>.</p>
<p>If that’s not possible for your use case, could you post the error message you are getting?</p>
Augmentation with new images

Frida: <p>Hi<br>
How can I add processed images to the original dataset and train on the new dataset meaning Original and processed. Thanks in advance!</p>

ptrblck: <p>You could create a <code>Dataset</code> without any data augmentation, and another one with some augmentations.<br>
Once both are created, you could use <code>ConcatDataset</code> to concatenate them into a single <code>Dataset</code>, which can be passed to a <code>DataLoader</code>.</p>
Unnecessary gradients in GAN training

f3ba: <p>The following is a quote from this Github issue: <a href="https://github.com/pytorch/pytorch/issues/5349" class="inline-onebox" rel="noopener nofollow ugc">Determining `requires_grad` automatically · Issue #5349 · pytorch/pytorch · GitHub</a>, about the ability to automatically detect variables for which a gradient is not needed. The quote:</p>
<blockquote>
<p>Suppose we are training a GAN, with a very simple generator G(z) = tanh(W₁z + b₁) and discriminator D(x) = W₂x + b₂. The loss is a function of D(G(z)), and when calling loss.backward(), all of the gradients in the network are computed. However, during GAN training, the generator and discriminator are trained separately in alternation, therefore</p>
<p>When training D, the gradients do not need to back propagate to G.<br>
When training G, we need only ∂D/∂x, but not ∂D/∂W₂ or ∂D/∂b₂.</p>
</blockquote>
<p>The Github issue is old and there have been some API improvements as far as I can tell that help to mitigate this issue. I just started reading the PyTorch tutorials, and  was wondering if this issue could be relevant in this section of the docs: <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">DCGAN Tutorial — PyTorch Tutorials 2.1.1+cu121 documentation</a>. Since this issue is not mentioned explicitly in the tutorial, I want to ask: The code from the tutorial is computing the unnecessary gradients, or not?</p>
Torch jit script method bugs: expected a value of type Tensor for argument 'self' but found Tensor[]

Peter_Ham: <p>I’m trying to code up a module from jit.ScriptModule that looks like this:</p>
<pre><code>class Model(jit.ScriptModule):
    def __init__(self, size):
        super().__init__()
        self.rnn = nn.GRUCell(size, size)

    @jit.script_method
    def forward(self, hidden, inputs):
        something = [torch.empty(0)] * 10
        return self.rnn(inputs, hidden)
</code></pre>
<p>however, it triggers the following errors:<br>
RuntimeError:<br>
arguments for call are not valid:</p>
<p>for operator aten::mul(Tensor self, Tensor other) -&gt; Tensor:<br>
expected a value of type Tensor for argument ‘self’ but found Tensor[]</p>
<p>I found the reason is that list comp is not available it jit, is there any fix?</p>
Using Pytorch Android under Kotlin

Chame_call: <p>Can I use Kotlin for that purpose or it’s possible only with Java?</p>

jspisak: <p><a class="mention" href="/u/david_reiss">@David_Reiss</a> - knows better. We do actually support Kotlin… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

David_Reiss: <p>Yep.  All Java code is fairly easy to use from Kotlin: <a href="https://kotlinlang.org/docs/reference/java-interop.html">https://kotlinlang.org/docs/reference/java-interop.html</a> .  That includes the PyTorch Java bindings.  We’ve tested it out briefly, and they seem to work fine, so let us know if you run into any trouble!</p>
Problem in using Resnet50

phantom90: <p>Hi,</p>
<p>I am currently using Resnet50 and calculate the gradient w.r.t the input, which is something like:</p>
<p><code>grad_val = pred.backward(create_graph=True)</code></p>
<p>Then I use the grad to further calculate:</p>
<p><code>grad_val.requires_grad_()</code><br>
<code>result = autograd.grad(grad_val, model.parameters())</code></p>
<p>I find that the last item of the result is always None. Normally, it should be the same shape as bias term, right?</p>
<p>Thanks,</p>
How to Vectorize/Parallelize Reinforcement Learning Environments?

RylanSchaeffer: <p>I feel like this is such an obvious problem, but I can’t find any clear answers. I have a Python class that conforms to OpenAI’s environment API, but it’s written in a way that it receives one input action per step and returns one reward per step. How do I parallelize this environment? I haven’t been able to find any clear answer online. A few people suggested baselines or stable_baselines, but these don’t appear to work with PyTorch and they’re currently broken by the switch to TensorFlow 2.0.</p>
<p>There are some other RL libraries (e.g. <a href="https://github.com/zuoxingdong/lagom" rel="nofollow noopener">https://github.com/zuoxingdong/lagom</a>, <a href="https://github.com/astooke/rlpyt" rel="nofollow noopener">https://github.com/astooke/rlpyt</a>), but they don’t appear to have professional support, so I’m concerned that if I use one, it’ll quickly become unusable.</p>

ptrblck: <p>I’m not an RL expert, but <a href="https://github.com/catalyst-team/catalyst">Catalyst</a> is in the PyTorch ecosystem, so it might get some future support.<br>
Have you had a look at this library already and would it fit your needs?</p>

RylanSchaeffer: <p>I decided to simply vectorize my environments and that appears to have given me the speed boost I need. I used the code provided here: <a href="https://stackoverflow.com/a/59599319/4570472" rel="nofollow noopener">https://stackoverflow.com/a/59599319/4570472</a></p>
Simple nn.Transformer Example

Emad_Elazhary: <p>I am a new “convert” from tensorflow…</p>
<p>I want to use nn.Transformer for a non NLP job, mainly a seq2seq job…</p>
<p>I need a simple example, where I would overfit it on one example (let’s say srcseq=[1,2,3], dstseq=[4,5,6])</p>
<p>I need both the training &amp; the inference code…</p>
<p>Can someone help a new convert <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

zhangguanheng66: <p>Welcome. See the tutorial <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noopener">here</a></p>

Emad_Elazhary: <p>Tutorial is just demonstrating the encoder layer not the Transformer model…</p>
<p>I need to know how to train the model. And how to autoregressivly generate seq</p>

zhangguanheng66: <p>Depending on the problem, you need TransformerEncoder or Transformer (encoder + decoder). Training the model is similar to the training function in the tutorial. For non-NLP problems, you need to figure out how to calculate the loss. Unless you show more details about your problem, I cannot give more help. The tutorial is a word-language-model problem and, more concise, predicting the next word following a sequence.</p>
Delete/unregister a parameter in a module

Carol_Ye_Liu: <p>I want to delete or unregister a parameter in a module by using erase(const Key &amp;key) method in OrderedDict class, but from the result i see no change in the OrderedDict. What is the right way to do this?</p>
<p>void deleteparm(torch::nn::AnyModule &amp; module, const std::string inName){</p>
<pre><code>torch::NoGradGuard no_grad;
std::cout &lt;&lt;"named parameters before erase "&lt;&lt;inName&lt;&lt; ":"&lt;&lt; std::endl;
for (auto&amp; p : module.ptr()-&gt;named_parameters(true)) {
    std::cout &lt;&lt;p.key() &lt;&lt; std::endl;
    std::cout &lt;&lt; p.value().sizes() &lt;&lt; std::endl;
}
module.ptr()-&gt;named_parameters(true).erase(inName);
//TODO: erase method in CLASS OrderedDict is available online(https://pytorch.org/cppdocs/api/classtorch_1_1_ordered_dict.html#classtorch_1_1_ordered_dict)
// but not in local version: update to nightly build?

std::cout &lt;&lt;"named parameters after erase:" &lt;&lt; std::endl;
for (auto&amp; p : module.ptr()-&gt;named_parameters(true)) {
    std::cout &lt;&lt;p.key() &lt;&lt; std::endl;
    std::cout &lt;&lt; p.value().sizes() &lt;&lt; std::endl;
}
</code></pre>
<p>}</p>
<pre><code>auto conv2D= torch::nn::Conv2d(torch::nn::Conv2dOptions(512, 512, 3).padding(1));
torch::nn::init::normal_(conv2D.ptr()-&gt;weight, 0, 1);//checked
conv2D.ptr()-&gt;bias.zero_();//conv2D.bias.zero_();
torch::nn::AnyModule my_module(conv2D);
deleteparm(my_module, "weight");
</code></pre>
<p>Result:<br>
named parameters before erase weight:<br>
weight<br>
[512, 512, 3, 3]<br>
bias<br>
[512]<br>
named parameters after erase:<br>
weight<br>
[512, 512, 3, 3]<br>
bias<br>
[512]</p>
Computation graph integrity

razvanc92: <p>I’m having a technical question. Let’s consider the following example where I want to select for only the top 10 values from the last dimension. The example works as intended but I am worried that I might break the computation graph since I’m creating a new tensor in line 2. Can anyone confirm if this is the right approach, or how I could verify it?</p>
<pre><code class="lang-auto"> topk, indices = torch.topk(support, 10)
 support = torch.zeros([B, N, N], device=self.device).scatter_(2, indices, topk)
</code></pre>

albanD: <p>Hi,</p>
<p>You can use this is a toy example to see which gradient are propagated back to support with something like:</p>
<pre><code class="lang-python">support = torch.rand(20, requires_grad=True)
indices = # Whatever they should be

topk, indices = torch.topk(support, 10)
out = torch.zeros([B, N, N], device=self.device).scatter_(2, indices, topk)

out.sum().backward()
print(support.grad)
# That will give you a 1 for each entry that was selected by the topk
</code></pre>
<p>But the gradients will propagate as you expect here, no problem !</p>
Correctness of an implementation of a Paper

Nagesh: <p>Hey guys,<br>
I have tried to implement the CNN-LSTM in the paper <a href="https://arxiv.org/pdf/1703.04046.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1703.04046.pdf</a>, could someone help me out with the correctness of the implementation.Their implementation in tensorflow is available- for anybody who can understand that. The parameters to layers have been given just to see if the model works. The test loss looks like a step function and the train and test accuracy is always 50% through any epoch, so please help me out.<br>
Here’s a link to my code<br>
<a href="https://controlc.com/eaacf3fd" rel="nofollow noopener">https://controlc.com/eaacf3fd</a></p>
Cross-entropy loss in VAE

Amperius: <p>Hello there,</p>
<p>I’m currently trying to implement a VAE for dimensionality reduction purposes.</p>
<p>As a base, I went on from pytorchs VAE example considering the MNIST dataset.<br>
My own problem however, does not rely on images, but on a 17 dimensional vector of continuous values.<br>
I want to use the VAE to reduce the dimensions to something smaller. Additionally, I use a “history” of these values, to transport information about previous values into the network.</p>
<p>So, overall i get an input for my network, which is of the following shape:<br>
[BATCH_SIZE x HISTORY_LENGTH x FEATURE_SIZE]</p>
<p>In a small example I used the following:<br>
[64 x 3 x 17]</p>
<p>Long story short, when it comes to the calculation of the loss function (I started from the pytorch example <a href="https://github.com/pytorch/examples/blob/master/vae/main.py#L72" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/vae/main.py#L72</a>) I get into some problems.</p>
<p>Due to the fact that I don’t use images as input, I changed the binary cross-entropy to normal cross-entropy, utilizing <code>F.cross_entropy(reconstructed_x, x, reduction='sum')</code> instead of <code>F.binary_cross_entropy(reconstructed_x, x, reduction='sum')</code> and now I get the following error message:</p>
<p><code>ValueError: Expected target size (64, 17), got torch.Size([64, 3, 17])</code></p>
<p>Can somebody tell me what I am missing?</p>
<p>Thanks in advance!</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects targets passed as a <code>LongTensor</code> containing class indices, as it’s usually used for multi-class classification use cases.</p>
<p>I’m not sure, how your input tensor is defined, but I assume it contains float values.<br>
In this case, you won’t be able to use <code>nn.CrossEntropyLoss</code> to reconstruct the input tensor.</p>
<p>Was the mentioned loss function from the VAE implementation not working?</p>

Amperius: <p>Your assumption is correct, my input tensor consists of float values, so this explains to me why it is not working properly.</p>
<p>The <code>F.binary_cross_entropy()</code> function produces very high losses, so I tried with <code>F.mse_loss()</code> instead, which produced better loss values but I assumed that this is not correct, since the MSE loss is not mentioned in the Kingma paper (<a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1312.6114.pdf</a>, see section 3 loss function).</p>
<p>So overall I`m not sure which loss function would suit my case best. I am open to any suggestions at this point.</p>
<p>Maybe there other things, that are likely to be the source of high losses, but I still have to find them.</p>
<p>Thanks for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
JIT Trace model that computes gradients

mark_jimenez: <p>Hi, I just wanted to ask if it’s possible to compute gradients inside a traced (torch.jit.trace) model.</p>
<p>We want to use torchscript to manually train models in mobile devices by computing gradients and recomputing new parameters. I tried to create simple model like this.</p>
<pre><code class="lang-auto">class ReturnGradientsModel(torch.nn.Module):
    
    def forward(self, input, w, b):
        result = input * w + b
        L = (10 - result).sum()
        L.backward()
        
        w = w - w.grad * 1e-2
        b = b - b.grad * 1e-2
        
        return w, b

torch.jit.trace(ReturnGradientsModel(), (torch.rand(2,2, requires_grad=True), 
                                        torch.rand(2,2, requires_grad=True), 
                                         torch.rand(2,2, requires_grad=True)))
</code></pre>
<p>but it just returns the error</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>even though I explicitly passed tensors with grad enabled.</p>
<p>I found this issue : <a href="https://github.com/pytorch/pytorch/issues/15644" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/15644</a> . But it’s for onnx, not sure if it applies to torch.jit.trace as well.</p>
<p>I’m grateful for any help.</p>
How to free graph manually after using retain_graph=True?

cyanM: <p>For some reasons, I use retain_graph = True and hook to get the gradient while backward, but this will lead to the gpu memory leak because the computation graph is not released. so how can i free graph manually?</p>

ptrblck: <p>If I’m not mistaken, the graph should be cleared once all attached tensors are deleted.<br>
Since Python uses function scoping, you could wrap your calls in a function and the computation graph should be freed once you leave the function scope (if you don’t return the output tensor and keep it alive of course).</p>

cyanM: <p>yes, some inner variables have not been released while using hook, there is no memory leak after delete all of them.</p>
Rotate an image using torchvision.transform for a tensor?

Peter_Ham: <p>I found that torchvision.transform only works for PIL image type of objects. However, I want to apply random rotations on torch tensor and I want to get the gradient of the input images, so I can’t convert it to PIL first and then apply the transform.</p>

ptrblck: <p><a href="https://github.com/kornia/kornia">kornia</a> might provide, what you are looking for.<br>
CC <a class="mention" href="/u/edgarriba">@edgarriba</a></p>

Peter_Ham: <p>I want to pass an image into a network and perform gradient update on the image. But I also want to add random perturbation on the image such as rotations and jittering, but not the PIL type of image but on the tensor. However, all torchvision transformations of these types are on PIL type image not on tensor.</p>

ptrblck: <p>I haven’t used kornia that much, but it should work on tensors as seen <a href="https://github.com/kornia/kornia/blob/master/examples/warp_affine.py">here</a>.</p>

ProGamerGov: <p>I would recommend searching for NumPy solutions to issues like these, as they are extremely easy to recreate in PyTorch. PyTorch has many of the same functions as NumPy, and you can also use indexing (which is the exact same in both libraries).</p>

ProGamerGov: <p>It looks like you could generate a random tensor, and then add it to the tensor you want to add jitter. If you want more of a random shifting type jitter, I wrote code for that a few days ago here: <a href="https://discuss.pytorch.org/t/how-to-split-a-c-x-h-x-w-tensor-into-tiles/64729/2" class="inline-onebox">How to split a (C x H x W) tensor into tiles?</a></p>
<p>For the rotations, you could probably also use <code>Torch.flip()</code>, and <code>Torch.diag()</code> if the rotations are using 90 degree intervals.</p>

Peter_Ham: <p>Thanks! That is also what I did!</p>

edgarriba: <p>take a look at the augmentation api we are developing. It’s all tensor based and tries to match torchvision interface: <a href="https://kornia.readthedocs.io/en/latest/augmentation.html" rel="nofollow noopener">https://kornia.readthedocs.io/en/latest/augmentation.html</a></p>
<p>More to come: <a href="https://github.com/kornia/kornia/issues/312" rel="nofollow noopener">https://github.com/kornia/kornia/issues/312</a></p>
Quaternions Loss function

Jaideep_Valani: <p>Is there any impmentation of quaternions loss function basically to regress Angles correctly.<br>
for a poses.</p>

edgarriba: <p>in kornia we have some utilities for this: <a href="https://kornia.readthedocs.io/en/latest/geometry.conversions.html" rel="nofollow noopener">https://kornia.readthedocs.io/en/latest/geometry.conversions.html</a></p>
Jacobian matrix for the ouput

aishahss: <p>hello every one,<br>
Is there a way in PyTorch to get the derivation from the net.<br>
I want to find the distance between two points, say x and y as follows:</p>
<p>The output of the encoder is x and y:</p>
<p>output_feature ,class=model(input)</p>
<p>x and y are the output_feature [2,2048]</p>
<p>the input is [2,8,3,128,128]</p>
<p>The distance to be calculated : (x-y)(transpose)   J(transpose)  J  (x-y).</p>
<p>with J being the Jacobian matrix wrt the encoder.</p>
<p>how can I calculate J in PyTorch?</p>
Can't get any inference speed up using FP16 in 2080ti compared with fp32

Edwardmark: <p>I convert a model trained in FP32, and I use model.half() and input.half() to do inference in fp16 precision, but the inference speed is almost same with fp32 in 2080ti, my batch size is fixed to 1, why is that? How can I speed up using fp16? Thanks.</p>

ptrblck: <p>Answered <a href="https://discuss.pytorch.org/t/cnn-fp16-slower-than-fp32-on-tesla-p100/12146/4">here</a>.</p>
I want to the pretrained model of ResNet 101

JasonChenhx: <p>I want to the pretrained model of ResNet101. I can’t download the pretrained mode in github successfully.</p>

ptrblck: <p>You should be able to get the pretrained <code>resnet101</code> from <code>torchvision</code> as:</p>
<pre><code class="lang-python">import torchvision.models as models
model = models.resnet101(pretrained=True)
</code></pre>
<p>Do you get an error using this code?</p>
The behavior of model.parameters()

phantom90: <p>Hi,</p>
<p>I am writing a pytorch model by myself and use model.parameters() and model.named_parameters() to access the parameters.</p>
<p>As one can see, both of them are iterators. When we use them to get a list of parameter tensors,</p>
<ol>
<li>Do they follow the same order?</li>
<li>If I save/load the model again, is the order identical to the original order.</li>
</ol>
<p>Thanks very much~</p>

JuanFMontesinos: <ol>
<li>Yes, in fact model.parameters just yield values from named_parameters</li>
<li>It depends on the code rather than the weights, if the code is the same, the order is the same.</li>
</ol>
Self.class_to_ind: {'p': 4, 'o': 1, 'l': 2, 'y': 3}

111215: <p>The result I want is  self. Class_to_ind: {‘polyp’: 0}.But the result of the output is the  self. Class_to_ind: {’ p ', 4 ‘o’ : 1, ‘l’, 2, ‘y’ : 3}.Here’s my code.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/16eeca35d863af90d67887e9f6e1a801cd432a86.png" data-download-href="https://discuss.pytorch.org/uploads/default/16eeca35d863af90d67887e9f6e1a801cd432a86" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16eeca35d863af90d67887e9f6e1a801cd432a86_2_345x159.png" alt="image" data-base62-sha1="3gS6Mdk8AOq3zQWDx7h2bhPLcRE" width="345" height="159" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16eeca35d863af90d67887e9f6e1a801cd432a86_2_345x159.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16eeca35d863af90d67887e9f6e1a801cd432a86_2_517x238.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16eeca35d863af90d67887e9f6e1a801cd432a86_2_690x318.png 2x" data-dominant-color="EFF0F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">982×453 59.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Eta_C: <p>Check your <code>VOC_CLASSES</code>, it should be <code>["polyp"]</code>, instead of <code>"polyp"</code>.</p>

111215: <p>thank you,Problem solved</p>

111215: <p>Hi, I debugged my code.The above problem is solved, but a new problem appears: “RuntimeError: copy_if failed to synchronize: device-side assert triggered”.I’m pretty sure this problem is caused by num_class, so I don’t think VOC_CLASSES = ([“polyp”]) can be rewritten this way because it changes the category, right?What should I do?</p>
How to load custom TorchScript ops in PyTorch Mobile?

dalalaa: <p>Is there any tutorials or examples about loading custom TorchScript ops in PyTorch Mobile</p>

dalalaa: <p><a class="mention" href="/u/david_reiss">@David_Reiss</a>  <a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a>  Could I load custom TorchScript ops in PyTorch Mobile?</p>

jspisak: <p>what are the custom ops you are trying to script? If the model is scriptable you shouldnt need to do anything else (unless I am missing something).</p>

tom: <p>If you create a library (.so) that features your custom op registration as a static variable and link that into your app, it’ll work. That’s how I did it back in 2018 and it should still work.</p>
<p>Best regards</p>
<p>Thomas</p>

David_Reiss: <p>Thomas is correct.  Here’s a demo of using the C++ API directly, which should allow custom op registration: <a href="https://github.com/ljk53/pytorch-android-cpp-demo">https://github.com/ljk53/pytorch-android-cpp-demo</a></p>
<p>I know the process is a bit cumbersome.  We’re looking at simplifying this over time.</p>
How to add preprocess into the model

MarcSteven: <p>Is it possible to add preprocess into the model? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Please don’t tag specific people in a post, as this might discourage others to post an answer and also the tagged user might not even know the best answer.</p>
<p>That being said: as explained in your other threads, yes that’s possible.<br>
E.g. you could add it in your <code>forward</code> method.</p>

MarcSteven: <p>Thank you , but the machine learning engineer told me it’s impossible to add the preprocess into the model. I reviewed the code and then find it doesn’t have the code (No normalize), does it influence the effect?<br>
We compose a sequence of transformation to pre-process the image:</p>
<pre><code class="lang-auto">import torchvision.transforms as transforms

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
</code></pre>

ptrblck: <p>It is possible as shown here:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self, transform):
        super(MyModel, self).__init__()
        self.conv = nn.Conv2d(3, 1, 3, 1, 1)
        self.transform = transform
        
    def forward(self, x):
        xs = []
        for x_ in x:
            x_ = self.transform(x_)
            xs.append(x_)
        xs = torch.stack(xs)
        x = self.conv(xs)
        return x

transform = transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
model = MyModel(transform)
x = torch.randn(1, 3, 24, 24)
output = model(x)
</code></pre>
<p>If you were using normalization in your PyTorch model, you have to use the same preprocessing pipeline in your CoreML model.</p>

MarcSteven: <p>Yeah if I didn’t do the normalize, it means I won’t do preprocess in coreML, right?</p>

ptrblck: <p>Yes, if you remove the preprocessing in PyTorch, retrain the model and the training looks good without any preprocessing (which might be hard), you could skip the preprocessing as well in your CoreML deployment.</p>

MarcSteven: <p>But in CoreML you must tell the range of pixel buffer,or it doesn’t know how to deal with the data. In other deep learning framework it has image preprocess. But the machine learning engineer told me it’s no way to add preprocess in the model. You must make a standard rule to let us know each other , so the image preprocess is the basic rule to let us know each other.</p>
Question about crossentropyloss

chunchun: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f79dc840411ed5ae8d8d9b852ce34dd137d837e7.png" alt="image" data-base62-sha1="zkvYT4UXXss87usqIPGElYbv0mr" width="253" height="205"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/8/7878bad17df7070b96e6be6e12f68b5376b1b8fc.png" alt="image" data-base62-sha1="hbJXrvGQUkv6KNBvKi9uYfoba6o" width="249" height="142"></p>
<p>I want to konw how can I get the loss=0.8133<br>
could you please tell me the details such as which one multiply which one to get this result?<br>
Thanks.</p>

albanD: <p>According to the formula in <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss">the doc</a>:</p>
<pre><code class="lang-auto">sample 0:
- x[1] + log ( sum(exp(x)) ) = 0.3133 + log( 0.2689 + 0.7310 ) = 0.3133 - 3.831e-5 = 0.3133
sample 1:
- x[0] + log ( sum(exp(x)) ) = 1.3133 + log( 0.2689 + 0.7310 ) = 1.3133 - -3.831e-5 = 1.3133
average of the results:
(1.3133 + 0.3133)*0.5 = 0.8133
</code></pre>

chunchun: <p>Thanks a lot.That’s very clear.</p>
Building custom LSTM with packed Sequences

amirshane: <p>I’m trying to build an LSTM variant, specifically the multiplicative LSTM (<a href="https://arxiv.org/abs/1609.07959" rel="nofollow noopener">https://arxiv.org/abs/1609.07959</a>). I want to use this to train a language model on data that varies quite a bit in sentence length so packing is very appealing. I think I can build an mLSTM using a slight modification of the LSTM operations as well as calls to LSTMCell, but I’m not sure how to use packing. All of the examples I’ve seen using torch.nn.utils.pack_padded plug the packed sequence into a built-in RNN method like nn.LSTM, which handles everything under the hood. Im not sure how to approach this and I’m not entirely certain as to how the parallelization is done with the packed sequence (how does the model know what to evaluate when computing losses?). Would appreciate any advice</p>
Defining Number of Kernels for Multi channel cnn

the_coder: <p>I am implementing a multichannel CNN. It is in time series data. and i am implementitng 1D convolution. I could not find how i can define the number of kernels. Through search i am aware that the number of output channel is equal to number of filters. But i have output channels already defined. which is 252. Shall i just omit 252 and put my desired number of filters in there? which is 8 for the first conv layer and for the second is 4. Also do i have to write nn.sequential for the second layer too ? This is my code below:</p>
<pre><code class="lang-auto">self.conv1 = nn.Sequential(
                    nn.Conv1d(in_channels=256, out_channels=252, kernel_size=5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                    nn.Dropout(p=0.25))  
self.conv2 = nn.Conv1d(in_channels=126, out_channels=122, kernel_size= 5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                     nn.Dropout(p=0.25)
</code></pre>
Pytorch 0.4.1 getting gradient at a particular node

Naman-ntc: <p>What is the standard way to get the gradient value at a particular node in pytorch 0.4.1<br>
When I try to access the gradient of a particular tensor in graph using the .grad attribute I get <code>None</code>.</p>
<p>Is <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.grad" rel="nofollow noopener">torch.autograd.grad</a> the approach I should follow.</p>
<p>Also is the None coming to free up memory. (I also tried using retain_graph=True in backward function but it didnt change anything ). <a href="https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94/3">Here</a> is a past link to this kind of discussion but it mainly talks about Variable API.</p>

ptrblck: <p>You have to call the backward pass before accessing <code>.grad</code>, otherwise it will be <code>None</code>:</p>
<pre><code class="lang-python">model = nn.Linear(10, 2)
x = torch.randn(1, 10)
output = model(x)
output.mean().backward()
print(model.weight.grad)
</code></pre>

Naman-ntc: <p>Yeahh I called the backward function!!</p>

ptrblck: <p>After the backward call you tried to print the gradients directly or did you perform some other operations, e.g. <code>optimizer.step()</code>?<br>
In the former case, could you post your model code? Maybe the computation graph is somehow detached at a certain point.</p>

Naman-ntc: <p>Hii, I forgot posting my code after replying earlier</p>
<p>This is the relavant code</p>
<pre><code class="lang-auto">         data, target = data.to(device), target.to(device)
         optimizer.zero_grad()
         output = model1(data)
         finalOutput = model2(data - factor*output)
         loss = F.nll_loss(finalOutput, target)
         loss.backward()
         print(output.grad)
         print(finalOutput.grad)
         optimizer.step()
</code></pre>
<p><code>None</code> is being printed!!</p>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94/2?u=ptrblck">this thread</a>.<br>
<code>register_hook</code> is probably, what you want.</p>

Naman-ntc: <p>Oh I looked at that link but missed that answer while skimmin:sweat_smile:<img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=6" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=6" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
Thanks!!</p>
Word prediction with RNN

FrancescoMandru: <p>I’m looking for a detailed tutorial / explanation about building a RNN for predicting the next word of a phrase. I have the embeddings of each word obtained with Word2Vec. Now I’m trying to understand how to build the network for the prediction of the next word given a phrase of length N, for example.</p>
<p>INPUT —&gt; The cat is on the<br>
OUTPUT --&gt; Softmax of some vectors (Table, Sofà, Kitchen etc…)</p>

zhangguanheng66: <p>the word language example might help. <a href="https://github.com/pytorch/examples/tree/master/word_language_model" rel="nofollow noopener">https://github.com/pytorch/examples/tree/master/word_language_model</a></p>
libtorch cpu inference servering multi request thread

leowgyang: <p>I’ve been using libtorch to run my pytorch model in a single thread.<br>
The process is:</p>
<ol>
<li>module = jit::load(mode)</li>
<li>out_tensor = module.forward()</li>
</ol>
<p>In the online environment, we use multiple threads to handle requests. Like in tf, the process is:</p>
<ol>
<li>model = load_graph() in parent thread</li>
<li>session = create_session(model), create_session in each child threads. These child threads share the weight of the model</li>
<li>run_session in each child theads.</li>
</ol>
<p>How do I implement this structure in libtorch?<br>
Please help me! Thanks.</p>

novioleo: <p>you ask yourself…ahahahahah.</p>
Some problem with hook

Weng_zhiqiang: <p>Hi everyone,<br>
I’m tring to visualizing the feature map in my network. The code is as follow:</p>
<pre><code class="lang-auto">def hook_fn_backward(module, grad_input, grad_output):
    # print(module)
    # print('grad_output', grad_output)
    # print('grad_input', grad_input)
    total_grad_input.append(grad_input)
    total_grad_output.append(grad_output)

...
  PD2SE = PD2SEModel()

    # PD2SE = torch.load("./models/55.pth")
    PD2SE.to(device)
    modules = PD2SE.named_children()
    test = list(modules)
    for module in test[0][1]:
        if isinstance(module, nn.ReLU):
            module.register_backward_hook(hook_fn_backward)
...
</code></pre>
<p>By doing this, it exactally could output the feature map as I want(maybe not?) But I don’t know how to close it, it takes CUDA out of memory error. And I’ve read some documents about this, which says I should remove the hook. So my question is where should I remove it. Indeed, I think I could set a function save the feature map in local memory. Then I could close it since the code use the hook again. Looking forward your reply.<br>
<a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

albanD: <p>Hi,</p>
<p>Just as a note, check the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_backward_hook">doc</a> for register backward hook. In particular the warning that it might not return what you want in some cases at the moment.</p>
<p>The memory problem here is that you do <code>total_grad_input.append(grad_input)</code>. So that list will contain more and more Tensors. You want to make sure to remove them when you don’t need them anymore.</p>

Weng_zhiqiang: <p>Thanks for your reply. So I just need remove some data in that list I won’t use then this error will be done. if so, maybe I could save the data in my pc local memery then remove them all? Maybe I should try just use the hook with Tensor. After I change it, I will put the result there. And sorry to ask this stupid question. If you  have free time, could you check this code which I inplement the Focal Loss. Since I use the loss, the value of loss return just like -2000, -1980…</p>
<pre><code class="lang-auto">class FocalLoss(nn.Module):

    def __init__(self, alpha=0, gamma=0, eps=1e-7):
        super(FocalLoss, self).__init__()
        self.alpha = alpha
        self.gamma = gamma
        self.eps = eps
        self.ce = nn.CrossEntropyLoss()

    def forward(self, input, target):
        logit = self.ce(input, target)
        p = torch.exp(-logit)

        if self.alpha:
            loss = -1 * self.alpha * (1 - p) ** self.gamma * logit
        else:
            loss = torch.clamp(-((1 - p) ** self.gamma) * logit, self.eps)  # focal loss

        return loss.mean()

</code></pre>
Is hidden_state in place operation as shown in this example?

HuangChuan: <p>In the following example, the variable <strong>hidden</strong>  is changed  every time step, is this assignment <strong>in place operation</strong>?  If it is, does calling <strong>backward()</strong> yield correct gradient?<br>
Thank you!</p>
<pre><code class="lang-auto">def train(category_tensor, line_tensor):
    rnn.zero_grad()
    hidden = rnn.init_hidden()
    
    for i in range(line_tensor.size()[0]):
        output, hidden = rnn(line_tensor[i], hidden)

    loss = criterion(output, category_tensor)
    loss.backward()

    optimizer.step()

    return output, loss.data[0]
</code></pre>

albanD: <p>When you do <code>hidden = </code> this is not an inplace operation. It only change the python object which is associated with the name “hidden”.</p>
<p>Also the autograd makes sure the inplace operations are valid and will raise an error if useful data are deleted.</p>

HuangChuan: <p>Thanks. Just to make it clear. So these objects, <strong>hidden</strong>, are still in memory when autograd is called?</p>

albanD: <p>It depends.<br>
If they are needed to compute gradients, yes the autograd will keep them in memory.<br>
If they are not needed to compute gradients, they will be freed when you associate a new Tensor to the python variable.</p>
Is this the correct way?

phantom90: <p>Hi</p>
<p>Suppose we have a expression: z = x^2 *y + y, where x, y are both Tensors.</p>
<p>We want to calculate d(d z / d x) / dy. Here d is the partial derivative operator.</p>
<p>I do as follows:</p>
<pre><code class="lang-auto">import torch

x = torch.Tensor([3])
y = torch.Tensor([17])

x.requires_grad_()
y.requires_grad_()

loss = x**2 * y + y
loss.backward(create_graph=True, retain_graph=True)

xx = x.grad
xx.requires_grad_()

result = torch.autograd.grad(xx, y)
</code></pre>
<p>This piece of code can get correct result. However, I wander if this is the correct way to calculate, or I just get the correct result by coincidence.</p>
<p>Thanks!</p>

Mazhar_Shaikh: <p>This is correct. The x.grad tensor is the dz/dx term. By setting the create_graph parameter to True, the graph of the derivatives is created and a backward function is attached to the x.grad tensor. Hence, deriving gradient of x.grad w.r.t y is the correct way to calculate the value of d(dz/dx)/dy</p>
Pytorch dataloader for a dataset with a lot of images

Isaac_Kargar: <p>Hi,</p>
<p>I have a very big dataset with a lot of images. Is there any way faster than loading single images in <strong>getitem</strong> method? I want to increase the batch size and want to find the bottleneck. the batch size, size of my network, …</p>
<p>Thank you</p>

Tahir: <p>The loading of images onto RAM and GPU can be speed up through custom loaders but speeding the training through larger batch sizes is something different.</p>
<p>if you just want to speed up the load of images, have a look here on copying the data to GPU in parallel.</p>
<aside class="quote quote-modified" data-post="2" data-topic="48762">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tahir/48/14280_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-do-i-copy-data-to-gpu-in-parallel/48762/2">How do I copy data to GPU in parallel?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    How about writing your own data loader in a separate thread? This can run in parrallel to your training loop. 
The data loading can look like this (I wrote the data caching in RAM but it can work for parrallel caching in GPU as well, i think): 
(cache_size is the number of images or data to be cached parrallelly) 
from threading import Thread
from time import sleep
import queue 
import time


trfqueue = queue.Queue(maxsize=20) 
tsfqueue = queue.Queue(maxsize=20) 

 
cache_size= 10
templist = []
…
  </blockquote>
</aside>


Isaac_Kargar: <p>So using this way I can increase the training speed right? My final goal is to train the network faster. I added a second RTX 2080Ti and use nn.DataParallel and now it’s almost twice faster, but I want it to be faster.</p>

Eta_C: <p>Maybe you can use <code>hdf5</code> or <code>lmdb</code> format.</p>

cgpeter96: <p>The tools come from NVIDIA----&gt;DALI  . But I never use it <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>

Tahir: <p>yes, it should make the training faster by loading the data faster. You might can do other optimizations to make faster as well, e.g. batch size, network size etc.</p>
About build Embedding_lookup.cc

liuchanfeng165: <p>I’m trying to find an equivalent of embedding_lookup in Caffe2 built in VS2015.<br>
Seems there is a C++ interface, but the compiler  couldn’t  compile it   successfully because ’ caffe2::DeviceTypeProto’ is not defined and I found it in caffe.proto but didnt know how to include  caffe.proto  .<br>
Could u tel me ? thanks.</p>
KeyError: 'polyp'

111215: <p>Experts, ‘polyp’ is my category, and only this category, and I’m running on a network called ‘RefineDet.’ how can I fix this?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/8/78133d5f9ac08abef402d3cdadf3ef3cdd63a2c3.png" data-download-href="https://discuss.pytorch.org/uploads/default/78133d5f9ac08abef402d3cdadf3ef3cdd63a2c3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78133d5f9ac08abef402d3cdadf3ef3cdd63a2c3_2_690x270.png" alt="image" data-base62-sha1="h8ew4Uxw2EHWvvH0pQcrjxxvhWX" width="690" height="270" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78133d5f9ac08abef402d3cdadf3ef3cdd63a2c3_2_690x270.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78133d5f9ac08abef402d3cdadf3ef3cdd63a2c3_2_1035x405.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/78133d5f9ac08abef402d3cdadf3ef3cdd63a2c3_2_1380x540.png 2x" data-dominant-color="F1EEF6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1895×742 139 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>It seems the error is raised, since <code>'polyp'</code>  isn’t defined in the <code>self.class_to_ind</code> dict.<br>
Also it seems your data loading pipeline is implemented in <code>RefineDet.PyTorch-master/data</code>, so you might get a better answer from the maintainers of the used repository.</p>

111215: <p>Thank you for your valuable advice. The problem has been solved</p>