akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
About Save Files(.pt) and Reload Inference Accuracy

username: xiaoliu_xiaoliu
cooked: <p>There is a problem happened on my case at Libtorch1.1.0<br>
Training and Directly Inference get a normal and correct result, however, Training-Save and Load-Inference get a innormal and absolutely wrong result.<br>
Any HELP?</p>
<p>Here is my flow path of my code, Step 6 output is same but Step 7 not and different:<br>
Step 1 Model Initialization<br>
Step 2 Model Training(printf:output result with an all-ones intput at last training step)<br>
Step 3 torch::save(model,“D:\path\to\model.pt”);<br>
Step 4 Initialization model2<br>
Step 5 torch::load(model2,“D:\path\to\model.pt”);<br>
Step 6 std::cout&lt;parameters()[i]&lt;&lt;std::endl;<br>
std::cout&lt;parameters()[i]&lt;&lt;std::endl;<br>
Step 7 torch::Tensor t1=model-&gt;forward(torch::ones(1,1,224,224).to(torch::kCUDA));<br>
torch::Tensor t2=model2-&gt;forward(torch::ones(1,1,224,224).to(torch::kCUDA));</p>

username: CF-chen-feng-CF
cooked: <p>:失望的:<br>
me too<br>
What can i do？</p>
Handle variable sized sentences for text classification with CNN(+RNN)

username: scarecrow21
cooked: <p>Hi all,</p>
<p>I’m trying to implement a text classifier using Conv1d. My dilemma is - how do we construct the linear layer to consume the output from the final CNN/MaxPool layers.</p>
<p>I can pad all the sentences in a batch to have the same length as the longest sentence in that batch and set the input to linear layer accordingly but if the next batch has a different seq_len, then I’ll still get an error.</p>
<p>Eg:</p>
<pre><code class="lang-auto">batch_size = 2

Batch 1:
[1, 2, 3, 4, 5]
[1, 2, 3, 0, 0]

Batch 2:
[1, 2, 3, 4, 5, 6, 7, 8]
[1, 2, 3, 4, 5, 0, 0, 0]

# The CNN/Maxpool output for both these batches would be different. 
# As a result, when I flatten this out to feed into the linear layer,
# the number of neurons in the linear layer would have to be different. 
</code></pre>
<p>The only idea I have right now is to pad all the sentences in my dataset to have the same length(mean/median of sentence lengths) and pad/remove the words from each sentence according to the length criteria.</p>
<p>Is there any other way to get around this?</p>
<p>Edit:<br>
How would we go about implementing:</p>
<ul>
<li>CNN -&gt; Linear</li>
<li>CNN -&gt; RNN -&gt; Linear</li>
</ul>

username: tom
cooked: <p>The typical thing is to do a reduction over the time axis, like max or mean.<br>
fast.ai’s (with the reference being Howard and Ruder’s ULMFiT paper) advocate using max, mean <em>and</em>last element for RNNs. As for CNNs the last element is hard / not that meaningful, you would concatenate max and mean.</p>
<p>Best regards</p>
<p>Thomas</p>

username: scarecrow21
cooked: <p>Please correct me if I understood this wrong.</p>
<p>Say, the penultimate layer after multiple Conv1D and Maxpool operations is of size (10, 100, 5) where:</p>
<pre><code class="lang-auto">batch_size = 10
filters(out_channel) = 100
seq_len = 5
</code></pre>
<p><strong>CNN -&gt; Linear</strong><br>
So, I will obtain the max (or mean) of the last dimension ie. the one with 5 elements. So, the shape now would be (10, 100). This (10, 100) layer will now be flattened into (1, 1000) layer and passed into the linear layer to obtain the output ie. a single class.</p>
<p><strong>CNN -&gt; RNN -&gt; Linear</strong><br>
How would I go about doing something similar for CNN-&gt;RNN-&gt;Linear?</p>

username: tom
cooked: <p>CNN -&gt; Linear:</p>
<p>No. The batch size should never be flattened.<br>
You keep  (10, 100) and feed it into a (100, 1) linear (or whatever) to get a 10, 1 - one prediction per batch item.<br>
You could concatenate the (10, 100) max with the mean (over the seq_len=5) to get a (10, 200).</p>
<p>CNN -&gt; RNN -&gt; Linear</p>
<p>Here you would feed (after permuting the axes) the (5, 10, 100) into the RNN (say 100-&gt;100) and then get something of the same shape.<br>
Then you could take <code>torch.concat([o.max(0), o.mean(0), o[1], -1)</code> of shape 10, 300 and feed to a linear as above.</p>
<p>Best regards</p>
<p>Thomas</p>
Libtorch Input Batch

username: airwater
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae9caf4d5b672273188db1ae51ea57573997d814.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae9caf4d5b672273188db1ae51ea57573997d814" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae9caf4d5b672273188db1ae51ea57573997d814_2_690x118.png" alt="image" data-base62-sha1="oUGM4MHCn7MmAYv9yFZAFibvsos" width="690" height="118" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae9caf4d5b672273188db1ae51ea57573997d814_2_690x118.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae9caf4d5b672273188db1ae51ea57573997d814_2_1035x177.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae9caf4d5b672273188db1ae51ea57573997d814.png 2x" data-dominant-color="252625"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1101×189 32.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The above is the data preprocessing stage. The batch input to the model is one.<br>
Now, I want to set batch to four, because will process four channels of video at the same time.<br>
How should I do ? Hope get your help. Thank you !</p>

username: 111169
cooked: <p>You can input a batch when testing like this:<br>
at::Tensor input1 = torch::ones({1,3,128,128});<br>
at::Tensor input2 = torch::ones({1,3,128,128});<br>
at::Tensor inputs = torch::cat({input1,input2},0);<br>
at::Tensor output = module-&gt;forward({inputs}).toTensor();</p>
Create dictionary of module layers

username: wasiahmad
cooked: <p>In pytorch, we can create list of modules using <code>ModuleList</code>. Can we create a dictionary of modules? I need to set a name along with a layer. Any help would be appreciated.</p>

username: ptrblck
cooked: <p>Have a look at the <a href="http://pytorch.org/docs/master/nn.html#torch.nn.Sequential" rel="nofollow noopener">Sequential example</a>. There they use an <code>OrderedDict</code> for this purpose.</p>
<p>Hope this helps!</p>

username: wasiahmad
cooked: <p>In that example, they wrapped the orderdict using nn.Sequential to make a single layer. In my case, I want to store a certain number of layers in a dictionary and later I want to use them based on conditions.</p>

username: dhpollack
cooked: <p>The modules are stored as an ordered dictionary in nn.Sequential.  You can directly access them with ._modules.</p>
<pre><code>model = nn.Sequential(OrderedDict([
            ("layer_one", nn.Linear(1, 10)), ("layer_two", nn.Linear(10, 3))
        ]))
model._modules["layer_one"]
model._modules["layer_two"]</code></pre>

username: jefflee
cooked: <p>We have ModuleDict now, which is easier to use <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
Ideal dataset format for Adversarial Autoencoder?

username: gpr
cooked: <p>Hi,</p>
<p>I am building an Adversarial Autoencoder and I am interested in hearing others’ feedback about the ideal way of representing my dataset which is not imagery / pixel data but rather a large set of records related to a digital acquisition system. I have been pulling sample data from a temperature thermistor along with a bunch of other environmental variables from a piece of equipment and would like to create a generative model that is able to mimic the behavior of the system by using the thermistor temperature column as a label for each CSV row within the dataset file.</p>
<p>The dataset file looks like this:</p>
<p>aaa,aab,aac,aad,aae,aaf…mmm,mmn,mmo,mmp,label<br>
bytes,bytes,bytes,bytes,bool,bool,bool…int,int,int,float</p>
<p>So I’ve got a mixture of different datatypes and my thoughts were to first cast each column to an integer and then create a vector from each int, followed by normalization. Those vectors would would then be the input to the encoder network to create the latent space for the AAE.</p>
<p>Is this a logical approach or should I be handing everything to the encoder network input layer as binary data similar to as how pixelwise AAEs do?</p>
<p>I am essentially looking to sample from the latent space similar exemplars pulled from the thermistor in the future, for determining from the generative model what state the rest of the machine should be in based upon generated samples if that makes sense.</p>
<p>Thanks in advance for any feedback!</p>
How to avoid overfitting

username: Prak_kas
cooked: <p>I am implementing logistic regression using pytorch. I am getting 100% accuracy for training. To reduce overfitting, I used PCA and scaled the data ,<br>
changed the shapes and ran the program again… But still I am getting same 100% accuracy !!</p>
<p>Iteration: 100. Loss: 0.8057927637150751.Correct:601. total:601. Accuracy: 100.0.<br>
Iteration: 200. Loss: 0.7555791968181238.Correct:601. total:601. Accuracy: 100.0.<br>
Iteration: 300. Loss: 0.8404111710965548.Correct:601. total:601. Accuracy: 100.0.<br>
Iteration: 400. Loss: 0.7050717906166214.Correct:601. total:601. Accuracy: 100.0.<br>
Iteration: 500. Loss: 0.7241441765733534.Correct:601. total:601. Accuracy: 100.0.<br>
Iteration: 600. Loss: 0.8043209660671171.Correct:601. total:601. Accuracy: 100.0.<br>
Iteration: 700. Loss: 0.6430073846611701.Correct:601. total:601. Accuracy: 100.0.<br>
Iteration: 800. Loss: 0.7107452274836563.Correct:601. total:601. Accuracy: 100.0.<br>
Iteration: 900. Loss: 0.7869175416749621.Correct:601. total:601. Accuracy: 100.0.</p>
<p>suggestion to overcome  overfitting is is appreciated.</p>

username: ptrblck
cooked: <p>How are your classes distributed and how many classes do you have?<br>
Also, what’s the validation and final test accuracy?</p>

username: Prak_kas
cooked: <p>I have only two classes (class 1 and class 0)</p>
<p>I have 947 points in class “1” and   2980 points in class “0”</p>

username: ptrblck
cooked: <p>Thanks for the information.<br>
Did you check the validation and test accuracy?</p>

username: Prak_kas
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> , thing is that I am getting training accuracy as 75.884 %  and Validation accuracy as 100%   quite weird !!</p>

username: mailcorahul
cooked: <p>Could be because of bad train-val split. Make sure the data distribution remains same across train and val splits.</p>
Downsample got error when converting model to .pt using script method

username: RebirthT
cooked: <p>The error comes from the torchvision itself:</p>
<pre><code class="lang-auto">attribute 'downsample' of type 'NoneType' is not usable in a script method (did you forget to add it __constants__?):
at /nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torchvision/models/resnet.py:109:12
        out = self.relu(out)
</code></pre>
<p>Original post is here --&gt; <a href="https://stackoverflow.com/questions/57602574/pytorch-torchscript-error-attribute-downsample-of-type-nonetype-is-not-u" rel="nofollow noopener">original post</a></p>
<p>Thanks in advance!</p>

username: driazati
cooked: <p>This was a bug in torchvision, it’s fixed with <a href="https://github.com/pytorch/vision/pull/1343" rel="nofollow noopener">this PR</a>.</p>
<p>We’re working on pushing a new release soon that will include this fix (among many others!). To get the fix locally today, you can <a href="https://github.com/pytorch/vision#installation" rel="nofollow noopener">install torchvision master from source</a>.</p>

username: RebirthT
cooked: <p><a class="mention" href="/u/driazati">@driazati</a> Thanks for your prompt reply<br>
After adding <code>__constant__ = ['downsample']</code> I got the error below:</p>
<pre><code class="lang-auto">RuntimeError: attempting to re-assign constant 'downsample' in WeakScriptModuleProxy
</code></pre>
<p>Thanks</p>

username: driazati
cooked: <p>Are you on the latest version of PyTorch (v.1.3.0)? You can check with <code>print(torch.__version__)</code>. <code>WeakScriptModuleProxy</code> was removed prior to the 1.3 release, so you may need to update to get the right fixes.</p>

username: RebirthT
cooked: <p><a class="mention" href="/u/driazati">@driazati</a>  I’ll check my libtorch version,<br>
Thanks for sharing !!!</p>
Segmentation Fault when assign tensor on Raspberry Pi

username: elepherai
cooked: <p>HW: Raspberry Pi 3B+<br>
OS: Raspbian Buster<br>
Pytorch: 1.1.0<br>
Build from source.</p>
<p>The code bellow works fine on PC, but there’s a Segmentation Fault on Raspberry Pi.</p>
<p><strong>import torch</strong><br>
<strong>x = torch.tensor([[1.]])</strong><br>
<strong>y = torch.tensor([[0.]])</strong><br>
<strong>y[:, 0] = x[:, 0]    # Segmentation Fault here</strong></p>
Practical use case for _extra_files?

username: wnnaone
cooked: <p>Can someone help me understand when and how to use the _extra_files parameter is <code>torch.jit.save</code> and <code>torch.jit.load</code>? The code example in the documents (with <code>extra_files['foo.txt'] = 'bar'</code>) didn’t help me understand what its practical use would be.</p>

username: driazati
cooked: <p>It lets you wrap up any extra data you want into the output of <code>torch.save</code> and have it be decoded for you by <code>torch.load</code>. Maybe you want to ship some documentation or versioning metadata with your saved model to help with deployment, <code>extra_files</code> lets you do that without having to ship files separately alongside your <code>.pt</code> file.</p>
<p>The <code>.pt</code> file from <code>torch.jit.save</code> is a zip file, so any <code>extra_files</code> just get added to the zip archive.</p>
<p>To dig into the docs example a little more, for this code:</p>
<pre><code class="lang-auto">class M(nn.Module):
    def forward(self):
        pass

m = torch.jit.script(M())
extra_files = torch._C.ExtraFilesMap()
extra_files['foo.txt'] = 'bar'
torch.jit.save(m, 'scriptmodule.pt', _extra_files=extra_files)
</code></pre>
<p>Then in a shell</p>
<pre><code class="lang-auto">$ unzip scriptmodule.pt
Archive:  scriptmodule.pt
 extracting: scriptmodule/version    
 extracting: scriptmodule/extra/foo.txt  
 extracting: scriptmodule/data.pkl   
 extracting: scriptmodule/code/__torch__.py  
 extracting: scriptmodule/code/__torch__.py.debug_pkl  
 extracting: scriptmodule/constants.pkl  
$ cat scriptmodule/extra/foo.txt
bar
</code></pre>
Chaining pytorch modules

username: markl
cooked: <p>I am trying to chain two <code>torch.nn.Module</code>s in python and then jointly train their parameters. Suppose they are <code>module_a</code> and <code>module_b</code>.</p>
<p>When simplified, my code looks something like this</p>
<pre><code class="lang-auto">import torch

device = torch.device("cuda:0")

module_a = ModuleA()
module_b = ModuleB()
module_a.to(device)
module_b.to(device)

all_params = chain(module_a.parameters(), module_b.parameters())
optimizer = torch.optim.Adam(all_params)

critereon = torch.nn.CrossEntropyLoss(ignore_index=UNCERTAIN)

net_input = get_input_tensor()
label = get_label()

for i in range(100000):
    optimizer.zero_grad()

    out_a = module_a.forward(net_input)
    out_b = module_b.forward(backbone_out)
    loss = critereon(out_b, label.to(device))
    loss.backward()

    if i % 10 == 0:
        print("loss", loss.item())

    optimizer.step()
</code></pre>
<p>I found that the loss goes down initially, but then gets stuck at a very high value. My hypothesis is that the backprop is not going through to <code>module_a</code>'s parameters. When I hook the loss up directly to <code>out_a</code>, the loss decreases to a value very close to zero.</p>
<p>Is this use case supported by pytorch?</p>

username: markl
cooked: <p>It seems I have made a mistake in my implementation of module_b and now it seems like it’s working.</p>
Conversion to .bfloat16() makes require_grad False

username: rohithkrn
cooked: <pre><code class="lang-auto">&gt;&gt;&gt; x = torch.randn(2,2, requires_grad=True)
&gt;&gt;&gt; x.requires_grad
True
&gt;&gt;&gt; y = x.half()
&gt;&gt;&gt; y.requires_grad
True
&gt;&gt;&gt; z = x.bfloat16()
&gt;&gt;&gt; z.requires_grad
False
</code></pre>
<p>Could someone help with fixing this?</p>

username: albanD
cooked: <p>It does look like a bug, thanks for reporting this.<br>
You can follow the progress in that issue: <a href="https://github.com/pytorch/pytorch/issues/28548" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/28548</a></p>
Training a torch::jit::script::Module

username: markl
cooked: <p>There is an example for training a <code>torch::nn::Module</code> <a href="https://pytorch.org/cppdocs/frontend.html#end-to-end-example" rel="nofollow noopener">https://pytorch.org/cppdocs/frontend.html#end-to-end-example</a> but when I try to use a <code>torch::jit::script::Module</code>, I cannot call <code>.parameters()</code> like in the example.</p>
<p>Is there a way to train a  <code>torch::jit::script::Module</code> in C++?</p>

username: driazati
cooked: <p>Weirdly unlike in Python, <code>jit::script::Module</code> doesn’t have the same API as <code>nn::Module</code>.</p>
<p>You should be able to access a <code>script::Module</code>'s parameters via something like:</p>
<pre><code class="lang-cpp">for (auto param_slot : my_module.get_parameters()) {
    auto my_tensor = param_slot.value().toTensor();
}
</code></pre>

username: markl
cooked: <p>The <code>nn::Module::parameters()</code> returns a thing called a <code>slot_list_impl&lt;&gt;</code>.</p>
<p>Do you have advice on how I can make one of these from the tensors returned by the <code>script::Module::get_parameters()</code>?</p>

username: driazati
cooked: <p><code>slot_list_impl&lt;NameValue&gt;</code> is an iterator over the one of a <code>script::Module</code>'s internal lists of Parameters/Attributes. Can you elaborate on your use case for constructing one yourself? You should be able to mess with a <code>script::Module</code> by using <code>script::Module::register_parameter</code> to change any parameter value.</p>

username: markl
cooked: <p>I was trying to adapt the training loop to work for <code>torch::jit::script::Module</code> so I needed the same interface that the optimizer expects. However, I realized that it accepts <code>std::vector&lt;at::Tensor&gt;</code>, so I actually don’t have to create <code>slot_list_impl</code> after all.</p>
<p>Maybe you can take a look at what I was trying to do here <a href="https://discuss.pytorch.org/t/jit-module-parameters-are-not-updating-when-training/58945/8" class="inline-onebox">jit::script::Module parameters are not updating when training</a></p>
<p>I think for the code to actually work, I need to use <code>register_parameter</code> or <code>set_parameter</code> to push the updated tensor back into the <code>script::Module</code> after the optimizer has acted on it. Does that sound right? Or could there be a way that the optimizer can update the tensor values in place?</p>
Custom loss function(Noise Reduction Loss)

username: Aman_Oswal
cooked: <p>I am trying to write a custom loss function(<strong>Noise Reduction Loss</strong>) in PyTorch. It is very similar to cross-entropy loss with the difference that it puts some confidence in the answer it predicts(highest probability in the predicted matrix), assuming some labels are incorrect in training data. Here pred represents the predicted [m * L] matrix where m is the number of examples and L is the number of labels, y_true is [m*1]matrix of actual labels and “ro” is a hyperparameter deciding the impact of the each of the two criterions used.</p>
<p>def lossNR(pred, y_true, ro):<br>
outputs = torch.log(pred)   # compute the log of softmax values<br>
out1 = outputs.gather(1, y_true.view([-1,1])) # pick the values corresponding to the labels<br>
l1 = -((ro)* torch.mean(out1))<br>
l2 = -(1-ro) * torch.mean((torch.max(outputs,1)[0]))<br>
print(“l1=”, l1)<br>
print("l2 = ", l2)<br>
return (l1+l2)</p>
<p>I have tried the loss function on various datasets but it does not work any good. Please provide suggestions.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Is that expected that both l1 and l2 will have the same sign?<br>
Also why were you expecting this to perform well?</p>

username: Aman_Oswal
cooked: <p>l1 and l2 both have logarithm of a value less then 1 and some other constants. Hence they will have the same sign.<br>
It should work as it is almost same as Categorical CrossEntropy and concept is taken from a recent CVPR research paper. I think there is some flaw in implementation.<br>
If I put ro=1, then it will become Categorical CrossEntropy. Am I correct?</p>

username: albanD
cooked: <p>Cross entropy already contains the log softmax <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="nofollow noopener">doc</a> so it won’t be the same.<br>
But what you compute will be close the NLLLoss with mean reduction from what I can see yes.</p>
Pytorch IR with Line Numbers

username: esqu1
cooked: <p>Hi,</p>
<p>I was wondering how to generate the PyTorch IR for a Torch Script module with line numbers, like the one shown in the example code:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/docs/source/jit.rst#interpreting-graphs" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/docs/source/jit.rst#interpreting-graphs" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/docs/source/jit.rst#interpreting-graphs</a></h4>
<pre><code class="lang-rst">TorchScript
===========

.. toctree::
   :maxdepth: 1
   :caption: Builtin Functions
   :hidden:

   torch.jit.supported_ops &lt;jit_builtin_functions&gt;

.. contents:: :local:

.. automodule:: torch.jit
.. currentmodule:: torch.jit

TorchScript is a way to create serializable and optimizable models from PyTorch code.
Any TorchScript program can be saved from a Python
process and loaded in a process where there is no Python dependency.

We provide tools to incrementally transition a model from a pure Python program
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/docs/source/jit.rst#interpreting-graphs" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Currently, <code>foo.graph</code> does not get you the IR with line numbers, so I was wondering if it was possible to get the IR with the line numbers included.</p>
<p>Thank you!</p>

username: driazati
cooked: <p>Are you on the latest version of PyTorch (v1.3.0)? You can check with <code>print(torch.__version__)</code>. This feature was recently added so if you’re on an older version (e.g. v1.0) it might not show up.</p>
<p>For a simple example like:</p>
<pre><code class="lang-python">@torch.jit.script
def x(e):
    return e + 10

print(x.graph)
</code></pre>
<p>You should get something like</p>
<pre><code class="lang-auto">graph(%e.1 : Tensor):
  %3 : int = prim::Constant[value=1]()
  %2 : int = prim::Constant[value=10]() # ../test.py:33:15
  %4 : Tensor = aten::add(%e.1, %2, %3) # ../test.py:33:11
  return (%4)
</code></pre>
How to get the device of a torch::jit::script::Module?

username: markl
cooked: <p>I want to know if <code>to(at::kCUDA)</code> has been called on the module</p>

username: driazati
cooked: <p><code>script::Module</code>s don’t track their device since the device only affects the Tensor parameters stored in the module. The only way to check the device would be to check one of the Tensor parameters on the module and view its device.</p>
<p>We definitely need to have a more accessible documentation page, but you can see the definition of <code>script::Module</code> <a href="https://github.com/pytorch/pytorch/blob/0aa694ebe5615c1de01c8075a5084bc30c79e04d/torch/csrc/jit/script/module.h" rel="nofollow noopener">here</a> for a complete list of its methods.</p>
Compute Squared distance b/w two tensors inside a model in PyTorch

username: Rafael_R
cooked: <p>Compute Squared distance b/w two tensors inside a model in PyTorch :</p>
<blockquote>
<p>D = | P1− P2 | ^ 2</p>
</blockquote>
<p>options:</p>
<pre><code>    torch.norm(p1 - p2, dim=0)
    (p1 - p2).pow(2).sum(1)
    torch.dist(p1, p2, 2)
</code></pre>
<p>which one is correct for just the squared distance between two tensors?</p>

username: albanD
cooked: <p>Hi,</p>
<ul>
<li>The first one computes the 2 norm, so you should square it if you want a squared norm.</li>
<li>The second one computes what you want but also sums over dimension 1.</li>
<li>The last one also computes what you want but does not do any reduction.</li>
</ul>
Loss backward with distributed setting

username: sherdencooper
cooked: <p>I am using torch.distributed to run federated learning. One worker trains an extra model besides the distributed model. However, I find that when the worker trains his own model, the gradient is averaged by other workers’ gradients for the main task. Can the loss.backward() for the extra model be not influenced by gradient average? Thx a lot in advance.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m afraid it cannot be as <code>.backward()</code> accumulates in the <code>.grad</code> field all the time.<br>
You can avoid this though by computing gradients as:<br>
<code>grad = torch.autograd.grad(loss, model.parameters())</code>.</p>
Strange behavior for DataLoader with random numbers

username: Yuval_Reina
cooked: <p>I was trying to use some randomness in a dataset I built, and got a very strange behavior.</p>
<p>For simplicity I reproduced this behavior in the following toy example:</p>
<pre><code class="lang-auto">import torch
import torch.utils.data as D

class TestDataset(D.Dataset):

    def __init__(self, mysize=20):

        super(TestDataset, self).__init__()
        self.mysize = mysize

    def __len__(self):
        return self.mysize

    def __getitem__(self, idx):
        
        return torch.tensor(np.random.randint(0,10,size=4))

td=TestDataset(16)
dl=D.DataLoader(td,batch_size=2,num_workers=4)

for d in dl:
    print (d)
</code></pre>
<p>The results I get are</p>
<pre><code class="lang-auto">tensor([[6, 2, 0, 6],
        [8, 4, 5, 3]])
tensor([[6, 2, 0, 6],
        [8, 4, 5, 3]])
tensor([[6, 2, 0, 6],
        [8, 4, 5, 3]])
tensor([[6, 2, 0, 6],
        [8, 4, 5, 3]])
tensor([[7, 4, 7, 0],
        [5, 6, 0, 6]])
tensor([[7, 4, 7, 0],
        [5, 6, 0, 6]])
tensor([[7, 4, 7, 0],
        [5, 6, 0, 6]])
tensor([[7, 4, 7, 0],
        [5, 6, 0, 6]])
</code></pre>
<p>Instead of random output, the output repeat itself according to the number of workers.</p>
<p>A Bug? A Feature?</p>

username: ptrblck
cooked: <p>Could reproduce this in <code>1.3.0</code> and it seems all workers use the same seed for each spawn.<br>
<a class="mention" href="/u/alband">@albanD</a> is this expected  as I cannot reproduce it in <code>1.3.0.dev20190919</code>?</p>

username: albanD
cooked: <p>I does ring a bell to have an PR fixing this.</p>

username: albanD
cooked: <p>Here is the <a href="https://pytorch.org/docs/stable/data.html#randomness-in-multi-process-data-loading" rel="nofollow noopener">doc</a> section that corresponds to it.</p>

username: Yuval_Reina
cooked: <p>Thanks for the blazing fast response!</p>
<p>I understand it’s a feature and I need to use Pytorch’s randoms to get real randomness</p>
NLL training using a weighted sum of softmax-normalized predictions

username: stangelid
cooked: <p>I have developed a classification model, where the final class prediction is an attention-weighted sum of multiple softmax-normalized probability distributions.</p>
<p>This is an instantiation of <em>Multiple Instance Learning</em> (MIL) – a classic use-case is computer vision: You make separate predictions about many individual patches in the image, but only have training labels for the image as a whole. The prediction for the whole image is a sum of the individual ones, weighted by attention.</p>
<p>Here is a stripped-down example with 5 classes, where the final prediction is a weighted sum of 3 individual predictions (I use a batch size of 1 for simplicity):</p>
<pre><code class="lang-python"># individual predictions obtained via softmax 
# [batch_size x num_patches x num_classes]
individual_preds = torch.tensor([[[0.10, 0.20, 0.20, 0.25, 0.25],
                                  [0.15, 0.15, 0.20, 0.20, 0.30],
                                  [0.25, 0.25, 0.20, 0.30, 0.00]]],
                                requires_grad=True)

# weights of individual predictions [batch_size x num_patches x 1]
attn = torch.tensor([[[0.25],
                      [0.30],
                      [0.45]]])

# final prediction as a weighted sum
# of individual ones [batch_size x num_classes]
pred = torch.sum(attn * individual_preds, dim=1)

# 'manually' taking the log probabilities for NLL
log_pred = pred.log()

# pseudo-target for loss [batch_size]
target = torch.tensor([1])

# NLL loss
crit = nn.NLLLoss()
loss = crit(pred, target)
loss.backward()
</code></pre>
<p>I know that it is recommended that classification tasks use a combination of LogSoftMax and NLLLoss (or simply CrossEntropyLoss), to avoid numerical instabilities.</p>
<p>In this case, I cannot think of a way to avoid computing softmax-normalized probabilities and then <em>‘manually’</em> taking the log, before pushing through NLLLoss. The problem is that a weighted sum of predictions is required, and this only makes sense in probability space – if individual predictions were already in log space, it is impossible to combine them (the log of a sum doesn’t decompose to anything usable).</p>
<p>Should I worry about “numerical instability” because I’m using SoftMax+Log+NLLLoss separately? Is there a way to achieve what I want that I am currently missing?</p>
<p>Thanks!</p>
Why no eval() and train() mode switch in the DQN tutorial?

username: Finspire13
cooked: <p>Hi, it really confuses me that why the <a href="http://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" rel="nofollow noopener">DQN tutorial</a> has no model.eval() and model.train(), since there are several BatchNorm layers in the model.</p>
<p>As I understand, model.eval() should be put before calling select_action(). And in function optimize_model(), model.train() should be put before calling the first model.forward() while model.eval() should be put before calling the second one. Is it correct?</p>

username: pedroveloso13
cooked: <p>I am also confused and I was going to post a similar question.</p>
<p>My guess is that you should run everything in train mode or only the action selection should be with model.eval(). I think training might become unstable if you use results from one model in eval and another one in train mode.</p>
Should the aspect ratio of test set same as train set?

username: Arta_A
cooked: <p>Hi, I trained my UNET model with images with size 320*320, I want to test(predict for new dataset) my model using a dataset with different image sizes, I heard that is possible but I got an error for mismatching the dimension sizes.<br>
<strong>I have used model.eval() mode.</strong><br>
error:</p>
<pre><code class="lang-auto">torch.Size([1, 3, 813, 1024])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
 in 
     21         img.cuda()
     22         print(img.shape)
---&gt; 23         out = model(img)
     24         temp = torch.Tensor.cpu(x).detach().np()
     25         print(type(temp), temp.shape)

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

 in forward(self, inputs)
     48 
     49         center = self.center(maxpool4)       # 256*32*32
---&gt; 50         up4 = self.up_concat4(center,conv4)  # 128*64*64
     51         up3 = self.up_concat3(up4,conv3)     # 64*128*128
     52         up2 = self.up_concat2(up3,conv2)     # 32*256*256

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

 in forward(self, high_feature, *low_feature)
     53         outputs0 = self.up(high_feature)
     54         for feature in low_feature:
---&gt; 55             outputs0 = torch.cat([outputs0, feature], 1)
     56         return self.conv(outputs0)

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 100 and 101 in dimension 2 at ..\aten\src\TH/generic/THTensor.cpp:711
</code></pre>
<p>Please help, Thanks</p>

username: ptrblck
cooked: <p>Apparently your code has some restrictions on the input shapes, as the height of your current input is 100 and 101 for <code>center</code> and <code>conv4</code>.</p>
<p>Usually it is easier to deal with shapes of powers of two instead of completely random input shapes.<br>
If you really want to use all possible input shapes, you could add some shape checks manually and use cropping or padding to create matching shapes for your concat layers.</p>

username: Arta_A
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> What do you mean by deal with shapes of powers of two? can you explain please?<br>
Thanks for your response.<br>
Note that my work is image resizing so maintaining the aspect ratio is critical.</p>
How to implement this code without loop

username: dali_wang
cooked: <pre><code class="lang-auto">#batch is a tensor with shape HW

  for i in range(H):
      batch[i,:]=i*batch[i,:]
</code></pre>
<p>T T please</p>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">batch = torch.ones(10, 4)
batch = torch.arange(batch.size(0)).view(-1, 1) * batch
</code></pre>
<p>If you are dealing with more dimensions, you might need to change the <code>view</code> call.</p>
Why my training loss does not decrease overtime

username: Yifei
cooked: <p>I got a very weird problem here. I use a very simple deep learning model with only linear layers. However, the training loss does not decrease at all during training.</p>
<p>But when I input some random inputs to my model, it works well… So I don’t think it is the model problem. Must be something wrong with my inputs. I load features from ‘.npy’ file.</p>
<p>Any suggestions would be greatly appreciated.<br>
Thank you</p>

username: Tahir
cooked: <p>There could be so many reasons why loss may not decrease overtime, varying from network design to input. Without looking at the data and network it is difficult to say what might be the problem.</p>

username: mailcorahul
cooked: <p>Please post the dataloading and training code here.</p>

username: Yifei
cooked: <p>Hi, I figured out myself. The reason is that my input features are not in the same range. I normalized them and it works now. Thank you.!</p>
Segmentation Fault, Pytorch on Raspberry pi 3

username: Burning
cooked: <p>Hey, guys. I installed pytorch recently on Raspberry pi 3 for running SSD(single shot multibox detector) there. Here are some potential reasons (I guess) may cause the error in the title:</p>
<ol>
<li>
<p>By installation, I chose the branch <strong>0.4.0</strong>, but installed the version <strong>0.5.0a0+fa5d95a</strong>.</p>
</li>
<li>
<p>I trained the model with pytorch 0.4.0 and python 2.7, but on raspberry the python version is 3.5.</p>
</li>
</ol>
<p>Sometimes, for the first several images, it works fine but eventually the error, <strong>segmentation fault</strong>, arises.</p>
<p>Does anyone have any opinion about this? Appreciated in advance.</p>

username: Burning
cooked: <p>I found the solution to this myself.</p>
<p>Do not use torch.nn.Sequential, instead use a for loop or something else to do it.</p>

username: Ashutosh_Mishra
cooked: <p>Hi,</p>
<p>I also installed Pytorch 0.4.0 and I am running basic MNIST example from Pytorch’s github repo, but still facing an issue related to seg fault.</p>

username: Burning
cooked: <p>Hi, you could try increase your swap space to 4 GB. That helped me in the end and no more segment fault.</p>

username: elepherai
cooked: <p>4GB swap space does’t work.</p>
BatchNorm with requires_grad=False and training=True in Validation

username: Ir1d
cooked: <p>Hi,<br>
In validation process we set <code>model.eval()</code> to tell BN and Dropouts that we are in val mode.<br>
But what’s the difference if we use train mode?<br>
In the <a href="https://pytorch.org/docs/master/nn.html#torch.nn.BatchNorm2d" rel="nofollow noopener">docs</a> it said:</p>
<pre><code class="lang-auto">During training, this layer keeps a running estimate of its computed mean and variance. The running sum is kept with a default momentum of 0.1.

During evaluation, this running mean/variance is used for normalization.
</code></pre>
<p>what if I set <code>requires_grad=False</code> for the whole model, which means that the running mean/var will not be modified.<br>
In that case, is training mode same as validation mode?<br>
Thanks in advance.</p>
<p>upd: This question came to me when using vgg16 as perceptual loss. Do we need to set vgg16 model to eval mode?</p>

username: ptrblck
cooked: <p><code>requires_grad</code> does not change the train/eval mode, but will avoid calculating the gradients for the affine parameters (<code>weight</code> and <code>bias</code>).<br>
<code>bn.train()</code> and <code>bn.eval()</code> will change the usage of the running stats (<code>running_mean</code> and <code>running_var</code>).</p>

username: Ir1d
cooked: <p>the running mean/var are parameters, right?<br>
If <code>requires_grad</code> is set to False, the params shouldn’t be changed. Does that mean when  <code>requires_grad=False</code>, the running stats will not be changed, so <code>.train()</code> and <code>.eval()</code> are the same?</p>

username: ptrblck
cooked: <p>No, <code>running_mean</code> and <code>running_var</code> are buffers and thus do not require gradients, but will be updated (in <code>train</code> mode) using the current batch statistics and the <code>momentum</code>.</p>
Memory_cached and memory_allocated does not nvidia-smi result

username: laoreja
cooked: <p>Hi, I tried to understand memory_cached and memory_allocated, in the doc, it is said that the unit is bytes.<br>
I tried to match the results of <code>torch.cuda.memory_cached()</code> and <code>torch.cuda.memory_allocated()</code> with the results of nvidia-smi, but the matching failed, in both ipython and in my evaluation code for evaluating a model.</p>
<p>In Ipython3:</p>
<pre><code class="lang-auto">In [1]: a = torch.zeros(8192, 4, device='cuda')

In [2]: c = torch.ones(123456, 2, 3, device='cuda')

In [3]: torch.cuda.max_memory_cached(0)
Out[3]: 4063232

In [4]: torch.cuda.max_memory_allocated(0)
Out[4]: 4011008

In [5]: torch.cuda.memory_allocated(0)
Out[5]: 4011008

In [6]: torch.cuda.memory_cached(0)
Out[6]: 4063232

In [7]: (8192 * 4 + 123456 * 2 * 3) * 4
Out[7]: 3094016
</code></pre>
<p>while the number shown in nvidia-smi is always 1053 MB.</p>
<p>What causes the difference?<br>
What’s the difference between max_memory_cached/allocated and memory_cached/allocated?<br>
If I would like to measure the GPU memory usage by a model during evaluation, where should I put the evaluation code, after getting the output?</p>

username: SimonW
cooked: <p>Your tensor is too small. The caching allocator allocates memory with a minimum size and a block size so it may allocate a bit more memory than the number of elements in your tensors.</p>
<p>Things like cuda ctx, cuda rng state, cudnn ctx, cufft plans, and other gpu memory your other libraries may use are not counted in the <code>torch.cuda.*</code> stats.</p>

username: colesbury
cooked: <p>The <code>nvidia-smi</code> number includes space the allocated by the CUDA driver when it loads PyTorch. This can be quite large because PyTorch includes many CUDA kernels. On a P100, I’ve seen an overhead of ~487 MB. On M40, I’ve seen ~303 MB.</p>
<p>This memory isn’t reported by the torch.cuda.xxx_memory functions because it’s not allocated by the program and there isn’t a good way to measure it outside of looking at nvidia-smi.</p>

username: Debasish_Jyotishi
cooked: <p>Hi, I tried the following commands;</p>
<p>torch.cuda.max_memory_allocated(device=0)</p>
<p>torch.cuda.max_memory_cached(0)</p>
<p>In both cases, 0 is returned. Can anyone explain what does that mean?</p>
<p>Previously when I ran my code the following error came.</p>
<p>RuntimeError: CUDA error: out of memory</p>
<p>If I run,<br>
torch.cuda.get_device_properties(device).total_memory</p>
<p>I get output:34078654464</p>

username: ptrblck
cooked: <p>Are you printing these stats right before the OOM error is raised and are you sure you’ve already allocated some tensors on the specified device?</p>
C extension for radon transform

username: AlbertZhang
cooked: <p>Hi,<br>
I am trying to implement a C extension of radon transform for pytorch <a href="https://github.com/AlbertZhangHIT/pytorch_radon/tree/master/radon_transform" rel="nofollow noopener">source code</a>. However, when I try to build it, a link error pops</p>
<pre><code class="lang-auto">_th_radon.obj : warning LNK4197: export 'PyInit__th_radon' specified multiple times; using first specification
   Creating library .\Release\_th_radon.lib and object .\Release\_th_radon.exp
_th_radon.obj : error LNK2001: unresolved external symbol cpu_radon
th_radon.obj : error LNK2001: unresolved external symbol __imp_THFloatTensor_data
th_radon.obj : error LNK2001: unresolved external symbol __imp_THFloatTensor_size
.\_th_radon.pyd : fatal error LNK1120: 3 unresolved externals
...
cffi.VerificationError: LinkError: command 'D:\\Programs\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\link.exe' failed with exit status 1120
</code></pre>
<p>I have no idea what happened. The evironment of my PC is</p>
<pre><code class="lang-auto">Windows 10, 
Visual Studio 2015,
PyTorch 0.4.1
Python 3.5.4
</code></pre>
<p>Any suggestion is appreciated.<br>
It is said that the ATen Library in PyTorch 1.0 is better <a href="https://discuss.pytorch.org/t/converting-torch-utils-ffi-create-extension-to-cppextension/26504">see here</a>, however I’m not clear how to convert the code using <code>cffi</code> to the <code>CppExtension</code>.</p>
<p>Thanks for your help.</p>

username: AlbertZhang
cooked: <p>Hi, thanks for your attention. I have made it by referring to an excellent example <a href="https://github.com/facebookresearch/maskrcnn-benchmark" rel="nofollow noopener">mask_rcnn_benchmark</a>.<br>
Thanks again.</p>

username: JiangYi0311
cooked: <p>could you please tell me how to solve this problem?</p>
Function of model.eval()

username: nik1996
cooked: <p>What is the default mode (training or inference) for a Pytorch model if model.eval() or model.train() is not executed? I was recently working on a pet project and was loading a trained model using torch.load(). I was only using the loaded model for inference. Now if I don’t execute model.eval(), it’s behaving as if the weights are untrained and giving a very bad performance. But just setting the eval flag with model.eval() fixes the issue. What might be causing this behavior? Isn’t a model supposed to be in eval mode by default.</p>

username: alex.veuthey
cooked: <p>They are initialized with train=True by default.</p>
<p>However, the difference only lies on a few things: batch nom layers and dropout layers. It has nothing to do with the weights being trained or not… So if you have batch norm or dropout layers, setting the eval flag will modify them accordingly, so that you can have correct values in the output! The weights of the model are unchanged, though.</p>
Running libtorch with g++-9

username: jose
cooked: <p>Hello everyone. The solution to this issue might be trivial; however, I haven’t been able to find it, and I have tried several things already. I am constrained to using g++@&gt;=5.</p>
<p>I implemented <a href="https://github.com/pytorch/examples/blob/master/cpp/dcgan/dcgan.cpp" rel="nofollow noopener">this example</a> some time ago, and was able to run it with no problems.</p>
<p>Now, due to issues when compiling libtorch with g++-9, I have come back to it, to get it working with this compiler.</p>
<p>My CMakeLists.txt looks as follows:</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(LibtorchExample)

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ")
#add_compile_definitions(D_GLIBCXX_USE_CXX11_ABI=1)

set(Torch_DIR /Users/ananias/Developer/libtorch/share/cmake/Torch)
find_package(Torch REQUIRED)
set(CMAKE_CXX_STANDARD 11)

add_executable(LibtorchExample main.cpp discriminator.h generator.h)

target_link_libraries(LibtorchExample "${TORCH_LIBRARIES}")
set_property(TARGET LibtorchExample PROPERTY CXX_STANDARD 11)
</code></pre>
<p>I am getting the following kinds of errors:</p>
<pre><code class="lang-auto">/usr/local/Cellar/gcc/9.2.0_1/include/c++/9.2.0/x86_64-apple-darwin18/bits/c++config.h:273:27: error: #if with no expression
  273 | #if _GLIBCXX_USE_CXX11_ABI

/usr/local/Cellar/gcc/9.2.0_1/include/c++/9.2.0/x86_64-apple-darwin18/bits/c++config.h:554:3: error: operator '&amp;&amp;' has no right operand
  554 |   &amp;&amp; _GLIBCXX_USE_DUAL_ABI &amp;&amp; __cpp_transactional_memory &gt;= 201500L \

/usr/local/Cellar/gcc/9.2.0_1/include/c++/9.2.0/string:58:53: error: operator '&amp;&amp;' has no right operand
   58 | #if __cplusplus &gt;= 201703L &amp;&amp; _GLIBCXX_USE_CXX11_ABI

/usr/local/Cellar/gcc/9.2.0_1/include/c++/9.2.0/stdexcept:236:28: error: operator '||' has no left operand
  236 | #if _GLIBCXX_USE_CXX11_ABI || _GLIBCXX_DEFINE_STDEXCEPT_COPY_OPS
</code></pre>
<p>I have tried a couple of things, including installing the latest libtorch version, according to <a href="https://github.com/pytorch/pytorch/issues/17492" rel="nofollow noopener">this issue</a>, to no avail.</p>
<p>I have tried setting the ABI flags in one way or another, but my attempts haven’t had any effect even on the type of error I am getting.</p>
<p>Any recommendations are very welcome.</p>
Is autograd.grad producing a wrong grad for l1 and l2(linear layers in example)? (sorry, edited post, I pasted wrong initially)

username: MLSlayer
cooked: <p>Here is the code that I am running and the corresponding output</p>
<pre><code class="lang-auto">l1 = t.nn.Linear(1, 1, bias=False)
l1.weight.data[:] = 2
l2 = t.nn.Linear(1, 1, bias=False)
l2.weight.data[:] = 3

inputs = t.tensor([5.], requires_grad=True)
a = l2(l1(inputs))
g = t.autograd.grad(outputs=a, inputs=inputs, create_graph=True)[0]
print('inputs.grad is ', g)
gp = g * 3
gp.backward()
print('l1_grad is now ', l1.weight.grad)
print('l2_grad is now ', l2.weight.grad)


</code></pre>
<pre><code class="lang-auto">
inputs.grad is  tensor([6.], grad_fn=&lt;SqueezeBackward1&gt;)
l1_grad is now  tensor([[9.]])
l2_grad is now  tensor([[6.]])
</code></pre>
<p>Now, to my understanding, the gradients should actually be<br>
<code>45 for l1_grad</code><br>
<code>30 for l2_grad</code></p>
<p>however, it seems that the inputs is not taken into account when computing the gradients for l1 and l2?</p>

username: albanD
cooked: <p>Hi,</p>
<p>So what you do here is the following.<br>
I use l1 for l1.weight and l2 for l2.weight for readability.</p>
<p><code>a = l1*(l2*inputs)</code><br>
grad wrt a:<br>
<code>inputs.grad = 1 * l1 * l2</code><br>
<code>gp = 3 * inputs.grad = 3 * l1 * l2</code></p>
<p>grad wrt gp now<br>
<code>dgp/dl1 = 3*l2 = 9</code><br>
<code>dgp/dl2 = 3*l1 = 6</code></p>
<p>So I would agree with the autograd result.<br>
Could you explain what is your logic to get 45 and 30?</p>

username: MLSlayer
cooked: <p>Ah, sorry. Your line</p>
<blockquote>
<p>gp = 3 * inputs.grad = 3 * l1 * l2</p>
</blockquote>
<p>made me realize my mistake. When taking the gradient of gp wrt l1, I was thinking that the <code>inputs</code> was still attached to the graph so something like<br>
<code>gp = 3 * l1 * l1 * inputs</code> which is wrong</p>
<p>Thank you!</p>