akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use BCE loss and CrossEntropyLoss correctly?

username: SU801T
cooked: <p>Hi,</p>
<p>I have defined a pretrained resnet50 for data parallelism using multiple classes and use nn.CrossEntropyLoss() .</p>
<pre><code class="lang-auto">model = models.resnet50(pretrained=True)
model = torch.nn.DataParallel(model)
for p in model.parameters():
      p.requires_grad = False
num_ftrs = model.module.fc.in_features
model.module.fc = nn.Linear(num_ftrs, num_classes)
model = model.to(device)
</code></pre>
<p>However, I’m unsure of how to use BCE Loss. I have read that it is better to use nn.BCELoss() for two classes, however, I don’t know if I need to define a sigmoid layer. And if so, where to put this layer…</p>
<p>Cheers,<br>
T</p>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/docs/master/generated/torch.nn.BCELoss.html">docs</a> will give you some information about these loss functions as well as small code snippets.</p>
<p>For a binary classification, you could either use <code>nn.BCE(WithLogits)Loss</code> and a single output unit or <code>nn.CrossEntropyLoss</code> and two outputs.<br>
Usually <code>nn.CrossEntropyLoss</code> is used for a multi-class classification, but you could treat the binary classification use case as a (multi) 2-class classification, but it’s up to you which approach you would like to use.</p>
<p>If you are using the former approach, we generally recommend to use <code>nn.BCEWithLogitsLoss</code> and pass raw logits to this criterion, as it will yield better numerical stability than <code>sigmoid</code> + <code>nn.BCELoss</code>.</p>
<p>The latter use case also expects raw logits, which can be passed to <code>nn.CrossEntropyLoss</code>.</p>

username: SU801T
cooked: <p>Ah ok cool.</p>
<p>So i’m under the assumption that softmax is automatically computed in the <code>nn.CrossEntropyLoss</code> module and <code>nn.BCE(WithLogits)Loss</code> will also have sigmoid computed within, whereas for <code>nn.BCELoss</code> you need to apply sigmoid first? And if so, I can send the output of <code>sigmoid</code> + <code>nn.BCELoss</code> to <code>nn.CrossEntropyLoss</code>?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="SU801T" data-post="3" data-topic="89049">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/48.png" class="avatar"> SU801T:</div>
<blockquote>
<p>And if so, I can send the output of <code>sigmoid</code> + <code>nn.BCELoss</code> to <code>nn.CrossEntropyLoss</code> ?</p>
</blockquote>
</aside>
<p>No, this wouldn’t work, since <code>nn.BCELoss</code> already calculates the loss.<br>
<code>nn.CrossEntropyLoss</code> expects a model output and targets, not another loss.</p>

username: SU801T
cooked: <p>Apologies, I just understood what you meant!. Many thanks for your help.</p>
Multi Class ROC

username: Deb_Prakash_Chatterj
cooked: <p>Hey, I am making a multi-class classifier with 4 classes. Now I have printed Sensitivity and Specificity along with a confusion matrix. Now I want to print the ROC plot of 4 class in the curve. As ROC is binary metric, so it is ‘given class vs rest’, but I want to add all 4 classes in the same plot.<br>
With this code, I have got my probability  -</p>
<pre><code class="lang-auto">output = model.forward(images)
p = torch.nn.functional.softmax(output, dim=1)
</code></pre>
<p>Can you guys please help me to get this Multi class ROC graph? Thanks.</p>

username: ptrblck
cooked: <p>If you already have the metrics (TP, FP, TN, FN), you could probably just use scikit-learn’s methods as explained <a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html" rel="nofollow noopener">here</a>.</p>

username: Deb_Prakash_Chatterj
cooked: <p>Well, I have done as it is saying, but in this -</p>
<pre><code class="lang-auto">```#Compute ROC curve and ROC area for each class 
fpr = dict() 
tpr = dict() 
roc_auc = dict() 
for i in range(n_classes):
 fpr[i], tpr[i], _ = roc_curve(y_test[:, i], y_score[:, i]) 
 roc_auc[i] = [auc](fpr[i], tpr[i]) 

#Compute micro-average ROC curve and ROC area 
fpr["micro"], tpr["micro"], _ = roc_curve(y_test.ravel(), y_score.ravel()) 
roc_auc["micro"] = auc(fpr["micro"], tpr["micro"])```
</code></pre>
<p>Do I need to change my fpr and tpr with my FP and TP in -<br>
TP = conf_matrix.diag() and FP = conf_matrix[idx, c].sum()?</p>
<p>Also, I have changed n_class to nb_class = 4<br>
How can I represent this -</p>
<pre><code class="lang-auto">for i in range(nb_classes):
    fpr[i], tpr[i], _ = roc_curve(y_test[:, i], y_score[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])
</code></pre>
<p>I do not have y_test, but I have dataloader_test, and what is y_score?</p>

username: qaixerabbas
cooked: <p>fpr and tpr are False Positive Rate and True Positive Rate respectively while your metrics are different FP and TP. You will get FPR and TPR from roc_curve() function.</p>
<p>y_test refers to the True Predictions i.e Ground Truth(y_true) and y_score are predictions generated by your model (y_pred).</p>

username: saba
cooked: <p>Hi Ptrblck,</p>
<p>Can we have ROC curve for the target with float numbers?</p>
<p>I try these codes when the Target is 0.9 instead of 1. but the error was that it is not supported for the continuous format</p>
<pre><code class="lang-auto">    AUC.append(roc_auc_score(TargetWholev3.reshape(-1,1),predictedwhole_v3.reshape(-1,1)),pos_label=max(TargetWholev3))

    fpr1, tpr1, thresholds =  metrics.roc_curve(TargetWholev3.reshape(-1,1),predictedwhole_v3.reshape(-1,1), pos_label=max(TargetWholev3))
</code></pre>

username: ptrblck
cooked: <p><code>metrics.roc_curve</code> seems to expect integer labels, so you would have to undo the label smoothing.</p>
How to check if irecv got a message?

username: maris_ancans
cooked: <p>say I have:</p>
<pre><code class="lang-auto">t = tensor.torch(0)
req = dist.irecv(t, rank)
</code></pre>
<p>how can i check if i have messages from some rank (stored in buffer or something). From c++ examples ive seen there are probe commands:</p>
<pre><code class="lang-auto">MPI_Probe(0, 0, MPI_COMM_WORLD, &amp;status);
</code></pre>
<p>to check message length.<br>
One possible solution would be to check if tensor != 0 but that doesnt seem to be the right way to do it</p>

username: Yong-gui
cooked: <p>Do you get a better solution?</p>

username: mrshenli
cooked: <p>We probably should set the <code>complete_</code> flag in the send/recv work object and expose that as a Python API. Similar issue exists in our <a href="https://pytorch.org/docs/master/futures.html" rel="nofollow noopener"><code>Future API</code></a>.</p>
<p><a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a> mentioned that this is also asked in <a href="https://github.com/pytorch/pytorch/issues/30723" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/30723</a></p>

username: mrshenli
cooked: <p>added an issue to track: <a href="https://github.com/pytorch/pytorch/issues/41428" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/41428</a></p>
Grad-cam working for PyTorch with custom model

username: NishantN
cooked: <p>I have been trying to use grad-cam for a custom model I made in pytorch but can’t figure out how to do it. The model I made is a classifier model using ResNet50. Any help on how I can use gradcam to create a heatmap on images with based on my model would be really appreaciated.</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p><code>Captum</code> provides a Guided <code>GradCam</code> implementation <a href="https://captum.ai/api/guided_grad_cam.html">here</a>, which might be useful. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: NishantN
cooked: <p>Thanks! I’ll have a look</p>
Faster R-CNN output adaption

username: juliwern
cooked: <p>Hi,</p>
<p>I was using the Faster R-CNN model from the torchvision library for a transfer learning task. Everything works perfectly and I get good results. But I’m wondering about the outputs of the whole model.</p>
<p>The output for inference is currently</p>
<ul>
<li>boxes [N, 4] with the bounding box coordinates</li>
<li>labels [N] with the predicted labels for each image</li>
<li>scores [N] with the scores or each prediction</li>
</ul>
<p>There is currently no interface investigate the class probabilities for the classes, that are not the most probable class prediction. This information would be pretty interesting for me to get a deeper insight for false positive predictions or wrong classifications.</p>
<p>Is there already a preferred way of doing this? If not, I would probably suggest this as a feature in Github.</p>
<p>Kind regards</p>
Pytorch loss backward error

username: lifeblack
cooked: <p>Hi there!</p>
<pre><code class="lang-auto">one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1024, 112]], which is output 0 of struct torch::autograd::CopySlices, is at version 112; expected version 111 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>How could I fix this problem?</p>

username: albanD
cooked: <p>Hi,</p>
<p>What does the extra backtrace show is the issue?<br>
It should point to the forward op whose result was modified inplace even though it needed it to compute some gradients.</p>
SSD object detection model overfitting even with very high weight decay

username: mukul54
cooked: <p>Actually I am struggling for a long time with this problem and had probably tried more than 100 experiments.</p>
<p>I am working on an object detection model for medical imaging problems using the SSD architecture with various resnet backbone. I started and experimented with a lot of hyperparameters with a resnet34 backbone. Training loss curve seems to be okay in almost all the case but validation loss starts to increase even with <code>weight_decay = 0.1</code>, which is really high value. <code>voc_map</code> on <code>train_set</code> comes out to be more than <code>90%</code> but on <code>val_set</code> the maximum which I have achieved using <code>resnet34</code> is <code>35%</code>. I see that model is heavily overfitting. I have used different augmentations that have been told to be okay by the medical experts.</p>
<p>Another weird thing about the loss curve which I observed is that, generally, validation classification loss starts to increase early and after training on more epochs localization loss also starts to increase.</p>
<p>Some other details:</p>
<p>Optimizer: Amsgrad<br>
Classification loss: Cross-Entropy with Hard Negative Mining<br>
Localization Loss: Smooth L1 loss</p>
<p>I am sorry if this is not the correct forum but since I am using PyTorch, I thought maybe someone else in the community might have faced similar issues.</p>
Tensorboard `add_graph`: visualize model partially composed from torch.distributions

username: sid
cooked: <p>I’m experimenting with a model that uses a GMM at the output, implemented using functionality from <code>torch.distributions</code>. Can this be visualized via tensorboard?</p>
<p>When calling <code>writer.add_graph</code> on the model, I get the following error:</p>
<pre><code class="lang-auto">/opt/conda/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py in add_graph(self, model, input_to_model, verbose)
    705         if hasattr(model, 'forward'):
    706             # A valid PyTorch model should have a 'forward' method
--&gt; 707             self._get_file_writer().add_graph(graph(model, input_to_model, verbose))
    708         else:
    709             # Caffe2 models do not have the 'forward' method

/opt/conda/lib/python3.7/site-packages/torch/utils/tensorboard/_pytorch_graph.py in graph(model, args, verbose)
    289             print(e)
    290             print('Error occurs, No graph saved')
--&gt; 291             raise e
    292 
    293     if verbose:

/opt/conda/lib/python3.7/site-packages/torch/utils/tensorboard/_pytorch_graph.py in graph(model, args, verbose)
    283     with torch.onnx.set_training(model, False):  # TODO: move outside of torch.onnx?
    284         try:
--&gt; 285             trace = torch.jit.trace(model, args)
    286             graph = trace.graph
    287             torch._C._jit_pass_inline(graph)

/opt/conda/lib/python3.7/site-packages/torch/jit/__init__.py in trace(func, example_inputs, optimize, check_trace, check_inputs, check_tolerance, _force_outplace, _module_class, _compilation_unit)
    873         return trace_module(func, {'forward': example_inputs}, None,
    874                             check_trace, wrap_check_inputs(check_inputs),
--&gt; 875                             check_tolerance, _force_outplace, _module_class)
    876 
    877     if (hasattr(func, '__self__') and isinstance(func.__self__, torch.nn.Module) and

/opt/conda/lib/python3.7/site-packages/torch/jit/__init__.py in trace_module(mod, inputs, optimize, check_trace, check_inputs, check_tolerance, _force_outplace, _module_class, _compilation_unit)
   1025             func = mod if method_name == "forward" else getattr(mod, method_name)
   1026             example_inputs = make_tuple(example_inputs)
-&gt; 1027             module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, _force_outplace)
   1028             check_trace_method = module._c._get_method(method_name)
   1029 

RuntimeError: Tracer cannot infer type of (MixtureSameFamily(
  Categorical(&lt;redacted&gt;),
  MultivariateNormal(&lt;redacted&gt;)))
:Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type MixtureSameFamily.
</code></pre>
<p>The last line clarifies in no uncertain terms <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> . I am curious what workarounds there are.</p>
Freezing just a few weights (instead of the whole layer)

username: numbpy
cooked: <p>Heya, I was trying to replicate the pruning experiments mentioned in the paper, <a href="https://arxiv.org/abs/1803.03635" rel="nofollow noopener">The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks</a> which is written in Tensorflow. I have managed to create the mask and apply it but unable to figure out how to freeze the pruned weights. As much as I checked, it is only possible to freeze one layer or a parameter at once, not the individual weights.</p>
<p>Tried looking at <a href="https://github.com/Eric-mingjie/rethinking-network-pruning/tree/master/cifar/lottery-ticket/weight-level" rel="nofollow noopener">Rethinking network pruning (code)</a> as it has a implementation but couldn’t find how they froze the pruned weights. In another code <a href="https://github.com/rahulvigneswaran/Lottery-Ticket-Hypothesis-in-Pytorch" rel="nofollow noopener">Lottery ticket in Pytorch</a>, they explicitly zeroed the gradients for pruned weights which seems cumbersome and extra computation for auto_grad as the gradients are still being computed. Is there a workaround?</p>
<p>Also, if there is no other way can backward hooks make the process a little bit easier as I am trying to use this with fastai package. Any help is really appreciated.</p>

username: Michela
cooked: <p>Check out the pruning functionalities added to PyTorch v1.4 under <code>torch.nn.utils.prune</code>. They were built with lottery ticket hypothesis research in mind. The masks are added as forward pre-hooks, and are therefore automatically included as an operation in the backward pass too, thus effectively freezing pruned parameters.</p>
<p>You might also want to check out the PyTorch pruning tutorial: <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/pruning_tutorial.html</a></p>
Issues with deepcopy on pruned model

username: George3d6
cooked: <p>After applying pruning to the layers of a model, by iterating over all modules and applying:</p>
<pre><code class="lang-auto">prune.l1_unstructured(layer, name='weight', amount=0.1)
prune.l1_unstructured(layer, name='bias', amount=0.1)
</code></pre>
<p>to nn.Linear modules.</p>
<p>I get the error <code>RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment</code>, when trying to <code>deepcopy</code> my<code> nn.Sequential</code> model, or rather, an object containing said <code>nn.Sequential</code> model.</p>

username: Michela
cooked: <p>Please see the discussion on this at this github issue:<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/37322" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/37322" target="_blank" rel="nofollow noopener">copy.deepcopy() breaks when pruning is set on sequential</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-26" data-time="18:46:12" data-timezone="UTC">06:46PM - 26 Apr 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-04-30" data-time="23:11:51" data-timezone="UTC">11:11PM - 30 Apr 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/Ge0rges" target="_blank" rel="nofollow noopener">
          <img alt="Ge0rges" src="https://avatars0.githubusercontent.com/u/3957136?v=4" class="onebox-avatar-inline" width="20" height="20">
          Ge0rges
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
Given a a subclass of nn.module which has nn.Sequential() parameters. If we register a pruning method on a subset of...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: nn</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to sparsify a ResNet-101

username: cyril9227
cooked: <p>Hello,</p>
<p>My question is pretty straight-forward. Let’s say I got a <strong>pruned</strong> ResNet-101 using <code>torch.nn.utils.prune</code> module. How can I sparsify the network to boost inference speed and reduce the memory footprint ?</p>
<p>The ResNet is from Detectron2 library.</p>
<p>Thank’s in advance for your advice</p>

username: Michela
cooked: <p>Hi <a class="mention" href="/u/cyril9227">@cyril9227</a>, please see my answer to a similar question and links therein:</p><aside class="quote quote-modified" data-post="2" data-topic="83429">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michela/48/18830_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/weight-pruning-on-bert/83429/2">Weight Pruning on BERT</a> 
  </div>
  <blockquote>
    The point of PyTorch pruning, at the moment, is not necessarily to guarantee inference time speedups or memory savings. It’s more of an experimental feature to enable pruning research. 
In any case, answers to questions similar to yours were given <a href="https://github.com/pytorch/tutorials/pull/605#issuecomment-585994076" rel="noopener nofollow ugc">here</a> and <a href="https://discuss.pytorch.org/t/how-to-improve-inference-time-of-pruned-model-using-torch-nn-utils-prune/78633/4">here</a>. 
TL;DR: You can save space by calling .to_sparse() which brings your sparse tensor into coordinate representation. You cannot expect any inference speedups unless you use a custom sparse matrix algebra library to power your computation. t…
  </blockquote>
</aside>
How to prune weights in a network

username: Tommy
cooked: <p>Hello everyone,<br>
I am a new Pytorch user, so I am fairly inexperienced. How can I zero out some weights of a fully connected layer in Pytorch? Effectively i want to set some weights equal to zero and avoid changing that value as the training of the network progresses. Can anyone point me to some solutions?</p>

username: SpandanMadan
cooked: <p>You basically need to access the right parameters and set them to 0. Then set their property .requires_grad = False to make sure they don’t change in value.</p>
<p>The way to access weights in PyTorch is to use .parameters() function. Any model’s children can be accessed using model.children().</p>
<pre><code>for child in model.children():
    for param in child.parameters():
        &lt;access specific parameters and set them to 0, and set requires_grad to False&gt;
</code></pre>
<p>Hope this helps!</p>

username: Tommy
cooked: <p>If i follow this solution I can deactivate only all the weights at the same time but not a specific element of the weight matrix. At the end of the day i want to have 2 layers where the connections are sparse, so y = Wx where W the weight matrix with some elements zero during the whole training process</p>

username: SpandanMadan
cooked: <p>Here, a more detailed answer.</p>
<p>I have a tutorial which shows accessing specific elements in detail. It doesn’t set the weight to zero and requires grad to False, but shows how to access the parameters. Here’s the github repo for the tutorial - <a href="https://github.com/Spandan-Madan/A-Collection-of-important-tasks-in-pytorch" rel="nofollow noopener">https://github.com/Spandan-Madan/A-Collection-of-important-tasks-in-pytorch</a></p>
<p>In the tutorial go to cell 22 and modify the lines to be.</p>
<p>for child in model.children():<br>
for param in child.parameters():<br>
param =  torch.zeros(param.size())<br>
param.requires_grad = False</p>
<p>This will basically prune all nodes of the model. To specify which nodes are to be pruned just add if statements in the above code to prune only the required nodes! Just add another for loop to loop over the elements within child.parameters().</p>
<p>Hope this helps!</p>

username: wxwx
cooked: <p>hello,i have same question.do you solve your problem?</p>

username: Michela
cooked: <p>There are some out-of-the-box pruning functionalities directly in PyTorch now, starting from v1.4, that will handle this for you, for a subset of pruning techniques. You can find them under  <code>torch.nn.utils.prune</code> .</p>
Iterative pruning

username: EliB
cooked: <p>Are there any PyTorch libraries, that can serve to prune networks?<br>
Tensorflow 2.0 is arrives with built in functionality for pruning: <a href="https://www.tensorflow.org/model_optimization/guide/pruning/pruning_with_keras" rel="nofollow noopener">https://www.tensorflow.org/model_optimization/guide/pruning/pruning_with_keras</a></p>

username: Michela
cooked: <p>There are some out-of-the-box pruning functionalities in PyTorch too now, starting from v1.4. You can find them under <code>torch.nn.utils.prune</code>.</p>
Getting a gradient d(df/dx)/df

username: janbednarik
cooked: <p>Let us define the following real valued functions f and g as follows:</p>
<p>f(x) = y (for instance f(x) = x^2)<br>
g(y) = dy/dx</p>
<p>The objective is to compute dg/dy.</p>
<p>The g(y) is computed using torch.autograd.grad. If I then want to compute dg/dy using torch.autograd.grad again, it fails with the message “RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.”</p>
<p>It seems that y is not part of the computation graph created after calling the first torch.autograd.grad, but I do not understand why. Here is a minimal example to reproduce the problem:</p>
<pre><code class="lang-auto">import torch
import torch.autograd as ag

# Input x
x = torch.empty((1, ), dtype=torch.float32).uniform_(-1., 1.)
x.requires_grad = True

# f(x) = y
y = x ** 2.

# g(y) = dy/dx
A = ag.grad(y, x, torch.ones((1, ), dtype=torch.float32), create_graph=True)[0]

# dg / dy - fails
B = ag.grad(A, y, torch.ones((1, ), dtype=torch.float32), create_graph=True)[0]
</code></pre>
<p>I assumed that the the computation graph would look as follows:</p>
<pre><code class="lang-auto">  ________________
 /   f         g   \|     dg/dy
x ------&gt; y ------&gt;  A ----------&gt;  B 
           \_______________________/|
</code></pre>
<p>But it seems that the branch connecting <strong>y</strong> to <strong>A</strong> does not exist (not created by the first call to torch.grad.autograd).</p>
<p>Would anyone have an idea what is going on here and how I could get such a derivative? Thank you!</p>

username: albanD
cooked: <p>Hi,</p>
<p>A is computed as <code>2 * x</code>. So A only depends on <code>x</code>, not <code>y</code>.<br>
In particular, you might want to change your notation to avoid mixing symbolic variables and values.<br>
If we write<br>
<code>f: x -&gt; y</code>. Then we can evaluate it at a given value <code>x0</code> to get <code>y0 = f(x0)</code>.<br>
But then <code>A = (df/dx)(x0)</code> (the derivative of f evaluated at the point <code>x0</code>).<br>
<code>y0</code> is not actually an input of <code>g</code> here (it is just an intermediary result), so writing <code>g(y)</code> (or <code>g(y0)</code>) does not really make sense. It is actually <code>g(x0)</code>.</p>
Best way to check if my model architecture is correct?

username: krishnab
cooked: <p>Hello folks. I am new to Pytorch and so far I like it a lot more than Tensorflow. But my question was about checking the architecture of my models. I am still getting used to the API for the layers in Pytorch, and I was hoping to find a way to visually validate the architecture of my models.</p>
<p>In other words, I write a stacked RNN or an LSTM with multiple layers, and I would like to know that the architecture that is coded is the architecture that I actually <em>intended</em>, if that makes sense. Without some explicit error on tensor shape, etc., it can be hard to make sure that I used the right argument to specify the number of hidden layers that I wanted, or the number of hidden states that I am using to plug into my fully connected layer. Especially for longer models, or models with a lot of cutting and pasting, it can be hard to ensure that the model architecture is what I intended.</p>
<p>Would the Tensorboard visualization fulfill this purpose, or are there other tools that take the model specification and then convert it into a visual representation of the model. I think a visual check would be the easiest way for me to tell that the model looks correct.</p>
<p>Any suggestions would be appreciated.<br>
Krishna</p>

username: ptrblck
cooked: <p><a href="https://github.com/szagoruyko/pytorchviz">torchviz</a> could visualize the graph for a specific input, which might be helpful.<br>
I’m not sure, how detailed e.g. the LSTM layer will be visualized, so you might try it out and see, if it helps.</p>
<p>Also, <a class="mention" href="/u/tom">@tom</a> has recently worked on a visualization package for notebooks, if I’m not mistaken, which might be helpful. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: krishnab
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for the info. I am still new to pytorch, so I don’t really know the package ecosystem yet. I will check out <code>torchviz</code>. That is great. Right now I am just putting the model code into <code>pdb</code> and then checking the sizes of the output tensors. That seems to work well–better than tensorflow, anyway <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> But it would be nice to have some visual way to visually inspect the model to make sure I did not miss anything.</p>
<p>Thanks again.</p>
How to find test accuracy after training

username: vishnu_vardhan1
cooked: <p>Hi,<br>
I’m a beginner and I trained a image classification model with training data and validation data using resnet50. But now I want to test my model performance on test data. Can anyone help me with this?</p>

username: ptrblck
cooked: <p>If you have the test data including the targets, you could reuse the validation approach and just swap the test data for the validation set.<br>
Since the validation code should neither update the model not use any data augmentation techniques, it should be OK to use the same workflow for the test case.</p>

username: AladdinPerzon
cooked: <p>The first thing you need to do is to separate your dataset into a training, validation and test set. For each of these you create a loader exactly like you’ve done for the training loader. Assuming you’ve done that and have a training_loader, validation_loader, and test_loader, you could then define a separate function to check the accuracy which will be general in the way that you just need to send in the loader you’ve created.</p>
<p>This could look something like this:</p>
<pre><code class="lang-auto">def check_accuracy(loader, model):
    num_correct = 0
    num_samples = 0
    model.eval()
    
    with torch.no_grad():
        for x, y in loader:
            x = x.to(device=device)
            y = y.to(device=device)
            
            scores = model(x)
            _, predictions = scores.max(1)
            num_correct += (predictions == y).sum()
            num_samples += predictions.size(0)
        
        print(f'Got {num_correct} / {num_samples} with accuracy {float(num_correct)/float(num_samples)*100:.2f}') 
    
    model.train()
</code></pre>
<p>Where you need to have defined device, but you could also send this to the function.</p>

username: vishnu_vardhan1
cooked: <p>yeah, Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: vishnu_vardhan1
cooked: <p>Thank so much <a class="mention" href="/u/aladdinperzon">@AladdinPerzon</a>  for sharing the code</p>
Arithmetics along batches

username: Podidiving
cooked: <p>Hello everyone! I have 2 tensors: first (tensor A) is (b, c, h, w) shape and second (tensor B) is (b) shape. I want to make substraction, A - B, so from every tensor (c, h, w) were substracted corresponding number. But I got <code>RuntimeError: The size of tensor a (w) must match the size of tensor b (b) at non-singleton dimension 3</code></p>

username: AladdinPerzon
cooked: <p>I think if you add additional dimensions for <strong>tensor B</strong> then you could utilize broadcasting and obtain the results you desire. For example:</p>
<pre><code class="lang-auto">x = torch.ones((2, 3, 64, 64))
y = torch.ones((2))

x - y # This will bring an error
x - y.view(-1, 1, 1, 1) # This will work
</code></pre>

username: Podidiving
cooked: <p>Hi! Thank you for respond<br>
However <code>y.expand(2, 1, 1, 1)</code> gives me an error <code>RuntimeError: The expanded size of the tensor (1) must match the existing size (2) at non-singleton dimension 3.  Target sizes: [2, 1, 1, 1].  Tensor sizes: [2]</code><br>
I’ve made it the next way:<br>
`<br>
x = torch.ones((2, 3, 64, 64))<br>
y = torch.ones((2))</p>
<p>x.view(2, -1) - y[:, None]<br>
`<br>
It works, but I’m not sure, it is the most optimal way</p>

username: AladdinPerzon
cooked: <p>Sorry, made a mistake when I tested it, I edited my previous answer and used <code>view</code> instead. Does this work for you now?</p>

username: Podidiving
cooked: <aside class="quote no-group" data-username="AladdinPerzon" data-post="2" data-topic="89162">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aladdinperzon/48/13045_2.png" class="avatar"> AladdinPerzon:</div>
<blockquote>
<pre><code class="lang-auto">x - y.view(-1, 1, 1, 1) # This will work
</code></pre>
</blockquote>
</aside>
<p>Yeah! Now It’s working. Thank you</p>
Error recognizing my GPU

username: Vendrick17
cooked: <p>Hi everyone, I have started working with multiple GPUs (K80) but when I run <code>torch.cuda.is_available()</code> the output is <code>False</code>, and the following error message appears in another section of my code:</p>
<pre><code class="lang-auto">AssertionError: 
The NVIDIA driver on your system is too old (found version 9010).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.
</code></pre>
<p>I have already checked my CUDA version with the nvcc–version command, it is 10.1, the Pytorch version is 1.5.1+cu101, torchvision 0.6.1+cu101, these are the specifications of the GPUs:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be2e0ab14f80297dd0ae5f14e6baca310ead65dd.png" data-download-href="https://discuss.pytorch.org/uploads/default/be2e0ab14f80297dd0ae5f14e6baca310ead65dd" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be2e0ab14f80297dd0ae5f14e6baca310ead65dd.png" alt="image" data-base62-sha1="r8pmwYMAm7WYXY8exqKEoM2557v" width="481" height="375" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">646×503 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks in advance for your help.</p>

username: naveen139
cooked: <p>I suspect that the problem is exactly what the error mentions, that the driver versions are pretty old. Check the <a href="https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#major-components" rel="nofollow noopener">nvidia docs</a>, the minimum requirements for cuda 10.1 is atleast 418. And the message shows your version is 390.</p>

username: Vendrick17
cooked: <p>Thank you so much, I will update it.</p>
Torch determinant function fails for batched inputs

username: AlphaBetaGamma96
cooked: <p>Hi everyone,</p>
<p>I’ve been following the torch.det(x) example <a href="https://pytorch.org/docs/stable/torch.html#torch.det" rel="nofollow noopener">here</a>. Yet, I get the following error and I can’t seem to find a solution online. My code is below,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; A = torch.randn(3,3) #initialise tensor (array) of size 3,3
&gt;&gt;&gt; torch.det(A) #calc determinant
tensor(0.4273)
&gt;&gt;&gt; 
&gt;&gt;&gt; A = torch.randn(3,2,2) #initialise tensor (array) of 3,2,2
&gt;&gt;&gt; A
tensor([[[-0.6285, -0.2763],
         [-0.8588,  1.3266]],

        [[ 1.3004, -0.7479],
         [-1.1130, -0.2943]],

        [[-1.6380, -1.3417],
         [-0.0915,  0.8072]]])
&gt;&gt;&gt; A.det() #calculate determinant along axis 0 (i.e. calc determinant for [i,:,:])
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: det(torch.FloatTensor{[3, 2, 2]}): expected a 2D square tensor of floating types
&gt;&gt;&gt; torch.det(A)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: det(torch.FloatTensor{[3, 2, 2]}): expected a 2D square tensor of floating types
</code></pre>
<p>This is the same as the example in the link above, yet I get an error? I assume it might be a version issue? (I’ve copied the example from link below for completeness)</p>
<pre><code class="lang-auto">&gt;&gt;&gt; A = torch.randn(3, 3)
&gt;&gt;&gt; torch.det(A)
tensor(3.7641)

&gt;&gt;&gt; A = torch.randn(3, 2, 2)
&gt;&gt;&gt; A
tensor([[[ 0.9254, -0.6213],
         [-0.5787,  1.6843]],

        [[ 0.3242, -0.9665],
         [ 0.4539, -0.0887]],

        [[ 1.1336, -0.4025],
         [-0.7089,  0.9032]]])
&gt;&gt;&gt; A.det()
tensor([1.1990, 0.4099, 0.7386])
</code></pre>
<p>Versions for my code are: PyTorch version: 1.1.0  with py3.7_cuda10.0.130_cudnn7.5.1_0</p>
<p>Thank you in advance!</p>

username: AladdinPerzon
cooked: <p>When looking at the documentation for <code>torch.det</code> for Pytorch version 1.1.0 they don’t have this feature as an example, so it’s most likely what you mentioned that it’s a feature that was made available at later releases. It seems that they introduced this in 1.2.0 (this is when the docs have an example on this).</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/aladdinperzon">@AladdinPerzon</a>, thanks for the response! It indeed seems like a version issue! Apologises for the silly question! Cheers!</p>
Is there a linear solver for xA=B like mrdivide in matlab?

username: msqiqi10
cooked: <p>I know that torch.solve solves Ax=B for x, but Is there a linear solver for xA=B like <a href="https://www.mathworks.com/help/matlab/ref/mrdivide.html" rel="nofollow noopener">mrdivide</a> in matlab?<br>
Thank you!</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think we have that function. But you can use compute a pseudo inverse for A with <code>torch.pinv()</code> to get the same result.</p>
Forward and derivative functions for error criterion

username: pytorcher
cooked: <p>I have some functions that require storing error values at a current layer as they are passed through.  For example if the nonlinearity of a layer is sigmoid the line of code looks like:</p>
<pre><code class="lang-auto">grad_output[0].detach()*(Value + sigmoidPrime(F.sigmoid(self.totalOut)))
</code></pre>
<p>and I have a defined</p>
<pre><code class="lang-auto">def sigmoidPrime(x):
    return x * (1-x)
</code></pre>
<p>I am looking to do the same thing for the top layer which would require functions for the actual forward and backward steps of what is happening within loss functions.  Questions 1 is if there is any generic way to do this or a place I can look within the pytoch definitions for these functions.  2 is, if not, if anyone can help me write what these might look like for some common loss functions like MSE and CrossEntropy?</p>
<pre><code class="lang-auto">def crossEntropyForward(x):
    ???
def crossEntropyPrime(x):
    ???
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>I’m not sure what <code>sigmoidPrime</code> is supposed to represent here?<br>
Also not sure what the <code>grad_output</code> is in your formula above. Can you share more context?</p>

username: pytorcher
cooked: <p>Ah yes sorry, this code is in a register backward hook function</p>
<pre><code class="lang-auto">layer.register_backward_hook(saveAverageD)
...
def saveAverageD(self, grad_input, grad_output):
</code></pre>
<p>Sigmoid prime is the derivative of the sigmoid function to calculate the gradient during the backward pass if autograd wasn’t doing it already under the hood.</p>
<p>The module just has a conv2D or Linear layer and then a sigmoid for the earlier layers and it seems to be doing what I want it to do.  But at the top layer where it is just a Linear into the loss function, I do not do this step and just use grad_output it does not appear to be working.</p>

username: albanD
cooked: <aside class="quote no-group" data-username="pytorcher" data-post="3" data-topic="89048">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/7ea924/48.png" class="avatar"> pytorcher:</div>
<blockquote>
<p>Ah yes sorry, this code is in a register backward hook function</p>
</blockquote>
</aside>
<p>I’m still unsure what your goal here is <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>But if you want the forward and backward functions, you just need to differentiate the functions.<br>
For example, you have mse that does this in the forward: <code>mse(x, y) = (x - y).norm(2).mean()</code><br>
Then you can differentiate wrt to x and do <code>mse_backward_x(grad_out, x, y) = 2/N * grad_out.expand_as(x) * (x - y)</code></p>
Better-Engineering Label

username: justhube
cooked: <p>What does this label mean for issues on the github repo?</p>

username: albanD
cooked: <p>Hi,</p>
<p>We use it for improvement that are not new features but would be nice to have from an engineering point of view: it makes the code easier to read/easier to extend.</p>

username: justhube
cooked: <p>Is the contribution process any different for these? Should I open an issue with design details?</p>

username: albanD
cooked: <p>If it is just a simple cleanup, you can send a PR.<br>
But if you have any doubt, do open an issue to discuss this!</p>
How to use torch.save() when the model architecture is changing?

username: gebrahimi
cooked: <p>I have asked this before and did not get an answer. So asking again to see if someone has an idea about it.</p>
<p>I am working on a project that prunes neurons (i.e. the architecture of the network is consistently changing by removing some neurons from layers). I need to save models during training which have different architectures. As far as I know, i have two options:</p>
<p>First option:</p>
<pre><code>torch.save(net.state_dict(), PATH)
model = Net()
model.load_state_dict(torch.load(PATH))
</code></pre>
<p>Second option</p>
<pre><code>torch.save(net, PATH)
</code></pre>
<p>I am currently using second option, however it is saving a lot of stuff somehow and although the memory it should take for networks like alexnet and resnet should not be more than 100 mb, my models take 10-15 GB. In other words it is saving some other tensors which i don’t know what they are.</p>
<p>I would like to use the first option, however since i am pruning the network and the architecture is changing constantly, i will not be able to load it back unless i know the architecture and also have a code that make that arch for me which is not convenient.</p>
<p>Now my question is that, is there a way to use the second option while i also save the model architecture so i can load them back? I would appreciate any help. Thanks.</p>

username: Michela
cooked: <p>Are you using torch.nn.utils.prune to prune your models?</p>

username: gebrahimi
cooked: <p><a class="mention" href="/u/michela">@Michela</a> No, I have my own algo to prune.</p>
Session Crashed, RAM out of memory in Colab

username: cskarthik7
cooked: <p>I am trying to implement self attention network and here is the code</p>
<pre><code class="lang-auto">class FANet(nn.Module):
  def __init__(self,ipchannels):
    super(FANet,self).__init__()
    self.conv1=nn.Conv2d(in_channels=ipchannels,out_channels=128,kernel_size=1,stride=1)
    self.bn1=nn.BatchNorm2d(128)
    self.conv2=nn.Conv2d(in_channels=128,out_channels=512,kernel_size=1,stride=1)
    self.bn2=nn.BatchNorm2d(512)
    self.relu=nn.ReLU()

  def forward(self,x):
    bs=x.size(0)
    h=x.size(2)
    q=self.conv1(x)
    q=self.bn1(q)
    k=self.bn1(self.conv1(x))
    v=self.bn1(self.conv1(x))
    q=F.normalize(q.view(128,-1),p=2,dim=0)
    k=F.normalize(k.view(128,-1),p=2,dim=0)
    v=self.relu(v.view(-1,128))
    result1=torch.mm(v,k)
    result2=torch.mm(q,result1)
    print(result2.shape)
    finalresult=self.bn2(self.conv2(result2.view(bs,128,h,-1)))
    print(finalresult.size())
    return finalresult
</code></pre>
<p>And my RAM always crashes when I try to pass an image tensor through  the FANet()</p>
Data Augmentation

username: gamebo5000
cooked: <p>I’m working on a CNN project and I’m trying to improve it. I was told me how I augment my data can greatly influence my results. I was told to look into inception style data augmentation, but I can’t find anything on. Does anyone know what that is and where I can find out about it, as well as in any other augmentation techniques/styles?</p>
<p>This is what I have so far:</p>
<blockquote>
<p>train_transform = transforms.Compose([<br>
transforms.Resize(224),<br>
transforms.CenterCrop(224),<br>
transforms.RandomRotation(10),      # rotate +/- 10 degrees<br>
transforms.RandomHorizontalFlip(),  # reverse 50% of images<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406],<br>
[0.229, 0.224, 0.225])<br>
])</p>
<p>test_transform = transforms.Compose([<br>
transforms.Resize(224),<br>
transforms.CenterCrop(224),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406],<br>
[0.229, 0.224, 0.225])<br>
])</p>
</blockquote>
<p>I am using the torchvison transform page on PyTorch, but are there any other augmentation styles or techniques anyone can point me to?</p>

username: ptrblck
cooked: <p>I guess you might be looking for <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomResizedCrop"><code>RandomResizedCrop</code></a>, which was used in Inception models.<br>
The <a href="https://arxiv.org/pdf/1409.4842.pdf">original paper</a> might also mention other augmentation techniques.</p>
<p>Besides the <code>torchvision</code> transformations, you could also have a look at <a href="https://github.com/kornia/kornia">kornia</a>.</p>