akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
‘gelu_backward_cuda’ is not a member of ‘at::native’

hs_matt: <p>Hi, it seems PyTorch 1.10 has changed its C++ API for gelu and softmax backward function. Previously, I can make customized backward functions by calling</p>
<pre><code class="lang-auto">at::native::gelu_backward_cuda(grad_output, input);
at::native::softmax_backward_cuda(grad_output, output, dim, self);
</code></pre>
<p>But now we get the errors:</p>
<pre><code class="lang-auto">error: ‘gelu_backward_cuda’ is not a member of ‘at::native’
error: ‘softmax_backward_cuda’ is not a member of ‘at::native’
</code></pre>
<p>Are there any solutions for that?</p>
<p>Kind regards</p>

ptrblck: <p>You should be able to replace them with <code>gelu_backward_out_cuda</code> and <code>softmax_backward_cuda_out</code>.</p>

hs_matt: <p>Thanks for your quick response!</p>
<p>I have tried to use <code>gelu_backward_out_cuda</code> and <code>softmax_backward_cuda_out</code> . However, the same errors still appear. I was wondering if the arguments for the two functions have changed as well?</p>
<p>My code:</p>
<pre><code class="lang-auto">return at::native::gelu_backward_cuda_out(grad_output, input);
...
return at::native::softmax_backward_cuda_out(grad_output, output, dim, self);
</code></pre>
<p>The errors:</p>
<pre><code class="lang-auto">error: ‘gelu_backward_cuda_out’ is not a member of ‘at::native’
error: ‘softmax_backward_cuda_out’ is not a member of ‘at::native’
</code></pre>
Learnable LR, not getting gradients

zilong: <p>Hi everyone,</p>
<p>I am trying to implement MetaSGD, where learning rates are learnable. I am having trouble getting gradients for them.</p>
<p>The code:</p>
<pre><code class="lang-auto">        for j in range(iterations):
            self.optim.zero_grad()
            inner_output = self.model(inner_x)
            inner_loss = self.criterion(inner_output, inner_y)
            grads = torch.autograd.grad(inner_loss, self.model.parameters(), create_graph=True)
            for i, param in enumerate(self.model.parameters()):
                param.grad = self.lrs[i] * grads[i]
             self.optim.step()

        outer_output = self.model(outer_x)
        outer_loss = self.criterion(outer_output, outer_y)
        
        # These lines should create the grads for the learning rates and update them accordingly.
        outer_loss.backward()
        # lrs.grad = None
        other_optimizer.step()
</code></pre>
<p>What happens is that <code>lrs.grad = None</code>. Meaning that <code>param.grad = self.lrs[i] * grads[i]</code> probably doesnt add to the computation graph for calculating the gradients.</p>
<p>And if I change the code to:</p>
<pre><code class="lang-auto">        for j in range(iterations):
            self.optim.zero_grad()
            inner_output = self.model(inner_x)
            inner_loss = self.criterion(inner_output, inner_y)
            grads = torch.autograd.grad(inner_loss, self.model.parameters(), create_graph=True)
            for i, param in enumerate(self.model.parameters()):
                param -= self.lrs[i] * grads[i]

        outer_output = self.model(outer_x)
        outer_loss = self.criterion(outer_output, outer_y)
        
        # These lines should create the grads for the learning rates and update them accordingly.
        outer_loss.backward()
        other_optimizer.step()
</code></pre>
<p>Then <code>param -= self.lrs[i] * grads[i]</code> gives a <code>RunTimeError</code> because of an in place operation. Which are not allowed for variables that requires grad.</p>
<p>&lt;edit&gt;<br>
<code>param = param - self.lrs[i] * grads[i]</code>, here no learning happens as well because the first param actually a new variable probably. And self.lrs.grad = None as well.<br>
&lt;/edit&gt;</p>
<p>Does anyone know a solution, i.e. getting gradients for the learning rates?</p>
Debugger question about tensor

The_duke: <p>Hi,</p>
<p>I started using spyder and a debugger not long ago.<br>
Essentially, I just want to have the effect of print(tensor) without having to write print(tensor) everywhere in my code.</p>
<p>So I take a look inside the variable explorer:<br>
while I can peek inside a dataframe just fine (the “value” column coincides with what I’d get if I were to print(dataframe)). In the case of pytorch tensor however, the variable explorer shows a tensor object  alongside all its built-in methods, I’m not too sure how I can peek inside of it.</p>
<p>What am I doing wrong? If I’m not looking at the right IDE, is there any other IDE that would allow me to do so?</p>
<p>Thanks</p>

ptrblck: <p>There is an <a href="https://github.com/spyder-ide/spyder/issues/7042">open feature request</a> for Spyder wich a posted workaround in case you are comfortable changing the spyder internals.</p>

googlebot: <p>pycharm can auto-peek, but for GPU tensors this lags pretty badly (because of cross-device data transfers, I don’t blame IDE here), so pycharm’s on demand variable loading mode is arguably better - it loads data on a mouse click</p>
Creating a custom GPU DataLoader

jacobbuckman: <p>I was wondering whether there was any good guide on best practices for creating my own DataLoader. To be clear: I’m <strong>not</strong> looking to create a custom Dataset, and am <strong>not</strong> looking to create a DataLoader subclass that adds features. I’m looking to build, from the ground-up, my own DataLoader replacement.</p>
<p>For single-process loading, this is easy. But for multi-process loading, it gets extremely hairy. I keep running into issues. Here’s a few examples of things I ran into:</p>
<ul>
<li>Sending big tensors across pipes is extremely slow. (I eventually discovered that I needed to use torch.multuprocessing.Queue instead of Python’s built-in Queue.)</li>
<li>Calling .pin_memory() on my worker process causes torch to initiaize CUDA and load CUDA libraries, which has a bunch of GPU memory overhead; this means that I run out of memory whenever I have more than a few workers. (Haven’t solved this one yet…)</li>
</ul>
<p>The torch DataLoader class doesn’t have these issues, but the code is quite impenetrable, and it has not been very helpful in guiding me toward what I need to do. Are there any simple tutorials for how to do this? Or could anybody offer me some guidance?</p>
Different ways to freeze a layer?

flpgrz: <p>If you want to freeze a sub-module of an nn.Module object (e.g. you want to freeze some layer of a NN), what’s the difference between:</p>
<p>A) <code>model.layer_to_freeze.eval()</code></p>
<p>B) <code>model.layer_to_freeze.requires_grad_(False)</code></p>
<p>?</p>
<p>Thanks</p>

ptrblck: <p><code>train()/eval()</code> will change the behavior of some layers and are used during training and evaluatio, respectively. E.g. batchnorm layers will use the running stats to normalize the input during evaluation instead of the input activation stats and dropout layers will be disabled during evaluation.<br>
Setting the <code>requires_grad</code> attribute of parameters will disable gradient calculation for these parameters and is commonly referred to when you are “freezing” layers.</p>
Using Sparsemax along with ke_padding mask for a batch_size >1

Rini: <p>Hi,</p>
<p>I have been recently toying with Sparsemax instead of Softmax to get sparsity in the attention matrix.</p>
<p>What  I have noticed is that with a batch_size of 1, I get sparsity in the attention matrix (For a given query, some of the keys are set to 0). There is no key padding mask used in this case.<br>
However when I use a batch_size&gt;1 and provide a key_padding mask, the attention matrix has zeros only in the masked position(For a given query, only the keys in the masked position are set to 0). I am not getting sparsity among the keys that actually matter (the unmasked positions).<br>
I think based on how sparsemax works (<a href="https://pypi.org/project/sparsemax/" class="inline-onebox" rel="noopener nofollow ugc">sparsemax · PyPI</a>) this is probably expected but then this might prevent from using a batch_size&gt;1 .</p>
<p>Has anyone come across this issue?</p>
<p>Thanks!</p>
<p>Below my MAB class</p>
<blockquote>
<p>Blockquote</p>
</blockquote>
<pre><code>class MAB(nn.Module):
def __init__(self, dim_Q, dim_K, dim_V, num_heads, ln=False):
    super(MAB, self).__init__()
    self.dim_V = dim_V
    self.num_heads = num_heads
    self.fc_q = nn.Linear(dim_Q, dim_V)
    self.fc_k = nn.Linear(dim_K, dim_V)
    self.fc_v = nn.Linear(dim_K, dim_V)
    if ln:
        self.ln0 = nn.LayerNorm(dim_V)
        self.ln1 = nn.LayerNorm(dim_V)
    self.fc_o = nn.Linear(dim_V, dim_V)

def forward(self, Q, K, key_mask=None, attn_mask=None):
    Q = self.fc_q(Q)
    K, V = self.fc_k(K), self.fc_v(K)

    dim_split = self.dim_V // self.num_heads
    Q_ = torch.cat(Q.split(dim_split, 2), 0)
    K_ = torch.cat(K.split(dim_split, 2), 0)
    V_ = torch.cat(V.split(dim_split, 2), 0)

    A = Q_.bmm(K_.transpose(1, 2)) / math.sqrt(self.dim_V)

    if key_mask is not None:
        attn_mask = key_mask
        A = A.masked_fill(attn_mask == 1, -9e15)

    sparsemax = Sparsemax(dim=-1)
    A = sparsemax(A)


    O = torch.cat((Q_ + A.bmm(V_)).split(Q.size(0), 0), 2)
    O = O if getattr(self, "ln0", None) is None else self.ln0(O)
    O = O + F.relu(self.fc_o(O))
    O = O if getattr(self, "ln1", None) is None else self.ln1(O)

    return (O, A)
</code></pre>
Pytorch traning on android/iOS devices

srohit0: <p>Is there pytorch.js library (similar to tensorflow.js) to train models  directlly on mobile edevices?</p>

dhruvbird: <p>Hello Rohit! Unfortunately there’s isn’t a JS library for on-device training available at this point in time. However, you could train models directly using the C++ API since the full-jit interpreter can run on mobile platforms.</p>

srohit0: <p>Thanks Dhruv.</p>
<p>Keep this thread updated when pyTorch decides to have one to compate with TF.</p>
How to run on gpu in apple M1

blade: <p>The (not so) new apple M1 chip has integrated GPU cores. I was trying running a simple PyTorch code on GPU on this machine, but <code>torch.cuda.is_available():</code> returns <code>False</code>. I was wondering if this is because of PyTorch not supporting this GPU or is there any additional step that I should take before running?</p>

albanD: <p>Hi,</p>
<p>The GPU part of the M1 chip is not based on nvidia CUDA so it won’t work with our CUDA backend, but The CPU part should run just fine.</p>

blade: <p>Thanks for the response! <a class="mention" href="/u/alband">@albanD</a> Is there a work in progress/plan to support M1 chip GPUs later?</p>

blade: <p>OK, so for the folks ending up here, here is the latest update: <a href="https://github.com/pytorch/pytorch/issues/47702#issuecomment-965625139" class="inline-onebox" rel="noopener nofollow ugc">GPU acceleration for Apple's M1 chip? · Issue #47702 · pytorch/pytorch · GitHub</a></p>
Is there a loss function for multiclassification in pytorch?

KNEC: <p>Hello, I am doing textclassification (3 classes) with huggigface library and Would like to know which loss function should use ? I used crossentropyloss but since reading forums , I found categorical cross entropy.</p>
<p>So which one is better ?</p>

KFrank: <p>Hi Emmanuelle!</p>
<aside class="quote no-group" data-username="KNEC" data-post="1" data-topic="138433" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/db5fbb/48.png" class="avatar"> KNEC:</div>
<blockquote>
<p>I used crossentropyloss but since reading forums , I found categorical cross entropy.</p>
<p>So which one is better ?</p>
</blockquote>
</aside>
<p>The choice will depend on how your data is “annotated.”</p>
<p>If your ground-truth labels (annotations) are integer class labels, then<br>
you would use what I believe you are calling “categorical cross entropy.”</p>
<p>That is, each sample in your data set would be labelled with a single<br>
integer – <code>0</code> for class-0, <code>1</code> for class-1, and <code>2</code> for class-2.</p>
<p>If your labels are probabilistic “soft” labels, then you would use the<br>
more general full cross entropy.</p>
<p>In this case, each sample would be labelled with three floating-point<br>
probabilities, <code>P[c]</code>, where <code>P[0]</code> would be the probability of that sample<br>
being in class-0, <code>P[1]</code> the probability of being in class-1, and so on.</p>
<p>As of the latest stable release, 1.10.0, pytorch’s <code>CrossEntropyLoss</code><br>
supports both categorical and probabilistic labels.</p>
<p>But again, the kind of labels you have determines which version of<br>
cross-entropy loss you should use.</p>
<p>Best.</p>
<p>K. Frank</p>
AttributeError: module 'torch' has no attribute 'permute'

Lady_Hangaku: <p>I tried to run the code below for training a sequence tagging model (didn’t list all of the code because it works fine). But I get the following error:<br>
<code>AttributeError: module 'torch' has no attribute 'permute'</code><br>
<code>torch</code> is definitely installed, otherwise other operations made with torch wouldn’t work, too. The code works on Windows 10, conda environment, pip installed torch. But not on Ubuntu 16.04, conda environment, pip installed torch. Any clue how that could be?</p>
<pre><code class="lang-auto">import sys, os
from pathlib import Path
import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.optim as optim
from torch.nn.utils.rnn import pad_packed_sequence, pack_padded_sequence, pad_sequence
from torch.utils.data import DataLoader, random_split
import torch.nn.functional as F
import re
from tqdm import tqdm, trange
import fasttext
import fasttext.util


# Preprocessing of data and initialising models not listed...
for epoch in trange(epochs, desc="Epoch"):
    for step, batch in enumerate(train_loader):

        docs = batch[0][0].to(device)
        tags = batch[1].to(device)
        
        model.zero_grad()
        
        doc_forward = docs
        doc_backward = torch.clone(doc_forward)
        doc_backward = torch.flip(doc_backward, [1])

        tag_scores = model(doc_forward, doc_backward)
        tag_scores = torch.permute(tag_scores, (0, 2, 1))

        loss = loss_function(tag_scores, tags)
        loss.backward()
        optimizer.step()
</code></pre>

tom: <aside class="quote no-group" data-username="Lady_Hangaku" data-post="1" data-topic="138448">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/ce7236/48.png" class="avatar"> Lady_Hangaku:</div>
<blockquote>
<p><code>tag_scores = torch.permute(tag_scores, (0, 2, 1))</code></p>
</blockquote>
</aside>
<p><code>tag_scores = tag_scores.permute((0, 2, 1))</code></p>
Nan values in confusion matrix

JanaR: <p>Hi,<br>
I’ve got a warning that there are 5 nan values in confusion matrix that were replaced with 0.<br>
What does that mean? What does it mean to have a nan values in the confusion matrix?</p>

ptrblck: <p>Could you describe your use case a bit more and which method raises the issue?<br>
Based on your current description I guess you are using a 3rd party package (e.g. <code>sklearn.metrics.confusion_matrix</code>) which tries to normalize the confusion matrix and might be running into a zero division?</p>

JanaR: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
this is the line of code I am using:</p>
<pre><code class="lang-auto"> confusion_matrix = tm.ConfusionMatrix(num_classes=nb_classes, threshold=0.5, normalize='true', multilabel=True)
</code></pre>
<p>and this is the warning:<br>
Testing: 0it [00:00, ?it/s]C:\ProgramData\Miniconda3\lib\site-packages\torchmetrics\utilities\prints.py:37: UserWarning: 16 nan values found in confusion matrix have been replaced with zeros.</p>

ptoews: <p>I had the same problem, because when I forwarded the ConfusionMatrix metric (<code>metric()</code>) at every training step it hasn’t had encountered every class yet, so it couldn’t properly normalize the counts (dividing zero by zero). I solved it by only updating the metric (<code>metric.update()</code>) at every training step, and since I log only every 50 training steps where every class has been encountered by then, I have no warnings anymore.</p>
Understanding nn.Conv2d class

Zhihan_Yang: <p>I’m trying to understand how nn.Conv2d is implemented.</p>
<p>As described by the example in <a href="https://pytorch.org/docs/stable/notes/extending.html" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 2.1 documentation</a>, implementing a custom linear layer from absolute scratch means that we need to first implement a custom linear function from scratch that has properly defined forward and backward static methods.</p>
<p>After reading the source code of nn.Conv2d (which is amazingly short), I find that it also relies on a function called F.conv2d, which does ALL the heavy lifting. I can’t find the code for this function, but I’ve found my answer here:</p>
<aside class="quote" data-post="1" data-topic="35424">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/a5b964/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/where-is-the-torch-nn-functional-conv1d-code/35424">Where is the torch.nn.functional.conv1d code?</a> 
  </div>
  <blockquote>
    Hi 
I searched torch files but did not find the main body codes of “torch.nn.functional.conv1d” . Is this code available to the public?
  </blockquote>
</aside>

<p>So, my question is:</p>
<p><strong>Does F.conv2d define both forward and backward code in a lower-level language? If so, which file is it? The path provided in the link above has changed.</strong></p>

my3bikaht: <p>Source is here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/conv.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/conv.py at master · pytorch/pytorch · GitHub</a><br>
backward method is inherited from torch.nn.Module</p>
Regular BatchNorm triggers buffer broadcast?

vadimkantorov: <p>Are regular BatchNorm buffers (<code>running_mean</code>, <code>running_var</code> I guess) also broadcast and synchronized during forward pass when using DistributedDataParallel?</p>
<p>I thought that only SyncBatchNorm does this.</p>
<p>Also, will this broadcast/sync also happen in eval mode? (for SyncBatchNorm)</p>
<p>How is it controlled which buffers are synchronized and which are not?</p>

wanchaol: <p>Hi <a class="mention" href="/u/vadimkantorov">@vadimkantorov</a> I think only SyncBatchNorm will broadcast and sync when using DDP, but they are syncing mean/invstd/count across GPUs instead of syncing buffers, the buffers maintained locally and updated locally. You can refer to the detailed implementaion in <a href="https://github.com/pytorch/pytorch/blob/251686fc4cb1962944ed99c938df2d54f3d62e46/torch/nn/modules/_functions.py#L6" class="inline-onebox" rel="noopener nofollow ugc">pytorch/_functions.py at 251686fc4cb1962944ed99c938df2d54f3d62e46 · pytorch/pytorch · GitHub</a></p>
<p>For SyncBatchNormed, broadcast/sync will not happen in eval mode, only in train mode.</p>

vadimkantorov: <p>I’m asking because we had this problem recently when a model without SyncBatchNorm was getting deadlocked during boradcasting of buffers.</p>
<p>The reason of the deadlock was different: <a href="https://github.com/speechbrain/speechbrain/issues/1166" class="inline-onebox" rel="noopener nofollow ugc">Checkpointing may cause the NCCL error · Issue #1166 · speechbrain/speechbrain · GitHub</a>, but it was still strange that it still went into the code path of broadcasting the buffers. In debugging the buffers were the stats of regular BatchNorm1d.</p>
How can I get __add__ feature map?

Eta_C: <p>Hi, I have a model</p>
<pre><code class="lang-python">class Model(nn.Module):
    def __init__(self):
        pass
    
    def forward(self, x):
        x1 = x * 2 
        x2 = x1 + x 
        x3 = x2 + 1
        return x3
</code></pre>
<p>I  want to get <code>x1</code>, <code>x2</code> and <code>x3</code>.<br>
As a universal method, I <strong>CAN NOT</strong> <code>return x1, x2, x3</code>. I try <code>forward_hook</code> but it can only trace the module which defined in <code>__init__</code>.</p>

Eta_C: <p>I have an idea.</p>
<ol>
<li>Find all <code>add</code>, <code>sub</code>, <code>mul</code>, <code>div</code> operators using <code>fx</code>
</li>
<li>Define <code>Add</code>, <code>Sub</code>, <code>Mul</code>, <code>Div</code> class.</li>
<li>Add those modules using <code>Module.add_module</code>
</li>
<li>Run <code>fx</code> again and replace <code>a+b</code> to <code>self.add(a, b)</code>
</li>
</ol>
<p>Then those operators would be like a <code>Module</code> instead of  a operator. So that I can use <code>forward_hook</code>.<br>
<strong>BUT IT’S EXTREMELY COMPLEX</strong>， do you have any better suggestions? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

tom: <p>Can you be a bit more specific how your goal differs from what the <a href="https://pytorch.org/blog/FX-feature-extraction-torchvision/" rel="noopener nofollow ugc">fx show case for torchvision</a> does?</p>
<p>The snippet I have in mind is:</p>
<pre><code class="lang-python">from torchvision.models.feature_extraction import create_feature_extractor
# Confused about the node specification here?
# We are allowed to provide truncated node names, and `create_feature_extractor`
# will choose the last node with that prefix.
feature_extractor = create_feature_extractor(
	model, return_nodes=['blocks.0', 'blocks.1', 'blocks.2', 'blocks.3'])
# `out` will be a dict of Tensors, each representing a feature map
out = feature_extractor(torch.zeros(1, 3, 32, 32))
</code></pre>
<p>(And from the blog post, people do appear to believe that this is the best method to do it.)</p>
<p>Best regards</p>
<p>Thomas</p>

Eta_C: <p>What a great new feature!<br>
I woule like to have a try today, thank you so much. <a class="mention" href="/u/tom">@tom</a></p>
<p>Edit: I have other more custom operators. <code>create_feature_extractor</code> is not enough but it inspires me. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"> Thanks again, tom.</p>
Modify gradient computation of `torch.linalg.svd`

souryadey: <p>I am performing a singular value decomposition during the forward pass using <code>torch.linalg.svd</code>. There are some singular values that are repeated. As a result, the gradients explode during the backward pass  since <code>1/(sigma_i**2-sigma_j**2)</code> becomes <code>nan</code>.</p>
<p>What I want to do is modify the SVD backward calculation by adding <code>torch.eps</code> to the denominator of the gradient computation, so that <code>nan</code>s are avoided. How can I do this?</p>

ptrblck: <p>You could write a custom <code>autograd.Function</code> use the <code>linalg.svd</code> method in the <code>forward</code> and implement your custom <code>backward</code> function using <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">this tutorial</a>.</p>
How to change shape of matrix?

ADONAI_TZEVAOT: <p>I want to convert a 1x6 matrix into a 2x3 matrix B and then convert it back to 1x6 C. How do I do it?</p>

zilong: <pre><code class="lang-auto">B = A.view(2, 3)
C = B.view(1, 6)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">B = A.reshape(2, 3)
C = B.reshape(1, 6)
</code></pre>

ADONAI_TZEVAOT: <p>But how to do it in batches?</p>
Dynamic output of nn.Upsample causes InstanceNorm2d error in onnx

quqixun: <p>Environment:<br>
ubuntu 20.04, python 3.8, pytorch 1.10-cu113 via pip installing, numpy 1.21.4</p>
<p>Here is an example.</p>
<pre><code class="lang-python">import torch
import torch.nn as nn


class TestModel(nn.Module):

    def __init__(self, num_features, init_size=None):
        super(TestModel, self).__init__()

        if init_size is None:
            self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)
        else:
            self.upsample = nn.Upsample(size=init_size * 2, mode='bilinear', align_corners=True)

        self.innorm = nn.InstanceNorm2d(num_features, affine=False)

    def forward(self, x):
        out = self.upsample(x)
        out = self.innorm(out)
        return out


if __name__ == '__main__':

    init_size = 64
    num_features = 32
    x = torch.randn(4, num_features, init_size, init_size)

    model = TestModel(num_features, init_size=None)
    # model = TestModel(num_features, init_size=init_size)
    output = model(x)
    print(output.size())
    # torch.Size([4, 32, 128, 128])

    torch.onnx.export(
        model, x, 'test.onnx',
        export_params=True,
        verbose=True,
        opset_version=14,  # 9 ~ 14
        input_names=['x'],
        output_names=['output']
    )
</code></pre>
<p>Following infomation is obtained from <a href="https://github.com/pytorch/pytorch/blob/v1.10.0/torch/onnx/symbolic_opset9.py#L1390" rel="noopener nofollow ugc">instance_norm in onnx/symbolic_opset9.py</a> by <code>print(input)</code>, which means that the output of nn.Upsample in onnx is dynamic.</p>
<pre><code class="lang-auto">8 defined in (%8 : Float(*, *, *, *, strides=[524288, 16384, 128, 1], requires_grad=0, device=cpu) = onnx::Resize[coordinate_transformation_mode="align_corners", cubic_coeff_a=-0.75, mode="linear", nearest_mode="floor"](%input.1, %7, %6) # /home/quqixun/miniconda3/envs/pt10/lib/python3.8/site-packages/torch/nn/functional.py:3731:0)
</code></pre>
<p>This leads to the following error in <a href="https://github.com/pytorch/pytorch/blob/v1.10.0/torch/onnx/symbolic_opset9.py#L1390" rel="noopener nofollow ugc">instance_norm in onnx/symbolic_opset9.py</a> because <code>channel_size</code> is None while affine=False in nn.InstanceNorm2d.</p>
<pre><code class="lang-auto">RuntimeError: Unsupported: ONNX export of instance_norm for unknown channel size.
</code></pre>
<p>Questions:<br>
Is there any method to make above code work?</p>
<p>Thanks a lot.</p>
Force not to cache generated static tensor

zfzhang: <p>Hi,<br>
My model has a weight tensor <strong>W</strong>, which contains <em>duplicate entries</em> due to symmetry.<br>
I compacted the weight tensor as <strong>U</strong> which is 1/4 the size of <strong>W</strong>, and generate an <em>index</em> tensor <strong>B</strong> (same size as <strong>W</strong>) to deflate it back to <strong>W</strong> <em>on the fly</em>, as <code>W = U[B]</code>, note that <strong>B</strong>’s content is static/fixed.<br>
I was hoping that since <strong>B</strong> is generated on the fly, we can save some memory, but training in <em>Pytorch</em> actually uses more memory.<br>
What is the reason behind this? If index <strong>B</strong> is cache because it can be predetermined, can I force it to be always generated <em>on the fly</em>?<br>
Thank you very much!</p>

ptrblck: <p>I’m not sure to understand the use case correctly.<br>
Based on the code it seems you are creating the entire <code>W</code> tensor as well as the additional index tensor <code>B</code> and are also holding <code>U</code> in memory. I would guess you might save some memory before creating <code>B</code> and <code>W</code>, but would use more memory once the weight is created or where would the memory saving be coming from?</p>

zfzhang: <p>Hi,<br>
Let’s say a layer has input <strong>X</strong>, and produces output <strong>Y</strong>.<br>
In the original form, we need to store <em>at each layer</em> a weight matrix <strong>W</strong> of size <code>m*m</code>.<br>
In the compact form, we need to store <em>at each layer</em> a weight matrix <strong>U</strong> of size <code>(m/2)*(m/2)</code>, and on-the-fly generate a deflated version <strong>W = U[B]</strong>. We use <strong>W</strong> to generate the layer output <strong>Y</strong>, and <strong>W &amp; B</strong> can be discarded. Back propagation only need to retain <strong>U &amp; Y</strong> <em>at each layer</em> on the graph.<br>
Not sure if my understanding of autograd is correct, or is there a way to get this behavior?<br>
Thanks!</p>

ptrblck: <p>Thanks for the description as it fits my understanding. The peak memory would be increased, since you would have to store <code>U</code>, <code>B</code>, <code>W</code>, and <code>Y</code> at one point during the forward pass. Since Autograd needs to backpropagate through <code>Y</code>, the intermediates will be kept.<br>
If you have a way to delete <code>W</code> and recalculate it in the backward pass, you try to could implement a custom <code>autograd.Function</code> with this logic.</p>

zfzhang: <p>I came across <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="noopener nofollow ugc">torch.utils.checkpoint</a>, wondering if it would solve the problem.</p>
<hr>
<p>Edit: I confirm that checkpoint solves the problem, at the expense of more computational time.</p>
How to compute magnitude of gradient of each loss function?

torchimport: <p>Hi,</p>
<p>I have a question about computing the magnitude of gradient by each loss function.</p>
<p>For example, if I use two losses for one deep model,<br>
how can I compute the magnitude of gradient by loss1 and the magnitude of gradient by loss2 in model parameters?:</p>
<pre><code class="lang-auto">model = nn.Sequential(
    nn.Linear(1, 10),
    nn.Linear(10, 1)
)
...
output = model(input)
total_loss = loss1(output) + loss2(output)
total_loss.backward()
...
</code></pre>
<p>The meaning of the magnitude of gradient is below, and this is the same thing in the snippets of “torch.nn.utils.clip_grad_norm_”.</p>
<pre><code class="lang-auto">torch.norm(torch.stack([torch.norm(p.grad.detach(), 2.0) for p in list(model.parameters())]), 2.0)
</code></pre>

tom: <p>The short answer is that you cannot easily do this without doing the backward for each separately, i.e.</p>
<pre><code class="lang-auto">grads1 = torch.autograd.grad(loss1(output), model.parameters(), retain_graph=True)
grads2 = torch.autograd.grad(loss2(output), model.parameters())
</code></pre>
<p>This gives you two lists of gradients, ordered just like the parameters, now you can use these in your formula.</p>
<p>Best regards</p>
<p>Thomas</p>

torchimport: <p>Thank you for your quick and detailed solution!</p>
Does computational graph or the hooks break?

Asgard: <p>I’ve a case where I need to call the <code>forward </code> method like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    #This defines the structure of the NN.
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()  #Dropout
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2)) 
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)
      
    def sampler(self,x):
        print("*",end=" ",flush=True)
        y = self.forward(x)

        # Do something to y!
          ....
        return y


model = Net()
y = model.sampler(x)
loss = F.nll_loss()(y,target)
loss.backward()
</code></pre>
<p>Will the loss backpropagation perform normally without breaking any internal hooks? If it is breaking it, what’s the alternative without changing the model structure?</p>
<p>Thanks in advance.</p>
PyTorch CTC Loss implementation

ThomasGk: <p>Hi everyone. I am studying about CTC from this wonderful article <a href="https://distill.pub/2017/ctc/" rel="noopener nofollow ugc">Sequence Modeling with CTC </a>, and I’d like to ask something regarding PyTorch’s way of computing CTC loss.</p>
<p>In the article it says that to compute the loss, a typical way would be to sum the probability of all valid alignments. However, this might become too costly, as the valid alignments may be too many.</p>
<p>It then goes on to explain an alternative, less costly way to compute the loss, which approximates the above-mentioned calculation.</p>
<p>Reading PyTorch’s <a href="https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html#torch.nn.CTCLoss" rel="noopener nofollow ugc">documentation</a>, I think that it computes the loss using the first way, that is summing over <em>all</em> possible alignments. Is my understanding correct, and if yes, is there a way to use the second approach described in the article, which uses a dynamic programming algorithm?</p>
<p>Any help will be greatly appreciated, thanks</p>
Bounding box does not fit the object

hlopez: <p>I’m doing EDA on the <a href="https://paperswithcode.com/dataset/mju-waste" rel="noopener nofollow ugc">MJU-Waste dataset</a>, and I have found a problem in the visualization of the bounding box for each image. The bounding boxes are in COCO format.</p>
<p>I have tried reading the EXIF orientation tag and rotating the image but it didn’t work. The following image is an example:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce3ad714a2898f826b12654d40d46087b9c9ed65.png" alt="imagen_mal_box" data-base62-sha1="tqoqfr98LrO5Qb46DaRpqTeecaV" width="329" height="248"></p>
<p>What else can I try?</p>

tom: <p>For a reason I don’t quite understand and lazily trying a dozen examples or so, it appears as if you would have to multiply the bounding boxes by a factor 0.7 to get the right thing.<br>
Or you could take the polygon annotations and use those.</p>
<p>Note that you should probably link images to annotations via the annotation-&gt;image_id and image-&gt;id, not go by the indices.</p>
<p>Here is my silly snippet of code:</p>
<pre><code class="lang-python">import json
%matplotlib inline
from matplotlib import pyplot
import PIL.Image
import numpy

d = json.load(open('./train.json', 'rb'))
ann = {x['image_id']:x for x in d["annotations"]}


idx = 300


im = numpy.array(PIL.Image.open(d["images"][idx]['file_name']))
seg = numpy.array(ann[d["images"][idx]["id"]]['segmentation']).reshape(-1, 2)
bbox = numpy.array(ann[d["images"][idx]["id"]]['bbox'])

x, y, w, h = (numpy.array(bbox) * 0.7).astype(int)
im[y, x:x+w, 0] = 255
im[y+h, x:x+w, 0] = 255
im[y:y+h, x, 0] = 255
im[y:y+h, x+w, 0] = 255

pyplot.figure(figsize=(15, 10))
pyplot.imshow(im)
pyplot.scatter(seg[:, 0], seg[:, 1])
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

hlopez: <p>Thank you Thomas, multiplying the bounding boxes by a factor of 0.7 works really well. Although I think I will compute the bounding boxes from the polygons as you suggested, It seems to be a better solution in the long term.</p>
<p>Thanks!</p>
TensorAccessor operations

mrfox321: <h2>
<a name="tensoraccessor-apis-1" class="anchor" href="#tensoraccessor-apis-1"></a>TensorAccessor APIs</h2>
<p>If one uses <code>TensorAccessor&lt;T, N&gt;</code>, is there a way to leverage high level operations with the performance benefits of bypassing dynamic typing?</p>
<p>In other words, can we do things like</p>
<pre><code class="lang-auto">auto X_ = X.accessor&lt;float, 1&gt;();
auto Y_ = Y.accessor&lt;float, 1&gt;();
at::Tensor Z = torch::dot(X_, Y_);
</code></pre>
<p>Similar to how we work with <code>Tensor</code> types?</p>

tom: <p>No. Accessors are more or less “array-like” interfaces to the memory of strided tensors.</p>
<p>Best regards</p>
<p>Thomas</p>