akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
_convolution_double_backward() parameters

username: tomguluson92
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, thanks for your explanation about the definition of  <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp" rel="nofollow noopener">_convolution_double_backward()</a>. When reading this code, I was confused about the parameter like <code>ggI</code>, <code>ggW_r</code>, <code>ggb</code> and <code>gO_r</code> and <code>output_mask</code> and the target of calculating <code>ggW</code>, <code>gI</code> and <code>gW</code>.</p>
<p>Under my superficial understanding of all these, <code>ggW_r</code> means the gradient of convolution kernel? <code>xx_r</code> means a requirement, <code>ggb</code> is the gradient of bias.</p>
<p>Really hope your advice on how to realize these parameters and the calculation of <code>ggW</code>, <code>gI</code> and <code>gW</code>.</p>
<p>Thank you very much!</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m not sure about the <code>*_r</code> variables. Most likely some some cpp reasons.<br>
The <code>gg*</code> variables contain the gradient of the gradient of something.<br>
Since the forward is I, W, b -&gt; O<br>
the backward is gO, I, W -&gt; gI, gW, gb<br>
and the double backward ggI, ggW, ggb -&gt; ggO, gI, gW</p>

username: tomguluson92
cooked: <p>very appreciate to your answer, thank you !</p>
(Mat)Mul layers don't convert well to ONNX models

username: Aeroxander
cooked: <p>So I have 3 PyTorch models where 2 of them give me trouble when loading in ONNX, the inference works fine in PyTorch and the conversion is all ok too.</p>
<p>So in the first model I get this error when reading the ONNX model in OpenCV:</p>
<blockquote>
<p>Assertion failed (inputs.size()) in cv::dnn::dnn4_v20190122::Layer::getMemoryShapese</p>
</blockquote>
<p>With the other model I get this error:</p>
<blockquote>
<p>Blob 64 not found in const blobs</p>
</blockquote>
<p>This has all something to do with the input where in the first error the inputs.size() is 0 and in the second model the blob isn’t big enough somehow.</p>
<p>I just don’t know where the error lies, as the PyTorch model alone is valid, I suppose it’s either in the PyTorch → ONNX model conversion or in OpenCV itself.</p>
<p>Here is the project including the first onnx model file <a href="https://github.com/Aeroxander/decodererror/" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Aeroxander/decodererror</a></p>
Negative indexing of a 1D tensor: is this something to do with ignore_index in NLLLoss?

username: sonsus
cooked: <p>Days ago I tried to use <code>LabelSmoothingLoss</code> from the openNMT and something I have question about its implementation.</p>
<p>in its implementation it has <code>one_hot[self.ignore_index] = 0  </code> and the index is default <code>self.ignore_index = -100 </code>. I don’t get what does it mean nor whether it is related with <code>nn.NCELoss</code> 's <code>ignore_index</code>.</p>
<p>Could somebody kindly explain what it does? I tried to follow source codes and still not sure what it’s doing.</p>
<pre><code class="lang-auto">class LabelSmoothingLoss(nn.Module):
    """
    With label smoothing,
    KL-divergence between q_{smoothed ground truth prob.}(w)
    and p_{prob. computed by model}(w) is minimized.
    """
    def __init__(self, label_smoothing, tgt_vocab_size, ignore_index=-100):
        assert 0.0 &lt; label_smoothing &lt;= 1.0
        self.ignore_index = ignore_index
        super(LabelSmoothingLoss, self).__init__()

        smoothing_value = label_smoothing / (tgt_vocab_size - 2)
        one_hot = torch.full((tgt_vocab_size,), smoothing_value)
        one_hot[self.ignore_index] = 0 # here is the thing. 
        self.register_buffer('one_hot', one_hot.unsqueeze(0))

        self.confidence = 1.0 - label_smoothing

  
</code></pre>
<p><a href="http://opennmt.net/OpenNMT-py/_modules/onmt/utils/loss.html#LossComputeBase" rel="nofollow noopener">you can see openNMT_LabelSmoothingLoss source code here</a></p>
How are the intermediate results freed during the backward() call?

username: Sanyuan
cooked: <p>Hi, I am trying to calculate the maximum memory usage during the backward pass. I understand the intermediate results can be freed during the backward() call, but I am wondering the intermediate feature maps and their gradients are freed right after the backward pass or as soon as they are not needed anymore?</p>
<p>For example, if I have a simple 3-layer cnn, have the intermediate result of the second layer and its gradient be freed when computing the gradient of the first layer?</p>
<p>Thanks!</p>

username: jih332
cooked: <p>sorry for bringing up this old question again, but I’ve been wondering over the same problem for a while.<br>
I understand that once the backProp is done on a module, the gradients of non-leaf parameters will be freed (by checking <code>.grad</code> attribute). But what about intermediate outputs in forwardProp? Will they be freed at the same time?<br>
Thanks.</p>