Pytorch, loss.backward() has no effect

username: Jourdelune
cooked: <p>Hello, I train a transformer model to generate a latent space and then decode this latent space to produce raw audio, which I use to compute my loss. I don’t want to train the decoder model itself (it works well and I don’t have a lot of RAM), so I disabled its gradients by using:</p>
<pre data-code-wrap="py"><code class="lang-py">for param in self.autoencoder.parameters():
            param.requires_grad = False
self.autoencoder.eval()
</code></pre>
<p>in the dataset class.</p>
<p>Then, I compute the loss of the decoded audio against the base waveform using MSE loss. The loss gives me a number, and then I call the <code>backward()</code> function.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/287d747ebade3852c6ee1f714b3a411226b42253.png" data-download-href="https://discuss.pytorch.org/uploads/default/287d747ebade3852c6ee1f714b3a411226b42253" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/287d747ebade3852c6ee1f714b3a411226b42253_2_690x469.png" alt="image" data-base62-sha1="5MbSS6dtgja82RZxO8NubVDYrYL" width="690" height="469" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/287d747ebade3852c6ee1f714b3a411226b42253_2_690x469.png, https://discuss.pytorch.org/uploads/default/original/3X/2/8/287d747ebade3852c6ee1f714b3a411226b42253.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/8/287d747ebade3852c6ee1f714b3a411226b42253.png 2x" data-dominant-color="242525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">750×510 53.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The problem is that my model doesn’t progress at all. I don’t think the <code>loss_total.backward()</code> has any effect because the loss doesn’t change.</p>
<pre><code class="lang-auto">EPOCH 0.0 (0.01%) - STEP 10: Loss 0.0229 LR 9.999e-05
EPOCH 0.0 (0.03%) - STEP 20: Loss 0.0344 LR 9.997e-05
EPOCH 0.0 (0.04%) - STEP 30: Loss 0.0252 LR 9.996e-05
EPOCH 0.0 (0.05%) - STEP 40: Loss 0.0149 LR 9.995e-05
EPOCH 0.0 (0.07%) - STEP 50: Loss 0.0325 LR 9.993e-05
EPOCH 0.0 (0.08%) - STEP 60: Loss 0.0124 LR 9.992e-05
EPOCH 0.0 (0.09%) - STEP 70: Loss 0.0191 LR 9.991e-05
EPOCH 0.0 (0.11%) - STEP 80: Loss 0.029 LR 9.989e-05
EPOCH 0.0 (0.12%) - STEP 90: Loss 0.0179 LR 9.988e-05
EPOCH 0.0 (0.13%) - STEP 100: Loss 0.0289 LR 9.987e-05
EPOCH 0.0 (0.15%) - STEP 110: Loss 0.0381 LR 9.985e-05
EPOCH 0.0 (0.16%) - STEP 120: Loss 0.0263 LR 9.984e-05
EPOCH 0.0 (0.17%) - STEP 130: Loss 0.0234 LR 9.983e-05
</code></pre>
<p>If I change the loss to compute the MSE loss on <code>y_hat</code> (the latent space) and <code>y</code> (the target latent space), the model works. Do you have any idea why the loss on the decoded audio doesn’t have any effect?</p>
<p>The decoder is just a set of strides convolution block (from the DAC model)</p><aside class="onebox githubblob" data-onebox-src="https://github.com/descriptinc/descript-audio-codec/blob/main/dac/model/dac.py#L115">
  <header class="source">

      <a href="https://github.com/descriptinc/descript-audio-codec/blob/main/dac/model/dac.py#L115" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/descriptinc/descript-audio-codec/blob/main/dac/model/dac.py#L115" target="_blank" rel="noopener nofollow ugc">descriptinc/descript-audio-codec/blob/main/dac/model/dac.py#L115</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="105" style="counter-reset: li-counter 104 ;">
          <li>            ),</li>
          <li>            ResidualUnit(output_dim, dilation=1),</li>
          <li>            ResidualUnit(output_dim, dilation=3),</li>
          <li>            ResidualUnit(output_dim, dilation=9),</li>
          <li>        )</li>
          <li></li>
          <li>    def forward(self, x):</li>
          <li>        return self.block(x)</li>
          <li></li>
          <li></li>
          <li class="selected">class Decoder(nn.Module):</li>
          <li>    def __init__(</li>
          <li>        self,</li>
          <li>        input_channel,</li>
          <li>        channels,</li>
          <li>        rates,</li>
          <li>        d_out: int = 1,</li>
          <li>    ):</li>
          <li>        super().__init__()</li>
          <li></li>
          <li>        # Add first conv layer</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Jourdelune
cooked: <p>If you want the chain rule: <a href="https://jmp.sh/s/nOZLHWAO9Ep1BsCouLM3" class="inline-onebox" rel="noopener nofollow ugc">torchviz-sample</a></p>

username: Jourdelune
cooked: <p>Finally it works, it’s just the training that was kinda random<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25963f2f296c618706a180ec3bd13d6a34e251ac.png" data-download-href="https://discuss.pytorch.org/uploads/default/25963f2f296c618706a180ec3bd13d6a34e251ac" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25963f2f296c618706a180ec3bd13d6a34e251ac_2_611x500.png" alt="image" data-base62-sha1="5mvzg9PcVjVJIuBJW4NjCys1h3S" width="611" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25963f2f296c618706a180ec3bd13d6a34e251ac_2_611x500.png, https://discuss.pytorch.org/uploads/default/original/3X/2/5/25963f2f296c618706a180ec3bd13d6a34e251ac.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/5/25963f2f296c618706a180ec3bd13d6a34e251ac.png 2x" data-dominant-color="423637"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">865×707 96.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>