Regularisation of dense output layer does not seem to work as expected

dunefox: <p>I have a neural network that I pretrain on Dataset A and then finetune on Dataset B - before finetuning I add a dense layer on top of the model (red arrow) that I would like to regularise. However, it does not seem to work properly: either the performance drops very low even with tiny regularisation weights (0.01 - 0.08 weight range, f1 drops from around 22% to 12% on the dev set) <em>or I get the exact same F1 score for all weights (as if I train the same model several times with no changes).</em> ← this does not occur anymore</p>
<p>(Since new users can only post one image for whatever reason, I have to resort to a screenshot here…)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bd5f541e821339a544b4fd88d899bde088efb0d.png" data-download-href="https://discuss.pytorch.org/uploads/default/2bd5f541e821339a544b4fd88d899bde088efb0d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bd5f541e821339a544b4fd88d899bde088efb0d_2_479x500.png" alt="image" data-base62-sha1="6fMVV6dN4GF3tHzH8iFVvbx280d" width="479" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bd5f541e821339a544b4fd88d899bde088efb0d_2_479x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bd5f541e821339a544b4fd88d899bde088efb0d_2_718x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bd5f541e821339a544b4fd88d899bde088efb0d.png 2x" data-dominant-color="F8F9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">777×811 143 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Reg. layer initialisation:<br>
<code>self.reg_matrix =</code> <code>torch.cat((torch.eye(42), torch.zeros(42, 200)), dim=1).cuda()</code></p>
<p>Reg. term (adapt is the dense layer <strong>W</strong>):<br>
<code>squared_sum = torch.sub(self.model.adapt.weight, self.reg_matrix).pow(2).sum()</code></p>
<p>Loss:<br>
<code>loss += self.weight * squared_sum</code></p>
<p>I originally assumed that the tape wasn’t recording all operations properly since I hadn’t used PyTorch operations for the calculation of the L2 reg. term, but even when I do it doesn’t seem to work.</p>
<p>This is the function I’m adapting, so the previous code is around line 42 here (loss function is unchanged):</p><aside class="onebox githubblob" data-onebox-src="https://github.com/yuhaozhang/tacred-relation/blob/master/model/rnn.py#L39">
  <header class="source">

      <a href="https://github.com/yuhaozhang/tacred-relation/blob/master/model/rnn.py#L39" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/yuhaozhang/tacred-relation/blob/master/model/rnn.py#L39" target="_blank" rel="noopener nofollow ugc">yuhaozhang/tacred-relation/blob/master/model/rnn.py#L39</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="29" style="counter-reset: li-counter 28 ;">
          <li>        inputs = [b.cuda() for b in batch[:7]]</li>
          <li>        labels = batch[7].cuda()</li>
          <li>    else:</li>
          <li>        inputs = [b for b in batch[:7]]</li>
          <li>        labels = batch[7]</li>
          <li></li>
          <li>    # step forward</li>
          <li>    self.model.train()</li>
          <li>    self.optimizer.zero_grad()</li>
          <li>    logits, _ = self.model(inputs)</li>
          <li class="selected">    loss = self.criterion(logits, labels)</li>
          <li>    </li>
          <li>    # backward</li>
          <li>    loss.backward()</li>
          <li>    torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.opt['max_grad_norm'])</li>
          <li>    self.optimizer.step()</li>
          <li>    loss_val = loss.data.item()</li>
          <li>    return loss_val</li>
          <li></li>
          <li>def predict(self, batch, unsort=True):</li>
          <li>    """ Run forward prediction. If unsort is True, recover the original order of the batch. """</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Results (F1 (micro)):</p>
<pre><code class="lang-auto">0.0 =&gt; 22.361%
0.01 =&gt; 13.309%
0.05 =&gt; 11.350%
1.0 =&gt; 12.084%
</code></pre>
<p>F1 scores from evaluation would be something like 42% with 0.0 and around 17% to 13% or so with regularisation. The results of the original model without pretraining is 65%.</p>
<p>Any ideas would be appreciated.</p>

ptrblck: <p>Could you explain how the regularization term is working?<br>
Currently it seems you are subtracting a matrix of the shape <code>[42, 242]</code> with 42 ones in the first diagonal from the weight before squaring and summing it.<br>
Usually you would add the weight norm to the loss and I’m unfamiliar with your approach.<br>
Did you try to use the standard weight decay?</p>

dunefox: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="82380" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Currently it seems you are subtracting a matrix of the shape <code>[42, 242]</code> with 42 ones in the first diagonal from the weight before squaring and summing it.</p>
</blockquote>
</aside>
<p>That is exactly the idea, yes. This regularisation is supposed to keep the weights of the new dense layer close to the identity matrix (in the regularisation matrix) so that with increasing reg. weight the output of the previous dense layer is not transformed much. I realise now that I could have explained the core idea better.<br>
For this you can just assume that the regularisation is correct. How do I regularise with a custom regularisation scheme like this?</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="82380" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Did you try to use the standard weight decay?</p>
</blockquote>
</aside>
<p>That doesn’t seem to offer me a way to incorporate the regularisation matrix if I understand it correctly.</p>
<p>More detail:</p>
<ol>
<li>I pretrain the model with data A: [emb → LSTM → attention → dense → softmax]</li>
<li>Afterwards I freeze all the ‘old’ layers and add a dense layer after the original dense (output) layer, so now it is [emb → LSTM → attention → dense → <strong>dense</strong> → softmax], the new dense layer has the dimensions of the original output dense layer and the LSTM layer combined: so dense1(42, 42) + lstm(42, 200) = dense2(42, 242)</li>
<li>Now, while finetuning the new dense layer receives the output of the original output dense layer and the output of the lstm, so: <strong>dense2([dense1();lstm()])</strong> → softmax</li>
<li>The weights of this new dense layer are now supposed to be regularised through the formula that I posted (controlled by the reg. weight)</li>
<li>For this to work I need to add this term to the loss (or at least that is what I have done so far, is there another way in PyTorch?):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a749a0119e156dea8bfd00a465306aaf31883ca.png" alt="image" data-base62-sha1="aCF9kH1BpCqDZZHX0JtmBLZKmnw" width="329" height="51"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74574116e5f915e4eafefb9aaf8bd7d779dd2459.png" data-download-href="https://discuss.pytorch.org/uploads/default/74574116e5f915e4eafefb9aaf8bd7d779dd2459" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74574116e5f915e4eafefb9aaf8bd7d779dd2459_2_265x250.png" alt="image" data-base62-sha1="gBckkEiEYSYaynwoscKXTNxLiOZ" width="265" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74574116e5f915e4eafefb9aaf8bd7d779dd2459_2_265x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74574116e5f915e4eafefb9aaf8bd7d779dd2459_2_397x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74574116e5f915e4eafefb9aaf8bd7d779dd2459_2_530x500.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">551×518 14.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b40eadf68cc1c52452a29e44a2553313a356224.png" alt="image" data-base62-sha1="6aDCcWvfKph1mjKIAt59Lw9Oxus" width="143" height="45"></li>
</ol>

ptrblck: <aside class="quote no-group" data-username="dunefox" data-post="3" data-topic="82380">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dunefox/48/24578_2.png" class="avatar"> dunefox:</div>
<blockquote>
<p>For this you can just assume that the regularisation is correct.</p>
</blockquote>
</aside>
<p>I’m still unsure if your approach is right.<br>
Note that you are subtracting a constant tensor with 1s at some positions from <code>self.model.adapt.weight</code>, while it seems your approach would rather subtract the values of <code>self.weight</code> at these positions from <code>self.model.adapt.weight</code>?</p>
<aside class="quote no-group" data-username="dunefox" data-post="3" data-topic="82380">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dunefox/48/24578_2.png" class="avatar"> dunefox:</div>
<blockquote>
<p>How do I regularise with a custom regularisation scheme like this?</p>
</blockquote>
</aside>
<p>If you are convinced that the formula is right and I’m misunderstanding the use case, then the approach should work. You could just add the regularization loss to the final loss and call <code>backward</code> on it.</p>

dunefox: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="82380" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>If you are convinced that the formula is right and I’m misunderstanding the use case, then the approach should work.</p>
</blockquote>
</aside>
<p>It is, I just explained it badly. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="82380" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You could just add the regularization loss to the final loss and call <code>backward</code> on it.</p>
</blockquote>
</aside>
<p>That’s what I’m doing; at least now I know that I’m not misunderstanding PyTorch itself in this case.</p>
<p>Thank you!</p>