Regularization to desired number

Frida: <p>Hi<br>
Does anyone know how we can enforce the sum of few parameters to be equal to a given value?</p>

KFrank: <p>Hi Frida!</p>
<aside class="quote no-group" data-username="Frida" data-post="1" data-topic="100881" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/57b2e6/48.png" class="avatar"> Frida:</div>
<blockquote>
<p>Does anyone know how we can enforce the sum of few parameters to be equal to a given value?</p>
</blockquote>
</aside>
<p>You can add an L1 or L2 penalty (regularizing term) to your loss<br>
function:</p>
<pre data-code-wrap="python"><code class="lang-python">l1_reg = torch.abs (param1 + param2 + param3 - desired_sum)
# or l2_reg = (param1 + param2 + param3 - desired_sum)**2
loss_total = loss + l1_reg
# or loss_total = loss + l2_reg
</code></pre>
<p>Then call <code>loss_total.backward()</code> and optimize as you normally would.</p>
<p>Best.</p>
<p>K. Frank</p>

Frida: <p>Any idea why it becomes lower and then becomes greater, although is greater then the loss before?</p>

KFrank: <p>Hi Frida!</p>
<aside class="quote no-group" data-username="Frida" data-post="3" data-topic="100881" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/57b2e6/48.png" class="avatar"> Frida:</div>
<blockquote>
<p>Any idea why it becomes lower and then becomes greater, although is greater then the loss before?</p>
</blockquote>
</aside>
<p>The short answer that I should have included a regularization<br>
“importance factor” as a hyperparameter:<br>
<code>loss_total = loss + reg_weight * l1_reg</code></p>
<p>My apologies for leaving out this important detail.</p>
<p>I am assuming that the “it” that “becomes lower and then becomes<br>
greater” is <code>l1_reg</code> (or <code>l2_reg</code>, if that’s the one you’re using).</p>
<p>First, some context:</p>
<p>If you use just <code>loss</code> for training, training will push your model<br>
parameters to make <code>loss</code> smaller (presumably making better<br>
predictions).  But the training won’t care what your parameter<br>
sum is.</p>
<p>Similarly, if you use just <code>l1_reg</code> for training, you will push your<br>
parameter sum to its desired value.  But the training won’t care<br>
about <code>loss</code>, and your model won’t learn anything about making<br>
good predictions.</p>
<p>These two goals – “good predictions” and “desired sum” – can<br>
potentially be in competition with one another.  <code>reg_weight</code> tunes<br>
this trade-off.</p>
<p>Add <code>reg_weight</code> to your training, and try increasing it until you get<br>
adequately close to <code>desired_sum</code>.</p>
<p>(It is sometimes helpful to start with a smallish value of <code>reg_weight</code><br>
and increase it gradually as training progresses.  But there’s no real<br>
wisdom about whether or not this will be helpful – like so much in<br>
neural networks, sometimes you just have to try it.)</p>
<p>Best.</p>
<p>K. Frank</p>