How to optimize the weights and bias in your neural network in order to get as results only positive values?

Benja: <p>Hi!. I am using a neural network with the following layers: (784,784),(784,784),(784,2). I want the values in last layer to be positives, but I do not want to use any activation function such as ReLU that would solve the problem. What I want to know if there is a way to guide the weights and bias (in pytorch), in that last layer, in order to get only positive values while using nn.Linear()?. Any advice would be grateful. Thanks in advance!!</p>

KFrank: <p>Hi Benja!</p>
<aside class="quote no-group" data-username="Benja" data-post="1" data-topic="178758" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/benja/48/46127_2.png" class="avatar"> Benja:</div>
<blockquote>
<p>I want the values in last layer to be positives</p>
</blockquote>
</aside>
<p>I assume you mean that you want the outputs of your network to be positive,<br>
rather than the weight and bias parameters of your network to be positive.<br>
(If you want your parameters to be positive, similar comments will apply, but<br>
the details will be different.)</p>
<p>One simple approach would be to add a penalty to your loss function (where<br>
<code>pred</code> is the output of your network):</p>
<pre data-code-wrap="python"><code class="lang-python">loss = loss_fn (pred, target)
penalty = (torch.nn.functional.relu (-pred)**2).sum()   # for example
loss_with_penalty = loss + alpha * penalty
</code></pre>
<p>Note that <code>penalty</code> will not force the elements of <code>pred</code> to be positive, but<br>
it will encourage them to be positive.  However, by increasing the value<br>
of the penalty-weight, <code>alpha</code>, you can push <code>pred</code> harder and harder not<br>
to be negative.</p>
<p>To perform an “official” constrained optimization that requires <code>pred</code> to be<br>
non-negative (an <em>inequality constraint</em>), you can add <a href="https://en.wikipedia.org/wiki/Slack_variable" rel="noopener nofollow ugc">slack variables</a> for the<br>
elements of <code>pred</code>:</p>
<pre><code class="lang-auto">pred_with_slack = pred - slack**2
</code></pre>
<p>and constrain (with an <em>equality constraint</em>) <code>pred_with_slack</code> to be zero<br>
element by element (where <code>slack</code> has the same shape as <code>pred</code>).  Because<br>
<code>slack**2</code> can be positive, <code>pred</code> is free to become positive, but because<br>
<code>slack**2</code> can never be negative, the constraint on <code>pred_with_slack</code><br>
prevents <code>pred</code> from becoming negative.</p>
<p>You can use <a href="https://en.wikipedia.org/wiki/Lagrange_multiplier" rel="noopener nofollow ugc">Lagrange multipliers</a> to perform such an optimization where<br>
<code>pred_with_slack</code> is constrained to be zero.  However, because the optimum<br>
for the Lagrange-multiplier optimization occurs at a saddle point (rather than<br>
at a minimum), you can’t use gradient descent to perform the optimization<br>
without tweaking it so that you use, in effect, gradient <em>ascent</em> on the Lagrange<br>
multiplier.</p>
<p>This is explained by <a class="mention" href="/u/tom">@tom</a>, <a href="https://discuss.pytorch.org/t/how-to-implement-lagrange-multiplier-with-pytorch/89011/9">here,</a> and a sound approach to implementing<br>
such mixed gradient descent / ascent optimization using pytorch (with its<br>
gradient-descent-based optimizers) is give by <a class="mention" href="/u/t_naumenko">@t_naumenko</a>, <a href="https://discuss.pytorch.org/t/gradient-ascent-and-gradient-modification-modifying-optimizer-instead-of-grad-weight/62777/6">here.</a></p>
<p>Note that using the Lagrange-multiplier technique will not force <code>pred</code> to be<br>
non-negative <em>during</em> the training process; only after the optimization has<br>
converged to its (saddle-point) optimum will <code>pred</code> be non-negative.</p>
<p>(You can think of the Lagrange multiplier as being being like the <code>alpha</code><br>
penalty-weight in the <code>loss_with_penalty</code> optimization approach except<br>
that the optimization process tunes the penalty-weight automatically.)</p>
<aside class="quote no-group">
<blockquote>
<p>but I do not want to use any activation function such as ReLU that would solve the problem.</p>
</blockquote>
</aside>
<p>As an aside, this naturally raises the question why not?  Depending on your use<br>
case, it may make perfect sense to pass the output of your network through<br>
something like <code>relu()</code> or, perhaps better, <code>exp()</code> to ensure positive values.</p>
<p>Good luck!</p>
<p>K. Frank</p>

Benja: <p>Okey, thank you!! I think it works</p>