Proper way to constraint the final linear layer to have non negative weights?

username: Richard_S
cooked: <p>I am trying to constraint the final layer of my NN to have non negative weights in the final layer, for my binary classification task ( the reason for me wanting to have non negative weights does not matter right now)</p>
<p>This is basically what my code looks like :</p>
<pre><code class="lang-auto">class Classifier(nn.Module):
    def __init__(self, in_dim, hidden_dim1 , hidden_dim2 , hidden_dim3 , n_classes):
        super(Classifier, self).__init__()

        # other layers

        self.classify = nn.Linear(hidden_dim3 , n_classes)

    def forward(self, g , h ):
 
        # other layers

        hg = self.classify(h)
        self.classify.weight.data = self.classify.weight.data.clamp(min=0)

        hg = torch.sigmoid ( hg )

        return  hg
</code></pre>
<p>so am i doing this right? is this proper way of forcing the final layer to only have positive weights and therefore only looks for “positive” features to do classification ?</p>
<p>wouldn’t there be problems because sigmoid with only positive input only outputs +50% probabilities? the bias should fix this problem, right?</p>
<p>Note that keras has</p>
<pre><code class="lang-auto">tf.keras.constraints.NonNeg()
</code></pre>
<p>which does the same thing and i am trying to do that in pytorch.</p>

username: KFrank
cooked: <p>Hi Richard!</p>
<p>It looks like this is related to your previous post:</p>
<aside class="quote" data-post="1" data-topic="103495">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/3be4f8/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-use-sigmoid-on-only-positive-values/103495">How to use sigmoid on only positive values?</a> 
  </div>
  <blockquote>
    I have a binary classification model, that in the latest linear layer, it outputs only positive values (don’t ask why, that’s a different matter), now when i give the final layer’s output to torch.sigmoid, all the results are above 50%, because the final linear layer is only outputting positive values, how can i fix this and output probability? is there any “positive only” sigmoid in pytorch?
  </blockquote>
</aside>

<aside class="quote no-group" data-username="Richard_S" data-post="1" data-topic="103498" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/3be4f8/48.png" class="avatar"> Richard_S:</div>
<blockquote>
<p>I am trying to constraint the final layer of my NN to have non negative weights in the final layer, for my binary classification task ( the reason for me wanting to have non negative weights does not matter right now)</p>
</blockquote>
</aside>
<p>My guess is that you’re not going about your problem in a sensible way.<br>
But unless you describe your actual use case, it’s hard to know.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        self.classify.weight.data = self.classify.weight.data.clamp(min=0)
</code></pre>
<p>is this proper way of forcing the final layer to only have positive weights</p>
</blockquote>
</aside>
<p><code>.data</code> is deprecated, and the forum experts will threaten you with<br>
the specter of computation-graph gremlins if you use it.</p>
<p>If you really want to do this, something like:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    self.classify.weight.copy_ (self.classify.weight.data.clamp(min=0))

</code></pre>
<p>might be better.</p>
<aside class="quote no-group">
<blockquote>
<p>and therefore only looks for “positive” features to do classification ?</p>
</blockquote>
</aside>
<p>Is this the key to what you are trying to do?  What would it mean to<br>
build a classifier that “only looks for ‘positive’ features?”</p>
<aside class="quote no-group">
<blockquote>
<p>wouldn’t there be problems because sigmoid with only positive input only outputs +50% probabilities?</p>
</blockquote>
</aside>
<p>Well, if you only look for positive features, I suppose that it would<br>
be natural to only find positive features, and therefore only output<br>
probabilities greater than 50%.</p>
<p>But yes, this does seem problematic to me.</p>
<aside class="quote no-group">
<blockquote>
<p>the bias should fix this problem, right?</p>
</blockquote>
</aside>
<p>I doubt anything I say where will be relevant to your actual problem,<br>
but here are some observations:</p>
<p>Just because your last layer has only positive <em>weights</em> (as distinct<br>
from biases) doesn’t mean that your output values (even if you had<br>
no biases) would be positive.  <em>Negative</em> inputs to a positive-weight<br>
layer will produce negative outputs.</p>
<p>But yes, a negative bias could flip the value of what would have been<br>
a positive output to negative.</p>
<p>However, as it stands, I don’t see how your optimizer would know that<br>
you want positive weights.  So even if your biases could “fix” the<br>
problem, your optimizer could well just take steps that leave your<br>
biases alone, and make your weights negative (even though you<br>
then force them by hand to be positive).</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Richard_S
cooked: <p>I’m trying to implement the idea of this paper for a more robust model :</p>
<aside class="onebox pdf">
  <header class="source">
      <a href="https://arxiv.org/pdf/1806.06108.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>
  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1806.06108.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://arxiv.org/pdf/1806.06108.pdf" target="_blank" rel="noopener nofollow ugc">1806.06108.pdf</a></h3>

<p class="filesize">375.14 KB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>so the “attacker” adding more benign features will not evade the model, the core idea is that by forcing the weights to be positive only, adding benign features will not affect the output, because only features that drive the model towards positive will affect the output, pretty simple but effective.</p>
<p>they implemented this in keras using</p>
<pre><code class="lang-auto">tf.keras.constraints.NonNeg()
</code></pre>
<p>So what is the most optimal way of implementing this in a multi layer NN in pytorch? (for binary classification), meaning how should i force the weights to be 0, what activation function and optimization parameters should i use?</p>

username: KFrank
cooked: <p>Hi Richard!</p>
<aside class="quote no-group" data-username="Richard_S" data-post="3" data-topic="103498" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/3be4f8/48.png" class="avatar"> Richard_S:</div>
<blockquote>
<p>they implemented this in keras using</p>
<pre><code class="lang-auto">tf.keras.constraints.NonNeg()
</code></pre>
<p>So what is the most optimal way of implementing this in a multi layer NN in pytorch?</p>
</blockquote>
</aside>
<p>According to the keras documentation, <a href="https://keras.io/api/layers/constraints/" rel="noopener nofollow ugc">Layer weight constraints</a>:</p>
<p>“They are per-variable projection functions applied to the target<br>
variable after each gradient update.”</p>
<p>So following along with what keras claims it does, you could try:</p>
<pre data-code-wrap="python"><code class="lang-python">optimizer.step()
with torch.no_grad():
    self.classify.weight.copy_ (self.classify.weight.data.clamp(min=0))
</code></pre>
<p>to force the constraint after each optimizer step.</p>
<p>You would then hope that the training process causes the optimizer<br>
to move the last layer’s <code>bias</code> into negative territory so that you get<br>
predicted logits centered around zero, and therefore make “negative”<br>
as well as “positive” predictions.</p>
<p>Such an approach would seem to be training the model with one hand<br>
tied behind its back, but, in principle, it ought to be able to train the<br>
biases to become negative.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Richard_S
cooked: <p>Thank you for the answer, any suggestion on which optimizer &amp; loss function to choose? does it really matter in this case? i am currently using :</p>
<pre><code>    criterion = nn.BCELoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001)</code></pre>

username: KFrank
cooked: <p>Hi Richard!</p>
<aside class="quote no-group" data-username="Richard_S" data-post="5" data-topic="103498" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/3be4f8/48.png" class="avatar"> Richard_S:</div>
<blockquote>
<pre><code>    criterion = nn.BCELoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001)
</code></pre>
</blockquote>
</aside>
<p>From your previous posts it appears that you have a multi-label,<br>
multi-class problem.  For this, <code>BCELoss</code> is reasonable.  For<br>
improved numerical stability, however, you should prefer using<br>
<code>BCEWithLogitsLoss</code> and remove the <code>sigmoid()</code> call from your<br>
<code>forward()</code> function.</p>
<p>Would it be practical for you to experiment with different optimizers,<br>
or is your model too slow and expensive to play around with?</p>
<p>I would recommend starting with plain-vanilla <code>SGD</code>, if only to get a<br>
baseline for comparison.</p>
<p>Both <code>SGD</code> with momentum and <code>Adam</code> “remember” things from<br>
previous steps, so the fact that you will be munging your weights<br>
after the optimization step might confuse this process.  It’s not that<br>
you shouldn’t try <code>SGD</code> with momentum or <code>Adam</code> – they might work<br>
fine – but be on the lookout for potential issues and use plain-vanilla<br>
<code>SGD</code> as a sanity check.</p>
<p>In general, weight decay is worthwhile, so you should probably turn<br>
it on (but run a baseline without it).</p>
<p>Your <code>BCEWithLogitsLoss</code> (or <code>BCELoss</code>) loss function should cause<br>
your model to train to make some “negative” predictions (assuming<br>
that your training data is sensible), but your weight-munging scheme<br>
is, at best, going to make your training process more difficult.  So you<br>
might have to carry out training runs that are longer than normal.<br>
(Also, experiment with your learning rate.)</p>
<p>Lastly, if your training data is *unbalanced," that is, some given class<br>
has many more “negative” samples than “positive,” you should also<br>
consider using <code>BCEWithLogitsLoss</code>’s <code>pos_weight</code> argument to<br>
account for this.</p>
<p>Good luck.</p>
<p>K. Frank</p>