Predict Multiple Binary feature

TeDataPro: <p>Hey <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I would like to add manualy an output layer to BERT in order to predict multiple features which are binary.</p>
<p>For example, these outputs would answer the questions:</p>
<ul>
<li>Is the text positive? 1 if yes, 0 otherwise.</li>
<li>Is this text about sports? 1 if yes, 0 if not.</li>
<li>Is this text about business? 1 if yes, 0 if not.</li>
</ul>
<p>My first idea was to do it as a regression task, adding an output layer with 3 neurons: one for each question.</p>
<p>The pseudo-code:</p>
<pre><code class="lang-auto">def __init__(self):
      self.bert = CamembertModel.from_pretrained('camembert-base')
      self.regressor = nn.Sequential(nn.Linear(dim_in, 3))

def forward(self, input):
      outputs = self.bert(input) # "Bert Layers"
      outputs = self.regressor(outputs) # LinearSequential with output of 3
      return outputs
</code></pre>
<p>But I’ll get values above 1 and below 0. So, which kind of layer could I add in order to get probability (values between 0 and 1) to deal with this problem ?</p>
<p>I hope my question was clear, thank you for your help.</p>

Matias_Vasquez: <p>You can add a <a href="https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html" rel="noopener nofollow ugc">softmax</a> layer after your regressor.</p>
<pre><code class="lang-python">m = nn.Softmax(dim=1)
input = torch.randn(4, 3)
output = m(input)

# Output: 
#tensor([[0.7552, 0.0566, 0.1882],   --&gt; adds up to 1
#        [0.0267, 0.6366, 0.3367],
#        [0.0937, 0.8218, 0.0845],
#        [0.1545, 0.6051, 0.2404]])
</code></pre>

Andrei_Cristea: <p>Softmax is good if the probabilities sum to 1. In your example though perhaps they don’t - a text could be both about business and positive. You can try a nn.Sigmoid() layer, whose output is always in the interval [0,1].</p>

Matias_Vasquez: <p>You are right. Thanks for the correction <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

KFrank: <p>Hi Te!</p>
<aside class="quote no-group" data-username="TeDataPro" data-post="1" data-topic="147828" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tedatapro/48/48115_2.png" class="avatar"> TeDataPro:</div>
<blockquote>
<p>predict multiple features which are binary.</p>
<p>For example, these outputs would answer the questions:</p>
<ul>
<li>Is the text positive? 1 if yes, 0 otherwise.</li>
<li>Is this text about sports? 1 if yes, 0 if not.</li>
<li>Is this text about business? 1 if yes, 0 if not.</li>
</ul>
</blockquote>
</aside>
<p>As you describe it, you are performing a <em>multi-label, multi-class</em><br>
classification.  It is multi-class because you have three classes:<br>
“positive,” “sports,” and “business.”  It is multi-label because for<br>
any given sample, none, some, or all of the classes can be active,<br>
so each sample can be labelled with multiple classes at the same<br>
time.</p>
<p>As you have recognized, this is, in essence, three binary classification<br>
problems that are run through your network at the same time.</p>
<p>The most typical approach for such a problem is to have your final<br>
layer be a <code>Linear</code> with <code>out_features = 3</code> (your number of classes)<br>
and to use <code>BCEWithLogitsLoss</code> as the loss criterion.</p>
<aside class="quote no-group">
<blockquote>
<p>But I’ll get values above 1 and below 0. So, which kind of layer could I add in order to get probability (values between 0 and 1) to deal with this problem ?</p>
</blockquote>
</aside>
<p>The output of your network (which is the output of your final <code>Linear</code><br>
layer) should be interpreted as raw-score <em>logits,</em> one (per sample)<br>
for each of your three classes.</p>
<p>If you <em>need</em> to convert the logits to probabilities (and you usually<br>
don’t), you do so by passing them through a <code>sigmoid()</code> function.</p>
<p>Note, however, you should pass the logits directly to <code>BCEWithLogitsLoss</code>,<br>
<em>without</em> converting them to probabilities.  <code>BCEWithLogitsLoss</code> expects<br>
logits and has, in effect, a <code>sigmoid()</code> built into it.  (For reasons of<br>
numerical stability, you should <em>not</em> use <code>BCELoss</code>, which does expect<br>
probabilities.)</p>
<p>If you do need to convert your logits to probabilities, for whatever reason,<br>
you should do it after computing your loss criterion, and not have it be<br>
part of your backpropagation.</p>
<p>Best.</p>
<p>K. Frank</p>