Singlelabel and Multilabel text classification by a LSTM

username: Huseyin
cooked: <p>Hello,</p>
<p>I have a problem where i would like to predict single class “d” [000001] and multilabel [ “d”,“z”] [010100] class at the same time in a classifier with LSTM. So I mean my final Network will be able to predict both single label and multilabel class.</p>
<p>I want to know what would be the best aproach to this problem. Because I have seen either Single label or Multilabel Classifiers so far.</p>
<p>EXAMPLE:  a Classifier with LSTM</p>
<p>sequence 1 --&gt; Target1<br>
[“a” , “b” , “c”]  = “d”<br>
“[000001]” , “[000010]” , “[000100]” = “[001000]”</p>
<p>sequence 2 --&gt; Target 2<br>
[“a”, “b”, “y”] =  [“d”, “z”]<br>
“[000001]” , “[000010]” , “[010000]” = “[101000]”</p>
<p>Many Thanks</p>

username: Abhilash_Srivastava
cooked: <p>It’s hard to understand what exactly you’re trying to do here? Better explain more.<br>
Are you trying to create <code>n</code> binary classifiers?</p>

username: Huseyin
cooked: <p>Hi <a class="mention" href="/u/abhilash_srivastava">@Abhilash_Srivastava</a> I have updated the description more in Detail, I hope it is more clear.<br>
I dont want to have n binary classifiers. Only one Classifier using LSTM, which will be capable of computing single and multilabel classification.</p>

username: KFrank
cooked: <p>Hello Hüseyin!</p>
<aside class="quote no-group" data-username="Huseyin" data-post="3" data-topic="99973" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/b782af/48.png" class="avatar"> Huseyin:</div>
<blockquote>
<p>I dont want to have n binary classifiers.</p>
</blockquote>
</aside>
<p>Some clarifying comments:</p>
<p>A <em>multi-label</em>, multi-class classifier should be thought of as <code>n</code> binary<br>
classifiers that all run together in a single network in single pass.</p>
<p>The predicted output is (logits / probabilities) predictions for a class-“0”<br>
binary classifier, yes vs. no, class-“1”, yes vs. no, and so on.</p>
<aside class="quote no-group">
<blockquote>
<p>Only one Classifier … which will be capable of computing single and multilabel classification.</p>
</blockquote>
</aside>
<p>Not commenting specifically on LSTM:</p>
<p>Single-label and multi-label classifiers are somewhat different.  How<br>
do you propose to build one that can do both?  How would you tell<br>
it that you want a single-label vs. multi-label output?</p>
<p>(One possibility is that your network has separate outputs for both<br>
the single-label and multi-label prediction, and you train on a loss<br>
that is a sum of a single-label and multi-label loss function.)</p>
<p>One thing you can do that does make sense is have the upstream<br>
“backbone” of your network be “pre-trained” to find features that are<br>
relevant for your classes, and then build two separate networks for<br>
each of which you train a couple of downstream layers, once set for<br>
the single-label case, and a second set for the multi-label case.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Huseyin
cooked: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="99973">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(One possibility is that your network has separate outputs for both<br>
the single-label and multi-label prediction, and you train on a loss<br>
that is a sum of a single-label and multi-label loss function.)</p>
</blockquote>
</aside>
<p>Hi Frank,</p>
<p>Thank you for your proposals.</p>
<p>(One possibility is that your network has separate outputs for both<br>
the single-label and multi-label prediction, and you train on a loss<br>
that is a sum of a single-label and multi-label loss function.)</p>
<p>This means that instead of computing n binary classification, I should have two branches (two output layer), where one computes the multilabel output loss and the other branch computes the single binary output loss. Then combining the both loss will get me the total loss. Just for approval <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Thanks</p>

username: KFrank
cooked: <p>Hello Hüseyin!</p>
<aside class="quote no-group" data-username="Huseyin" data-post="5" data-topic="99973" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/b782af/48.png" class="avatar"> Huseyin:</div>
<blockquote>
<p>This means that instead of computing n binary classification, I should have two branches (two output layer), where one computes the multilabel output loss and the other branch computes the single binary output loss.</p>
</blockquote>
</aside>
<p>I <em>think</em> what you propose is correct, but I wouldn’t say it quite this<br>
way.</p>
<p>Let’s say you have <code>nClass</code> classes, and you want to perform both<br>
single-label and multi-label multi-class classification.</p>
<p>In order to have two “branches” you could have a final <code>Linear</code> layer<br>
with <code>2 * nClass</code> outputs.  The first <code>nClass</code> of those would be your<br>
single-label branch, and would be fed into <code>CrossEntropyLoss</code> with<br>
a single integer class label (per sample in your batch) as the <code>target</code><br>
for <code>CrossEntropyLoss</code>.</p>
<p>The second set of <code>nClass</code> output values would be your multi-label<br>
branch, and would be fed into <code>BCEWithLogitsLoss</code> with a set<br>
<code>nClass</code> per-class probabilities (which could just be <code>0.0</code> and <code>1.0</code><br>
labels) as the <code>target</code> for <code>BCEWithLogitsLoss</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>Then combining the both loss will get me the total loss.</p>
</blockquote>
</aside>
<p>Yes, then sum the two losses together to get your <code>total_loss</code>, call<br>
<code>total_loss.backward()</code>, and optimize as usual.</p>
<p>You will likely want to perform a <em>weighted</em> sum of your two losses to<br>
balance the relative performance of your two types of classification,<br>
as dictated by your specific needs.</p>
<p>Good luck.</p>
<p>K. Frank</p>