Converting single label classification to multi-label classification

username: kerrangcash
cooked: <p>I am currently using a LSTM model to do some binary classification on a text dataset and was wondering how to go about extending this model to perform multi-label classification.  The current model is as follows:</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers, bidirectional,
                 dropout_rate, pad_index):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_index)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, bidirectional=bidirectional,
                            dropout=dropout_rate, batch_first=True)
        self.fc = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim, output_dim)
        self.dropout = nn.Dropout(dropout_rate)
        
    def forward(self, ids, length):

        embedded = self.dropout(self.embedding(ids))

        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, length, batch_first=True, 
                                                            enforce_sorted=False)
        packed_output, (hidden, cell) = self.lstm(packed_embedded)

        output, output_length = nn.utils.rnn.pad_packed_sequence(packed_output)

        if self.lstm.bidirectional:
            hidden = self.dropout(torch.cat([hidden[-1], hidden[-2]], dim=-1))

        else:
            hidden = self.dropout(hidden[-1])

        prediction = self.fc(hidden)

        return prediction
</code></pre>
<p>Currently, the output of the model is 2 dimensions as the labels can either be 1 or 0.  But I would be looking to convert it to classify a one hot encoding of the form [0, 0, 1, 0, 0, 1, 0].  I am aware I would probably need to change the criterion from nn.crossEntropyLoss but I am unsure what other steps to take.</p>

username: KFrank
cooked: <p>Hi Kerrangcash!</p>
<aside class="quote no-group" data-username="kerrangcash" data-post="1" data-topic="148272" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/d9b06d/48.png" class="avatar"> kerrangcash:</div>
<blockquote>
<p>how to go about extending this model to perform multi-label classification.</p>
</blockquote>
</aside>
<p>When you say “multi-label” I assume that you mean “multi-label, multi-class”<br>
classification.  That is, you have multiple classes, presumably, but not<br>
necessarily, more than two.  And each sample can be labelled with any<br>
number of those several classes, including none or all.</p>
<aside class="quote no-group">
<blockquote>
<p>I would be looking to convert it to classify a one hot encoding of the form [0, 0, 1, 0, 0, 1, 0].</p>
</blockquote>
</aside>
<p>First of all, the example vector you gives represents not “one-hot encoding,”<br>
but *multi-hot encoding."  (A one-hot encoded vector would contain exactly<br>
one <code>1</code> with all the rest <code>0</code>s.)</p>
<p>Your example vector has length 7.   Therefore I assume that you have 7<br>
classes (which should be that value of <code>output_dim</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>I am aware I would probably need to change the criterion from nn.crossEntropyLoss.</p>
</blockquote>
</aside>
<p>No.  If I am correct that you are performing a multi-label, multi-class<br>
classification, then you should use <code>BCEWithLogitsLoss</code>.  The point is<br>
that a multi-label problem is, in some sense, n-class binary-classification<br>
problems run through your network at the same time.  (Is your sample in<br>
“class-1” – yes or no?  Is it separately in “class-2” – yes or no?  And so on.)</p>
<p>So you would want the output of your model to be a final <code>Linear</code> layer<br>
with <code>out_features = out_dim</code> (so, for this example, 7), as you have<br>
in the code you posted.  Feed these <em>logits</em> directly to <code>BCEWithLogitsLoss</code><br>
without any intervening “activation” layer.</p>
<p>The <code>target</code> (ground-truth labels) you pass to <code>BCEWithLogitsLoss</code> can<br>
be your multi-hot-encoded label vector, except that it has to be <code>float</code>,<br>
rather than <code>int</code>.  (This is because <code>BCEWithLogitsLoss</code> accepts<br>
probabilistic targets that can range from <code>0.0</code> to <code>1.0</code>, but having them<br>
be exactly <code>0.0</code> or <code>1.0</code> is perfectly fine.)</p>
<p>Best.</p>
<p>K. Frank</p>