Multi-label approach for emotion recoginition advice needed

username: pretbc
cooked: <p>Hello all,</p>
<p>Ive a data as below:</p>
<pre><code class="lang-auto">| clip | happy | sad | anger | surprise | disgust | fear | neutral |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 0.44 | 0.0 | 0.0 | 0.0  | 0.0 | 0.0 | 0.0 | 
| 2 | 0.67 | 0.0 | 0.0 | 0.11 | 0.0 | 0.0 | 0.0 |
| 3 | 0.11 | 0.0 | 0.0 | 0.0  | 0.0 | 0.6 | 0.0 | 
| 4 | 0.44 | 0.0 | 0.0 | 0.11 | 0.0 | 0.0 | 0.0 | 
| 5 | 0.67 | 0.0 | 0.0 | 0.0  | 0.0 | 0.0 | 0.0 |

</code></pre>
<p>emotion range 0-1 ( data were normalized from original scores 0 - 3 )</p>
<p>each emotion was annotated by 3 judges after movie clip watched ( CMU-MOSEI ).<br>
And as You can see multiple emotions can occur for a clip</p>
<p>Right now Im trying to solve this as regression problem where I crop 32 frames from each clip and perform MSEloss with AdamW</p>
<pre><code class="lang-auto">        outputs = model(inputs=inputs)
        logits = outputs.logits

        loss = F.mse_loss(logits, labels)
        loss.backward()
        optimizer.step()
</code></pre>
<p>But is there possibility to convert this problem into multilabel classification problem ? To perform BinaryCrossEntropy or CrossEntropy ?<br>
Or any other to rather perform classification than regression</p>
<p>TY</p>

username: KFrank
cooked: <p>Hi pretbc!</p>
<aside class="quote no-group" data-username="pretbc" data-post="1" data-topic="195886" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pretbc/48/63592_2.png" class="avatar"> pretbc:</div>
<blockquote>
<p>emotion range 0-1</p>
</blockquote>
</aside>
<p>Because the score for each emotion ranges from zero to one (and there<br>
is no constraint that the values sum to one), it is reasonable to treat each<br>
emotion score as if it were a probability.</p>
<aside class="quote no-group">
<blockquote>
<p>And as You can see multiple emotions can occur for a clip</p>
</blockquote>
</aside>
<p>Because multiple emotions can occur for a single clip <em>and</em> the emotion<br>
scores are (in some sense) independent of one another (in particular,<br>
there is no constraint that they sum to one), treating your problem as<br>
a multi-label classification could well be a good fit for your use case.</p>
<aside class="quote no-group">
<blockquote>
<p>Right now Im trying to solve this as regression problem</p>
</blockquote>
</aside>
<p>Treating this as a regression could also be a good fit, although regressing<br>
values that are constrained to run from zero to one is not fully natural.  One<br>
sign of this is that there is nothing that prevents the <em>predictions</em> of such a<br>
regression model from lying outside the zero-to-one range.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        outputs = model(inputs=inputs)
        logits = outputs.logits

        loss = F.mse_loss(logits, labels)
        loss.backward()
        optimizer.step()
</code></pre>
<p>But is there possibility to convert this problem into multilabel classification problem ? To perform BinaryCrossEntropy or CrossEntropy ?</p>
</blockquote>
</aside>
<p>Yes, it would be perfectly reasonable to convert this into a multi-label<br>
classification.  Whether this will work better for your use case than a<br>
regression is an empirical question – if you have the time, try both<br>
approaches.</p>
<p>As it stands, all you need to do is change:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = F.mse_loss (logits, labels)
</code></pre>
<p>to:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = F.binary_cross_entropy_with_logits (logits, labels)
</code></pre>
<p>(where I assume that your <code>outputs.logits</code> run from <code>-inf</code> to <code>inf</code> and<br>
would typically be the output of a final <code>Linear</code> layer with <code>out_features</code><br>
equal to <code>9</code> (your number of classes) and where that <code>Linear</code> layer is <em>not</em><br>
followed by any <code>sigmoid()</code> or other non-linear activation).</p>
<p>(You could also use the class form, <code>BCEWithLogitsLoss</code> if you prefer<br>
that stylistically to the functional form.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: pretbc
cooked: <p>Thanks for Your  answer  <a href="/u/KFrank">KFrank</a></p>
<p>So after I did a switch to BCEWithLogits another question comes to my mind</p>
<p>How to evaluate this problem. I wrote metrics function:</p>
<pre><code class="lang-auto">from torchmetrics.classification import (
    MultilabelAccuracy, MultilabelAveragePrecision, MultilabelF1Score,
)

def multilabel_metrics(predictions: torch.Tensor, target: torch.Tensor) -&gt; dict:
    """Calculate multilabel classification various metrics"""
    num_outputs = target.shape[1]
    # change targets to binary [0,1] for metrics 
    th_target = torch.as_tensor((target - 0.5) &gt; 0, dtype=torch.int64)
    multilabel_acc = MultilabelAccuracy(num_labels=num_outputs, average='macro')
    multilabel_ap = MultilabelAveragePrecision(num_labels=num_outputs, average="macro")
    multilabel_f1 = MultilabelF1Score(num_labels=num_outputs, average='macro')

    return {
        'multilabel-acc': multilabel_acc(predictions, th_target).item(),
        'multilabel-ap': multilabel_ap(predictions, th_target).item(),
        'multilabel-f1': multilabel_f1(predictions, th_target).item(),
    }
</code></pre>
<p>and validation loop</p>
<pre><code class="lang-auto">    model.eval()
    test_loss = 0
    preds = torch.tensor([])
    targets = torch.tensor([])
    loop = tqdm(test_loader, leave=True)
    with torch.no_grad():
        for (inputs, labels) in loop:
            inputs = inputs.to(envi_builder.config.device)
            labels = labels.to(envi_builder.config.device)

            outputs = model(inputs=inputs)
            logits = outputs.logits

            # call binary_cross_entropy_with_logits (logits, labels)
            loss = call_task_loss(envi_builder.config.task)(logits, labels)
            test_loss += loss.item()
  
            #concat all preds and target to use later on metrics
            preds = torch.cat((preds, logits.detach().cpu()), 0)
            targets = torch.cat((targets, labels.detach().cpu()), 0)

    test_loss = test_loss / len(test_loader)
    metrics = call_task_metrics(envi_builder.config.task)(preds, targets)
</code></pre>
<p>and metrics seems to be very strange ( after first 5 epoch )</p>
<pre><code class="lang-auto">Metric multilabel-acc: 0.925
Metric multilabel-ap: 0.177
Metric multilabel-f1: 0.000
</code></pre>
<p>example:</p>
<pre><code class="lang-auto">from torchmetrics.classification import MultilabelAccuracy
target = tensor([[0, 0, 0, 0, 0, 0, 0], [1, 0, 0, 0, 0, 0, 0]])
preds = tensor([[-0.8832, -1.3300, -1.2242, -1.1196, -0.6935, -0.7516, -0.4022], [-0.8940, -1.2300, -1.2766, -1.1840, -0.8486, -0.6994, -0.4034]])
metric = MultilabelF1Score(num_labels=7)
metric(preds, target)
&gt;&gt;tensor(0.9286)

--------------------

from torchmetrics.classification import MultilabelF1Score
target = tensor([[0, 0, 0, 0, 0, 0, 0], [1, 0, 0, 0, 0, 0, 0]])
preds = tensor([[-0.8832, -1.3300, -1.2242, -1.1196, -0.6935, -0.7516, -0.4022], [-0.8940, -1.2300, -1.2766, -1.1840, -0.8486, -0.6994, -0.4034]])
metric = MultilabelF1Score(num_labels=7)
metric(preds, target)
&gt;&gt;tensor(0.)
</code></pre>
<p>Is there something that I missing ?</p>

username: pretbc
cooked: <p>I see Your point <a href="/u/rtwolfe94022">rtwolfe94022</a></p>
<p>but still I wondering how to evaluate model</p>
<p>lets say I have emotion → <code>label = tensor([[0.67, 0.0, 0.11, 0.0, 0.0, 0.0, 0.0]])</code></p>
<p>I than convert this into binary:</p>
<pre><code class="lang-auto">torch.as_tensor(label &gt; 0, dtype=torch.float32)
&gt;&gt; tensor([[1., 0., 1., 0., 0., 0., 0.]])

</code></pre>
<p>and put into loss fun() # BCEwithlogits</p>
<p>next:<br>
during metric calculation Im gonna convert logits into prob</p>
<pre><code class="lang-auto">sigmoid(preds)
tensor([[0.6789, 0.2092, 0.3643, 0.1256 0.3333, 0.3205, 0.4008],

</code></pre>
<p>so after MultilabelAccuracy th = 0.5 I will get ~ [1, 0, 0, 0, 0, 0, 0]</p>
<ol>
<li>I assume <code>th</code> of metric has to be changed ( th &gt; 0 ) ?</li>
<li>But that will only tell me that the emotion is there, but I won’t know its strength, right?</li>
</ol>