Multi-class BERT Model: Class weights and where to use them

Bersk: <p>Hi all,</p>
<p>I am considering that similar or related topics have been asked before (such as in <a href="https://discuss.pytorch.org/t/weights-of-cross-entropy-loss-for-validation-dev-set/46132">weights-of-cross-entropy-loss-for-validation-dev-set</a> and <a href="https://discuss.pytorch.org/t/apply-weightedrandomsampler-to-validation-test-splits-makes-sense/93733">apply-weightedrandomsampler-to-validation-test-splits-makes-sense</a>). However, I still have some confusion as those and previous post include different code snipets for me to understand the complete scenario.</p>
<p>My case is as follows: I am working on a model for multi-class classification with a BERT Model with customized layers. However, the classes are not balanced, and I am trying to validate if working on this can improve the result metrics.<br>
I found 3 main suggestions (besides working with the data as-is) in different articles and previous posts, which are undersampling, oversampling and use of class weights.<br>
I am currently testing the last one, but I am not sure if I am doing so in the right way, or if there is any reasoning that I am missing from those examples.</p>
<p>The steps that I have seen in previous examples are:</p>
<ol>
<li>
<p>Sample the data into test, train and validation (standard step, no issue here)</p>
</li>
<li>
<p>Generate the class weights for later cross entropy calculation. For this step, I have seen different approaches, but the one I have found the most is something similar to this:</p>
</li>
</ol>
<pre><code class="lang-auto">from sklearn.utils.class_weight import compute_class_weight

# compute the class weights
class_wts = compute_class_weight('balanced', np.unique(train_labels), train_labels)

print(class_wts)

# convert class weights to tensor
weights= torch.tensor(class_wts,dtype=torch.float)
weights = weights.to(device)
</code></pre>
<p>2.5. Sometimes, the weights are immediately assigned to a temporary new function:</p>
<pre><code class="lang-auto"># loss function
cross_entropy  = nn.NLLLoss(weight=weights) 
</code></pre>
<ol start="3">
<li>After defining the model and the hyperparameters, proceed with the training. If the weights were not incorporated in a cross_entropy function before, they are directly created now:</li>
</ol>
<pre><code class="lang-auto">def train():
  
  model.train()
    total_loss, total_accuracy = 0, 0
    total_preds=[]
    for step,batch in enumerate(train_dataloader):
        batch = [r.to(device) for r in batch]
        sent_id, mask, labels = batch
        model.zero_grad()        
        preds = model(sent_id, mask)
        loss = cross_entropy(preds, labels)
        # or # criterion = torch.nn.CrossEntropyLoss(weight=weights,reduction='mean')
        # and then # loss = criterion(outputs[1], b_labels)
        total_loss = total_loss + loss.item()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        preds=preds.detach().cpu().numpy()
        total_preds.append(preds)
  
    avg_loss = total_loss / len(train_dataloader)
    total_preds  = np.concatenate(total_preds, axis=0)
    return avg_loss, total_preds
</code></pre>
<p>Until here I had little issues following the logic. But the next step is the one that confuses me.</p>
<ol start="4">
<li>After training, the validation is being done, but also adding the cross_entropy funtion.</li>
</ol>
<pre><code class="lang-auto">def evaluate():  
    model.eval()
    total_loss, total_accuracy = 0, 0
    total_preds = []
    for step,batch in enumerate(val_dataloader):
        if step % 50 == 0 and not step == 0:
            elapsed = format_time(time.time() - t0)
            print('  Batch {:&gt;5,}  of  {:&gt;5,}.'.format(step, len(val_dataloader)))
        batch = [t.to(device) for t in batch]
        sent_id, mask, labels = batch
        with torch.no_grad():
            preds = model(sent_id, mask)
            loss = cross_entropy(preds,labels)  #&lt;------- Here
            total_loss = total_loss + loss.item()
            preds = preds.detach().cpu().numpy()
            total_preds.append(preds)
    avg_loss = total_loss / len(val_dataloader) 
    total_preds  = np.concatenate(total_preds, axis=0)
    return avg_loss, total_preds
</code></pre>
<p>However, the cross_entropy function is the same as it was when created, which means that it is using the weights calculated from the train dataset.<br>
In that regard, my questions are:</p>
<ol>
<li>Is it correct to use weights during the validation method, or should the validation method calculate the loss as usual?</li>
<li>If we are supposed to use weights, should they be recalculated from the validation dataset? Or is it correct to use the ones from the training dataset?</li>
<li>I am assuming that for the test dataset no weight should be used, since the goal is to test the model with what is real-world data. Is this correct or there am I confusing the logic for this final part?</li>
</ol>
<p>Thanks in advance for any help and suggestion!</p>

ptrblck: <ol>
<li>Yes, the weighted loss can be reused to get a signal about overfitting as <a class="mention" href="/u/kfrank">@KFrank</a> explains in <a href="https://discuss.pytorch.org/t/how-can-overfitting-be-detected-using-weighted-binary-cross-entropy/169317/3">this post</a>.</li>
<li>Both datasets should come from the same domain and have the same or similar distributions as also explained in the linked post.</li>
<li>For the final run on the test data you should calculate the metric which represents your target the closest. E.g. calculating the accuracy for an imbalanced use case could show a high value, while your model might be useless and predict only the majority class due to the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">accuracy paradox</a>.</li>
</ol>

Bersk: <p>I think now I have a more clear idea of the differences in the steps between oversampling and the use of class weights.</p>
<p>Thanks for the reply and links!</p>