How can I fine-tune XLM for sentence classification?
Author: AMR-KELEG
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I am using the `XLMTokenizer` and `XLMForSequenceClassification` for fine-tuning the `xlm-mlm-en-2048` model to work on a sentence classification problem. 
I am using the same configuration as the one that I have used for fine-tuning BERT.
Surprisingly, XLM seems not to be improving at all (The loss is decreasing but the accuracy isn't affected!).
Actually, the model has overfitted to always select the dominant class in the dataset!
```
EPOCH 0:
Iteration: 0. Loss: 1.0213737487792969. Accuracy: 66.17283950617283%
Iteration: 1000. Loss: 0.9081503748893738. Accuracy: 66.29629629629629%
EPOCH 1:
Iteration: 0. Loss: 0.6950288414955139. Accuracy: 66.29629629629629%
Iteration: 1000. Loss: 0.648954451084137. Accuracy: 66.29629629629629%
EPOCH 2:
Iteration: 0. Loss: 0.7168332934379578. Accuracy: 66.29629629629629%
Iteration: 1000. Loss: 0.38551628589630127. Accuracy: 66.29629629629629%
```

The function used to tokenize a sentence is:
```
def prepare_features(tokenizer, seq_1, max_seq_length = 100, 
             zero_pad = True, include_CLS_token = True, include_SEP_token = True):
    ## Tokenzine Input
    tokens_a = tokenizer.tokenize(seq_1)

    ## Truncate
    if len(tokens_a) > max_seq_length - 2:
        tokens_a = tokens_a[0:(max_seq_length - 2)]
    ## Initialize Tokens
    tokens = []
    if include_CLS_token:
        tokens.append(tokenizer.cls_token)
    ## Add Tokens and separators
    for token in tokens_a:
        tokens.append(token)

    if include_SEP_token:
        tokens.append(tokenizer.sep_token)

    input_ids = tokenizer.convert_tokens_to_ids(tokens)
    ## Input Mask 
    input_mask = [1] * len(input_ids)
    ## Zero-pad sequence length
    if zero_pad:
        while len(input_ids) < max_seq_length:
            input_ids.append(0)
            input_mask.append(0)
    return torch.tensor(input_ids).squeeze(0), input_mask
```

What do you advise me to do in order to investigate this strange result?

Thanks,
Amr

Comments:
First of, your learning rate might be too low but even then it is odd to see the exact same accuracy all the time. You'll have to have a look at your dataset. Are all your classes in both training, validation, and test set? Are some classes weighted? It's quite hard to help with this.

Also, accuracy is a crude measure. Have a look at how your f1 evolves over time. IIRC sklearn has an utility to also calculate a "test report" where you can see how well all classes are predicted. Might be worth investigating too. 

> First of, your learning rate might be too low but even then it is odd to see the exact same accuracy all the time. You'll have to have a look at your dataset. Are all your classes in both training, validation, and test set? Are some classes weighted? It's quite hard to help with this.
> 
> Also, accuracy is a crude measure. Have a look at how your f1 evolves over time. IIRC sklearn has an utility to also calculate a "test report" where you can see how well all classes are predicted. Might be worth investigating too.

Hmm, The learning rate is the default `1e-5`. I am sure the classes are available in the training and validation datasets.
Since the model is overfitting, sklearn generates a warning message that f1 score is ill-defined since the model always predict 0.
This extreme over-fitting seems strange to me, I will actually try lowering down the learning rate.
Here is the Google Colab Notebook url in case you want to have a look: https://colab.research.google.com/drive/1VLt_a-lxLdibYFGnZFDncm1Ib28es57A 
Thanks :smile: 

Some thing: batch_size seems rather small but not so much that it could explain the issue. You do have a big difference in input data length (median of 37 and max of 165 tokens), and with a small batch size this may not average well. I'm not sure if it's standard practice to evaluate every 1000 steps in the training loop (I'd evaluate each epoch after all training data has been seen, at least for a dataset this size), but also that won't explain the problem.

You can try printing out the accuracy during training, too, and see if it is overfitting. 

I am having a similar problem with my own data. It was predicting the majority class all the time. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I have the same problem when I finetune XLM for  two class sentence classification task. not only does it predict the majority class all the time, but also gives the exactly same probablity for different cases !  Does anyone find a solution to that?