Trouble fine tuning multiple choice
Author: KerenzaDoxolodeo
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi! I have issues with fine-tuning the multi-choice BERT and I am stuck on an error and I can use some help. When I tried to fine-tune it with my own dataset, it threw the Error  ```RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /opt/conda/conda-bld/pytorch_1570711300255/work/aten/src/THNN/generic/ClassNLLCriterion.c:97````

According to what I have found, this normally happened due to the dimension mismatch between the label and the output layer. When I printed the model, it seems that the model does not have a suitable output layer.

```
` (dropout): Dropout(p=0.1, inplace=False)
  (classifier): Linear(in_features=768, out_features=1, bias=True)
)`
```
I have made sure that the Bert Config has received the correct number of the label (`num_labels`)

```
{
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 3,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 30522
}
```

I am using the code from examples/run_multiple_choice.py and examples/utils_multiple_choice.py without touching its logic but I have modified it the data processor.

```
class CSProcessor(object):

    def get_train_examples(self):
        return self._create_examples(df_all[:8000] , 
                                     df_answer[:8000], "train")

    def get_test_examples(self):
        """See base class."""
        return self._create_examples(df_sentence[8000:] ,
                                     df_answer[8000:], "test")

    def get_labels(self):
        """See base class."""
        return ["0", "1", "2"]
    

    def _create_examples(self, df_sentence, df_answer, set_type):
        """Creates examples for the training and dev sets."""
        examples = []
        for I in range(len(df_sentence)):
            race_id = "%s-%s" % (set_type, I)
            
            truth = str(ord(df_answer[1][I]) - ord("A"))
            illogicalAnswer = df_sentence["FalseSent"][I]
            examples.append(
                InputExample(
                example_id=race_id,
                question="Why it doesn't make senses?",
                contexts=[illogicalAnswer], 
                endings=[df_sentence['OptionA'][I], 
                         df_sentence['OptionB'][I],
                         df_sentence['OptionC'][I]],
                label=truth,
                )
            )
        return examples
```

I appreciate any help :)

Comments:
Hi! I believe this issue could stem from your label being negative as well. Could you check that it doesn't fail when computing the loss with a negative label?

I'm not sure what is a negative label in the context of multiple-choice,  but here's what I did: it successfully computed loss when the label is 0 (which should be representing the first option?) but it fails for 1, 2, and -1.

@KerenzaDoxolodeo Hello,I am also trying to finetune RACE on bert model. I am wondering if you had fixed this problem.
Also can you post your fine tuning command with exact hyperparameters? Thanks.

@KerenzaDoxolodeo I ran into a similar problem. You are right in that the error is due to a shape mismatch. However you don't need to change the config file. Instead adapt your processor class by changing the context parameter of your InputExample from
`contexts=[illogicalAnswer]` 
to
`contexts=[illogicalAnswer, illogicalAnswer, illogicalAnswer]`

If you look at the original SwagProcessor, they copied the context several times such that both the context as well as the endings are lists of size num_labels.

Unfortunately, the code does not raise an error if you ignore this requirement. If you look closely at what happens within the convert_examples_to_features() method in examples/utils_multiple_choice.py, you'll notice the line
`enumerate(zip(example.contexts, example.endings))`
This is where everything breaks. Since you only provided a single context per example, this line will lead to also only taking into account a single ending from example.endings.

You don't need to adapt the config file as it will adapt to the number of labels automatically. If, for some reason, you still like to change the config, I think you should not manually overwrite the num_labels parameter as this is likely to introduce further errors (like in the config you showed above). Instead load the config from pretrained and provide the number of labels, e.g. like this
`config = BertConfig.from_pretrained(
        'bert-base-uncased',
        num_labels=5,
)`
This will also change the mappings "id2label" as well as "label2id" appropriately (both are not set properly in your posted example).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.