Strange error while using the `LongformerForMultipleChoice`
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

I am trying to use `LongformerForMultipleChoice` model, and the code I am using is the following:
```python
# import the pre-trained HuggingFace Longformer tokenizer.
longformer_tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')

# get the pre-trained HuggingFace Longformer
best_model_longformer = LongformerForMultipleChoice.from_pretrained('allenai/longformer-base-4096',
                                                         output_hidden_states = True)

# my multiple choice question has 4 options.
question_list = [main_question, main_question, main_question, main_question]
options_list = [option1, option2, option3, option4]
mc_labels = torch.tensor([my_answer])

encoded_dict = longformer_tokenizer(question_list, options_list,
                             return_tensors = 'pt',
                             add_prefix_space = True,
                             padding = True)
input_hidden_state = best_model_longformer(
                       **{k: v.unsqueeze(0) for k,v in encoded_dict.items()}, 
                        labels = mc_labels)[2][0][:,:,:].detach()
```

and I am getting the error below:
```
/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py:71: UserWarning: This overload of nonzero is deprecated:
        nonzero()
Consider using one of the following signatures instead:
        nonzero(*, bool as_tuple) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)
  sep_token_indices = (input_ids == sep_token_id).nonzero()
Traceback (most recent call last):
  File "SEED_125_V20_15_LONGFORMER.py", line 427, in <module>
    main_function('/home/ec2-user/G1G2.txt','/home/ec2-user/G1G2_answer_num.txt', num_iter)
  File "SEED_125_V20_15_LONGFORMER.py", line 389, in main_function
    best_model_longformer)
  File "SEED_125_V20_15_LONGFORMER.py", line 198, in fill_MC_loss_accuracy_tensor
    input_hidden_state = best_model_longformer(**{k: v.unsqueeze(0) for k,v in encoded_dict.items()}, labels = mc_labels)[2][0][:,:,:].detach()
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 1808, in forward
    loss = loss_fct(reshaped_logits, labels)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 948, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2422, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2218, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 1 is out of bounds.
```
How can I fix this error?

I also tried solving this issue with:
```python
# group question_list and options_list as a single list rather than specifying them seperately
encoded_dict = longformer_tokenizer([question_list, options_list],
                             return_tensors = 'pt',
                             add_prefix_space = True,
                             padding = True)
```
But this generates a different error, saying:
```
ValueError: 2 expected but found 1
```
PS: I don't think my Longformer model is correctly getting that my multiple-choice questions have 4 options...is there any way to make Longformer to take my mutiple-choice questions with the ones with 4 options (instead of 2 options)?

Thank you.

PS: I am interested in extracting hidden embedding more than the error or the logit themselves

Comments:
The way `xxxForMultipleChoice` models work is actually a bit tricky. It works as follows (based on the [original explanation by the author of BERT](https://github.com/google-research/bert/issues/38)):

Given a question, and several options, the question + options are processed by the model independently. So they will look as follows:  `[CLS] question [SEP] option 1 [SEP]` first, then `[CLS] question [SEP] option 2 [SEP]` second, and so on. 

So when you're using the tokenizer to encode the input, it should be used as follows:
```
# my multiple choice question has 4 options.
question = "this is a question"
option1 = "option 1"
option2 = "option 2"
option3 = "option 3"
option4 = "option 4"

encoded_input = longformer_tokenizer([question, question, question, question], 
                                 [option1, option2, option3, option4], 
                                return_tensors='pt', 
                                padding='max_length')
```
we need to `unsqueeze` the values of that dictionary, so to make sure they are all of shape (batch_size, num_choices, seq_len), or thus (1, 4, 4096). Also, the answer should be a tensor having shape (batch_size,), so in our case this is just a tensor containing a single element, containing the index of the correct option. Suppose the correct option is 3, then `answer` will be `torch.tensor([2])` (since indexing starts at zero). Next, we can run the forward pass as follows:

```
mc_labels = torch.tensor([2])
outputs = model(**{k: v.unsqueeze(0) for k,v in encoding.items()}, labels=mc_labels,
                                 return_dict=True)  # batch size is 1
```

The `outputs` will be a `MultipleChoiceOutput` containing the loss and the logits. The logits are of shape (batch_size, number of choices), so (1, 4). 

If you want to train the model, simply get the loss using `outputs.loss` and perform `loss.backward()`. If you want to get the predictions of the model, convert the logits into predictions by typing `outputs.logits.argmax(-1)`. 

UPDATE: fixed the fact that the answer shouldn't be unsqueezed, since it's just a tensor of shape (batch_size,). 
UPDATE: fix indexing of answer.



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.