Repository: huggingface/transformers

Issue:
Cannot use camembert for question answering
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
Camembert
Language I am using the model on (English, Chinese ...):
French
The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: Squad
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load camembert for Q&A
2. Use the script for Q&A from the HuggingFace Doc
3. Get a Runtimeerror

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

from [https://huggingface.co/transformers/usage.html#question-answering](https://huggingface.co/transformers/usage.html#question-answering) : 

Note : I tried with `camembert-base`, `illuin/camembert-base-fquad` and `fmikaelian/camembert-base-fquad`
```
from transformers import AutoTokenizer, CamembertForQuestionAnswering
import torch

tokenizer = AutoTokenizer.from_pretrained("camembert-base")
model = CamembertForQuestionAnswering.from_pretrained("camembert-base")

text = r"""Some text in french"""
questions = ["Just one question in french"]

for question in questions:
    inputs = tokenizer.encode_plus(question, text, add_special_tokens=True, return_tensors="pt")
    input_ids = inputs["input_ids"].tolist()[0]

    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
    answer_start_scores, answer_end_scores = model(**inputs)

    answer_start = torch.argmax(
        answer_start_scores
    )  # Get the most likely beginning of answer with the argmax of the score
    answer_end = torch.argmax(answer_end_scores) + 1  # Get the most likely end of answer with the argmax of the score

    answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))

    print(f"Question: {question}")
    print(f"Answer: {answer}\n")
```
It fails as well with the `pipeline` method 
```
q_a_pipeline = pipeline("question-answering", model=model, tokenizer=tokenizer)

q_a_pipeline({'question': question, 'context': text}
```
Stack trace : 
```
Traceback (most recent call last):
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/sanic/app.py", line 976, in handle_request
    response = await response
  File "test_server_sanic.py", line 72, in get_answer
    results = [q_a_pipeline({'question': question, 'context': doc})
  File "test_server_sanic.py", line 72, in <listcomp>
    results = [q_a_pipeline({'question': question, 'context': doc})
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/transformers/pipelines.py", line 1109, in __call__
    start, end = self.model(**fw_args)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/transformers/modeling_roberta.py", line 663, in forward
    outputs = self.roberta(
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/transformers/modeling_bert.py", line 728, in forward
    embedding_output = self.embeddings(
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/transformers/modeling_roberta.py", line 64, in forward
    return super().forward(
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/transformers/modeling_bert.py", line 175, in forward
    token_type_embeddings = self.token_type_embeddings(token_type_ids)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 112, in forward
    return F.embedding(
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 1724, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Run the exemple in french as with Bert in english

Note : I am able to run the exemple with a Hugging Face Pipeline (with all the different camembert model, comunity or not)
```
bert_tok = AutoTokenizer.from_pretrained("camembert-base")
bert = CamembertForQuestionAnswering.from_pretrained("camembert-base")

nlp = pipeline('question-answering', model=bert, tokenizer=bert_tok)

answer = nlp({'question': "A question in french",
    'context': a_big_string_in_french})
print(answer)
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->


- `transformers` version: 2.8.0
- Platform: Linux-5.6.10-arch1-1-x86_64-with-glibc2.2.5
- Python version: 3.8.2
- PyTorch version (GPU?): 1.4.0 (True) (Same with 1.5)
- Tensorflow version (GPU?): 2.2.0-rc4 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Discussion:

Resolving patch:
diff --git a/src/transformers/tokenization_camembert.py b/src/transformers/tokenization_camembert.py
--- a/src/transformers/tokenization_camembert.py
+++ b/src/transformers/tokenization_camembert.py
@@ -102,6 +102,7 @@ class CamembertTokenizer(PreTrainedTokenizer):
     vocab_files_names = VOCAB_FILES_NAMES
     pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
     max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
+    model_input_names = ["attention_mask"]
 
     def __init__(
         self,
@@ -200,14 +201,7 @@ def create_token_type_ids_from_sequences(
     ) -> List[int]:
         """
         Creates a mask from the two sequences passed to be used in a sequence-pair classification task.
-        A CamemBERT sequence pair mask has the following format:
-
-        ::
-
-            0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
-            | first sequence  | | second sequence |
-
-        if token_ids_1 is None, only returns the first portion of the mask (0s).
+        CamemBERT, like RoBERTa, does not make use of token type ids, therefore a list of zeros is returned.
 
         Args:
             token_ids_0 (:obj:`List[int]`):
@@ -216,15 +210,15 @@ def create_token_type_ids_from_sequences(
                 Optional second list of IDs for sequence pairs.
 
         Returns:
-            :obj:`List[int]`: List of `token type IDs <../glossary.html#token-type-ids>`_ according to the given
-            sequence(s).
+            :obj:`List[int]`: List of zeros.
+
         """
         sep = [self.sep_token_id]
         cls = [self.cls_token_id]
 
         if token_ids_1 is None:
             return len(cls + token_ids_0 + sep) * [0]
-        return len(cls + token_ids_0 + sep + sep) * [0] + len(token_ids_1 + sep) * [1]
+        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]
 
     @property
     def vocab_size(self):


Test patch: