Question Answering pipeline not working
Author: paras55
Pull request: no
Labels: name: wontfix

This is error while running the question answering pipeline !

convert squad examples to features:   0%|          | 0/1 [00:00<?, ?it/s]

---------------------------------------------------------------------------

RemoteTraceback                           Traceback (most recent call last)

RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/usr/local/lib/python3.6/dist-packages/transformers/data/processors/squad.py", line 198, in squad_convert_example_to_features
    p_mask = np.array(span["token_type_ids"])
KeyError: 'token_type_ids'
"""


The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)

<ipython-input-3-3c4dd3618524> in <module>()
      2 
      3 nlp_qa = pipeline('question-answering')
----> 4 nlp_qa(context='Hugging Face is a French company based in New-York.', question='Where is based Hugging Face ?')

8 frames

/usr/local/lib/python3.6/dist-packages/transformers/data/processors/squad.py in squad_convert_example_to_features()
    196         # p_mask: mask with 1 for token than cannot be in the answer (0 for token which can be in an answer)
    197         # Original TF implem also keep the classification token (set to 0) (not sure why...)
--> 198         p_mask = np.array(span["token_type_ids"])
    199 
    200         p_mask = np.minimum(p_mask, 1)

KeyError: 'token_type_ids'



Comments:
Had the same Issue, I think it was already fixed in an older commit, but the pip package doesnt seem to be updated. Try installing transformers from this repo:
`git clone https://github.com/huggingface/transformers`
`cd transformers`
`pip install .`

I also had the same issue as @paras55, and the solution provided by @mowoe fixed it! Thanks!

this is still an issue in 2.7.0.

Are you sure? I think it was fixed in this commit: [c76c3ce](https://github.com/huggingface/transformers/commit/c76c3cebed3c707178d9f721349c5abd5206a57f). 

I cloned the repo, checked out the 2.7.0 tag and built and installed the wheel and was running into the same issue.

Result of `pip list`

```
Package                Version     
---------------------- ------------
absl-py                0.9.0       
astor                  0.8.1       
astroid                2.3.3       
asttokens              2.0.3       
attrs                  19.3.0      
autopep8               1.5         
boto3                  1.12.27     
botocore               1.15.27     
cachetools             4.0.0       
certifi                2019.11.28  
chardet                3.0.4       
click                  7.1.1       
dataclasses            0.7         
decorator              4.4.2       
docutils               0.15.2      
entrypoints            0.3         
filelock               3.0.12      
flake8                 3.7.9       
flake8-aaa             0.7.1       
gast                   0.2.2       
google-auth            1.11.3      
google-auth-oauthlib   0.4.1       
google-pasta           0.2.0       
grpcio                 1.27.2      
h5py                   2.10.0      
idna                   2.9         
importlab              0.5.1       
importlib-metadata     1.5.0       
isort                  4.3.21      
jmespath               0.9.5       
joblib                 0.14.1      
Keras-Applications     1.0.8       
Keras-Preprocessing    1.1.0       
lazy-object-proxy      1.4.3       
Markdown               3.2.1       
mccabe                 0.6.1       
more-itertools         8.2.0       
mypy                   0.761       
mypy-extensions        0.4.3       
networkx               2.4         
ninja                  1.9.0.post1 
numpy                  1.18.2      
oauthlib               3.1.0       
opt-einsum             3.2.0       
packaging              20.1        
pandas                 1.0.3       
pip                    20.0.2      
pluggy                 0.13.1      
protobuf               3.11.3      
py                     1.8.1       
pyasn1                 0.4.8       
pyasn1-modules         0.2.8       
pycodestyle            2.5.0       
pyflakes               2.1.1       
pylint                 2.4.4       
pyparsing              2.4.6       
pytest                 5.3.5       
python-dateutil        2.8.1       
pytype                 2020.2.20   
pytz                   2019.3      
PyYAML                 5.3.1       
regex                  2020.2.20   
requests               2.23.0      
requests-oauthlib      1.3.0       
rsa                    4.0         
s3transfer             0.3.3       
sacremoses             0.0.38      
scikit-learn           0.22.2.post1
scipy                  1.4.1       
sentencepiece          0.1.85      
setuptools             45.2.0      
six                    1.14.0      
tensorboard            2.0.2       
tensorflow             2.0.0       
tensorflow-determinism 0.3.0       
tensorflow-estimator   2.0.1       
termcolor              1.1.0       
tokenizers             0.5.2       
tqdm                   4.43.0      
transformers           2.7.0       
typed-ast              1.4.1       
typing-extensions      3.7.4.1     
urllib3                1.25.8      
wcwidth                0.1.8       
Werkzeug               1.0.0       
wheel                  0.34.2      
wrapt                  1.11.2      
zipp                   3.0.0  
```


the code I ran (unnecessary parts excluded):
```
for a, b in zip(train_text_a, train_text_b):
    tokens_dict = tokenizer.encode_plus(a, b, max_length=10,
                                        pad_to_max_length=True)
    train_input_ids.append(np.asarray([tokens_dict["input_ids"]]))
    train_input_masks.append(np.asarray([tokens_dict["attention_mask"]]))
    train_input_segment_ids.append(np.asarray([tokens_dict["token_type_ids"]]))
```

and the traceback I received:

```
Traceback (most recent call last):
  File "<input>", line 104, in <module>
KeyError: 'token_type_ids'
```

Yeah I think we are talking about a completely different problem here. @amoux and @paras55 had this exception in the squad.py of this repo (Which shouldnt raise one, when used this way). Your Exception however is a KeyError in your code (line 104). Maybe try adding `return_token_type_ids=True` as argument to your tokenizer. This is still an issue, as this should be default. Maybe open a new one, as this isnt the same problem.

You're definitely correct. `return_token_type_ids` solved the problem. Much appreciated @mowoe!


An example for question answering with DistilBERT

```
from transformers import DistilBertTokenizer, DistilBertForQuestionAnswering
import torch

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased',return_token_type_ids = True)
model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased-distilled-squad')

context = "The US has passed the peak on new coronavirus cases, President Donald Trump said and predicted that some states would reopen this month.The US has over 637,000 confirmed Covid-19 cases and over 30,826 deaths, the highest for any country in the world."
question = "What was President Donald Trump's prediction?"

# question = "How many deaths have been reported from the virus?"

encoding = tokenizer.encode_plus(question, context)


input_ids, attention_mask = encoding["input_ids"], encoding["attention_mask"]

start_scores, end_scores = model(torch.tensor([input_ids]), attention_mask=torch.tensor([attention_mask]))

ans_tokens = input_ids[torch.argmax(start_scores) : torch.argmax(end_scores)+1]
answer_tokens = tokenizer.convert_ids_to_tokens(ans_tokens , skip_special_tokens=True)

all_tokens = tokenizer.convert_ids_to_tokens(input_ids)

print ("\nAnswer Tokens: ")
print (answer_tokens)

answer_tokens_to_string = tokenizer.convert_tokens_to_string(answer_tokens)

print ("\nFinal Answer : ")
print (answer_tokens_to_string)

```

Output is - 
Answer Tokens:
['some', 'states', 'would', 're', '##open', 'this', 'month']

Final Answer :
some states would reopen this month


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.