QA pipeline fails when using fast tokenizer
Author: bdalal
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Linux-5.3.0-1032-gcp-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu 
 documentation: @sgugger
 -->
@LysandreJik @mfuntowicz 

## Information

Model I am using (Bert, XLNet ...): "sshleifer/tiny-distilbert-base-cased-distilled-squad"

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Modify unit test https://github.com/huggingface/transformers/blob/master/tests/test_pipelines.py#L644-L647 to use fast tokenizer
`nlp = pipeline(task="question-answering", model=model_name, tokenizer=(model_name, {"use_fast": True}))`
2. Run modified unit test

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
The test fails with error `ValueError: TextInputSequence must be str`

Complete failure result:
```
Testing started at 16:06 ...
Launching pytest with arguments test_pipelines.py::QAPipelineTests::test_torch_question_answering in /home/lttazz99/transformers_perf_tests/transformers/tests

============================= test session starts ==============================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /home/lttazz99/miniconda3/envs/qna/bin/python
cachedir: .pytest_cache
rootdir: /home/lttazz99/transformers_perf_tests/transformers
collected 1 item                                                               

test_pipelines.py::QAPipelineTests::test_torch_question_answering FAILED [100%]HuggingFace None
was None
founded None
in None
Paris. None

tests/test_pipelines.py:642 (QAPipelineTests.test_torch_question_answering)
self = <tests.test_pipelines.QAPipelineTests testMethod=test_torch_question_answering>

    @require_torch
    def test_torch_question_answering(self):
        for model_name in QA_FINETUNED_MODELS:
            nlp = pipeline(task="question-answering", model=model_name, tokenizer=(model_name, {"use_fast": True}))
>           self._test_qa_pipeline(nlp)

test_pipelines.py:647: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_pipelines.py:626: in _test_qa_pipeline
    mono_result = nlp(valid_inputs[0])
../src/transformers/pipelines.py:1675: in __call__
    tqdm_enabled=False,
../src/transformers/data/processors/squad.py:369: in squad_convert_examples_to_features
    is_training=is_training))
../src/transformers/data/processors/squad.py:165: in squad_convert_example_to_features
    return_token_type_ids=True,
../src/transformers/tokenization_utils_base.py:2043: in encode_plus
    **kwargs,
../src/transformers/tokenization_utils_fast.py:458: in _encode_plus
    **kwargs,
../src/transformers/tokenization_utils_fast.py:369: in _batch_encode_plus
    is_pretokenized=is_pretokenized,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Tokenizer(vocabulary_size=28996, model=BertWordPiece, unk_token=[UNK], sep_token=[SEP], cls_token=[CLS], pad_token=[PA...sk_token=[MASK], clean_text=True, handle_chinese_chars=True, strip_accents=None, lowercase=False, wordpieces_prefix=##)
sequence = [2777, 1108, 20164, 10932, 2271, 7954, ...]
pair = ['Hu', '##gging', '##F', '##ace', 'was', 'founded', ...]
is_pretokenized = False, add_special_tokens = True

    def encode(
        self,
        sequence: InputSequence,
        pair: Optional[InputSequence] = None,
        is_pretokenized: bool = False,
        add_special_tokens: bool = True,
    ) -> Encoding:
        """ Encode the given sequence and pair. This method can process raw text sequences as well
        as already pre-tokenized sequences.
    
        Args:
            sequence: InputSequence:
                The sequence we want to encode. This sequence can be either raw text or
                pre-tokenized, according to the `is_pretokenized` argument:
    
                - If `is_pretokenized=False`: `InputSequence` is expected to be `str`
                - If `is_pretokenized=True`: `InputSequence` is expected to be
                    `Union[List[str], Tuple[str]]`
    
            is_pretokenized: bool:
                Whether the input is already pre-tokenized.
    
            add_special_tokens: bool:
                Whether to add the special tokens while encoding.
    
        Returns:
            An Encoding
        """
        if sequence is None:
            raise ValueError("encode: `sequence` can't be `None`")
    
>       return self._tokenizer.encode(sequence, pair, is_pretokenized, add_special_tokens)
E       ValueError: TextInputSequence must be str

../../../miniconda3/envs/qna/lib/python3.7/site-packages/tokenizers/implementations/base_tokenizer.py:212: ValueError

Assertion failed

Assertion failed


=================================== FAILURES ===================================
________________ QAPipelineTests.test_torch_question_answering _________________

self = <tests.test_pipelines.QAPipelineTests testMethod=test_torch_question_answering>

    @require_torch
    def test_torch_question_answering(self):
        for model_name in QA_FINETUNED_MODELS:
            nlp = pipeline(task="question-answering", model=model_name, tokenizer=(model_name, {"use_fast": True}))
>           self._test_qa_pipeline(nlp)

test_pipelines.py:647: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_pipelines.py:626: in _test_qa_pipeline
    mono_result = nlp(valid_inputs[0])
../src/transformers/pipelines.py:1675: in __call__
    tqdm_enabled=False,
../src/transformers/data/processors/squad.py:369: in squad_convert_examples_to_features
    is_training=is_training))
../src/transformers/data/processors/squad.py:165: in squad_convert_example_to_features
    return_token_type_ids=True,
../src/transformers/tokenization_utils_base.py:2043: in encode_plus
    **kwargs,
../src/transformers/tokenization_utils_fast.py:458: in _encode_plus
    **kwargs,
../src/transformers/tokenization_utils_fast.py:369: in _batch_encode_plus
    is_pretokenized=is_pretokenized,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Tokenizer(vocabulary_size=28996, model=BertWordPiece, unk_token=[UNK], sep_token=[SEP], cls_token=[CLS], pad_token=[PA...sk_token=[MASK], clean_text=True, handle_chinese_chars=True, strip_accents=None, lowercase=False, wordpieces_prefix=##)
sequence = [2777, 1108, 20164, 10932, 2271, 7954, ...]
pair = ['Hu', '##gging', '##F', '##ace', 'was', 'founded', ...]
is_pretokenized = False, add_special_tokens = True

    def encode(
        self,
        sequence: InputSequence,
        pair: Optional[InputSequence] = None,
        is_pretokenized: bool = False,
        add_special_tokens: bool = True,
    ) -> Encoding:
        """ Encode the given sequence and pair. This method can process raw text sequences as well
        as already pre-tokenized sequences.
    
        Args:
            sequence: InputSequence:
                The sequence we want to encode. This sequence can be either raw text or
                pre-tokenized, according to the `is_pretokenized` argument:
    
                - If `is_pretokenized=False`: `InputSequence` is expected to be `str`
                - If `is_pretokenized=True`: `InputSequence` is expected to be
                    `Union[List[str], Tuple[str]]`
    
            is_pretokenized: bool:
                Whether the input is already pre-tokenized.
    
            add_special_tokens: bool:
                Whether to add the special tokens while encoding.
    
        Returns:
            An Encoding
        """
        if sequence is None:
            raise ValueError("encode: `sequence` can't be `None`")
    
>       return self._tokenizer.encode(sequence, pair, is_pretokenized, add_special_tokens)
E       ValueError: TextInputSequence must be str

../../../miniconda3/envs/qna/lib/python3.7/site-packages/tokenizers/implementations/base_tokenizer.py:212: ValueError
----------------------------- Captured stdout call -----------------------------
HuggingFace None
was None
founded None
in None
Paris. None
=============================== warnings summary ===============================
tests/test_pipelines.py::QAPipelineTests::test_torch_question_answering
  /home/lttazz99/transformers_perf_tests/transformers/src/transformers/tokenization_utils_base.py:1319: FutureWarning: The `max_len` attribute has been deprecated and will be removed in a future version, use `model_max_length` instead.
    FutureWarning,

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED test_pipelines.py::QAPipelineTests::test_torch_question_answering - Va...
========================= 1 failed, 1 warning in 2.64s =========================

Process finished with exit code 1

Assertion failed

Assertion failed
```


## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Test should pass.
---
NOTES:
This errors appears to be coming because [here](https://github.com/huggingface/transformers/blob/98ee802023a4db76879c761e7ce3677eb4555871/src/transformers/tokenization_utils_fast.py#L366) the inputs are the query tokens' ids and the context token text.
This in turn is passed to the [fast tokenizer](https://github.com/huggingface/tokenizers/blob/5d8728a26b654784572612ae119edc451205c2ff/bindings/python/py_src/tokenizers/implementations/base_tokenizer.py#L211) which expects `str` for both the sequence and the pair but instead gets a sequence of ints for the query and the text for the context.
The pipeline fails with when any Bert-like fast tokenizer is used irrespective of the model.
The unit test is the easiest way to reproduce this error.

Comments:
I also have run into this issue. @bdalal did you find a fix? I see the above PR, but seems you closed it. 

The fix is not very straightforward because it seems like both the tokenizers expose vastly different APIs and support very different functionalities. You can check this in `tokenization_utils_base.py` and `tokenization_utils_fast.py`.
In the context of the pipeline therefore, the tokenizer change is not transparent and requires a fair bit of work to integrate correctly.
I'll probably have a better fix up sometime next month if I do end up including the fast tokenizer in my project.

I've got a hacky working version here: https://github.com/bdalal/transformers
Also contains additional performance optimizations

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@bdalal  How does the link you provided help solving the issue?

@MHDBST you can navigate to the [QA pipeline](https://github.com/bdalal/transformers/blob/36a19915ea4fc3dc337a310e4a1af43eb3c81c9a/src/transformers/pipelines.py#L1656) and look at the changes I made to get them working. This was a while ago, so I don't exactly recall what changes I made.

That being said, I believe my code is now redundant because HF has come a long way in integrating fast tokenizers and they're now the default I believe. So you'll be better off just playing around with their latest release.