`run_squad_trainer` doesn't actually use a Rust tokenizer + errors in `squad_convert_example_to_features` when using a Rust tokenizer
Author: k8si
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Platform: Linux-4.14.35-1902.303.4.1.el7uek.x86_64-x86_64-with-oracle-7.8
- Python version: 3.6.8
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

and

- `transformers` version: 3.3.1
- Platform: macOS-10.15.6-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@mfuntowicz
@LysandreJik
@patil-suraj 

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

Firstly, in `run_squad_trainer.py`, I noticed that the "use_fast" arg doesn't get propagated into the tokenizer instantiation: https://github.com/huggingface/transformers/blob/0acd1ffa09a06084efa7cfa0e4e9d97cffdda5f9/examples/question-answering/run_squad_trainer.py#L107  Probably should be 
```
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=model_args.use_fast
    )
```
However, when I make that change, the script hangs at the call to `squad_convert_examples_to_features` in SquadProcessor. 

So, I did a little digging. The error is in `squad_convert_example_to_features` and seems to be due to inconsistencies in the behavior of `tokenizer.encode_plus` between the Python and Rust tokenizers, detailed below. I've also provided a [gist](https://gist.github.com/k8si/a143346dfa875c28d98e95cba1f82f1b) that hopefully elucidates & will help reproduce each of these points. I tested both BertTokenizer/BertTokenizerFast and GPT2Tokenizer/GPT2TokenizerFast.

1) Python tokenizers handle negative values for `stride`, Rust tokenizers throw an exception (`OverflowError: can't convert negative int to unsigned`)

2) For sequence pairs, Python tokenizers are fine if the first arg (`text`) is a list of ints and the second arg (`text_pair`) is a list of strings. The Rust tokenizers throw an exception `ValueError: PreTokenizedInputSequence must be Union[List[str], Tuple[str]]`. (Furthermore, the typehints for these arguments indicate that a string, a list of strings, or a list of ints are all fine.)

3) Leaving the `is_split_into_words` kwarg at its default value (`False`), then running `tokenizer.encode_plus(list of ints)` works fine for the Python tokenizers. The Rust tokenizers raise an exception `ValueError: TextInputSequence must be str`.

4) When running on a pair of sequences and setting `return_tensors=None`, the Python tokenizers return an output dict with input_ids (and other elements) as a list of ints i.e.`input_ids = [id1, id2, ...]` whereas the Rust tokenizers return a dict with input_ids as a list of list of ints i.e. `input_ids = [[id1, id2, ...]]`. I also noticed that if you set `return_tensors="pt"`, both the Python and Rust tokenizers return `input_ids = tensor([[id1, id2, ...]])`. 

5) When `return_overflowing_tokens=True`, the Python tokenizers return a list of the overflowing tokens at key `overflowing_tokens` as expected. The Rust tokenizers return them at key `overflow_to_sample_mapping` which is not documented anywhere, as far as I can tell. The values seem to be different for the Python output vs. Rust output.

6) Running the same procedure on the same input twice produces the same result each time for the Python tokenizer. For the Rust tokenizer, the result of the second run is **different**. I am not familiar enough with the Rust tokenizer internals at this point to have a theory as to why this is the case. Anyway, this is the point at which I stopped debugging and decided to file an issue.

## To reproduce

Steps to reproduce the behavior:

1. Download squad 2.0 dataset from ["official" squad website](https://rajpurkar.github.io/SQuAD-explorer/)
2. Make fix in `run_squad_training.py` described above to correctly instantiate a Rust tokenizer
3. Run script: `python examples/question-answering/run_squad_trainer.py --model_name_or_path bert-base-uncased --use_fast --output_dir "./outputs-squad" --do_train --data_dir "./squad-data" --version_2_with_negative`

Also see gist detailing issues described above: https://gist.github.com/k8si/a143346dfa875c28d98e95cba1f82f1b

## Expected behavior

1) I expected `run_squad_trainer.py` to use a Rust tokenizer when the `use_fast` arg was set to True
2) I expected `SquadProcessor.squad_convert_example_to_features` to not raise exceptions when processing squad data when using a Rust tokenizer
3) I expected `tokenizer.encode_plus` to return the same outputs given the same inputs, regardless of whether the tokenizer is a Rust tokenizer or a Python tokenizer


Comments:
Hello! Indeed, the Rust tokenizers are not handled by the SQuAD data processing. This is one item we would like to resolve when refactoring the data processing methods, which will soon be implemented directly in `datasets` rather than in `transformers`.

Thank you for your detailed issue!

For what it's worth, my main issue is with the behavioral issues with the Python vs. Rust tokenizers, not with the SQuAD data processing itself (I can easily write my own SQuAD processor but writing my own tokenizers is more work--the tokenizers are part of why I've been using the library in the first place). 

It places a sizeable burden on people coding against the library that different kwargs result in completely different behaviors (and different errors) between the Rust vs. Python implementations, and these differences often aren't documented. And Item # 6 seems like a fundamental error somewhere in the Rust codebase.

Are there any plans to address these issues in the near future? For what it's worth, I've never had an issue with the Python tokenizers. I'd like to use the Rust ones because they're Fast, plus I can train them myself easily, but navigating the API weirdness has been a slog.

You're correct that there is currently a mismatch between the python and rust tokenizers, and thank you for writing such a detailed issue explaining all of your pain points. We'll keep a close eye to the issues mentioned here as we continue working and improving the compatibility between the two APIs, which is something we will be focusing on in the near future.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.