Inconsistencies and possible bugs in different tokenizers
Author: alanakbik
Pull request: no
Labels: name: Core: Tokenization

name: Should Fix

# 🐛 Bug

## Information

Over in [Flair](https://github.com/flairNLP/flair/pull/1494) we are integrating your awesome library in our embeddings interfaces. We are using the `AutoTokenizer` class to create one interface for all embeddings. We use the tokenizers to encode strings with special tokens. 

However, we note some inconsistencies: (1) Most, but not all encodings do not include the BOS token and (2) some encodings behave differently depending on how the tokenizer is called. In both cases, this is detrimental to downstream task performance. 

This can be reproduced with the following script: 

```python
from transformers import AutoTokenizer

# example string to tokenize
text = "CRICKET1 MATCH"

# different models
for tokenizer_name in [
    'bert-base-cased',
    'openai-gpt',
    'transfo-xl-wt103',
    'gpt2',
    'xlnet-base-cased',
    'xlm-mlm-ende-1024',
    'roberta-base',
    'distilbert-base-uncased',
    'ctrl',
    'camembert-base',
    'albert-base-v2',
    'xlm-roberta-base',
    'distilgpt2',
    'bart-large',
    'distilroberta-base',
]:

    # for each tokenizer model, print name and result of checks
    print('------------')
    print(tokenizer_name)
    print('------------')

    # get tokenizer
    tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)

    # method 1: tokenizer.encode() with add_special_tokens=True
    ids = tokenizer.encode(text, add_special_tokens=True)
    subtokens_encode_special = tokenizer.convert_ids_to_tokens(ids)

    # method 2: tokenizer.encode() with add_special_tokens=False and subsequent build_inputs_with_special_tokens()
    ids = tokenizer.encode(text, add_special_tokens=False)
    ids_extended = tokenizer.build_inputs_with_special_tokens(ids)
    subtokens_encode_and_build = tokenizer.convert_ids_to_tokens(ids_extended)

    # check if both methods yield the same result
    if subtokens_encode_special != subtokens_encode_and_build:
        print("DIFFERENCE IN ENCODING!")
        print(f'Method 1 - Encode (+ special):\t{str(subtokens_encode_special)}')
        print(f'Method 2 - Encode and build:  \t{str(subtokens_encode_and_build)}')

    # check if the BOS token is included
    bos_token = tokenizer.bos_token
    if bos_token and bos_token not in subtokens_encode_and_build:
        print("DOES NOT CONTAIN BOS TOKEN!")
        print(f"BOS token '{bos_token}' not in {str(subtokens_encode_and_build)}")
```

This outputs the following inconsistencies, at least some of which likely are bugs.

There are two encodings that do not contain the BOS token: 

```console
------------
xlm-mlm-ende-1024
------------
DOES NOT CONTAIN BOS TOKEN!
BOS token '
' not in ['
', 'crick', 'et', '1</w>', 'match</w>', '
']
```

So, the XLM encoding of the string "CRICKET1 MATCH" strangely starts with a **`
`** (EOS) even though it should probably start with a **`
`**. 

```console
------------
xlnet-base-cased
------------
DOES NOT CONTAIN BOS TOKEN!
BOS token '
' not in ['▁CR', 'ICK', 'ET', '1', '▁M', 'ATCH', '<sep>', '<cls>']
```

XLNet encoding does not contain BOS and EOS at all. This is consistent with the documentation but is detrimental to performance. In our experiments, it works a lot better if we include `
` and `
` in the sequence. 

There are also two tokenizers for which the two methods (encode with special tokens and encode and build) give slightly different results, namely  RoBERTa and BART:

```console
------------
roberta-base
------------
DIFFERENCE IN ENCODING!
Method 1 - Encode (+ special):	['
', 'ĠCR', 'ICK', 'ET', '1', 'ĠM', 'ATCH', '
']
Method 2 - Encode and build:   	['
', 'CR', 'ICK', 'ET', '1', 'ĠM', 'ATCH', '
']
```

This was already noted by @stefan-it in #1196 and strangely, even though the tokenization output by method 1 seems to make more sense, method 2 gives better results. 

## Expected behavior

Consistent output :) 

## Environment info     
- `transformers` version: 2.8
- Platform: Ubuntu
- Python version: 3.7
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: o


Comments:
Hi, and thanks for your report! Indeed some of these seem to be bugs.

## XLM

This seems to be a bug. It doesn't behave as the documentation says, I'm looking into it. Encoding sequences with the bos token instead of the cls token should do the trick.

## XLNet

The way XLNet encode sequences is with the format `A <sep> B <sep> <cls>`, as it can be seen in the [original repository](https://github.com/zihangdai/xlnet/blob/0b642d14dd8aec7f1e1ecbf7d6942d5faa6be1f0/data_utils.py#L481-L487). I'm not finding any usage of the `
` and `
` tokens in the repo, even though they're declared.
It's interesting that you obtained better results when using `
` and `
`!

## RoBERTa and BART

This is a slightly more complicated issue. The #1196 issue describes it well, and the https://github.com/huggingface/transformers/pull/2778 PR addresses this as well. Here the correct tokenization is the first one:

```
['
', 'ĠCR', 'ICK', 'ET', '1', 'ĠM', 'ATCH', '
']
```

It is interesting that you're getting better results with the second one. I believe the original implementation outputs the same results as the result pasted above. I'm guessing you obtained the second result with the following:

```py
tokenizer.build_inputs_with_special_tokens(
    tokenizer.encode('She likes <mask> cats.', add_special_tokens=False)
)
```

which indeed yields the tokenization you mentioned. This happens because when encoding without special tokens, no space is added between the initial special token and the first sequence token (seeing as there's not special tokens). When using this method, you would need to specify you want that prefix space so that it adds it. You can do so with the `add_prefix_space` option for the `encode` method:

```py
tokenizer.build_inputs_with_special_tokens(
    tokenizer.encode('She likes <mask> cats.', add_special_tokens=False, add_prefix_space=True)
)
```

This yields the same results as the first method. Let me know if I can be of further help.


Thanks, that clarifies it. 

You're right that the XLNet implementation declares the ` 
` and `
` but then does not seem to use them, which is strange. Also strange that we are seeing better results with these tags but this could also be a problem in our code. Perhaps you could then set the `tokenizer.bos_token` and `tokenizer.eos_token` fields to `None` for the `XLNetTokenizer` if they are not used?