T5 Tokenization of unique masked tokens (<extra_id_1>) is incorrect
Author: mansimov
Pull request: no

Hi!

Thanks for the awesome library!

I am trying to tokenize the following text using the T5Tokenizer `tokenizer = T5Tokenizer.from_pretrained("t5-base")` 

`text = "The dog <extra_id_1> in the park"`
`tokenized_text = tokenizer.tokenize(text)`
`print (tokenized_text)`

And get the following output:
`['▁The', '▁dog', '▁', '<', 'extra', '_', 'i', 'd', '_', '1', '>', '▁in', '▁the', '▁park']`

The `<extra_id_1>` is tokenized incorrectly. Any idea on how to solve this issue?

I am using transformers version 2.8.0 and tokenizers version 0.7.0





Comments:
I tried the following code:
```python
tokenizer = T5Tokenizer.from_pretrained("t5-base")

text = "The dog <extra_id_1> in the park"
tokenized_text = tokenizer.tokenize(text)
print (tokenized_text)
```

With the same settings, it works fine with the following output:

```python
['▁The', '▁dog', '<extra_id_1>', '▁in', '▁the', '▁park']
```

FYI, I just installed transformers on Google Colab using `pip install transformers`.

@girishponkiya thanks for comment!

which version of tokenizers and transformers are you using ?

torch: 1.5.0+cu101
transformers: 2.8.0
tokenizers: 0.7.0

I tried with the following environment as well:

-  torch: 1.5.0+cu101
-  transformers: 2.8.0
-  tokenizers: **0.5.2**

..and got the following output:
```
['▁The', '▁dog', '<extra_id_1>', '▁in', '▁the', '▁park']
```

We found the same problem.

By gitting bisect, the following commit raises this ?

```
* commit 96ab75b8dd48a9384a74ba4307a4ebfb197343cd
| Author: Funtowicz Morgan <mfuntowicz@users.noreply.github.com>
| Date:   Mon Apr 6 22:29:15 2020 +0000
(Pull Request # 3185)
```

In src/transformers/tokenization_utils.py # 272 (at the current master, Line 500),
https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L500
Did you forget "setattr(self, key, value)" after assert.

See:
https://github.com/huggingface/transformers/commit/96ab75b8dd48a9384a74ba4307a4ebfb197343cd#diff-66606105b55d0ec62ae34112ea3e3d20R272
(please scroll down tokenization_utils.py and "Load diff")

Best,

P.S.) bisect results
```
$ git bisect log
git bisect start
# bad: [b1ff0b2ae7d368b7db3a8a8472a29cc195d278d8] Fix bug in examples: double wrap into DataParallel during eval
git bisect bad b1ff0b2ae7d368b7db3a8a8472a29cc195d278d8
# good: [e52d1258e010b88d3507a4f527c6201616c119ad] Fix RoBERTa/XLNet Pad Token in run_multiple_choice.py (#3631)
git bisect good e52d1258e010b88d3507a4f527c6201616c119ad
# bad: [c59b1e682d6ebaf7295c63418d4570228904e690] [examples] unit test for run_bart_sum (#3544)
git bisect bad c59b1e682d6ebaf7295c63418d4570228904e690
# bad: [31baeed614bf7f65aafde545f20a95e84cd293b4] Update quotes
git bisect bad 31baeed614bf7f65aafde545f20a95e84cd293b4
# bad: [e344e3d4021421ec0d631d076daf17f8a4e82e69] [examples] SummarizationDataset cleanup (#3451)
git bisect bad e344e3d4021421ec0d631d076daf17f8a4e82e69
# bad: [5aa8a278a3f13b8f83a0deb9b6d743f159cea23c] Fix roberta checkpoint conversion script (#3642)
git bisect bad 5aa8a278a3f13b8f83a0deb9b6d743f159cea23c
# bad: [0a9d09b42a9c7c1ccc00da48486a1188078e8594] fixed TransfoXLLMHeadModel documentation (#3661)
git bisect bad 0a9d09b42a9c7c1ccc00da48486a1188078e8594
# bad: [96ab75b8dd48a9384a74ba4307a4ebfb197343cd] Tokenizers v3.0.0 (#3185)
git bisect bad 96ab75b8dd48a9384a74ba4307a4ebfb197343cd
# first bad commit: [96ab75b8dd48a9384a74ba4307a4ebfb197343cd] Tokenizers v3.0.0 (#3185)
```

@takahiro971 

Here is the output of me running `git bisect log`
```
git bisect start

# bad: [b1ff0b2ae7d368b7db3a8a8472a29cc195d278d8] Fix bug in examples: double wrap into DataParallel during eval
git bisect bad b1ff0b2ae7d368b7db3a8a8472a29cc195d278d8
# good: [e52d1258e010b88d3507a4f527c6201616c119ad] Fix RoBERTa/XLNet Pad Token in run_multiple_choice.py (#3631)
git bisect good e52d1258e010b88d3507a4f527c6201616c119ad
```

Can you point out how git bisect solves this issue?
Thanks!

@mansimov

Hi!
The 'git bisect' is a tool for determine when (which commit) contains bugs.

This issue is not still solved.

Old version (before at Apr 6) is right.
But the commit 96ab75b8dd48a9384a74ba4307a4ebfb197343cd have a bug and raises this issue.

Also, the current master (latest source) have the same problem.

BTW, How did you installed transformer?
clone from GitHub and 'pip install --editable .' ?
or 'pip install transformers==2.8.0' ?

If first one (and use newer version), it may have the bug.


We can three choices:

1) Use older version (before above commit on Apr 6).
  or 'pip install transformers==2.8.0' is also OK, because it seems older than the above commit.
2) Clone repo to local and,  Hack the transformers code.
3) Wait until somebody fix this bug, and release next version.

Best,


FYI; my hack is like:

```
$ g log -p
(snip)
Date:   Fri May 8 10:52:13 2020 +0900

    不具合修正

diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
index a2d258a..91f345b 100644
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -498,6 +498,7 @@ class SpecialTokensMixin:
             if key in self.SPECIAL_TOKENS_ATTRIBUTES:
                 if key == "additional_special_tokens":
                     assert isinstance(value, (list, tuple)) and all(isinstance(t, str) for t in value)
+                    setattr(self, key, value)
                 elif isinstance(value, AddedTokenFast):
                     setattr(self, key, str(value))
                 elif isinstance(value, str):

```

Be careful, the line number may be changed in the different revision.

Thanks @takahiro971 this solved my issue!

I think this issue is solved with #4353