when I encode [unused1], return not one token
Author: jxyxiangyu
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using: tokenizer.encode('[unused1]')
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:relation extraction
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. tokenizer.encode("[unused1]")
2. but return not one token, if using keras-bert, it will return me only one token
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: least version
- Platform: 
- Python version: 3.7
- PyTorch version (GPU?):  1.1.0
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Comments:
I can reproduce this. cc @n1t0 @mfuntowicz Special "unused" tokens are not tokenised correctly. This happens in the fast tokenizers as well as the slow ones. See test case below.

```python
from transformers import BertTokenizer
USE_FAST = True

tokenizer = BertTokenizer.from_pretrained("bert-base-cased", use_fast=USE_FAST)
print('"[unused1]" in vocab?', "[unused1]" in tokenizer.vocab)
print('"[unused1]" index in vocab', tokenizer.vocab["[unused1]"] if "[unused1]" in tokenizer.vocab else "NA")

idxs = tokenizer.encode("[unused1]", add_special_tokens=False)
print("indices", idxs)
recoded = tokenizer.decode(idxs)
print("recoded", recoded)
```

Hi @jxyxiangyu, thanks for reporting this, thanks @BramVanroy to making a code to reproduce.

So far, the behavior you want to achieve needs to be done by deactivating the `do_basic_tokenize` feature on `BertTokenizer`, otherwise the input will be splitted on ponctuation chars before actually going through the wordpiece tokenizer.

_I don't think we have an equivalent on the Rust implementation of Bert, let me check internally and get back to you on this point._

Here a snippet of code which should achieve the desired behavior:

```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-cased", do_basic_tokenize=False)
tokenizer.tokenize("[unused1]")

>>> ['[unused1]']

tokenizer.encode("[unused1]", add_special_tokens=False)
>>> [1]

tokenizer.decode([1])
>>> '[unused1]'
```

Thanks for responding to my query. After I tried the method you gave, ‘[unused1]’ could indeed be tokenized correctly, but I want to use '[unused1]' to concatenate two words with little relation. In my opinion,  may I  set other words do_basic_tokenize as True, and '[unused1]' as False?

Hi @jxyxiangyu! Thank you @BramVanroy & @mfuntowicz for the help on this!

I think in this case the easiest way to handle this, is by adding the tokens you plan to use as special tokens. After all, that's what they are. They are not added by default since only a handful of them are actually used so you need to do it manually with
```python
tokenizer.add_special_tokens({ "additional_special_tokens": [ "[unused1]" ] })
```

Then, it should work for both fast and slow tokenizers:
```python
>>> from transformers import AutoTokenizer

>>> slow = AutoTokenizer.from_pretrained("bert-base-cased", use_fast=False)
>>> fast = AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True)

>>> slow.add_special_tokens({ "additional_special_tokens": [ "[unused1]" ] })
>>> fast.add_special_tokens({ "additional_special_tokens": [ "[unused1]" ] })

>>> slow.encode("[unused1]", add_special_tokens=False)
[1]
>>> fast.encode("[unused1]", add_special_tokens=False)
[1]
```

Thank you very much for your response, which solved my confusion

Awesome! Closing the issue, do not hesitate to reopen if needed!

Hi @n1t0 , I had a related question. I want to re-register a token's id in the vocab so that I don't need to add a new token and expand the size of the vocab. Specifically, for example I want to register "qazwsx" to id 1, which means I want to replace "[unused1]" : 1 by "qazwsx" : 1. Do you know how to achieve this?
Another question is how to synchronize two tokens with the same id, to avoid expanding the size of vocab. For example, instead of replacing [unused1]" : 1 by "qazwsx" : 1, I want to keep both in the new vocab.

Thank you so much for the help!