Loading a tokenizer from the Tokenizers library doesn't transfer over padding/truncation behavior correctly
Author: Ubadub
Pull request: no
Labels: name: Core: Tokenization

### System Info

Not especially relevant, but included for completeness:

- `transformers` version: 4.29.2
- Platform: macOS-13.2-x86_64-i386-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

(n.b.: originally posted a similar query in the [transformers forum](https://discuss.huggingface.co/t/padding-not-working-when-loading-a-tokenizer-trained-via-the-tokenizers-library-into-transformers/42326/1) but got no answer there.)

I trained a simple WhitespaceSplit/WordLevel tokenizer using the `tokenizers` library. I added padding by calling `enable_padding(pad_token="<pad>")` on the Tokenizer instance. Then I saved it to a JSON file and then loaded it into transformers using [the instructions here](https://huggingface.co/docs/transformers/fast_tokenizers):

```py
fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")
```

When using the `tokenizers.Tokenizer` object directly, `encode` correctly adds the padding tokens. However, if I try padding when tokenizing using the `PreTrainedTokenizerFast` instance, I get the exception:

```py
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
```

Sure enough, if I follow the instructions and add the pad token as a special token, it works. Alternatively, I can pass the argument `pad_token="<pad>"` to the `PreTrainedTokenizerFast` constructor call, to the same effect.

To reproduce the problem, you can use the code below. Most of it is from the [tokenizers Quicktour](https://huggingface.co/docs/tokenizers/quicktour), so you'll need to download the data files as per the instructions there (or modify `files` if using your own files). The rest is from the official transformers docs on [how to load a tokenizer from `tokenizers` into `transformers`](https://huggingface.co/docs/transformers/fast_tokenizers):

```py
from tokenizers import BpeTrainer, Tokenizer
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import Whitespace
from transformers import PreTrainedTokenizerFast

files = [f"data/wikitext-103-raw/wiki.{split}.raw" for split in ["test", "train", "valid"]]
sentences = ["Hello, y'all!", "How are you 😁 ?"]

tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
tokenizer.pre_tokenizer = Whitespace()

tokenizer.train(files, trainer)

# Enable padding
tokenizer.enable_padding(pad_id=3, pad_token="[PAD]")

# Now use this tokenizer to tokenize a couple of sentences.
output = tokenizer.encode_batch(sentences)

# The output is padded, as it should be:
print(output[0].tokens)
# ['Hello', ',', 'y', "'", 'all', '!']
print(output[1].tokens)
# ['How', 'are', 'you', '[UNK]', '?', '[PAD]']

# But now let's say we load the tokenizer into transformers- let's try loading it directly from the tokenizer object:

fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)

# Tokenize two strings of different token length with padding
fast_output = fast_tokenizer(sentences, padding=True)
```

This gives us the error:
```
Using pad_token, but it is not set yet.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2548, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2634, in _call_one
    return self.batch_encode_plus(
  File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2816, in batch_encode_plus
    padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies(
  File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2453, in _get_padding_truncation_strategies
    raise ValueError(
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
```

We can resolve the issue by explicitly specifying the special tokens when initializing the `PreTrainedTokenizerFast`:

```py
fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer,  pad_token="[PAD]", unk_token="[UNK]")

# Now padding works as expected
fast_output = fast_tokenizer(sentences, padding=True)

print(fast_output[0].tokens)
# ['Hello', ',', 'y', "'", 'all', '!']
print(fast_output[1].tokens)
# ['How', 'are', 'you', '[UNK]', '?', '[PAD]']
```

The code above uses the `tokenizer_object` parameter to load the fast tokenizer as a `PreTrainedTokenizerFast` instance, but as you can confirm for yourselves, the same behavior occurs if you first save the tokenizer to file, then load it into `PreTrainedTokenizerFast` using the `tokenizer_file` parameter instead.

**First, I wanted to check- am I doing something wrong/missing something? Or is this just how it works?**

If the latter, as follows, an explanation of how I feel it should work and why.

### Expected behavior

I understand that I can get the desired behavior by either:

1. Add the pad token as a special token i.e. `fast_tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
2. Alternatively, I can pass the argument `pad_token='[PAD]'` to the `PreTrainedTokenizerFast` constructor call, to the same effect.

But I want the tokenizer to work *out of the box identically as the `tokenizer.Tokenizer` instance does* (to the extent that is reasonably possible), including in terms of padding behavior

I find it confusing and awkward that I have to enable padding for the `tokenizer.Tokenizer` instance, and then *again* for the `PreTrainedTokenizerFast` instance.

Imagine if your system architecture/workflow has two entirely different processes for tokenizing a document vs. training a model on it using `transformers` (as I imagine is often the case for people). Then you would need to hardcode the pad token in both locations, and if for some reason you wanted to change it, also update it in both locations.

On the other hand, if `PreTrainedTokenizerFast` really behaved exactly like the fast tokenizer it was created from, the training code could be entirely agnostic to how the tokenizer was created. All it would need was a path to the saved tokenizer config, and it could proceed without needing to know anything else. This is the behavior I think most people would naturally expect.

It could make sense to keep the `pad_token` parameter in the `PreTrainedTokenizerFast` *as an optional override*, or for cases where the fast tokenizer didn't have a padding token set, but the default should be to copy over the padding behavior as-is.

Put another way, the tokenizer object/config file should uniquely determine the tokenization behavior of a tokenizer, whether it is a `tokenizers.Tokenizer` instance or its equivalent `PreTrainedTokenizerFast` (to the extent it can; I understand some misalignment is probably inevitable, but this seems not to be one of those cases).

**Bottom line:** If the padding information is already in the tokenizer (or in the saved tokenizer config file), you should not need to explicitly specify the padding token again when transferring the tokenizer. This introduces a lot of totally unnecessary friction and leads to brittle code. The tokenizer object/config should be self-contained (i.e. I should not need to hardcode the pad token in two places), and information already encapsulated in the tokenizer object or its saved config file should be preserved on transfer.

EDIT: I later observed that the same behavior is true of truncation. See my followup comment for what I believe to be the responsible section of code.

Comments:
The responsible bit of code seems to be [in the `set_truncation_and_padding` function](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_fast.py#L319) of `PreTrainedTokenizerFast`.  And it affects truncation, too, not just padding. Basically, this function erases the backend fast tokenizer's truncation/padding strategy, and then adds in the user-supplied overrides (e.g. as passed via `encode`).

This seems to me to be the opposite of what we want. The truncation/padding strategy should *start* with the backend fast tokenizer's truncation/padding strategy as-is, and if override arguments are provided, then those and only those arguments should be selectively overriden.

Do the devs have thoughts on this? I am working on implementing the changes that I am proposing, but curious if there is a reason it was done this way.

Also, the function docstring for that function says

> The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section.

What does "managed section" refer to?

Hey! Thanks for reporting, I’ll have a look asap 

I should be able to get to this in the coming weeks, I don't think the fix is complicated as you isolated well. 

Hey! Started working on this, the problem is in the initialization. This is tricky because  potentially breaking. Will be adding tests to make sure this is fixed

Fixed it! Make sure to use the latest version of transformers