`CodeLlamaTokenizerFast` behavior changes permanently after encoding a string containing `"<FILL_ME>"`
Author: rfriel
Pull request: no

### System Info

- `transformers` version: 4.34.0.dev0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu118 (False)
- Tensorflow version (GPU?): 2.12.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (cpu)
- Jax version: 0.4.14
- JaxLib version: 0.4.14
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no



### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The `CodeLlamaTokenizerFast` tokenizer behaves differently after calling `.encode()` on a string containing `'<FILL_ME>'`.

Here's a very brief example showing the gist:

```python
>>> tokenizer = transformers.AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-hf")
>>>
>>> a = tokenizer.encode("foo")
>>> tokenizer.encode("first <FILL_ME> second")
>>> b = tokenizer.encode("foo")
>>>
>>> a == b
False
```

The specific effects I've noticed are:

1. The tokenizer no longer includes a prefix space
2. The tokenizer no longer includes the BOS token, even with `add_special_tokens=True`

It seems like maybe the tokenizer is going into a state where it behaves more like [`encode_infilling` from the Facebook repo](https://github.com/facebookresearch/codellama/blob/e064c1c24c377cc0875711440ef4c0a6eaf0147b/llama/tokenizer.py#L50-L52), and not properly exiting that state afterward?

The following script demonstrates the issue in more detail.

```python
from transformers import AutoTokenizer

model_name = "codellama/CodeLlama-7b-hf"

def show_tokens(token_ids):
    print()
    print(f"\ttokens IDs:             {token_ids}")
    print(f"\tstring representations: {test_tokenizer.convert_ids_to_tokens(token_ids)}")
    print()

def demo(use_fast: bool):
    for add_special_tokens in [False, True]:
        test_tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=use_fast)

        TEST_STR = 'foo'

        TEST_STR_FILL = "first <FILL_ME> second"

        token_lists, descriptions = [], []

        token_ids = test_tokenizer.encode(TEST_STR, add_special_tokens=add_special_tokens)
        print(f'Before <FILL_ME>\nCalling `tokenizer.encode({repr(TEST_STR)}, add_special_tokens={add_special_tokens})`')
        show_tokens(token_ids)

        test_tokenizer.encode(TEST_STR_FILL)

        token_ids = test_tokenizer.encode(TEST_STR, add_special_tokens=add_special_tokens)
        print(f'After <FILL_ME>\nCalling `tokenizer.encode({repr(TEST_STR)}, add_special_tokens={add_special_tokens})`')
        show_tokens(token_ids)

        print('---------------------------------------------------\n')

demo(use_fast=True)

demo(use_fast=False)
```

When we run the line `demo(use_fast=True)`, it prints:

```
Before <FILL_ME>
Calling `tokenizer.encode('foo', add_special_tokens=False)`

	tokens IDs:             [7953]
	string representations: ['▁foo']

After <FILL_ME>
Calling `tokenizer.encode('foo', add_special_tokens=False)`

	tokens IDs:             [5431]
	string representations: ['foo']

---------------------------------------------------

Before <FILL_ME>
Calling `tokenizer.encode('foo', add_special_tokens=True)`

	tokens IDs:             [1, 7953]
	string representations: ['
', '▁foo']

After <FILL_ME>
Calling `tokenizer.encode('foo', add_special_tokens=True)`

	tokens IDs:             [5431]
	string representations: ['foo']

---------------------------------------------------
```

That is, the tokenizer gives different outputs for the same inputs, depending on whether we have encoded a FILL_ME string yet or not.


The line `demo(use_fast=False)` prints:

```
before fill, add_special_tokens=False

	tokens IDs:             [7953]
	string representations: ['▁foo']

after fill, add_special_tokens=False

	tokens IDs:             [7953]
	string representations: ['▁foo']

---------------------------------------------------

before fill, add_special_tokens=True

	tokens IDs:             [1, 7953]
	string representations: ['
', '▁foo']

after fill, add_special_tokens=True

	tokens IDs:             [1, 7953]
	string representations: ['
', '▁foo']

---------------------------------------------------
```

So the slow tokenizer behaves consistently before and after FILL_ME.

### Expected behavior

The `encode` method should not modify the state of the tokenizer.

If I call `encode` multiple times, without doing anything else in between, I should expect the outputs to be independent of the order in which the calls are made.

Comments:
Good catch! The `set_infilling_processor` is missing it's early return statement!