Unexpected result `tok.convert_tokens_to_string(["café"]) == 'mycaf�' for some tokenizers like gpt2
Author: trianxy
Pull request: no
Labels: name: Core: Tokenization

name: bug

### System Info

- `transformers` version: 4.44.2
- Platform: macOS-14.7-arm64-arm-64bit
- Python version: 3.11.5
- Huggingface_hub version: 0.25.0
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: NO

### Who can help?

@ArthurZucker @itazap

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2", use_fast=False)

print(tok.convert_tokens_to_string(["café"]))  # -> error 1: prints 'caf�' instead of expected `café`

print(tok.encode(tok.decode([113]))[0])  # error 2: prints 4210 (the id of '�') instead of expected 113 (related to error1)
```

### Expected behavior

The expected behavior is mentioned above inside `Reproduction`. The same behavior appears for a few other models, which use similar code to `gpt2`.

### Cause
The [line](https://github.com/huggingface/transformers/blob/5af7d41e49bbfc8319f462eb45253dcb3863dfb7/src/transformers/models/gpt2/tokenization_gpt2.py#L295)

```python
        text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors)
```

uses `errors="replace"` which will replace `é` with `�` since `é` is not inside the dictionary `self.byte_encoder`.

### Possible solutions
If others also feel that this is a problem, then there are a few ways to improve this behavior. I can create a PR if you like:
1. Improve the docstring of `tok.decode()`, similarly to how the library `tiktoken` includes a [WARNING](https://github.com/openai/tiktoken/blob/c0ba74c238d18b4824c25f3c27fc8698055b9a76/tiktoken/core.py#L249) that this operation is lossy
2. Improve the docstring of `tok.convert_tokens_to_string()` to have a warning that you should ONLY input tokens (!) (i.e. strings which are represented by a specific token_id)
3. Finally, I can refactor `tok.convert_tokens_to_string()` to work for all strings, but I am not sure if that's worth it, because this would need to be done at a lot of models. And it might break production code which may rely on the above (wrong?) behavior.

Comments:
Thanks for your input! 😊 `convert_tokens_to_string` expects a sequence of **tokens** passed in which implies that the text  has already been tokenized to produce these tokens. This function is useful for making sense of tokens by returning a coherent string / phrase. As you suggested, `café` is not a token as it is not in tokenized format.  For example you can use it to do: `tok.convert_tokens_to_string(tok.tokenize("café"))`, which would pass in the token sequence! Hope this helps!

Thank you @itazap for the support and your time!

I agree on how the function can/should be used. 

I just felt that we could (at least) improve the wording in the docstrings of `tok.convert_tokens_to_string(...)` and `tok.decode(...)`, as mentioned above. I had tripped over these 2 functions and I am afraid others have/will, too.

However, if others don't feel there is a need to change anything, then I am fine if the issue is closed.

Thanks @trianxy , yes it can be more explicitly described that `convert_tokens_to_string ` should be used on a list of tokens, while `decode` on a list of strings. Feel free to open a PR for the docstrings if you'd like! 🤗 

Sounds good. I'll look to add a PR soonish

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.