Mistral Tokenizer.decode() add a space when use_fast=True
Author: HugeHeart
Pull request: no

```python
# transformers version is 4.38.2

from transformers import AutoTokenizer
model_path="mistralai/Mistral-7B-v0.1"
fast = AutoTokenizer.from_pretrained(model_path, use_fast=True)
slow = AutoTokenizer.from_pretrained(model_path, use_fast=False)
text = "hi"
print(f"fast tokenize={fast.encode(text)}")
print(f"slow tokenize={slow.encode(text)}")

# fast decode add a space after 

print(f"fast decode={fast.decode(fast.encode(text))}")
print(f"slow decode={slow.decode(slow.encode(text))}")
```

output is
```
fast tokenize=[1, 12014]
slow tokenize=[1, 12014]
fast decode=
 hi
slow decode=
hi
```

fast and slow tokenizer have same encode result, but fast add a space after "\<s\>" when decode

I also noticed issue huggingface/tokenizers#1448 , @ArthurZucker said "use metaspace with prepend_scheme="first" and no normalizer", this already exists in transformers version 4.38.2 and doesn't seem to work. 

Are there any useful info I missed? how can I delete the space after "\<s\>" when use_fast=True? 

Thank you to those who have contributed to the Transformers lib.

Comments:
Hey! Thanks for reporting. 
If you check this: 
```python 
In [4]: slow.convert_ids_to_tokens([1, 12014])
Out[4]: ['
', '▁hi']
```
This issue emerges because `convert_tokens_to_string` is "wrong":
```python 
In [14]: slow.convert_tokens_to_string(['
', '▁hi'])
Out[14]: '
hi'
```

```diff
    def convert_tokens_to_string(self, tokens):
        """Converts a sequence of tokens (string) in a single string."""
        # since we manually add the prefix space, we have to remove it when decoding
        if tokens[0].startswith(SPIECE_UNDERLINE) and self.add_prefix_space:
            tokens[0] = tokens[0][1:]

        current_sub_tokens = []
        out_string = ""
        prev_is_special = False
        for i, token in enumerate(tokens):
            # make sure that special tokens are not decoded using sentencepiece model
            if token in self.all_special_tokens:
                if not prev_is_special and i != 0 and self.legacy:
                    out_string += " "
                out_string += self.sp_model.decode(current_sub_tokens) + token
                prev_is_special = True
                current_sub_tokens = []
            else:
+               if prev_is_special and i==1 and self.add_prefix_space:
+                   out_string += " "
                current_sub_tokens.append(token)
                prev_is_special = False
        out_string += self.sp_model.decode(current_sub_tokens)
        return out_string
```
this should fix it. 
Now this should bring both implementation closer while keeping:
```python
In [4]: slow.convert_tokens_to_string(['▁hi'])
Out[4]: 'hi'
```



> Hey! Thanks for reporting. If you check this:
> 
> ```python
> In [4]: slow.convert_ids_to_tokens([1, 12014])
> Out[4]: ['
', '▁hi']
> ```
> 
> This issue emerges because `convert_tokens_to_string` is "wrong":
> 
> ```python
> In [14]: slow.convert_tokens_to_string(['
', '▁hi'])
> Out[14]: '
hi'
> ```
> 
>  ```diff
>     def convert_tokens_to_string(self, tokens):
>         """Converts a sequence of tokens (string) in a single string."""
>         # since we manually add the prefix space, we have to remove it when decoding
>         if tokens[0].startswith(SPIECE_UNDERLINE) and self.add_prefix_space:
>             tokens[0] = tokens[0][1:]
> 
>         current_sub_tokens = []
>         out_string = ""
>         prev_is_special = False
>         for i, token in enumerate(tokens):
>             # make sure that special tokens are not decoded using sentencepiece model
>             if token in self.all_special_tokens:
>                 if not prev_is_special and i != 0 and self.legacy:
>                     out_string += " "
>                 out_string += self.sp_model.decode(current_sub_tokens) + token
>                 prev_is_special = True
>                 current_sub_tokens = []
>             else:
> +               if prev_is_special and i==1 and self.add_prefix_space:
> +                   out_string += " "
>                 current_sub_tokens.append(token)
>                 prev_is_special = False
>         out_string += self.sp_model.decode(current_sub_tokens)
>         return out_string
> ```
> 
> this should fix it. Now this should bring both implementation closer while keeping:
> 
> ```python
> In [4]: slow.convert_tokens_to_string(['▁hi'])
> Out[4]: 'hi'
> ```

Thanks for your reply!
When a text contains special tokens, I want the text after tokenizer's encode and decode to be consistent with the original text.
But even with the code you mentioned, I still can't achieve this effect.

Here is an example:
```python
from transformers import AutoTokenizer
model_path="mistralai/Mistral-7B-v0.1"
fast = AutoTokenizer.from_pretrained(model_path, use_fast=True, add_bos_token=False)
slow = AutoTokenizer.from_pretrained(model_path, use_fast=False, add_bos_token=False)
text = "
user: hi
assistant: hello
"
print(f"text={text}")

print(f"fast tokenize={fast.encode(text)}")
print(f"slow tokenize={slow.encode(text)}")

fast_decode_text = fast.decode(fast.encode(text))
slow_decode_text = slow.decode(slow.encode(text))
print(f"fast decode text={fast_decode_text}")
print(f"slow decode text={slow_decode_text}")
```

output:
```
text=
user: hi
assistant: hello

fast tokenize=[1, 2188, 28747, 12014, 2, 13892, 28747, 6312, 28709, 2]
slow tokenize=[1, 2188, 28747, 12014, 2, 13892, 28747, 6312, 28709, 2]
fast decode text=
 user: hi
 assistant: hello

slow decode text=
  user: hi
 assistant: hello

```

fast/slow decode text add space after special tokens.

Is there any parameter that can be controlled not to add space after special tokens?

That is expected 😉 
The problem here is mentioned in #28881 which is a PR suppose to fix it. 
You can fix this with: 
```python 
slow = AutoTokenizer.from_pretrained(model_path, use_fast=False, add_bos_token=False, legacy=False)
```

check the doc for this argument 🤗 

> That is expected 😉 The problem here is mentioned in #28881 which is a PR suppose to fix it. You can fix this with:
> 
> ```python
> slow = AutoTokenizer.from_pretrained(model_path, use_fast=False, add_bos_token=False, legacy=False)
> ```

Thank you for your prompt response. But when use_fast=True, space will still appear after sp tokens.

Here is an example:
```python
from transformers import AutoTokenizer
model_path="mistralai/Mistral-7B-v0.1"
fast = AutoTokenizer.from_pretrained(model_path, use_fast=True, add_bos_token=False, legacy=False)
slow = AutoTokenizer.from_pretrained(model_path, use_fast=False, add_bos_token=False, legacy=False)
text = "
user: hi
assistant: hello
"
print(f"text={text}")

print(f"fast tokenize={fast.encode(text)}")
print(f"slow tokenize={slow.encode(text)}")

fast_decode_text = fast.decode(fast.encode(text))
slow_decode_text = slow.decode(slow.encode(text))
# a space appear between "
" and "assistant"
print(f"fast decode text={fast_decode_text}")
print(f"slow decode text={slow_decode_text}")
```
output is
```
text=
user: hi
assistant: hello

fast tokenize=[1, 2188, 28747, 12014, 2, 13892, 28747, 6312, 28709, 2]
slow tokenize=[1, 1838, 28747, 12014, 2, 489, 11143, 28747, 6312, 28709, 2]
fast decode text=
 user: hi
 assistant: hello

slow decode text=
 user: hi
assistant: hello

```

Also, I found that there is a difference between fast and slow encoding when legacy=False, which one does mistralai/Mistral-7B-V0.1 use?


Yes, as I said this is being fixed by #28881 but it is not in main yet.
Mistral uses legacy = True by default. It was train with it