Llama: Generating text token by token removes whitespaces 
Author: qtrrb
Pull request: no

### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.10
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help?

 @ArthurZucker @sgugger 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run the following script

```python
import torch
from transformers import (
    LlamaForCausalLM,
    LlamaTokenizer,
    LogitsProcessorList,
    RepetitionPenaltyLogitsProcessor,
    TemperatureLogitsWarper,
    TopPLogitsWarper,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = LlamaTokenizer.from_pretrained("./path/to/llama")
model = LlamaForCausalLM.from_pretrained(
    "./models/llama-7b-hf", torch_dtype=torch.float16
).to(device)


@torch.no_grad()
def stream_generate(
    prompt: str,
    temperature=1.0,
    max_new_tokens=512,
    top_p=1.0,
    repetition_penalty=1.0,
):
    global tokenizer, model
    if tokenizer is None or model is None:
        return {"error": "Model not loaded."}

    input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

    original_size = len(input_ids[0])

    logits_processor = LogitsProcessorList(
        [
            TemperatureLogitsWarper(temperature=temperature),
            RepetitionPenaltyLogitsProcessor(penalty=repetition_penalty),
            TopPLogitsWarper(top_p=top_p),
        ]
    )
    while True:
        # Get logits for the next token
        logits = model(input_ids).logits[:, -1, :]

        logits = logits_processor(input_ids, logits)

        probs = torch.nn.functional.softmax(logits, dim=-1)
        next_token_id = torch.multinomial(probs, num_samples=1)

        next_token = tokenizer.decode(next_token_id[0], skip_special_tokens=True)
        print(next_token, end="", flush=True)

        input_ids = torch.cat((input_ids, next_token_id), dim=1)

        if len(input_ids[0]) >= original_size + max_new_tokens:
            break


stream_generate("In a shocking finding, ")
```

### Expected behavior

Text should be printed in a streaming manner, similar to OpenAI's playground, this behaviour properly happens with models like GPT-2 or GPT-J, however, with LLaMA, there are no whitespaces inbetween words.

I believe this is related to this, mentionned in the [documentation](https://huggingface.co/docs/transformers/main/en/model_doc/llama) 

> The LLaMA tokenizer is based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. “Banana”), the tokenizer does not prepend the prefix space to the string. To have the tokenizer output the prefix space, set decode_with_prefix_space=True in the LlamaTokenizer object or in the tokenizer configuration.

However, it seems that `decode_with_prefix_space` has been removed

Comments:
cc @gante 
Note that the streaming API is new and still experimental :-)

Also note that we aim to match the original tokenizer 1 to 1, while trying not to add super model specific pieces of code in the more general api 😉 

> Note that the streaming API is new and still experimental :-)

I may be wrong as I'm not sure what you are mentionning, but I don't believe I am using the streaming API
The example code is something I wrote based on the generate() function in transformers, It's an old piece of code that I attempted to run with LLaMa and that seems to pose issues



Oh my bad, I read to fast. I don't know what this `decode_with_prefix_space` mentioned in the documentation is. It does not exist in the codebase at all (not for LLaMA and not for any other sentencepiece model either).

> I don't know what this decode_with_prefix_space mentioned in the documentation is. It does not exist in the codebase at all (not for LLaMA and not for any other sentencepiece model either).

I haven't had much time to check why , but i noticed that it was removed in the latest commit to LLaMA's tokenizer
https://github.com/huggingface/transformers/commit/c0f99b4d2ec73090595914dde4c16da207e21d73

> Also note that we aim to match the original tokenizer 1 to 1, while trying not to add super model specific pieces of code in the more general api 

I understand, is it then normal for the tokenizer to do this as it is based on sentencepiece? 

Not sure I understand your question. Yes it is normal, if you use the sentencepiece tokenizer, you will get the same results when decoding. 
```python 
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained('huggyllama/llama-30b')
>>> tokenizer.batch_decode(tokenizer.encode("Hey how are you doing?"), skip_special_tokens = True)
['', 'Hey', 'how', 'are', 'you', 'doing', '?']
``` 
whether you are using the fast or slow tokenizer, both will output the same. 
If you use `tokenizer.sp_model.decode` (which is basically the sentencpiece model that the original llama uses) then you have no additional prefix space (see the original codebase [here](https://github.com/facebookresearch/llama/blob/main/llama/tokenizer.py)). However, the doc should be updated to remove the `decode_with_prefix_space` thanks for catching this. 

> Not sure I understand your question. Yes it is normal, if you use the sentencepiece tokenizer, you will get the same results when decoding.

I apologize, my question was meant to ask if it is normal for the tokenizer to not have an option to `decode_with_prefix_space`
As it is the case, I will close this issue, I will find another way to fix my piece of code 

This feature was added on my request so I do not think it is justified that this got closed.
decode_with_prefix_space was meant as a workaround for the fact that the tokenizer is unsuitable for downstream tasks like the one mentioned in this issue. The default behavior was off, but it allowed implementations to turn it on and get proper generations.
I do not know if it ever worked right, but without this my users keep complaining that every generation lacks the space. If this is not added back I will have to do this myself in my own code, but that would be unfortunate for the rest of the ecosystem.

@henk717 While I can't help with the removal of `decode_with_prefix_space` from the library, here is how I dealt with it, hope this helps

```python
    while True:
        # Get logits for the next token
        logits = model(input_ids).logits[:, -1, :]
        # Apply logits processor
        next_tokens_scores = logits_processor(input_ids, logits)

        probs = torch.nn.functional.softmax(next_tokens_scores, dim=-1)
        next_token_id = torch.multinomial(probs, num_samples=1)

        # Note: This is done to handle Sentencepiece based tokenizers,
        # as they don't preprend the prefix space to the start of a word
        tokens_previous = tokenizer.decode(input_ids[0], skip_special_tokens=True)
        input_ids = torch.cat((input_ids, next_token_id), dim=1)
        tokens = tokenizer.decode(input_ids[0], skip_special_tokens=True)

        new_tokens = tokens[len(tokens_previous) :]
```
This is a fairly unclean way to fix the issue, but it works.


> @henk717 While I can't help with the removal of `decode_with_prefix_space` from the library, here is how I dealt with it, hope this helps
> 
> ```python
>     while True:
>         # Get logits for the next token
>         logits = model(input_ids).logits[:, -1, :]
>         # Apply logits processor
>         next_tokens_scores = logits_processor(input_ids, logits)
> 
>         probs = torch.nn.functional.softmax(next_tokens_scores, dim=-1)
>         next_token_id = torch.multinomial(probs, num_samples=1)
> 
>         # Note: This is done to handle Sentencepiece based tokenizers,
>         # as they don't preprend the prefix space to the start of a word
>         tokens_previous = tokenizer.decode(input_ids[0], skip_special_tokens=True)
>         input_ids = torch.cat((input_ids, next_token_id), dim=1)
>         tokens = tokenizer.decode(input_ids[0], skip_special_tokens=True)
> 
>         new_tokens = tokens[len(tokens_previous) :]
> ```
> 
> This is a fairly unclean way to fix the issue, but it works.

it works for me, thank you!

Is there a better way to fix this than calling the tokenizer.decode on the entire sequence? It seems a very ineffiecient workaround.

You could use the `convert_ids_to_tokens` which should be more efficient no? 

Hello, do not plan to fix this?

You can already set the `add_prefix_space` parameter no? 

Arthur, I recall that always added a space which is alao bad on continuous generations. I think most people in this issue ended up programming their own workarounds after decode_with_prefix_space was dropped.

In KoboldAI we ended up adding a dummy character to the input to force the actual generation to be continuous and then after the tokenization is done we delete the first token before feeding it to the generator.

The real issue was fixed prior to the llama PR after my back and forth with its maintainer, but removed during HF's review phase. So I assume it has never worked correct and the projects who could afford the effort to implement workarounds figured that was easier than continuing the discussion.

Don't worry will be all fixed next release ! 

For completeness the same issue also exists in TGI which prevented us from adopting it. Its a bug / "feature" of sentencepiece itself so anything using Llama's tokenizer is effected.

Okay, what I don't understand is that the prefix space addition only comes from the `tokenizer.normalizer`. There is no API clean way to de-activate it yet, tho #28881 will introduce it, you can quite simply just do:
```python 
from transformers import AutoTokenizer
from tokenizers import normalizers
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer.normalizer = normalizers.Replace(" ", "▁")
```


Anyways the PR is completely ready, and will really fix a lot of the pain we have been having with llama. 
cc @OlivierDehaene for TGI

> Okay, what I don't understand is that the prefix space addition only comes from the `tokenizer.normalizer`. There is no API clean way to de-activate it yet, tho #28881 will introduce it, you can quite simply just do:
> 
> ```python
> from transformers import AutoTokenizer
> from tokenizers import normalizers
> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
> tokenizer.normalizer = normalizers.Replace(" ", "▁")
> ```

How about CJK characters? Which do not have so many spaces.

I don't understand the question. But you could remove the normalizer. 

> Anyways the PR is completely ready, and will really fix a lot of the pain we have been having with llama. 
> 
> cc @OlivierDehaene for TGI

Hi @ArthurZucker, what is the status of this work? I also have the issue that progressive decoding of tokens using the `LlamaTokenizerFast` doesn't prefix the token with a space when required. It does this because it only gets one token at a time to decode; the behavior in this case is to remove the space prefix, this should not happen for progressive decoding.


It has been merged a few month ago, you can enable and disable the addition of the prefix space when calling from pretrained and the bug itself was fixed

Some tokenizers has a strip step in its decoder, which is the direct cause of the missing whitespace. You can check the `decoder.decoders` field in `tokenizer.json` to see if it is your case. If it is, you can do something like `tokenizer.decoder.decoders.pop()`.