Stopping criteria does not work for Llama-2-13B
Author: rafael-wampfler
Pull request: no

### System Info

- `transformers` version: 4.35.0
- Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
- Python version: 3.9.0
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker
@younesbelkada
@gante
@ArthurZucker

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm using LLama-2 13B with the following stopping criteria:

```
    stop_words = ["Human:", "Chatbot:", "###"]
    stop_words_ids = [tokenizer(stop_word, return_tensors='pt')['input_ids'].squeeze() for stop_word in stop_words]
    stopping_criteria = StoppingCriteriaList([StoppingCriteriaSub(stops=stop_words_ids)])
    generation_config = GenerationConfig( ... stopping_criteria=stopping_criteria )
    prompt = tokenizer(text, return_tensors='pt', truncation="only_first", max_length=4096)
    prompt = {key: value.to("cuda") for key, value in prompt.items()}
    out = model.generate(**prompt, generation_config=generation_config)
    res = tokenizer.decode(out[0])
```

The model does not stop at the provided stop words. For example, if I have a response of the model `I'm feeling good, how about you?### Human: I'm also feeling good.### Chatbot: That's good.` the model should stop generating at the first `###`.

Why does this not work and how can this be fixed?

I have fine-tuned the model (with Axolotl) on a dataset so that the model produces responses as shown above.

### Expected behavior

The model should stop producing output at the first occurrence of a stop word.

Comments:
Hey! 🤗 
I don't  have access to `StoppingCriteriaSub` (missing form the reproducer) but this is very similar to #23852, and #26959, which most probably has the answers you are looking for. 
Now what you need to check thoroughly is not the strings that are decoder, but the token ids that you feed to logit processor. You should be using `tokenizer.convert_tokens_to_ids` to check if these are indeed tokens or not, then you should make sure you encode the raw tokens without the prefix space this is added to the tokens. For this we'll add a `add_prefix_space` option that you can set to `False` soon, in the meantime you should just use `[tokenizer._tokenize(word) for word in subword]`

Dear Arthur
Thank you for your response. So I'm using `###` to separate turns in a conversation. I will check if `###` is a single token as you proposed. What is `add_prefix_space` exactly doing? Currently I'm separating turns as follows: `I'm feeling good, how about you?### Human: I'm also feeling good.### Chatbot: That's good.` So there is no white space before `###` but a white space after. Is that good or should I also add one white space before?

Sentencepiece based tokenizers like Llama or T5 always add a prefix space to the input tokens. This means that when you are trying to get the encoding for `###` you are actually getting the encoding for ` ###` which is why it does not stop. 

So is it better to have also a white space before `###` in my training data? This mean `I'm feeling good, how about you? ### Human: I'm also feeling good. ### Chatbot: That's good.` instead of `I'm feeling good, how about you?### Human: I'm also feeling good.### Chatbot: That's good.`

You proposed to use `[tokenizer._tokenize(word) for word in subword]`. How and where in my code (first post) should I use this?

No no, it's better to just use `tokenizer._tokenize` for a slow tokenizer instead of `tokenizer.tokenize` to get the actual tokens. If `' #'` is encoded as `' ','#'` then you are good, otherwise `' #'` can be tokenized a `' #'` which is a token itself

I'm a bit confused. The follwing response `Would you like to chat about something interesting?### Human: Yes please.` gets encoded by the model as following:

```
x = tokenizer._tokenize('Would you like to chat about something interesting?### Human: Yes please.')
print(x)
y = tokenizer.convert_tokens_to_ids(x)
print(y)
['W', 'ould', '▁you', '▁like', '▁to', '▁chat', '▁about', '▁something', '▁interesting', '?', '##', '#', '▁Human', ':', '▁Yes', '▁please', '.']
[29956, 483, 366, 763, 304, 13563, 1048, 1554, 8031, 29973, 2277, 29937, 12968, 29901, 3869, 3113, 29889]
```

I have tried to add `model.config.eos_token_id = 2277`. I also tried to use the following stopping criteria:

```
from transformers import StoppingCriteria
class EosListStoppingCriteria(StoppingCriteria):
    def __init__(self, eos_sequence=[2277, 29937]):
        self.eos_sequence = eos_sequence

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        last_ids = input_ids[:, -len(self.eos_sequence):].tolist()
        return self.eos_sequence in last_ids
    generation_config = GenerationConfig( ... stopping_criteria=[EosListStoppingCriteria()])
    prompt = tokenizer(text, return_tensors='pt', truncation="only_first", max_length=4096)
    prompt = {key: value.to("cuda") for key, value in prompt.items()}
    out = model.generate(**prompt, generation_config=generation_config)
    res = tokenizer.decode(out[0])
```

Both approaches do not work and the model is not stopping producing output at `###`.

You need to account for all possible token combinations or juste check the ids that are generated by the model. 
Does it not stop when you set ` 2277, 29937` in the custom logits processor on the linked issues?

I have found the error:

The following works:

```
        generation_config = GenerationConfig(
            min_length=self.min_length,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            top_k=top_k,
            top_p=top_p,
            temperature=temperature,
            repetition_penalty=1.1,
            no_repeat_ngram_size=no_repeat_ngram_size,
            use_cache=True,
            pad_token_id=self.tokenizer.eos_token_id,
            max_time=5.0
        )
out = self.model.generate(**prompt, generation_config=generation_config, stopping_criteria=[EosListStoppingCriteria()])
```

But when I provide the `stopping_criteria` as part of the `GenerationConfig `it does not stop anymore. Should I not use the `GenerationConfig ` and provide all parameters directly in the generate method? I am now unsure if the parameters set in `GenerationConfig` are used at all or if I was using default values (and did not recognize it).

Hi @Eichhof 👋 Thank you for opening this issue

`stopping_criteria` is not part of `GenerationConfig` and should be passed separately :) 

The issue on our end is the lack of an informative exception, which would have enabled you to catch and fix the issue immediately! I will open a PR that will catch these sorts of issues 🤗