Support reverse_prompt
Author: louisabraham
Pull request: no
Labels: name: WIP

### Feature request

Add a simple option to provide a stopping criteria in the generate method. This option could also be a stopping criteria.

### Motivation

Llama.cpp supports an option `--reverse-prompt`. It is very useful for chat models to stop at words like "USER:".

### Your contribution

This is how I implemented it, but it is not ideal:

```
def count_stop_words(text, stop_words):
    return Counter({expr: text.count(expr) for expr in stop_words})


class MyStoppingCriteria(StoppingCriteria):
    def __init__(self, stop_counter):
        self.stop_counter = stop_counter

    def __call__(self, input_ids, scores, **kwargs):
        # Get the generated text as a string
        generated_text = tokenizer.decode(input_ids[0])
        counter = count_stop_words(generated_text, self.stop_counter)
        if counter - self.stop_counter:
            return True
        return False

```

Then you can use this parameter:

```
            stopping_criteria=StoppingCriteriaList(
                [
                    MyStoppingCriteria(
                        count_stop_words(PROMPT, ["ASSISTANT:", "USER:", "SYSTEM:"])
                    )
                ]
            ),
```

Comments:
Hey! I am not sure I understand what you are requesting here, the custom logits processors are specifically designed for this! 

I'm interested in addressing this issue. Here are two concise solutions:

1. `stopping_criteria Parameter`: Introduce a stopping_criteria parameter directly in the generate method. Users can provide a list of stop words, and the generation process will halt when any of these words are encountered.
2. `Extend StoppingCriteria Class` : Enhance the StoppingCriteria class to include a predefined set of stop words and logic for checking against the generated text.

I also encountered a similar issue where it was necessary to stop the chatbot when specific user inputs like "bye" or "thank you" were detected. Due to the absence of this feature, I had to manually check for these user inputs to end the chatbot with a fixed response. This highlights the importance of implementing a more automated stopping mechanism. 

I would greatly appreciate your guidance in the process. Please assign this to me for implementation.

> Hey! I am not sure I understand what you are requesting here, the custom logits processors are specifically designed for this!

What is your idea? Using a custom logit processor to detect the stopping word and output EOS instead? I think this is a bit overkill and such a common use case could benefit from a custom parameter or a pre-defined LogitsProcessor.

> I'm interested in addressing this issue. Here are two concise solutions:
> 
>     1. `stopping_criteria Parameter`: Introduce a stopping_criteria parameter directly in the generate method. Users can provide a list of stop words, and the generation process will halt when any of these words are encountered.
> 
>     2. `Extend StoppingCriteria Class` : Enhance the StoppingCriteria class to include a predefined set of stop words and logic for checking against the generated text.
> 
> 
> I also encountered a similar issue where it was necessary to stop the chatbot when specific user inputs like "bye" or "thank you" were detected. Due to the absence of this feature, I had to manually check for these user inputs to end the chatbot with a fixed response. This highlights the importance of implementing a more automated stopping mechanism.
> 
> I would greatly appreciate your guidance in the process. Please assign this to me for implementation.

IMO a `StoppingCriteria` is not able to do this for batched generation queries as they don't get the prompt. A `LogitsProcessor` cannot filter words with multiple tokens either as they would need to erase the end of the output. Another filtering step / condition is needed to erase the last words after the `LogitsProcessor` stopped the generation.

As I mentioned in #23852, (see [this comment](https://github.com/huggingface/transformers/issues/23852#issuecomment-1578399293)) a StoppingCriteria is a good way to stop on a user specific list of tokens and is not an overkill. 
Generate already has a `stopping_criteria` parameter. See the doc [here](https://huggingface.co/docs/transformers/internal/generation_utils#transformers.StoppingCriteria).

Now that chat template are becoming a lot more, this simple criteria could make it's way to the library, with a more general `generation_config.eos_sequence`. But from my understanding, the[ `eos_token_id`](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig.eos_token_id) can already be a list:

>         eos_token_id (`Union[int, List[int]]`, *optional*):
>            The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens.

so I don't really see the point of creating a new list. 

I don't understand why it would not work for batches could you enlighten me?


Maybe my understanding is wrong but as I said above, `StoppingCriteria` just doesn't get enough information to do the job.

I think `eos_token_id` can check for multiple _posible_ stopping tokens, but not a _sequence_ of tokens. Please tell me if I'm mistaken. It will work for batches but not for reverse prompts with multiple tokens.



cc @gante who is more expert on this! 


Adding support for multiple tokens as EOS would be useful indeed. I've seen similar requests (often under the name of sentinel token). 

However, it requires a bit of engineering, as:
a) a single EOS token is not a `StoppingCriteria`, but it should be
b) we could then expand it to accept arbitrary lists of tokens -- please not that it has to be done at a token level, not at a text level, otherwise it will be painfully slow

I've added this to my todo list :)