Add a stop_sequence option to text generation pipeline
Author: Jcwscience
Pull request: no
Labels: name: Good Second Issue

### Feature request

A stop sequence option to allow text generation models to stop generating when a specific token is reached.

### Motivation

When I use GPT-J on a slower machine every extra generated token counts. If I need the model to answer a question for example, the only way I can ensure it isn’t cut off is to set the max length well above what I expect the answer to be. That takes a considerable amount of extra processing power for useless data.

### Your contribution

I am only beginning to understand the core workings of model inference, so I’m not sure what I can do to help. I might be able to gather documentation, or test code.

Comments:
cc @Narsil 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Unstale.

This is actually a great suggestion I've been meaning to add for quite a while.

If you're willing to do a PR here's the high level vision I have for this:

- Add a new parameter `stop_sequence`  (Here's some doc on how to add a pipeline which should cover parameters)
- within `_sanitized_parameters` consume `stop_sequence`, tokenize it. Raise a warning if sequence is multiple tokens long (being able to stop on a multiple tokens sequence is not yet covered in `transformers` and would require even more work, we can start small here). Set within the `forward_parameters["generate_kwargs"]` `eos_token_id` to the new stop sequence first token.
- Add the docstring about this new parameter
- Add some tests ( I can help with that as adding tests should be relatively forward, but as the test are really attempting to cover ALL models and all variants, they can fail in odd ways, or worse, the test could easily miss some configurations and fail to see any regression in the future).

Cheers.
I would really like to add such a parameter myself, but at the moment I don't really have the time to dedicate to this, guidance on a PR is the best I can offer 100% !


Hi @Narsil I'd love to take this on if it's still open.

I think it's still open. Thanks for taking this !

Hi @Narsil is this issue still open? Is there anything else I can help with?

It seems like this is mostly done, except for documentation [here](https://huggingface.co/docs/transformers/main_classes/pipelines?highlight=transformers.TextGenerationPipeline#transformers.TextGenerationPipeline.__call__), which I'm happy to take.  @KMFODA is there a reason why that wasn't added? 

Hey @pruksmhc good spot. I missed adding the docs for this, I'll add it as soon as I can. The PR was merged last year though so the functionality should be available in the main branch. Once docs are added I'll post here so we can close this Issue.

Hi, it seems that this issue is still open just waiting for a docstring to be added :thinking: 
@KMFODA will you be able to add it any time soon? if not and @pruksmhc is busy with other stuff, I'm happy to be the second backup :wink: 

If this is to be added, can it be a list of stop sequences instead of a single one?

Toolformer, for example, has the AI invoke tools, so I would want two stop sequences: one for tool invocation, and one for message completion

```
User: What files are on my desktop?
Assistant: On your desktop, there are [Get-ChildItem ~/Desktop ->
```

Here, the `->` arrow is meant to pause generation, execute the command, append it to the prompt, and then continue

```
User: What files are on my desktop?
Assistant: On your desktop, there are [Get-ChildItem ~/Desktop -> ["a.txt","b.txt","c.md"]] two text files called "a" and "b", and a markdown file called "c"<stop>
```

The agent uses the command output to provide context when formulating its response. The second stop token, `<stop>`, is used to prevent the assistant from continuing to generate the conversation by simulating what the user might say.

 

This issue seems to have been duplicated in #26280.

Documentation was written and merged for #26280, and the issue was recently closed. If that work was satisfactory, then this issue can be closed as well.

Indeed, thanks @Tanman2001