Add custom stop token ids for generation
Author: tokestermw
Pull request: yes

### Update (using eos_token_id instead): https://github.com/huggingface/transformers/pull/20727#issuecomment-1355219288

# What does this PR do?

Hi 🤗  team!

This adds stop token ids inside, e.g. `model.generate(..., stop_token_ids=[10, 25])`, and syntactic sugar for the generation pipelines, e.g. `pipeline(..., stop_tokens=['\n'])`. When the generation detects the specified token ids for all examples in the batch, it will stop.

Rationale
* It's common to set a stop id/token for text generation tasks. For example for dialogue, we may want to stop it when the speaker changes.
* It's convenient to have arguments for stop tokens similar to `max_new_tokens` without digging into `StoppingCriterion`.
* Some servers like [DeepSpeed MII](https://github.com/microsoft/DeepSpeed-MII/issues/109) uses gRPC and it's difficult to pass `StoppingCriteria` objects.

## Usage Example

```python
# in pipeline
prompt = """Hello I believe in"""
text_generator = pipeline("text-generation", model="hf-internal-testing/tiny-random-gpt2", stop_tokens=[' fe'])
text_generator(prompt)

# from generate
gpt2_tokenizer = GPT2Tokenizer.from_pretrained("hf-internal-testing/tiny-random-gpt2")
gpt2_model = GPT2LMHeadModel.from_pretrained("hf-internal-testing/tiny-random-gpt2").to(torch_device)
input_ids = gpt2_tokenizer(prompt, return_tensors="pt").input_ids.to(torch_device)

stop_token_ids = gpt2_tokenizer.encode(" fe")
gpt2_model.generate(input_ids=input_ids, stop_token_ids=stop_token_ids)
```

## How to Test

```shell
pytest tests/generation/test_stopping_criteria.py::StoppingCriteriaTestCase::test_stop_token_id_criteria
pytest tests/generation/test_utils.py::GenerationIntegrationTests::test_stop_token_ids_stopping_criteria
pytest tests/pipelines/test_pipelines_text_generation.py::TextGenerationPipelineTests::test_stop_token_ids_stopping_criteria
pytest tests/pipelines/test_pipelines_text_generation.py::TextGenerationPipelineTests::test_stop_tokens_stopping_criteria
```

## Related PR(s)

There is a `stop_sequence` argument for the `TextGeneration` pipeline: https://github.com/huggingface/transformers/pull/18444

But it's limited to a single token, only in the text generation pipeline, and overwrites `eos_token_id`. Instead, we use `StoppingCriteria` directly.

This PR is a bit overlapping with above, so please let me know if this approach is not optimal.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation). **(☢️ noting i've tried to update the docs from the instructions, but they don't seem correct)**
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten @Narsil



Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @gante 

Think we could actually allow `eos_token_id` to be both an integer and a list of integers no ? Both in the config and in the input. 


Hi @tokestermw 👋 

Like my colleagues, I also think this would be a helpful feature! I also agree with @patrickvonplaten, allowing the existing argument (`eos_token_id`) to also accept list of integers would result in a cleaner interface and fewer lines of code to maintain :) It is also to port to TF/FLAX, which do not use `StoppingCriterion`.

In a nutshell, if `eos_token_id` can be a list of integers, we can replace [the existing check](https://github.com/huggingface/transformers/blob/26dd041c6e45379141302e2d293ab4cd9cf805d4/src/transformers/generation/utils.py#L2154) with 
```python
unfinished_sequences = unfinished_sequences.mul((sum(next_tokens == i for i in eos_token_id)).long())
```
as long as we always cast `eos_token_id` to a list before the generation loop. In other words, 2 lines of change (per generation method) would probably do the trick!

@tokestermw WDYT?

Got it thanks for the suggestion! I can certainly make it so we use
eos_token_id.

​It is also to port to TF/FLAX, which do not use StoppingCriterion.

ah good to know :)

I can look at this again this weekend



On Fri, Dec 16, 2022 at 8:59 AM, Joao Gante ***@***.***>
wrote:

> Hi @tokestermw <https://github.com/tokestermw> [image: 👋]
>
> Like my colleagues, I also think this would be a helpful feature! I also
> agree with @patrickvonplaten <https://github.com/patrickvonplaten>,
> allowing the existing argument (eos_token_id) to also accept list of
> integers would result in a cleaner interface and fewer lines of code to
> maintain :) It is also to port to TF/FLAX, which do not use
> StoppingCriterion.
>
> In a nutshell, if eos_token_id can be a list of integers, we can replace the
> existing check
> <https://github.com/huggingface/transformers/blob/26dd041c6e45379141302e2d293ab4cd9cf805d4/src/transformers/generation/utils.py#L2154>
> with
>
> unfinished_sequences = unfinished_sequences.mul((sum(next_tokens == i for i in eos_token_id)).long())
>
> as long as we always cast eos_token_id to a list before the generation
> loop. In other words, 2 lines of change (per generation method) would
> probably do the trick!
>
> @tokestermw <https://github.com/tokestermw> WDYT?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/20727#issuecomment-1355219288>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABEA3R6MBS7EMZQVFCS4POTWNSNXPANCNFSM6AAAAAAS3PYPPE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Hi @gante,

* Made `eos_token_id` into `Union[int, List[int]]` type. I convert into a list at the beginning of the respective functions. Also, looks like `eos_token_id` is used in a few more places, e.g. `beam_search.py`.
* Some parts where we insert the `eos_token_id`, I only insert the first token id, [here](https://github.com/tokestermw/transformers/pull/1/files#diff-f208583b02617e3684c26a002cea6640d42ddd04bbd49eb55fa8f8460e701586R857) and [here](https://github.com/tokestermw/transformers/pull/1/files#diff-26783ca033d92b4ce2e01eb691dbf5b05dd972b0cbfdc69fc726cf77a9dcb011R1343)

You can see the changes here: https://github.com/tokestermw/transformers/pull/1/files

If this change looks good, I can merge into this PR, and start polishing (fixing tests, docs, remove dead code, etc.).

thanks!


@tokestermw that's a comprehensive set of changes, it looks great to me! ❤️ 

I was actually thinking more about directly adapting all those lines: https://github.com/huggingface/transformers/blob/17292440c069118fbdb992b9a17da2098fab5b87/src/transformers/generation/utils.py#L2154

To **also** accept a list of `eos_token_ids` so that we don't even need a new criterion class for this but can just make it work out of the box by passing `model.generate(..., eos_token_id=[0, 1])`

Awesome this looks nice to me, @gante @sgugger ok for you?

Is this feature already available on the transformers version available through pip (4.25.1)? I have tried enabling it and the generation continued on even though I set `eos_token_id`, in my case to

    tokenizer.encode('\n', return_tensors='pt')[1]

(I'm also not sure why 2 integers are returned by `encode` instead of just 1)

**EDIT**

Nevermind, I got it working

    n = tokenizer.encode('\n', return_tensors='pt')[0][1]
    output = model.generate(input_ids, eos_token_id=n).cuda()