Allow Exclusion of Input IDs from RepetitionPenaltyLogitsProcessor
Author: alex-jw-brooks
Pull request: yes

This PR adds a flag for allowing the exclusion of input IDs when using the `RepetitionPenaltyLogitsProcessor` - currently there are some workarounds that may be used for language models, e.g., passing input embeddings instead of input ids, however, these workarounds are a bit trickier with multimodal models, since we generally pass the input ids first to create the merged multimodal embeddings in the model.

Adding such a flag would be really helpful for models like granite speech, which @avihu111 and I had recently added support for. In experiments, the model works very well with a super high repetition penalty on just the newly generated token IDs, but performance degrades severely when the input IDs are included.

By default, input IDs are included to avoid changing existing default behaviors.

Fixes https://github.com/huggingface/transformers/issues/36642 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->

Probably @gante @zucchini-nlp @eustlb would have the most context!

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Hi Alex 👋  Thank you for opening this PR!

This is the first time I'm reading about this feature request. As such, there are a few workarounds I'd like to ask you to try before adding your proposal -- minimizing flags greatly improves the UX of the library in the long run 🤗 If none of them solves your problem (or if they have unwanted secondary effects), let me know! 

_______________________________
1. If we combine `EncoderRepetitionPenaltyLogitsProcessor` (modifier on the prompt tokens) with `RepetitionPenaltyLogitsProcessor` (modifier on all tokens) with the right penalty values, we can cancel out the modifier applied to the prompt. The difference is what happens if a token that is in the prompt gets generated: with the existing solution, that token will never get a modifier, while with your solution it gets a modifier.
2. If we run a forward pass with the prompt to extract the KV cache, and then call generate passing the KV cache without passing the prompt tokens, the penalty only gets applied to new tokens. The caveat here is that you need to generate the next token from the prompt manually after the forward pass, and feed it to `generate` as input (so generate receives the cache + 1st generated token). You'll also need to hand-craft the attention mask for `generate`. It's possible there are bugs. If this works and it's not too complex on your side, I can open a PR to automate the creation of the attention mask when partial input tokens + KV cache are passed to `generate`.



Hi @gante, thanks a lot for the quick response, I really appreciate the suggestions! That all makes total sense and I agree. I'm not a fan of the flag either, but thought it might be nice to avoid changing the default behavior. 

Do you have any thoughts on keeping the change to the `RepetitionPenaltyLogitsProcessor`, but just avoiding exposing it through`.generate`, like the current state of this PR?

This might be simpler, because then if people do end up hitting this in the future, there is a clear workaround by passing it as a custom logit processor, and it avoids touching the generate api 🙂  

```python
from transformers import RepetitionPenaltyLogitsProcessor

...

logits_processor = [
    RepetitionPenaltyLogitsProcessor(
        penalty=3.0,
        input_ids_seq_length=model_inputs["input_ids"].shape[-1],
    )
]

model_outputs = model.generate(
    **model_inputs,
    ...
    logits_processor=logits_processor,
)
```

Yeah, I'd be happy to add the flag to the processor, but not to the generation config -- it would indeed be the best of both worlds 🤗 

Awesome, thanks a lot for the quick reviews! 🙂