Constraints in constrained beam search can be satisfied by the inputs.
Author: zawedcvg
Pull request: no
Labels: name: Generation

### System Info

- `transformers` version: 4.42.0.dev0
- Platform: Linux-5.15.0-97-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.2
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): 2.13.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA A40

### Who can help?

@ArthurZucker  @younesbelkada @gante 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1RpUejeoW6-ExXK6b9h_iey254bSAfDoC?usp=sharing

### Expected behavior

One would expect the constraints to be applied to the generated output, but as seen in the above example, the constraints can be satisfied by the inputs.  As can be seen [here](https://github.com/huggingface/transformers/blob/7729b7747872eabe461a8d2d4ce1068ae0e716a8/src/transformers/generation/beam_search.py#L615) and [here](https://github.com/huggingface/transformers/blob/7729b7747872eabe461a8d2d4ce1068ae0e716a8/src/transformers/generation/beam_search.py#L719C1-L724C56), the code checks the list of constraints against the whole hypothesis, which includes the inputs.
This becomes an issue when few-shot prompting on chat models, as the constraints are generally already satisfied in the system prompts. I am not sure if this is the intended result, but if it isn't it can be fixed relatively easily as the decoder_prompt_len is already being passed as a parameter during the beam search.

Comments:
Similar issue was addressed [here](https://github.com/huggingface/transformers/issues/30295#issuecomment-2088852000)

I see. So is this supposed to be the intended outcome? Since including the inputs feels a bit counterintuitive. If it's a bug, I could take up the issue.

@zawedcvg I would say that it's a known issue but with a very low priority since we are not maintaining constraints actively. Regarding the fix, if it's not a major change in code (except for the Constraints classes) we are open to a PR. Otherwise using "sequence_bias" should do similar generation

Let's get @gante 's opinion on community PR, in case there are any plans for constraints I wasn't aware of  :)

If the constraints shouldn't be satisfied by the inputs, it is a relatively minor fix since decoder_prompt_len is already a parameter that is being passed through.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.