candidate beam search : Add candidate-based response generation functionality to the generate function
Author: ckdrjs96
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR enhances the generate functionality by pre-defining desired responses as candidates. The GPT model will generate responses exclusively from within this candidate set. This is particularly useful for generating verified responses, feature generation, or when dealing with a large number of classifications (hundreds to thousands).

It pre-encodes the desired candidates and modifies the beam search function to exclude token IDs not present in the candidates. The scores for excluded tokens are set to -inf, ensuring that only the desired candidates are generated.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi @ckdrjs96 👋 Thank you for opening the PR!

`transformers` is a large library, so we must be conservative when adding new features to ensure we can maintain them. To be considered for addition, a feature must be one of the following:
1. It is a very small feature, with negligible code footprint and maintenance requirements;
2. It is a feature that is widely requested by the community, ideally linking requests for the feature;
3. The feature substantially improves a certain aspect of the library and/or model usage, and there is data to back the claim.

Unfortunately, I can't fit this feature in any of the above buckets at the moment. Let me know if the situation changes, or if you have data to support your proposed feature, based on the criteria above 🤗  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hello @gante .
thank you for the reply.

I think Candidate beam search is powerful when used as a feature generator.

When using GPT for zero-shot or with a slight finetuning to produce features, it is often difficult to obtain the desired consistent form of response.

Therefore, by utilizing this, you can obtain consistent forms of responses without additional training or with slight finetuning.

1. Often, it is difficult to obtain the desired response without fine-tuning when receiving a prompt.
    1. For instance, if I request a response of 'yes or no' from GPT, it might answer yes, true, no, false, and so on. However, using candidate beam search can only provide answers of yes or no.
    2. It is useful when connecting specific words to the desired space, such as with Hard verbalizer.
        1. In the case of sentiment analysis, I want to receive a response like "Ihappy," but GPT can provide very similar synonyms such as cheerful, delighted, in a good mood, glad, content, and so on.
```python
keyword_list = ['happy','sad', 'angry','love' ...]
candidates_words_ids = tokenizer(keyword_list, add_special_tokens=False,).input_ids

outputs = model.generate(
    **input_ids,
    candidate_words_ids=candidates_words_ids,
    num_beams=3,
    num_return_sequences=3,
    remove_invalid_values=True,
    max_length=512,
    eos_token_id = tokenizer.eos_token_id,
    return_dict_in_generate=True,
    output_scores=True
)
```
2. It is known that for classification, it is recommended to create embeddings with GPT and utilize softmax. However, in cases of too many classification problems, after tuning with a small dataset (about 1 to 5 labels per label), you can use candidate beam search to obtain the desired response. (The existing generation often answers with labels that have not been learned.)
    - For example, when creating approximately 1000 topic classifiers with various depths, you can store those topics in JSON and load them for use in the desired format.
    - `keyword_list = ['sports>football',''sports>baseball','movie>action', 'movie>comedy', ...] `
3. Building a trustworthy chatbot
    1. GPT often hallucinates. However, companies want safe responses.
    2. After implementing reviewed response lists in candidates, you can always receive verified responses when generating.

@ckdrjs96 isn't it similar to libraries like https://github.com/outlines-dev/outlines ? (which support `transformers` models)

yes. It looks like the library includes all the features I need. Thanks for letting me know.








@ckdrjs96 perfect 💛 We prefer to support external libraries, as opposed to adding similar features in `transformers` -- community power 💪 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.