Generate: Add new decoding strategy "DoLa" in `.generate()`
Author: voidism
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #29524

We add the support for a new decoding strategy proposed in a [recent paper](https://arxiv.org/abs/2309.03883) of ICLR 2024.
The main revisions are in [src/transformers/generation/utils.py](https://github.com/voidism/transformers-dola/blob/main/src/transformers/generation/utils.py) and [src/transformers/generation/configuration_utils.py](https://github.com/voidism/transformers-dola/blob/main/src/transformers/generation/configuration_utils.py)

We also update the documentation and add the test code. Run the test by:
```
CUDA_VISIBLE_DEVICES=0 python examples/pytorch/text-generation/run_generation_dola.py --model_name_or_path huggyllama/llama-7b --model_type llama --dola_layers 'low'
```

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), 
         Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Yes, in #29524
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests? Yes, in [examples/pytorch/text-generation/run_generation_dola.py](https://github.com/voidism/transformers-dola/blob/main/examples/pytorch/text-generation/run_generation_dola.py)


## Who can review?

@gante is the main contributor of the part of `.generate()` function, which this PR focuses on.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi @gante !

Thanks so much for your suggestions! I spent some time to add the code for test cases, and fixed the issues you mentioned. 
All the CI checks were passed as well. Can you take a look at my latest commits of the code?

Please let me know if you have any other concerns or suggestions for me to fix! I would be happy to address any of the issues you may have! 🤗

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29619). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @gante !

Thanks so much for your great suggestions! I have fixed all the issues you mentioned. Just let me know if you have any other concerns or suggestions!
Thanks for requesting a review from the core maintainer! 🤗

Hi @gante !

While waiting for the core maintainer's approval, I found that the validation of the parameter ranges in the generation config mainly happens in `tsrc/transformers/generation/configuration_utils.py` instead of `src/transformers/generation/utils.py`. Thus, I simply moved the warning of repetition penalty of dola generation to `configuration_utils.py`, and the warning will also only occur once!

However, after I committed the new code. A test case of XLM model failed, and it seems to have nothing to do with my commit. The failed case seems related to #29297

I tried syncing with the upstream but it didn't solve the issue. I wonder if you know what's the reason for this failed test case. Sorry for bothering you again!


```
Some tests failed!

============================= FAILURES SHORT STACK =============================
____________________ XLMModelTest.test_batching_equivalence ____________________

tests/test_modeling_common.py:745: in recursive_check
    self.assertTrue(
E   AssertionError: tensor(False) is not true : Batched and Single row outputs are not equal in XLMForQuestionAnswering for key=end_top_index. Difference=1.


FAILED tests/models/xlm/test_modeling_xlm.py::XLMModelTest::test_batching_equivalence - AssertionError: tensor(False) is not true : Batched and Single row outputs are not equal in XLMForQuestionAnswering for key=end_top_index. Difference=1.

Exited with code exit status 255
```

The failed test case was solved after syncing with the upstream! Please ignore my previous comment.
It's ready to merge now!

Hi @amyeroberts !

This PR is ready to merge after some iterations! Would you be able to review it and give me any suggestions you have? 
Thanks a lot for the help! 🤗

Hi @amyeroberts !

Thanks so much for all of your great suggestions! They are very helpful and they improved my code and the test cases!
I have tried my best to fix all the issues you mentioned above. Let me know if there are still concerns or suggestions so I can address them further! 🤗

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@voidism are you intending to continue the PR? 🤗 Or do you need a hand?

Hi @gante 

Sorry that I was busy with my midterm for the past few weeks 😔 so I forgot to fix this for a while... I will continue fixing the PR this or next week!
Thanks for the reminder and sorry for the delay!

@voidism no worries, focus on your midterms 💪 we'll be here when you're ready to continue 🙌 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @gante and @amyeroberts 

I am back and fixed all the suggestions from @amyeroberts last time!

Sorry that I was busy with midterm exams and paper deadlines last month 😔, so I stopped fixing this PR for a while. 🥲
And last week I just traveled to ICLR 2024 to present the DoLa paper! Now I finally get some free time to fix this.

It's my fault that you guys might need to spend more time recalling our discussions from almost two months ago. I am really sorry about that! 🥲

In addition to fixing all the suggestions from last time, I have synced this PR with the latest transformers `v4.41.0`, and it passed all the CI tests. I found that the new version of `generation/utils.py` becomes more concise and cleaner than before. Thanks so much for your efforts in making it better!

Let me know if you have any other concerns or suggestions. I recently have more free time so I can assure you guys that I will fix any of your new suggestions as soon as I can! No more procrastination I promise!

Thanks @amyeroberts so much for approving the changes! 🙌

Hi @gante Just let me know if the current version looks good or not. I will be happy to fix any suggestions or concerns you have! Thanks! 🤗

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@voidism my turn to apologise for the delay, I'm catching up with issues :)

I've re-checked the PR and I'm happy with it! I'm going to merge this Monday (to avoid breaking our CI on a weekend 😉 )

Hi @gante 

No problem! Thanks so much for your help!! 🤗

rebased yet again (previous `main` had unrelated issues that was making CI red), fixing resulting issues in next commits

Ran the following slow tests locally (with the expected results):
1. `RUN_SLOW=1 py.test tests/models/ -k dola -vv`
2. `RUN_SLOW=1 py.test -vv tests/models/llama/test_modeling_llama.py`
3. `RUN_SLOW=1 py.test -vv tests/generation/test_utils.py`
4. `RUN_SLOW=1 py.test -vv tests/utils/test_cache_utils.py`


@voidism finally all CI issues were sorted -- thank you for bearing with us 🤗 I will communicate about this feature tomorrow! 💪 

Hi @gante 

Thanks a lot for your help! Handling these CI tests isn't easy (I learned a lot from it 😂). I really appreciate your effort. So happy that we finally made it! 🤗

> Handling these CI tests isn't easy

@voidism hehe it looks annoying, but it is essential to ensure all our features are playing nicely with each other 🤗 

@vanpelt hello,Can dola only be used in llama? Is it supported for qwen?