Adding new decoding strategy "DoLa" into the `model.generate()` function
Author: voidism
Pull request: no
Labels: name: Feature request

name: Generation

### Feature request

Paper: https://openreview.net/pdf?id=Th6NyL07na
Code: https://github.com/voidism/DoLa

DoLa (**D**ecoding by C**o**ntrasting **La**yers) is a new decoding strategy that can make LLM decoding more factual. 
The key idea is to obtain the next-token distribution by contrasting the differences in logits obtained from projecting the later layers versus earlier layers to the vocabulary space, exploiting the fact that factual knowledge in an LLMs has generally been shown to be localized to particular transformer layers. 

The paper has been peer-reviewed and accepted at the ICLR 2024 conference.

### Motivation

The decoding strategy was proven useful in a variety of tasks, including TruthfulQA, Chain-of-thought reasoning, and instruction following, as shown in the experiments of [the paper](https://openreview.net/pdf?id=Th6NyL07na)

For example, the performance on StrategyQA/GSM8K/TruthfulQA and VicunaQA is shown as follows:
| Model          | StrategyQA | GSM8K | TruthfulQA |
|----------------|-------|-------|-----------------------|
| LLaMa-7B       | 60.1  | **10.8**  | 26.9                  |
| + DoLa         | **64.1**  | 10.5  | **40.8**              |
| LLaMa-13B      | 66.6  | 16.7  | 32.4                  |
| + DoLa         | **67.6**  | **18.0**  | **44.6**              |
| LLaMa-33B      | 69.9  | 33.8  | 31.7                  |
| + DoLa         | **72.1**  | **35.5**  | **49.1**              |
| LLaMa-65B      | 70.5  | 51.2  | 34.8                  |
| + DoLa         | **72.9**  | **54.0**  | **49.2**              |

<img width="361" alt="image" src="https://github.com/huggingface/transformers/assets/26344602/d3ac5abb-a18c-4167-8ded-fa2e38acbf67"> _Vicuna QA results_

Currently, the code is open-source in a [separated repo](https://github.com/voidism/DoLa), which is a little bit difficult for people to use.
It would be much more convenient for people to use this decoding method if it's merged into the transformers package.

### Your contribution

I have implemented the DoLa decoding method based on a fork from the latest transformers `(4.39.0.dev0)` package. The code is here: https://github.com/voidism/transformers-dola/

The method doesn't rely on any external modules or modification to model architectures, so it can be easily incorporated into the transformers package mainly by modifying [src/transformers/generation/utils.py](https://github.com/voidism/transformers-dola/tree/main/src/transformers/generation/utils.py)

This implementation does NOT need to modify any of the model architectures of LLMs like LLaMA, so it will be quite easy to merge. And it can be directly used by any other models like Mistral/Gemma as well.

I can submit a PR for this request feature ASAP.


Comments:
Hi @gante 

Thanks for your great contribution to the Transformers package!

I noticed that you are the main contributor to [generation/utils.py](https://github.com/huggingface/transformers/commits/main/src/transformers/generation/utils.py). Could you take a look at our feature requests? We believe this feature will be useful to the research community of text generation. Please let us know if you have any suggestions!

Thanks very much for your help!

Hi @voidism 👋 

This is a very interesting idea indeed! It's nice to see existing models get a performance boost out of your strategy. Have you benchmarked your technique using other models?

I've looked at the diff on your fork, it looks good! Looking forward to your PR 🔥 

Hi @gante

Yes! In our paper, we also tried DoLa on MPT-7B, which got a similar performance boost.
We recently tried it on LLaMA-2 and it also worked similarly.

<img width="361" alt="image" src="https://github.com/huggingface/transformers/assets/26344602/6b0dcebc-fbb7-4d8d-92b1-089045c92088">

Thanks a lot for your review! I will submit a PR soon! 🔥