---
license: other
license_name: gemma-terms-of-use
license_link: https://ai.google.dev/gemma/terms
library_name: transformers
base_model: google/gemma-2b
tags:
- trl
- orpo
- generated_from_trainer
model-index:
- name: gemma-2b-orpo
  results: []
datasets:
- alvarobartt/dpo-mix-7k-simplified
language:
- en
---

<img src="./assets/gemma-2b-orpo.png" width="450"></img>
# gemma-2b-orpo

This is an ORPO fine-tune of [google/gemma-2b](https://huggingface.co/google/gemma-2b) with
[`alvarobartt/dpo-mix-7k-simplified`](https://huggingface.co/datasets/alvarobartt/dpo-mix-7k-simplified).

## ORPO
[ORPO (Odds Ratio Preference Optimization)](https://arxiv.org/abs/2403.07691) is a new training paradigm that combines the usually separated phases
of SFT (Supervised Fine-Tuning) and Preference Alignment (usually performed with RLHF or simpler methods like DPO).
- Faster training
- Less memory usage (no reference model needed)
- Good results!

## 🏆 Evaluation

### Nous

gemma-2b-orpo performs well for its size on Nous' benchmark suite.

(evaluation conducted using [LLM AutoEval](https://github.com/mlabonne/llm-autoeval)).

| Model | Average | AGIEval | GPT4All | TruthfulQA | Bigbench |
|---|---:|---:|---:|---:|---:|
| [**anakin87/gemma-2b-orpo**](https://huggingface.co/anakin87/gemma-2b-orpo) [📄](./assets/gemma-2b-orpo-Nous.md) | **39.45** | 23.76 | 58.25 | 44.47 | 31.32 |
| [mlabonne/Gemmalpaca-2B](https://huggingface.co/mlabonne/Gemmalpaca-2B) [📄](https://gist.github.com/mlabonne/4b638752fc3227df566f9562064cb864) | 38.39 | 24.48 | 51.22 | 47.02 | 30.85 |
| [google/gemma-2b-it](https://huggingface.co/google/gemma-2b-it) [📄](https://gist.github.com/mlabonne/db0761e74175573292acf497da9e5d95) | 36.1 | 23.76 | 43.6 | 47.64 | 29.41 |
| [google/gemma-2b](https://huggingface.co/google/gemma-2b) [📄](https://gist.github.com/mlabonne/7df1f238c515a5f63a750c8792cef59e) | 34.26 | 22.7 | 43.35 | 39.96 | 31.03 |


## 🙏 Dataset
[`alvarobartt/dpo-mix-7k-simplified`](https://huggingface.co/datasets/alvarobartt/dpo-mix-7k-simplified)
is a simplified version of [`argilla/dpo-mix-7k`](https://huggingface.co/datasets/argilla/dpo-mix-7k).
You can find more information [here](https://huggingface.co/alvarobartt/Mistral-7B-v0.1-ORPO#about-the-dataset).

## 🎮 Model in action
### Usage notebook
[📓 Chat and RAG using Haystack](./notebooks/usage.ipynb)
### Simple text generation with Transformers
The model is small, so runs smoothly on Colab. *It is also fine to load the model using quantization*.
```python
# pip install transformers accelerate
import torch
from transformers import pipeline
pipe = pipeline("text-generation", model="anakin87/gemma-2b-orpo", torch_dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "Write a rap song on Vim vs VSCode."}]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False)
outputs = pipe(prompt, max_new_tokens=500, do_sample=True, temperature=0.7,  top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```
## Training
The model was trained using HF TRL.
[📓 Training notebook](./notebooks/training.ipynb)

### Framework versions

- Transformers 4.39.1
- Pytorch 2.2.0+cu121
- Datasets 2.18.0
- Tokenizers 0.15.2