Hugging Face's logo

RichardErkhov
/
Columbia-NLP_-_LION-Gemma-2b-odpo-v1.0-8bits

Safetensors
gemma
8-bit precision
bitsandbytes
Model card Files Community
Columbia-NLP_-_LION-Gemma-2b-odpo-v1.0-8bits
File size: 4,852 Bytes 
d310b81
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
 
Quantization made by Richard Erkhov.

[Github](https://github.com/RichardErkhov)

[Discord](https://discord.gg/pvy7H8DZMG)

[Request more models](https://github.com/RichardErkhov/quant_request)


LION-Gemma-2b-odpo-v1.0 - bnb 8bits
- Model creator: https://huggingface.co/Columbia-NLP/
- Original model: https://huggingface.co/Columbia-NLP/LION-Gemma-2b-odpo-v1.0/




Original model description:
---
library_name: transformers
tags: []
---

# Model Card for LION-Gemma-2b-odpo-v1.0

The LION-series are trained using an **empirically optimized pipeline** that consists of three stages: SFT, DPO, and online preference learning (online DPO). We find simple techniques such as sequence packing, loss masking in SFT, increasing the preference dataset size in DPO, and online DPO training can significantly improve the performance of language models. Our best models (the LION-series) **exceed the performance of the official instruct models** tuned with closed-source data and algorithms.

For training datasets, code, and evaluation scripts, please refer to our [paper](https://arxiv.org/abs/2407.06542) and [codebase](https://github.com/Columbia-NLP-Lab/LionAlignment).


## Model description

This model is finetuned from [`Columbia-NLP/LION-Gemma-2b-dpo-v1.0`](https://huggingface.co/Columbia-NLP/LION-Gemma-2b-dpo-v1.0) using online DPO from the LION pipeline.

- **Model type:** [`gemma-2b`](https://huggingface.co/google/gemma-2b)
- **Language(s) (NLP):** Primarily English
- **License:** Gemma Terms of Use
- **Finetuned from model:** [`Columbia-NLP/LION-Gemma-2b-dpo-v1.0`](https://huggingface.co/Columbia-NLP/LION-Gemma-2b-dpo-v1.0)


## Performance

| Model | Method | Size | Arena-Hard | AlpacaEval-2 | MT-Bench | OpenLLM |
|-------------|--------|------|------:|------:|---------:|-------:|
|[Gemma-2b](https://huggingface.co/google/gemma-2b) | - | 2B | - | - | - | 46.69 |
|[Gemma-2b-it](https://huggingface.co/google/gemma-2b-it) | SFT+RLHF | 2B | 3.4 | 5.44 | 5.63 | 42.75 |
|[Gemma-2b-zephyr](https://huggingface.co/wandb/gemma-2b-zephyr-dpo) | SFT+DPO | 2B | 0.9 | 2.65 | 4.13 | 46.92 |
|[LLaMA-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | SFT | 7B | 4.6 | 5.35 | 6.22 | 53.16 |
|[Vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5) | SFT | 7B | 2.5 | 7.62 | 6.57 | 52.06 |
|[LION-Gemma-2b-sft-v1.0 (ours)](https://huggingface.co/Columbia-NLP/LION-Gemma-2b-sft-v1.0) | SFT | 2B | 2.4 | 7.79 | 6.37 | 54.78 |
|[LION-Gemma-2b-dpo-v1.0 (ours)](https://huggingface.co/Columbia-NLP/LION-Gemma-2b-dpo-v1.0) | SFT+DPO | 2B | 4.6 | 8.75 | 6.58 | 55.35 |
|⮕ [LION-Gemma-2b-odpo-v1.0 (ours)](https://huggingface.co/Columbia-NLP/LION-Gemma-2b-odpo-v1.0) | SFT+DPO+ODPO | 2B | 5.0 | 9.57 | 6.75 | 55.98 |


## Intended uses

To ensure reproducibility, please use the following chat templates:

```python
import torch
from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="Columbia-NLP/LION-Gemma-2b-odpo-v1.0",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
messages = [
    {
        "role": "system",
        "content": "",
    },
    {
        "role": "user", 
        "content": "Write a short paragraph where every sentence start with the letter A."
    },
]
outputs = pipe(
    messages,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.7,
    top_p=0.7,
    stop_sequence="<|im_end|>",
)
print(outputs[0]["generated_text"][-1]["content"])
# Alice always aspired to acquire ample adventure.
# Astonishingly, amid abundant allurements, Alice allocated ample attention to each activity, ensuring an array of adventures.
# Albeit anxieties arose, Alice assuaged them with affirmations, ardently advancing ambitiously towards an array of adventures.
```

to inspect the chat template/manually do generation:

```python
tokenizer = AutoTokenizer.from_pretrained("Columbia-NLP/LION-Gemma-2b-odpo-v1.0")
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(prompt)
# tokenize prompt and use model.generate
```


### Training details

Please refer to our [paper](https://arxiv.org/abs/2407.06542) and [codebase](https://github.com/Columbia-NLP-Lab/LionAlignment).


## Citation Information

If you find this model useful in your work, please consider citing our paper:

```
@misc{yu2024lionsempiricallyoptimizedapproach,
      title={LIONs: An Empirically Optimized Approach to Align Language Models}, 
      author={Xiao Yu and Qingyang Wu and Yu Li and Zhou Yu},
      year={2024},
      eprint={2407.06542},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.06542}, 
}
```

## Acknowledgements

We thank the Columbia-NLP group and [articulate.ai](https://www.articulateai.com/) for providing OpenAI API credits and computational resources to conduct our experiments.