Update README.md

f4b8032 verified 3 months ago

No virus

4.42 kB

	---
	language:
	- en
	license: apache-2.0
	library_name: transformers
	tags:
	- orpo
	- qlora
	- trl
	datasets:
	- alvarobartt/dpo-mix-7k-simplified
	- argilla/dpo-mix-7k
	base_model: mistralai/Mistral-7B-v0.1
	pipeline_tag: text-generation
	inference: false
	---

	## ORPO fine-tune of Mistral 7B v0.1 with DPO Mix 7K

	![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/60f0608166e5701b80ed3f02/hRyhnTySt-KQ0gnnoclSm.jpeg)

	> Stable Diffusion XL "A capybara, a killer whale, and a robot named Ultra being friends"

	This is an ORPO fine-tune of [`mistralai/Mistral-7B-v0.1`](https://huggingface.co/mistralai/Mistral-7B-v0.1) with
	[`alvarobartt/dpo-mix-7k-simplified`](https://huggingface.co/datasets/alvarobartt/dpo-mix-7k-simplified).

	⚠️ Note that the code is still experimental, as the `ORPOTrainer` PR is still not merged, follow its progress
	at [🤗`trl` - `ORPOTrainer` PR](https://github.com/huggingface/trl/pull/1435).

	## About the fine-tuning

	In order to fine-tune [`mistralai/Mistral-7B-v0.1`](https://huggingface.co/mistralai/Mistral-7B-v0.1) using ORPO, the branch
	`orpo` from [🤗`trl`](https://github.com/huggingface/trl) has been used, thanks to the invaluable and quick contribution of
	@kashif.

	ORPO stands for Odds Ratio Preference Optimization, and defines a new paradigm on fine-tuning LLMs, “combining” both the SFT
	and the PPO/DPO stage into a single stage, thanks to the proposed loss function starting off from a preference dataset i.e.
	chosen-rejected pairs.

	Some key features about ORPO:
	- ⚡️ Faster to train as it’s now a single stage fine-tuning
	- 👨🏻‍🏫 Requires preference data i.e. (prompt, chosen, rejected)-like datasets
	- ⬇️ Less memory than PPO/DPO as doesn’t need a reference model
	- 🏆 SOTA results for Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) when fine-tuned using single-turn UltraFeedback

	Some notes on the experiments mentioned in the paper:
	- 📌 Up to 7B parameter LLMs were fine-tuned, achieving better performance compared to 7B counterparts and even 13B LLMs
	- 📌 Not yet trained with multi-turn datasets as Capybara (may be an interesting experiment to run)
	- 📌 For OPT models fine-tuned with HH-RLHF from Anthropic, truncated and padded to 1024 tokens, filtering out filtering the prompts with > 1024 tokens
	- 📌 For Phi-2, Mistral (7B) and Llama 2 (7B), or UltraFeedback from OpenBMB (truncated and padded to 2048 tokens), filtering out filtering the prompts with > 1024 tokens
	- 📌 Fine-tuned for 10 epochs, and using the evaluation loss as the metric for selecting the best models

	For more information about ORPO, I highly recommend reading their paper titled [`ORPO: Monolithic Preference Optimization without Reference Model`](https://huggingface.co/papers/2403.07691),
	as it contains a lot of information and details not only on the ORPO method, but also on the experiment they ran, the results they got, and much more.

	📅 Fine-tuning code will be shared soon, stay tuned!

	## About the dataset

	The dataset used for this fine-tune is [`alvarobartt/dpo-mix-7k-simplified`](https://huggingface.co/datasets/alvarobartt/dpo-mix-7k-simplified),
	which is a simplified version of [`argilla/dpo-mix-7k`](https://huggingface.co/datasets/argilla/dpo-mix-7k).

	The simplification comes from the fact that the `prompt` column is detached from both the `chosen` and `rejected`
	columns so that there's no need for extra pre-processing while applying the chat template to the dataset before the
	fine-tuning. So on, the dataset remains as is, with an additional column for the `prompt`.

	The dataset is a small cocktail combining Argilla's latest efforts on DPO datasets, mixing the following datasets:

	* [`argilla/distilabel-capybara-dpo-7k-binarized`](https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized)
	* [`argilla/distilabel-intel-orca-dpo-pairs`](https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs)
	* [`argilla/ultrafeedback-binarized-preferences-cleaned`](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned)

	The samples have been randomly selected from the original datasets with a proportion of 0.33 each, as can be seen via the `dataset` column of the dataset.

	For more information about the original dataset check [the `README.md` file of `argilla/dpo-mix-7k`](https://huggingface.co/datasets/argilla/dpo-mix-7k/blob/main/README.md).