|
--- |
|
library_name: transformers |
|
language: |
|
- en |
|
--- |
|
|
|
# Phi3-mini-128k-it ORPO model |
|
|
|
[Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) fine-tuned on Text-to-SQL downstream task using [Odds Ratio Preference Optimization (ORPO)](https://arxiv.org/pdf/2403.07691). |
|
|
|
## Details |
|
|
|
A 4-bit quantized version of the [Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) model was used to fine-tuned on [zerolink/zsql-sqlite-dpo](https://huggingface.co/datasets/zerolink/zsql-sqlite-dpo). |
|
Used PEFT to merge the trained adapters. |
|
|
|
### Odds Ratio Preference Optimization (ORPO) |
|
|
|
The goal of ORPO is to penalize the "rejected" samples, and increase the likelihood of "accepted" samples.This builds upon DPO but incorporates a ranking of preferences. This means we not only learn which outputs are preferred but also their relative ranking |
|
|
|
#### Dataset |
|
The model was fine-tuned on [zerolink/zsql-sqlite-dpo](https://huggingface.co/datasets/zerolink/zsql-sqlite-dpo) dataset. |
|
Total entries in the dataset: 250,000 |
|
|
|
The dataset needs to be in the following format: |
|
You need at least 3 columns: |
|
- Schema |
|
- Question |
|
- Rejected |
|
- Chosen |
|
- Weight |
|
|
|
For example: |
|
- Schema: "CREATE TABLE table_name_56 (location TEXT, year INTEGER)" |
|
- Question: "What location is previous to 1994?" |
|
- Rejected: "SELECT location FROM table_name_56 WHERE year < 1994" |
|
- Chosen: "SELECT "location" FROM "table_name_56" WHERE "year" < 1994" |
|
- Weight: 0.056641 |
|
|
|
### Training Parameters |
|
|
|
* QLoRA Parameters |
|
- r = 16 |
|
- target_modules = ["q_proj", "k_proj", "v_proj", "o_proj","gate_proj", "up_proj", "down_proj",] |
|
- lora_alpha = 16 |
|
- lora_dropout = 0 |
|
- bias = None |
|
- random_state = 3407 |
|
|
|
* [ORPO Trainer](https://huggingface.co/docs/trl/main/en/orpo_trainer) Config |
|
- num_epochs = 1 |
|
- max_steps = 30 |
|
- per_device_train_batch_size = 2 |
|
- gradient_accumulation_step = 4 |
|
- optim = "adamw_8it" |
|
- lr_scheduler_type = "linear," |
