---
library_name: transformers
language:
- en
---

# Phi3-mini-128k-it ORPO model

[Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) fine-tuned on Text-to-SQL downstream task using [Odds Ratio Preference Optimization (ORPO)](https://arxiv.org/pdf/2403.07691).

## Details

A 4-bit quantized version of the [Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) model was used to fine-tuned on [zerolink/zsql-sqlite-dpo](https://huggingface.co/datasets/zerolink/zsql-sqlite-dpo).
Used PEFT to merge the trained adapters.

### Odds Ratio Preference Optimization (ORPO)

The goal of ORPO is to penalize the "rejected" samples, and increase the likelihood of "accepted" samples.This builds upon DPO but incorporates a ranking of preferences. This means we not only learn which outputs are preferred but also their relative ranking

#### Dataset
The model was fine-tuned on [zerolink/zsql-sqlite-dpo](https://huggingface.co/datasets/zerolink/zsql-sqlite-dpo) dataset.
Total entries in the dataset: 250,000

The dataset needs to be in the following format:
You need at least 3 columns:
- Schema
- Question
- Rejected
- Chosen
- Weight

For example:
- Schema: "CREATE TABLE table_name_56 (location TEXT, year INTEGER)"
- Question: "What location is previous to 1994?"
- Rejected: "SELECT location FROM table_name_56 WHERE year < 1994"
- Chosen: "SELECT "location" FROM "table_name_56" WHERE "year" < 1994"
- Weight: 0.056641

### Training Parameters

* QLoRA Parameters
  - r = 16
  - target_modules = ["q_proj", "k_proj", "v_proj", "o_proj","gate_proj", "up_proj", "down_proj",]
  - lora_alpha = 16
  - lora_dropout = 0
  - bias = None
  - random_state = 3407

* [ORPO Trainer](https://huggingface.co/docs/trl/main/en/orpo_trainer) Config
  - num_epochs = 1
  - max_steps = 30
  - per_device_train_batch_size = 2
  - gradient_accumulation_step = 4
  - optim = "adamw_8it"
  - lr_scheduler_type = "linear,"