ThakrePranjal/pharma-tinyllama-dpo-lora

Pharma TinyLlama — DPO Preference LoRA Adapter (Stage 3)

This is the Stage 3 DPO preference-tuning LoRA adapter, trained on top of the instruction-tuned merged model (ThakrePranjal/pharma-tinyllama-instruct-merged) using Direct Preference Optimization (DPO).

Full 3-stage training pipeline

TinyLlama (base)
    │
    ├── Stage 1: Domain Pretraining (LoRA)
    │       [ThakrePranjal/pharma-tinyllama-domain-lora]
    │
    ├── Stage 1 Merged Model → used as base for Stage 2
    │
    ├── Stage 2: Instruction Fine-Tuning (LoRA / SFT)
    │       [ThakrePranjal/pharma-tinyllama-instruct-lora]
    │
    ├── Stage 2 Merged Model → used as base for Stage 3
    │       [ThakrePranjal/pharma-tinyllama-instruct-merged]
    │
    └── Stage 3: DPO Preference Tuning (LoRA)
            [THIS ADAPTER]

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load Stage 2 merged model as base
base_model = AutoModelForCausalLM.from_pretrained(
    "ThakrePranjal/pharma-tinyllama-instruct-merged"
)

# Attach Stage 3 DPO LoRA adapter
model     = PeftModel.from_pretrained(base_model, "ThakrePranjal/pharma-tinyllama-dpo-lora")
tokenizer = AutoTokenizer.from_pretrained("ThakrePranjal/pharma-tinyllama-dpo-lora")
model.eval()

Inference (Alpaca-style prompt)

def generate(instruction, input_text="", max_new_tokens=150):
    if input_text.strip():
        prompt = (
            f"### Instruction:\n{instruction}\n\n"
            f"### Input:\n{input_text}\n\n"
            f"### Response:\n"
        )
    else:
        prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
            repetition_penalty=1.1,
            pad_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(out[0], skip_special_tokens=True)

print(generate("Explain the primary mechanism of action of metformin."))

DPO training config

Param	Value
Beta (KL penalty)	0.1
Epochs	3 (max_steps=5)
Learning rate	5e-5
Batch size	1
Grad accum steps	8
Max seq length	512
Max prompt length	256
LoRA rank (r)	16
LoRA alpha	32

Dataset

ThakrePranjal/pharma-preference-dataset

Limitations

Trained on a small pharma corpus. Not validated for clinical or production use. Intended for educational/research purposes only.