SHN157/dpo-model

Direct Preference Optimization (DPO) - GPT-2 Fine-Tuned Model

View Model on Hugging Face

Student Information

• Name: Soe Htet Naing
• ID: st125166

📌 Model Information

This repository hosts the fine-tuned GPT-2 model using Direct Preference Optimization (DPO) for preference-based response ranking. The model was trained on the UltraFeedback dataset to align model outputs with human preferences.

🔗 Hugging Face Model Repository

📂 Files in This Repository

• pytorch_model.bin – The fine-tuned model weights.
• config.json – Configuration file for the GPT-2 model.
• tokenizer.json & tokenizer_config.json – Tokenizer files for text preprocessing.
• README.md – This documentation file.

📊 Dataset Citation

This model was trained using the UltraFeedback dataset:

• Source: UltraFeedback Dataset
• Dataset Used: argilla/ultrafeedback-binarized-preferences-cleaned

If you use this dataset in your research, please cite the original UltraFeedback dataset:

@article{ultrafeedback2023,
  title={UltraFeedback: A Large-Scale Preference Dataset for Reinforcement Learning from Human Feedback},
  author={OpenBMB Team},
  journal={Hugging Face Datasets},
  year={2023},
  url={https://huggingface.co/datasets/openbmb/UltraFeedback}
}

🚀 How to Use

To use this model in your projects, you can load it with the Hugging Face Transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model and tokenizer
model_name = "SHN157/dpo-model"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Ensure padding token is set
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

print("✅ Model successfully loaded!")

📊 Training Details

• Base Model: GPT-2
• Optimization Method: Direct Preference Optimization (DPO)
• Dataset: UltraFeedback (preprocessed to include prompt, chosen, and rejected fields)
• Training Split: 80% Train, 20% Test
• Best Hyperparameters:
  • Learning Rate: 0.002
  • Batch Size: 8
  • Optimizer: AdamW
  • Beta: 0.3

🖥 Deployment

The model can be integrated into Streamlit, FastAPI, or any other inference framework. Example usage for text generation:

def generate_response(prompt, max_tokens=100):
    input_ids = tokenizer(prompt, return_tensors="pt").input_ids
    output = model.generate(input_ids, max_new_tokens=max_tokens)
    return tokenizer.decode(output[0], skip_special_tokens=True)

prompt = "How can I improve my productivity?"
print("Response:", generate_response(prompt))

📢 Acknowledgments

This model was trained as part of an NLP research project to enhance model alignment with human feedback. If you use this model in your research or projects, please consider citing the UltraFeedback dataset.

💡 For further details, visit the full project documentation: GitHub Repository