license: apache-2.0
datasets:
- Anthropic/hh-rlhf
language:
- en
pipeline_tag: text-generation
tags:
- text-generation-inference
Model Card for OpenBezoar-HH-RLHF-DPO
The OpenBezoar-HH-RLHF-DPO is an LLM that has been fine tuned for human preferences alignment using Direct Preference Optimization (DPO), on top of OpenBezoar-HH-RLHF-SFT model on a subset of Anthropic's HH-RLHF Dataset.
Model Details
- Base Model: OpenBezoar-HH-RLHF-SFT
- Dataset used for SFT: First 100K examples of the HH-RLHF dataset
- Alignment Method: Direct Preference Optimization (DPO)
- Epochs: 1
Model Description
OpenBezoar-HH-RLHF-SFT is an LLM that is built upon the OpenLLaMA 3B v2 architecture. This model has been fine-tuned for human preferences alignment using DPO. Alignment has been performed on top of the OpenBezoar-HH-RLHF-SFT model. For more information please refer to our paper.
Model Sources
- Repository: [More Information Needed]
- Paper : [More Information Needed]
Instruction Format
We follow the typical format for instruction-based prompt templates, with a system prompt followed up by the user prompt. Both begins with a prefix and ends with two newline characters as described below. It is important to utilize this template in order to obtain best responses for instruction fine-tuning related tasks.
### System: {system}
### Instruction: {instruction}
### Response:
Notice that no end-of-sentence (eos) token is being appended.
Limitations
- The model might not consistently show improved abilities to follow instructions, and it could respond inappropriately or get stuck in loops.
- Although this model is aligned to human preferences and has been evaluated for performance, it is not guaranteed that it will refrain from generating harmful content exclusively.
- Caution is urged against relying on this model for production or adjacent use-cases.
Citation
If you find our work useful, please cite our paper as follows:
[More Information Needed]
Model Authors
Chandeepa Dissanayake, Lahiru Lowe, Sachith Gunasekara, and Yasiru Ratnayake