OpenLlama-Platypus-3B

OpenLlama-Platypus-3B is an instruction fine-tuned model based on the OpenLLaMA-3B transformer architecture.

Model Details

• Trained by: Robbe De Sutter
• Model type: OpenLlama-Platypus-3B is an auto-regressive language model based on the OpenLLaMA-3B transformer architecture.
• Language(s): English
• License for base weights: Non-Commercial Creative Commons license (CC BY-NC-4.0)

Prompt Template

### Instruction:
<prompt> (without the <>)
### Response:

Training Dataset

RobbeD/OpenLlama-Platypus-3B trained using STEM and logic based dataset garage-bAInd/Open-Platypus.

Please see their paper and project webpage for additional information.

Training Procedure

RobbeD/OpenLlama-Platypus-3B was instruction fine-tuned using LoRA on 1 RX 6900 XT 16GB.

Citations

@article{platypus2023,
    title={Platypus: Quick, Cheap, and Powerful Refinement of LLMs}, 
    author={Ariel N. Lee and Cole J. Hunter and Nataniel Ruiz},
    booktitle={arXiv preprint arxiv:2308.07317},
    year={2023}
}

@software{openlm2023openllama,
  author = {Geng, Xinyang and Liu, Hao},
  title = {OpenLLaMA: An Open Reproduction of LLaMA},
  month = May,
  year = 2023,
  url = {https://github.com/openlm-research/open_llama}
}

@inproceedings{
    hu2022lora,
    title={Lo{RA}: Low-Rank Adaptation of Large Language Models},
    author={Edward J Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
    booktitle={International Conference on Learning Representations},
    year={2022},
    url={https://openreview.net/forum?id=nZeVKeeFYf9}
}

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	36.76
ARC (25-shot)	41.21
HellaSwag (10-shot)	71.67
MMLU (5-shot)	29.86
TruthfulQA (0-shot)	36.45
Winogrande (5-shot)	65.98
GSM8K (5-shot)	1.14
DROP (3-shot)	11.01