|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- cerebras/SlimPajama-627B |
|
- bigcode/starcoderdata |
|
- sam-mosaic/orca-gpt4-chatml |
|
language: |
|
- en |
|
--- |
|
<div align="center"> |
|
|
|
# TinyLlama-1.1B |
|
|
|
Finetuned with ORCA-GPT4 (chatml format) |
|
|
|
</div> |
|
|
|
This is a fine-tuned version of [TinyLlama-1.1B-intermediate-step-240k-503b](https://huggingface.co/PY007/TinyLlama-1.1B-intermediate-step-240k-503b) using the [sam-mosaic/orca-gpt4-chatml](https://huggingface.co/datasets/sam-mosaic/orca-gpt4-chatml) dataset. |
|
|
|
### Training |
|
- **Method**: QLORA |
|
- **Quantization**: fp16 |
|
- **Time**: 20h on a RTX 4090 (from runpod.io) |
|
- **Cost**: About $15 |
|
- **Based on**: [https://colab.research.google.com/drive/1Zmaceu65d7w4Tcd-cfnZRb6k_Tcv2b8g](https://colab.research.google.com/drive/1Zmaceu65d7w4Tcd-cfnZRb6k_Tcv2b8g) |
