|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- databricks/databricks-dolly-15k |
|
--- |
|
|
|
# Instruct_Phi2_Dolly15K |
|
Fine-tuned from phi2，used Dolly15k for the dataset. 90% for training, 10% validation. Trained for 2.0 epochs using QLora. Trained with 1024 context window. |
|
|
|
# Model Details |
|
* **Trained by**: trained by HenryJJ. |
|
* **Model type:** **Instruct_Phi2_Dolly15K** is an auto-regressive language model based on the phi 2 transformer architecture. |
|
* **Language(s)**: English |
|
* **License for HenryJJ/Instruct_Phi2_Dolly15K**: apache-2.0 license |
|
|
|
|
|
# Prompting |
|
|
|
## Prompt Template With Context |
|
chatml format |
|
|
|
``` |
|
<|im_start|>system |
|
{instruction}<|im_end|> |
|
<|im_start|>user |
|
{prompt}<|im_end|> |
|
<|im_start|>assistant |
|
``` |
|
|
|
## Prompt Template Without Context |
|
``` |
|
<|im_start|>system |
|
{instruction}<|im_end|> |
|
<|im_start|>assistant |
|
``` |
|
|
|
# Training script: |
|
Fully opensourced at: https://github.com/hengjiUSTC/learn-llm/blob/main/trl_finetune.py. Run on 1 A10G instance for 4 hours. |
|
|
|
``` |
|
python3 trl_finetune.py --config configs/phi2-dolly.yml |
|
``` |
