Merge branch 'main' of https://huggingface.co/HenryJJ/Instruct_Phi2_Dolly15K

README.md

@@ -1,3 +1,42 @@
 ---
 license: apache-2.0
+datasets:
+- databricks/databricks-dolly-15k
 ---
+
+# Instruct_Phi2_Dolly15K
+Fine-tuned from phi2，used Dolly15k for the dataset. 90% for training, 10% validation.  Trained for 2.0 epochs using QLora.  Trained with 1024 context window.
+
+# Model Details
+* **Trained by**: trained by HenryJJ.
+* **Model type:**  **Instruct_Phi2_Dolly15K** is an auto-regressive language model based on the phi 2 transformer architecture.
+* **Language(s)**: English
+* **License for HenryJJ/Instruct_Phi2_Dolly15K**: apache-2.0 license
+
+
+# Prompting
+
+## Prompt Template With Context
+chatml format
+
+```
+<|im_start|>system
+{instruction}<|im_end|>
+<|im_start|>user
+{prompt}<|im_end|>
+<|im_start|>assistant
+```
+
+## Prompt Template Without Context
+```
+<|im_start|>system
+{instruction}<|im_end|>
+<|im_start|>assistant
+```
+
+# Training script:
+Fully opensourced at: https://github.com/hengjiUSTC/learn-llm/blob/main/trl_finetune.py. Run on 1 A10G instance for 4 hours.
+
+```
+python3 trl_finetune.py --config configs/phi2-dolly.yml
+```