HenryJJ commited on
Commit
27585b5
1 Parent(s): 93b2a35

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +39 -0
README.md CHANGED
@@ -1,3 +1,42 @@
1
  ---
2
  license: apache-2.0
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ datasets:
4
+ - databricks/databricks-dolly-15k
5
  ---
6
+
7
+ # Instruct_Phi2_Dolly15K
8
+ Fine-tuned from phi2, used Dolly15k for the dataset. 90% for training, 10% validation. Trained for 2.0 epochs using QLora. Trained with 1024 context window.
9
+
10
+ # Model Details
11
+ * **Trained by**: trained by HenryJJ.
12
+ * **Model type:** **Instruct_Phi2_Dolly15K** is an auto-regressive language model based on the phi 2 transformer architecture.
13
+ * **Language(s)**: English
14
+ * **License for Instruct_Yi-6B_Dolly15K**: apache-2.0 license
15
+
16
+
17
+ # Prompting
18
+
19
+ ## Prompt Template With Context
20
+ chatml format
21
+
22
+ ```
23
+ <|im_start|>system
24
+ {instruction}<|im_end|>
25
+ <|im_start|>user
26
+ {prompt}<|im_end|>
27
+ <|im_start|>assistant
28
+ ```
29
+
30
+ ## Prompt Template Without Context
31
+ ```
32
+ <|im_start|>system
33
+ {instruction}<|im_end|>
34
+ <|im_start|>assistant
35
+ ```
36
+
37
+ # Training script:
38
+ Fully opensourced at: https://github.com/hengjiUSTC/learn-llm/blob/main/trl_finetune.py. Run on 1 A10G instance for 4 hours.
39
+
40
+ ```
41
+ python3 trl_finetune.py --config configs/phi2-dolly.yml
42
+ ```