Update README.md
Browse files
README.md
CHANGED
@@ -3,3 +3,28 @@ license: other
|
|
3 |
license_name: phi-3
|
4 |
license_link: https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/raw/main/LICENSE
|
5 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3 |
license_name: phi-3
|
4 |
license_link: https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/raw/main/LICENSE
|
5 |
---
|
6 |
+
Finetune of [microsoft/Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) on [m-a-p/CodeFeedback-Filtered-Instruction](https://huggingface.co/datasets/m-a-p/CodeFeedback-Filtered-Instruction) for ~9-10h using a single 3090 24GB.
|
7 |
+
|
8 |
+
Due to limited resources and time, the training was only on half (0.5136) of the epoch.
|
9 |
+
|
10 |
+
```
|
11 |
+
train_loss: 0.43311
|
12 |
+
```
|
13 |
+
|
14 |
+
```
|
15 |
+
learning_rate=5e-5,
|
16 |
+
lr_scheduler_type="cosine",
|
17 |
+
max_length=1024,
|
18 |
+
max_prompt_length=512,
|
19 |
+
overwrite_output_dir=True,
|
20 |
+
beta=0.1,
|
21 |
+
gradient_accumulation_steps=8,
|
22 |
+
optim="adamw_torch",
|
23 |
+
num_train_epochs=1,
|
24 |
+
evaluation_strategy="steps",
|
25 |
+
eval_steps=0.2,
|
26 |
+
logging_steps=1,
|
27 |
+
warmup_steps=50,
|
28 |
+
fp16=True,
|
29 |
+
save_steps=50
|
30 |
+
```
|