Update README.md
Browse files
README.md
CHANGED
@@ -6,11 +6,12 @@ language:
|
|
6 |
- en
|
7 |
- id
|
8 |
---
|
9 |
-
|
10 |
|
11 |
**Merak-7B-v3-Mini-Orca** is Ichsan2895's [Merak-7B-v3](https://huggingface.co/Ichsan2895/Merak-7B-v3) fine-tuned on psmathur's [orca_mini_v1_dataset](https://huggingface.co/datasets/psmathur/orca_mini_v1_dataset). Dataset was machine translated into Bahasa Indonesia with Google Translate.
|
12 |
|
13 |
-
[
|
|
|
14 |
#### Training details
|
15 |
Merak-7B-v3-Mini-Orca was instruction fine-tuned on 2 x 3090-24GB for 6 hours. [LoRA](https://github.com/microsoft/LoRA), [DeepSpeed ZeRO-2](https://github.com/microsoft/DeepSpeed), and [FlashAttention](https://github.com/Dao-AILab/flash-attention) were implemented during training using [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl).
|
16 |
Hyperparameter | value |
|
@@ -27,6 +28,7 @@ lora rank | 16 |
|
|
27 |
lora dropout | 0.05 |
|
28 |
lora target modules | q_proj, v_proj, k_proj, o_proj |
|
29 |
cutoff length | 4096 |
|
|
|
30 |
#### Training loss
|
31 |
Step |Train Loss
|
32 |
| ------ | ------ |
|
|
|
6 |
- en
|
7 |
- id
|
8 |
---
|
9 |
+
# 🦚Merak-7B-v3-Mini-Orca🐳
|
10 |
|
11 |
**Merak-7B-v3-Mini-Orca** is Ichsan2895's [Merak-7B-v3](https://huggingface.co/Ichsan2895/Merak-7B-v3) fine-tuned on psmathur's [orca_mini_v1_dataset](https://huggingface.co/datasets/psmathur/orca_mini_v1_dataset). Dataset was machine translated into Bahasa Indonesia with Google Translate.
|
12 |
|
13 |
+
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
|
14 |
+
|
15 |
#### Training details
|
16 |
Merak-7B-v3-Mini-Orca was instruction fine-tuned on 2 x 3090-24GB for 6 hours. [LoRA](https://github.com/microsoft/LoRA), [DeepSpeed ZeRO-2](https://github.com/microsoft/DeepSpeed), and [FlashAttention](https://github.com/Dao-AILab/flash-attention) were implemented during training using [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl).
|
17 |
Hyperparameter | value |
|
|
|
28 |
lora dropout | 0.05 |
|
29 |
lora target modules | q_proj, v_proj, k_proj, o_proj |
|
30 |
cutoff length | 4096 |
|
31 |
+
|
32 |
#### Training loss
|
33 |
Step |Train Loss
|
34 |
| ------ | ------ |
|