Unggi commited on
Commit
eab90d2
1 Parent(s): b786edd

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +136 -1
README.md CHANGED
@@ -1,3 +1,138 @@
1
  ---
2
- license: llama3
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ license: other
3
+ base_model: beomi/Llama-3-Open-Ko-8B
4
+ tags:
5
+ - generated_from_trainer
6
+ model-index:
7
+ - name: out-llama-8b-ko-slimorca_45000
8
+ results: []
9
  ---
10
+
11
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
12
+ should probably proofread and complete it, then remove this comment. -->
13
+
14
+ [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
15
+ <details><summary>See axolotl config</summary>
16
+
17
+ axolotl version: `0.4.0`
18
+ ```yaml
19
+ base_model: beomi/Llama-3-Open-Ko-8B
20
+ model_type: LlamaForCausalLM
21
+ tokenizer_type: AutoTokenizer
22
+
23
+ load_in_8bit: false
24
+ load_in_4bit: false
25
+ strict: false
26
+
27
+ # datasets:
28
+ # - path: /workspace/axolotl/datasets/mix_corpus_extended_validated_stage1.json
29
+ # type: completion
30
+ # field: text
31
+ # /workspace/axolotl/datasets/slimorca_20000.jsonl
32
+ datasets:
33
+ - path: /workspace/axolotl/datasets/slimorca_ko_45000.jsonl
34
+ type: sharegpt
35
+ conversation: chatml
36
+ dataset_prepared_path: last_run_prepared
37
+ val_set_size: 0.05
38
+ eval_sample_packing: False
39
+ output_dir: ./out-llama-8b-ko-slimorca_45000
40
+
41
+ sequence_len: 8192
42
+ sample_packing: true
43
+ pad_to_sequence_len: true
44
+
45
+ wandb_project:
46
+ wandb_entity:
47
+ wandb_watch:
48
+ wandb_name:
49
+ wandb_log_model:
50
+
51
+ gradient_accumulation_steps: 8
52
+ micro_batch_size: 1
53
+ num_epochs: 1
54
+ optimizer: paged_adamw_8bit
55
+ lr_scheduler: cosine
56
+ learning_rate: 2e-5
57
+
58
+ train_on_inputs: false
59
+ group_by_length: false
60
+ bf16: auto
61
+ fp16:
62
+ tf32: false
63
+
64
+ gradient_checkpointing: true
65
+ gradient_checkpointing_kwargs:
66
+ use_reentrant: false
67
+ early_stopping_patience:
68
+ resume_from_checkpoint:
69
+ logging_steps: 1
70
+ xformers_attention:
71
+ flash_attention: true
72
+
73
+ warmup_steps: 100
74
+ evals_per_epoch: 1
75
+ eval_table_size:
76
+ saves_per_epoch: 1
77
+ debug:
78
+ deepspeed:
79
+ weight_decay: 0.0
80
+ fsdp:
81
+ fsdp_config:
82
+ special_tokens:
83
+ pad_token: <|end_of_text|>
84
+
85
+ ```
86
+
87
+ </details><br>
88
+
89
+ # out-llama-8b-ko-slimorca_45000
90
+
91
+ This model is a fine-tuned version of [beomi/Llama-3-Open-Ko-8B](https://huggingface.co/beomi/Llama-3-Open-Ko-8B) on the None dataset.
92
+ It achieves the following results on the evaluation set:
93
+ - Loss: 0.8945
94
+
95
+ ## Model description
96
+
97
+ More information needed
98
+
99
+ ## Intended uses & limitations
100
+
101
+ More information needed
102
+
103
+ ## Training and evaluation data
104
+
105
+ More information needed
106
+
107
+ ## Training procedure
108
+
109
+ ### Training hyperparameters
110
+
111
+ The following hyperparameters were used during training:
112
+ - learning_rate: 2e-05
113
+ - train_batch_size: 1
114
+ - eval_batch_size: 1
115
+ - seed: 42
116
+ - distributed_type: multi-GPU
117
+ - num_devices: 4
118
+ - gradient_accumulation_steps: 8
119
+ - total_train_batch_size: 32
120
+ - total_eval_batch_size: 4
121
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
122
+ - lr_scheduler_type: cosine
123
+ - lr_scheduler_warmup_steps: 100
124
+ - num_epochs: 1
125
+
126
+ ### Training results
127
+
128
+ | Training Loss | Epoch | Step | Validation Loss |
129
+ |:-------------:|:-----:|:----:|:---------------:|
130
+ | 1.0058 | 0.99 | 102 | 0.8945 |
131
+
132
+
133
+ ### Framework versions
134
+
135
+ - Transformers 4.40.0.dev0
136
+ - Pytorch 2.2.0+cu121
137
+ - Datasets 2.15.0
138
+ - Tokenizers 0.15.0