daekeun-ml commited on
Commit
823da3d
1 Parent(s): 20093ea

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +43 -0
README.md ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ ---
3
+ language:
4
+ - ko
5
+ tags:
6
+ - llama-2
7
+ - instruct
8
+ - instruction
9
+ pipeline_tag: text-generation
10
+ license: llama2
11
+ ---
12
+
13
+ # Llama-2-ko-OpenOrca-gugugo-13B
14
+
15
+ This model was trained for PoC purposes. This is part of an experiment to check whether model performance improves when fine-tuned with large data of about 1 million samples.
16
+
17
+ ### Model Details
18
+ - Base Model: [beomi/llama-2-koen-13b](https://huggingface.co/beomi/llama-2-koen-13b)
19
+
20
+ ### Datasets
21
+ Trained on 1 million samples from the dataset. The training infrastructure used AWS g5.12xlarge x 2ea (total of NVIDIA A10G 8 GPUs).
22
+
23
+ - [OpenOrca-gugugo-ko](https://huggingface.co/datasets/squarelike/OpenOrca-gugugo-ko)
24
+
25
+ ### Hyperparameters
26
+ The hyperparameters are simply heuristic values. For reference only:
27
+
28
+ ```python
29
+ learning_rate = 3e-5
30
+ lr_scheduler = "constant_with_warmup"
31
+ batch_size = 1
32
+ gradient_accumulation_steps = 8
33
+ lora_alpha = 16
34
+ lora_r = 16
35
+ lora_dropout = 0.1
36
+ lora_target_modules = "[gate_proj, down_proj, up_proj, q_proj, k_proj, o_proj, v_proj]"
37
+ use_flash_attention_2 = True
38
+ ```
39
+
40
+ ### License
41
+ - Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License, under LLAMA 2 COMMUNITY LICENSE AGREEMENT
42
+
43
+ This model was created as a personal experiment, unrelated to the organization I work for.