Heng666 commited on
Commit
6393978
1 Parent(s): 419d8ca

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +59 -0
README.md ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - generated_from_trainer
4
+ - mistral
5
+ model-index:
6
+ - name: taiwan-kapok-300M
7
+ results: []
8
+ license: apache-2.0
9
+ language:
10
+ - zh
11
+ ---
12
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
+ should probably proofread and complete it, then remove this comment. -->
14
+
15
+ # taiwan-kapok-300M
16
+
17
+
18
+ - Loss: 2.7894
19
+
20
+ ## Model description
21
+
22
+ More information needed
23
+
24
+ ## Intended uses & limitations
25
+
26
+ More information needed
27
+
28
+ ## Training and evaluation data
29
+
30
+ More information needed
31
+
32
+ ## Training procedure
33
+
34
+ ### Training hyperparameters
35
+
36
+ The following hyperparameters were used during training:
37
+ - learning_rate: 0.0006
38
+ - train_batch_size: 4
39
+ - eval_batch_size: 4
40
+ - seed: 42
41
+ - distributed_type: multi-GPU
42
+ - num_devices: 2
43
+ - gradient_accumulation_steps: 64
44
+ - total_train_batch_size: 512
45
+ - total_eval_batch_size: 8
46
+ - optimizer: Adam with betas=(0.9,0.95) and epsilon=0.0001
47
+ - lr_scheduler_type: cosine
48
+ - lr_scheduler_warmup_steps: 1000
49
+ - num_epochs: 5
50
+ - mixed_precision_training: Native AMP
51
+ ### Training results
52
+ | Training Loss | Epoch | Step | Validation Loss |
53
+ |:-------------:|:-----:|:----:|:---------------:|
54
+ | 2.6962 | 3.35 | 5000 | 2.7894 |
55
+ ### Framework versions
56
+ - Transformers 4.35.2
57
+ - Pytorch 2.1.2+cu121
58
+ - Datasets 2.14.5
59
+ - Tokenizers 0.14.1