File size: 1,345 Bytes
f56ed71
e838dfa
f56ed71
e838dfa
 
 
 
 
 
 
 
 
2eed76d
e838dfa
f56ed71
e838dfa
 
c86f8fb
e838dfa
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c86f8fb
e838dfa
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
---
pipeline_tag: text-generation
license: apache-2.0
language:
- en
tags:
- SOLAR-10.7B-v1.0
- Open-platypus-Commercial
base_model: upstage/SOLAR-10.7B-v1.0
datasets:
- kyujinpy/Open-platypus-Commercial
model-index:
- name: T3Q-Platypus-SOLAR
  results: []
---
Update @ 2024.03.07

## T3Q-platypus-SOLAR

This model is a fine-tuned version of upstage/SOLAR-10.7B-v1.0

**Model Developers** Chihoon Lee(chlee10), T3Q

## Training hyperparameters

The following hyperparameters were used during training:

```python
  # ๋ฐ์ดํ„ฐ์…‹๊ณผ ํ›ˆ๋ จ ํšŸ์ˆ˜์™€ ๊ด€๋ จ๋œ ํ•˜์ดํผ ํŒŒ๋ผ๋ฏธํ„ฐ
  batch_size = 16
  num_epochs = 1
  micro_batch = 1
  gradient_accumulation_steps = batch_size // micro_batch
  
  # ํ›ˆ๋ จ ๋ฐฉ๋ฒ•์— ๋Œ€ํ•œ ํ•˜์ดํผ ํŒŒ๋ผ๋ฏธํ„ฐ
  cutoff_len = 4096
  lr_scheduler = 'cosine'
  warmup_ratio = 0.06 # warmup_steps = 100
  learning_rate = 4e-4
  optimizer = 'adamw_torch'
  weight_decay = 0.01
  max_grad_norm = 1.0
  
  # LoRA config(QLoRA)
  lora_r = 16
  lora_alpha = 16
  lora_dropout = 0.05
  lora_target_modules = ["gate_proj", "down_proj", "up_proj"]
  
  # Tokenizer์—์„œ ๋‚˜์˜ค๋Š” input๊ฐ’ ์„ค์ • ์˜ต์…˜
  train_on_inputs = False
  add_eos_token = False
  
  # NEFTune params
  noise_alpha: int = 5
```

## Framework versions

  - Transformers 4.34.1
  - Pytorch 2.1.0+cu121
  - Datasets 2.13.0
  - Tokenizers 0.14.1