|
--- |
|
pipeline_tag: text-generation |
|
license: apache-2.0 |
|
language: |
|
- en |
|
tags: |
|
- SOLAR-10.7B-v1.0 |
|
- Open-platypus-Commercial |
|
base_model: upstage/SOLAR-10.7B-v1.0 |
|
datasets: |
|
- kyujinpy/Open-platypus-Commercial |
|
model-index: |
|
- name: T3Q-Platypus-SOLAR |
|
results: [] |
|
--- |
|
Update @ 2024.03.07 |
|
|
|
## T3Q-platypus-SOLAR |
|
|
|
This model is a fine-tuned version of upstage/SOLAR-10.7B-v1.0 |
|
|
|
**Model Developers** Chihoon Lee(chlee10), T3Q |
|
|
|
## Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
|
|
```python |
|
# ๋ฐ์ดํฐ์
๊ณผ ํ๋ จ ํ์์ ๊ด๋ จ๋ ํ์ดํผ ํ๋ผ๋ฏธํฐ |
|
batch_size = 16 |
|
num_epochs = 1 |
|
micro_batch = 1 |
|
gradient_accumulation_steps = batch_size // micro_batch |
|
|
|
# ํ๋ จ ๋ฐฉ๋ฒ์ ๋ํ ํ์ดํผ ํ๋ผ๋ฏธํฐ |
|
cutoff_len = 4096 |
|
lr_scheduler = 'cosine' |
|
warmup_ratio = 0.06 # warmup_steps = 100 |
|
learning_rate = 4e-4 |
|
optimizer = 'adamw_torch' |
|
weight_decay = 0.01 |
|
max_grad_norm = 1.0 |
|
|
|
# LoRA config(QLoRA) |
|
lora_r = 16 |
|
lora_alpha = 16 |
|
lora_dropout = 0.05 |
|
lora_target_modules = ["gate_proj", "down_proj", "up_proj"] |
|
|
|
# Tokenizer์์ ๋์ค๋ input๊ฐ ์ค์ ์ต์
|
|
train_on_inputs = False |
|
add_eos_token = False |
|
|
|
# NEFTune params |
|
noise_alpha: int = 5 |
|
``` |
|
|
|
## Framework versions |
|
|
|
- Transformers 4.34.1 |
|
- Pytorch 2.1.0+cu121 |
|
- Datasets 2.13.0 |
|
- Tokenizers 0.14.1 |
|
|
