|
--- |
|
library_name: transformers |
|
license: apache-2.0 |
|
basemodel: beomi/llama-2-koen-13b |
|
datasets: |
|
- Saxo/total_ko_train_set_small_basic |
|
- beomi/KoAlpaca-v1.1a |
|
- kyujinpy/KOR-OpenOrca-Platypus-v2 |
|
- nlpai-lab/databricks-dolly-15k-ko |
|
language: |
|
- ko |
|
- en |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
AI 와 빅데이터 분석 전문 기업인 Linkbricks의 데이터사이언티스트인 지윤성 박사(Saxo)가 beomi/llama-2-koen-13b 베이스모델을 GCP상의 A100-40G 4개를 통해 SFT 훈련을 한(2048 Tokens) 인스트럭션 모델. |
|
Accelerate, Deepspeed Zero-3 라이브러리를 사용했으며 Flash Attention 은 Disable 로 설정 |
|
|
|
Dr. Yunsung Ji (Saxo), a data scientist at Linkbricks, a company specializing in AI and big data analytics, trained the beomi/llama-2-koen-13b base model on 4 A100-40Gs on GCC for 4 hours of instructional training (2048 Tokens). |
|
Accelerate, Deepspeed Zero-3 libraries were used. |
|
|
|
www.linkbricks.com, www.linkbricks.vc |
|
|
|
## Configuration including BitsandBytes |
|
--- |
|
learning_rate = 2e-4 |
|
num_epochs = 5 |
|
batch_size = 4 |
|
block_size = 2048 |
|
trainer = "sft" |
|
warmup_ratio = 0.1 |
|
weight_decay = 0.01 |
|
gradient_accumulation = 4 |
|
mixed_precision = "fp16" |
|
peft = True |
|
quantization = "int4" |
|
lora_r = 64 |
|
lora_alpha = 16 |
|
lora_dropout = 0.1 |
|
model_max_length = 2048 |
|
--- |
|
## Dataset Format |
|
Alpaca Format Prompt Text |
