|
--- |
|
license: cc-by-nc-sa-4.0 |
|
datasets: |
|
- HumanF-MarkrAI/Korean-RAG-ver2 |
|
language: |
|
- ko |
|
tags: |
|
- Retrieval Augmented Generation |
|
- RAG |
|
- Multi-domain |
|
--- |
|
|
|
# MarkrAI/RAG-KO-Mixtral-7Bx2-v2.1 |
|
|
|
# Model Details |
|
|
|
## Model Developers |
|
MarkrAI - AI Researchers |
|
|
|
## Base Model |
|
[DopeorNope/Ko-Mixtral-v1.4-MoE-7Bx2](https://huggingface.co/DopeorNope/Ko-Mixtral-v1.4-MoE-7Bx2). |
|
|
|
## Instruction tuning Method |
|
Using QLoRA. |
|
``` |
|
4-bit quantization |
|
Lora_r: 64 |
|
Lora_alpha: 64 |
|
Lora_dropout: 0.05 |
|
Lora_target_modules: [embed_tokens, q_proj, k_proj, v_proj, o_proj, gate, w1, w2, w3, lm_head] |
|
``` |
|
|
|
## Hyperparameters |
|
``` |
|
Epoch: 10 |
|
Batch size: 64 |
|
Learning_rate: 1e-5 |
|
Learning scheduler: linear |
|
Warmup_ratio: 0.06 |
|
``` |
|
|
|
## Datasets |
|
Private datasets: [HumanF-MarkrAI/Korean-RAG-ver2](https://huggingface.co/datasets/HumanF-MarkrAI/Korean-RAG-ver2) |
|
``` |
|
Aihub datasets νμ©νμ¬μ μ μν¨. |
|
``` |
|
|
|
## Implmentation Code |
|
``` |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
import torch |
|
|
|
repo = "MarkrAI/RAG-KO-Mixtral-7Bx2-v2.1" |
|
markrAI_RAG = AutoModelForCausalLM.from_pretrained( |
|
repo, |
|
return_dict=True, |
|
torch_dtype=torch.float16, |
|
device_map='auto' |
|
) |
|
markrAI_RAG_tokenizer = AutoTokenizer.from_pretrained(repo) |
|
``` |
|
|
|
# Model Benchmark |
|
- Coming soon... |
