Information
- Model Pretrain: VietAI/gpt-neo-1.3B-vietnamese-news
Training procedure
batch_size: int = 128
micro_batch_size: int = 2
num_epochs: int = 1
learning_rate: float = 3e-4
cutoff_len: int = 256
val_set_size: int = 0
Lora hyperparams
lora_r: int = 6
lora_alpha: int = 12
lora_dropout: float = 0.05
lora_target_modules: str = "q_proj k_proj v_proj"
Framework versions
- transformer 4.32.1
- pytorch
- PEFT 0.5.0
How to use
BASE_MODEL = "VietAI/gpt-neo-1.3B-vietnamese-news"
PEFT_WEIGHTS = "NghiemAbe/QAvietnamese"
model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, PEFT_WEIGHTS, torch_dtype=torch.bfloat16)
- Downloads last month
- 3