Bakpia
Collection
Open models instructioned-tuned to respond in Javanese! Comes in 0.5B, 1.5B, and 9B parameters.
•
4 items
•
Updated
•
1
Bakpia is a family of open language models capable of responding in Javanese language. Version one of Bakpia is the first generative Javanese LLM gain functional instruction performance using solely synthetic data.
Beta preview
This repository contains the 4bit version of Bakpia V1 9B.
Version | Base Model | URL | Training |
---|---|---|---|
V1 0.5B | Qwen 2 0.5B Instruct | fp16 | Epoch = 1, Batch = 16*8, lr = 5e-5, linear schedule |
V1 1.5B | Qwen 2 1.5B Instruct | fp16 | Epoch = 1, Batch = 16*8, lr = 5e-5, linear schedule |
V1 9B | Gemma 2 9B Instruct | fp16/4bit | Batch size = 16*8, lr = 4e-5, linear schedule |
Training data is accessible here.
This is the first version of Bakpia.
✨ Training
✨ Features
# Update transformers for Gemma 2 compatibility + install accelerate and bitsandbytes for loading 4bit model
!pip install -q git+https://github.com/huggingface/transformers.git
!pip install -q accelerate bitsandbytes
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
tokenizer = AutoTokenizer.from_pretrained("afrizalha/Bakpia-V1-9B-Javanese-4bit")
model = AutoModelForCausalLM.from_pretrained("afrizalha/Bakpia-V1-9B-Javanese-4bit", quantization_config=BitsAndBytesConfig(load_in_4bit=True))
model.to("cuda")
template = """<start_of_turn>user
{prompt}<end_of_turn>
<start_of_turn>model
"""
input = template.format(prompt="Kados pundi kulo saged nyinaoni Basa Jawa kanthi sae?")
input = tokenizer([input], return_tensors = "pt").to("cuda")
outputs = model.generate(**input, max_new_tokens = 1024, streamer= TextStreamer(tokenizer), temperature=.5, use_cache=True, do_sample=True)