YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
Indic Language Bloom Model Training
This repository contains the code and resources for fine-tuning the Huggingface Bloom model on the Indic language dataset using Low-Rank Adaptation (LoRA). The goal is to create a high-performance language model specifically tailored to Indic languages.
Dataset
The dataset used for training is provided by AI4Bharat. I have uploaded it to huggingface hub at:
Progress
Completed
- Low-Rank Adaptation fine-tuning of the Bloom model on streaming data
- Single checkpoint available (training logs at Weights & Biases)
To Do
- Benchmark current multilingual LLMs on IndicGLUE using lm-evaluation-harness
- Integrate DeepSpeed for better resource utilization
- Convert current instruction dataset to Indic languages and train (dolly v2 dataset, distilled from GPT, etc.)
- Model doesn't stop producing text - how to fix?
- Deploy RLHF community app using Cheese
Using the Model
import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
peft_model_id = "aashay96/indic-BloomLM"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
# Load the Lora model
model = PeftModel.from_pretrained(model, peft_model_id)
batch = tokenizer("आप कैसे हैं", return_tensors='pt')
with torch.cuda.amp.autocast():
output_tokens = model.generate(**batch, max_new_tokens=10)
print('\n\n', tokenizer.decode(output_tokens[0], skip_special_tokens=True))