Indic Language Bloom Model Training

This repository contains the code and resources for fine-tuning the Huggingface Bloom model on the Indic language dataset using Low-Rank Adaptation (LoRA). The goal is to create a high-performance language model specifically tailored to Indic languages.

Dataset

The dataset used for training is provided by AI4Bharat. I have uploaded it to huggingface hub at:

Processed Indic Language Corpus

Progress

Completed

Low-Rank Adaptation fine-tuning of the Bloom model on streaming data
Single checkpoint available (training logs at Weights & Biases)

To Do

Benchmark current multilingual LLMs on IndicGLUE using lm-evaluation-harness
Integrate DeepSpeed for better resource utilization
Convert current instruction dataset to Indic languages and train (dolly v2 dataset, distilled from GPT, etc.)
Model doesn't stop producing text - how to fix?
Deploy RLHF community app using Cheese

Using the Model

import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

peft_model_id = "aashay96/indic-BloomLM"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)

# Load the Lora model
model = PeftModel.from_pretrained(model, peft_model_id)



batch = tokenizer("आप कैसे हैं", return_tensors='pt')

with torch.cuda.amp.autocast():
  output_tokens = model.generate(**batch, max_new_tokens=10)

print('\n\n', tokenizer.decode(output_tokens[0], skip_special_tokens=True))