Pastaaaaa2003/hindi-llama-omni-model

🦙🎧 Hindi LLaMA-Omni: Hindi Speech Interaction with Large Language Models

Hindi LLaMA-Omni is a Hindi speech-to-speech model built upon LLaMA-Omni. It uses Whisper for speech understanding, a fine-tuned Hindi LLaMA-Omni backbone for response generation, and IndicF5 to synthesize answers in one fixed default Hindi voice.

• 🗣️ Hindi speech interaction: accepts Hindi speech questions and returns Hindi speech answers.
• 🧠 Built on LLaMA-Omni: keeps the Whisper encoder and speech projector structure from the original speech-language architecture.
• 🇮🇳 Fine-tuned for Hindi instruction following: trained with Hindi instruction data converted into speech-question form.
• 🎙️ IndicF5 speech generation: replaces the original unit-vocoder output path with a fixed default Hindi voice.
• 📊 Evaluated in Hindi: includes IndicQA, MT-Bench-Hi, IFEval-Hi, and GSM8K-Hi evaluation scripts.

💡 Architecture

The runtime flow is:

1. A user records or uploads a Hindi speech question.
2. Whisper encodes the speech input.
3. The speech projector maps audio features into the LLaMA-Omni language backbone.
4. The fine-tuned Hindi backbone generates a Hindi text response.
5. A bundled reference recording provides the fixed speaker identity for IndicF5.
6. IndicF5 synthesizes the final answer in the fixed default voice.

📚 Data

This project uses Hindi instruction-following examples converted into speech-question form. Since public datasets with Hindi speech questions paired with text answers do not exist, the training data was generated from Hindi instruction corpora and converted into audio for speech-instruction fine-tuning.

Dataset card: Pastaaaaa2003/Hindi-speech-instruct

The text instruction mixture includes AI4Bharat Indic-Instruct style sources such as anudesh, flan_v2, hh-rlhf, and lm_sys.

⚖️ License and attribution

The Hindi stage-2 adapter is published at Pastaaaaa2003/hindi-llama-omni-model. This is a thin release: this repository contains code only, and the model repository contains only the final stage-2 adapter. The other checkpoints are downloaded from their upstream owners. Follow their respective terms, including the academic, non-commercial restriction for LLaMA-Omni, and cite the original LLaMA-Omni paper.

🛠️ Install

The supported runtime is Linux with an NVIDIA GPU (about 24 GB VRAM) and CUDA 12.1. Keep about 40 GB free disk space for the environment and checkpoints. Apple Silicon is not currently a supported runtime for the Gradio server.

1. Clone this repository and create an environment

Create and activate any Python 3.11.14 environment. The example below uses the standard-library venv:

git clone https://github.com/Asthag29/llama_omni_hindi_.git
cd llama_omni_hindi_
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

2. Install dependencies

Install the CUDA-enabled PyTorch build appropriate for your system, then install the project packages:

pip install torch==2.1.2+cu121 torchvision==0.16.2+cu121 torchaudio==2.1.2+cu121 \
  --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
pip install "f5_tts @ git+https://github.com/AI4Bharat/IndicF5.git@13f7c4d627cc10111aea8fe9c0039462cacacdc7"

⚡ Download model checkpoints

1. Download the Llama-3.1-8B-Omni model from 🤗 Huggingface and place it in models/llama/.

2. Download the Whisper-large-v3 model.

import whisper
model = whisper.load_model("large-v3", download_root="models/speech_encoder/")

3. IndicF5 is a gated repository. Request access on that page first, then download it:

hf download ai4bharat/IndicF5 --local-dir models/indicf5

4. Download this project's Hindi stage-2 adapter from 🤗 Huggingface:

hf download Pastaaaaa2003/hindi-llama-omni-model --local-dir models/hindi

Then run this check from the repository root. It confirms that every required checkpoint file is present and that Whisper matches the official SHA-256:

python check_models.py

Do not start Gradio or inference.py until this check succeeds.

Only the final stage-2 adapter is published by this project. Stage-1 checkpoints are training artifacts and are not required for inference.

The final layout must be:

models/
├── llama/            # LLaMA-Omni base model
├── speech_encoder/   # Whisper large-v3 weights
├── indicf5/          # IndicF5 model snapshot
└── hindi/            # Hindi stage-2 adapter and speech projector

IndicF5 load warning

IndicF5 was saved with Transformers 4.49.0 and stores GRN affine weights and Vocos ConvNeXt layer-scale as gamma / beta. This repo pins transformers==4.43.4 for LLaMA-Omni, which rewrites those substrings to weight / bias (old LayerNorm convention). Those tensors then miss the live modules, so the log reports unused grn.weight / convnext.N.weight and newly initialized grn.gamma / convnext.N.gamma. The snapshot itself is complete; this is a load-time rename, not a truncated checkpoint.

Resolution: do not bump Transformers, or LLaMA-Omni will break. Load IndicF5 with safetensors.torch.load_file("models/indicf5/model.safetensors") and load_state_dict(..., strict=False) so keys are not rewritten. Speech-to- speech still runs if you ignore the warning; only those 16 tensors are wrong.

🎧 Run the Gradio demo

Start each command in a separate terminal. Activate the environment and run the commands from the repository root.

Terminal 1 — controller

source .venv/bin/activate
python -m omni_speech.serve.controller --host 127.0.0.1 --port 21001

Terminal 2 — Hindi model worker

source .venv/bin/activate
python -m omni_speech.serve.model_worker \
  --host 127.0.0.1 \
  --port 21002 \
  --worker-address http://127.0.0.1:21002 \
  --controller-address http://127.0.0.1:21001 \
  --model-path models/llama \
  --model-name llama-omni-hindi \
  --checkpoint models/hindi \
  --config configs/stage_2.yaml \
  --device cuda

Wait until the worker reports that it has registered with the controller.

Terminal 3 — web interface

source .venv/bin/activate
python -m omni_speech.serve.gradio_web_server \
  --host 127.0.0.1 \
  --port 7860 \
  --controller-url http://127.0.0.1:21001 \
  --indicf5-model-path models/indicf5

Open http://127.0.0.1:7860/ and record or upload a Hindi speech question. The demo returns a Hindi audio answer using data/inference.wav as the fixed IndicF5 reference voice. Its fixed reference transcript is तुम कौन हो; Whisper is not used to transcribe reference audio.

🧪 Speech-input text-response test

This command tests the speech-understanding and Hindi-text generation stages without starting the web server. It prints the Hindi response in the terminal; use the Gradio demo above for the complete speech-to-speech response.

python -m omni_speech.infer.inference \
  --audio path/to/hindi-question.wav \
  --checkpoint models/hindi

The tracked data/inference.wav file can be used as the default test audio.

📊 Evaluation

Evaluation scripts live in evaluations/, and the full results discussion is maintained in evaluations/results/summary.md.

Run the evaluations from an activated environment:

python evaluations/indicQA.py
python evaluations/mt_bench_hi.py
python evaluations/if_eval_hi.py
python evaluations/gsm8k_hi.py

The scripts write result files under evaluations/results/.

Summary

Across the available evaluations, the fine-tuned model improves Hindi QA, answer overlap, semantic similarity, extraction, STEM, humanities, writing, and instruction-following metrics. These gains align with the Hindi instruction data used for training, which emphasizes direct question answering, transformation, extraction, formatting, and concise assistant responses.

🙏 Acknowledgements

• LLaMA-Omni: base speech-language architecture and code structure.
• Whisper: speech encoder for spoken input.
• IndicF5: Hindi/Indic speech-synthesis backend.
• AI4Bharat: Indic instruction and speech resources.