Kaushalb11/seamless-s2t-finetune

Direct Speech-to-Text Translation Fine-tuning

This repository contains scripts and resources for fine-tuning direct speech-to-text (S2T) translation models for English to Indian languages (Hindi, Gujarati, and Telugu) without intermediate text generation.

📁 Repository Structure

.
├── finetune_scripts/
│   ├── data_collect_clean.py      # Data collection and cleaning
│   ├── dataset_making.py           # Dataset preparation for fine-tuning
│   ├── finetune.py                 # Model fine-tuning script
│   ├── mt_eval.py                  # Evaluation and comparison script
│   ├── eval_finetuned/             # Evaluation results and predictions
│   ├── final-data/                 # Processed datasets (language-wise)
├──finetuned_models/           # Fine-tuned model checkpoints (.pt files)

🎯 Translation Tasks

• Hindi → English (hin-eng)
• Gujarati → English (guj-eng)
• Telugu → English (tel-eng)

Note: All translations are performed using direct speech-to-text without intermediate text generation.

🚀 Pipeline Overview

1. Data Collection and Cleaning (data_collect_clean.py)

Downloads and cleans audio data from five AI4Bharat datasets:

• ai4bharat/IndicVoices-ST
• ai4bharat/NPTEL
• ai4bharat/WordProject
• ai4bharat/Spoken-Tutorial
• ai4bharat/Mann-ki-Baat

Output: Cleaned dataset saved to Kaushalb11/multi-lang-s2t-dataset

2. Dataset Preparation (dataset_making.py)

Processes and structures the cleaned data for fine-tuning purposes.

Output: Language-wise datasets saved in final-data/ directory

3. Model Fine-tuning (finetune.py)

Fine-tunes the speech-to-text translation models for each target language.

Output: Model checkpoints saved as .pt files in finetuned_models/ directory

4. Evaluation (mt_eval.py)

Evaluates model performance by:

• Generating baseline metrics (BLEU and chrF scores)
• Loading fine-tuned model weights from .pt files
• Comparing baseline vs fine-tuned model performance
• Generating predictions

Output: Evaluation metrics and predictions saved in eval_finetuned/ directory

📊 Evaluation Metrics

• BLEU Score: Measures translation quality based on n-gram precision
• chrF Score: Character-level F-score for translation evaluation

🤗 Dataset

The cleaned dataset is available on Hugging Face:
Kaushalb11/multi-lang-s2t-dataset

📄 Output Files

final-data/

Contains processed datasets organized by target language:

• Hindi (hin)
• Gujarati (guj)
• Telugu (tel)

finetuned_models/

Contains trained model checkpoints in PyTorch format (.pt files)

eval_finetuned/

Contains:

• Baseline evaluation metrics
• Fine-tuned model evaluation metrics
• Comparison results
• Translation predictions

🔬 Model Architecture

Direct speech-to-text translation without intermediate text generation, enabling:

• Faster inference
• Reduced error propagation
• End-to-end optimization

🙏 Acknowledgments

• LTRC, IIIT Hyderabad for providing GPU facilities and mentorship
• AI4Bharat for providing the Indic language datasets