--- language: ca datasets: - CommonVoice - ParlamentParla --- # Casper: a Catalan Automatic Speech Recognition Model ## Table of Contents - [Model Details](#model-details) - [Uses](#uses) - [Limitations](#risks-limitations-and-biases) - [Training](#training) - [Evaluation](#evaluation) - [How to Get Started With the Model](#how-to-get-started-with-the-model) ## Model Details - **Model Description:** Casper is a state-of-the-art automatic speech recognition (ASR) model for Catalan by finetuning [whisper-small](https://huggingface.co/openai/whisper-small) on Catalan datasets. - **Developed by:** Yongjian Chen, Xavier Bonet-Casals, Mireia Farrús. - **Model Type:** Transformer-based encoder-decoder model - **Language(s):** Catalan - **Parent Model:** See the [whisper-small](https://huggingface.co/openai/whisper-small) for more information about the Whisper model. ## Uses #### Direct Use This model can be used for Catalan transcription task. ## Limitations - This model does not do punctuation and casing. - This model only supports audio samples of up to 30 seconds in duration as an inherent property from the parent model. To transcribe audio samples of more than 30 seconds, an additional chunking algorithm is needed to preprocess the samples. ## Training #### Training Data This model was fine-tuned on Common Voice and ParlamentParla Ctalan datasets. The [Common Voice](https://labs.mozilla.org/projects/common-voice/) project seeks to provide a platform where everyone can contribute their own voice to an open source multilingual data bank. The model developers used the the version [mozilla-foundation/common_voice_11_0 ca](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0) for training. The [ParlamentParla](https://huggingface.co/datasets/projecte-aina/parlament_parla) speech corpus contains audio segments extracted from recordings from the Catalan Parliment plenary sessions between 2007/07/11 - 2018/07/1. #### Training Procedure | Step | Training Loss | Epoch | Validation Loss | Validation WER | |:------:|:-------------:|:-----:|:---------------:|:--------------:| | 10000 | 0.11 | 0.43 | 0.14 | 6.49% | | 20000 | 0.09 | 0.86 | 0.13 | 6.28% | | 30000 | 0.05 | 1.28 | 0.13 | 5.91% | | 40000 | 0.06 | 1.71 | 0.12 | 5.90% | | 50000 | 0.03 | 2.14 | 0.13 | 5.70% | | 60000 | 0.03 | 2.57 | 0.13 | 5.82% | | 70000 | 0.03 | 3.00 | 0.13 | 5.56% | | 80000 | 0.01 | 3.43 | 0.14 | 5.64% | | 90000 | 0.01 | 3.85 | 0.14 | 5.59% | | 100000 | 0.01 | 4.28 | 0.14 | 5.50% | | 110000 | 0.01 | 4.71 | 0.14 | 5.42% | | 120000 | 0.01 | 5.14 | 0.15 | 5.83% | | 130000 | 0.01 | 5.57 | 0.15 | 5.65% | | 140000 | 0.01 | 6.00 | 0.15 | 5.54% | | 150000 | 0.003 | 6.42 | 0.15 | 5.56% | ## Evaluation #### Evaluation Data The evaluation dataset was created by the developer Xavier using the webinars from the University of Barcelona and is mostly domain-specific, surrounding topics of linguistics and language policy. The distribution of different specifications in the evaluation set is as follows: | Specification | Category | # | % | | ------------- | -------- | ---- | ------ | | Register | Formal | 88 | 57.14% | | | Informal | 66 | 42.86% | | Accent | Central | 33 | 21.43% | | | Balearic | 44 | 28.57% | | | Valencian| 44 | 28.57% | | | Western | 33 | 21.43% | | Gender | Male | 66 | 42.86% | | | Female | 88 | 57.14% | #### Evaluation Metrics The model developers evaluated Casper using two metrics: Word Error Rate (WER) and BLEU for machine translation (MT) from Catalan to Spanish and to English. Our fine-tuned Whisper model Casper significantly outperforms the zero-shot performance of the pre-trained Whisper model across different specifications on the evaluation dataset and such improvements lead to better outcomes in the MT downstream task. ##### WER | Specification | Category | Whisper-small | Fine-tuned Whisper-small | |:-------------:|:--------:|:-------------:|:------------------------:| | Register | Formal | 31.21% | 17.71% | | | Informal | 53.10% | 22.10% | | Accent | Central | 16.38% | 14.39% | | | Balearic | 29.76% | 29.68% | | | Valencian| 77.28% | 16.15% | | | Western | 21.10% | 17.48% | | Gender | Male | 57.49% | 15.14% | | | Female | 24.60% | 23.39% | | Total | / | 40.12% | 19.50% | ##### BLEU | Language | Target | Correct Transcript | Whisper-small | Fine-tuned whisper-small | |:--------:|:-------------------------:|:------------------:|:-------------:|:------------------------:| | Spanish | Human Translation | 83.5498 | 54.0836 | 63.7367 | | | Machine-assisted Translation | 84.219 | 54.5868 | 63.9436 | | English | Human Translation | 32.7 | 29.5 | 30.8 | | | Machine-assisted Translation | 33.5 | 30.3 | 31.6 | ## How to Get Started With the Model ```python from transformers import WhisperProcessor, WhisperForConditionalGeneration, WhisperConfig import torch import torchaudio # Load Casper and its processor : processor = WhisperProcessor.from_pretrained("maximilianchen/casper") model = WhisperForConditionalGeneration.from_pretrained("maximilianchen/casper"） # Load an audio sample ## Please make sure that the audio sample has been resampled to 16kHz before being loaded sa, sr = torchaudio.load(filename) sa = sa.squeeze(0) # Convert input audio sample into features inputs = processor(sa, sampling_rate=sr, return_tensors="pt").input_features # Generate token ids with torch.no_grad(): generated_ids = model.generate(inputs=inputs) # Decode token ids to text transcription = processor.batch_decode(generated_ids, skip_special_tokens=True) print(transcription) ['miraré de destacar molt breument què té d específic i essencial la coordinació aquesta estructura aparentment trivial on diem que coordinem dues categories aparentment iguals què té d especial què té de específic perquè és complicat si té raó o eies per això es parla d equivalència sintàctica i semàntica i llavors el repte és veure exactament què què té de sintàctica què té de semàntica'] ```