1
---
2
language: ta
3
#datasets:
4
#- Interspeech 2021
5
metrics:
6
- wer
7
tags:
8
- audio
9
- automatic-speech-recognition
10
- speech
11
license: mit
12
model-index:
13
- name: Wav2Vec2 Vakyansh Tamil Model by Harveen Chadha
14
  results:
15
  - task:
16
      name: Speech Recognition
17
      type: automatic-speech-recognition
18
    dataset:
19
      name: Common Voice ta
20
      type: common_voice
21
      args: ta
22
    metrics:
23
    - name: Test WER
24
      type: wer
25
      value: 53.64
26
---
27
28
## Pretrained Model
29
30
Fine-tuned on Multilingual Pretrained Model [CLSRIL-23](https://arxiv.org/abs/2107.07402). The original fairseq checkpoint is present [here](https://github.com/Open-Speech-EkStep/vakyansh-models). When using this model, make sure that your speech input is sampled at 16kHz.
31
32
**Note: The result from this model is without a language model so you may witness a higher WER in some cases.**
33
34
## Dataset
35
36
This model was trained on 4200 hours of Hindi Labelled Data. The labelled data is not present in public domain as of now.
37
38
## Training Script
39
40
Models were trained using experimental platform setup by Vakyansh team at Ekstep. Here is the [training repository](https://github.com/Open-Speech-EkStep/vakyansh-wav2vec2-experimentation).
41
42
In case you want to explore training logs on wandb they are [here](https://wandb.ai/harveenchadha/tamil-finetuning-multilingual).
43
44
45
## [Colab Demo](https://github.com/harveenchadha/bol/blob/main/demos/hf/tamil/hf_tamil_tnm_4200_demo.ipynb)
46
47
## Usage
48
49
The model can be used directly (without a language model) as follows:
50
51
```python
52
import soundfile as sf
53
import torch
54
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
55
import argparse
56
57
def parse_transcription(wav_file):
58
    # load pretrained model
59
    processor = Wav2Vec2Processor.from_pretrained("Harveenchadha/vakyansh-wav2vec2-tamil-tam-250")
60
    model = Wav2Vec2ForCTC.from_pretrained("Harveenchadha/vakyansh-wav2vec2-tamil-tam-250")
61
62
    # load audio
63
    audio_input, sample_rate = sf.read(wav_file)
64
65
    # pad input values and return pt tensor
66
    input_values = processor(audio_input, sampling_rate=sample_rate, return_tensors="pt").input_values
67
68
    # INFERENCE
69
    # retrieve logits & take argmax
70
    logits = model(input_values).logits
71
    predicted_ids = torch.argmax(logits, dim=-1)
72
73
    # transcribe
74
    transcription = processor.decode(predicted_ids[0], skip_special_tokens=True)
75
    print(transcription)
76
77
```
78
79
80
## Evaluation
81
The model can be evaluated as follows on the hindi test data of Common Voice. 
82
83
```python
84
85
import torch
86
import torchaudio
87
from datasets import load_dataset, load_metric
88
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
89
import re
90
91
test_dataset = load_dataset("common_voice", "ta", split="test")
92
wer = load_metric("wer")
93
94
processor = Wav2Vec2Processor.from_pretrained("Harveenchadha/vakyansh-wav2vec2-tamil-tam-250")
95
model = Wav2Vec2ForCTC.from_pretrained("Harveenchadha/vakyansh-wav2vec2-tamil-tam-250")
96
model.to("cuda")
97
98
resampler = torchaudio.transforms.Resample(48_000, 16_000)
99
100
chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“]'
101
102
# Preprocessing the datasets.
103
# We need to read the aduio files as arrays
104
def speech_file_to_array_fn(batch):
105
  batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
106
  speech_array, sampling_rate = torchaudio.load(batch["path"])
107
  batch["speech"] = resampler(speech_array).squeeze().numpy()
108
  return batch
109
110
test_dataset = test_dataset.map(speech_file_to_array_fn)
111
112
# Preprocessing the datasets.
113
# We need to read the aduio files as arrays
114
def evaluate(batch):
115
  inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
116
117
  with torch.no_grad():
118
      logits = model(inputs.input_values.to("cuda")).logits
119
120
      pred_ids = torch.argmax(logits, dim=-1)
121
      batch["pred_strings"] = processor.batch_decode(pred_ids, skip_special_tokens=True)
122
      return batch
123
124
result = test_dataset.map(evaluate, batched=True, batch_size=8)
125
126
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
127
128
```
129
130
**Test Result**: 53.64 %
131
132
[**Colab Evaluation**](https://github.com/harveenchadha/bol/blob/main/demos/hf/tamil/hf_vakyansh_tamil_tnm_4200_evaluation_common_voice.ipynb) 
133
134
## Credits
135
Thanks to Ekstep Foundation for making this possible. The vakyansh team will be open sourcing speech models in all the Indic Languages.