MMS

## [](#overview)Overview

The MMS model was proposed in [Scaling Speech Technology to 1,000+ Languages](https://arxiv.org/abs/2111.09296) by Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, Michael Auli

The abstract from the paper is the following:

_Expanding the language coverage of speech technology has the potential to improve access to information for many more people. However, current speech technology is restricted to about one hundred languages which is a small fraction of the over 7,000 languages spoken around the world. The Massively Multilingual Speech (MMS) project increases the number of supported languages by 10-40x, depending on the task. The main ingredients are a new dataset based on readings of publicly available religious texts and effectively leveraging self-supervised learning. We built pre-trained wav2vec 2.0 models covering 1,406 languages, a single multilingual automatic speech recognition model for 1,107 languages, speech synthesis models for the same number of languages, as well as a language identification model for 4,017 languages. Experiments show that our multilingual speech recognition model more than halves the word error rate of Whisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data._

Tips:

-   MMS is a speech model that accepts a float array corresponding to the raw waveform of the speech signal. The raw waveform should be pre-processed with [Wav2Vec2FeatureExtractor](/docs/transformers/v4.30.0/en/model_doc/wav2vec2#transformers.Wav2Vec2FeatureExtractor).
-   MMS model was trained using connectionist temporal classification (CTC) so the model output has to be decoded using [Wav2Vec2CTCTokenizer](/docs/transformers/v4.30.0/en/model_doc/wav2vec2#transformers.Wav2Vec2CTCTokenizer).
-   MMS can load different language adapter weights for different languages via [load\_adapter()](/docs/transformers/v4.30.0/en/model_doc/wav2vec2#transformers.Wav2Vec2ForCTC.load_adapter). Language adapters only consists of roughly 2 million parameters and can therefore be efficiently loaded on the fly when needed.

Relevant checkpoints can be found under [https://huggingface.co/models?other=mms](https://huggingface.co/models?other=mms).

MMS’s architecture is based on the Wav2Vec2 model, so one can refer to [Wav2Vec2’s documentation page](wav2vec2).

The original code can be found [here](https://github.com/facebookresearch/fairseq/tree/main/examples/mms).

## [](#inference)Inference

By default MMS loads adapter weights for English, but those can be easily switched out for another language. Let’s look at an example.

First, we load audio data in different languages using the [Datasets](https://github.com/huggingface/datasets).

```
from datasets import load_dataset, Audio


stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "en", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
en_sample = next(iter(stream_data))["audio"]["array"]


stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "fr", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
fr_sample = next(iter(stream_data))["audio"]["array"]```

Next, we load the model and processor

```
from transformers import Wav2Vec2ForCTC, AutoProcessor
import torch

model_id = "facebook/mms-1b-all"

processor = AutoProcessor.from_pretrained(model_id)
model = Wav2Vec2ForCTC.from_pretrained(model_id)```

Now we process the audio data, pass the processed audio data to the model and transcribe the model output, just like we usually do for [Wav2Vec2ForCTC](/docs/transformers/v4.30.0/en/model_doc/wav2vec2#transformers.Wav2Vec2ForCTC).

```
inputs = processor(en_sample, sampling_rate=16_000, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs).logits

ids = torch.argmax(outputs, dim=-1)[0]
transcription = processor.decode(ids)
```

We can now keep the same model in memory and simply switch out the language adapters by calling the convenient [load\_adapter()](/docs/transformers/v4.30.0/en/model_doc/wav2vec2#transformers.Wav2Vec2ForCTC.load_adapter) function for the model and [set\_target\_lang()](/docs/transformers/v4.30.0/en/model_doc/wav2vec2#transformers.Wav2Vec2CTCTokenizer.set_target_lang) for the tokenizer. We pass the target language as an input - `"fra"` for French.

```
processor.tokenizer.set_target_lang("fra")
model.load_adapter("fra")

inputs = processor(fr_sample, sampling_rate=16_000, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs).logits

ids = torch.argmax(outputs, dim=-1)[0]
transcription = processor.decode(ids)
```

In the same way the language can be switched out for all other supported languages. Please have a look at:

```
processor.tokenizer.vocab.keys()```

to see all supported languages.