M-CTC-T

## [](#overview)Overview

The M-CTC-T model was proposed in [Pseudo-Labeling For Massively Multilingual Speech Recognition](https://arxiv.org/abs/2111.00161) by Loren Lugosch, Tatiana Likhomanenko, Gabriel Synnaeve, and Ronan Collobert. The model is a 1B-param transformer encoder, with a CTC head over 8065 character labels and a language identification head over 60 language ID labels. It is trained on Common Voice (version 6.1, December 2020 release) and VoxPopuli. After training on Common Voice and VoxPopuli, the model is trained on Common Voice only. The labels are unnormalized character-level transcripts (punctuation and capitalization are not removed). The model takes as input Mel filterbank features from a 16Khz audio signal.

The abstract from the paper is the following:

_Semi-supervised learning through pseudo-labeling has become a staple of state-of-the-art monolingual speech recognition systems. In this work, we extend pseudo-labeling to massively multilingual speech recognition with 60 languages. We propose a simple pseudo-labeling recipe that works well even with low-resource languages: train a supervised multilingual model, fine-tune it with semi-supervised learning on a target language, generate pseudo-labels for that language, and train a final model using pseudo-labels for all languages, either from scratch or by fine-tuning. Experiments on the labeled Common Voice and unlabeled VoxPopuli datasets show that our recipe can yield a model with better performance for many languages that also transfers well to LibriSpeech._

This model was contributed by [cwkeam](https://huggingface.co/cwkeam). The original code can be found [here](https://github.com/flashlight/wav2letter/tree/main/recipes/mling_pl).

## [](#documentation-resources)Documentation resources

-   [Automatic speech recognition task guide](../tasks/asr)

Tips:

-   The PyTorch version of this model is only available in torch 1.9 and higher.

## [](#transformers.MCTCTConfig)MCTCTConfig

### class transformers.MCTCTConfig

[](#transformers.MCTCTConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/mctct/configuration_mctct.py#L29)

( vocab\_size = 8065hidden\_size = 1536num\_hidden\_layers = 36intermediate\_size = 6144num\_attention\_heads = 4attention\_head\_dim = 384max\_position\_embeddings = 920layer\_norm\_eps = 1e-05layerdrop = 0.3hidden\_act = 'relu'initializer\_range = 0.02hidden\_dropout\_prob = 0.3attention\_probs\_dropout\_prob = 0.3pad\_token\_id = 1bos\_token\_id = 0eos\_token\_id = 2conv\_glu\_dim = 1conv\_dropout = 0.3num\_conv\_layers = 1conv\_kernel = (7,)conv\_stride = (3,)input\_feat\_per\_channel = 80input\_channels = 1conv\_channels = Nonectc\_loss\_reduction = 'sum'ctc\_zero\_infinity = False\*\*kwargs )

This is the configuration class to store the configuration of a [MCTCTModel](/docs/transformers/v4.30.0/en/model_doc/mctct#transformers.MCTCTModel). It is used to instantiate an M-CTC-T model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the M-CTC-T [speechbrain/m-ctc-t-large](https://huggingface.co/speechbrain/m-ctc-t-large) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import MCTCTConfig, MCTCTModel

>>> 
>>> configuration = MCTCTConfig()

>>> 
>>> model = MCTCTModel(configuration)

>>> 
>>> configuration = model.config```

## [](#transformers.MCTCTFeatureExtractor)MCTCTFeatureExtractor

( feature\_size = 80sampling\_rate = 16000padding\_value = 0.0hop\_length = 10win\_length = 25win\_function = 'hamming\_window'frame\_signal\_scale = 32768.0preemphasis\_coeff = 0.97mel\_floor = 1.0normalize\_means = Truenormalize\_vars = Truereturn\_attention\_mask = False\*\*kwargs )

Constructs a M-CTC-T feature extractor.

This feature extractor inherits from [SequenceFeatureExtractor](/docs/transformers/v4.30.0/en/main_classes/feature_extractor#transformers.SequenceFeatureExtractor) which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. This code has been adapted from Flashlight’s C++ code. For more information about the implementation, one can refer to this [notebook](https://colab.research.google.com/drive/1GLtINkkhzms-IsdcGy_-tVCkv0qNF-Gt#scrollTo=pMCRGMmUC_an) that takes the user step-by-step in the implementation.

( raw\_speech: typing.Union\[numpy.ndarray, typing.List\[float\], typing.List\[numpy.ndarray\], typing.List\[typing.List\[float\]\]\]padding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Falsemax\_length: typing.Optional\[int\] = Nonetruncation: bool = Falsepad\_to\_multiple\_of: typing.Optional\[int\] = Nonereturn\_attention\_mask: typing.Optional\[bool\] = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = Nonesampling\_rate: typing.Optional\[int\] = None\*\*kwargs )

Main method to featurize and prepare for the model one or several sequence(s). sequences. It returns the log-mel spectrogram of the input audio, as implemented in the original Flashlight MFSC feature extraction code.

## [](#transformers.MCTCTProcessor)MCTCTProcessor

### class transformers.MCTCTProcessor

[](#transformers.MCTCTProcessor)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/mctct/processing_mctct.py#L24)

( feature\_extractortokenizer )

Parameters

-   [](#transformers.MCTCTProcessor.feature_extractor)**feature\_extractor** (`MCTCTFeatureExtractor`) — An instance of [MCTCTFeatureExtractor](/docs/transformers/v4.30.0/en/model_doc/mctct#transformers.MCTCTFeatureExtractor). The feature extractor is a required input.
-   [](#transformers.MCTCTProcessor.tokenizer)**tokenizer** (`AutoTokenizer`) — An instance of [AutoTokenizer](/docs/transformers/v4.30.0/en/model_doc/auto#transformers.AutoTokenizer). The tokenizer is a required input.

Constructs a MCTCT processor which wraps a MCTCT feature extractor and a MCTCT tokenizer into a single processor.

[MCTCTProcessor](/docs/transformers/v4.30.0/en/model_doc/mctct#transformers.MCTCTProcessor) offers all the functionalities of [MCTCTFeatureExtractor](/docs/transformers/v4.30.0/en/model_doc/mctct#transformers.MCTCTFeatureExtractor) and [AutoTokenizer](/docs/transformers/v4.30.0/en/model_doc/auto#transformers.AutoTokenizer). See the [**call**()](/docs/transformers/v4.30.0/en/model_doc/mctct#transformers.MCTCTProcessor.__call__) and [decode()](/docs/transformers/v4.30.0/en/model_doc/mctct#transformers.MCTCTProcessor.decode) for more information.

When used in normal mode, this method forwards all its arguments to MCTCTFeatureExtractor’s [**call**()](/docs/transformers/v4.30.0/en/model_doc/mctct#transformers.MCTCTFeatureExtractor.__call__) and returns its output. If used in the context `as_target_processor()` this method forwards all its arguments to AutoTokenizer’s `__call__()`. Please refer to the doctsring of the above two methods for more information.

#### from\_pretrained

[](#transformers.MCTCTProcessor.from_pretrained)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/processing_utils.py#L153)

( pretrained\_model\_name\_or\_path\*\*kwargs )

Parameters

-   [](#transformers.MCTCTProcessor.from_pretrained.pretrained_model_name_or_path)**pretrained\_model\_name\_or\_path** (`str` or `os.PathLike`) — This can be either:
    
    -   a string, the _model id_ of a pretrained feature\_extractor hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`.
    -   a path to a _directory_ containing a feature extractor file saved using the [save\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/feature_extractor#transformers.FeatureExtractionMixin.save_pretrained) method, e.g., `./my_model_directory/`.
    -   a path or url to a saved feature extractor JSON _file_, e.g., `./my_model_directory/preprocessor_config.json`. \*\*kwargs — Additional keyword arguments passed along to both [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/feature_extractor#transformers.FeatureExtractionMixin.from_pretrained) and `~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`.
    

Instantiate a processor associated with a pretrained model.

This class method is simply calling the feature extractor [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/feature_extractor#transformers.FeatureExtractionMixin.from_pretrained), image processor [ImageProcessingMixin](/docs/transformers/v4.30.0/en/internal/image_processing_utils#transformers.ImageProcessingMixin) and the tokenizer `~tokenization_utils_base.PreTrainedTokenizer.from_pretrained` methods. Please refer to the docstrings of the methods above for more information.

#### save\_pretrained

[](#transformers.MCTCTProcessor.save_pretrained)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/processing_utils.py#L91)

( save\_directorypush\_to\_hub: bool = False\*\*kwargs )

Parameters

-   [](#transformers.MCTCTProcessor.save_pretrained.save_directory)**save\_directory** (`str` or `os.PathLike`) — Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist).
-   [](#transformers.MCTCTProcessor.save_pretrained.push_to_hub)**push\_to\_hub** (`bool`, _optional_, defaults to `False`) — Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs — Additional key word arguments passed along to the [push\_to\_hub()](/docs/transformers/v4.30.0/en/internal/image_processing_utils#transformers.ImageProcessingMixin.push_to_hub) method.

Saves the attributes of this processor (feature extractor, tokenizer…) in the specified directory so that it can be reloaded using the [from\_pretrained()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Processor.from_pretrained) method.

This class method is simply calling [save\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/feature_extractor#transformers.FeatureExtractionMixin.save_pretrained) and [save\_pretrained()](/docs/transformers/v4.30.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.save_pretrained). Please refer to the docstrings of the methods above for more information.

This method forwards all its arguments to AutoTokenizer’s [batch\_decode()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.batch_decode). Please refer to the docstring of this method for more information.

This method forwards all its arguments to AutoTokenizer’s [decode()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.decode). Please refer to the docstring of this method for more information.

## [](#transformers.MCTCTModel)MCTCTModel

### class transformers.MCTCTModel

[](#transformers.MCTCTModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/mctct/modeling_mctct.py#L661)

( config )

Parameters

-   [](#transformers.MCTCTModel.config)**config** ([MCTCTConfig](/docs/transformers/v4.30.0/en/model_doc/mctct#transformers.MCTCTConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The bare M-CTC-T Model transformer outputting raw hidden-states without any specific head on top. This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.MCTCTModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/mctct/modeling_mctct.py#L671)

( input\_features: Tensorattention\_mask: typing.Optional\[torch.Tensor\] = Nonehead\_mask: typing.Optional\[torch.Tensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.BaseModelOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutput) or `tuple(torch.FloatTensor)`

The [MCTCTModel](/docs/transformers/v4.30.0/en/model_doc/mctct#transformers.MCTCTModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoProcessor, MCTCTModel
>>> from datasets import load_dataset
>>> import torch

>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate

>>> processor = AutoProcessor.from_pretrained("speechbrain/m-ctc-t-large")
>>> model = MCTCTModel.from_pretrained("speechbrain/m-ctc-t-large")

>>> 
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_ids = torch.argmax(logits, dim=-1)

>>> 
>>> transcription = processor.batch_decode(predicted_ids)
>>> transcription[0]
[1, 195, 1536]

>>> inputs["labels"] = processor(text=dataset[0]["text"], return_tensors="pt").input_ids

>>> 
>>> loss = model(**inputs).loss```

## [](#transformers.MCTCTForCTC)MCTCTForCTC

### class transformers.MCTCTForCTC

[](#transformers.MCTCTForCTC)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/mctct/modeling_mctct.py#L721)

( config )

Parameters

-   [](#transformers.MCTCTForCTC.config)**config** ([MCTCTConfig](/docs/transformers/v4.30.0/en/model_doc/mctct#transformers.MCTCTConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

MCTCT Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.MCTCTForCTC.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/mctct/modeling_mctct.py#L741)

( input\_features: Tensorattention\_mask: typing.Optional\[torch.Tensor\] = Nonehead\_mask: typing.Optional\[torch.Tensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = Nonelabels: typing.Optional\[torch.LongTensor\] = None ) → [transformers.modeling\_outputs.CausalLMOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.CausalLMOutput) or `tuple(torch.FloatTensor)`

The [MCTCTForCTC](/docs/transformers/v4.30.0/en/model_doc/mctct#transformers.MCTCTForCTC) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoProcessor, MCTCTForCTC
>>> from datasets import load_dataset
>>> import torch

>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate

>>> processor = AutoProcessor.from_pretrained("speechbrain/m-ctc-t-large")
>>> model = MCTCTForCTC.from_pretrained("speechbrain/m-ctc-t-large")

>>> 
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_ids = torch.argmax(logits, dim=-1)

>>> 
>>> transcription = processor.batch_decode(predicted_ids)
>>> transcription[0]
"Mr. Quilter is the apostle of the middle classes, and we're glad to welcome his gospel."

>>> inputs["labels"] = processor(text=dataset[0]["text"], return_tensors="pt").input_ids

>>> 
>>> loss = model(**inputs).loss
>>> round(loss.item(), 2)
1885.65```