M-CTC-T Model
Author: cwkeam
Pull request: yes

# M-CTC-T Model

See: https://twitter.com/PatrickPlaten/status/1504875528469331972

Very early stage WIP for the mCTC model from the paper [Pseudo-Labeling For Massively Multilingual Speech Recognition](https://arxiv.org/pdf/2111.00161.pdf). 

Making this WIP PR in this particularly early stage to track many of the links & issues that keep coming up in development (PR code itself is not worth reading into atm.

## NOTES

1. This work is mainly based on the paper [Pseudo-Labeling For Massively Multilingual Speech Recognition](https://arxiv.org/pdf/2111.00161.pdf), which is essentially a continuation of the following works (ASR -> slimIPL -> mCTC):
    - [SlimIPL: Language-Model-Free Iterative Pseudo-Labeling](https://arxiv.org/abs/2010.11524)
    - [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://arxiv.org/abs/1911.08460)
    - which were all developed with the [Flashlight wav2letter++](https://github.com/flashlight/wav2letter) package
2. The original C++ code reference for `mCTC` is [here](https://github.com/flashlight/wav2letter/blob/main/recipes/mling_pl/mling_large.cpp)
    - Which is an adoption of the original [slimIPL code](https://github.com/flashlight/wav2letter/blob/main/recipes/slimIPL/100h_supervised.cpp) which provides useful reference. 
    - Both of which are very simple scripts with major abstractions (like `layer = std::make_shared<fl::Transformer>(768, 192, 3072, 4, 920, 0.3, 0.3, false, false)`, so the following code must be referenced:
        - [Transformers](https://github.com/flashlight/flashlight/blob/main/flashlight/fl/contrib/modules/Transformer.cpp)
        - [Multi-Head Self Attention](https://github.com/flashlight/flashlight/blob/main/flashlight/fl/autograd/Functions.h)
        - [Conv2D](https://github.com/flashlight/flashlight/blob/main/flashlight/fl/nn/modules/Conv2D.cpp)
    - And the weights for the `mctc-large` model can be found [here](https://dl.fbaipublicfiles.com/wav2letter/mling_pl/checkpoint_cv_finetune.bin). 
    - Weights for the other model sizes reported in the paper are not released.
3. [Position Embeddings](https://github.com/flashlight/flashlight/blob/main/flashlight/fl/contrib/modules/Transformer.cpp#L93): `posEmb = tile(params_[0].as(encoderInput.type()), af::dim4(1, 1, nHeads_ * bsz));`
    - Pseudo-Labeling Paper: "...self-attention dimension 768, using the relative position embeddings of [15]."
```
# https://github.com/flashlight/flashlight/blob/main/flashlight/fl/contrib/modules/Transformer.cpp#L93
auto encoderInput = input.at(input.size() - 2);
int n = input[0].dims(1), bsz = input[0].dims(2);
# ...
posEmb = tile(params_[0].as(encoderInput.type()), af::dim4(1, 1, nHeads_ * bsz));

# https://github.com/flashlight/flashlight/blob/main/flashlight/fl/autograd/Functions.cpp#L1276
auto scores = matmulNT(q, k);
if (!posEmb.isempty()) {
  int n = posEmb.dims(0) / 2 - offset;
  auto pscores =
      relativePositionEmbeddingRotate(matmulNT(posEmb.as(q.type()), q));
  scores = scores + transpose(pscores.rows(n, n + k.dims(0) - 1));
}
```

4. Gated Linear Unit. Though not stated in the original paper, the slimIPL paper, or the End-to-End ASR paper, the code shows:
```
convFrontend_->add(
    // std::make_shared<fl::Conv2D>(nFeature, 1536, 7, 1, 3, 1, -1, 0, 1,
    // 1));
    std::make_shared<fl::Conv2D>(nFeature, 3072, 7, 1, 3, 1, -1, 0, 1, 1));
convFrontend_->add(std::make_shared<fl::GatedLinearUnit>(2));
...
# The documentation doesn't state its parameters and I did a long search of the code base but couldn't find this info.
# But I think it's safe to assume 2 refers to the dimensions of GLU
```

## Issue Tracking
- [ ]  **Naming**: Currently we've opted for `mCTC` since throughout the above works the author refer to it as a "CTC Transformer Model", but this leads to awkward uses later on when talking about `mCTCForCTC` for downstream tasks (and even `CTCForCTC` for single-language models). Seems like using a common loss function to define a model might lead to troubles later down the line.
- [x] **Feature Extraction (raw audio -> model inputs)**: Unlike other audio processing modules in `transformers`, the input data is a log-mel spectrogram. As explained in the paper: 

    [From the original Pseudo-Labeling paper]
    *The input to the encoder is a sequence of **80**-dimensional **log-mel filterbank** frames, extracted using **25ms Hamming** windows every **10ms** from the **16kHz** audio signal.* 

    [For more details, from the slimIPL paper]
    *We keep the original **16kHz** sampling rate and compute **log-mel** filterbanks with **80** coefficients for a **25ms** sliding window, strided by **10ms**. All features are **normalized** to have zero mean and unit variance per input sequence before feeding them into the acoustic model.*

     For us, such feature processing could be implemented with the following process:
    - [librosa melspectrogram](https://librosa.org/doc/main/generated/librosa.feature.melspectrogram.html) function that calls the [Scipy Hamming windows](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.windows.hamming.html#scipy.signal.windows.hamming) function.
    - I'm conflicted between creating a `SpectrogramFeatureExtractor` in the `transformers` library that would involve `import librosa` in `feature_extraction_sequence_utils.py`, or adding a `spectrogram.py` in [datasets.features](https://github.com/huggingface/datasets/blob/master/src/datasets/features/audio.py). The latter separates this task of `audio signal -> spectrogram` to the user-side, while providing an easy interface with the `datasets` library.
- [x] **Testing**: Under development
- [x] **MCTCModel**
- [x] **MCTCForCTC**
- [x] **MCTCTokenizer**
- [x] **MCTCFeatureExtractor**: 
- [x] **MCTCProcessor**



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
Pinging @patrickvonplaten @anton-l @patil-suraj for a midway check-in.

Still have the following to resolve (some mentioned in slack):

- [ ] Loading weights from the binary file found [here](https://github.com/flashlight/wav2letter/tree/main/recipes/mling_pl).
- [ ] `Wav2Vec2` has a `Wav2Vec2Tokenizer` and a `Wav2Vec2CTCTokenizer`. Should there be both for this too (`MCTCTokenizer`, `MCTCCTCTokenizer`)? For now I've just made an `MCTCTokenizer`, taking the role analogous to `Wav2Vec2CTCTokenizer`.
- [ ] As the initial description states, the original model code uses a GLU layer @ dim=2, which requires the input to that layer to have even length. Due to CNN resizing, this isn't guaranteed, and I've encountered a related error during development.  
- [ ] Properly integrating attention masks (right now it's ignored in several places due to temporary issues)
- [ ] vocabulary in the original code doesnt have analogous tokens to <pad> 
 and such that are index 0~3 for Wav2Vec2Tokenizer implementation. So currently the vocab.json doesn’t have tokens like them (its copied directly from the token list in the original repo)




Hey @cwkeam,

Thanks for making so much progress already! 

I think in a next step now it's very important that we are able to run the following original code:
https://github.com/flashlight/wav2letter/tree/main/recipes/mling_pl#inference
to be able to compare intermediate results of the model in C++ with the ported model in PyTorch.

Once we get this script to run, we should try to save the C++ weights somehow in PyTorch. Think for this we need to adapt the code here:
https://github.com/flashlight/flashlight/blob/942414d530f19a1fca965499c8a5441405bd3a16/flashlight/app/asr/Test.cpp#L82


Model working after feature extraction step:
https://colab.research.google.com/drive/10-03jHBUcp2P4DMF-OeeSE0lDS0xux7y?usp=sharing

This was a result of a lot of debugging and manual C++ translations, the results of which still need a lot of optimizing and cleaning. The process I took can be found in this separate branch: https://github.com/cwkeam/transformers/tree/mctc-model-debugging/src


Amazing job @cwkeam ! Feature extraction seems to be :heavy_check_mark: :-) Next the modeling code should be tackled? Think a similar debugging strategy as for the feature extraction can be used here :-)

@patrickvonplaten yep I've started the feature extraction debugging process in [this notebook](https://colab.research.google.com/drive/1Gw4DcnwvPNKxgMFkeDaADbJ7G6fhhcQk?usp=sharing). Hopefully I'll have a solution soon!

_The documentation is not available anymore as the PR was closed or merged._

MFSC feature extraction is done and the functionality works as expected!

```python
!pip install gdown
!gdown --id 1GvsB9aqrlpJSvLNlz1Yv6lT_Je9jyNtB # audio.flac

import torch
import torchaudio
from transformers import MCTCForCTC, MCTCTokenizer, MCTCFeatureExtractor

model = MCTCForCTC.from_pretrained("cwkeam/mctc-large")
tokenizer = MCTCTokenizer.from_pretrained("cwkeam/mctc-large")
feature_extractor = MCTCFeatureExtractor.from_pretrained("cwkeam/mctc-large")

waveform, sample_rate = torchaudio.load("audio.flac")
features = feature_extractor(waveform[0])
input_features = torch.Tensor(features.input_features)

my_logits = model(input_features).logits[0]

pad_token = tokenizer.convert_tokens_to_ids(["<pad>"])[0]

pred_ids = torch.stack([token for token in pred_ids if token != pad_token])

output = tokenizer.batch_decode(pred_ids)
print(output)
>>> ['', 'H', 'e', '', 'm', 'u', 's', 's', 't', '', 'h', 'a', 'v', 'e', 'e', '', 'r', 'e', 'e', 'a', 'a', 'l', 'l', 'i', 'z', 'e', 'd', '', '', 'I', '', 'w', 'a', 's', 's', '', '', 'a', '', 's', 't', 'r', 'a', 'n', 'n', 'g', 'e', 'r', 'r', '', 'a', 'n', 'd', '', 'w', 'i', 's', 'h', 'e', 'd', '', 'y', 'o', 'u', 'u', '', 't', 'u', 'n', 'n', 'd', 'e', 'r', '', '', 'h', 'i', 's', '', '', 'h', 'o', 's', 's', 'p', 'i', 't', 'a', 'l', 'l', 'i', 'i', 't', 'y', '', 't', 'o', '', 'm', 'e', ';', '', 'I', '', 'a', 'c', 'c', 'e', 'p', 't', 't', 'e', 'd', '', '', 'i', 't', '', '', 'g', 'r', 'a', 'a', 't', 'e', 'f', 'f', 'u', 'l', 'l', 'y', ',', '', '', 'I', '', 'c', 'l', 'a', 's', 'p', '', 'h', 'i', 's', '', 'h', 'a', 'n', 'd', '', 'h', 'e', '', 'p', 'r', 'e', 's', 's', 'e', 'd', '', '', 'm', 'i', 'n', 'd', '.', '']
```

The following notebook shows the replicable step-by-step process behind developing the feature extraction process: [Debugging Flashlight MFSC Feature Extraction.ipynb](https://colab.research.google.com/drive/1Gw4DcnwvPNKxgMFkeDaADbJ7G6fhhcQk?usp=sharing)

[This notebook](https://colab.research.google.com/drive/1NnL819XoCGzo3fqKn3BySz0pyOnjGKpQ) directly plugs in the results of the above notebook, and [this notebook](https://colab.research.google.com/drive/16I-lJvG5EwlwY5f7AucTEO49pPGwIyGS?usp=sharing) tests directly importing `MCTCXXX` modules from `cwkeam/transformers`, loading the weights with `from_pretrained("cwkeam/mctc-large")`, and running ASR on the same audio.

Few things that might need fixing & added:
- [ ] Might need to add tests & further optimize some of my manual translations of the C++ code in the modeling & feature extraction code.
- [ ] The character/word offset values are a bit different from those found in `Wav2Vec2ForCTC`. The Wav2Vec2 one is [here](https://github.com/huggingface/transformers/blob/main/tests/wav2vec2/test_tokenization_wav2vec2.py#L654) and mine is [here](https://github.com/cwkeam/transformers/blob/mctc-model/tests/mctc/test_tokenization_mctc.py#L426).
- [ ] Been using the name `MCTC` because thats what I started off with initially, now wondering if I should replace everything with `MCTCT`; though after development I've felt like this might be preferrable over ending up with  `MCTCTTokenizer` `MCTCTForCTC`. 
- [ ] Language identification head since I think the published weights include them too
- [ ] Replacing the \<pad\> token with "\<blank\>" 


@cwkeam - amazing job on making this model work! This is an extremely difficult port. 

I think there are still some things left to clean up, but overall the PR is already in a great shape. For now, I'd like to focus on two things:

1. IMO, we can fully remove the tokenizer no? The tokenizeir is 1-to-1 the same as the Wav2Vec2 tokenizer. I adapted your repo here: https://huggingface.co/cwkeam/mctc-large/commit/ce4eb45713b5f6873dcbb0e7bf35902c440cccb4 so that the Wav2Vec2CTCTokenizer can be used by default. I.e. now the following code snippet should work out of the box:

```python
#!/usr/bin/env python3
import torch
import torchaudio
from transformers import MCTCForCTC, AutoTokenizer, MCTCFeatureExtractor

model = MCTCForCTC.from_pretrained("./mctc-large")
tokenizer = AutoTokenizer.from_pretrained("./mctc-large")
feature_extractor = MCTCFeatureExtractor.from_pretrained("./mctc-large")

waveform, sample_rate = torchaudio.load("audio.flac")

features = feature_extractor(waveform[0], sampling_rate=sample_rate, return_tensors="pt")
input_features = features.input_features

my_logits = model(input_features).logits

# CTC
pred_ids = torch.argmax(my_logits, dim=-1)
output = tokenizer.batch_decode(pred_ids)

print("output", output)
```

-> As you can see one can use the `AutoTokenizer` functionality and the tokenizer re-uses to 100% the existing Wav2Vec2 one. Would it be ok if we delete all tokenizer logic in this PR? The advantage would be:
- less maintenance 
- All features of Wav2Vec2 are tested and can be used (time stamps, etc...)
- Phoneme tokenizers would work as well

2. Let's discuss the naming of the model maybe a bit in Slack :-) Once we have a good name (or we keep this one) - let's update everything

After 1. and 2. I think we can take another review iteration.

@patrickvonplaten 100% agree with the tokenization part and have removed them accordingly. Everything else should be fixed as well! You can also find the model card [here](https://huggingface.co/cwkeam/mctct-large) that we should move over to SpeechBrain's page. 

@cwkeam, let me know if you need help with anything or when your model is ready for a final review

@patrickvonplaten I was stuck on some importing issue but now it's all working! The one test that it's failing right now is [this](https://app.circleci.com/pipelines/github/huggingface/transformers/39945/workflows/62fc3038-3d21-45f5-a097-003932b2d3cb/jobs/450467?invite=true#step-111-5088)

```
SKIPPED [1] tests/pipelines/test_pipelines_image_segmentation.py:300: test is slow
FAILED tests/pipelines/test_pipelines_image_classification.py::ImageClassificationPipelineTests::test_pt_DeiTConfig_DeiTForImageClassification_notokenizer_DeiTFeatureExtractor
===== 1 failed, 464 passed, 350 skipped, 526 warnings in 113.76s (0:01:53) =====
```

I think it's ready for a final review!

@cwkeam - amazing job! Think we are really close to merging the PR here :-) 

I've run the slow tests locally and I'm getting some errors for the following tests:

```bash
FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_normal - AssertionE...
FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_normal_batched - Ty...
FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_robust_batched - Ty...
```

The rest all passes - could you try fixing those? 

Also we should make sure that the documentation example works correctly. The doc tests seem to fail at the moment. Could you follow the explanations here: https://github.com/huggingface/transformers/tree/main/docs#testing-documentation-examples  to test the example doc strings? :-)

Let me know if something is not clear!



> @cwkeam - amazing job! Think we are really close to merging the PR here :-)
> 
> I've run the slow tests locally and I'm getting some errors for the following tests:
> 
> ```shell
> FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_normal - AssertionE...
> FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_normal_batched - Ty...
> FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_robust_batched - Ty...
> ```
> 
> The rest all passes - could you try fixing those?
> 
> Also we should make sure that the documentation example works correctly. The doc tests seem to fail at the moment. Could you follow the explanations here: https://github.com/huggingface/transformers/tree/main/docs#testing-documentation-examples to test the example doc strings? :-)
> 
> Let me know if something is not clear!

Also let me know if you'd like me to take over the final steps of the PR if you're busy with other things, happy to help you out here!

Thanks for the final fixes! Could you maybe resolve the conflicting files and then we're ready for a final review :heart_eyes: ? :-)

@patrickvonplaten just checked that the docs are correct as well!

Thanks for your patience on this project I've been so preoccupied with other work

Hey @cwkeam,

I'll can do the final changes (and move the checkpoints) once the tests are green and you give me the green light :-) 

is there an specific reason why it's named "input_features" instead of "input_values" as in wav2vec2 and there is no pad function in the processor?

> is there an specific reason why it's named "input_features" instead of "input_values" as in wav2vec2 and there is no pad function in the processor?

Yeah good question! It's `input_features` because the input is a sequnce of vectors not a sequence of just float values

I guess there probably should be a pad function in the processor 

@lorenlugosch right I missed that one. Should be good now!

@patrickvonplaten re: removing `encoder_hidden_states` logic: to be clear, you want to remove the (unused) `encoder_hidden_states` but not `output_hidden_states`? I personally would prefer keeping the ability to access the intermediate layer outputs --- e.g. I used them for this thread https://twitter.com/lorenlugosch/status/1533457200819183617 , and they might be useful for people trying transfer learning stuff

Other than that, I've tested it out, and everything seems good to me (would love to add the LID logits but we can do that in another PR as you said)

> @patrickvonplaten re: removing `encoder_hidden_states` logic: to be clear, you want to remove the (unused) `encoder_hidden_states` but not `output_hidden_states`? I personally would prefer keeping the ability to access the intermediate layer outputs --- e.g. I used them for this thread https://twitter.com/lorenlugosch/status/1533457200819183617 , and they might be useful for people trying transfer learning stuff
> 
> Other than that, I've tested it out, and everything seems good to me (would love to add the LID logits but we can do that in another PR as you said)

Hey @lorenlugosch,

exactly the field `output_hidden_states` should be kept

Amazing job @cwkeam! Test failure is unrelated -> merging!