Fix M-CTC-T chunking
Author: samwaterbury
Pull request: yes

## Summary

Chunking doesn't currently work for the M-CTC-T architecture, which this PR attempts to fix. The change is fairly minor but I am not an expert on how M-CTC-T works, so definitely open to feedback. In my usage, it works as intended.

Paging @patrickvonplaten since I think you originally implemented the chunking mechanism. :slightly_smiling_face:

I will add some PR comments explaining the changes.

## Testing

I added one test to cover the ASR pipeline with M-CTC-T. I also ran these tests:

```shell
RUN_SLOW=True RUN_PIPELINE_TESTS=True pytest \
    tests/models/mctct \
    tests/pipelines/test_pipelines_automatic_speech_recognition.py \
    tests/pipelines/test_pipelines_common.py
```

Some of these tests fail, but I was able to confirm they're also failing on the main branch. Here are the failures:

```
FAILED tests/pipelines/test_pipelines_common.py::CommonPipelineTest::test_iterator_data_tf - tensorflow.python.framework.errors_impl.InternalError: Exception encountered when calling layer...
FAILED tests/pipelines/test_pipelines_common.py::PipelineUtilsTest::test_load_default_pipelines_tf - tensorflow.python.framework.errors_impl.InternalError: Exception encountered when calli...
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_chunking_fast_with_lm - AssertionError: 'e
eh' != '
 <s'
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_large_model_pt_with_lm - AssertionError: 'ctc' != 'ctc_with_lm'
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_with_lm_fast - AssertionError: 'ctc' != 'ctc_with_lm'
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_with_local_lm_fast - AssertionError: 'ctc' != 'ctc_with_lm'
FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_normal - RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.76 GiB tot...
FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_normal_batched - RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.76...
FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_robust_batched - RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.76...
```

Some of these look to just be issues with my machine (memory errors).


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18949). All of your documentation changes will be reflected on that endpoint.

In addition to running tests, as a sanity check I ran the code snippet from [this blog post](https://huggingface.co/blog/asr-chunking):

```python
from transformers import pipeline

pipe = pipeline(model="facebook/wav2vec2-base-960h")
pipe("very_long_file.mp3", chunk_length_s=10)
```

and it works as expected. I also modified it for M-CTC-T and it still works:

```python
from transformers import AutomaticSpeechRecognitionPipeline, MCTCTForCTC, MCTCTProcessor

model = MCTCTForCTC.from_pretrained("speechbrain/m-ctc-t-large")
processor = MCTCTProcessor.from_pretrained("speechbrain/m-ctc-t-large")
pipe = AutomaticSpeechRecognitionPipeline(
    feature_extractor=processor.feature_extractor,
    tokenizer=processor.tokenizer,
    model=model,
    framework="pt",
)

print(pipe("very_long_file.mp3", chunk_length_s=10))
```

Thanks for the PR - this all looks good to me! cc @sanchit-gandhi and @Narsil for a quick second review :-) 

Thanks for the PR - that's a great addition! Generally this all looks more or less all good to me! Just a bit unsure about the changes in the pipeline, but ok for me since all tests pass. @Narsil what do you think? 

Also cc @sanchit-gandhi for info

Thanks for the PR @samwaterbury! LGTM - happy with the changes to computing the inputs : logits ratio :-) 

@Narsil if you have 10 minutes it'd be super nice to get your review here :-) 

Thanks all! 🙂 @Narsil any chance for your 👀

> FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_large_model_pt_with_lm - AssertionError: 'ctc' != 'ctc_with_lm'

This is because you don't have `kenlm` installed. Since the pipeline makes this a soft error instead of a strong one, you're seeing a different error. 

I think we could update the tests to skip the ones requiring kenlm if you don't have it installed (but `kenlm` is a tricky dependency iirc)

Hi @samwaterbury ,

Sorry for the long delay before review, I'm pretty far behind on some stuff.

The first and biggest issue I have, is that I am not sure that the approach is **sound**.
For the pipeline to work correctly with chunking/striding we **need** a very big property, which is that every data point in audio space corresponds to a single logits.

This enables to do this: https://huggingface.co/blog/asr-chunking

However, I fear that M-CTC models uses mel spectrogram, which means the spectrograms themselves are distributing single data points into multiple feature points in the sequence length. It probably depends on the parameters of the feature extractor, but it seems like something should be done.

Since there is overlap in the feature space, it become very hard to attribute logits (hence letters) to their origin, and to *stitch* back together the original string when running inference on 2 different blocks with striding.

The current PR is actually I fear quite wrong, and is only correct by accident, because the `inputs_to_logits_ratio` is set to 1 and the audio is rather small, all the input audio ends up being in the first chunk, and since striding is broken, all the first chunk is used, and afterwards none of the chunks are used.

In order to see what I'm talking about, run the example with `batch_size=2` and print the `stride` within `postprocess`. You will see the numbers don't add up.

I created another PR https://github.com/huggingface/transformers/pull/19338 to recreate what you have done here in hopefully a more correct version. Unfortunately, it seems the output is wrong, because the stiching cannot be done properly.

I might have made a mistake in my calculations though so take it with a grain of salt.
But calculating the ratio like I'm doing in the PR is too incorrect, and that's why we rely on `config.inputs_to_logits_ratio` instead for wav2vec2. (The differences should be very minor since it's only about the padding options of convolutions and such, but it can lead to subtle bugs on some splitting).

Hi @Narsil sorry for the delay and thanks for the long and detailed review and response. What you're saying makes sense and I appreciate the time you took to write it all out!

I'm going to close this PR since it looks like your PR is a better starting point for continued work. (I've also personally moved from M-CTC-T to Whisper since opening this PR 😄)