Wav2Vec2ProcessorWithLM in pipeline issue
Author: anderleich
Pull request: no

Opening a new issue for better tracking purposes. This issue follows: https://github.com/huggingface/transformers/issues/16759

> Hey @gxbag,
> 
> Please make sure to provide a reproducible code snippet. I cannot run the above snippet because I don't have access to `"language_model/vocabulary.txt"`.
> 
> Regarding the issue, you should not pass a processor object as the model object. The model object should only be used for models of type `PreTrainedModel`. To pass the model with the processor you could do the following:
> 
> ```python
> from transformers import AutoProcessor
> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-large-960h-lv60-self")
> vocab_dict = processor.tokenizer.get_vocab()
> 
> from pyctcdecode import build_ctcdecoder
> unigrams_file = open("language_model/vocabulary.txt", "r")
> unigrams_list = unigrams_file.readlines()
> decoder = build_ctcdecoder(
>     labels=list(vocab_dict.keys()),
>     kenlm_model_path="language_model/5gram.bin",
>     unigrams=unigrams_list
> )
> 
> from transformers import Wav2Vec2ProcessorWithLM
> processor_with_lm = Wav2Vec2ProcessorWithLM(
>     feature_extractor=processor.feature_extractor,
>     tokenizer=processor.tokenizer,
>     decoder=decoder
> )
> 
> from transformers import pipeline
> pipe = pipeline("automatic-speech-recognition", model="facebook/wav2vec2-large-960h-lv60-self", tokenizer=processor_with_lm, feature_extractor=processor_with_lm.feature_extractor, decoder=processor_with_lm.decoder, device=0)
> ```
> 
> This should correctly initialize the pipeline.

Hi @patrickvonplaten ,

I've just tried your solution. However, it does not use the LM for decoding. `self.type` is always `"ctc"` as `feature_extractor._processor_class` is alway `None`. See here:

https://github.com/huggingface/transformers/blob/b487096b02307cd6e0f132b676cdcc7255fe8e74/src/transformers/pipelines/automatic_speech_recognition.py#L127

And this is my code:

``` python

model = Wav2Vec2ForCTC.from_pretrained("./results/checkpoint-11600").to("cuda")
tokenizer = Wav2Vec2CTCTokenizer.from_pretrained("./", unk_token="[UNK]", pad_token="[PAD]", word_delimiter_token="|")
feature_extractor = Wav2Vec2FeatureExtractor(feature_size=1, sampling_rate=16000, padding_value=0.0, do_normalize=True, return_attention_mask=True)
processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)

vocab_dict = processor.tokenizer.get_vocab()
sorted_vocab_dict = {k.lower(): v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}

from pyctcdecode import build_ctcdecoder
decoder = build_ctcdecoder(
	labels=list(sorted_vocab_dict.keys()),
	kenlm_model_path="lm.small_3gram_correct.arpa",
)

processor_with_lm = Wav2Vec2ProcessorWithLM(
	feature_extractor=processor.feature_extractor,
	tokenizer=processor.tokenizer,
	decoder=decoder
)

pipe = AutomaticSpeechRecognitionPipeline(
	model=model,
	tokenizer=processor_with_lm.tokenizer,
	feature_extractor=processor_with_lm.feature_extractor,
	decoder=processor_with_lm.decoder,
	device=0)
```

Any clues?

Comments:
Hi @anderleich,

I am encountering the same issue when I try to use the AutomaticSpeechRecognitionPipeline in combination with a Languague Model. Have you been able to find a solution? I've tracked the issue down to the same lines of code as you, I cannot get self.type to evaluate to "ctc_with_lm" with the models I am using, even though they work fine when I use them outside of the pipeline. 

Best wishes,
Judith

Hi @judithvdw ,

Not yet. I decided to use the LM outside the pipeline for the moment

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @patrickvonplaten ,

Any suggestions on this?

cc @sanchit-gandhi 

Hey @anderleich,

As a temporary fix, could you set the feature extractor's `_processor_class` attribute manually?

```python
model = Wav2Vec2ForCTC.from_pretrained("./results/checkpoint-11600").to("cuda")
tokenizer = Wav2Vec2CTCTokenizer.from_pretrained("./", unk_token="[UNK]", pad_token="[PAD]", word_delimiter_token="|")
feature_extractor = Wav2Vec2FeatureExtractor(feature_size=1, sampling_rate=16000, padding_value=0.0, do_normalize=True, return_attention_mask=True)
processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)

vocab_dict = processor.tokenizer.get_vocab()
sorted_vocab_dict = {k.lower(): v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}

from pyctcdecode import build_ctcdecoder
decoder = build_ctcdecoder(
	labels=list(sorted_vocab_dict.keys()),
	kenlm_model_path="lm.small_3gram_correct.arpa",
)

# set class manually
feature_extractor._set_processor_class("Wav2Vec2ProcessorWithLM")

processor_with_lm = Wav2Vec2ProcessorWithLM(
	feature_extractor= feature_extractor,
	tokenizer=tokenizer,
	decoder=decoder,
)

pipe = AutomaticSpeechRecognitionPipeline(
	model=model,
	tokenizer=processor_with_lm.tokenizer,
	feature_extractor=processor_with_lm.feature_extractor,
	decoder=processor_with_lm.decoder,
	device=0,
)
```
I'll take a deeper look into why the class is defaulting to None

Did you happen get a chance to look further into this? Not to push you, but just to make sure the bot doesn't close the issue again for a lack of activity.

I haven't had the chance sadly - keeping the bot from closing the issue! Maybe if you have the chance to look into this @hollance?

Sure, I'll have a look.


I can't reproduce this. I used the following code:

```python
from transformers import (
    AutomaticSpeechRecognitionPipeline,
    Wav2Vec2ForCTC, 
    Wav2Vec2CTCTokenizer, 
    Wav2Vec2FeatureExtractor, 
    Wav2Vec2Processor, 
    Wav2Vec2ProcessorWithLM
)

model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-100h")
tokenizer = Wav2Vec2CTCTokenizer.from_pretrained("facebook/wav2vec2-base-100h")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("facebook/wav2vec2-base-100h")
processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-100h")

# without LM
pipe = AutomaticSpeechRecognitionPipeline(
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor
)

print(pipe.type)  # "ctc"
print(processor.feature_extractor._processor_class)  # None

pipe("https://huggingface.co/spaces/Matthijs/speecht5-asr-demo/resolve/main/examples/hmm_i_dont_know.wav")

# {'text': "I DON'T KNOW I THINK MAY BE ITS EASY TO GET A NEW ONE WE CAN GO TO THE STORL LATER TO SEE IF THEY HAVE ANY IN STOCK"}

# note that STORL is spelled wrong


# with LM
processor_with_lm = Wav2Vec2ProcessorWithLM.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm")

pipe_with_lm = AutomaticSpeechRecognitionPipeline(
    model=model,
    tokenizer=processor_with_lm.tokenizer,
    feature_extractor=processor_with_lm.feature_extractor,
    decoder=processor_with_lm.decoder
)

print(pipe_with_lm.type)  # "ctc_with_lm"
print(processor_with_lm.feature_extractor._processor_class)  # Wav2Vec2ProcessorWithLM

pipe_with_lm("https://huggingface.co/spaces/Matthijs/speecht5-asr-demo/resolve/main/examples/hmm_i_dont_know.wav")

# {'text': "I DON'T KNOW I THINK MAY BE ITS EASY TO GET A NEW ONE WE CAN GO TO THE STORE LATER TO SEE IF THEY HAVE ANY IN STOCK"}

# and now STORE is spelled correctly
```

I verified that the decoder was indeed called on the `pipe_with_lm` pipeline.

There might still be an issue with your own models but I can't tell that without having access to those models.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Going to leave this one as is since the issue is not reproducible using a public checkpoint and we don't have access to a local model that demonstrates this behaviour, so we're unable to pinpoint where the bug potentially lies in transformers

The thread did result in two workarounds for this issue that you can try:
* Start from a pre-trained checkpoint like [facebook/wav2vec2-base-100h](hf.co/faceobook/wav2vec2-base-100h) that works as expected
* Use the 'hack' described in https://github.com/huggingface/transformers/issues/18825#issuecomment-1410416281

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.