How to use Wav2Vec2ProcessorWithLM in pipeline?
Author: gxbag
Pull request: no

I created a `Wav2Vec2ProcessorWithLM` as described in the blog post (https://huggingface.co/blog/wav2vec2-with-ngram).

How can I use it in the `pipeline`?

```python
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained("facebook/wav2vec2-large-960h-lv60-self")
vocab_dict = processor.tokenizer.get_vocab()

from pyctcdecode import build_ctcdecoder
unigrams_file = open("language_model/vocabulary.txt", "r")
unigrams_list = unigrams_file.readlines()
decoder = build_ctcdecoder(
    labels=list(vocab_dict.keys()),
    kenlm_model_path="language_model/5gram.bin",
    unigrams=unigrams_list
)

from transformers import Wav2Vec2ProcessorWithLM
processor_with_lm = Wav2Vec2ProcessorWithLM(
    feature_extractor=processor.feature_extractor,
    tokenizer=processor.tokenizer,
    decoder=decoder
)

from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model=processor_with_lm, device=0)
```
outputs
```
[AttributeError: 'Wav2Vec2ProcessorWithLM' object has no attribute 'config']()
```

Comments:
cc @patrickvonplaten 

Hey @gxbag,

Please make sure to provide a reproducible code snippet. I cannot run the above snippet because I don't have access to `"language_model/vocabulary.txt"`.

Regarding the issue, you should not pass a processor object as the model object. The model object should only be used for models of type `PreTrainedModel`. To pass the model with the processor you could do the following:

```py
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained("facebook/wav2vec2-large-960h-lv60-self")
vocab_dict = processor.tokenizer.get_vocab()

from pyctcdecode import build_ctcdecoder
unigrams_file = open("language_model/vocabulary.txt", "r")
unigrams_list = unigrams_file.readlines()
decoder = build_ctcdecoder(
    labels=list(vocab_dict.keys()),
    kenlm_model_path="language_model/5gram.bin",
    unigrams=unigrams_list
)

from transformers import Wav2Vec2ProcessorWithLM
processor_with_lm = Wav2Vec2ProcessorWithLM(
    feature_extractor=processor.feature_extractor,
    tokenizer=processor.tokenizer,
    decoder=decoder
)

from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model="facebook/wav2vec2-large-960h-lv60-self", tokenizer=processor_with_lm, feature_extractor=processor_with_lm.feature_extractor, decoder=processor_with_lm.decoder, device=0)
```

This should correctly initialize the pipeline.

Hi @patrickvonplaten, thank you very much for answering. Your above provided code does not seem to use the language model. Here is the minimal working example to reproduce the error:
```python
from transformers import Wav2Vec2ProcessorWithLM
processor_with_lm = Wav2Vec2ProcessorWithLM.from_pretrained("gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm")

from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model="gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm", tokenizer=processor_with_lm, feature_extractor=processor_with_lm.feature_extractor, decoder=processor_with_lm.decoder)
```

I believe I should be able to just
```python
from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model="gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm")
```
if I had set it up correctly, which I unfortunately have not.

I manually copied config.json from https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self/blob/main/config.json and added it to my repository as I believed this missing file could be the cause of the problem but I think there are possibly more problems. Could you help me out?

Hey @gxbag,

Yeah, your model here: https://huggingface.co/gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm/tree/main should indeed work out of the box.

Why doesn't it work? Can you show a codesnippet that shows how it doesn't work?

Hey @patrickvonplaten,

When I run this snippet:
```python
from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model="gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm")
```

The error output is exactly the following:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/media/home/run.ipynb Cell [1](vscode-notebook-cell://ssh-remote%2Bstudent2/media/home/run.ipynb#ch0000000vscode-remote?line=0)' in <cell line: 2>()
      1[ from transformers import pipeline
----> ]()[2](vscode-notebook-cell://ssh-remote%2Bstudent2/media/home/run.ipynb#ch0000000vscode-remote?line=1)[ pipe = pipeline("automatic-speech-recognition", model="gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm")

File ~/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py:549, in pipeline(task, model, config, tokenizer, feature_extractor, framework, revision, use_fast, use_auth_token, model_kwargs, pipeline_class, **kwargs)
    ]()[545](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=544)[ # Infer the framework from the model
    ]()[546](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=545)[ # Forced if framework already defined, inferred if it's None
    ]()[547](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=546)[ # Will load the correct model if possible
    ]()[548](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=547)[ model_classes = {"tf": targeted_task["tf"], "pt": targeted_task["pt"]}
--> ]()[549](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=548)[ framework, model = infer_framework_load_model(
    ]()[550](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=549)[     model,
    ]()[551](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=550)[     model_classes=model_classes,
    ]()[552](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=551)[     config=config,
    ]()[553](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=552)[     framework=framework,
    ]()[554](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=553)[     revision=revision,
    ]()[555](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=554)[     task=task,
    ]()[556](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=555)[     **model_kwargs,
    ]()[557](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=556)[ )
    ]()[559](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=558)[ model_config = model.config
    ]()[561](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py?line=560)[ load_tokenizer = type(model_config) in TOKENIZER_MAPPING or model_config.tokenizer_class is not None

File ~/mambaforge/lib/python3.9/site-packages/transformers/pipelines/base.py:255, in infer_framework_load_model(model, config, model_classes, task, framework, **model_kwargs)
    ]()[252](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/base.py?line=251)[             continue
    ]()[254](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/base.py?line=253)[     if isinstance(model, str):
--> ]()[255](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/base.py?line=254)[         raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
    ]()[257](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/base.py?line=256)[ framework = "tf" if model.__class__.__name__.startswith("TF") else "pt"
    ]()[258](file:///home/ubuntu/mambaforge/lib/python3.9/site-packages/transformers/pipelines/base.py?line=257)[ return framework, model

ValueError: Could not load model gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCTC'>, <class 'transformers.models.auto.modeling_auto.AutoModelForSpeechSeq2Seq'>, <class 'transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForCTC'>).]()
```

When I run the longer snippet:
```python
from transformers import Wav2Vec2ProcessorWithLM
processor_with_lm = Wav2Vec2ProcessorWithLM.from_pretrained("gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm")

from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model="gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm", tokenizer=processor_with_lm, feature_extractor=processor_with_lm.feature_extractor, decoder=processor_with_lm.decoder)
```
The output is exactly the same:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/media/home/run.ipynb Cell 2' in <cell line: 5>()
      2 processor_with_lm = Wav2Vec2ProcessorWithLM.from_pretrained("gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm")
      4 from transformers import pipeline
----> 5 pipe = pipeline("automatic-speech-recognition", model="gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm", tokenizer=processor_with_lm, feature_extractor=processor_with_lm.feature_extractor, decoder=processor_with_lm.decoder)

File ~/mambaforge/lib/python3.9/site-packages/transformers/pipelines/__init__.py:549, in pipeline(task, model, config, tokenizer, feature_extractor, framework, revision, use_fast, use_auth_token, model_kwargs, pipeline_class, **kwargs)
    545 # Infer the framework from the model
    546 # Forced if framework already defined, inferred if it's None
    547 # Will load the correct model if possible
    548 model_classes = {"tf": targeted_task["tf"], "pt": targeted_task["pt"]}
--> 549 framework, model = infer_framework_load_model(
    550     model,
    551     model_classes=model_classes,
    552     config=config,
    553     framework=framework,
    554     revision=revision,
    555     task=task,
    556     **model_kwargs,
    557 )
    559 model_config = model.config
    561 load_tokenizer = type(model_config) in TOKENIZER_MAPPING or model_config.tokenizer_class is not None

File ~/mambaforge/lib/python3.9/site-packages/transformers/pipelines/base.py:255, in infer_framework_load_model(model, config, model_classes, task, framework, **model_kwargs)
    252             continue
    254     if isinstance(model, str):
--> 255         raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
    257 framework = "tf" if model.__class__.__name__.startswith("TF") else "pt"
    258 return framework, model

ValueError: Could not load model gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCTC'>, <class 'transformers.models.auto.modeling_auto.AutoModelForSpeechSeq2Seq'>, <class 'transformers.models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForCTC'>).
```

Hey @gxbag,

Your repo does not have a model PyTorch file. Could you add the correct `pytorch_model.bin` to your folder here: https://huggingface.co/gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm/tree/main ? 

Hey @patrickvonplaten,

Thank you so much for the hint! With this almost everything is solved as the model with the above snippet can now produce a result and it correctly uses the language model.

Something still seems off: When I use a longer audio file and use the striding method (as per this blog post: https://huggingface.co/blog/asr-chunking) of the pipeline to process longer audio, the last bit of text output is cut off.

To reproduce:
```python
from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model="gxbag/wav2vec2-large-960h-lv60-self-with-wikipedia-lm")
output = pipe("/any/long/audio/file.wav", chunk_length_s=30, stride_length_s=(6, 3))
output
```

This will cut off the last 3 seconds (as specified in stride_length_s) of the audio file from generating output. I didn't see this behavior when I used regular models without an added language model.

What could be the cause here?

Ah yeah this was a bug in Transformes that we recently fixed I think :-) See https://github.com/huggingface/transformers/pull/16730 . Could you check whether everything works correctly on master? :-) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@patrickvonplaten Is it possible to use Wav2Vec2ProcessorWithLM during _training_ to use a LM at train time? Or, is there another way to do this with some other HF tool?

Sure this is possibleb @kaleko - you just need to adapt the training script a bit, but it should be pretty trivial :-) 

> Hey @gxbag,
> 
> Please make sure to provide a reproducible code snippet. I cannot run the above snippet because I don't have access to `"language_model/vocabulary.txt"`.
> 
> Regarding the issue, you should not pass a processor object as the model object. The model object should only be used for models of type `PreTrainedModel`. To pass the model with the processor you could do the following:
> 
> ```python
> from transformers import AutoProcessor
> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-large-960h-lv60-self")
> vocab_dict = processor.tokenizer.get_vocab()
> 
> from pyctcdecode import build_ctcdecoder
> unigrams_file = open("language_model/vocabulary.txt", "r")
> unigrams_list = unigrams_file.readlines()
> decoder = build_ctcdecoder(
>     labels=list(vocab_dict.keys()),
>     kenlm_model_path="language_model/5gram.bin",
>     unigrams=unigrams_list
> )
> 
> from transformers import Wav2Vec2ProcessorWithLM
> processor_with_lm = Wav2Vec2ProcessorWithLM(
>     feature_extractor=processor.feature_extractor,
>     tokenizer=processor.tokenizer,
>     decoder=decoder
> )
> 
> from transformers import pipeline
> pipe = pipeline("automatic-speech-recognition", model="facebook/wav2vec2-large-960h-lv60-self", tokenizer=processor_with_lm, feature_extractor=processor_with_lm.feature_extractor, decoder=processor_with_lm.decoder, device=0)
> ```
> 
> This should correctly initialize the pipeline.

Hi @patrickvonplaten ,

I've just tried your solution. However, it does not use the LM for decoding. `self.type` is always `"ctc"` as `feature_extractor._processor_class` is alway `None`. See here:

https://github.com/huggingface/transformers/blob/b487096b02307cd6e0f132b676cdcc7255fe8e74/src/transformers/pipelines/automatic_speech_recognition.py#L127

And this is my code:

``` python

model = Wav2Vec2ForCTC.from_pretrained("./results/checkpoint-11600").to("cuda")
tokenizer = Wav2Vec2CTCTokenizer.from_pretrained("./", unk_token="[UNK]", pad_token="[PAD]", word_delimiter_token="|")
feature_extractor = Wav2Vec2FeatureExtractor(feature_size=1, sampling_rate=16000, padding_value=0.0, do_normalize=True, return_attention_mask=True)
processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)

vocab_dict = processor.tokenizer.get_vocab()
sorted_vocab_dict = {k.lower(): v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}

from pyctcdecode import build_ctcdecoder
decoder = build_ctcdecoder(
	labels=list(sorted_vocab_dict.keys()),
	kenlm_model_path="lm.small_3gram_correct.arpa",
)

processor_with_lm = Wav2Vec2ProcessorWithLM(
	feature_extractor=processor.feature_extractor,
	tokenizer=processor.tokenizer,
	decoder=decoder
)

pipe = AutomaticSpeechRecognitionPipeline(
	model=model,
	tokenizer=processor_with_lm.tokenizer,
	feature_extractor=processor_with_lm.feature_extractor,
	decoder=processor_with_lm.decoder,
	device=0)
```

Any clues?

Hey @anderleich, sorry could you add a new issue for the problem? It's always a bit hard to keep track of already answered issues :sweat_smile: 

Done! ;)

Dear @patrickvonplaten 
Does vocabulary.txt in the above discussion refer to unigrams.txt in the language_model folder?
I didn't see in the [totorial](https://huggingface.co/blog/wav2vec2-with-ngram) there is a section to create vocabulary.txt.