Wav2vec code not working with kenlm n-gram
Author: harveenchadha
Pull request: no

I followed all the steps listed in https://discuss.huggingface.co/t/how-to-create-wav2vec2-with-language-model/12703/7

to create LM but seems like I am getting no output at all. I tried to replicate all the steps listed for Hindi and failed.

Also I tried to run the Spanish repo as pointed out by @patrickvonplaten but even after executing the [file](https://huggingface.co/patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm/blob/main/run_ngram_wav2vec2.py), I was not able to run the code.

![Screenshot from 2022-01-19 15-58-57](https://user-images.githubusercontent.com/30959215/150115123-0caac09d-d68a-4e11-aff3-d244f362ceac.png)

Can we please start a discussion on the potential issues with this because the Community week is approaching soon and hence language model decoding can be the game changer.

Comments:
Hey @harveenchadha, 

Could you try to instead follow the steps of the official blog post: https://huggingface.co/blog/wav2vec2-with-ngram . It's more up-to-date and contains more in-detail information.

Hi, 
I created a processor with LM using my own data in Urdu Language. 
When decoding, I am getting the following error.

ValueError: Input logits of size 412, but vocabulary is size 46


Do I also need to train the model with this processor?  

@ypirkani, 

It seems like the alphabet of the LM and the vocab of the model doesn't match. Could you upload all files (kenLM, alphabet.json, vocab.json, pt_model.bin, ...) to a HF Hub repo so that we can take a look together? :-)

https://huggingface.co/yapak1994/LM_Test_1

I Have uploaded some of the relevant filles om the above repository.

Hey @ypirkani, 

The file structure doesn't look correct. Could you please try to follow the guide here: https://huggingface.co/blog/wav2vec2-with-ngram#4-combine-an-n-gram-with-wav2vec2 to create a pyctcdecode beam search decoder with your `.arpa` kenLM and your model? 

BTW, we usually don't upload the model as a zipped file as this way it cannot be loaded with `transformers`. Hope this helps

Hi Patrick,
Thanks a lot for your reply. 
I am still facing the same issue and I created the decoded as per the guidelines on the link you provided.

Let me point out a few things which I am following.

I am fine-tuning on my own data of Urdu Language.
My training System is as follows: 


```python
tokenizer = Wav2Vec2CTCTokenizer("./vocab.json", unk_token="[UNK]", pad_token="[PAD]", word_delimiter_token="|", bos_token="
", eos_token="
", do_lower_case=False )
feature_extractor = Wav2Vec2FeatureExtractor(feature_size=1, sampling_rate=16000, padding_value=0.0, do_normalize=True, return_attention_mask=False)
processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)
processor.save_pretrained("Processor/wav2vec2-base-Urdu")

class DataCollatorCTCWithPadding:


    processor: Wav2Vec2Processor
    padding: Union[bool, str] = True
    max_length: Optional[int] = None
    max_length_labels: Optional[int] = None
    pad_to_multiple_of: Optional[int] = None
    pad_to_multiple_of_labels: Optional[int] = None

    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
        # split inputs and labels since they have to be of different lenghts and need
        # different padding methods
        input_features = [{"input_values": feature["input_values"]} for feature in features]
        label_features = [{"input_ids": feature["labels"]} for feature in features]

        batch = self.processor.pad(
            input_features,
            padding=self.padding,
            max_length=self.max_length,
            pad_to_multiple_of=self.pad_to_multiple_of,
            return_tensors="pt",
        )
        with self.processor.as_target_processor():
            labels_batch = self.processor.pad(
                label_features,
                padding=self.padding,
                max_length=self.max_length_labels,
                pad_to_multiple_of=self.pad_to_multiple_of_labels,
                return_tensors="pt",
            )

        # replace padding with -100 to ignore loss correctly
        labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)

        batch["labels"] = labels

        return batch
data_collator = DataCollatorCTCWithPadding(processor=processor, padding=True)

model = Wav2Vec2ForCTC.from_pretrained(
    "facebook/wav2vec2-base", 
    ctc_loss_reduction="mean", 
    pad_token_id=processor.tokenizer.pad_token_id,
    vocab_size=len(processor.tokenizer)
)

training_args = TrainingArguments(
  output_dir="model_Urdu/wav2vec2-xlsr-Urdu",
  group_by_length=True,
  per_device_train_batch_size=4,
  evaluation_strategy="steps",
  num_train_epochs=5,
  fp16=True,
  gradient_checkpointing=True,
  save_steps=500,
  eval_steps=500,
  logging_steps=500,
  learning_rate=1e-4,
  weight_decay=0.005,
  warmup_steps=1000,
  save_total_limit=12,
)

trainer = Trainer(
    model=model,
    data_collator=data_collator,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=data["train"],
    eval_dataset=data["test"],
    tokenizer=processor.feature_extractor,
)

trainer.train()

```
The training goes smoothly and I am getting a WER of 40%. 

To create a Language model using KenLM, I am using the same cleaned text which I used in the above process.
After creating the LM, I corrected it to include both eos and bos symbols.

Now in order to combine LM with my previous processor, I am doing the following:

```py

processor = Wav2Vec2Processor.from_pretrained("Processor/wav2vec2-base-Urdu")
vocab_dict = processor.tokenizer.get_vocab()
sorted_vocab_dict = {k.lower(): v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}
decoder = build_ctcdecoder(
    labels=list(sorted_vocab_dict.keys()),
    kenlm_model_path="Urdu.arpa",
)
processor_with_lm = Wav2Vec2ProcessorWithLM(
    feature_extractor=processor.feature_extractor,
    tokenizer=processor.tokenizer,
    decoder=decoder
)

`
```
Now vocab_dict of processor and processor_with_lm is the same i.e. has the same alphabets.

Now when I try to decode using Processor_with_lm it is giving me an error:
`ValueError: Input logits of size 412, but vocabulary is size 46`


The main, issue is that vocab.json on non LM processor has been derived from my the same text which I have used to create LM.

Hey @ypirkani,

Sadly, the model is still in zipped format on the Hub here: https://huggingface.co/yapak1994/LM_Test_1/tree/main . Could you please not zip any files and upload them as standalone files to the repo? E.g. the following command should work:

```py
from transformers import AutoModelForCTC, AutoProcessor

model = AutoModelForCTC.from_pretrained("yapak1994/LM_Test_1")
processor = AutoProcessor.from_pretrained("yapak1994/LM_Test_1")
```

but it currently doesn't

I am making my arpa file from the same text I am using for training the model and generating the processor.
can Be downloaded from here
https://huggingface.co/yapak1994/Urdu_ASR_wav2vec2_base_Model/blob/main/5gram_correct_1.arpa
for inference code is follows,
```

import soundfile as sf
import torch
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import os

from transformers import AutoProcessor, AutoModelForCTC

processor = AutoProcessor.from_pretrained("yapak1994/Urdu_ASR_wav2vec2_base_Model")

model = AutoModelForCTC.from_pretrained("yapak1994/Urdu_ASR_wav2vec2_base_Model")`

path = "urdu/"

#print(os.listdir(path))
for file in os.listdir(path):
# load audio
	audio_input, sample_rate = sf.read(path + file)

	# pad input values and return pt tensor
	input_values = processor(audio_input, sampling_rate=sample_rate, return_tensors="pt").input_values

	# INFERENCE

	# retrieve logits & take argmax
	logits = model(input_values).logits
	predicted_ids = torch.argmax(logits, dim=-1)

	# transcribe
	transcription = processor.decode(predicted_ids[0])
#	print(transcription)
	print(transcription + " (" + file + ")" )
```


This code is working correctly and giving me output.

Now 

When I use merge this model with my LM.arpa

```
from transformers import AutoProcessor, AutoModelForCTC

processor = AutoProcessor.from_pretrained("yapak1994/Urdu_ASR_wav2vec2_base_Model")

model = AutoModelForCTC.from_pretrained("yapak1994/Urdu_ASR_wav2vec2_base_Model")`


from transformers import Wav2Vec2ProcessorWithLM
decoder = build_ctcdecoder(
    labels=list(sorted_vocab_dict.keys()),
    kenlm_model_path="yapak1994/Urdu_ASR_wav2vec2_base_Model/5gram_correct_1.arpa ",
)

processor_lm = Wav2Vec2ProcessorWithLM(feature_extractor=feature_extractor, tokenizer=tokenizer , decoder=decoder)`

```
Now when I try to repeat the same code for previous inference, I get error.

`ValueError: Input logits of size 586, but vocabulary is size 44`

```
for file in os.listdir(path):
# load audio
	audio_input, sample_rate = sf.read(path + file)

	# pad input values and return pt tensor
	input_values = processor(audio_input, sampling_rate=sample_rate, return_tensors="pt").input_values

	# INFERENCE

	# retrieve logits & take argmax
	logits = model(input_values).logits
	predicted_ids = torch.argmax(logits, dim=-1)

	# transcribe
	transcription = processor_lm.decode(predicted_ids[0])
#	print(transcription)
	print(transcription + " (" + file + ")" )
```



Secondly, How to I push the Wav2Vec2ProcessorWithLM with Language_model to HF repo ? 

Hey @ypirkani, 

I've fixed the language model structure of your repo so that you can now correctly load it with `AutoProcessor`. Doing the following works as expected:

```python
import numpy as np
from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="yapak1994/Urdu_ASR_wav2vec2_base_Model")
asr(np.array(16_000 * [0.0]))
```





For the future, please read through this blog post to understand how to correctly integrate the LM with Wav2Vec2:
https://huggingface.co/blog/wav2vec2-with-ngram

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.