Inconsistencies in Evaluation: Manual Inference vs. Huggingface Pipeline
Author: monk1337
Pull request: no

Hi, 

I noticed a significant difference between the Word Error Rate (WER) scores when evaluating with manual inference and when using a custom pipeline. I would appreciate any guidance on whether I might be making a mistake in my approach.

1. Manual Inference:

```
model.eval()
for step, batch in enumerate(tqdm(eval_dataloader)):
    with torch.cuda.amp.autocast():
        with torch.no_grad():
            generated_tokens = (
                model.generate(
                    input_features=batch["input_features"].to("cuda"),
                    forced_decoder_ids=forced_decoder_ids,
                    max_new_tokens=255,
                )
                .cpu()
                .numpy()
            )
            labels = batch["labels"].cpu().numpy()
            labels = np.where(labels != -100, labels, processor.tokenizer.pad_token_id)
            decoded_preds = processor.tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
            decoded_labels = processor.tokenizer.batch_decode(labels, skip_special_tokens=True)
            predictions.extend(decoded_preds)
            references.extend(decoded_labels)
            normalized_predictions.extend([normalizer(pred).strip() for pred in decoded_preds])
            normalized_references.extend([normalizer(label).strip() for label in decoded_labels])
        del generated_tokens, labels, batch
    gc.collect()
wer = 100 * metric.compute(predictions=predictions, references=references)
normalized_wer = 100 * metric.compute(predictions=normalized_predictions, references=normalized_references)
eval_metrics = {"eval/wer": wer, "eval/normalized_wer": normalized_wer}

print(f"{wer=} and {normalized_wer=}")
print(eval_metrics)
```

The score is 

`{'eval/wer': 167.3913043478261, 'eval/normalized_wer': 75.28089887640449}`


2. Pipeline Evaluation:


```
class TranscriptionPipeline:
    def __init__(self, model_name):
        self.pipe = self._inference_pipeline(model_name)

    def _inference_pipeline(self, model_name):
        return pipeline(model=model_name)

    def transcribe(self, audio_path):
        result = self.pipe(audio_path)
        return result

    def wer_score(self, audio_path, ground_truth):
        result = self.transcribe(audio_path)["text"]
        wer = 100 * WER_score.compute(predictions=[result], references=[ground_truth])
        return {"wer": wer}

    def wer_score_bulk(self, audio_paths, ground_truths):
        results = [result_.get("text", "") for result_ in self.transcribe(audio_paths)]
        wer = 100 * WER_score.compute(predictions=results, references=ground_truths)
        return {"wer": wer}

    def evaluate_dataset(self, dataset):
        
        audio_paths = []
        sentences = []
    
        # Populate the lists
        for item in dataset:
            audio_paths.append(item['audio']['path'])
            sentences.append(item['sentence'])
    
        # Get WER score using the bulk method
        result = self.wer_score_bulk(audio_paths, sentences)
        return result


ty = TranscriptionPipeline("aaditya/Whisper_a948aa64-4c75-4246-ab98-0a48a9f2cd14")
ty.evaluate_dataset(common_voice_["test"])
```

While Pipeline evaluation score is

`{'wer': 106.5217391304348}`

Concern:
The discrepancy between the two WER scores is quite large. Could someone help identify if I've made an error in my approach or clarify why such a difference might occur?




### Who can help?

pipelines: @Narsil @Vaibhavs10 @sanchit-gandhi

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

### Expected behavior

WER should be same.




Comments:
The issue has been solved. I had to use `AutomaticSpeechRecognitionPipeline`

Thanks for sharing the solution!