Whisper is unable to transcribe in spoken language
Author: markakash
Pull request: no

### System Info

Currently, when I use any other Whisper models except for large-v3, instead of it auto detecting the language and transcribing it, it translates it and then transcribes it. I looked but couldn't find any flag/config to remove this behaviour. If I use the large-v3, then it detects the language perfectly and transcribes it in the detected language. I am using the latest version of transformer and huggingface-hub. I am also not sure, if this is a known issue, in any case would be grateful if anyone could point me to a fix for it.

### Who can help?

@sac

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

model_id = "openai/whisper-tiny"

    model = AutoModelForSpeechSeq2Seq.from_pretrained(
        model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
    )
    model = model.to_bettertransformer()
    model.to(device)
    processor = AutoProcessor.from_pretrained(model_id)

    pipe = pipeline(
        "automatic-speech-recognition",
        model=model,
        tokenizer=processor.tokenizer,
        feature_extractor=processor.feature_extractor,
        max_new_tokens=128,
        chunk_length_s=30,
        batch_size=16,
        return_timestamps=True,
        torch_dtype=torch_dtype,
        device=device,
    )
    
    This is my pipeline configuration

### Expected behavior

Expected behaviour should be, Whisper detecting the spoken language and then transcribing it in the detected language, instead of it translating and then transcribing.

Expected:
Get audio data -> detect language -> transcribe

Comments:
It's important to note that the behavior you described might be influenced by various factors, including the quality of the audio input and the specific model being used. While it's not accurate to label anyone as "stupid," as you rightly pointed out, there are ways to address this concern.

Firstly, when using any Whisper model other than large-v3, it seems to translate the content before transcribing it, rather than automatically detecting the language and transcribing in the detected language.

To address this, you can try enforcing the language or transcription via flags in the generate method. Here's an example of how you might modify your code:

![image](https://github.com/huggingface/transformers/assets/48595927/237e8c96-2dce-4d88-9670-d1ff91fbc404)
In the above code snippet (which we should definitely add to the doc somewhere cc @sanchit-gandhi  I'll open a PR), 

This is not an issue with the code but with the documentation lacking this and your understanding 🤗 


There's some detailed documentation on the audio transformers course on setting the language/task parameters: https://huggingface.co/learn/audio-course/chapter5/asr_models#graduation-to-seq2seq

We'll definitely add this to the model cards + docs in our next efforts on Whisper generation (cc @Vaibhavs10)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.