How to return detected language using whisper with asr pipeline?
Author: arso1er
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@sanchit-gandhi, @Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hello,

I'm trying to use asr pipeline with whisper, in other to detect an audio language and transcribe it. I get the transcribed audio successfully, but I have not found a way to return the detected language too.
I search the GitHub issues, and it seems this was added by [#21427](https://github.com/huggingface/transformers/pull/21427), but I don't know how to return the detected language. Here is my code:
```
from transformers import pipeline
import torch

speech_file = "input.mp3"
device = "cuda:0" if torch.cuda.is_available() else "cpu"

whisper = pipeline("automatic-speech-recognition", max_new_tokens=448, model="openai/whisper-small", device=device)
whisper_result = whisper(speech_file)
print(whisper_result)
```

### Expected behavior

Be able to return detected language.

Comments:
Probably the easiest here is to use the `processor` + `model` API:
```python
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from datasets import load_dataset

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")

librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = librispeech_dummy[0]["audio"]

input_features = processor(sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt").input_features

pred_tokens = model.generate(input_features, max_new_tokens=448)
pred_text = processor.batch_decode(pred_tokens, skip_special_tokens=True)
pred_language = processor.batch_decode(pred_tokens[:, 1:2], skip_special_tokens=False)

print(pred_text)
print(pred_language)
```
**Print Output:**
```
[' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.']
['
']
```

The pipeline discards the 'special' task/language tokens from the predictions when merging chunks, so we loose this information.


OK. I will try that.
Thank you.

Hi, there is an easy way now to get the language detected by the pipeline. According to this [PR](https://github.com/huggingface/transformers/issues/21311) you can add the parameter `return_language=True` to the pipe and you can get the language of the text.

@matallanas Wow that's what I call timing; you posted this 4h before I was looking for that answer, on a 4 months old thread I came across first in my search results 😄 

Great addition! Is there a simple way to get the probability of the detected language as well? It is quite easy using the original model but I haven't found a way to make it work with the huggingface implementation.

The `pipeline` is designed to be a high-level wrapper that goes from audio inputs -> text outputs. Anytime we want something more granular than that, it's best to use the `model` + `processor` API:
```python
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from datasets import load_dataset
import torch

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")

librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = librispeech_dummy[0]["audio"]

input_features = processor(sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt").input_features

outputs = model.generate(
    input_features, output_scores=True, return_dict_in_generate=True, max_new_tokens=128
)

transition_scores = model.compute_transition_scores(
    outputs.sequences, outputs.scores, normalize_logits=True
)

pred_text = processor.batch_decode(outputs.sequences, skip_special_tokens=True)
pred_language = processor.batch_decode(outputs.sequences[:, 1:2], skip_special_tokens=False)
lang_prob = torch.exp(transition_scores[:, 0])

print(pred_text)
print(pred_language)
print(lang_prob)
```
**Print Output:**
```
[' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.']
['
']
tensor([1.])
```

> Hi, there is an easy way now to get the language detected by the pipeline. According to this [PR](https://github.com/huggingface/transformers/issues/21311) you can add the parameter `return_language=True` to the pipe and you can get the language of the text.

would you be kind to show the example code? I have tried following code abut it fails to return detected language type

`   
    device = "cuda:0" if torch.cuda.is_available() else "cpu"
    torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

    model_id = "openai/whisper-large-v3"

    model = AutoModelForSpeechSeq2Seq.from_pretrained(
        model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
    )
    model.to(device)

    processor = AutoProcessor.from_pretrained(model_id)

    pipe = pipeline(
        "automatic-speech-recognition",
        model=model,
        tokenizer=processor.tokenizer,
        feature_extractor=processor.feature_extractor,
        max_new_tokens=128,
        chunk_length_s=30,
        batch_size=16,
        return_timestamps=True,
        torch_dtype=torch_dtype,
        device=device,
        return_language=True
    )

    result = pipe(sample, return_timestamps="word", generate_kwargs={"task": "transcribe"}, return_language=True)`

Is there any way we can get no_speech_probability with pipeline? 

> > Hi, there is an easy way now to get the language detected by the pipeline. According to this [PR](https://github.com/huggingface/transformers/issues/21311) you can add the parameter `return_language=True` to the pipe and you can get the language of the text.
> 
> would you be kind to show the example code? I have tried following code abut it fails to return detected language type
> 
> ` device = "cuda:0" if torch.cuda.is_available() else "cpu" torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
> 
> ```
> model_id = "openai/whisper-large-v3"
> 
> model = AutoModelForSpeechSeq2Seq.from_pretrained(
>     model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
> )
> model.to(device)
> 
> processor = AutoProcessor.from_pretrained(model_id)
> 
> pipe = pipeline(
>     "automatic-speech-recognition",
>     model=model,
>     tokenizer=processor.tokenizer,
>     feature_extractor=processor.feature_extractor,
>     max_new_tokens=128,
>     chunk_length_s=30,
>     batch_size=16,
>     return_timestamps=True,
>     torch_dtype=torch_dtype,
>     device=device,
>     return_language=True
> )
> 
> result = pipe(sample, return_timestamps="word", generate_kwargs={"task": "transcribe"}, return_language=True)`
> ```

For this case you only need to put return language in the declaration of the pipe. Based on your code this should be like:
```python
model_id = "openai/whisper-large-v3"

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    chunk_length_s=30,
    batch_size=16,
    torch_dtype=torch_dtype,
    device=device,
    return_language=True
)

result = pipe(sample, return_timestamps="word", generate_kwargs={"task": "transcribe"})
```
This is how I declare it to obtain de language. I hope this help you.

hello, how to set lanuage? 

 self.model.generation_config.language = "
"  # for hindi
            self.model.generation_config.task = "transcribe" 

not work

> hello, how to set lanuage?
> 
> self.model.generation_config.language = "
" # for hindi self.model.generation_config.task = "transcribe"
> 
> not work

as additional arguments when defining processor

WhisperProcessor.from_pretrained("openai/whisper-large-v3", language="german", task="transcribe")


Hi, I think it's not the way.