[whisper] added dropping of attention weights after DTW calculations related to word timestamps if these weights are not requested in the output
Author: jacekc3
Pull request: yes

# What does this PR do?
We observed significantly increased memory usage when using word-level timestamps  (`return_timestamps='word'`), which increased with each segment processed - especially when processing longer recordings.
It turned out that for DTW calculations, cross-attention weights are extracted for each segment, and then they are accumulated in intermediate segments. However, when the user doesn't need the model output - they are not used any more - just sequences are returned. This accumulation of weights unnecessarily increases memory consumption.

It's simple whisper patch for when attention weights are not explicitly requested (but need to be extracted to be used in DTW for timestamps), so weights are not accumulated across segment objects, minimizing memory usage.

Probably related to #27834


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
Thanks for your PR! cc @ylacombe :)

Hello! Any updates on this?

Can we merge it? It's critical for obtaining word-level timestamps in longer recordings.

gentle ping @ylacombe!

Most specifically, it'd be great to know if you're facing this issue while using the pipeline or while using the model directly, and with which hyper-parameters

I tried both options (using the pipeline or not) and I didn't see any difference in VRAM usage with or without your PR.

<details><summary>Code snippet</summary>
<p>

```py
from transformers import AutoTokenizer, set_seed, AutoModelForCausalLM
import soundfile as sf
import torch
import time


from transformers import WhisperFeatureExtractor, WhisperForConditionalGeneration, WhisperTokenizerFast, WhisperProcessor, pipeline
from datasets import load_dataset

model_id = "openai/whisper-large-v3-turbo"

torch_dtype = torch.float16
device = "cuda:1"

model = WhisperForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch_dtype,
).to(device)

processor = WhisperProcessor.from_pretrained(model_id)
tokenizer = WhisperTokenizerFast.from_pretrained(model_id)
feature_extractor = WhisperFeatureExtractor.from_pretrained(model_id)

dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
sample = dataset[0]["audio"]

# Use the model and processor to transcribe the audio:
input_features = processor(
    sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt"
).input_features.to(device).to(torch_dtype)


NB_LOOPS = 5

set_seed(0)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=tokenizer,
    feature_extractor=feature_extractor,
    torch_dtype=torch_dtype,
    device=device,
)

def measure_latency_and_memory_use(model, nb_loops=NB_LOOPS, chunk_length_s=0.0, batch_size=None):
    # define Events that measure start and end of the generate pass
    start_event = torch.cuda.Event(enable_timing=True)
    end_event = torch.cuda.Event(enable_timing=True)

    # reset cuda memory stats and empty cache
    torch.cuda.reset_peak_memory_stats(device)
    torch.cuda.empty_cache()
    torch.cuda.synchronize()

    # get the start time
    start_event.record()

    # actually generate
    for _ in range(nb_loops):
        # set seed for reproducibility
        set_seed(0)
        outputs = pipe(
            inputs={"raw":sample["array"], "sampling_rate":sample["sampling_rate"]},
            chunk_length_s=chunk_length_s,
            batch_size=batch_size,
            generate_kwargs={
                "task": "transcribe",
                "language": "en",
            },
            return_timestamps="word",  # Enable word-level timestamps
        )
    # get the end time
    end_event.record()
    torch.cuda.synchronize()

    # measure memory footprint and elapsed time
    max_memory = torch.cuda.max_memory_allocated(device)
    elapsed_time = start_event.elapsed_time(end_event) * 1.0e-3

    print("Execution time:", elapsed_time / nb_loops, "seconds")
    print("Max memory footprint", max_memory * 1e-9, " GB")

    return outputs

def measure_latency_and_memory_use_generate(model, nb_loops=NB_LOOPS):
    # define Events that measure start and end of the generate pass
    start_event = torch.cuda.Event(enable_timing=True)
    end_event = torch.cuda.Event(enable_timing=True)

    # reset cuda memory stats and empty cache
    torch.cuda.reset_peak_memory_stats(device)
    torch.cuda.empty_cache()
    torch.cuda.synchronize()

    # get the start time
    start_event.record()

    # actually generate
    for _ in range(nb_loops):
        # set seed for reproducibility
        set_seed(0)
        outputs = model.generate(input_features, task="transcribe", language="en", return_token_timestamps=True, return_timestamps=True,
                       return_segments=True)
    # get the end time
    end_event.record()
    torch.cuda.synchronize()

    # measure memory footprint and elapsed time
    max_memory = torch.cuda.max_memory_allocated(device)
    elapsed_time = start_event.elapsed_time(end_event) * 1.0e-3

    print("Execution time:", elapsed_time / nb_loops, "seconds")
    print("Max memory footprint", max_memory * 1e-9, " GB")

    return outputs


print("--------------------BS = 0 -------------")
print("USING PÏPELINE CHUNKS")

outputs = measure_latency_and_memory_use(model, chunk_length_s=30)
print("---")

print("USING PÏPELINE WHISPER LONGFORM")

outputs = measure_latency_and_memory_use(model, chunk_length_s=0)
print("---")



print("USING MODEL DIRECTLY")


outputs = measure_latency_and_memory_use_generate(model)
print("---")


print("--------------------BS = 24 -------------")


print("USING PÏPELINE CHUNKS")

outputs = measure_latency_and_memory_use(model, chunk_length_s=30, batch_size=24)
print("---")

print("USING PÏPELINE WHISPER LONGFORM")

outputs = measure_latency_and_memory_use(model, chunk_length_s=0 , batch_size=24)
print("---")


```

</p>
</details> 

<details><summary>Results (more or less the same with and without your PR)</summary>
<p>

--------------------BS = 0 -------------
USING PÏPELINE CHUNKS
Execution time: 6.43879140625 seconds
Max memory footprint 4.697205248  GB

---
USING PÏPELINE WHISPER LONGFORM
Execution time: 5.48822421875 seconds
Max memory footprint 4.691129856  GB

---
USING MODEL DIRECTLY
Execution time: 2.241147265625 seconds
Max memory footprint 4.6902394880000005  GB

---
--------------------BS = 24 -------------
USING PÏPELINE CHUNKS
Execution time: 5.332154296875 seconds
Max memory footprint 10.696115712000001  GB

---
USING PÏPELINE WHISPER LONGFORM
Execution time: 5.643804687499999 seconds
Max memory footprint 4.691129856  GB


</p>
</details> 

I do see a time speed-up using the model directly though.

hey!

thanks all, for all of your responses and @ylacombe for detailed analysis. unfortunately I think that you're missing the essence of the problem, let me explain it in more detail;
 - first of all - it only occurs with transformers's chunking disabled - for our application, intentionally disabling this chunking significantly improved the transcription quality (WER dropped in some cases by 15-20%)
 - it's all about RAM usage, not VRAM - mentioned attention weights are cumulated in RAM (by copying to cpu - `generation_whsiper.py:967:972`)
 - that attention weights are dropped from memory once the generate() method is done (generate from `generation_whisper.py:277`) - so processing multiple short audio files won't expose this issue
 - you've taken very short piece of audio - if I looked correctly, the first sample from 'distil-whisper/librispeech_long' is only a little over a minute long


from my perspective it looks like these attention weights (which take up so much RAM) are intentionally put aside, so that word-level timestamps can be calculated from them, but they are redundantly accumulated in segments (`current_segments[prev_i] += segments - generation_whisper.py:714`) (redunantly because they are no longer needed - don't need them in the result), and they should only be returned if the user explicitly selects them in `output_attention=True` - while the request to return word-level timestamps sets output_attentions=True (`generation_whisper.py:1182`).

so we have a case where a request for word-level timestamps sets a request to return attention weights to the user, where in fact these weights are only needed at the segment level for DTW and completely unnecessarily accumulated in RAM later.

and this case is fixed by this PR - when the user did not explicitly request attention weights at the output - hence the copy of the value what was orginally passed to `output_attention` ​​in `org_gen_config_output_attention` - and despite this they are needed, they are simply deleted after their use


below is a script that demonstrates the problem, hope this clarifies things a bit;

first of all - generate a 1 hour mono silent audio (content ofc doesn't matter);
`ffmpeg -f lavfi -i anullsrc=r=44100:cl=mono -t 360 silent_audio.wav`

```python
import os
import time
from datetime import datetime

import psutil
import torch
import torchaudio
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline, WhisperTokenizer

# ==================================
def print_memory_usage():
    pid = os.getpid()
    process = psutil.Process(pid)

    try:
        while True:
            memory_usage = process.memory_info().rss / 1024 ** 2  # Memory in MB
            timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
            print(f"{timestamp} - Memory usage: {memory_usage:.2f} MB")
            time.sleep(1)
    except KeyboardInterrupt:
        print("Memory monitoring stopped.")

from threading import Thread

monitor_thread = Thread(target=print_memory_usage)
monitor_thread.daemon = True
monitor_thread.start()

# ===================================

device = "cuda:0"
chunk_length_s = 0
stride_length_s = 0

torch_dtype = torch.float16
model_id = "openai/whisper-large-v3"
batch_size = 2
processor = AutoProcessor.from_pretrained(model_id)
processor.tokenizer = WhisperTokenizer.from_pretrained(model_id)

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    torch_dtype=torch_dtype,
    low_cpu_mem_usage=True,
    use_safetensors=True,
    attn_implementation="sdpa"
)

asr_pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    chunk_length_s=chunk_length_s,
    stride_length_s=stride_length_s,
    batch_size=batch_size,
    torch_dtype=torch_dtype,
    device=device,
)

long_waveform, sr = torchaudio.load('silent_audio.wav')
transcription = asr_pipe(long_waveform.squeeze(0).numpy(), return_timestamps="word")
print(transcription)

monitor_thread.join()
```


results for `chunk_length_s = 0` -> OOM:
```
2024-12-06 11:21:33 - Memory usage: 517.21 MB
2024-12-06 11:21:34 - Memory usage: 551.32 MB
2024-12-06 11:21:35 - Memory usage: 570.48 MB
Device set to use cuda:0
2024-12-06 11:21:36 - Memory usage: 2325.56 MB
/home/jacek/virtualenvs/whisper-memory/lib/python3.12/site-packages/transformers/models/whisper/generation_whisper.py:512: FutureWarning: The input name `inputs` is deprecated. Please make sure to use `input_features` instead.
  warnings.warn(
Due to a bug fix in https://github.com/huggingface/transformers/pull/28687 transcription using a multilingual Whisper will default to language detection followed by transcription instead of translation to English.This might be a breaking change for your use case. If you want to instead always translate your audio to English, make sure to pass `language='en'`.
Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.43.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.
WhisperModel is using WhisperSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `layer_head_mask` not None. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.
2024-12-06 11:21:37 - Memory usage: 1401.03 MB
From v4.47 onwards, when a model cache is to be returned, `generate` will return a `Cache` instance instead by default (as opposed to the legacy tuple of tuples format). If you want to keep returning the legacy format, please set `return_legacy_cache=True`.
2024-12-06 11:21:38 - Memory usage: 4404.74 MB
2024-12-06 11:21:39 - Memory usage: 7163.54 MB
2024-12-06 11:21:40 - Memory usage: 10623.50 MB
2024-12-06 11:21:41 - Memory usage: 13052.23 MB
2024-12-06 11:21:42 - Memory usage: 15276.43 MB
2024-12-06 11:21:43 - Memory usage: 17121.79 MB
2024-12-06 11:21:44 - Memory usage: 18584.11 MB
2024-12-06 11:21:45 - Memory usage: 20160.75 MB
2024-12-06 11:21:46 - Memory usage: 20567.50 MB
2024-12-06 11:21:47 - Memory usage: 20972.07 MB
2024-12-06 11:21:48 - Memory usage: 21369.73 MB
2024-12-06 11:21:49 - Memory usage: 21633.43 MB
2024-12-06 11:21:50 - Memory usage: 21478.69 MB
2024-12-06 11:21:51 - Memory usage: 21484.09 MB
2024-12-06 11:21:52 - Memory usage: 23784.99 MB
2024-12-06 11:21:53 - Memory usage: 25467.99 MB
2024-12-06 11:21:54 - Memory usage: 25819.21 MB
2024-12-06 11:21:55 - Memory usage: 25838.82 MB
[...OOM...]
```



and as expected, for `chunk_length_s=30` nothing happens;
```
2024-12-06 11:22:46 - Memory usage: 505.65 MB
2024-12-06 11:22:47 - Memory usage: 538.84 MB
2024-12-06 11:22:48 - Memory usage: 559.10 MB
Device set to use cuda:0
2024-12-06 11:22:49 - Memory usage: 2571.82 MB
/home/jacek/virtualenvs/whisper-memory/lib/python3.12/site-packages/transformers/models/whisper/generation_whisper.py:512: FutureWarning: The input name `inputs` is deprecated. Please make sure to use `input_features` instead.
  warnings.warn(
Due to a bug fix in https://github.com/huggingface/transformers/pull/28687 transcription using a multilingual Whisper will default to language detection followed by transcription instead of translation to English.This might be a breaking change for your use case. If you want to instead always translate your audio to English, make sure to pass `language='en'`.
2024-12-06 11:22:50 - Memory usage: 1177.64 MB
Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.43.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
WhisperModel is using WhisperSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `layer_head_mask` not None. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.
From v4.47 onwards, when a model cache is to be returned, `generate` will return a `Cache` instance instead by default (as opposed to the legacy tuple of tuples format). If you want to keep returning the legacy format, please set `return_legacy_cache=True`.
2024-12-06 11:22:51 - Memory usage: 3399.20 MB
2024-12-06 11:22:52 - Memory usage: 2089.69 MB
2024-12-06 11:22:53 - Memory usage: 4586.82 MB
2024-12-06 11:22:54 - Memory usage: 2137.42 MB
2024-12-06 11:22:55 - Memory usage: 5610.87 MB
2024-12-06 11:22:56 - Memory usage: 2116.91 MB
2024-12-06 11:22:57 - Memory usage: 5842.59 MB
2024-12-06 11:22:58 - Memory usage: 2178.12 MB
2024-12-06 11:22:59 - Memory usage: 7128.38 MB
2024-12-06 11:23:00 - Memory usage: 3479.24 MB
2024-12-06 11:23:01 - Memory usage: 2138.89 MB
2024-12-06 11:23:02 - Memory usage: 4664.70 MB
2024-12-06 11:23:03 - Memory usage: 2137.42 MB
2024-12-06 11:23:04 - Memory usage: 5723.35 MB
2024-12-06 11:23:05 - Memory usage: 2089.82 MB
2024-12-06 11:23:06 - Memory usage: 6976.41 MB
2024-12-06 11:23:07 - Memory usage: 3315.44 MB
2024-12-06 11:23:08 - Memory usage: 2137.42 MB
2024-12-06 11:23:09 - Memory usage: 4577.36 MB
2024-12-06 11:23:10 - Memory usage: 2093.48 MB
2024-12-06 11:23:11 - Memory usage: 5654.59 MB
2024-12-06 11:23:12 - Memory usage: 2104.46 MB
2024-12-06 11:23:13 - Memory usage: 6949.06 MB
2024-12-06 11:23:14 - Memory usage: 3281.61 MB
2024-12-06 11:23:15 - Memory usage: 2111.79 MB
2024-12-06 11:23:16 - Memory usage: 4573.09 MB
2024-12-06 11:23:17 - Memory usage: 2111.79 MB
2024-12-06 11:23:18 - Memory usage: 5577.07 MB
2024-12-06 11:23:19 - Memory usage: 2115.45 MB
2024-12-06 11:23:20 - Memory usage: 6562.89 MB
Whisper did not predict an ending timestamp, which can happen if audio is cut off in the middle of a word. Also make sure WhisperTimeStampLogitsProcessor was used during generation.
{'text': ' you you you you you you you you you you you you you you you you you you you you you you you you you you you you you you you you you you', 'chunks': [{'text': ' you', 'timestamp': (29.98, 29.98)}, {'text': ' you', 'timestamp': (59.98, 59.98)}, {'text': ' you', 'timestamp': (89.98, 89.98)}, {'text': ' you', 'timestamp': (119.98, 119.98)}, {'text': ' you', 'timestamp': (149.98, 149.98)}, {'text': ' you', 'timestamp': (179.98, 179.98)}, {'text': ' you', 'timestamp': (209.98, 209.98)}, {'text': ' you', 'timestamp': (239.98, 239.98)}, {'text': ' you', 'timestamp': (269.98, 269.98)}, {'text': ' you', 'timestamp': (299.98, 299.98)}, {'text': ' you', 'timestamp': (329.98, 329.98)}, {'text': ' you', 'timestamp': (359.98, 359.98)}, {'text': ' you', 'timestamp': (389.98, 389.98)}, {'text': ' you', 'timestamp': (419.98, 419.98)}, {'text': ' you', 'timestamp': (449.98, 449.98)}, {'text': ' you', 'timestamp': (479.98, 479.98)}, {'text': ' you', 'timestamp': (509.98, 509.98)}, {'text': ' you', 'timestamp': (539.98, 539.98)}, {'text': ' you', 'timestamp': (569.98, 569.98)}, {'text': ' you', 'timestamp': (599.98, 599.98)}, {'text': ' you', 'timestamp': (629.98, 629.98)}, {'text': ' you', 'timestamp': (659.98, 659.98)}, {'text': ' you', 'timestamp': (689.98, 689.98)}, {'text': ' you', 'timestamp': (719.98, 719.98)}, {'text': ' you', 'timestamp': (749.98, 749.98)}, {'text': ' you', 'timestamp': (779.98, 779.98)}, {'text': ' you', 'timestamp': (809.98, 809.98)}, {'text': ' you', 'timestamp': (839.98, 839.98)}, {'text': ' you', 'timestamp': (869.98, 869.98)}, {'text': ' you', 'timestamp': (899.98, 899.98)}, {'text': ' you', 'timestamp': (929.98, 929.98)}, {'text': ' you', 'timestamp': (959.98, 959.98)}, {'text': ' you', 'timestamp': (989.98, 989.98)}, {'text': ' you', 'timestamp': (990.0, 990.2)}]}

```


with this PR applied + `chunk_length_s=0`:
```
2024-12-06 11:58:17 - Memory usage: 509.18 MB
2024-12-06 11:58:18 - Memory usage: 543.35 MB
2024-12-06 11:58:19 - Memory usage: 561.73 MB
Device set to use cuda:0
2024-12-06 11:58:20 - Memory usage: 2139.02 MB
/home/jacek/virtualenvs/whisper-memory/lib/python3.12/site-packages/transformers/models/whisper/generation_whisper.py:512: FutureWarning: The input name `inputs` is deprecated. Please make sure to use `input_features` instead.
  warnings.warn(
Due to a bug fix in https://github.com/huggingface/transformers/pull/28687 transcription using a multilingual Whisper will default to language detection followed by transcription instead of translation to English.This might be a breaking change for your use case. If you want to instead always translate your audio to English, make sure to pass `language='en'`.
Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.43.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.
WhisperModel is using WhisperSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `layer_head_mask` not None. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.
2024-12-06 11:58:21 - Memory usage: 1383.41 MB
From v4.47 onwards, when a model cache is to be returned, `generate` will return a `Cache` instance instead by default (as opposed to the legacy tuple of tuples format). If you want to keep returning the legacy format, please set `return_legacy_cache=True`.
2024-12-06 11:58:22 - Memory usage: 1625.78 MB
2024-12-06 11:58:23 - Memory usage: 1626.64 MB
2024-12-06 11:58:24 - Memory usage: 2533.76 MB
2024-12-06 11:58:25 - Memory usage: 3430.64 MB
2024-12-06 11:58:26 - Memory usage: 4339.90 MB
2024-12-06 11:58:27 - Memory usage: 1630.66 MB
2024-12-06 11:58:28 - Memory usage: 1938.96 MB
2024-12-06 11:58:29 - Memory usage: 2806.08 MB
2024-12-06 11:58:30 - Memory usage: 3665.20 MB
2024-12-06 11:58:31 - Memory usage: 1634.67 MB
2024-12-06 11:58:32 - Memory usage: 1635.47 MB
2024-12-06 11:58:33 - Memory usage: 2193.27 MB
2024-12-06 11:58:34 - Memory usage: 3102.22 MB
2024-12-06 11:58:35 - Memory usage: 3973.34 MB
2024-12-06 11:58:36 - Memory usage: 1639.48 MB
2024-12-06 11:58:37 - Memory usage: 1640.29 MB
2024-12-06 11:58:38 - Memory usage: 2459.57 MB
2024-12-06 11:58:39 - Memory usage: 3364.53 MB
2024-12-06 11:58:40 - Memory usage: 4263.71 MB
2024-12-06 11:58:41 - Memory usage: 1644.35 MB
2024-12-06 11:58:42 - Memory usage: 1830.81 MB
2024-12-06 11:58:43 - Memory usage: 2735.92 MB
2024-12-06 11:58:44 - Memory usage: 3624.87 MB
2024-12-06 11:58:45 - Memory usage: 1648.31 MB
2024-12-06 11:58:46 - Memory usage: 1649.11 MB
2024-12-06 11:58:47 - Memory usage: 2119.06 MB
2024-12-06 11:58:48 - Memory usage: 3023.55 MB
2024-12-06 11:58:49 - Memory usage: 3671.62 MB
{'text': ' you you you you you you you you you you you you you you you you you you you you you you you you you you you you you you you you you Thank you.', 'chunks': [{'text': ' you', 'timestamp': (29.98, 29.98)}, {'text': ' you', 'timestamp': (59.98, 59.98)}, {'text': ' you', 'timestamp': (89.98, 89.98)}, {'text': ' you', 'timestamp': (119.98, 119.98)}, {'text': ' you', 'timestamp': (149.98, 149.98)}, {'text': ' you', 'timestamp': (179.98, 179.98)}, {'text': ' you', 'timestamp': (209.98, 209.98)}, {'text': ' you', 'timestamp': (239.98, 239.98)}, {'text': ' you', 'timestamp': (269.98, 269.98)}, {'text': ' you', 'timestamp': (299.98, 299.98)}, {'text': ' you', 'timestamp': (329.98, 329.98)}, {'text': ' you', 'timestamp': (359.98, 359.98)}, {'text': ' you', 'timestamp': (389.98, 389.98)}, {'text': ' you', 'timestamp': (419.98, 419.98)}, {'text': ' you', 'timestamp': (449.98, 449.98)}, {'text': ' you', 'timestamp': (479.98, 479.98)}, {'text': ' you', 'timestamp': (509.98, 509.98)}, {'text': ' you', 'timestamp': (539.98, 539.98)}, {'text': ' you', 'timestamp': (569.98, 569.98)}, {'text': ' you', 'timestamp': (599.98, 599.98)}, {'text': ' you', 'timestamp': (629.98, 629.98)}, {'text': ' you', 'timestamp': (659.98, 659.98)}, {'text': ' you', 'timestamp': (689.98, 689.98)}, {'text': ' you', 'timestamp': (719.98, 719.98)}, {'text': ' you', 'timestamp': (749.98, 749.98)}, {'text': ' you', 'timestamp': (779.98, 779.98)}, {'text': ' you', 'timestamp': (809.98, 809.98)}, {'text': ' you', 'timestamp': (839.98, 839.98)}, {'text': ' you', 'timestamp': (869.98, 869.98)}, {'text': ' you', 'timestamp': (899.98, 899.98)}, {'text': ' you', 'timestamp': (929.98, 929.98)}, {'text': ' you', 'timestamp': (959.98, 959.98)}, {'text': ' you', 'timestamp': (989.98, 989.98)}, {'text': ' Thank', 'timestamp': (1019.52, 1019.52)}, {'text': ' you.', 'timestamp': (1019.52, 1019.6)}]}
```

Hey @jacekc3, many thanks for this reproducer and for the explanation. I was indeed missing the point of your PR.

I was able to reproduce and confirm that what you're proposing solves your issue.
That said, I wasn’t able to reproduce this using the model itself on a standalone basis:
<details><summary>Reproducer</summary>
<p>

```python
import os
import time
from datetime import datetime

import psutil
import torch
import torchaudio
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, WhisperTokenizer

# ==================================
def print_memory_usage():
    pid = os.getpid()
    process = psutil.Process(pid)

    try:
        while True:
            memory_usage = process.memory_info().rss / 1024 ** 2  # Memory in MB
            timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
            print(f"{timestamp} - Memory usage: {memory_usage:.2f} MB")
            time.sleep(1)
    except KeyboardInterrupt:
        print("Memory monitoring stopped.")

from threading import Thread

monitor_thread = Thread(target=print_memory_usage)
monitor_thread.daemon = True
monitor_thread.start()

# ===================================

device = "cuda:1"
chunk_length_s = 0
stride_length_s = 0

torch_dtype = torch.float16
model_id = "openai/whisper-large-v3"
batch_size = 1
processor = AutoProcessor.from_pretrained(model_id)
processor.tokenizer = WhisperTokenizer.from_pretrained(model_id)

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    torch_dtype=torch_dtype,
    low_cpu_mem_usage=True,
    use_safetensors=True,
    attn_implementation="sdpa"
)


long_waveform, sr = torchaudio.load('silent_audio.wav')
long_waveform = torchaudio.functional.resample(long_waveform, sr, processor.feature_extractor.sampling_rate)
input_features = processor(
    long_waveform[0], return_tensors="pt"
).input_features.to(device).to(torch_dtype)
model = model.to(device)
transcription = model.generate(input_features, task="transcribe", language="en", return_token_timestamps=True, return_timestamps=True,
                       return_segments=True)

print(processor.batch_decode(transcription["sequences"]))


monitor_thread.join()
``` 

</p>
</details> 

This lead me to believe it might has to do with the way pipelines are implemented. 


I don't see an issue merging your PR, but the solution you've proposed is a hack, and I wonder if there are cleaner solutions: cc @Rocketknight1 and @Narsil since it seems related to the pipeline code. cc @eustlb as well for visibility.