[Whisper] TypeError: '<=' not supported between instances of 'NoneType' and 'float'
Author: felipehertzer
Pull request: no
Labels: name: Core: Tokenization

name: bug

name: Audio

### System Info

- `transformers` version: 4.44.2
- Platform: macOS-15.0-arm64-arm-64bit
- Python version: 3.12.6
- Huggingface_hub version: 0.24.7
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.6.0.dev20240916 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No

### Who can help?

@kamilakesbi @ArthurZucker @itazap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi, I am attempting to transcribe several audio files; however, the process intermittently encounters an exception with some of the files. The transcription works successfully in approximately 90% of the cases, but certain files trigger this exception unexpectedly. I am attaching one of the audio files that generates this exception for your review. Thank you.

- I was able replicate it on a MacOS on CPU and Linux on CUDA.

1 Install Stable TS
`pip install stable-ts`


2 Run the code:
```python
import stable_whisper

model = stable_whisper.load_hf_whisper('medium')
result = model.transcribe(
    audio = 'radio_18596_1726554951_1726554981.mp3',
)
print(result.text)
```

Audio sample: https://filebin.net/hivqswoer298m65m

Than I receive the follow exception:
```
Traceback (most recent call last):
  File "/tests/test.py", line 4, in <module>
    result = model.transcribe(
             ^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/stable_whisper/whisper_word_level/hf_whisper.py", line 236, in transcribe
    return transcribe_any(
           ^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/stable_whisper/non_whisper.py", line 342, in transcribe_any
    result = inference_func(**inference_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/stable_whisper/whisper_word_level/hf_whisper.py", line 116, in _inner_transcribe
    output = self._pipe(audio, **pipe_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 284, in __call__
    return super().__call__(inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/transformers/pipelines/base.py", line 1255, in __call__
    return next(
           ^^^^^
  File "/.venv/lib/python3.12/site-packages/transformers/pipelines/pt_utils.py", line 125, in __next__
    processed = self.infer(item, **self.params)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 587, in postprocess
    text, optional = self.tokenizer._decode_asr(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/transformers/models/whisper/tokenization_whisper.py", line 835, in _decode_asr
    return _decode_asr(
           ^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/transformers/models/whisper/tokenization_whisper.py", line 1086, in _decode_asr
    resolved_tokens, resolved_token_timestamps = _find_longest_common_sequence(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/transformers/models/whisper/tokenization_whisper.py", line 1193, in _find_longest_common_sequence
    matches = sum(
              ^^^^
  File "/.venv/lib/python3.12/site-packages/transformers/models/whisper/tokenization_whisper.py", line 1198, in <genexpr>
    and left_token_timestamp_sequence[left_start + idx]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<=' not supported between instances of 'NoneType' and 'float'
```

### Expected behavior

To be able to transcibe the audio files without this exception.

Comments:
Thanks for raising, looks like the below indeed does happen:

https://github.com/huggingface/transformers/blob/db72894b48990860a9b8d1ebbfdd9a0342309b8f/src/transformers/models/whisper/tokenization_whisper.py#L1058-L1061

since this loops over tokens and the last index + 1 will be out of range:

https://github.com/huggingface/transformers/blob/db72894b48990860a9b8d1ebbfdd9a0342309b8f/src/transformers/models/whisper/tokenization_whisper.py#L971

cc @eustlb @ylacombe wdyt about how the last timestamp should be handled ?

Im experiencing this as well on stable whisper 
![image](https://github.com/user-attachments/assets/d17deffa-1861-4fa5-9cea-0371e96a6451)


Any news here? Will it be fixed anytime soon ? Or is there a version where this is not a problem?

Hey, working on a fix but having trouble consistently reproducing this (sometimes breaks sometimes not) 🤔 Are you experiencing the same? @aklacar1 @felipehertzer ? 

@itazap Yes I am, however it happens on one of my videos, but not on other. I have no idea why one works and other does not. The one that does not work is almost 20 minutes long.

NOTE: To make sure I am giving you correct info, I am rerunning it now. Just waiting for Docker build to finish.

This time I used:

torch==2.0.0
torchvision==0.15
torchaudio==2.0.1
transformers=4.45.1
stable-ts[hf]==2.17.4

Last time I believe I had 2.4.x version of Torch, this time its 2.0.0. However, I get same results

  File "/function/stable_whisper/whisper_word_level/hf_whisper.py", line 236, in transcribe
    return transcribe_any(
  File "/function/stable_whisper/non_whisper.py", line 342, in transcribe_any
    result = inference_func(**inference_kwargs)
  File "/function/stable_whisper/whisper_word_level/hf_whisper.py", line 116, in _inner_transcribe
    output = self._pipe(audio, **pipe_kwargs)
  File "/function/transformers/pipelines/automatic_speech_recognition.py", line 284, in __call__
    return super().__call__(inputs, **kwargs)
  File "/function/transformers/pipelines/base.py", line 1260, in __call__
    return next(
  File "/function/transformers/pipelines/pt_utils.py", line 125, in __next__
    processed = self.infer(item, **self.params)
  File "/function/transformers/pipelines/automatic_speech_recognition.py", line 598, in postprocess
    text, optional = self.tokenizer._decode_asr(
  File "/function/transformers/models/whisper/tokenization_whisper.py", line 835, in _decode_asr
    return _decode_asr(
  File "/function/transformers/models/whisper/tokenization_whisper.py", line 1034, in _decode_asr
    resolved_tokens, resolved_token_timestamps = _find_longest_common_sequence(
  File "/function/transformers/models/whisper/tokenization_whisper.py", line 1193, in _find_longest_common_sequence
    matches = sum(
  File "/function/transformers/models/whisper/tokenization_whisper.py", line 1198, in <genexpr>
    and left_token_timestamp_sequence[left_start + idx]
TypeError: '<=' not supported between instances of 'NoneType' and 'float'

Hi @itazap,

Thank you for looking into the issue. I am able to reproduce it using the following audio file and stable-ts code:

Audio: https://file.io/wi9gbaf1GMvt

does #33625 fix your issue ? 🤗 

@felipehertzer apologies for the delay but I believe the link has expired, can you please reshare the file? 🙏 

Hi @itazap I reuploaded the file https://drive.google.com/file/d/1oqqnVygU7-8pviRS1CmZ6R86wqv68Uf5/view?usp=sharing

Thanks.

@felipehertzer Thanks! Okay I am not super familiar with the whisper model but I think it has to do with stable-ts not adding a special token to end the text, but you can try this branch: https://github.com/huggingface/transformers/pull/33625 and it should address the fix for now! 😊 

Hi @itazap, Sorry for the delay, I just tested it and I can confirm that it have fixed the issue. Thank you.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Reopening until a solution is merged into main

@felipehertzer do you still have the audio snippet ? the link has expired 

Hey @eustlb here is the link updated. Thanks.

https://drive.google.com/file/d/1BNUV7K8XMYCRC-YE_6QJ4PpmktgTIuNS/view?usp=sharing

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Sorry for necrodumping but we are still running into this on 4.48.0 (and pulled the patch versions and looks like the same thing still applies.) In particular if there's even a heuristic for what types of this happens on it would be super helpful. Thanks in advance!

cc @eustlb - maybe we need to merge #33625?

still an issue. is this possibly due to the audio encoding type?


@eustlb  still it looks like a bug, not ability to handle some files, specially with long lengths, [tokenization_whisper.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/tokenization_whisper.py) 

with lines 1210 to 1221 
```
            if token_timestamp_sequences:
                # Get length of longest subsequence of tokens that match
                # and have timestamps that are in order
                matches = sum(
                    1
                    for idx, elem in enumerate(left)
                    if (
                        elem == right[idx]
                        and left_token_timestamp_sequence[left_start + idx]
                        <= token_timestamp_sequences[seq_idx + 1][right_start + idx]
                    )
                )
```
and needs conditional pass as` tokens is not None`.
But, it could be some other handling issue. The issue comes only at the word level time sequences. If the batch size is bigger, than [36093](https://github.com/huggingface/transformers/issues/36093) error comes (again with word level).  With normal transcription it performs so far fine. It looks some unstable transformers? or bugs?

@Tortoise17 @Noah-Jaffe @Jemoka hello all, I'm sorry that you're still facing this issue. I've tested the original audio clip in the issue description on transformers v4.49.0 with this PR branch https://github.com/huggingface/transformers/pull/33625 . On this branch, can you please confirm if you are still experiencing this issue on your end? and if so, can you please share the audio file? Thank you! 🤗 

@itazap This commit perfectly worked. So far there is no crash and it performed successfully. I will check some more tests. If there is stil any problem, I will write you here. But your solution [#33625](https://github.com/huggingface/transformers/pull/33625) is perfect. Thank you again. 

@itazap #33625 works well on my end; I'd love it if a fix could be pushed to pypi (even pypi-test) soon so we can pin to it for our package :)

Thanks again for all your help!

Hey @felipehertzer @Tortoise17 @Noah-Jaffe @Jemoka 

First of all, I am really sorry for the time it took to solve this issue. To provide a correct fix, I had to dive a bit into OpenAI's original codebase to understand what was going on, and unfortunately, higher priorities stacked up. 

Thanks for your patience 🤗 #36632 provides a fix!

Great to hear, thanks so much for all your help @eustlb 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I moved passed this error by downgrading my transformers version to `transformers==4.38.2`

Currently experiencing this issue with `transformers==4.51.3`

> Currently experiencing this issue with `transformers==4.51.3`

Please try this version `transformers==4.38.2`