AttributeError in automatic_speech_recognition.py when return_segments and return_timestamps are both True
Author: jdalegonzalez
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.48.0
- Platform: macOS-15.2-arm64-arm-64bit
- Python version: 3.12.3
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.4.5
- Accelerate version: 1.1.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.3.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. With Whisper long-form automatic speech recognition, set return_segments=True and return_timestamps=True.
2. AttributeError: 'dict' object has no attribute 'dtype' in `automatic_speech_recognition.py:postprocess`

line 579 in postprocess `    if self.framework == "pt" and outputs[key].dtype in (torch.bfloat16, torch.float16):` is expecting outputs['tokens'] to be a tensor.  In the case return_segments is true, outputs['tokens'] is not a tensor, but rather a dict with keys ['sequences', 'segments'].  The block of code at line 526 is accounting for this case when return_timestamps = "word" but not accounting for it when return_timestamps = True

### Expected behavior

I would expect the code in around 526 to do some format changing similar to what it does when return_timestamps = "word".  For what it's worth, setting return_timestamps to "word" is not a usable work-around because that causes output_attentions to get set to True, which is incompatible with SPDA

Comments:
The error occurs because the code doesn't properly handle cases where return_segments=True and return_timestamps=True. Specifically:

outputs['tokens'] is expected to be a tensor.
When return_segments=True, outputs['tokens'] becomes a dictionary containing ['sequences', 'segments'], which causes the AttributeError.

**Fix:**
Modify the block of code around line 526 in automatic_speech_recognition.py to handle the return_timestamps=True case similarly to how return_timestamps="word" is processed. This involves adapting the format of outputs['tokens'] to ensure compatibility.

**Test:**
Create unit tests for both return_timestamps=True and return_timestamps="word" cases to ensure the new implementation works as expected.
Verify the fix on both supported and edge cases.

Can I be assigned to this? I starting to work on it now

Hi @sh950, we don't assign issues! If you open a PR and link this issue, though, feel free to ping us f

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.