fix prompt strip to support tensors and np arrays
Author: AvivSham
Pull request: yes
Labels: name: Audio

# What does this PR do?

`WhisperTokenizer` does not strip the prompt ids when calling `decode` for `torch.Tensor`, `tf.Tensor`, and `np.ndarray`. It does not perform the slicing in `_strip_prompt` because the condition `isinstance(token_ids, list)` is not met.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sanchit-gandhi
@ArthurZucker

Comments:
I was not able to make it pass the CI/CD tests due to import issues. @sanchit-gandhi can you please guide?

Thank you for reviewing @sanchit-gandhi !
I was not aware of the framework agnostic constraint, I think we are ok now, can you please review it again?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This PR is not stale, waiting for @sanchit-gandhi to approve.

Gentle ping @sanchit-gandhi 

@amyeroberts @sanchit-gandhi 
Any news about this PR?

@amyeroberts This issue is not stale, still waiting for @sanchit-gandhi.

@sanchit-gandhi @ylacombe Can one of you review this?


Hi @sanchit-gandhi,
Added the requested changes (including adding the test), can you review again (hopefully for the last time 😄)?
BTW I also pulled transformers main branch so we will be aligned.

@sanchit-gandhi 
Done all the changes required including reverting and running the style. Please check again.

Hi @sanchit-gandhi ,
This test fails since `token_ids` is an empty list. IMO this should be handled elsewhere and much earlier maybe in [this line](https://github.com/huggingface/transformers/pull/27818/files#diff-b258211cb184b5aebf1a50b0f4a3f63683823996d367b68ac5afc504353d5c21R644), I cannot this about a case that user will try to decode empty list. Maybe `_preprocess_token_ids` would be a better place to add it.
WDYT?  
if you agree I can add this to the current PR:
```python
if not isinstance(token_ids, int):
    if len(token_ids) == 0:
        raise ValueError("No token ids provided for decoding.")
```
or if you see a reason for decoding an empty string we can return `token_ids` instead of raising the exception.

Hey @AvivSham - sorry I don't fully follow! If we look at the test [test_skip_special_tokens_with_timestamps](https://github.com/huggingface/transformers/blob/1e79eade41c346d1d8c2ad79c9e32e3186f20b54/tests/models/whisper/test_tokenization_whisper.py#L209), we see that we always pass a non-empty array of token-ids to the tokeniser's `.decode` method. Running this test on `main`, we see that we always get the expected decoding results. For this PR, we need to make sure the changes we implement do not change the current decoding behaviour, which means that the test should pass. I would recommend using the debugger to check the behaviour of `_preprocess_token_ids` and `_strip_prompt` before and after the changes, and try to unify this PR to how it is originally done. If you get stuck, let me know and I can take a deeper look!

Ok, so I debugged it and the empty `token_ids` comes from the method `_decode_with_timestamps` check line `553` under `tokenization_whisper.py` file.
I made a small adjustment to address this issue, and all the tests passed successfully.
@sanchit-gandhi please have another look at it 😄 

@amyeroberts @sanchit-gandhi 

Hi @AvivSham, you need to run `make fix-copies` to resolve the failing quality checks. 

cc @kamilakesbi For first review as @sanchit-gandhi is off 

@amyeroberts I ran it and committed. Let's see if the CI/CD pass this time.


@amyeroberts @kamilakesbi 
Gentle reminder 😃 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27818). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @amyeroberts,
This is a mistake we can resolve this comment. See [these lines](https://github.com/huggingface/transformers/pull/27818/files#diff-2fb1a089a6a7c4bbbab1eba43e5bfe7fdd29415e24fc62c1ea8601ea14f9ad48R606-R613).

@amyeroberts I keep receiving emails regarding CI/CD failed runs how can we prevent that? I thought it would stop once the PR is merged.
![image](https://github.com/user-attachments/assets/c7ef052e-9846-49e2-ae1f-0681083ea6ae)


Hi @AvivSham, this is because of the settings on your fork. 

You can disable the CI runs by going to the repo, then Settings > Actions > Disable actions

<img width="691" alt="image" src="https://github.com/user-attachments/assets/e93a7240-670f-47df-9b27-255f1898d616">


@amyeroberts Thank you very much for helping.