isin() received an invalid combination of arguments 
Author: pseudotensor
Pull request: no

### System Info

4.41.0
python 3.10


### Who can help?

@ArthurZucker @gante

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

This change: https://github.com/huggingface/transformers/commit/7130a22db9033e47b34a5e836b6014d531179f02

Here:

https://github.com/huggingface/transformers/blob/main/src/transformers/generation/utils.py#L486-L493

is leading to alot of different software to fail with the below error:

```
 isin() received an invalid combination of arguments - got (test_elements=int, elements=Tensor, ), but expected one of:
 * (Tensor elements, Tensor test_elements, *, bool assume_unique, bool invert, Tensor out)
 * (Number element, Tensor test_elements, *, bool assume_unique, bool invert, Tensor out)
 * (Tensor elements, Number test_element, *, bool assume_unique, bool invert, Tensor out)
```



A work-around patch is:
```
--- /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/generation/utils.py      2024-05-26 01:04:39.151177467 -0700
+++ new.py      2024-05-26 01:02:53.993095157 -0700
@@ -468,12 +468,14 @@
             raise ValueError(
                 "Can't infer missing attention mask on `mps` device. Please provide an `attention_mask` or use a different device."
             )
+        pad_token_tensor = torch.tensor([pad_token_id], device=inputs.device) if pad_token_id is not None else None
+        eos_token_tensor = torch.tensor([eos_token_id], device=inputs.device) if eos_token_id is not None else None
 
         is_pad_token_in_inputs = (pad_token_id is not None) and (
-            torch.isin(elements=inputs, test_elements=pad_token_id).any()
+            torch.isin(elements=inputs, test_elements=pad_token_tensor).any()
         )
         is_pad_token_not_equal_to_eos_token_id = (eos_token_id is None) or ~(
-            torch.isin(elements=eos_token_id, test_elements=pad_token_id).any()
+            torch.isin(elements=eos_token_tensor, test_elements=pad_token_tensor).any()
         )
         can_infer_attention_mask = is_pad_token_in_inputs * is_pad_token_not_equal_to_eos_token_id
         attention_mask_from_padding = inputs.ne(pad_token_id).long()
```

E.g. Coqui XTT (no longer maintained) fails like this without the above patch.

What is going on?

### Expected behavior

No failure.  I expect a Number (as it says is allowed) to be converted properly to a tensor on same device so no failure.

Comments:
@pseudotensor hi! Can you share a minimal reproducible code? 

If you're calling the `generate()`, all special tokens are converted to tensors [here](https://github.com/huggingface/transformers/blob/84c4b72ee99e8e65a8a5754a5f9d6265b45cf67e/src/transformers/generation/utils.py#L1608), before preparing attention mask

+1

@zucchini-nlp I have the same issue. This is effectively what I am doing:
```python
model_name = 'xtts-v2'
prompt = 'This is a test prompt'

config = XttsConfig()
# Load the configuration from the model's config.json file
config.load_json(path)
model = Xtts.init_from_config(config)

model.load_checkpoint(
    config, checkpoint_dir=checkpoint_dir, eval=True
)


gpt_cond_latent, speaker_embedding = model.get_conditioning_latents(
    audio_path=samples
)
gen = model.inference_stream(
    prompt,
    language=language,
    gpt_cond_latent=gpt_cond_latent,
    speaker_embedding=speaker_embedding,
)

for chunk in gen:
    print(chunk)
```

And I get the error:
![Screenshot 2024-05-28 at 12 09 13 AM](https://github.com/huggingface/transformers/assets/41707552/9a7fb72b-aab7-436a-8d5e-1cd8b58adca1)


I opened https://github.com/idiap/coqui-ai-TTS/issues/31 in our Coqui fork. This is due to the XTTS streaming code modifying `generate()` and calling internal methods that have been changed in #30624. PRs welcome to fix it on our side, I'm not very familiar with that code.

@eginhard Are you aware of an older version of `transformers` where this streaming works?

Anything below 4.41.0 should work. I've just released `coqui-tts` version 0.24.1 that limits `transformers` to lower versions to temporarily fix that until someone properly updates the streaming code. This issue can probably be closed because I don't think any action is needed here.

No problem, I patched coqui too to handle

https://github.com/h2oai/h2ogpt/blob/main/docs/xtt.patch

same issue

Can anyone share a minimal script to reproduce which would not depend much on external libraries? That would help us a lot to solve the issue if it still persists

But I believe this is issue from TTS library as calling the bare `generate()` on any model within transformers is expected to work. In that case it is recommended to open an issue/PR in corresponding libraries for them to either update their code or restrict transformers version to any other version that worked. I guess the external library is calling parts of `generate()` and not the whole pipeline

@binhoul Could you mention whether you're using the original repo or [our fork](https://github.com/idiap/coqui-ai-TTS)? The former is expected to be broken. The fork should currently work with `transformers>=4.43.0,<=4.46.2` and is restricted to those versions.

@zucchini-nlp Yes, the TTS library essentially reimplements `generate()` to allow streaming where not only the token, but also the embedding is returned, which is needed for the XTTS model. This is obviously very brittle because it relies on private functions that keep changing a lot. Not sure if there's a better way.

@eginhard i see, please lmk you have any ideas how we can help the TTS libraries to be more robust and use `generate()` with less changes. We are currently planning to modularize the generation utilities so that it can be used by more varieties of architectures. We have a tracker in https://github.com/huggingface/transformers/issues/30810

Feel free to go to the issue and comment any ideas/requests you might have :)

Thanks for the pointer! More flexible streaming outputs already seem to be part of the plan, so that should work for Coqui TTS.