MPS support broken for T5 models
Author: abdulfatir
Pull request: no

### System Info

- `transformers` version: 4.42.3
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.10.14
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.2 (False)



### Who can help?

@zucchini-nlp @ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The following breaks with an error on Apple Silicon Macs. It works on older versions (e.g., `transformers~=4.39.0`).

```py
import torch
from transformers import GenerationConfig
from transformers import T5ForConditionalGeneration

model = T5ForConditionalGeneration.from_pretrained(
    "google/t5-efficient-tiny", device_map="mps"
)
input_ids = torch.tensor([[4, 5, 6, 6, 7]], device="mps")
model.generate(
    input_ids=input_ids,
    generation_config=GenerationConfig(do_sample=True),
)
```

**Error**:

```
NotImplementedError                       Traceback (most recent call last)
Cell In[1], line 9
      5 model = T5ForConditionalGeneration.from_pretrained(
      6     \"google/t5-efficient-tiny\", device_map=\"mps\"
      7 )
      8 input_ids = torch.tensor([[4, 5, 6, 6, 7]], device=\"mps\")
----> 9 model.generate(
     10     input_ids=input_ids,
     11     generation_config=GenerationConfig(do_sample=True),
     12 )

File ~/miniconda3/envs/chronos-forecasting/lib/python3.10/site-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File ~/miniconda3/envs/chronos-forecasting/lib/python3.10/site-packages/transformers/generation/utils.py:1664, in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs)
   1661 batch_size = inputs_tensor.shape[0]
   1663 device = inputs_tensor.device
-> 1664 self._prepare_special_tokens(generation_config, kwargs_has_attention_mask, device=device)
   1666 # decoder-only models must use left-padding for batched generation.
   1667 if not self.config.is_encoder_decoder and not is_torchdynamo_compiling():
   1668     # If `input_ids` was given, check if the last id in any sequence is `pad_token_id`
   1669     # Note: If using, `inputs_embeds` this check does not work, because we want to be more hands-off.

File ~/miniconda3/envs/chronos-forecasting/lib/python3.10/site-packages/transformers/generation/utils.py:1513, in GenerationMixin._prepare_special_tokens(self, generation_config, kwargs_has_attention_mask, device)
   1510     logger.warning(f\"Setting `pad_token_id` to `eos_token_id`:{pad_token_id} for open-end generation.\")
   1512 # we can't infer attn mask if pad token is set to be eos token in model's generation config
-> 1513 if eos_token_id is not None and torch.isin(elements=eos_token_id, test_elements=pad_token_id).any():
   1514     if kwargs_has_attention_mask is not None and not kwargs_has_attention_mask:
   1515         logger.warning_once(
   1516             \"The attention mask is not set and cannot be inferred from input because pad token is same as eos token.\"
   1517             \"As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` \"
   1518             \"to obtain reliable results.\"
   1519         )

NotImplementedError: The operator 'aten::isin.Tensor_Tensor_out' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS."
```

### Expected behavior

Expected the script to work like in older versions. 

Comments:
Hi @abdulfatir, thanks for reporting! 

There's a PR open to fix this here: #31695 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing as #31695 is merged in

This config worked on my Mac M1:

```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch
import warnings
warnings.filterwarnings("ignore")

def get_device(device_name):
    if device_name == 'mps':
        return torch.device('mps') if torch.backends.mps.is_available() else torch.device('cpu')
    elif device_name == 'cuda':
        return torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
    else:
        return torch.device('cpu')

device = get_device('mps')

model_path = "google/flan-t5-small"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSeq2SeqLM.from_pretrained(
    model_path,
    torch_dtype=torch.float16)
model.to(device)

prompt = 'Q: What is the largest animal?\nA:'
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

generation_output = model.generate(
    input_ids=input_ids, max_new_tokens=32
)
print(tokenizer.decode(generation_output[0]))
```

The answer is:
`
<pad> a lion

`