All versions of Transformers after 4.52.4 break batch inference on MPS when padding_side = left with SmolVLM
Author: Matteo-Omenetti
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.55.2
- Platform: macOS-15.6-arm64-arm-64bit-Mach-O
- Python version: 3.13.3
- Huggingface_hub version: 0.34.4
- Safetensors version: 0.6.2
- Accelerate version: 1.9.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No


### Information
All versions of Transformers after 4.52.4 break batch inference on MPS when padding_side = left. 

When padding_side = "left" and inputs of difference length are given such that padding is introduced, then the output for the shortest padded input is token 0 repeated for ever.

This seems to be happening only on MPS, setting device = CPU solves the issue.


### Reproduction
Minimal reproducer:
```python
from transformers import AutoModelForImageTextToText, AutoProcessor
from transformers.image_utils import load_image

DEVICE = "mps"  # if device == "cpu", it works
model_name = "HuggingFaceTB/SmolVLM-256M-Instruct"
processor = AutoProcessor.from_pretrained(model_name)
processor.tokenizer.padding_side = "left" # if set to right, you get "wrong" output but not infinite loop of 0 token
model = AutoModelForImageTextToText.from_pretrained(model_name, device_map=DEVICE)
model.eval()


image_urls = [
    "https://upload.wikimedia.org/wikipedia/commons/7/76/GazettedeFrance.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/1/15/Cat_August_2010-4.jpg",
]
images = [load_image(u) for u in image_urls]


def get_prompt(query):
    messages = [
        {
            "role": "user",
            "content": [{"type": "image"}, {"type": "text", "text": query}],
        },
    ]
    prompt = processor.apply_chat_template(messages, add_generation_prompt=True)

    return prompt


prompts = [get_prompt("OCR this image"), get_prompt("Describe in details the content of this picture.")]


inputs = processor(
    text=prompts,
    images=images,
    return_tensors="pt",
    padding=True
)
inputs = inputs.to(DEVICE)

gen_kwargs = dict(
    max_new_tokens=1000 - inputs.input_ids.shape[1],
    use_cache=True,
    do_sample=False,
)

generated_ids = model.generate(**inputs, **gen_kwargs)
outputs = processor.batch_decode(
    generated_ids[:, inputs.input_ids.shape[1] :], skip_special_tokens=False
)

for i, output in enumerate(outputs):
    print(f"=========== Output {i} ===========")
    print(output)
    print("\n\n")
```



Output:

> =========== Output 0 ===========





























































































































> =========== Output 1 ===========
>  The image depicts a cat lying on a white surface, likely a ledge or a wall. The cat is positioned in a relaxed manner, with its front paws slightly outstretched and its hind legs bent. The cat has a striped pattern on its fur, which is a common feature in many cat breeds. The stripes are horizontal and run from the tip of its tail to the base of its tail, creating a pattern that is both eye-catching and visually appealing.
> 
> The cat's fur is light brown or black, and its eyes are a deep amber color, which is a common feature in

### Expected behavior

The model should produce the same output as on CPU.

Comments:
cc @zucchini-nlp

Unfortunately I can't test it yet if the issue is reproducible on MPS only. Leaving it here in case anyone want to take it 

This seems to be a PyTorch issue, specifically with the their MPS implementation of SDPA. When you left pad, the masked tokens introduce fully masked rows in the attention matrix (i.e. all False) which leads to `-inf` attention scores that MPS softmax computes as NaN, which then gets propagated throughout the model leading to NaN logits. Right padding works as those masked tokens can still attend to unmasked tokens to their left, though this is probably a different computation. You can also bypass MPS SDPA by setting `model.set_attn_implementation("eager")` at the cost of efficiency. Believe this was recently addressed in [pytorch/pytorch#157727](https://github.com/pytorch/pytorch/pull/157727) to align with cpu/cuda implementations but not released as of PyTorch 2.8.0

Ah didn't know it was still an issue for MPS, since that was a problem we had with older version of torch and had to manually workaround by "unmasking" attention [here](https://github.com/huggingface/transformers/blob/75d6f17de68f372284ecb5b40db6f83007ffa394/src/transformers/masking_utils.py#L463-L467)

If it was fixed on torch side, I believe we don't need to add another workaround for mps devices

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.