Bug in v4.49 where the attention mask is ignored during generation (t5-small)
Author: bdhammel
Pull request: no
Labels: name: bug

### System Info

Hi all!

First, thank you very much for your hard work and making these features avalible.

I'm seeing a bug after updating to v4.49 where the output changes even though the attention mask should be masking padded values. Below is a script to reproduce the error.

It will tokenize two prompts, and then call `.generate` on the shorter prompt while trying different slices of the padded `input_ids` and padded `attention_mask`. At some point, the generated response will change for v4.49 but not v4.48.


Enviroment information
```
- `transformers` version: 4.49.0
- Platform: macOS-15.3-arm64-arm-64bit
- Python version: 3.10.13
- Huggingface_hub version: 0.29.0
- Safetensors version: 0.5.2
- Accelerate version: not installed
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
```

output of `uv pip compile requirements.in`
```
transformers==4.48.0         # change this to 4.49.0 to reproduce the error

asttokens==3.0.0
certifi==2025.1.31
charset-normalizer==3.4.1
decorator==5.1.1
exceptiongroup==1.2.2
executing==2.2.0
filelock==3.17.0
fsspec==2025.2.0
huggingface-hub==0.29.0
idna==3.10
ipython==8.32.0
jedi==0.19.2
jinja2==3.1.5
markupsafe==3.0.2
matplotlib-inline==0.1.7
mpmath==1.3.0
networkx==3.4.2
numpy==2.2.3
packaging==24.2
parso==0.8.4
pexpect==4.9.0
prompt-toolkit==3.0.50
ptyprocess==0.7.0
pure-eval==0.2.3
pygments==2.19.1
pyyaml==6.0.2
regex==2024.11.6
requests==2.32.3
safetensors==0.5.2
sentencepiece==0.2.0
stack-data==0.6.3
sympy==1.13.1
tokenizers==0.21.0
torch==2.6.0
tqdm==4.67.1
traitlets==5.14.3
typing-extensions==4.12.2
urllib3==2.3.0
wcwidth==0.2.13
```

### Who can help?

@ArthurZucker 

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig


model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
tokenizer = AutoTokenizer.from_pretrained("t5-small")

cfg = GenerationConfig(
    max_new_tokens=512,
    do_sample=False,
    use_cache=True,    # same behavior with use_cache=False
)


shortprompt = ("summarize: Transformers v4.49 appears to have a bug where .generate stops respecting "
               "the attention_mask after some number of tokens.")
longprompt = ("summarize: I enjoy walking with my cute dog, especially in the early mornings "
              "when the air is crisp and the streets are quiet. Watching my dog happily trot along, "
              "always brings a smile to my face.")

# ---
print("# Single prompt ---")
inputs = tokenizer(
    [shortprompt], return_tensors="pt", padding=True
)

outputs = model.generate(**inputs, generation_config=cfg)

expected = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]
print(f"short prompt: '{expected}'")
print()


# ---
print("# Double prompt ---")
inputs = tokenizer(
    [shortprompt, longprompt], return_tensors="pt", padding=True
)

outputs = model.generate(**inputs, generation_config=cfg)

text = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(f"short prompt: '{text[0]}'")
print(f"long prompt: '{text[1]}'")
print()

# ---
print("# Single shortprompt with mask ---")
def run_sliced_input(slice_, show_text=False):
    shortprompt_tokens = inputs.input_ids[0:1, slice_]
    shortprompt_mask = inputs.attention_mask[0:1, slice_]

    outputs = model.generate(inputs=shortprompt_tokens, attention_mask=shortprompt_mask, generation_config=cfg)
    text = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]
    if show_text:
        print(f"'{text}'")
    return text != expected

# run a bisect search to find the first slice that fails
import bisect
start = inputs.attention_mask[0].sum().item()
full_range = inputs.attention_mask.size(1)
ends = range(start, full_range)
print(f"searching in range {start} to {full_range}")

first_failure = start + bisect.bisect_left(
    [slice(None, end) for end in ends], True, key=run_sliced_input
)
if first_failure == full_range:
    print("No failure found in the full range!")
else:
    print(f"First failing slice: {first_failure}")

    print(f"Output with slice at {first_failure-1}: ", end="")
    run_sliced_input(slice(None, first_failure-1), show_text=True)
    print(f"Output with slice at {first_failure}: ", end="")
    run_sliced_input(slice(None, first_failure), show_text=True)

```

### Expected behavior

version 4.48
```
# Single prompt ---
short prompt: 'v4.49 appears to have a bug where.generate stops respecting the attention_mask after some tokens.'

# Double prompt ---
short prompt: 'v4.49 appears to have a bug where.generate stops respecting the attention_mask after some tokens.'
long prompt: 'i enjoy walking with my cute dog, especially in the early mornings. watching my dog happily trot along brings a smile to my face.'

# Single shortprompt with mask ---
searching in range 36 to 46
No failure found in the full range!
```

version 4.49
```
# Single prompt ---
short prompt: 'v4.49 appears to have a bug where.generate stops respecting the attention_mask after some tokens.'

# Double prompt ---
short prompt: ''
long prompt: 'i enjoy walking with my cute dog, especially in the early mornings. watching my dog happily trot along brings a smile to my face.'

# Single shortprompt with mask ---
searching in range 36 to 46
First failing slice: 39
Output with slice at 38: 'v4.49 appears to have a bug where.generate stops respecting the attention_mask after some tokens.'
Output with slice at 39: 'Transformers v4.49 appears to have a bug where.generate stops respecting the attention_mask after some tokens.'
```

Comments:
Trying a little bit more debugging. It looks like the encoder outputs are the same. If I get some time tomorrow I'll try to isolate the cause in the decoder

```
encoder_out_no_mask = model.encoder(input_ids=inputs.input_ids[0:1,:start], attention_mask=inputs.attention_mask[0:1,:start])
encoder_out_masked = model.encoder(input_ids=inputs.input_ids[0:1], attention_mask=inputs.attention_mask[0:1])

last_token_idx = encoder_out_no_mask.last_hidden_state.size(1)

import numpy as np
np.testing.assert_array_equal(
    encoder_out_no_mask.last_hidden_state.detach(), encoder_out_masked.last_hidden_state.detach()[:,:last_token_idx,:]
) # passes
```

Hey @bdhammel !

Should have been fixed by https://github.com/huggingface/transformers/commit/77aa9fc0767439fab61108d960cec8ec37c6723e. Encoder mask was being ignored on models. Can you try to install from `main` branch and check if it works?

Confirmed it's fixed on `main`. Thanks guys!