transformers showing decoder model architecture detected so padding should be left
Author: sleepingcat4
Pull request: no
Labels: name: bug

### System Info

I had implemented some batching both using pipeline and model.generate method and for Qwen3 and Llama4 models it shows decoder only model detected so padding should mentioned in left. Which is weird as I inspected the model output and it was totally fine. 

However when I mentioned padding to be left, Qwen3 class returned an error. it is very weird. Can this waring/bug be fixed?

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

To reproduce please use my below code:

```
def load_pipeline(model_name: str):
    pipe = pipeline(
        "text-generation",
        model=model_name,
        model_kwargs={"torch_dtype": torch.bfloat16, "attn_implementation": "flash_attention_2"},
        device_map="auto"
    )
    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
    return pipe, tokenizer

system_prompt = (
    "Think step by step to answer the following question.\n"
    "Return the answer for the MC (Multiple Choice) question at the end of the response after a separator #####.\n"
)

def zero_eval(df, pipe, tokenizer, agieval=False, batch_size=4):
    full_prompts = []

    for _, row in tqdm(df.iterrows(), total=len(df)):
        question = row["question"]
        if agieval:
            choices_str = row["answer"]
        else:
            choices = eval(row["options"])
            choices_str = "\n".join([f"({chr(65+i)}) {c}" for i, c in enumerate(choices)])
        user_prompt = f"---\nQ: {question}\nChoices:\n{choices_str}\nA:"
        full_prompts.append(user_prompt)

    batch_messages = [[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": prompt}
    ] for prompt in full_prompts]

    prompts = [
        tokenizer.apply_chat_template(msg, tokenize=False, enable_thinking=False, add_generation_prompt=True)
        for msg in batch_messages
    ]

    outputs = []
    for i in tqdm(range(0, len(prompts), batch_size)):
        batch = prompts[i:i+batch_size]
        result = pipe(batch, batch_size=batch_size, max_new_tokens=2035)
        outputs.extend([r[0]["generated_text"] for r in result])

    df["full_prompt"] = full_prompts
    df["model_output"] = outputs

    model_name = pipe.model.config.name_or_path.split("/")[-1].replace(":", "_")
    timestamp = datetime.now().strftime("%H-%M_%Y-%m-%d")
    folder_name = model_name
    os.makedirs(folder_name, exist_ok=True)
    file_name = f"{model_name}_{timestamp}.csv"
    file_path = os.path.join(folder_name, file_name)

    df.to_csv(file_path, index=False)
```

### Expected behavior

When you will run the code, the warning will be shown for Qwen3 models and Llama4 models.

Comments:
cc @gante as well maybe, since I think that warning is generation-related?

Hi @sleepingcat4 👋 

Yes, the warning is thrown at generation-time: at text generation time, when padding is used, it should be left-padding (see [here](https://huggingface.co/docs/transformers/llm_tutorial#padding-side) for more details).

Can you try passing the `attention_mask` from tokenization into the pipeline as well? In general, it's good practice always to pass the attention mask to the forward pass. If it doesn't work, let me know :) 

@gante thanks but we didn't see any issues with our code in terms of what the model was generating. I have tried to do what the error told me but it produces garbage. Can you please try my code and see once?

@sleepingcat4 happy to try your code, but the snippet you shared above is not self-contained -- I should be able to copy/paste it and then run locally, with zero changes from my side :)

ughh! I was actually using the code for running some private bench, you can swap it with noise data? 

What's really triggering the warning is the decoder specific model like Qwen3 and Llama4 🦙 

I think you can get it any arbitary code as well. My problem is that the warning is misleading as if I do what it says setting padding to **right** then the results are gibberish and passing attention mask also produce some gibberish but if I let the warning persist then the generation is normal. 

That's what bugging me. 

Yes, it can be a super small model with dummy data, as long as I can reproduce the issue 🤗 Otherwise, from my end, it would be very difficult to pin the exact issue, and therefore suggest a fix!

@gante I have written this colab: https://colab.research.google.com/drive/1b0b1JykqRBjNHo9tjnFBcDeX2IDoCwv4?usp=sharing

I don't see the warning on Colab although I have used the exact same specification on the colab that I am using to eval the models. For me it is extremely important that the code is concrete as I am using the results for my paper. and having a slight doubt won't be acceptable for me and my team. If you want I can create another colab notebook and paste my entire code there and maybe you can help me to debug. 

**For me I don't see any errors in the model's generation despite seeing the warning which means it is working correctly. But, it should be noted the colab environment and my Lambda instance use a different version of transformers. Maybe the warning is only seen recent version of transformers. I have hand checked the model outputs and my team members as well and it looks normal without any weird generation. Would love some thoughts**

@gante I get this warning on my 3090 server which I don't get on colab

```bash
2025-06-01 17:47:30,208 — INFO — Loading model...
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 726/726 [00:00<00:00, 5.53MB/s]
model.safetensors.index.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32.8k/32.8k [00:00<00:00, 50.4MB/s]
model-00003-of-00003.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 99.6M/99.6M [00:01<00:00, 87.4MB/s]
model-00002-of-00003.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.99G/3.99G [00:07<00:00, 550MB/s]
model-00001-of-00003.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.96G/3.96G [00:07<00:00, 519MB/s]
Fetching 3 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:08<00:00,  2.99s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00,  3.10it/s]
generation_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 239/239 [00:00<00:00, 2.38MB/s]
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.73k/9.73k [00:00<00:00, 57.2MB/s]
vocab.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.78M/2.78M [00:00<00:00, 4.27MB/s]
merges.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.67M/1.67M [00:00<00:00, 3.12MB/s]
tokenizer.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.4M/11.4M [00:00<00:00, 14.2MB/s]
Device set to use cuda:0
2025-06-01 17:47:48,136 — INFO — Validating dataset...
2025-06-01 17:47:48,136 — INFO — Preparing data and prompts...
2025-06-01 17:47:48,152 — INFO — Starting model inference...
  0%|                                                                                                                                                                                        | 0/5 [00:00<?, ?it/s]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
 20%|███████████████████████████████████▏                                                                                                                                            | 1/5 [00:05<00:23,  5.83s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
 40%|██████████████████████████████████████████████████████████████████████▍                                                                                                         | 2/5 [00:11<00:16,  5.65s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
 60%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                                      | 3/5 [00:16<00:11,  5.59s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
 80%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                                   | 4/5 [00:22<00:05,  5.56s/it]A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:27<00:00,  5.59s/it]
2025-06-01 17:48:16,105 — INFO — Results saved to: Qwen3-4B/Qwen3-4B_17-48_2025-06-01.jsonl
```

@gante @ArthurZucker I have ran some test with what the warning was asking me to do **padding in the left for a decoder only architecture** and the results are bad. The model simply can't even answer simple questions like *what's the capital of France?* VS when I don't do anything and the model can easily answer and I used Qwen3-4B.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.