Gemma3
Author: fuchao01
Pull request: no
Labels: name: bug

### System Info

After installing the latest Transformers and reasoning about Gemma 3 video understanding, this error is reported

pip install -q git+https://github.com/huggingface/transformers@v4.49.0-Gemma-3

# -*- coding: utf-8 -*-

"""Let's load the model."""

import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration

ckpt = "google/gemma-3-4b-it"
model = Gemma3ForConditionalGeneration.from_pretrained(
    ckpt, device_map="auto", torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained(ckpt)

"""Download the video and downsample the frames from the video."""


import cv2
from PIL import Image
import numpy as np

def downsample_video(video_path):
    vidcap = cv2.VideoCapture(video_path)
    total_frames = int(vidcap.get(cv2.CAP_PROP_FRAME_COUNT))
    fps = vidcap.get(cv2.CAP_PROP_FPS)

    frames = []
    frame_indices = np.linspace(0, total_frames - 1, 10, dtype=int)

    for i in frame_indices:
        vidcap.set(cv2.CAP_PROP_POS_FRAMES, i)
        success, image = vidcap.read()
        if success:
            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Convert from BGR to RGB
            pil_image = Image.fromarray(image)
            timestamp = round(i / fps, 2)
            frames.append((pil_image, timestamp))

    vidcap.release()
    return frames

frames = downsample_video("30515642-b6d9-11ef-b24f-fa163ea3a38d_zip.mp4")

frames

"""Here's our system prompt and the instruction. We will add frames and images on top of it."""

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}]
    },

    {
        "role": "user",
        "content": [
            {"type": "text", "text": f"What is happening in this video? Summarize the events."}]
    }
]

messages[1]["content"][0]

for frame in frames:
    image, timestamp = frame
    messages[1]["content"].append({"type": "text", "text": f"Frame {timestamp}:"})
    image.save(f"image_{timestamp}.png")
    messages[1]["content"].append({"type": "image", "url": f"image_{timestamp}.png"})

messages

"""Preprocess our input and infer."""

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)

processed_chat = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    video_fps=32,
    video_load_backend="decord",
)


input_len = inputs["input_ids"].shape[-1]

generation = model.generate(**inputs, max_new_tokens=500, do_sample=True)
generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

Traceback (most recent call last):
  File "/picassox/intelligence-sfs-turbo-cv/fc3/work/Gemma_3_for_Video_Understanding.py", line 89, in <module>
    generation = model.generate(**inputs, max_new_tokens=500, do_sample=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/picassox/intelligence-sfs-turbo-cv/fc3/work/transformers/src/transformers/generation/utils.py", line 2314, in generate
    result = self._sample(
  File "/picassox/intelligence-sfs-turbo-cv/fc3/work/transformers/src/transformers/generation/utils.py", line 3294, in _sample
    outputs = model_forward(**model_inputs, return_dict=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/picassox/intelligence-sfs-turbo-cv/fc3/work/transformers/src/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
  File "/picassox/intelligence-sfs-turbo-cv/fc3/work/transformers/src/transformers/models/gemma3/modeling_gemma3.py", line 1352, in forward
    outputs = self.language_model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/picassox/intelligence-sfs-turbo-cv/fc3/work/transformers/src/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
  File "/picassox/intelligence-sfs-turbo-cv/fc3/work/transformers/src/transformers/models/gemma3/modeling_gemma3.py", line 976, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/picassox/intelligence-sfs-turbo-cv/fc3/work/transformers/src/transformers/models/gemma3/modeling_gemma3.py", line 754, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/picassox/intelligence-sfs-turbo-cv/fc3/work/transformers/src/transformers/models/gemma3/modeling_gemma3.py", line 443, in forward
    hidden_states, self_attn_weights = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/picassox/intelligence-sfs-turbo-cv/fc3/work/transformers/src/transformers/models/gemma3/modeling_gemma3.py", line 365, in forward
    attn_output, attn_weights = attention_interface(
  File "/picassox/intelligence-sfs-turbo-cv/fc3/work/transformers/src/transformers/integrations/sdpa_attention.py", line 54, in sdpa_attention_forward
    attn_output = torch.nn.functional.scaled_dot_product_attention(
RuntimeError: p.attn_bias_ptr is not correctly aligned

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

test

### Expected behavior

bug

Comments:
I guess this is torch version related issue, which version you have and can you try to update?

The torch version I use：
torch                        2.4.0+cu124
torchaudio                   2.4.0+cu124

I tried the new version and got the same error.
torch                        2.6.0
torchaudio                   2.6.0

I got the same error message with MWE below. Same bug maybe?

```
generator = transformers.pipeline(
            "text-generation",
            model="gemma-2-2b-it",
            model_kwargs={"torch_dtype": torch.bfloat16},
            device_map="auto",
            return_full_text=False
        )
generator(prompt, max_new_tokens=256, truncation=True, num_return_sequences=1, do_sample=True)
```

This problem was solved for me by https://github.com/google-deepmind/gemma/issues/169

You may need this file https://github.com/zhurui1995/gemma3--OpenAI-Compatible-API-Server/blob/main/server.py

### The Crux of the Problem

The problem you are facing is not related to HuggingFace or PyTorch but arises from the way FlashAttention works.

### How is FlashAttention Responsible?

The sole purpose of FlashAttention is speed, for which it utilizes the shared memory of the tensor cores of a GPU. This is where the "problem" lies:

1 - These tensor cores require all operations to be executed 8 bytes, 16 bytes, or 32 bytes.
2 - Warp-level execution on NVIDIA GPUs is optimized for multiples of 8/16/32 threads.

If, let's say, the tensors are misaligned, it causes the strides to be non-linear in nature which slows down computation heavily.

So, it would be wrong to say that **misaligned tensors are not allowed**. It's just that it is **very inefficient to work with misaligned tensors.**

***Therefore, one fix would be to skip FlashAttention and use EagerAttention instead.***

To ensure that these requirements are met, PyTorch calls the following macro to ensure that FlashAttention can be used:
```
#define CHECK_ALIGNED_PTR(PTR, ALIGNMENT) \
  TORCH_CHECK(                         \
      uint64_t(PTR) % ALIGNMENT == 0, #PTR " is not correctly aligned")
```

In `functional.py` file of the PyTorch repository, you can find the following note:

```
There are currently three supported implementations of scaled dot product attention:

            - `FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning`_
            - `Memory-Efficient Attention`_
            - A PyTorch implementation defined in C++ matching the above formulation

        The function may call optimized kernels for improved performance when using the CUDA backend.
        For all other backends, the PyTorch implementation will be used.

        All implementations are enabled by default. Scaled dot product attention attempts to automatically select the
        most optimal implementation based on the inputs. In order to provide more fine-grained control over what implementation
        is used, the following functions are provided for enabling and disabling implementations.
        The context manager is the preferred mechanism:

            - :func:`torch.nn.attention.sdpa_kernel`: A context manager used to enable or disable any of the implementations.
            - :func:`torch.backends.cuda.enable_flash_sdp`: Globally enables or disables FlashAttention.
            - :func:`torch.backends.cuda.enable_mem_efficient_sdp`: Globally enables or disables  Memory-Efficient Attention.
            - :func:`torch.backends.cuda.enable_math_sdp`: Globally enables or disables  the PyTorch C++ implementation.
```

***Therefore, the most optimal fix would be the following:***
```
pad_to_multiple_of: Optional[int]
```
Set this value to a multiple of 8 in the `processor` or the `tokenizer` as it is a `kwarg` for textual inputs in HuggingFace's Transformers.

## TL;DR
Fix 1 - Use Vanilla Attention (EagerAttention in PyTorch).
Fix 2 (Recommended) - Set the `pad_to_multiple_of` parameter in the `tokenizer` or `processor` to a multiple of 8.

I also got same error I was able to solve it by adding padding while generating inputs

code looks like 

`inputs = processor.apply_chat_template(
        messages, add_generation_prompt=True, tokenize=True,
        return_dict=True, return_tensors="pt", padding="max_length", max_length=4096, truncation=True
    ).to(model.device, dtype=torch.bfloat16)` 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.