Add Gemma 3 For Sequence Classification
Author: AhmedHashish123
Pull request: no
Labels: name: Feature request

### Feature request

Hello, I was wondering when will you add support for Gemma3Config for sequence generation as currently only GemmaConfig and Gemma2Config are supported

### Motivation

This would be extremely beneficial given that Gemma 2 2B Instruct excels as a sequence classifier. I would expect Gemma 3 4B to be even more performant. 

### Your contribution

I already did something to finetune Gemma 3 1B by using Gemma2ForSequenceClassification:
In the "modeling_gemma2.py", I import the Gemma 3 text model class as follows:
from ..gemma3.modeling_gemma3 import Gemma3TextModel

and then in "Gemma2ForSequenceClassification" class in the same file, I change this line:
self.model = Gemma2Model(config)
to
self.model = Gemma3TextModel(config)

Comments:
@AhmedHashish123 

We usually add a class for task when there is an official checkpoint for the task or when we many requets from community.

In case of gemma3, we can add the Sequence Classification since we've had a lot of request to add it for Vision LLMs with other models (Qwen2-VL and LLaVa). And currently none of the vision LLMs have a classification class. I am totally fine with the addition, and will be happy to review a PR :)

cc @ArthurZucker just in case you don't want the class to be added, unless we get more requests for Gemma3 specifically?

Thanks @zucchini-nlp !
I was mainly looking to add it for the text part of the models. Just like the regular LLMs' classifiers where we just remove the last layer and add a linear layer with the output being the desired number of classes. 

Ah oke. As I noted, we try to not add new classes unless there is a clear need or high community request, so I suggest to wait a bit for this issue to get community reaction. If there is high request, we can add support in the core library :)

Would love this

would also love to see this!

I came across this issue after searching for "sequence classification" and "Gemma 3", so one more person is interested in this feature ;)
Especially since Gemma is exceptionally good for its size, and most models, like BERT or DeBERTa, have a limited sequence length. 

Useful feature, look forward this.

Hey folks, given the high interest for this task, we can add it in core library.

The model will be mostly same as Gemma2 Sequence Classification, with the only difference that it will accept pixel values as input. That means we need a base model for multimodal Gemma3, which I am adding here (https://github.com/huggingface/transformers/pull/37033). After that we can add `Gemma3ForSequenceClassification` 

Hi, @zucchini-nlp, I'm really interested in using Gemma 3 for sequence classification myself, so I appreciate you moving forward with this!

I haven't contributed to open source before, but it is a goal of mine to start. Is there anything I could do to help with this? I'd be happy to assist if pointed in the right direction

@jacob-danner I would prefer the model to inherit from base `Gemma3Model` class, so we don't duplicate the same code for each possible task. It is annoying to read the code otherwise, let's wait until the base model is added and then I'll ping you here :)

Basically it will be same as Gemma2, with difference that the base model will accept images as well 

Hey guys, I'm interested in supporting QA for Gemma 3: https://github.com/huggingface/transformers/issues/36977. What do you think about this?

That would be wonderful thing to have `Gemma3ForSequenceClassification` supported with image support. Gemma 3 is a very very promising set of models. 

Is there any news on this topic?

I have also conducted fine-tuning on Gemma3 1B it would be wonderful if `Gemma3ForSequenceClassification` can be supported!

Support for this feature, would love to contribute as well!

It is very god idea to add this, especially because gemma 2 2b was even better than debert v3 large in my tasks as per my research!

So i expect that gemma  3 is even better or just smaller size and the same!

Classification can be added like this. With 4bits i get much lower accuracy, while bf16 works best, and quite fast.

It only works for gemma 3 1b model, but it works even better than gemma 2 2b model.

for gemma 3 4b Gemma3ForCausalLM wont work, you need to use one for conditional generation.

```
model_id = "unsloth/gemma-3-1b-pt"

from transformers import Gemma3ForCausalLM, BitsAndBytesConfig
import torch.nn as nn
import torch


bmodel = Gemma3ForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
)

bmodel.config.output_hidden_states = True          
bmodel.gradient_checkpointing_enable()          

class SequenceClassifier(nn.Module):
    def __init__(self, bmodel, hidden_size):
        super().__init__()
        self.model = bmodel
        self.dropout = nn.Dropout(0.1)
        self.classifier = nn.Linear(hidden_size, 2)

    def forward(self, input_ids, attention_mask=None, labels=None):
        outputs = self.model(input_ids=input_ids, attention_mask=attention_mask)
        hidden_state = outputs.hidden_states[-1]   # ← korišćenje skrivenih slojeva
        pooled       = hidden_state[:, -1, :]      # poslednji token
        logits       = self.classifier(self.dropout(pooled))

        loss = None
        if labels is not None:
            loss = nn.CrossEntropyLoss()(logits, labels)
        return {"loss": loss, "logits": logits} if loss is not None else {"logits": logits}

clf_model = SequenceClassifier(bmodel, bmodel.config.hidden_size).to("cuda")
```

I've also needed sequence classification for Gemma 3 models.
This monkey-patch was made with ChatGPT o3. I've tested with the text-only 1B model, and the multimodal 4B models, works fine for me.
You can just import this as a module, and use the `AutoModelForSequenceClassification` like anywhere else, it will load Gemma 3 models as expected.

```python
"""
Universal Gemma-3 sequence-classification heads.
Works with
  • google/gemma-3-1b-(base|it)  (text only)
  • google/gemma-3-4b-* and larger (multimodal)
Import once, early, then call
    AutoModelForSequenceClassification.from_pretrained(...)
"""
import torch
import torch.nn as nn
from transformers import AutoModelForSequenceClassification
from transformers.models.gemma3.configuration_gemma3 import (
    Gemma3Config, Gemma3TextConfig,
)
from transformers.models.gemma3.modeling_gemma3 import (
    Gemma3Model, Gemma3TextModel, Gemma3PreTrainedModel,
)
from transformers.modeling_outputs import SequenceClassifierOutputWithPast


# ───────────────────── helpers ─────────────────────────────────────────
def _txt(cfg):                              # text sub-config
    return getattr(cfg, "text_config", cfg)


def _pool_last(token_logits, input_ids, pad_id):
    if input_ids is None or pad_id is None:
        return token_logits[:, -1]
    ends = (~input_ids.eq(pad_id)).cumsum(-1).argmax(-1)
    return token_logits[torch.arange(token_logits.size(0)), ends]


def _compute_loss(config, logits, labels, num_labels):
    if labels is None:
        return None
    if config.problem_type is None:
        if num_labels == 1:
            config.problem_type = "regression"
        elif labels.dtype in (torch.long, torch.int):
            config.problem_type = "single_label_classification"
        else:
            config.problem_type = "multi_label_classification"

    if config.problem_type == "regression":
        return nn.MSELoss()(logits.squeeze(), labels.squeeze())
    if config.problem_type == "single_label_classification":
        return nn.CrossEntropyLoss()(logits, labels)
    return nn.BCEWithLogitsLoss()(logits, labels)


# ───────────────────── multimodal variant (4 B+) ──────────────────────
class Gemma3ForSequenceClassification(Gemma3Model):
    config_class = Gemma3Config
    _no_split_modules = ["GemmaBlock"]
    keys_to_ignore_at_inference = ["past_key_values"]

    def __init__(self, config):
        super().__init__(config)  # backbone
        self.num_labels = config.num_labels
        self.score = nn.Linear(_txt(config).hidden_size,
                               self.num_labels, bias=False)
        self.post_init()

    def forward(
        self,
        input_ids=None, attention_mask=None, position_ids=None,
        inputs_embeds=None, pixel_values=None, past_key_values=None,
        labels=None, use_cache=None, output_attentions=None,
        output_hidden_states=None, return_dict=None,
    ):
        return_dict = (return_dict if return_dict is not None
                       else self.config.use_return_dict)

        outputs = super().forward(
            input_ids=input_ids,
            attention_mask=attention_mask,
            position_ids=position_ids,
            inputs_embeds=inputs_embeds,
            pixel_values=pixel_values,
            past_key_values=past_key_values,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=True,
        )
        token_logits = self.score(outputs.last_hidden_state)
        pad_id = getattr(self.config, "pad_token_id",
                         getattr(_txt(self.config), "pad_token_id", None))
        pooled = _pool_last(token_logits, input_ids, pad_id)
        loss = _compute_loss(self.config, pooled, labels, self.num_labels)

        if not return_dict:
            out = (pooled,) + outputs[1:]
            return ((loss,) + out) if loss is not None else out

        return SequenceClassifierOutputWithPast(
            loss=loss, logits=pooled,
            past_key_values=outputs.past_key_values,
            hidden_states=outputs.hidden_states
                if output_hidden_states else None,
            attentions=outputs.attentions
                if output_attentions else None,
        )


# ───────────────────── text-only variant (1 B) ────────────────────────
class Gemma3TextForSequenceClassification(Gemma3PreTrainedModel):
    """
    **Wraps** Gemma3TextModel in `self.model` – keeps `model.*`
    prefixes so every pretrained weight loads.
    """
    config_class = Gemma3TextConfig
    keys_to_ignore_at_inference = ["past_key_values"]

    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels
        self.model = Gemma3TextModel(config)              # <- wrapper
        self.score = nn.Linear(config.hidden_size,
                               self.num_labels, bias=False)
        self.post_init()

    def forward(
        self,
        input_ids=None, attention_mask=None, position_ids=None,
        inputs_embeds=None, past_key_values=None, labels=None,
        use_cache=None, output_attentions=None,
        output_hidden_states=None, return_dict=None,
    ):
        return_dict = (return_dict if return_dict is not None
                       else self.config.use_return_dict)

        outputs = self.model(
            input_ids=input_ids,
            attention_mask=attention_mask,
            position_ids=position_ids,
            inputs_embeds=inputs_embeds,
            past_key_values=past_key_values,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=True,
        )
        token_logits = self.score(outputs.last_hidden_state)
        pooled = _pool_last(token_logits, input_ids, self.config.pad_token_id)
        loss = _compute_loss(self.config, pooled, labels, self.num_labels)

        if not return_dict:
            out = (pooled,) + outputs[1:]
            return ((loss,) + out) if loss is not None else out

        return SequenceClassifierOutputWithPast(
            loss=loss, logits=pooled,
            past_key_values=outputs.past_key_values,
            hidden_states=outputs.hidden_states
                if output_hidden_states else None,
            attentions=outputs.attentions
                if output_attentions else None,
        )


# ───────────────────── register with HF factory ───────────────────────
AutoModelForSequenceClassification.register(
    Gemma3Config, Gemma3ForSequenceClassification)
AutoModelForSequenceClassification.register(
    Gemma3TextConfig, Gemma3TextForSequenceClassification)
```

Oh btw, forgot about this issue. We have done the refactor on vision LLMs and a PR for gemma3 sequence classification would be more than welcome, if anyone wants to give it a try 🤗 

Following the same style as in Llama but using a base multimodal model will be the right way to implement it

Any updates on this? Sadly it's not in my expertise to help with the code here :(

Any updates on Gemma 3 for sequence classification? Would be great if vision LLMs have with a classification class.

Seems like no-one is working on it currently. I'll submit a PR then today

Yes, should be very very straighforward 😉