T5Gemma problem with tokenizer(?)
Author: jadermcs
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.53.2
- Platform: Linux-6.14.0-23-generic-x86_64-with-glibc2.41
- Python version: 3.13.3
- Huggingface_hub version: 0.33.4
- Safetensors version: 0.5.3
- Accelerate version: 1.8.1
- Accelerate config: 	- compute_environment: LOCAL_MACHINE
	- distributed_type: NO
	- mixed_precision: bf16
	- use_cpu: False
	- debug: False
	- num_processes: 1
	- machine_rank: 0
	- num_machines: 1
	- gpu_ids: all
	- rdzv_backend: static
	- same_network: True
	- main_training_function: main
	- enable_cpu_affinity: True
	- downcast_bf16: no
	- tpu_use_cluster: False
	- tpu_use_sudo: False
	- tpu_env: []
	- dynamo_config: {'dynamo_backend': 'INDUCTOR'}
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 5060 Ti

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction



```python
from datasets import load_dataset
import numpy as np
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from transformers import EvalPrediction
from transformers import (
    AutoTokenizer,
    AutoModelForSeq2SeqLM,
    Seq2SeqTrainer,
    Seq2SeqTrainingArguments,
)

# Convert to Hugging Face Dataset
dataset = load_dataset("super_glue", "wic")

# Initialize tokenizer and model
model_name = "google/t5gemma-b-b-ul2-it"
# model_name = "google-t5/t5-small"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name, attn_implementation="eager")


def compute_metrics(eval_pred: EvalPrediction):
    predictions, labels = eval_pred
    # Decode predicted token IDs to strings
    pred_str = tokenizer.batch_decode(predictions, skip_special_tokens=True)
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    label_str = tokenizer.batch_decode(labels, skip_special_tokens=True)
    print(pred_str)
    print(label_str)

    # Convert "true"/"false" strings to 1/0
    pred_labels = [1 if p.strip().lower() == "true" else 0 for p in pred_str]
    true_labels = [1 if l.strip().lower() == "true" else 0 for l in label_str]
    # compute precision, recall, f1
    precision, recall, f1_score, _ = precision_recall_fscore_support(
        true_labels, pred_labels, average="binary"
    )
    accuracy = accuracy_score(true_labels, pred_labels)

    return {
        "accuracy": accuracy,
        "precision": precision,
        "recall": recall,
        "f1_score": f1_score,
    }


# Preprocessing function
def preprocess(example):
    input_text = f"sentence1: {example['sentence1']} sentence2: {example['sentence2']} word: {example['word']}"
    target_text = "true" if example["label"] == 1 else "false"
    target_text = target_text + tokenizer.eos_token

    # Tokenize inputs and targets
    model_inputs = tokenizer(
        input_text, max_length=128, truncation=True, padding="max_length"
    )
    labels = tokenizer(target_text, max_length=5, truncation=True, padding="max_length")

    # Replace pad token id's in labels with -100 so they're ignored by loss
    labels_ids = labels["input_ids"]
    labels_ids = [
        label if label != tokenizer.pad_token_id else -100 for label in labels_ids
    ]

    model_inputs["labels"] = labels_ids
    return model_inputs


# Tokenize dataset
tokenized_dataset = dataset.map(
    preprocess, remove_columns=dataset["train"].column_names
)

# Training arguments
training_args = Seq2SeqTrainingArguments(
    output_dir="./t5-wic",
    eval_strategy="epoch",
    per_device_train_batch_size=32,
    num_train_epochs=10,
    save_strategy="epoch",
    save_total_limit=1,
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    predict_with_generate=True,
    bf16=True,
)
print(tokenized_dataset["train"][0])
print(tokenizer.decode(tokenized_dataset["train"][0]["input_ids"]))
# remove -100
labels = [
    label if label != -100 else tokenizer.pad_token_id
    for label in tokenized_dataset["train"][0]["labels"]
]
print(tokenizer.decode(labels))
# Initialize Trainer
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"],
    compute_metrics=compute_metrics,
)

# Train the model
trainer.train()
```

### Expected behavior

I am training T5Gemma for Word-in-Context binary classification as sentence-to-sentence problem (the same as original T5 paper). However the model is predicting the same label. Initially, I notice that the tokenizer do not add the end-of-string token so I manually did it in my code. It went from "falsetruetruetruetrue" until reaching maximum tokens. Now, after adding eos, it predicts only true.

Any help here? The code above works with "google-t5/t5-small"

Comments:
> ### System Info
> 
> * `transformers` version: 4.53.2
> * Platform: Linux-6.14.0-23-generic-x86_64-with-glibc2.41
> * Python version: 3.13.3
> * Huggingface_hub version: 0.33.4
> * Safetensors version: 0.5.3
> * Accelerate version: 1.8.1
> * Accelerate config: 	- compute_environment: LOCAL_MACHINE
>   
>   * distributed_type: NO
>   * mixed_precision: bf16
>   * use_cpu: False
>   * debug: False
>   * num_processes: 1
>   * machine_rank: 0
>   * num_machines: 1
>   * gpu_ids: all
>   * rdzv_backend: static
>   * same_network: True
>   * main_training_function: main
>   * enable_cpu_affinity: True
>   * downcast_bf16: no
>   * tpu_use_cluster: False
>   * tpu_use_sudo: False
>   * tpu_env: []
>   * dynamo_config: {'dynamo_backend': 'INDUCTOR'}
> * DeepSpeed version: not installed
> * PyTorch version (accelerator?): 2.7.1+cu128 (CUDA)
> * Tensorflow version (GPU?): not installed (NA)
> * Flax version (CPU?/GPU?/TPU?): not installed (NA)
> * Jax version: not installed
> * JaxLib version: not installed
> * Using distributed or parallel set-up in script?:
> * Using GPU in script?:
> * GPU type: NVIDIA GeForce RTX 5060 Ti
> 
> ### Who can help?
> 
> [@ArthurZucker](https://github.com/ArthurZucker)
> 
> ### Information
> 
> * [ ]  The official example scripts[x]  My own modified scripts
> 
> ### Tasks
> 
> * [x]  An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)[ ]  My own task or dataset (give details below)
> 
> ### Reproduction
> 
> from datasets import load_dataset
> import numpy as np
> from sklearn.metrics import accuracy_score, precision_recall_fscore_support
> from transformers import EvalPrediction
> from transformers import (
>     AutoTokenizer,
>     AutoModelForSeq2SeqLM,
>     Seq2SeqTrainer,
>     Seq2SeqTrainingArguments,
> )
> 
> # Convert to Hugging Face Dataset
> dataset = load_dataset("super_glue", "wic")
> 
> # Initialize tokenizer and model
> model_name = "google/t5gemma-b-b-ul2-it"
> # model_name = "google-t5/t5-small"
> tokenizer = AutoTokenizer.from_pretrained(model_name)
> model = AutoModelForSeq2SeqLM.from_pretrained(model_name, attn_implementation="eager")
> 
> 
> def compute_metrics(eval_pred: EvalPrediction):
>     predictions, labels = eval_pred
>     # Decode predicted token IDs to strings
>     pred_str = tokenizer.batch_decode(predictions, skip_special_tokens=True)
>     labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
>     label_str = tokenizer.batch_decode(labels, skip_special_tokens=True)
>     print(pred_str)
>     print(label_str)
> 
>     # Convert "true"/"false" strings to 1/0
>     pred_labels = [1 if p.strip().lower() == "true" else 0 for p in pred_str]
>     true_labels = [1 if l.strip().lower() == "true" else 0 for l in label_str]
>     # compute precision, recall, f1
>     precision, recall, f1_score, _ = precision_recall_fscore_support(
>         true_labels, pred_labels, average="binary"
>     )
>     accuracy = accuracy_score(true_labels, pred_labels)
> 
>     return {
>         "accuracy": accuracy,
>         "precision": precision,
>         "recall": recall,
>         "f1_score": f1_score,
>     }
> 
> 
> # Preprocessing function
> def preprocess(example):
>     input_text = f"sentence1: {example['sentence1']} sentence2: {example['sentence2']} word: {example['word']}"
>     target_text = "true" if example["label"] == 1 else "false"
>     target_text = target_text + tokenizer.eos_token
> 
>     # Tokenize inputs and targets
>     model_inputs = tokenizer(
>         input_text, max_length=128, truncation=True, padding="max_length"
>     )
>     labels = tokenizer(target_text, max_length=5, truncation=True, padding="max_length")
> 
>     # Replace pad token id's in labels with -100 so they're ignored by loss
>     labels_ids = labels["input_ids"]
>     labels_ids = [
>         label if label != tokenizer.pad_token_id else -100 for label in labels_ids
>     ]
> 
>     model_inputs["labels"] = labels_ids
>     return model_inputs
> 
> 
> # Tokenize dataset
> tokenized_dataset = dataset.map(
>     preprocess, remove_columns=dataset["train"].column_names
> )
> 
> # Training arguments
> training_args = Seq2SeqTrainingArguments(
>     output_dir="./t5-wic",
>     eval_strategy="epoch",
>     per_device_train_batch_size=32,
>     num_train_epochs=10,
>     save_strategy="epoch",
>     save_total_limit=1,
>     load_best_model_at_end=True,
>     metric_for_best_model="accuracy",
>     predict_with_generate=True,
>     bf16=True,
> )
> print(tokenized_dataset["train"][0])
> print(tokenizer.decode(tokenized_dataset["train"][0]["input_ids"]))
> # remove -100
> labels = [
>     label if label != -100 else tokenizer.pad_token_id
>     for label in tokenized_dataset["train"][0]["labels"]
> ]
> print(tokenizer.decode(labels))
> # Initialize Trainer
> trainer = Seq2SeqTrainer(
>     model=model,
>     args=training_args,
>     train_dataset=tokenized_dataset["train"],
>     eval_dataset=tokenized_dataset["validation"],
>     compute_metrics=compute_metrics,
> )
> 
> # Train the model
> trainer.train()
> 
> metrics = trainer.evaluate(tokenized_dataset["test"])
> print("Final metrics:")
> print(metrics)
> ### Expected behavior
> 
> I am training T5Gemma for Word-in-Context binary classification as sentence-to-sentence problem (the same as original T5 paper). However the model is predicting the same label. Initially, I notice that the tokenizer do not add the end-of-string token so I manually did it in my code. It went from "falsetruetruetruetrue" until reaching maximum tokens. Now, after adding eos, it predicts only true.
> 
> Any help here? The code above works with "google-t5/t5-small"



This isn't a transformers library bug - it's a model-specific behavior difference. T5Gemma uses different tokenization and expects different input formats than standard T5

> This isn't a transformers library bug - it's a model-specific behavior difference. T5Gemma uses different tokenization and expects different input formats than standard T5

This tokenization is described in the documentation, so it should be correct (or the documentation have to be update).

The origin of the bug came from `remove_unused_columns=True` when set to false (and concatenating the EOS token) it worked fine. So I still think the Tokenizer should handle the correct tokenization as other models does.

Okay, will close as it does not seem an issue, thanks for the deepdive!