T5Gemma returning 0 loss for s2s training
Author: jadermcs
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.53.0
- Platform: Linux-6.14.0-23-generic-x86_64-with-glibc2.41
- Python version: 3.13.3
- Huggingface_hub version: 0.33.1
- Safetensors version: 0.5.3
- Accelerate version: 1.8.1
- Accelerate config: 	- compute_environment: LOCAL_MACHINE
	- distributed_type: NO
	- mixed_precision: bf16
	- use_cpu: False
	- debug: False
	- num_processes: 1
	- machine_rank: 0
	- num_machines: 1
	- gpu_ids: all
	- rdzv_backend: static
	- same_network: True
	- main_training_function: main
	- enable_cpu_affinity: True
	- downcast_bf16: no
	- tpu_use_cluster: False
	- tpu_use_sudo: False
	- tpu_env: []
	- dynamo_config: {'dynamo_backend': 'INDUCTOR'}
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 5060 Ti

### Who can help?

@zach-huggingface @SunMarc 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```python
from datasets import load_dataset
import numpy as np
from sklearn.metrics import accuracy_score
from transformers import EvalPrediction
from transformers import (
    AutoTokenizer,
    T5GemmaForConditionalGeneration,
    Seq2SeqTrainer,
    Seq2SeqTrainingArguments,
)

# Convert to Hugging Face Dataset
dataset = load_dataset("super_glue", "wic")

# Initialize tokenizer and model
model_name = "google/t5gemma-s-s-ul2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = T5GemmaForConditionalGeneration.from_pretrained(
    model_name, attn_implementation="eager"
)


def compute_metrics(eval_pred: EvalPrediction):
    predictions, labels = eval_pred
    # Decode predicted token IDs to strings
    pred_str = tokenizer.batch_decode(predictions, skip_special_tokens=True)
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    label_str = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Convert "true"/"false" strings to 1/0
    pred_labels = [1 if p.strip().lower() == "true" else 0 for p in pred_str]
    true_labels = [1 if l.strip().lower() == "true" else 0 for l in label_str]

    return {"accuracy": accuracy_score(true_labels, pred_labels)}


# Preprocessing function
def preprocess(example):
    input_text = f"sentence1: {example['sentence1']} sentence2: {example['sentence2']} word: {example['word']}"
    target_text = "true" if example["label"] == 1 else "false"

    # Tokenize inputs and targets
    model_inputs = tokenizer(
        input_text, max_length=128, truncation=True, padding="max_length"
    )
    labels = tokenizer(target_text, max_length=5, truncation=True, padding="max_length")

    # Replace pad token id's in labels with -100 so they're ignored by loss
    labels_ids = labels["input_ids"]
    labels_ids = [
        label if label != tokenizer.pad_token_id else -100 for label in labels_ids
    ]

    model_inputs["labels"] = labels_ids
    return model_inputs


# Tokenize dataset
tokenized_dataset = dataset.map(
    preprocess, remove_columns=dataset["train"].column_names
)

# Training arguments
training_args = Seq2SeqTrainingArguments(
    output_dir="./t5-wic",
    eval_strategy="epoch",
    per_device_train_batch_size=16,
    num_train_epochs=10,
    save_strategy="epoch",
    save_total_limit=1,
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    predict_with_generate=True,
    bf16=True,
)
print(tokenized_dataset["train"][0])
# Initialize Trainer
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)

# Train the model
trainer.train()

metrics = trainer.evaluate(tokenized_dataset["test"])
print("Final metrics:")
print(metrics)
```

### Expected behavior

When I run this code I first get a:
```
`loss_type=ForMaskedLMLoss` was set in the config but it is unrecognised.Using the default loss: `ForCausalLMLoss`.
```

Then I have this for every epoch:
```bash
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 4.2661764705882354e-05, 'epoch': 1.47}
{'eval_loss': nan, 'eval_accuracy': 0.5, 'eval_runtime': 19.3888, 'eval_samples_per_second': 32.906, 'eval_steps_per_second': 4.126, 'epoch': 2.0}
```

Comments:
I see it was fixed in #39052