Generating <extra_id_0> Tokens Instead of Expected Text Responses After Fine-tuning mT5 Model
Author: calchora
Pull request: no

### System Info

### I have encountered an issue with the mT5 model generating `<extra_id_0>` tokens instead of meaningful text responses, despite successful fine-tuning on a custom dataset. This behavior persists even when interacting with the model using inputs similar to the training data.

**Environment:**
- transformers version: **4.38.2**
- PyTorch version:  **2.2.1**
- Python version: **3.9.6**
- Operating System: **Darwin MacBook-Air-Szymon.local 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:06 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T8103 arm64**

**Steps to reproduce**:
1. Fine-tuned the `google/mt5-small` model on a custom dataset following the standard procedure recommended in the transformers documentation. The dataset consists of simple question-answer pairs without any specific prefixes or format, structured similarly to the intents.json file I used for training.
2. After fine-tuning, when using the model to generate responses to inputs similar to the training data, the model consistently outputs `<extra_id_0>` tokens instead of expected text responses.

my code
train.py
```python
import torch
from functools import partial
from datasets import Dataset, load_dataset, DatasetDict, concatenate_datasets
from transformers import (
    T5ForConditionalGeneration,
    Seq2SeqTrainingArguments,
    Seq2SeqTrainer,
)

from typing import Any, Callable, List, Optional, TypeVar, Union
from transformers import AutoTokenizer
from functools import partialmethod
import numpy as np
from abc import ABC, abstractmethod
import torch.nn.functional as FT

tokenizer = AutoTokenizer.from_pretrained("google/mt5-small")
model = T5ForConditionalGeneration.from_pretrained("google/mt5-small")


K = TypeVar("K")
V = TypeVar("V")
T = TypeVar("T")


class Feature(object):
    """Features represent the schema of a record."""

    pass


class DatasetFeature(Feature):
    """A feature whose type is either known or unknown."""

    def __init__(self, feature_type: str, special_tokens: Optional[List[str]] = None):
        assert feature_type in ("value", "index"), "Invalid feature type."
        self.feature_type = feature_type
        self.special_tokens = [] if special_tokens is None else special_tokens

    def __repr__(self):
        return f"{self.__class__.__name__}(feature_type={'value' if self.feature_type=='value' else 'index'}, special_tokens={self.special_tokens}"


# Załaduj swój plik JSON jako dataset
data_file = "intents.json"
raw_dataset = load_dataset("json", data_files=data_file)


MAX_SOURCE_LENGTH = 64
MAX_TARGET_LENGTH = 16


def preprocess_function(examples):
    input_texts = ["question: " + " ".join(pat) for pat in examples["patterns"]]
    target_texts = ["answer: " + " ".join(resp) for resp in examples["responses"]]

    model_inputs = tokenizer(
        input_texts,
        max_length=MAX_SOURCE_LENGTH,
        padding="max_length",
        truncation=True,
        return_tensors="pt",
    )

    # Prepare target_texts
    with tokenizer.as_target_tokenizer():
        labels = tokenizer(
            target_texts,
            max_length=MAX_TARGET_LENGTH,
            padding="max_length",
            truncation=True,
            return_tensors="pt",
        )["input_ids"]

    model_inputs["labels"] = labels
    return model_inputs


processed_dataset = raw_dataset.map(preprocess_function, batched=True)

train_size = int(len(processed_dataset["train"]) * 0.8)
eval_size = len(processed_dataset["train"]) - train_size

train_dataset = processed_dataset["train"].select(range(train_size))
eval_dataset = processed_dataset["train"].select(
    range(train_size, len(processed_dataset["train"]))
)

print(len(train_dataset), len(eval_dataset))


# Define training arguments
training_args = Seq2SeqTrainingArguments(
    output_dir="./results",
    num_train_epochs=10,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    warmup_steps=1000,
    weight_decay=0.01,
    logging_dir="./logs",
    logging_steps=10,
    save_steps=30,
    evaluation_strategy="epoch",
    learning_rate=5e-5,
    lr_scheduler_type="linear",
)


class Metric(ABC, torch.nn.Module):
    @abstractmethod
    def reset(self):
        raise NotImplementedError

    @abstractmethod
    def update(self, predictions, references):
        raise NotImplementedError

    @abstractmethod
    def compute(self):
        raise NotImplementedError


class ExactMatch(Metric):
    def __init__(self):
        super().__init__()
        self.reset()

    def reset(self):
        self.total, self.correct = 0, 0

    def update(self, predictions, references):
        self.total += len(predictions)
        self.correct += sum(
            [np.array_equal(p, r) for p, r in zip(predictions, references)]
        )

    def compute(self):
        return {"exact_match": float(self.correct / self.total)}


def compute_metrics(pred):
    logits = pred.predictions
    labels = pred.label_ids

    # Jeśli logits jest więcej niż dwuwymiarowy, prawdopodobnie będziesz musiał zmodyfikować ten kod.
    if isinstance(
        logits, tuple
    ):  # Dodatkowe sprawdzenie, na wypadek gdyby logits było krotką
        logits = logits[0]

    print("Logits shape:", logits.shape)
    predictions = np.argmax(logits, axis=-1)

    # Upewnij się, że labels nie wymagają podobnego traktowania.
    # Jeśli labels jest krotką, być może będziesz musiał podobnie je dostosować.

    metric = ExactMatch()
    metric.update(predictions, labels)
    return metric.compute()


# Instantiate the Trainer object
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    compute_metrics=partial(compute_metrics),
)

# Train the model
trainer.train()

model.save_pretrained("./results")
tokenizer.save_pretrained("./results")

```

chat.py
```python
import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM


def main():
    model_name = "google/mt5-small"
    model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

    # Ustawienie urządzenia na CPU
    device = torch.device("cpu")
    model.to(device)

    # Przełączenie modelu w tryb ewaluacji
    model.eval()

    # Załadowanie tokenizera
    tokenizer = AutoTokenizer.from_pretrained(model_name)

    while True:
        user_input = input("You: ")
        if user_input.lower() == "exit":
            break

        # Bez dodawania jakiegokolwiek prefiksu, jeśli model nie był na nich trenowany
        prompt = user_input

        # Tokenizacja wejścia użytkownika
        input_tokens = tokenizer.encode(prompt, return_tensors="pt").to(device)

        # Generowanie odpowiedzi
        generated_response = model.generate(
            input_tokens, max_length=100, num_return_sequences=1
        )

        # Dekodowanie i wyświetlanie odpowiedzi
        output_sequence = tokenizer.decode(
            generated_response[0], skip_special_tokens=True
        )
        print(f"Bot: {output_sequence}")


if __name__ == "__main__":
    main()
```

intents.json looks like this (it's in polish):
```json
[
  {
    "patterns": [
      "cześć",
      "hej",
      "dzień dobry",
      "dzien dobry",
      "hejka",
      "witam",
      "siema",
      "witaj",
      "dobry wieczór",
      "czesc"
    ],
    "responses": [
      "Cześć! Jak mogę Ci dzisiaj pomóc?",
      "Witaj! Czym mogę służyć?",
      "Hej! Jak mogę Ci pomóc?",
      "Dzień dobry! Jakie masz dzisiaj pytanie?"
    ]
  }
...
]
```

### Expected behavior

The model should generate meaningful text responses based on the input, similar to the training question-answer pairs.

Comments:
Hi @calchora, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I am having this issue. I need this project done urgently. Can you please give a solution to this. I am getting exactly the same extra_id_0 error


did anyone end up fixing this problem ? 

I am also having this issue and would love a solution 


> I am also having this issue and would love a solution 
> 

Some suggestions from other posts recommended training with a higher number of epochs (60 if i remember correctly) 
In my case thats not feasible since my fine-tuning dataset is small. 

So i switched to another model, barthez (since my model is processing data and outputting in french) 

Maybe that could help 


thank you @VinDazy! I'll try that