Repository: huggingface/transformers (Python)
Date: 2024-05-19 13:18:57+00:00
Task type: Bug Fix

Issue:
transformers 4.41.0 breaks generate() for T5
### System Info

- `transformers` version: 4.41.0
- Platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
- Python version: 3.10.9
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.3
- Accelerate version: 0.30.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

 @ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The following code breaks in `v4.41.0` (it works on earlier versions).

```py
import torch
from transformers import GenerationConfig
from transformers import T5ForConditionalGeneration

model = T5ForConditionalGeneration.from_pretrained(
    "google/t5-efficient-tiny", device_map="cuda"
)
input_ids = torch.tensor([[4, 5, 6, 6, 7]], device="cuda")
model.generate(
    input_ids=input_ids,
    generation_config=GenerationConfig(do_sample=True),
)
```

Error: 
```
ValueError: `decoder_start_token_id` or `bos_token_id` has to be defined for encoder-decoder generation.
```

### Expected behavior

Expected generate to work like before without manually specifying `decoder_start_token_id` or `bos_token_id` in the `GenerationConfig`.


Issue discussion:

Solution patch:
diff --git a/src/transformers/generation/utils.py b/src/transformers/generation/utils.py
--- a/src/transformers/generation/utils.py
+++ b/src/transformers/generation/utils.py
@@ -1354,6 +1354,23 @@ def _get_static_cache(self, max_batch_size: int, max_cache_len: int) -> StaticCa
             self._static_cache.reset()  # reset the cache for a new generation
         return self._static_cache
 
+    def _get_decoder_start_token_id(
+        self, decoder_start_token_id: Union[int, List[int]] = None, bos_token_id: int = None
+    ) -> int:
+        decoder_start_token_id = (
+            decoder_start_token_id
+            if decoder_start_token_id is not None
+            else self.generation_config.decoder_start_token_id
+        )
+        bos_token_id = bos_token_id if bos_token_id is not None else self.generation_config.bos_token_id
+
+        if decoder_start_token_id is not None:
+            return decoder_start_token_id
+        elif bos_token_id is not None:
+            return bos_token_id
+        else:
+            return
+
     def _prepare_special_tokens(
         self,
         generation_config: GenerationConfig,
@@ -1378,11 +1395,16 @@ def _tensor_or_none(token, device=None):
                 return token
             return torch.tensor(token, device=device, dtype=torch.long)
 
+        # for BC we also try to get `decoder_start_token_id` from model's generation config (#30892)
+        if self.config.is_encoder_decoder:
+            generation_config.decoder_start_token_id = self._get_decoder_start_token_id(
+                generation_config.decoder_start_token_id, generation_config.bos_token_id
+            )
+
         bos_token_id = _tensor_or_none(generation_config.bos_token_id, device=device)
         eos_token_id = _tensor_or_none(generation_config.eos_token_id, device=device)
         pad_token_id = _tensor_or_none(generation_config.pad_token_id, device=device)
         decoder_start_token_id = _tensor_or_none(generation_config.decoder_start_token_id, device=device)
-        decoder_start_token_id = decoder_start_token_id if decoder_start_token_id is not None else bos_token_id
 
         # We can have more than one eos token. Always treat it as a 1D tensor (when it exists).
         if eos_token_id is not None and eos_token_id.ndim == 0:


Test patch:
diff --git a/tests/generation/test_utils.py b/tests/generation/test_utils.py
--- a/tests/generation/test_utils.py
+++ b/tests/generation/test_utils.py
@@ -65,6 +65,7 @@
         GenerateBeamEncoderDecoderOutput,
         GenerateDecoderOnlyOutput,
         GenerateEncoderDecoderOutput,
+        GenerationConfig,
         GreedySearchDecoderOnlyOutput,
         GreedySearchEncoderDecoderOutput,
         LogitsProcessorList,
@@ -2478,6 +2479,35 @@ def test_batched_decoder_start_id(self):
 
         self.assertListEqual(outputs.tolist(), outputs_batched_ids.tolist())
 
+    def test_decoder_start_id_from_config(self):
+        # Refer to: (#30899)
+        articles = [
+            "Justin Timberlake and Jessica Biel, welcome to parenthood.",
+            "Michael Phelps is arguably the most decorated Olympian of all time.",
+        ]
+        bart_tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/tiny-random-bart")
+        bart_model = BartForConditionalGeneration.from_pretrained("hf-internal-testing/tiny-random-bart").to(
+            torch_device
+        )
+        input_ids = bart_tokenizer(articles, return_tensors="pt", padding=True).input_ids.to(torch_device)
+        decoder_start_token_id = bart_model.generation_config.decoder_start_token_id
+
+        # we should be able to take `decoder_start_token_id` from model's generation config if user passes a `GenerationConfig` type
+        outputs = bart_model.generate(input_ids, generation_config=GenerationConfig(do_sample=False))
+
+        # If the generatoin config has no `decoder_start_token_id` or `bos_token_id`, we will raise an error unless user passes it in config
+        bart_model.generation_config.decoder_start_token_id = None
+        bart_model.generation_config.bos_token_id = None
+        outputs_with_user_id = bart_model.generate(
+            input_ids,
+            generation_config=GenerationConfig(do_sample=False, decoder_start_token_id=decoder_start_token_id),
+        )
+
+        self.assertListEqual(outputs.tolist(), outputs_with_user_id.tolist())
+
+        with self.assertRaises(ValueError):
+            outputs = bart_model.generate(input_ids, generation_config=GenerationConfig(do_sample=False))
+
     def test_contrastive_search_batched(self):
         # PT-only test: TF doesn't have constrained beam search
         # Tests that contrastive search works with batched inputs (i.e. has the same output as for non-batched inputs)