Repository: huggingface/transformers

Issue:
generation length always equal to 20 when using run_seq2seq.py script
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.4.0-197-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten  @LysandreJik 

Model I am using : any seq2seq model

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: cnn-dailymail / xsum
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
python examples/seq2seq/run_seq2seq.py     --model_name_or_path t5-small     --do_train     --do_eval     --task summarization     --dataset_name cnn_dailymail  --dataset_config_name 3.0.0     --output_dir tmp     --per_device_train_batch_size=4     --per_device_eval_batch_size=4     --overwrite_output_dir  --max_steps 250 --eval_steps 249  --save_steps 249  --max_val_samples 250  --max_target_length 100 --val_max_target_length 100 --predict_with_generate
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

```
02/05/2021 16:36:34 - INFO - __main__ -   ***** Eval results *****
02/05/2021 16:36:34 - INFO - __main__ -     epoch = 0.01
02/05/2021 16:36:34 - INFO - __main__ -     eval_gen_len = 19.0
02/05/2021 16:36:34 - INFO - __main__ -     eval_loss = 2.064879894256592
02/05/2021 16:36:34 - INFO - __main__ -     eval_rouge1 = 21.837379907220157
02/05/2021 16:36:34 - INFO - __main__ -     eval_rouge2 = 7.506564948396541
02/05/2021 16:36:34 - INFO - __main__ -     eval_rougeL = 18.074704390199546
02/05/2021 16:36:34 - INFO - __main__ -     eval_rougeLsum = 17.99211046381146
02/05/2021 16:36:34 - INFO - __main__ -     eval_runtime = 19.079
02/05/2021 16:36:34 - INFO - __main__ -     eval_samples_per_second = 13.103
```
`eval_gen_len` is never exceeding 20 for some reason. I tried with several models and datasets 



Discussion:
Hi @moussaKam 

Thank you for reporting this.
Could you try evaluating directly using the generate method and calculate `avg eval_gen_len` ?

Resolving patch:
diff --git a/examples/seq2seq/run_seq2seq.py b/examples/seq2seq/run_seq2seq.py
--- a/examples/seq2seq/run_seq2seq.py
+++ b/examples/seq2seq/run_seq2seq.py
@@ -167,9 +167,22 @@ class DataTrainingArguments:
             "value if set."
         },
     )
+    max_test_samples: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": "For debugging purposes or quicker training, truncate the number of test examples to this "
+            "value if set."
+        },
+    )
     source_lang: Optional[str] = field(default=None, metadata={"help": "Source language id for translation."})
     target_lang: Optional[str] = field(default=None, metadata={"help": "Target language id for translation."})
-    eval_beams: Optional[int] = field(default=None, metadata={"help": "Number of beams to use for evaluation."})
+    num_beams: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": "Number of beams to use for evaluation. This argument will be passed to ``model.generate``, "
+            "which is used during ``evaluate`` and ``predict``."
+        },
+    )
     ignore_pad_token_for_loss: bool = field(
         default=True,
         metadata={
@@ -336,8 +349,13 @@ def main():
     # We need to tokenize inputs and targets.
     if training_args.do_train:
         column_names = datasets["train"].column_names
-    else:
+    elif training_args.do_eval:
         column_names = datasets["validation"].column_names
+    elif training_args.do_predict:
+        column_names = datasets["test"].column_names
+    else:
+        logger.info("There is nothing to do. Please pass `do_train`, `do_eval` and/or `do_predict`.")
+        return
 
     # For translation we set the codes of our source and target languages (only useful for mBART, the others will
     # ignore those attributes).
@@ -440,6 +458,19 @@ def preprocess_function(examples):
             load_from_cache_file=not data_args.overwrite_cache,
         )
 
+    if training_args.do_predict:
+        max_target_length = data_args.val_max_target_length
+        test_dataset = datasets["test"]
+        if data_args.max_test_samples is not None:
+            test_dataset = test_dataset.select(range(data_args.max_test_samples))
+        test_dataset = test_dataset.map(
+            preprocess_function,
+            batched=True,
+            num_proc=data_args.preprocessing_num_workers,
+            remove_columns=column_names,
+            load_from_cache_file=not data_args.overwrite_cache,
+        )
+
     # Data collator
     label_pad_token_id = -100 if data_args.ignore_pad_token_for_loss else tokenizer.pad_token_id
     if data_args.pad_to_max_length:
@@ -523,7 +554,7 @@ def compute_metrics(eval_preds):
     if training_args.do_eval:
         logger.info("*** Evaluate ***")
 
-        results = trainer.evaluate()
+        results = trainer.evaluate(max_length=data_args.val_max_target_length, num_beams=data_args.num_beams)
 
         output_eval_file = os.path.join(training_args.output_dir, "eval_results_seq2seq.txt")
         if trainer.is_world_process_zero():
@@ -533,6 +564,34 @@ def compute_metrics(eval_preds):
                     logger.info(f"  {key} = {value}")
                     writer.write(f"{key} = {value}\n")
 
+    if training_args.do_predict:
+        logger.info("*** Test ***")
+
+        test_results = trainer.predict(
+            test_dataset,
+            metric_key_prefix="test",
+            max_length=data_args.val_max_target_length,
+            num_beams=data_args.num_beams,
+        )
+        test_metrics = test_results.metrics
+
+        output_test_result_file = os.path.join(training_args.output_dir, "test_results_seq2seq.txt")
+        if trainer.is_world_process_zero():
+            with open(output_test_result_file, "w") as writer:
+                logger.info("***** Test results *****")
+                for key, value in sorted(test_metrics.items()):
+                    logger.info(f"  {key} = {value}")
+                    writer.write(f"{key} = {value}\n")
+
+            if training_args.predict_with_generate:
+                test_preds = tokenizer.batch_decode(
+                    test_results.predictions, skip_special_tokens=True, clean_up_tokenization_spaces=True
+                )
+                test_preds = [pred.strip() for pred in test_preds]
+                output_test_preds_file = os.path.join(training_args.output_dir, "test_preds_seq2seq.txt")
+                with open(output_test_preds_file, "w") as writer:
+                    writer.write("\n".join(test_preds))
+
     return results
 
 


Test patch: