Repository: huggingface/transformers

Issue:
parameter `ignore_keys` of `trainer.predict` not accessible in `Trainer` or `TrainingArguments`
# 🚀 Feature request

The [`predict`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.Trainer.predict) and [`evaluate`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.Trainer.evaluate) methods of the Trainer class provide an excellent option of `ignore_keys`. Here is a small example:

```python
trainer.predict(dataset, ignore_keys=["ner_loss", "cls_loss", "ner_logits", "cls_logits"])
```

@sgugger 

This option is however, not accessible during the normal setup of defining `TrainingArguments` class nor the `Trainer` class so the a call to `trainer.train()` leads to errors during the mid-training evaluation.

## Motivation

I am unable to evaluate the model metrics on the validation set **during** the training to see if makes sense to continue.

## Your contribution

I am happy to make a PR if this is seen as a genuine problem. Like always, may be I am missing something.



Discussion:
We could add an argument for this (like `ignore_keys_for_eval`) yes. Let me know if you want to tackle this!
I very much wanna do this and will get right down to it!  👍 Admittedly though, I am relatively new to making contributions.
Hi! As a short update: I will not be able to work on this until 23rd of June... so if anyone wants to pick it up good otherwise I need 3 more weeks.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
OK so I'd like to work on this :)
Sure! Ping me when you open a PR :-)

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -989,6 +989,7 @@ def train(
         self,
         resume_from_checkpoint: Optional[Union[str, bool]] = None,
         trial: Union["optuna.Trial", Dict[str, Any]] = None,
+        ignore_keys_for_eval: Optional[List[str]] = None,
         **kwargs,
     ):
         """
@@ -1002,6 +1003,9 @@ def train(
                 training will resume from the model/optimizer/scheduler states loaded here.
             trial (:obj:`optuna.Trial` or :obj:`Dict[str, Any]`, `optional`):
                 The trial run or the hyperparameter dictionary for hyperparameter search.
+            ignore_keys_for_eval (:obj:`List[str]`, `optional`)
+                A list of keys in the output of your model (if it is a dictionary) that should be ignored when
+                gathering predictions for evaluation during the training.
             kwargs:
                 Additional keyword arguments used to hide deprecated arguments
         """
@@ -1322,13 +1326,13 @@ def train(
                     self.state.epoch = epoch + (step + 1) / steps_in_epoch
                     self.control = self.callback_handler.on_step_end(args, self.state, self.control)
 
-                    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
+                    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
 
                 if self.control.should_epoch_stop or self.control.should_training_stop:
                     break
 
             self.control = self.callback_handler.on_epoch_end(args, self.state, self.control)
-            self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
+            self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
 
             if DebugOption.TPU_METRICS_DEBUG in self.args.debug:
                 if is_torch_tpu_available():
@@ -1405,7 +1409,7 @@ def _load_state_dict_in_model(self, state_dict):
         if len(load_result.unexpected_keys) != 0:
             logger.warn(f"There were unexpected keys in the checkpoint model loaded: {load_result.unexpected_keys}.")
 
-    def _maybe_log_save_evaluate(self, tr_loss, model, trial, epoch):
+    def _maybe_log_save_evaluate(self, tr_loss, model, trial, epoch, ignore_keys_for_eval):
         if self.control.should_log:
             logs: Dict[str, float] = {}
             tr_loss_scalar = tr_loss.item()
@@ -1423,7 +1427,7 @@ def _maybe_log_save_evaluate(self, tr_loss, model, trial, epoch):
 
         metrics = None
         if self.control.should_evaluate:
-            metrics = self.evaluate()
+            metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
             self._report_to_hp_search(trial, epoch, metrics)
 
         if self.control.should_save:


Test patch: