Trainer bug > using dict (with single or multiple datasets) for eval_dataset arg throws error
Author: lenn-arts
Pull request: no

### System Info

Using python 3.10, transformers 4.28.1

Dear team @sgugger ,

When I tried passing multiple datasets to the eval_dataset argument of the trainer using a dict structure as per the documentation, I experienced that only metrics on the first of the passed datasets get compute at the evaluation step in the training process. For the subsequent dataset, I receive the following error:

Code:
```
eval_datasets = {"test": datasets["test"].shuffle(seed=SEED),
                     "val": datasets["val"].shuffle(seed=SEED), }
trainer = Trainer("model":model,
      "args":t_args,
      "train_dataset":datasets["train"],
      "eval_dataset":eval_datasets, 
      "compute_metrics": metric_fn)
```

Error:

```
Traceback (most recent call last):
...
  File "src/code_main.py", line 767, in run_experiment_
    trainer.train(resume_from_checkpoint=resume)
  File "lib/python3.10/site-packages/transformers/trainer.py", line 1662, in train
    return inner_training_loop(
  File "lib/python3.10/site-packages/transformers/trainer.py", line 2006, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "lib/python3.10/site-packages/transformers/trainer.py", line 2285, in _maybe_log_save_evaluate
    metrics.update(dataset_metrics)
TypeError: 'NoneType' object is not iterable
```

Notably, even if I only pass one dataset in the form of dictionary as opposed to directly passing its dataset instance, the same error appears, without any metrics being reported back.

I triple checked that all the datasets I pass are correct dataset instances with >0 samples.

Any suggestions?
Thanks!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

eval_datasets = {"test": datasets["test"].shuffle(seed=SEED),
                     "val": datasets["val"].shuffle(seed=SEED), }
trainer = Trainer("model":model,
      "args":t_args,
      "train_dataset":datasets["train"],
      "eval_dataset":eval_datasets, 
      "compute_metrics": metric_fn)

### Expected behavior

Metric function runs on both dataset instance values of the passed dict, one after the other. All metrics with the according prefixes are reported back. No error is thrown.

Comments:
It looks like one of your datasets might not have labels, so the metrics returned are None, maybe?

Thanks for the prompt answer! I don't think this is the problem for two reasons:
1st) running 
```
eval_datasets["val"].unique("labels")
eval_datasets["test"].unique("labels")
```
returns `[0,1]` for both eval datasets.

2nd) When passing the datasets directly as dataset instances, I get metrics that are fine, so it seems it has to do with the fact of them together or individually being passed as dict type.

Could you share a small reproducer of the issue I can execute?

Hi @sgugger, please find a minimum version of the code below.

```python
# by Lennart Schulze, July 2023

############## SETUP
import random
import numpy as np
import os

import torch
import torch.nn as nn
import torch.optim

CACHE_ROOT = "/.cache/hf/"

os.environ['TRANSFORMERS_CACHE'] = CACHE_ROOT
os.environ['HF_DATASETS_CACHE'] = CACHE_ROOT
import transformers as hft
import datasets as hfds
import evaluate as hfev

g = torch.Generator()
SEED = 3
def seed(seed_num=SEED):
  torch.manual_seed(seed_num)
  random.seed(seed_num)
  np.random.seed(seed_num)
  g.manual_seed(seed_num)
seed()
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print(DEVICE)

############## MODEL & TOKENIZER
def get_model():
  model_name = "microsoft/deberta-v3-large"

  tokenizer = hft.AutoTokenizer.from_pretrained(model_name)
  print(tokenizer)

  num_labels = 1
  model = hft.DebertaV2ForSequenceClassification.from_pretrained(model_name, num_labels=num_labels)
  print(type(model))
  print(model.config.id2label)
  print(model)
  return model, tokenizer

############## DATASET

def get_datasets():  
  cache_dir = os.path.join(CACHE_ROOT)
  datasets = hfds.load_dataset("rotten_tomatoes", cache_dir=cache_dir)
  datasets["test"] = hfds.load_dataset("rotten_tomatoes", split="test", cache_dir=cache_dir)
  
  print(datasets)
  print(type(datasets))
  print(datasets.keys())
  for key, ds in datasets.items():
      print(key, ":", len(ds))
  return datasets

# tokenize dataset & prepare for torch transformer
def tokenize_function(tokenizer, sample):
  return tokenizer(sample["text"], padding="max_length", truncation=True,
                   return_tensors="pt", max_length=128)

def preprocess_datasets(tokenizer, datasets):
  tokenize_fn = lambda sample: tokenize_function(tokenizer, sample)
  tokenized_datasets = datasets.map(tokenize_fn, batched=True) 

  tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
  new_features = tokenized_datasets["train"].features.copy()
  new_features["labels"] = hfds.ClassLabel(num_classes=2, names=["No", "Yes"])
  tokenized_datasets = tokenized_datasets.cast(new_features)
  tokenized_datasets = tokenized_datasets.with_format("torch")

  print("preprocess_datasets > train: \t", tokenized_datasets["train"].features)
  print("preprocess_datasets > val: \t", tokenized_datasets["validation"].features)
  print("preprocess_datasets > test: \t", tokenized_datasets["test"].features)
  print("preprocess_datasets > val:", tokenized_datasets["validation"][0])
  print("preprocess_datasets > test:", tokenized_datasets["test"][0])

  return tokenized_datasets

############## TRAINING

# metric for trainer
def get_metric_fn():
  metric = hfev.load("accuracy", cache_dir=CACHE_ROOT)
  def compute_metrics(eval_prediction):
    out = {}
    logits, gt_labels = eval_prediction
    predicted_labels = (torch.sigmoid(torch.tensor(logits))>=0.5).float().numpy()
    key = 0
    out[f"ACC_{key}"] = metric.compute(predictions=predicted_labels, references=gt_labels)["accuracy"]
    return out
  return compute_metrics


class CustomTrainer(hft.Trainer):
  def __init__(self, **kwargs):
    super().__init__(**kwargs)

  def evaluate(self, **kwargs):
    print("\n> EVALUATING >\n")
    super().evaluate(**kwargs)
    print("\n< EVALUATING DONE <\n")

  def compute_loss(self, model, inputs, return_outputs=False):
    inputs["labels"] = inputs["labels"].float()
    labels = inputs.get("labels")

    outputs = model(**inputs)
    logits = outputs.get("logits")
    threshold = 0.5
    preds = (torch.sigmoid(logits) >= threshold).float()

    loss_kwargs = {}
    loss_fn = nn.BCEWithLogitsLoss(**loss_kwargs)
    loss = loss_fn(logits.view(-1, 1), labels.view(-1, 1))
    
    return (loss, outputs) if return_outputs==True else loss


def get_trainer(t_args, model, datasets, 
                eval_ds="test", train_size=5000, eval_size=50):
  metric_fn = get_metric_fn()

  sample_size_train = train_size if train_size!="all" else len(datasets["train"])
  if eval_size!="all": 
    sample_size_eval_val = eval_size
    sample_size_eval_test = sample_size_eval_val
  else:
    sample_size_eval_val = len(datasets["validation"]) 
    sample_size_eval_test = len(datasets["test"])
  
  eval_datasets = {}
  if eval_ds!=None:
    eval_datasets = {"test": datasets["test"].shuffle(seed=SEED).select(range(sample_size_eval_test)),
                     "val": datasets["validation"].shuffle(seed=SEED).select(range(sample_size_eval_val)), 
                  }
    print("get trainer > eval_datasets val ",eval_datasets["val"][0])
    print("get trainer > eval_datasets test",eval_datasets["test"][0])
    print("get trainer > eval_datasets val ",eval_datasets["val"].unique("labels"))
    print("get trainer > eval_datasets test",eval_datasets["test"].unique("labels"))
    if eval_ds=="val":
      ###del eval_datasets["test"] # this won't work!
      eval_datasets = eval_datasets["val"] # this will work!
    elif eval_ds=="test":
      del eval_datasets["val"] # this won't work!
      #eval_datasets = eval_datasets["test"] # this will work!
  
    trainer_constructor_kwargs = {
      "model":model,
      "args":t_args,
      "train_dataset":datasets["train"].shuffle(seed=SEED).select(range(sample_size_train)),
      "eval_dataset":eval_datasets, 
      "compute_metrics": metric_fn
    }
    trainer = CustomTrainer(
        **trainer_constructor_kwargs,
    )
  return trainer

# training arguments for trainer
def get_trainer_args():
  output_dirs_root = os.path.join(os.path.dirname(os.path.abspath(__file__)), "../")
  output_dir = "chkpts"

  t_args = hft.TrainingArguments(
      output_dir=os.path.join(output_dirs_root, output_dir),
      evaluation_strategy="steps",
      logging_strategy="steps",
      per_device_train_batch_size=4, 
      per_device_eval_batch_size=4,
      gradient_accumulation_steps=1,
      eval_steps=1,
      logging_steps=1,
      num_train_epochs=1,
      learning_rate=1e-6,
      save_steps=1000,
      report_to="tensorboard",
  )
  return t_args

############## AUTOMATION

def run_experiment_(model=None, tokenizer=None, datasets=None):
  t_args = get_trainer_args()
  trainer = get_trainer(t_args, model, datasets)
  trainer.train()

def start_notebook():
  model, tokenizer = get_model()

  datasets = get_datasets()
  datasets = preprocess_datasets(tokenizer, datasets)

  return model, tokenizer, datasets


############## EXPERIMENTS
if __name__=="__main__":
    print(torch.version.cuda)
    print(torch.cuda.is_available())

    model, tokenizer, datasets = start_notebook()    
    run_experiment_(model=model, datasets=datasets)
``` 

The key is the `get_trainer` function, where the key argument is `eval_ds` (try "all"/"test"/"val").

Please let me know in case of questions.

In your code sample, your orverloaded `evaluate` function does not return anything, this is what causes the problem.
```py 
def evaluate(self, **kwargs):
    print("\n> EVALUATING >\n")
    result = super().evaluate(**kwargs)
    print("\n< EVALUATING DONE <\n")
    return result
``` 
this the issue on my side.

This works, thank you very much! It did not cross my mind that the results need to be returned at this level because they were still printed.