KeyError: 'eval_loss'
Author: monk1337
Pull request: no

I am trying to build a Question Answering Pipeline with the Hugginface framework but facing the `KeyError: 'eval_loss'` error. My goal is to train and save the best model at last and evaluate the validation test on the loaded model. My trainer configuration looks like this:



    args = TrainingArguments(f'model_training',
                          evaluation_strategy="epoch",
                          label_names = ["start_positions", "end_positions"],
                          logging_steps = 1,
                          learning_rate=2e-5,
                          num_train_epochs=epochs,
                          save_total_limit = 2,
                          load_best_model_at_end=True,
                          save_strategy="epoch",
                          logging_strategy="epoch",
                          report_to="none",
                          weight_decay=0.01,
                          fp16=True,
                          push_to_hub=False)


While training, getting this error:

    Traceback (most recent call last):
      File "qa_pipe.py", line 286, in <module>
        pipe.training(train_d, val_d, epochs = 2)
      File "qa_pipe.py", line 263, in training
        self.trainer.train()
      File "/home/admin/qa/lib/python3.7/site-packages/transformers/trainer.py", line 1505, in train
        ignore_keys_for_eval=ignore_keys_for_eval,
      File "/home/admin/qa/lib/python3.7/site-packages/transformers/trainer.py", line 1838, in _inner_training_loop
        self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
      File "/home/admin/qa/lib/python3.7/site-packages/transformers/trainer.py", line 2090, in _maybe_log_save_evaluate
        self._save_checkpoint(model, trial, metrics=metrics)
      File "/home/admin/qa/lib/python3.7/site-packages/transformers/trainer.py", line 2193, in _save_checkpoint
        metric_value = metrics[metric_to_check]
    KeyError: 'eval_loss'


The minimal working example is provided on [colab][1]


How to avoid this error and save the best model at last?

  [1]: https://colab.research.google.com/drive/1JNHK8CnMHTm6VMukvDFJq8nvaHhBkxgM?usp=sharing

### Who can help?

@LysandreJik
@sgugger
@patil-suraj

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1JNHK8CnMHTm6VMukvDFJq8nvaHhBkxgM?usp=sharing

### Expected behavior

It should run without the error.

Comments:
It doesn't look like you are providing any labels to your model (your functions preparing the dataset do not generate any `start_positions` and `end_positions`). More generally, please use the [forums](https://discuss.huggingface.co/) for any help to debug your code, as we keep the issues for bugs and feature requests only.

```
from transformers import AutoModelForQuestionAnswering
from transformers import TrainingArguments
from transformers import Trainer
from tqdm.auto import tqdm
from transformers import AutoTokenizer

import numpy as np
import collections
import evaluate
metric = evaluate.load("squad")

from datasets import load_dataset
from transformers import AutoTokenizer





from datasets import load_dataset
    
def sapl(data, n, split):
    data_sampled = data[split].shuffle(seed=42).select(range(n))
    return data_sampled
    
    
from datasets import load_dataset
raw_datasets = load_dataset('squad')
raw_train    = sapl(raw_datasets, 100, 'train') # 100 samples
raw_test     = sapl(raw_datasets, 100, 'validation') # 100 samples



n_best            = 20
max_answer_length = 30
predicted_answers = []


class QA_pipeline(object):
    
    
    def __init__(self, model_name, 
                 device     = 'cuda', 
                 max_length = 512, 
                 stride     = 128):
        
        
        self.model_name = model_name
        self.device     = device
        self.tokenizer  = AutoTokenizer.from_pretrained(self.model_name)
        self.model      = AutoModelForQuestionAnswering.from_pretrained(
            self.model_name).to(self.device)
        
        self.max_length = max_length
        self.stride     = stride
        
        
        
    def _tokenization_train2(self, examples):
        questions = [q.strip() for q in examples["question"]]
        inputs = tokenizer(
            questions,
            examples["context"],
            max_length=self.max_length,
            truncation="only_second",
            stride=self.stride,
            return_overflowing_tokens=True,
            return_offsets_mapping=True,
            padding="max_length",
        )
        
        return inputs
    
    
    
    def _tokenization_train(self, examples):
        questions = [q.strip() for q in examples["question"]]
        inputs = self.tokenizer(
            questions,
            examples["context"],
            max_length=self.max_length,
            truncation="only_second",
            stride=self.stride,
            return_overflowing_tokens=True,
            return_offsets_mapping=True,
            padding="max_length",
        )

        offset_mapping = inputs.pop("offset_mapping")
        sample_map = inputs.pop("overflow_to_sample_mapping")
        answers = examples["answers"]
        start_positions = []
        end_positions = []

        for i, offset in enumerate(offset_mapping):
            sample_idx = sample_map[i]
            answer = answers[sample_idx]
            start_char = answer["answer_start"][0]
            end_char = answer["answer_start"][0] + len(answer["text"][0])
            sequence_ids = inputs.sequence_ids(i)

            # Find the start and end of the context
            idx = 0
            while sequence_ids[idx] != 1:
                idx += 1
            context_start = idx
            while sequence_ids[idx] == 1:
                idx += 1
            context_end = idx - 1

            # If the answer is not fully inside the context, label is (0, 0)
            if offset[context_start][0] > start_char or offset[context_end][1] < end_char:
                start_positions.append(0)
                end_positions.append(0)
            else:
                # Otherwise it's the start and end token positions
                idx = context_start
                while idx <= context_end and offset[idx][0] <= start_char:
                    idx += 1
                start_positions.append(idx - 1)

                idx = context_end
                while idx >= context_start and offset[idx][1] >= end_char:
                    idx -= 1
                end_positions.append(idx + 1)

        inputs["start_positions"] = start_positions
        inputs["end_positions"] = end_positions
        return inputs
    
    
    def _tokenization_validation(self, examples):
        questions = [q.strip() for q in examples["question"]]
        inputs = self.tokenizer(
            questions,
            examples["context"],
            max_length=self.max_length,
            truncation="only_second",
            stride=self.stride,
            return_overflowing_tokens=True,
            return_offsets_mapping=True,
            padding="max_length",
        )

        sample_map = inputs.pop("overflow_to_sample_mapping")
        example_ids = []

        for i in range(len(inputs["input_ids"])):
            sample_idx = sample_map[i]
            example_ids.append(examples["id"][sample_idx])

            sequence_ids = inputs.sequence_ids(i)
            offset = inputs["offset_mapping"][i]
            inputs["offset_mapping"][i] = [
                o if sequence_ids[k] == 1 else None for k, o in enumerate(offset)
            ]

        inputs["example_id"] = example_ids
        return inputs
    
    
    
    def get_train_dataset(self, train_dataset):
        train_dataset = train_dataset.map(self._tokenization_train,
                                          batched=True,
                                          remove_columns=train_dataset.column_names,)
        
        print(len(train_dataset), len(train_dataset))
        return train_dataset
    
    
    
    def get_val_dataset(self, val_dataset):
        
        validation_dataset = val_dataset.map(
            self._tokenization_validation,
            batched=True,
            remove_columns=val_dataset.column_names,)
        
        print(len(val_dataset), len(validation_dataset))
        return validation_dataset
        
                
            
    def compute_metrics_eval(self, eval_pred):
        print("it is working")
        print(eval_pred)



    def compute_metrics(self, start_logits, end_logits, features, examples):
        example_to_features = collections.defaultdict(list)
        for idx, feature in enumerate(features):
            example_to_features[feature["example_id"]].append(idx)

        predicted_answers = []
        for example in tqdm(examples):
            example_id = example["id"]
            context = example["context"]
            answers = []

            # Loop through all features associated with that example
            for feature_index in example_to_features[example_id]:
                start_logit = start_logits[feature_index]
                end_logit = end_logits[feature_index]
                offsets = features[feature_index]["offset_mapping"]

                start_indexes = np.argsort(start_logit)[-1 : -n_best - 1 : -1].tolist()
                end_indexes = np.argsort(end_logit)[-1 : -n_best - 1 : -1].tolist()
                for start_index in start_indexes:
                    for end_index in end_indexes:
                        # Skip answers that are not fully in the context
                        if offsets[start_index] is None or offsets[end_index] is None:
                            continue
                        # Skip answers with a length that is either < 0 or > max_answer_length
                        if (
                            end_index < start_index
                            or end_index - start_index + 1 > max_answer_length
                        ):
                            continue

                        answer = {
                            "text": context[offsets[start_index][0] : offsets[end_index][1]],
                            "logit_score": start_logit[start_index] + end_logit[end_index],
                        }
                        answers.append(answer)

            # Select the answer with the best score
            if len(answers) > 0:
                best_answer = max(answers, key=lambda x: x["logit_score"])
                predicted_answers.append(
                    {"id": example_id, "prediction_text": best_answer["text"]}
                )
            else:
                predicted_answers.append({"id": example_id, "prediction_text": ""})

        theoretical_answers = [{"id": ex["id"], "answers": ex["answers"]} for ex in examples]
        return metric.compute(predictions=predicted_answers, references=theoretical_answers)

    
    
    def training(self, train_dataset, val_dataset, epochs = 2):
        
        self.args = TrainingArguments(f'{self.model_name}_training',
                              logging_steps = 1,
                              learning_rate=2e-5,
                              num_train_epochs=epochs,
                                      
                                      
                              save_total_limit = 2,
                              save_strategy = "epoch",
                              load_best_model_at_end=True,
                              
                              evaluation_strategy = "epoch", #To calculate metrics per epoch
                              logging_strategy="epoch", #Extra: to log training data stats for loss 
                              weight_decay=0.01,
                              fp16=True,
                              push_to_hub=False)
        
        self.trainer = Trainer(model = self.model,
                          args  = self.args,
                 compute_metrics=self.compute_metrics_eval,
        train_dataset           = train_dataset,
        eval_dataset            = val_dataset,
        tokenizer               = self.tokenizer,)
        self.trainer.train()
        self.trainer.save_model()
        
    
    def validation(self, val_dataset, raw_val_dataset):
        
        self.trainer       = Trainer(model=self.model)
        self.trainer.model = self.model.cuda()

        predictions, _, _ = self.trainer.predict(val_dataset)
        start_logits, end_logits = predictions
        output  = self.compute_metrics(start_logits, end_logits, val_dataset, raw_val_dataset)
        return output
```
    
```
pipe = QA_pipeline("emilyalsentzer/Bio_ClinicalBERT", device = 'cuda:0')
train_d   = pipe.get_train_dataset(raw_train)
val_d     = pipe.get_val_dataset(raw_test)

print(pipe.validation(val_d, raw_test))
pipe.training(train_d, val_d, epochs = 5)
```

That is my code, while `load_best_model_at_end=True` it's giving error 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@monk1337 It is actually very helpful just to know that this was initiated by using `load_best_model_at_end=True`!  I was struggling with the same error.  Using `load_best_model_at_end=False` solved it for me.

However, this shouldn't be necessary.  @sgugger this is a bug.

Just curious, I want to load the best model at the end, so I leave `load_best_model_at_end=True` and I want the best model at the end to be evaluated by `eval_loss`. My trainer already computes a train/loss value (as sseen on my wandb logger, so I'm curious why there's no eval/loss, given that I've provided a eval dataset. 

> Just curious, I want to load the best model at the end, so I leave `load_best_model_at_end=True` and I want the best model at the end to be evaluated by `eval_loss`. My trainer already computes a train/loss value (as sseen on my wandb logger, so I'm curious why there's no eval/loss, given that I've provided a eval dataset.

I have exactly the same question, any solution for this? 

I vaguely remember being annoyed by this, but I have no solution. :(

On Tue, Apr 16, 2024 at 7:34 AM Abhirama Subramanyam <
***@***.***> wrote:

> Just curious, I want to load the best model at the end, so I leave
> load_best_model_at_end=True and I want the best model at the end to be
> evaluated by eval_loss. My trainer already computes a train/loss value
> (as sseen on my wandb logger, so I'm curious why there's no eval/loss,
> given that I've provided a eval dataset.
>
> I have exactly the same question, any solution for this?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/19957#issuecomment-2058876863>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A5OJQKFSXYLHYHK3OHXNYUDY5UED7AVCNFSM6AAAAAARRLF2UGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJYHA3TMOBWGM>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


Btw, I fixed it, inside evaluate code, I had set ```return_loss = True```
For anyone who is facing this error, might try this fix.

Still an issue here :(

transformer 4.47.0
Still an issue here :(