Using PatchTST for self supervised pretraining. eval_loss key error even when the metric to load best model is "loss"
Author: nikhilajoshy
Pull request: no

### System Info

- `transformers` version: 4.37.2
- Platform: Linux-5.15.0-101-generic-x86_64-with-glibc2.31
- Python version: 3.11.8
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I want to pretrain PatchTST for self supervised learning where the model is trained to predict the time series data itself. But not able to pretrain because of "eval_loss" key error.



    `# Initialize model
      config = PatchTSTConfig(
          num_input_channels=len(forecast_columns),
          context_length=context_length,
          patch_length=patch_length,
          stride=patch_length,
          mask_type="random",
          random_mask_ratio=0.4,
          use_cls_token=True,
      )
      model = PatchTSTForPretraining(config)
      training_args = TrainingArguments(
          output_dir="encoder_models/",
          overwrite_output_dir=True,
          learning_rate=0.001,
          num_train_epochs=10,
          do_eval=True,
          do_train=True,
          evaluation_strategy="epoch",
          per_device_train_batch_size=batch_size,
          per_device_eval_batch_size=batch_size,
          dataloader_num_workers=1,
          save_strategy="epoch",
          logging_strategy="epoch",
          save_total_limit=2,
          logging_dir="encoder_models/",  # Make sure to specify a logging directory
          load_best_model_at_end=True,  # Load the best model when training ends
          metric_for_best_model="loss",  # Metric to monitor for early stopping
          greater_is_better=False,  # For loss
      )

    # Create the early stopping callback
    early_stopping_callback = EarlyStoppingCallback(
        early_stopping_patience=10,  # Number of epochs with no improvement after which to stop
        early_stopping_threshold=0.0001,  # Minimum improvement required to consider as improvement
    )

    # define trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=valid_dataset,
        callbacks=[early_stopping_callback],
    )

    # pretrain
    trainer.train()`

### Expected behavior

PatchTST for pretraining runs and stops on early callback depending on the value of loss.

Comments:
Hey! Could you try to update to the latest `transformers` release? otherwise cc @kashif 

@ArthurZucker I tried updating the transformers library but getting the same error. 
@kashif 

can you share the full traceback ? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi I am Facing the exact same issue, @nikhilajoshy could you solve it ? 

I found a trick to make it work but can cause other issues: 

refer to: https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524

added eval_output.metrics["eval_loss"] = "No log" above on evaluate at the train.py fiel and worked... i am doing some tests to see what else that breaks

I also live same issue.. it does not even enter to compute_metrics.

`    X_train, X_val, X_test = get_input_datasets(
        uri=input_uri,
        npy_files=['lb_train', 'lb_val', 'lb_test'])

    actual_train = X_train
    actual_val = X_val
    actual_test = X_test

    ds_train = TimeSeriesDataset(X_train, actual_train)
    ds_val = TimeSeriesDataset(X_val, actual_val)
    ds_test = TimeSeriesDataset(X_test, actual_test)

    config = PatchTSTConfig(
        num_input_channels=X_train.shape[-1],
        context_length=X_train.shape[1],
        patch_length=patch_length,
        prediction_length=actual_train.shape[1],
        random_mask_ratio=msk_rate,
        d_model=d_model,
        num_attention_heads=num_attention_heads,
        num_hidden_layers=num_hidden_layers,
        ffn_dim=ffn_dim,
        dropout=dropout,
        head_dropout=dropout,
        pooling_type=None,
        channel_attention=False,
        scaling=None,
        loss='mse')

    model = PatchTSTForPretraining(config)

    # Early stopping callback to stop training
    # when validation loss doesn't improve
    early_stopping_callback = EarlyStoppingCallback(
        early_stopping_patience=patience)

    training_args = TrainingArguments(
        output_dir=os.path.join('tmp'),
        overwrite_output_dir=True,
        learning_rate=learning_rate,
        num_train_epochs=num_train_epochs,
        do_eval=True,
        evaluation_strategy='epoch',
        per_device_train_batch_size=batch_size,
        per_device_eval_batch_size=batch_size,
        dataloader_num_workers=num_workers,
        save_strategy='epoch',
        logging_strategy='epoch',
        save_total_limit=3,
        logging_dir=None,
        load_best_model_at_end=True,
        metric_for_best_model='eval_loss',
        greater_is_better=False)

    # define trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=ds_train,
        eval_dataset=ds_val,
        callbacks=[early_stopping_callback],
        compute_metrics=compute_metrics)

    trainer.train()`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.