My customized function compute_metrics doesn't work when i train the CLIP model
Author: lchustc
Pull request: no

### System Info

**environment**
transformers==4.17.0

**hopes**
I would like to know how to output evaluation metrics when i train the CLIP? I want to get `eval_loss` the way I get `train_loss`.
Could anyone help me?

**details**
I have trained the CLIP demo(with the original training script `run_clip.py`： https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text) successfully and obtained the following logs：
<img width="1243" alt="image" src="https://user-images.githubusercontent.com/27990344/196113496-3582dd41-32bf-47c5-939c-289b5d8fcef6.png">
There seems to be no output evaluation metrics(such as `loss` or `acc`) after i specify `--do_eval ` before training. So i specified `compute_metrics=compute_metrics` in `Trainer` and got errors when the CLIP do evaluation
<img width="737" alt="image" src="https://user-images.githubusercontent.com/27990344/196118789-3587ea57-1654-4075-a7ef-33f43505f15b.png">

<img width="1238" alt="image" src="https://user-images.githubusercontent.com/27990344/196117202-106ee438-254e-492f-a6ea-c332d9a4b7b4.png">

Except the above errors, i also checked the src code(`line 3052`, https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py) and found that my customized function `compute_metrics` doesn't work when `all_labels is None`
<img width="855" alt="image" src="https://user-images.githubusercontent.com/27990344/196119730-ab1ae85d-324b-4580-bf64-ff322ee2a11e.png">



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run original training script `run_clip.py` https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text

### Expected behavior

I would like to know how to output evaluation metrics when i train the CLIP? I want to get `eval_loss` the way I get `train_loss`.

Comments:
@ydshieh
Can you take a moment to help out? Thx a lot！

Hi @lchwhut . Thank you for reporting. I will take a look

Hi @lchwhut 

Could you check what you get as `all_preds` here
https://github.com/huggingface/transformers/blob/b17a5e00749790895314ea33a4f156c918718dfe/src/transformers/trainer.py#L3071

Does it contain the loss value?

Well, a second look, I think you can comment out this block

```python
        # Metrics!
        if self.compute_metrics is not None and all_preds is not None and all_labels is not None:
            if args.include_inputs_for_metrics:
                metrics = self.compute_metrics(
                    EvalPrediction(predictions=all_preds, label_ids=all_labels, inputs=all_inputs)
                )
            else:
                metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
        else:
            metrics = {}
```
and put `metrics = {}` before the line `metrics = denumpify_detensorize(metrics)`.

Please let me know if this helps, thank you!

> Well, a second look, I think you can comment out this block
> 
> ```python
>         # Metrics!
>         if self.compute_metrics is not None and all_preds is not None and all_labels is not None:
>             if args.include_inputs_for_metrics:
>                 metrics = self.compute_metrics(
>                     EvalPrediction(predictions=all_preds, label_ids=all_labels, inputs=all_inputs)
>                 )
>             else:
>                 metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
>         else:
>             metrics = {}
> ```
> 
> and put `metrics = {}` before the line `metrics = denumpify_detensorize(metrics)`.
> 
> Please let me know if this helps, thank you!

Commenting out this block and putting `metrics = {}` before the line `metrics = denumpify_detensorize(metrics)` doesn't work. It will report error like this:
![image](https://user-images.githubusercontent.com/27990344/196848990-107c81db-525a-4f6e-985e-f065907dbab1.png)

Actually, this way is equivalent to executing the `else` module because `all_labels ` is None. 
I checked the src code 
`loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)`
This step seems to return (None, logits, None) when the model is CLIP,because in this function, `has_labels=False`.


Hi @lchwhut , could you check if the `inputs` at the line you mentioned have `"return_loss": True`, just like
https://github.com/huggingface/transformers/blob/bbe2c8b126b04a3250e7089cf507ec62ce8d716c/examples/pytorch/contrastive-image-text/run_clip.py#L215
If it exists there and being `True`, the model itself should be able to return a loss value. We then have to check if `has_labels=False` will play a role not to return it back.

> Hi @lchwhut , could you check if the `inputs` at the line you mentioned have `"return_loss": True`, just like
> 
> https://github.com/huggingface/transformers/blob/bbe2c8b126b04a3250e7089cf507ec62ce8d716c/examples/pytorch/contrastive-image-text/run_clip.py#L215
> 
> 
> If it exists there and being `True`, the model itself should be able to return a loss value. We then have to check if `has_labels=False` will play a role not to return it back.

Hi @ydshieh, i think the `inputs` does have `"return_loss": True` because the `inputs` is generated by dataloader which conducts function `collate_fn`.
<img width="940" alt="image" src="https://user-images.githubusercontent.com/27990344/197094616-779776fd-2ade-4ce2-95b5-5d9ec0e5d273.png">
`loss = None` will be specified if `have_label == False`
<img width="833" alt="image" src="https://user-images.githubusercontent.com/27990344/197096010-32dac0c7-57d9-4fd2-8232-146ecee848dd.png">
The `logits` returned by `self.prediction_step` is the output of the CLIP which like:
<img width="381" alt="image" src="https://user-images.githubusercontent.com/27990344/197096335-d91dfc31-8f61-41b1-8432-1b34de556cad.png">
The type of `text_model_output` and `vision_model_output` is not Tensor which causes(i think) the following error:
<img width="612" alt="image" src="https://user-images.githubusercontent.com/27990344/197096740-eb542f27-6894-4e30-962a-e0fd661fd768.png">


OK @lchwhut Thank you for all of these checks. I will figure it out a way.

Hi @lchwhut 

Since the commit [3951b9f39](https://github.com/huggingface/transformers/commit/3951b9f3908bfa30be7fd814cd2ad1039d3162d8) (PR #16526), we can get the evaluation loss. Could you try with the latest version? Thanks.

(You can try with a tiny dummy training)

```bash
***** train metrics *****
  epoch                    =        1.0
  train_loss               =     1.3123
  train_runtime            = 0:00:08.53
  train_samples_per_second =      1.876
  train_steps_per_second   =      0.938
[INFO|trainer.py:2412] 2022-10-22 11:28:29,243 >> ***** Running Evaluation *****
[INFO|trainer.py:2414] 2022-10-22 11:28:29,243 >>   Num examples = 16
[INFO|trainer.py:2417] 2022-10-22 11:28:29,243 >>   Batch size = 2
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  5.62it/s]
***** eval metrics *****
  epoch                   =        1.0
  eval_loss               =     0.8159
  eval_runtime            = 0:00:01.63
  eval_samples_per_second =      9.806
  eval_steps_per_second   =      4.903
```

Dummy training like
```bash
python ./run_clip.py \
    --output_dir ./outputs \
    --model_name_or_path ./clip-roberta \
    --data_dir $PWD/data \
    --dataset_name ydshieh/coco_dataset_script \
    --dataset_config_name=2017 \
    --image_column image_path \
    --caption_column caption \
    --remove_unused_columns=False \
    --do_train \
    --do_eval \
    --num_train_epochs 1 \
    --max_steps 8 \
    --max_train_samples 16 \
    --max_eval_samples 16 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --learning_rate="5e-5" \
    --warmup_steps="0" \
    --weight_decay 0.1 \
    --overwrite_output_dir \
```

> Hi @lchwhut
> 
> Since the commit [3951b9f39](https://github.com/huggingface/transformers/commit/3951b9f3908bfa30be7fd814cd2ad1039d3162d8) (PR #16526), we can get the evaluation loss. Could you try with the latest version? Thanks.
> 
> (You can try with a tiny dummy training)
> 
> ```shell
> ***** train metrics *****
>   epoch                    =        1.0
>   train_loss               =     1.3123
>   train_runtime            = 0:00:08.53
>   train_samples_per_second =      1.876
>   train_steps_per_second   =      0.938
> [INFO|trainer.py:2412] 2022-10-22 11:28:29,243 >> ***** Running Evaluation *****
> [INFO|trainer.py:2414] 2022-10-22 11:28:29,243 >>   Num examples = 16
> [INFO|trainer.py:2417] 2022-10-22 11:28:29,243 >>   Batch size = 2
> 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  5.62it/s]
> ***** eval metrics *****
>   epoch                   =        1.0
>   eval_loss               =     0.8159
>   eval_runtime            = 0:00:01.63
>   eval_samples_per_second =      9.806
>   eval_steps_per_second   =      4.903
> ```
> 
> Dummy training like
> 
> ```shell
> python ./run_clip.py \
>     --output_dir ./outputs \
>     --model_name_or_path ./clip-roberta \
>     --data_dir $PWD/data \
>     --dataset_name ydshieh/coco_dataset_script \
>     --dataset_config_name=2017 \
>     --image_column image_path \
>     --caption_column caption \
>     --remove_unused_columns=False \
>     --do_train \
>     --do_eval \
>     --num_train_epochs 1 \
>     --max_steps 8 \
>     --max_train_samples 16 \
>     --max_eval_samples 16 \
>     --per_device_train_batch_size 2 \
>     --per_device_eval_batch_size 2 \
>     --learning_rate="5e-5" \
>     --warmup_steps="0" \
>     --weight_decay 0.1 \
>     --overwrite_output_dir \
> ```

Hi @ydshieh
Thanks for your help!
I updated `transformers==4.17.0` to `transformers==4.21.3` and could get the evaluation loss now!
<img width="1235" alt="image" src="https://user-images.githubusercontent.com/27990344/197668516-35e48897-0728-40bf-9179-80862ddd4941.png">
But the error `'BaseModelOutputWithPoolingAndCrossAttentions' object has no attribute 'detach'` will still reproduce if i customize `compute_metrics`.
I check the code and find if don't customize `compute_metrics`, `prediction_loss_only=None` will be set and thus the `prediction_step` will return `(loss, None, None)` (just skip `nested_detach`)
<img width="850" alt="image" src="https://user-images.githubusercontent.com/27990344/197669277-debf27d2-306e-43cb-a5f6-84ca95a4befe.png">
<img width="442" alt="image" src="https://user-images.githubusercontent.com/27990344/197668702-0506b9bd-6c73-453e-84b8-e9b1c5632773.png">
I think it doesn't matter because the `eval_loss` is enough for me. I can remove `text_model_output` and `vision_model_output` from `CLIPOutput` if i need customized compute_metrics to evaluate.
Thank you very much!


Hi @lchwhut Yeah, `CLIP` is indeed special in terms of the output format :-) The `Trainer` class is designed to work with the most common use cases, but not a one-size-fits-all solution. Sometimes we need more customization



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.