compute_metrics with causal LM training
Author: steremma
Pull request: no
Labels: name: WIP

### Feature request

Besides loss, users often need to report additional metrics throughout the training in order to drive decision making and communicate results, which in the case of Seq2Seq models is elegantly done with the `compute_metrics` argument of the `Trainer`. Generative metrics easily fit this framework by setting `predict_with_generate=True`. The same is much less straightforward with a Causal underlying LM. The only "working" approach I found is this: https://github.com/huggingface/transformers/blob/5e11d72d4d0939138fbabfebe9a69d2061519547/examples/pytorch/language-modeling/run_clm.py#L578

But I think this is an erroneous calculation: the `logits.argmax(dim=-1)` call does not really generate in inference mode, it "cheats" because of teacher forcing and therefore any metric computed that way is probably inflated. Ideally it would be possible to make the argument passed to `compute_metrics`  include a proper `predictions` attribute that has been properly generated using the trainers generation config.

### Motivation

I am always frustrated when I can't observe the learning trajectory of my generative metric (say BLEU/ROUGE) when using a CML even though it is trivial to do when I am using a S2S

### Your contribution

If you confirm that this is an issue and important enough to justify a fix I may be able to make a PR but can't promise it

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Related issue: https://github.com/huggingface/trl/issues/1222

For causal LM fine-tuning with instruction tuning (i.e., completion only) I use the `SFTTrainer` from the [trl library](https://github.com/huggingface/trl) and it suffers from the same problem. Perhaps the solution I suggested there would provide some necessary insights.

I don't know why this issue is closed but it's really a necessity in order to avoid spending unnecessary computation and easily selecting the best possible model. We should have something that works well across different generative models.

At the moment most causal models generate predictions with shape not the same as labels which causes an error when computing different metrics. 

cc @gante @zucchini-nlp @muellerzr 

Same issue as #31462 and I am very much pro of having this as feature. After having tried to tune a VLMs with HF Trainer, noticed that there are some points that could be improved. 

The main issue here, as I see it, is that decoder-only models need different inputs for generation and for loss calculation. I can propose two options:
1. If users pass in already tokenized inputs, they could add smth like "generation_mask"  which will be used to crop out the generation part from `input_ids`, since I think passing whole `input_ids` and `generation_input_ids` is kinda double work. Maybe this can be part of the `DataCollatorForLanguageModeling` activated by a flag `predict_with_generate`
2. If users want to pass in simple text format dataset, we can have a data collator that is similar to `DataCollatorForCompletionTask` but will return `generation_input_ids` along with the usual inputs

Let me know what you think :)  

@zucchini-nlp asking our internal fine-tune folks for suggestions here would be great, I'm sure they have opinions about it (given the related thread in TRL)!

btw, let me know if you have bandwidth to take this issue. Since there are a LOT of people requesting this, I'm happy to take it if you're low on bandwidth :)

@gante definitely, I will post on out internal slack with other related issues I encountered on my way to tuning VLMs. And yes, I can take on this and open a PR next week (or after I come back in July) :)

(@zucchini-nlp assigning to you as per your comment above 🤗 )

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

How's going on this now?

@jacklanda we have a PR and I'll try to get review on it soon. But note that enabling generation on LLMs while training will be very slow as generation in general is a slow process due to auto regressive nature of LLMs. We haven't yet thought about speeding it up, and the PR is on initial design stage only

Thank you, I think this will be an amazing feature. Indeed generation can
be slow but the user can always address that by limiting the size of their
validation set, or running evaluation less frequently (or just not enabling
it). In many cases the advantage of computing metrics or even plain
eyeballing intermediate generations is very beneficial.


On Sun, 24 Nov 2024 at 06:11, Raushan Turganbay ***@***.***>
wrote:

> @jacklanda <https://github.com/jacklanda> we have a PR and I'll try to
> get review on it soon. But note that enabling generation on LLMs while
> training will be very slow as generation in general is a slow process due
> to auto regressive nature of LLMs. We haven't yet thought about speeding it
> up, and the PR is on initial design stage only
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/26474#issuecomment-2495943557>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACG2NY6YRWFYH6VD2YTM4AL2CGX4JAVCNFSM6AAAAAA5LFWFU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJVHE2DGNJVG4>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


> @jacklanda we have a PR and I'll try to get review on it soon. But note that enabling generation on LLMs while training will be very slow as generation in general is a slow process due to auto regressive nature of LLMs. We haven't yet thought about speeding it up, and the PR is on initial design stage only

You know we can enable `use_cache` when generating, it does help.

> Thank you, I think this will be an amazing feature. Indeed generation can be slow but the user can always address that by limiting the size of their validation set, or running evaluation less frequently (or just not enabling it). In many cases the advantage of computing metrics or even plain eyeballing intermediate generations is very beneficial.
> […](#)
> On Sun, 24 Nov 2024 at 06:11, Raushan Turganbay ***@***.***> wrote: @jacklanda <https://github.com/jacklanda> we have a PR and I'll try to get review on it soon. But note that enabling generation on LLMs while training will be very slow as generation in general is a slow process due to auto regressive nature of LLMs. We haven't yet thought about speeding it up, and the PR is on initial design stage only — Reply to this email directly, view it on GitHub <[#26474 (comment)](https://github.com/huggingface/transformers/issues/26474#issuecomment-2495943557)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACG2NY6YRWFYH6VD2YTM4AL2CGX4JAVCNFSM6AAAAAA5LFWFU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJVHE2DGNJVG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

Ditto, I think the users should notice the limit of generation samples and control it to avoid high time costs.

That's exactly what I do with Seq2Seq: just use a smaller validation set. Sure, it would be great to be able to specify another set of samples (not `eval_dataset` but, say, `generate_and_eval_dataset`) but for now just being able to use `predict_with_generate` on the  `SFTTrainer` would be amazing.

I'm happy to test the PR, can you please link to it?

EDIT: Nevermind, I think I found it: https://github.com/huggingface/transformers/pull/32346

Just run into [this Wandb tutorial](https://wandb.ai/capecape/alpaca_ft/reports/How-to-Fine-tune-an-LLM-Part-3-The-HuggingFace-Trainer--Vmlldzo1OTEyNjMy#(optional)-preprocessing:-masking-instructions-by-using-the-datacollatorforcompletiononlylm), it suggests a nice workaround of using a callback during training. The disadvantage of this approach is that the eval dataset will be evaluated twice: once to calculate the eval loss, inside SFTTrainer evaluation loop, and once inside the callback.

> Just run into [this Wandb tutorial](https://wandb.ai/capecape/alpaca_ft/reports/How-to-Fine-tune-an-LLM-Part-3-The-HuggingFace-Trainer--Vmlldzo1OTEyNjMy#(optional)-preprocessing:-masking-instructions-by-using-the-datacollatorforcompletiononlylm), it suggests a nice workaround of using a callback during training. The disadvantage of this approach is that the eval dataset will be evaluated twice: once to calculate the eval loss, inside SFTTrainer evaluation loop, and once inside the callback.

Very impressive! Thanks for sharing!