Feature/mean token accuracy trainer #40303
Author: badwriter123
Pull request: yes

# What does this PR do?

Feature : Add mean token accuracy to Trainer

Fixes #40303 

### Description
1. For the file -->  src/transformers/training_args.py  Added a new boolean argument to allow users to toggle the mean token accuracy computation.

2. For the file --> src/transformers/trainer.py. Added the new method  (_compute_mean_token_accuracy) . Improvised for both Language Models and Seq2seq Models.

The feature is now ready to be used by setting compute_mean_token_accuracy=True in TrainingArguments.

Comments:
Why are changes to gemma3n in this PR?

Similarly, why have dependencies changed?

Actually the changes other than the mean token accuracy feature were from a different pr, that has not been reviewed or merged. It got mixed up with the mean token accuracy feature.

They can be overwritten, and just the mean token accuracy feature can be reviewed.

You should clean up the PR then :)

Yeah I'll do that and get back with the update . 🤗

Hello @RylanSchaeffer ,
I have removed the unnecessary files from the pr, the required feature implementation can be reviewed.

Hello @AzizCode92 
I have removed the unnecessary python version constraint from setup.py and restored it to original state.
Feature Implementation is ready for review.

cc @SunMarc 

Hello @Rocketknight1 ,
Requesting review of mean token accuracy feature #40303 .

Out of curiosity, what are using the trainer for?
In case it's relevant, the SFTTrainer already computes and report the mean token accuracy.

Pretraining smallish autoregressive models. To be honest, I don't know what the difference is between transformers trainer and TRL's sft trainer, and I haven't found the time to investigate 😅

Pretraining smallish autoregressive models. To be honest, I don't know what the difference is between transformers trainer and TRL's sft trainer, and I haven't found the time to investigate 😅

`SFTTrainer` is `Trainer` with some extra features, like preparing your dataset if needed, packing, log useful metrics like the mean token accuracy. Technically you can use `SFTTrainer` to do pretraining.

Let's wait what @SunMarc think of upstreaming the mean token accuracy computation in `Trainer`. I don't have strong opinion on this.

`SFTTrainer is Trainer with some extra features, like preparing your dataset if needed, packing, log useful metrics like the mean token accuracy. Technically you can use SFTTrainer to do pretraining.`

Can you try to use SFTTrainer to train your model @RylanSchaeffer ? I prefer not adding this if this not very important feature as this will make Trainer more bloated 

@SunMarc a question and a comment: 

1. Is there any reason to prefer Trainer over SFTTrainer for pretraining?

2. From a user perspective, this is _very_ confusing. As I understand, your guidance is, "For pretraining, install a library called Transformer _Reinforcement Learning_ and then use its SFTTrainer." I don't think using an RL library's SFT implementation for pretraining is intuitive or stated anywhere in any documentation (but if I'm wrong, please let me know). I love TRL and have even made some contributions to it, but I'm a bit baffled by this conceptual organization across libraries.

Also, is packing not implemented for Trainer, only SFTTrainer?

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma3n

Hello @MekkCyber ,
Files Have been restored as previous.

> Is there any reason to prefer Trainer over SFTTrainer for pretraining?

SFTTrainer only add features, so I'd say no. But if you don't use any of these features, then you should use Trainer.

> From a user perspective, this is very confusing. As I understand, your guidance is, "For pretraining, install a library called Transformer Reinforcement Learning and then use its SFTTrainer." 

I'd phrase a bit differently:
For pretraining, I recommend using nanotron. However, if you want to use `transformers.Trainer`, You can sneak around and use `SFTTrainer` directly to benefit from some of its added features (like packing, accuracy logging...), while remaining aware that this is a roundabout way of using it.