Feature: print tokens per second during training
Author: tibor-reiss
Pull request: yes

Fixes #34471 

If `include_num_input_tokens_seen` was part of `args`, the requested metrics can be printed from `self.state.num_input_tokens_seen`.

@muellerzr and @SunMarc


Comments:
I saw #31501 and these two seem related to some extent. Proposal: remove `include_tokens_per_second` and use `num_tokens=self.state.num_input_tokens_seen` instead of `num_tokens=num_train_tokens` when reporting metrics for the full run. What do you think @muellerzr?

> We already have the option to set `include_num_input_tokens_seen`. Could you try it and tell us if it fits your use case. One thing that could be nice it to compare the results we get with `include_num_input_tokens_seen` and `include_tokens_per_second` cc @muellerzr

Hi @SunMarc, I think we are talking about the same. Someone requested in #34471 the option to be able to see the number of tokens inside the training loop - currently only the total is logged at the end of the training. `num_input_tokens_seen` is already tracked if `include_num_input_tokens_seen`, so I just added one line for the logging. This is the scope of the current PR.

My comment (https://github.com/huggingface/transformers/pull/34507#issuecomment-2447742561) is about simplification, because the current logic poses anyways issues with large datasets. This is not yet covered by this PR, but I could make another one if you agree with my proposal.

> My comment (https://github.com/huggingface/transformers/pull/34507#issuecomment-2447742561) is about simplification, because the current logic poses anyways issues with large datasets.

Before doing that, could you verify that these two metrics returns actually the same thing. if yes, then we should definitely do as you suggested ! 

> > My comment ([#34507 (comment)](https://github.com/huggingface/transformers/pull/34507#issuecomment-2447742561)) is about simplification, because the current logic poses anyways issues with large datasets.
> 
> Before doing that, could you verify that these two metrics returns actually the same thing. if yes, then we should definitely do as you suggested !

Results from some basic tests (for speed, I commented out the call to `self.training_step` because it does not influence the result) with the following args:

case 1: `--log_level info     --model_name_or_path=distilbert/distilgpt2    --dataset_name=Salesforce/wikitext     --dataset_config_name=wikitext-2-raw-v1     --block_size=1024     --per_device_train_batch_size=8     --do_train     --output_dir=Llama-3.2-1B-wikitext-2-raw-v1     --overwrite_output_dir     --seed=42     --logging_steps=10     --lr_scheduler_type=cosine     --num_train_epochs=3     --learning_rate=5e-05     --warmup_ratio=0.03     --dataloader_drop_last  --include_tokens_per_second --include_num_input_tokens_seen`

case 2: using different model, i.e. `model_name_or_path=meta-llama/Llama-3.2-1B`

case 3: using `--max_step=50`

In all 3 cases, the values matched:
- case 1: 7'102'464
- case 2: 7'348'224
- case 3: 409'600

The one test missing is when dataloader does not have a size. Not sure yet how to create that. However, looking at the formula, it is likely that it will also match.

cc @SunMarc @muellerzr :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34507). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.