noamaanMulla-03/transformers-issues huggingface dataset:
Feature request: Support evaluation every N epochs in TrainingArguments
Author: varuna-km-18267
Pull request: no
Labels: name: Feature request

### Feature request

Currently, Trainer supports evaluation strategies:
- "epoch": evaluate every epoch
- "steps": evaluate every N steps

However, there is no built-in way to evaluate every N epochs (e.g., every 5 epochs).

This is particularly useful when:
- evaluation is computationally expensive
- running large-scale training
- benchmarking periodically instead of every epoch

Suggested API:
- Add parameter like `eval_epochs` (int)
- Or extend `eval_strategy="epoch"` to support frequency

Example:
eval_strategy="epoch"
eval_epochs=5  # evaluate every 5 epochs

Currently, this requires custom Trainer overrides or callbacks.

### Motivation

-

### Your contribution

-

Comments:
cc @SunMarc 

Thanks for the feature request. I'm not sure if it is worth adding a new arguments for that, so i'll wait to see if the community is interested in that. For now, you can also just specify steps even it is a bit for complicated. 

Hi @SunMarc,

I’d like to work on this feature upon discussion and feedback

I agree that adding a new argument directly may increase API complexity. Instead of introducing a completely separate parameter, I was thinking of extending the existing `evaluation_strategy="epoch"` behavior with an optional frequency parameter like `eval_epochs`.

This way:
- backward compatibility is preserved (default = 1 epoch)
- no change for existing users
- allows flexibility for large-scale training where evaluation is expensive

Alternatively, we could also consider integrating this logic into the existing control flow without adding a completely new strategy.

Would love your thoughts before proceeding.

Hi @SunMarc,

I'd like to work on this. 

My Plan: add an eval_epochs argument to TrainingArguments (defaults to 1, so existing behavior is unchanged), wire it into DefaultFlowCallback.on_epoch_end so evaluation only fires on epochs that are multiples of eval_epochs, and always evaluate on the final epoch regardless, so users don't lose their last eval metrics.

I will add tests and open a PR shortly.
Fix train_batch_size and eval_batch_size to respect split_batches config
Author: MinuriRajapakse
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=45694)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45694)
<!-- ci-dashboard-badge:end -->

Fixes #45693

## Problem
When `split_batches=True` is set in `accelerator_config`, the 
`train_batch_size` and `eval_batch_size` properties were still 
multiplying `per_device_batch_size` by `n_gpu`, which is incorrect.

When `split_batches=True`, the batch is split across devices rather 
than replicated, so the total batch size equals `per_device_batch_size` 
directly.

## Fix
Added a check for `split_batches` in both `train_batch_size` and 
`eval_batch_size` properties in `TrainingArguments`.

## Testing
- Added a new test `test_batch_size_respects_split_batches`
- All 26 existing + new tests pass

Comments:
cc @sunmarc

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45694)
**Latest run:** [28564422708:2](https://github.com/huggingface/transformers/actions/runs/28564422708)
**Result:** `failure` | **Jobs:** 14 | **Tests:** 70,943 | **Failures:** 1 | **Duration:** 18h 19m

<!-- ci-dashboard-recap:end -->

The only failing check is `tests_processors [shard 5/8] ` which is unrelated to this PR, it's a `PIL.UnidentifiedImageError` flaky test in image processing. All 70,942 other tests pass. Could a maintainer please re-run CI or approve the PR? Thanks! @SunMarc
Allowing dataloader prefetching to work with iterable datasets.
Author: Lucaweihs
Pull request: yes

# What does this PR do?

This PR fixes #34867 which noted that dataloader prefetching was not occurring for iterable datasets. This PR also refactors the dataloader creation code in the trainer so that the train/eval/test dataloaders all use the same underlying function rather than repeating the same logic in multiple places. I added two tests to confirm that the prefetching works with iterable datasets as expected.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?

## Who can review?

@muellerzr @SunMarc

Comments:
Hi @Lucaweihs — I'm the author of issue #43431, which overlaps with this PR. I noticed this has been quiet since Dec 2024 and is now in a `dirty` mergeable state.
Are you still planning to push it forward? If you don't have the bandwidth, I'd like to pick it up: rebase onto curren `main`, address the earlier review feedback, and land it as part of the work I'm planning  for #43431 (which also adds `dataloader_multiprocessing_context` in a separate PR). I'd of course keep `Co-authored-by: @Lucaweihs` on the commit and link this PR in the description.
Happy to wait a week or so for your reply before opening anything. cc @SunMarc @Rocketknight1 in case maintainers want to weigh in.

Completely missed this @muyihao ;) Feel free to open a PR, i will do my best to get it merged
revert #47121
Author: eustlb
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47144)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47144)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

This reverts commit c2e101442d3e85cf7c8d006dffe8b3eeaa5e7a47, introduced in  #47121.

Indeed, not only did it not fix the mentioned DAC slow test, but introduced a new failure.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: dac

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47144)
**Latest run:** [28877713985](https://github.com/huggingface/transformers/actions/runs/28877713985)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47144). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] fix autolinks
Author: stevhliu
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46968)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46968)
<!-- ci-dashboard-badge:end -->

A dataclass field or `__init__` parameter can't be linked using the usual dot form syntax (`.`) and are usually just written as Markdown links. Using the `#` form instead of the dot form seems to resolve this workaround.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46968)
**Latest run:** [28408091615:1](https://github.com/huggingface/transformers/actions/runs/28409702513)
**Result:** `success` | **Jobs:** 2 | **Tests:** 24 | **Failures:** 0 | **Duration:** 1m 50s

<!-- ci-dashboard-recap:end -->

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46968). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @Rocketknight1
[docs] continuous batching (offloading behavior, max batch tokens, block size minimum)
Author: stevhliu
Pull request: yes

Adds docs for several recent continuous batching PRs (#46765, #46712, #46587):

- New section for `max_batch_tokens`,  the trade off and when to raise or lower it
- New section explaining `block_size`
- Reframe soft reset as the fallback path for offloading
- Describe new offloading behavior (demand-drive, newest first-eviction with batched CPU transfers)

Comments:
**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46925)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46925). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add eval_epochs to TrainingArguments to support evaluating every N epochs
Author: lakshmiprasanna07
Pull request: yes
Labels: name: Code agent slop

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47084)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47084)
<!-- ci-dashboard-badge:end -->

Fixes #45308

`eval_strategy` currently only supports `"steps"` and `"epoch"`, with no way to evaluate every N epochs. This is a problem when evaluation is expensive (large eval sets, generation-based metrics) and training runs for many epochs.

This PR adds a new `eval_epochs` argument (default `1`, so existing behavior is unchanged) to `TrainingArguments`, and uses it in `DefaultFlowCallback.on_epoch_end` to only trigger evaluation on epochs that are multiples of `eval_epochs`.

Two details worth calling out for review:
- The final training epoch always triggers evaluation, even if it isn't a multiple of `eval_epochs`, so users always get metrics for their last checkpoint.
- Setting `eval_epochs` while `eval_strategy` isn't `"epoch"` doesn't raise, but logs a warning, since it's a silent no-op otherwise and I'd rather not break scripts that set it defensively.

**Testing:** added `DefaultFlowCallbackEvalEpochsTest` in `tests/trainer/test_trainer_callback.py` covering default behavior, skipping intermediate epochs, forced final-epoch evaluation, interaction with `eval_delay`, the `eval_epochs < 1` validation error, and no-op behavior under `eval_strategy="steps"`. Full `tests/trainer/test_trainer_callback.py` suite passes (57 passed, 3 skipped). `ruff check` and `ruff format --check` are clean on all changed files.

Comments:
<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47084)
**Latest run:** [28825638099:2](https://github.com/huggingface/transformers/actions/runs/28825638099)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 171,556 | **Failures:** 6 | **Duration:** 24h 3m

<!-- ci-dashboard-recap:end -->
Skip caching_allocator_warmup on Neuron (no reuse pool to warm; currently OOMs)
Author: dacorvo
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47029)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47029)
<!-- ci-dashboard-badge:end -->

### What does this PR do?

`caching_allocator_warmup` pre-allocates `get_total_byte_count` bytes per device in a single
`torch.empty(...)`. The `cuda`/`xpu` branch bounds that allocation against device capacity via
`mem_get_info`; `mps` is skipped (#46239). Every **other** accelerator device — including `neuron`
(AWS Trainium/Inferentia, a PrivateUse1 backend) — falls through to the **unclamped** allocation, which
OOMs the NeuronCore during `from_pretrained`, before any weight is loaded:

```
RuntimeError: NRT allocation failed for size 233658313344 (OOM)
```

This happens because since `torch.neuron` exposes no `mem_get_info`, the allocation can't be bounded.

Incidentally, the warmup provides no benefit. on Neuron, since the backend keeps no CUDA-style reuse pool for a warm-up to fill: `torch_neuronx.memory_stats()` reports `reserved_bytes = 0` even while 2 GB is allocated.

Skipping therefore costs nothing and fixes the OOM — the same trade-off as the MPS skip (#46239).

### Before submitting
- [x] Read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).
- [x] Verified on AWS Trainium2: the OOM repro + allocator/timing measurements above, and a full support
      sweep with the skip applied — the warmup allocations are gone, models load up to their real footprint,
      and TP>1 models that previously passed still pass (no regression). transformers CI has no Neuron runner,
      as with the MPS skip.

### Who can review?
@Cyrilvallez @SunMarc (reviewed the related warmup PRs #36380 / #46145; cf. the MPS skip #46239)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47029). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Pushed without my consent by an overzealous agent, sorry about that.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47029)
**Latest run:** [28874754917:1](https://github.com/huggingface/transformers/actions/runs/28874754917)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 172,467 | **Failures:** 1 | **Duration:** 22h 23m

<!-- ci-dashboard-recap:end -->
Add linear_attn entries to Qwen3.5 base_model_tp_plan
Author: zaid646
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47009)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47009)
<!-- ci-dashboard-badge:end -->

Fixes #46846

Qwen3.5 is a hybrid model: ~75% of decoder layers use linear_attention (Gated DeltaNet) with their own projection matrices (`in_proj_qkv`, `in_proj_z`, `in_proj_b`, `in_proj_a`, `out_proj`). These were missing from `base_model_tp_plan`, causing:
- OOM at TP>1 (weights not sharded)
- RuntimeError in `model.generate()` (Conv1d channel mismatch after `in_proj_qkv`)

**Fix:** Add all `linear_attn.*` projections with `"colwise_gather_output"` pattern, which shards the weight matrix (fixing OOM) and all-gathers activations before the depthwise Conv1d (fixing the shape mismatch).

Updated both `modular_qwen3_5.py` (source) and `configuration_qwen3_5.py` (auto-generated).

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3_5

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47009)
**Latest run:** [28538882882:2](https://github.com/huggingface/transformers/actions/runs/28538882882)
**Result:** `success` | **Jobs:** 2 | **Tests:** 10 | **Failures:** 0 | **Duration:** 45s

<!-- ci-dashboard-recap:end -->

cc @cyrilvallez for TP!

Already taken care of
Restrict QuantizedCache to only full attention
Author: Cyrilvallez
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47160)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47160)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

As per the title. Supersedes https://github.com/huggingface/transformers/pull/46781 correctly cause I don't have time to explain stuff to code agents

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47160). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47160)
**Latest run:** [28924042165:1](https://github.com/huggingface/transformers/actions/runs/28924042165)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 172,237 | **Failures:** 36 | **Duration:** 21h 54m

<!-- ci-dashboard-recap:end -->
Diffusion gemma: fix failed test cases
Author: kaixuanliu
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47025)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47025)
<!-- ci-dashboard-badge:end -->

This PR skip 1 invalid test case: `tests/models/diffusion_gemma/test_modeling_diffusion_gemma.py::DiffusionGemmaVisionText2TextModelTest::test_model_parallelism`, and add 5 expected output on XPU for diffusion_gemma model tests:
```
tests/models/diffusion_gemma/test_modeling_diffusion_gemma.py::DiffusionGemmaIntegrationTest::test_minified_diffusion_gemma_forward_batched

tests/models/diffusion_gemma/test_modeling_diffusion_gemma.py::DiffusionGemmaIntegrationTest::test_minified_diffusion_gemma_forward_text_only

tests/models/diffusion_gemma/test_modeling_diffusion_gemma.py::DiffusionGemmaIntegrationTest::test_minified_diffusion_gemma_forward_with_image

tests/models/diffusion_gemma/test_modeling_diffusion_gemma.py::DiffusionGemmaIntegrationTest::test_minified_diffusion_gemma_generate_with_image_batched

tests/models/diffusion_gemma/test_modeling_diffusion_gemma.py::DiffusionGemmaIntegrationTest::test_minified_diffusion_gemma_generate_with_image_batched_long
```
@ydshieh pls help review, thx!

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: diffusion_gemma

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47025)
**Latest run:** [28596442015](https://github.com/huggingface/transformers/actions/runs/28596442015)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47025). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
add XPU Expectation for cosmos3_omni tests
Author: kaixuanliu
Pull request: yes

@ydshieh pls help review, thx!

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: cosmos3_omni

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46880)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46880). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
add XPU output expectations for minicpm3 tests
Author: kaixuanliu
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47092)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47092)
<!-- ci-dashboard-badge:end -->

@ydshieh pls help review, thx!

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: minicpm3

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47092)
**Latest run:** [28762107158](https://github.com/huggingface/transformers/actions/runs/28762107158)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47092). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
chore(qa): fix Windows compatibility for type checker excludes
Author: Partha-Shankar
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47167)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47167)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

This PR fixes type checker failures on Windows environments by enforcing the exclusion of generated Protocol Buffer files (`sentencepiece_model_pb2.py` and `sentencepiece_model_pb2_new.py`) when `utils/check_types.py` is executed.

Specifically, it:
1. Appends `--force-exclude` to the `ty check` arguments list to ensure that exclude rules are respected even when directories are passed to `ty` as explicit positional targets.
2. Specifies explicit relative file paths for the generated protobuf modules to ensure that Windows path resolution (backslashes) does not prevent glob matching.

Fixes # (no corresponding Github issue, found during developer environment setup on Windows)

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://huggingface.co/docs/transformers/contributing) and the [Pull Request](https://huggingface.co/docs/transformers/pr_checks) checks?
- [x] Was this discussed/approved via a Github issue or the forum?
- [x] Did you make sure to update the documentation with your changes according to the guidelines?
- [x] Did you write any new necessary tests? (No new tests required for developer tooling script modifications).

## Who can review?
- CIs/Tooling: @ydshieh
- Library / Typing: @ArthurZucker

Comments:
<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47167)
**Latest run:** [28928374742](https://github.com/huggingface/transformers/actions/runs/28928374742)
**Result:** `success` | Grafana metrics are not available yet.

<!-- ci-dashboard-recap:end -->