Fix gradient inflation when combining label smoothing with gradient accumulation
Author: Incheonkirin
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47261)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47261)
<!-- ci-dashboard-badge:end -->

> This replaces #46734, which I accidentally closed when its source fork was detached. The change has been rebased onto current `main`; the original discussion remains in #46734.

## What this fixes

`label_smoothing_factor > 0` with `gradient_accumulation_steps > 1` inflates gradients by ~`gradient_accumulation_steps` for any model whose forward accepts loss kwargs. Repro and write-up in #46731.

## Root cause

#34198 wired `num_items_in_batch` into `trainer.py` but not into `LabelSmoother` (`trainer_pt_utils.py`). When `model_accepts_loss_kwargs` is True, `training_step` skips the `1/gradient_accumulation_steps` normalization on the assumption the loss already divided by the effective batch — but `LabelSmoother` mean-reduces over the micro-batch instead. The same `num_items_in_batch` is already threaded into the adjacent `compute_loss_func` branch.

## Fix

- `LabelSmoother.__call__` takes `num_items_in_batch` and uses it as the denominator, falling back to the active-token count when it's `None` (so standalone calls are unchanged).
- `compute_loss` passes `num_items_in_batch` into both `label_smoother` calls.

## Test

`test_gradient_accumulation_grad_norm_with_label_smoothing`: the grad-norm ratio between a large-batch baseline and an equivalent accumulation run goes from ~9x at `gradient_accumulation_steps=8` to ~1.0. Existing `TrainerGradientAccumulationTest` / `TrainerLabelSmoothingTest` still pass.

## Backward compatibility

Changes loss/gradient magnitude by ~`gradient_accumulation_steps` for existing label-smoothing + accumulation runs (previously over-scaled). LRs tuned against the old magnitude may need re-checking.

## Scope

Single-device and default DDP token averaging become exact. Legacy DataParallel (`n_gpu > 1`) keeps the same `n_gpu` approximation the `compute_loss_func` path already has, so this doesn't add DP-specific behavior.

Closes #46731

## Validation
- `pytest -q tests/trainer/test_trainer.py::TrainerGradientAccumulationTest::test_gradient_accumulation_grad_norm_with_label_smoothing` (1 passed)

Comments:
@SunMarc, could you review this LabelSmoother gradient-accumulation fix? It replaces #46734; the full PR CI is green.

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/29543790283) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

Thanks, this looks good to me, I was able to repro:

before the fix

```python
import torch
from transformers.trainer_pt_utils import LabelSmoother

ls = LabelSmoother(epsilon=0.1)
torch.manual_seed(0)
logits = torch.randn(4, 3, 5)        # 4 sequences, 3 tokens, vocab 5
labels = torch.randint(0, 5, (4, 3)) # no padding -> 12 active tokens

full  = ls({"logits": logits}, labels)  # what bs=4, gas=1 computes
accum = sum(ls({"logits": logits[i:i+1]}, labels[i:i+1]) for i in range(4))  # what bs=1, gas=4 accumulates
print(full, accum, accum / full)
```

```
tensor(1.8192) tensor(7.2767) tensor(4.0000)
```

after

```diff
- accum = sum(ls({"logits": logits[i:i+1]}, labels[i:i+1]) for i in range(4))
+ from math import prod
+ accum = sum(ls({"logits": logits[i:i+1]}, labels[i:i+1], num_items_in_batch=prod(labels.shape)) for i in range(4))
```

```
tensor(1.8192) tensor(1.8192) tensor(1.0000)
```

I'll let @SunMarc give the final review and merge 

applied all three in 7de5122c, trainer GA + label smoothing tests still green locally

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47261)
**Latest run:** [29630724535:2](https://github.com/huggingface/transformers/actions/runs/29630724535)
**Result:** `success` | **Jobs:** 15 | **Tests:** 172,163 | **Failures:** 0 | **Duration:** 14h 50m

<!-- ci-dashboard-recap:end -->