Add repeated n-gram stopping criterion
Author: AliOsm
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47271)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47271)
<!-- ci-dashboard-badge:end -->

> **AI-generated PR disclosure:** This is an AI-generated PR. It is being submitted after three days of implementation, testing, benchmarking, and review in our own Cohere batch-inference script, [AliOsm/cohere-transcribe-arabic-batch-inference](https://github.com/AliOsm/cohere-transcribe-arabic-batch-inference). The production implementation that motivated this contribution is [here](https://github.com/AliOsm/cohere-transcribe-arabic-batch-inference/blob/5035d8ef5787bde147681d6760122a00f8a21303/transcribe.py#L1477-L1560).

# What does this PR do?

Adds an opt-in `RepeatedNGramCriteria` that stops individual batch rows when their suffix contains a configurable number of consecutive copies of the same token block.

It supports:

- one fixed n-gram size or an inclusive size range;
- a configurable repetition count (four by default, matching the production-tested setting);
- prompt exclusion through `prompt_length_to_skip`;
- a `min_new_tokens` gate to reduce premature matches;
- independent per-row results with no sticky criterion state;
- pure tensor operations compatible with `torch.compile(fullgraph=True)`;
- a fast fixed-size path and bounded temporary memory for wide ranges.

The criterion is exported from both `transformers` and `transformers.generation`, documented in the internal generation API, and covered by unit, randomized-reference, compile, stress, export, and real `generate()` integration tests.

Fixes #32902. The issue was explicitly approved for a `StoppingCriteria` contribution in [this maintainer comment](https://github.com/huggingface/transformers/issues/32902#issuecomment-2302674622).

## Scope and prior work

This PR adds the reusable criterion first. It intentionally does not add serialized `GenerationConfig` fields yet: the exact configuration surface (one size versus a range, repetition count, prompt handling, and a minimum-token gate) deserves maintainer agreement before becoming a persisted API. The criterion is fully usable today through `stopping_criteria=StoppingCriteriaList([...])`; config wiring can follow if maintainers prefer it.

Closed draft #44708 contains an adjacent scheduler callback, but not this implementation. That callback inspected only batch row 0, synchronized with `.item()`, counted non-consecutive occurrences, and retained mutable detection state. This PR detects consecutive suffix blocks per row, uses tensor-only operations, and remains stateless so beam/candidate reordering cannot attach completion state to the wrong row. `generate()` already retains stopped rows through its unfinished-sequence mask.

## Production evidence

The deployed Cohere ASR guard uses the same core detection policy used here: check every generated token after a 96-token gate for four consecutive copies of blocks sized 8 through 32. Its wrapper additionally caches invariant index tensors and records audit/EOS state, so these are production implementation results rather than a benchmark of this exact upstream diff.

Matched 500-clip ablation, with every other inference setting held constant (RTX 3060 12 GB, BF16, batch 24, `max_new_tokens=445`, 5,032.699 seconds of Arabic audio):

| Configuration | Wall time | Generation time | Lexical WER |
|---|---:|---:|---:|
| Projection cache, repetition guard off | 59.603 s | 37.034 s | 23.2035% |
| Projection cache, repetition guard on | 51.723 s | 29.116 s | 22.9551% |
| Change | **-13.22%** | **-21.38%** | **-0.2483 pp** |

Only 5 of 500 hypotheses changed. All five were visible punctuation, character, or phrase loops; their combined output length fell from 3,048 to 784 characters. This is why WER improved rather than regressed.

As supplemental scale evidence, a 24,414-clip / 36.393-hour evaluation changed 125 hypotheses (0.512%), reduced lexical WER from 32.2576% to 31.3205% (paired 95% CI for the delta: -1.4766 to -0.4781 percentage points), and reduced insertions from 9,800 to 7,629. Its timing comparison also includes an invariant encoder-projection cache, so the 23.54% generation-time reduction in that larger run is not attributed solely to this stopping policy.

## Exact criterion microbenchmark

This exact diff was measured independently on CPU and an RTX 3060 with PyTorch 2.11.0+cu128. The input was a non-repeating random batch of 24 rows; each trial invoked the criterion for all 350 eligible lengths from 96 through 445 tokens. Results are medians of 100 trials after 10 warmups, with one CUDA synchronization around each complete trial.

| Criterion | CPU, 350 calls | RTX 3060, 350 calls | CUDA per call |
|---|---:|---:|---:|
| Fixed 32-token block, four copies | 7.273 ms | 9.676 ms | 0.0276 ms |
| Range 8–32, four copies | 46.279 ms | 43.375 ms | 0.1239 ms |

The benchmark intentionally uses healthy, non-looping rows so it measures the full detector overhead without early termination. There are no timing assertions in the test suite.

## Validation

```text
make style
  4/4 checks passed

make check-repo
  24/24 checks passed

PYTHONPATH=src python -m pytest -q tests/generation/test_stopping_criteria.py
  31 passed

PYTHONPATH=src python -m pytest -q \
  tests/generation/test_utils.py::GenerationIntegrationTests::test_repeated_ngram_stopping_criteria
  1 passed
```

The tests cover constructor validation, fixed and ranged matching, mixed batch rows, exact and near-miss thresholds, prompt exclusion, minimum-token gating, row reordering/statelessness, empty batches, bounded wide-range processing, 100 randomized cases against a Python reference, fixed/range full-graph compilation, top-level exports, and deterministic generation stopping.

## Code Agent Policy

- [ ] (First-time contributors only): not applicable. `AliOsm` is an existing Transformers contributor, and the AI-generated nature of this PR is disclosed at the top rather than checking this attestation.

## Before submitting

- [x] I read the contributor and pull-request guidance.
- [x] This was discussed and approved in #32902.
- [x] The API documentation and public exports are updated.
- [x] Necessary tests were added and the repository checks pass.

Reviewers: @gante for the original issue context and @Cyrilvallez for generation.

Comments:
<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47271)
**Latest run:** [29280702776:2](https://github.com/huggingface/transformers/actions/runs/29280702776)
**Result:** `success` | **Jobs:** 2 | **Tests:** 24 | **Failures:** 0 | **Duration:** 1m 59s

<!-- ci-dashboard-recap:end -->