Fall back to the for-loop grouped_mm on CPU
Author: Sunt-ing
Pull request: yes


# What does this PR do?

The shared MoE experts path in `integrations/moe.py` runs each expert's matmul through `torch._grouped_mm` / `torch.nn.functional.grouped_mm`, and falls back to a per-expert for-loop of `torch.mm` when grouped matmul is not usable. `_can_use_grouped_mm` decides which path to take.

On CPU it returned `True` whenever the tensors were 16-byte aligned (the only CPU case it rejected was misalignment, for `torch <= 2.10`). But `torch._grouped_mm` has no CPU kernel on supported torch versions: on torch 2.8 (the library requires only `torch>=2.4`) `aten::_grouped_mm` is registered for CUDA/Meta only and raises `NotImplementedError` on CPU. So aligned CPU tensors reached the grouped-mm branch and every MoE model using this experts path crashed on CPU instead of falling back, even though the for-loop right below it is pure `torch.mm` and runs fine.

The fix drops the alignment sub-condition so CPU always takes the for-loop fallback for `torch <= 2.10`, which the existing comment already describes as the intended CPU path. This is deliberate: `torch._grouped_mm` is unreliable on CPU before 2.11 (no kernel at all on 2.8), and the for-loop is correct. The change only touches the `torch <= 2.10` CPU branch, so `torch >= 2.11` (where `is_torch_less_or_equal("2.10.0")` is already `False`, and a real CPU grouped-matmul kernel exists) is unchanged.

There is no existing issue for this; it was found while running MoE models on CPU. The eager-vs-grouped MoE tests only run on accelerators, so the CPU path was uncovered; this PR adds a small model-free regression test that exercises `_grouped_mm` on CPU.

<details>
<summary>Reproduction and before/after (CPU)</summary>

Environment: `transformers` 5.13.0.dev0, Python 3.12.3, PyTorch 2.8.0+cu128 (CPU run), Linux x86_64. Reproduces on any `torch <= 2.10`, where `aten::_grouped_mm` has no CPU kernel.

```python
import torch
from transformers.integrations.moe import _grouped_mm

inp = torch.randn(6, 8)
w = torch.randn(3, 8, 5)
offs = torch.tensor([2, 4, 6], dtype=torch.int32)
_grouped_mm(inp, w, offs=offs)
# before: NotImplementedError: Could not run 'aten::_grouped_mm' with the 'CPU' backend
# after:  returns the per-expert grouped matmul
```

Real models hit the same crash on CPU. Running these on CPU, every case fails with `NotImplementedError` at `integrations/moe.py` before the fix and passes after:

```
tests/models/mixtral/test_modeling_mixtral.py    -k "left_padding or batching_equivalence"   # 2 fail -> 2 pass
tests/models/qwen3_moe/test_modeling_qwen3_moe.py -k "left_padding or batching_equivalence"   # 2 fail -> 2 pass
tests/models/olmoe/test_modeling_olmoe.py        -k "left_padding or batching_equivalence"   # 2 fail -> 2 pass
tests/models/jamba/test_modeling_jamba.py        -k "left_padding"                            # 1 fail -> 1 pass
```

New regression test (passes after, fails before):

```
tests/utils/test_modeling_utils.py -k test_grouped_mm_dispatcher_runs_on_cpu
```
</details>

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests?

## Who can review?

cc @IlyasMoutawwakil @3outeille


Comments:
Fix looks right but I'm not sure about the test - our CI is `torch 2.11` anyway, so I don't think we'll ever trip that bug! 

i guess we also discussed this last time and converged towards only falling back when really necessary so maybe we can just extend the condition with an or torch_is_less_than(2.8). otherwise we would be not using it where it's usable. cc @vasqu 

2.8 is probably the first version to include grouped mm. Agree we can make an or condition instead to avoid overly applying this on 2.9+ 

Thanks all, agreed. Switched to extending the existing condition with an `or` so 2.9/2.10 keep using the CPU kernel when the tensors are aligned, instead of always falling back.

One correction on the cutoff: I checked `torch._grouped_mm` on CPU across versions, and 2.8 is the only one with no CPU kernel at all (it raises `NotImplementedError` for any shape). 2.9+ do have a CPU kernel that just needs 16 byte alignment. So `or is_torch_less_than(2.8)` would still leave 2.8 itself crashing; the guard has to cover 2.8, i.e. `not is_torch_greater_or_equal("2.9")`. The condition now falls back on CPU when `torch < 2.9`, or when the existing `data_ptr % 16` check says the tensors are misaligned; 2.9/2.10 aligned tensors keep hitting the kernel.

<details>
<summary>Per-version CPU check</summary>

| torch | `torch._grouped_mm` on CPU |
| --- | --- |
| 2.8 | `NotImplementedError` (no CPU kernel, any shape) |
| 2.9, 2.10, 2.11, 2.12 | works when 16 byte aligned, else `RuntimeError: strides should be multiple of 16 bytes` |

For example on 2.9 and 2.11, `torch._grouped_mm(torch.randn(8, 16), torch.randn(2, 16, 32), offs=torch.tensor([4, 8], dtype=torch.int32))` returns a result on CPU, while the `(6, 8) x (3, 8, 5)` shape is not aligned and raises.
</details>

I also dropped the dedicated test. The crash only happens on torch <= 2.8 CPU, which CI on 2.11 never exercises, and those small unaligned shapes would raise the alignment `RuntimeError` on 2.11 rather than exercise the fallback, so the test was not a useful guard.


**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46743)

Hi @IlyasMoutawwakil, any updates on the review?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46743). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.