Add support for BharatGen's Param2MoE model architecture
Author: bhargav-patel-29
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46395)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46395)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

This PR adds support for **[Param-2-17B-MoE-A2.4B](https://huggingface.co/bharatgenai/Param2-17B-A2.4B-Thinking)**, a large-scale Mixture-of-Experts (MoE) causal language model.

**Param-2-17B-MoE-A2.4B** uses a **Hybrid Dense + MoE architecture** with **17B total parameters while activating only 2.4B parameters per token**, enabling high model capacity with efficient inference cost.

The model is pretrained from scratch with strong multilingual capabilities and particular emphasis on linguistic diversity and Indian language representation. It is released as a pretrained base model intended for downstream fine-tuning.

This integration enables seamless loading, inference, and downstream fine-tuning using standard Transformers APIs.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
Text models: @ArthurZucker @Cyrilvallez

Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46395&sha=23356e

Hey @ArthurZucker the `test_tp_generation_quantized` failure on `Param2MoEModelTest` (37% match, threshold 75%) is something I'd like your input on before merging.

What we found:
After layer-by-layer instrumentation, the issue originates in the attention block, not the MoE routing. Float8 weight-only TP quantization introduces a consistent ~12% relative error in attn_out hidden states. This cascades under greedy decoding once one token flips, all subsequent tokens diverge (autoregressive amplification), hence the low match ratio.

Key facts confirming this is not a TP correctness bug:
- `test_tp_forward`, `test_tp_backward`, `test_tp_generation` all pass 
- Weight sharding is verified correct (shapes confirmed via forward hooks)
- Router picks identical expert indices on TP and non-TP paths
- The `q_norm`/`k_norm` TP plan entries are copied directly from Qwen3/Qwen3MoE (which pass this test), but Param2MoE's test config has fewer heads, giving worse relative quantization error per rank

I saw the `NOTE(3outeille)` comment in `test_tensor_parallel_mixin.py `acknowledging this is model-dependent. My proposed fix is to override `test_tp_generation_quantized` in `Param2MoEModelTest` with a relaxed threshold of 0.30 (above the observed 37%, well below what a real TP bug would produce).

Is this the right approach, or would you prefer a different strategy (e.g. exposing `match_threshold` as a parameter in `_test_tp_generation_quantized_impl` so model-specific overrides don't need to copy the whole function)?

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/29472245901) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, param2moe

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46395)
**Latest run:** [29566770921:2](https://github.com/huggingface/transformers/actions/runs/29568537285)
**Result:** `success` | **Jobs:** 15 | **Tests:** 172,484 | **Failures:** 1 | **Duration:** 14h 13m

<!-- ci-dashboard-recap:end -->