Add MiniCPM3
Author: bzantium
Pull request: yes
Labels: name: New model

# What does this PR do?

Adds native support for the **MiniCPM3** architecture from OpenBMB
([`openbmb/MiniCPM3-4B`](https://huggingface.co/openbmb/MiniCPM3-4B)).

MiniCPM3 combines:

- **Multi-head Latent Attention (MLA)** from DeepSeek-V2, with the original cos/sin rotary embedding convention preserved on the rotary part of the query/key heads.
- A standard SwiGLU MLP (no MoE).
- Three scalar scaling factors:
  - `scale_emb` — multiplier on input embeddings.
  - `scale_depth / sqrt(num_hidden_layers)` — multiplier on residual connections.
  - `hidden_size / dim_model_base` — divisor on hidden states before the LM head.

When `scale_depth` / `dim_model_base` are not provided, they default to no-op values (`sqrt(num_hidden_layers)` and `hidden_size` respectively).

Implemented as a single `modular_minicpm3.py` deriving from `LlamaConfig` / `LlamaModel`, with a custom `MiniCPM3Attention` for MLA. The model reuses the upstream `LlamaTokenizer` (no new tokenizer files); `minicpm3` is added to `MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS` so the auto loader falls back to `TokenizersBackend`. `AutoConfig` / `AutoModel` / `AutoModelForCausalLM` / `AutoModelForSequenceClassification` mappings, docs, and tests are wired up.

Fixes #41115.

A second, more recent PR (#45613) covers the same model. This PR (#41116) is the original; the implementations differ on the RoPE convention (this one preserves the original cos/sin RoPE; #45613 uses DeepSeek-V2's complex RoPE) and on a few config defaults.

AI assistance was used during the rebase/refactor; the changes have been reviewed and tested locally end-to-end (see follow-up comment for the example run on `openbmb/MiniCPM3-4B`).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. — #41115
- [x] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests?

## Who can review?

@ArthurZucker @Cyrilvallez

Comments:
cc @arthurzucker for text models

@ArthurZucker please check this out! I've tested with MiniCPM3-4B and it worked well.

@ArthurZucker I've tested MiniCPM3-4B and our in-house model for the case `q_lora_rank is None`. Please review this when you are available and let me know if more needed! thanks a lot :) I hope this can be merged because we are going to open-weights one of models using this modeling in the near future.

End-to-end check on `openbmb/MiniCPM3-4B`:

```python
import torch
from transformers import AutoTokenizer, MiniCPM3ForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openbmb/MiniCPM3-4B")
model = MiniCPM3ForCausalLM.from_pretrained(
    "openbmb/MiniCPM3-4B",
    dtype=torch.float16,
    device_map="auto",
    attn_implementation="eager",
).eval()

messages = [{"role": "user", "content": "Tell me a fun fact about the Eiffel Tower."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

Output:

```
user
Tell me a fun fact about the Eiffel Tower.
 assistant
The Eiffel Tower was originally intended to be a temporary structure, designed for the 1889 World's Fair to celebrate the 100th anniversary of the French Revolution. It was meant to be a symbol of the Industrial Revolution and a demonstration of France's engineering prowess. However, it was so popular that it was allowed to stand after the Fair. It
```

@aliyevaladddin Sorry I missed your email earlier. I just rebased this PR, would be happy to add you as coauthor since your message got me to come back to it.

@ArthurZucker @Cyrilvallez when you have a moment, would appreciate a review on this. Thanks!

Thanks for the rebase, @bzantium! I'm honored to be added as a co-author. I'm really excited to see MiniCPM3 support getting closer to being merged. Let me know if there's anything else I can help with regarding testing or further updates!

Will try to take a look today :hugs: 

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41116&sha=42b80e

We also need to sync with main and fix CI please 

@vasqu 

The `security-gate / security-check` failure comes from the "Enforce allowlist" step. The only offending file is `docs/source/en/_toctree.yml`, which the gate rejects because `.yml` is not allowed outside `tests/` (only `.py`, `.txt`, `.md` are permitted there). That `_toctree.yml` entry is required to register MiniCPM3 in the docs navigation, so it cannot be dropped.

This gate only runs for fork PRs, and the failure comment notes that a maintainer can manually approve CI when a finding is a false positive. Could you approve CI when you get a chance? All 11 other changed files pass the allowlist. Thanks!


Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`) or has a disallowed extension (only `.py`, `.txt`, `.md` are permitted outside `tests/`)
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/27735957407) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

gentle ping @bzantium 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, minicpm3

@vasqu Thanks for the careful check! I verified on an A100 (compute capability 8.0, same hardware the `("cuda", 8)` reference came from) using the current branch, which includes your modular cleanup. Nothing is broken.

**bf16 reproduction (A100, cc 8.0):**
```
slice : [0.765625, 3.640625, -0.189453, -0.835938, -0.835938]
ref   : [0.765625, 3.640625, -0.189453125, -0.8359375, -0.8359375]
diff  : [0, 0, 0, 0, 0]   # exact match
```
So your refactor (inheriting `Gemma3TextScaledWordEmbedding` and dropping the duplicated `MiniCPM3Model.forward`) is numerically a no op. That matches the code too: the new embedding base is identical to the old one, and the removed `forward` was a verbatim copy of the inherited `LlamaModel.forward`.

**On the 8 vs 8.6 gap:** both buckets run bf16 (the config `torch_dtype` is `bfloat16`, and the test loads with `dtype="auto"`), so the difference is purely cross GPU bf16 variation, not a dtype or code change. These particular logits sit near zero and the model is 62 layers deep, so bf16 rounding accumulates and the low magnitude entries wobble. To put a number on it, I also ran fp32 as a ground truth on the same A100:
```
fp32  : [0.535951, 3.610042, -0.174161, -1.174419, -1.174420]
```
The bf16 vs fp32 gap reaches about 0.34, which is larger than the 8.0 vs 8.6 gap of about 0.08. Most importantly, the argmax token is 11213 in bf16 and fp32 alike, so the actual prediction is stable across precision and only the near zero logit tail moves.

Net: your `(8, 6)` bucket is a legitimate per hardware entry and the existing `(8)` reference still reproduces exactly. No regression. Good to go from my side.


run-slow: minicpm3

Just sanity checking but should be good to go then :saluting_face: 

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28020375684)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/minicpm3"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28020375684)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [bc3a358c](https://github.com/huggingface/transformers/commit/bc3a358c70fb999098f039fd521b0a813feb66c7) | workflow commit (merge commit) |
| PR | [baf1d586](https://github.com/bzantium/transformers/commit/baf1d586961663ae30c1f71a5c3de13f262def35) | branch commit (from PR) |
| main | [43639365](https://github.com/huggingface/transformers/commit/4363936534ef47721b84a95bfdc758f056c3ee98) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41116). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Nice thanks for sticking through and gz on the merge :hugs: 

Finaly!