ForgePlex-M2.2-10M

ForgePlex-M2.2-10M is the definitive M2: the final and most refined release of the ForgePlex M2 architecture from ForgeWorks. Same ~9.99M-parameter (unique, tied embeddings) decoder-only design as M2 and M2.1, pushed as far as we could take it. Packaged as a Hugging Face custom causal LM (requires trust_remote_code=True).

M2.2 improves on M2.1 on HellaSwag, ARC, PIQA and ArithMark, and the Intelligence Index climbs from 10.70 to 12.12.

Unique parameters	9,991,938
Post-training	2B tokens on top of M2.1
License	Apache-2.0

Architecture

GQA + NeoX-style RoPE + RMSNorm + SwiGLU, with Qwen3.5-style attention output gates on every layer and GPT-S2-style refresh gates on inject layers [5, 10] (kernel 9). XSA is off. Weights keep the original key layout (no Llama remapping).

Component	Details
Position encoding	RoPE (theta=5,000, NeoX even/odd)
Normalization	RMSNorm (eps=1e-6)
Feed-forward	SwiGLU (intermediate 712)
Attention	GQA: 8Q / 2KV, head_dim=32 + attn output gate
Refresh	Layers 5, 10, kernel 9
Bias	None
Embedding	Weight tying
Depth x width	11 layers x 256 hidden
Context	1024 tokens
Vocab	4,096 custom BPE

Evaluation

All scores are 0-shot acc_norm. HellaSwag, ARC and PIQA are run with lm-eval; ArithMark-3 is reported as acc_norm. ARC is the mean of ARC-Easy and ARC-Challenge.

Benchmark	ForgePlex-M2.2-10M
HellaSwag	29.19
ARC-Easy	39.77
ARC-Challenge	22.70
ARC (avg)	31.24
PIQA	59.79
ArithMark-3	37.40
Intelligence Index	12.12

Compared with earlier M2 releases

Model	HellaSwag	ARC-e	ARC-c	ARC (avg)	PIQA	ArithMark-3	Index
ForgePlex-M2	28.02	-	-	30.00	57.18	34.60	9.14
ForgePlex-M2.1-10M	29.09	36.45	22.87	29.66	59.14	35.50	10.70
ForgePlex-M2.2-10M	29.19	39.77	22.70	31.24	59.79	37.40	12.12

Against M2.1, M2.2 is up on HellaSwag (+0.10), ARC (+1.58 avg, driven by +3.32 on ARC-Easy, with ARC-Challenge roughly flat at -0.17), PIQA (+0.65) and ArithMark (+1.90), for +1.42 on the Index. A balanced, jack-of-all-trades step up rather than a spike on one benchmark.

M2.2 leads on HellaSwag, ARC-Easy and PIQA against the strongest sub-10M models, while staying close on ARC-Challenge and ArithMark.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ForgeWorks/ForgePlex-M2.2-10M"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
    device_map="auto",
)

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Or run python usage.py from a local copy of this repo.

Citation

@misc{forgeplex_m22_10m,
  title={ForgePlex-M2.2-10M},
  author={ForgeWorks},
  year={2026},
  howpublished={\url{https://huggingface.co/ForgeWorks/ForgePlex-M2.2-10M}}
}