Maincode/Matilda-Platypus-7B-Base

Matilda Platypus 7B (base)

Any size you want. Matilda Platypus is a family of intermediate-size language models built by connecting the lower layers of a larger pretrained Qwen3 model to the upper layers of a smaller one. A learned linear Representation Bridge aligns the hidden representations at the connection. This checkpoint is the 7B member (cut 25+11): 6.96B parameters, 13.3 GFLOP per token, 84% of the larger parent's compute. It is a base model (no instruction tuning); see the technical report (technical_report.pdf) for the method and the full evaluation.

How this model is built

Component	Source	Layers	Notes
Lower block	Qwen/Qwen3-8B-Base (36 layers)	1–25	frozen at the released weights, loaded from the hub
Representation Bridge	learned	–	one linear map, 4096 → 2560 hidden units, 10.5M parameters (bridge.safetensors)
Upper block	Qwen/Qwen3-4B-Base (36 layers)	26–36 + final norm + output head	lightly adapted with low-rank updates (merged), shipped as upper_block/

The notation 25+11 means 25 layers of the larger parent followed by 11 layers of the smaller parent. The forward pass is: embedding and layers 1–25 of Qwen3-8B-Base → bridge → layers 26–36 of Qwen3-4B-Base → its final normalisation and output head. The tokenizer is the Qwen3 tokenizer.

Results

Base-model evaluation with lm-eval-harness 0.4.13, identical protocol for the model and both parents (0-shot multiple choice, MMLU 5-shot, GSM8K 8-shot chain-of-thought, BBH 3-shot chain-of-thought, HumanEval 0-shot and MBPP 3-shot with the generated code executed; HumanEval scored with markdown fences stripped). Scores are proportions.

Benchmark	Qwen3-4B-Base	Platypus 7B	Qwen3-8B-Base
HumanEval pass@1	0.628	0.689	0.677
GSM8K (8-shot CoT)	0.844	0.875	0.862
MMLU (5-shot)	0.730	0.763	0.768
BBH (3-shot CoT)	0.712	0.746	0.774
HellaSwag	0.737	0.758	0.786
ARC-Easy	0.761	0.782	0.798
ARC-Challenge	0.517	0.527	0.572
OpenBookQA	0.406	0.418	0.416
WinoGrande	0.702	0.710	0.723
PIQA	0.778	0.784	0.794
LAMBADA	0.691	0.693	0.711
MBPP pass@1	0.672	0.668	0.696
BoolQ	0.832	0.820	0.832

Above the smaller parent on 11 of 13 benchmarks and above the larger parent on 3 of 13, at 84% of the larger parent's compute (13.3 vs 15.7 GFLOP per token; the smaller parent is 8.7). Compute is the report's relative measure, normalised to the larger parent; it is not a verified latency or cost measurement. Full tables and figures: technical_report.pdf.

Training recipe (summary)

1. Initialise the bridge by least squares. Both released parents are run on a small calibration set and a linear map is fitted between their representations at the connection; initial language-modelling loss ranks candidate cuts.
2. Supervised adaptation. The bridge and light low-rank updates to the retained upper layers are trained on a few billion tokens of mixed web text, code, mathematics and worked solutions, with the lower block frozen at its released weights.
3. Reinforcement learning with verifiable rewards. A short policy-optimisation stage on code problems with unit tests and mathematics problems with checkable answers.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Maincode/Matilda-Platypus-7B-Base"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).cuda()
inputs = tok("The capital of Australia is", return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=32)
print(tok.decode(out[0], skip_special_tokens=True))

The remote code loads the lower block from Qwen/Qwen3-8B-Base, the upper block from upper_block/ and the bridge from bridge.safetensors, truncates both parents to the retained layers and runs them through their standard transformers forward passes with one KV cache per block. generate follows the standard transformers keyword arguments (max_new_tokens / max_length, min_new_tokens, stopping_criteria, logits_processor, do_sample, temperature, top_k, top_p, repetition_penalty, eos_token_id, pad_token_id, streamer), so the model runs unchanged under lm-eval-harness (--model hf --model_args pretrained=Maincode/Matilda-Platypus-7B-Base,trust_remote_code=True). Greedy and sampled decoding only; beam search raises NotImplementedError. Requires transformers >= 4.56 (tested on 4.56 and 5.x).

Remote-code revision 2 (2026-10-31): revision 1 ignored max_length and stopping_criteria and always produced 128 new tokens, which truncated chain-of-thought answers under lm-eval and under-reported generative benchmarks (GSM8K read ~0.58 instead of the table value); tie_weights() is now a no-op at the top level. Model weights are unchanged.

Limitations

A base model: it continues text and is not instruction-tuned or safety-tuned. Its knowledge and behaviour derive from the two Qwen3 parents and the adaptation data. The evaluation above is a base-model protocol shared by the model and its parents; differences of a few tenths of a point are within run-to-run variance.

Citation

Maincode Team (Yue Yang, Arsh Chawla, David Lemphers). Matilda Platypus: Any Size You Want. Intelligence You Need. Technical report, Maincode / Matilda Research, October 2026. Included in this repository as technical_report.pdf. Built on Qwen3 (Qwen Team, Qwen3 Technical Report, arXiv:2505.09388). Released under the Apache 2.0 license.