Nanbeige/Nanbeige4.2-3B-Base

1. Introduction

Nanbeige4.2-3B-Base is a new base model in the Nanbeige4 series, trained from scratch with improvements to both model architecture and training data. It delivers substantial gains over the previous generation and outperforms opensource base models of comparable scale, including Qwen3.5-4B and Gemma4-E4B, across a broad range of benchmarks. Key features include:

• Looped Architecture: Adopts a Looped Transformer architecture, in which hidden states are fed back into the same Transformer layers after completing a bottom-to-top pass. The reuse schema significantly increases the model capacity without increasing the number of parameters.

• Enhanced Training Data Recipe: Expands the total amount of training data from 23T to 28T tokens, with more aggressive upsampling of math, code, STEM, and Synthetic-QA data to strengthen reasoning and coding capabilities.

The accompanying modeling_nanbeige.py also includes our latest architectural improvements, including LoopSplit, mHC with depth attention, and concatenated n-gram embeddings. These features have been incorporated into Nanbeige4.5, whose training is underway for release later in 2026.

2. Model Performance

We compare Nanbeige4.2-3B-Base with other base models of comparable parameter scale on a diverse benchmark suite covering mathematical reasoning, general reasoning, code generation, and knowledge-intensive tasks:

Model	Parameters		Benchmarks
	Total Params	Non-emb. Params	GSM8K	BBH	MBPP	MMLU-Pro	SuperGPQA	GPQA
Gemma4-E4B-Base	8B	4B	61.8	62.5	53.5	37.6	23.3	27.5
Qwen3.5-4B-Base	5B	4B	84.4	79.1	57.1	51.8	32.1	43.1
Nanbeige4.1-3B-Base	4B	3B	85.9	70.7	60.7	47.6	24.8	36.2
Nanbeige4.2-3B-Base	4B	3B	92.7	81.6	67.6	63.8	35.2	53.3

The results demonstrate that Nanbeige4.2-3B-Base consistently outperforms Gemma4-E4B-Base, Qwen3.5-4B-Base, and our previous Nanbeige4.1-3B-Base across the benchmark suite.

3. Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
  'Nanbeige/Nanbeige4.2-3B-Base',
  use_fast=False,
  trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
  'Nanbeige/Nanbeige4.2-3B-Base',
  torch_dtype='auto',
  device_map='auto',
  trust_remote_code=True
)

prompt = "The capital of China is"
input_ids = tokenizer(prompt, return_tensors='pt').input_ids
output_ids = model.generate(input_ids.to('cuda'))
resp = tokenizer.decode(output_ids[0][len(input_ids[0]):], skip_special_tokens=True)
print(resp)

4. Limitations

While we place great emphasis on model safety throughout the training process, the model may still generate unexpected or inappropriate outputs due to its probabilistic nature. Such outputs may include inaccurate information, bias, discrimination, or other harmful content. Please do not propagate such content. We do not assume responsibility for consequences of disseminating inappropriate information.

5. Citation

If you find our model useful or would like to use it in your own work, please cite this Hugging Face project:

@misc{nanbeige2026,
  title        = {Nanbeige4.2-3B-Base},
  author       = {Nanbeige Team},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Nanbeige/Nanbeige4.2-3B-Base}}
}

6. Contact

If you have any questions, please open an issue in this repository or contact us at nanbeige@kanzhun.com.