ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage2_v2

NCP-ArchPreview 8.9B - Stage 2

Model collection | Technical report | Training code (coming soon) | Evaluation code

NCP-ArchPreview Stage 2 continues the Stage 1 model on Dolma 3 Dolmino, following the second stage of the OLMo-3 data curriculum. It retains joint Next Token Prediction (NTP) and Next Concept Prediction (NCP), allowing the model to refine token-level capabilities and its learned concept space during continued pretraining.

Developed by The NCP Team at Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University, the model has approximately 8.94B parameters and an 8,192-token training context. Stage 2 remains a base-model release for completion, evaluation, and further adaptation.

Highlights

• Continued training in token and concept space. Stage 2 preserves the 16-layer Encoder, 8-layer Concept Module, and 16-layer Decoder from Stage 1.
• Improved main Stage 2 average. Table 1 reports 57.57 Overall AVG, compared with 56.98 for the corresponding OLMo-3-7B model, with 83.02 GSM8K and 68.48 MMLU.
• Three matched-budget data recipes. Appendix C compares V1, V2, and V3 under a common inference and sampling setup. V1 leads on the reported code, mathematics, academic-STEM, and logical-reasoning tasks; V2/V3 lead on HellaSwag.
• A Target for concept-conditioned drafting. Stage 2 V1 is the paired Target for the separate NCPFlash draft model.

The report also finds that Stage 2 reaches the OLMo-3-7B final training loss with 66.2% of the training tokens, corresponding to 1.51x convergence in token budget. This is a training-loss comparison, not an inference throughput measurement.

Stage 2 Releases

Variant	Hugging Face repository
V1	NCP_ArchPreview_dolma3_8.9B_Stage2_v1
V2	NCP_ArchPreview_dolma3_8.9B_Stage2_v2
V3	NCP_ArchPreview_dolma3_8.9B_Stage2_v3

These labels identify alternative Stage 2 data recipes. Higher version numbers do not imply a stronger model: although aggregate training loss decreases from V1 to V3, several downstream capabilities decline. The quick-start example uses V1, which is the strongest of these recipes across the free-form capabilities evaluated in Appendix C.

Evaluation

Main Stage 2 Comparison

The following values are the report's main Stage 2 comparison with OLMo-3-7B and should not be assigned to all three released variants. Scores are percentages; deltas are absolute percentage points.

Metric	OLMo-3-7B Stage 2	NCP-ArchPreview Stage 2	Delta
Overall AVG	56.98	57.57	+0.59
MMLU	66.66	68.48	+1.82
GSM8K	79.68	83.02	+3.34
GSM-Symbolic	57.32	60.32	+3.00
MATH-500	43.44	43.91	+0.47
HumanEval	49.31	45.62	-3.69
MBPP	48.98	50.85	+1.87
ARC-Challenge	85.49	83.28	-2.21
PIQA	78.35	81.45	+3.10

Results by domain and likelihood evaluation

Domain average	OLMo-3-7B Stage 2	NCP-ArchPreview Stage 2
MMLU family	58.32	59.94
Mathematics	55.63	57.39
Code	39.42	38.77
Multiple-choice STEM	89.65	88.67
Multiple-choice non-STEM	76.85	77.76
GenQA	53.49	54.04

Likelihood is reported separately in bits per UTF-8 byte (BPB), where lower is better.

Likelihood metric	OLMo-3-7B Stage 2	NCP-ArchPreview Stage 2
BPB AVG	0.793	0.763

The main Stage 2 model improves the overall average while showing a mixed profile across individual tasks, including lower HumanEval and ARC-Challenge scores. The report discusses a possible mismatch between the continued-training mixture and downstream domains; lower aggregate loss alone does not establish better performance on every task.

V1 / V2 / V3 Recipe Comparison

All three Stage 2 variants share the same inference and sampling configuration within this comparison.

Benchmark	v1	v2	v3
HumanEval	45.60	42.19	39.96
MBPP	50.91	49.34	46.60
MATH-500	43.74	41.66	37.21
Minerva	42.20	40.39	36.70
MMLU-STEM	61.84	59.85	57.72
BBH	63.23	62.90	60.44
HellaSwag	66.40	67.30	67.25

For recipe selection, the report treats V2 and V3 as effectively tied on HellaSwag because they differ by only 0.05 percentage points. Its held-out proxy analysis supports the V1 > V2 > V3 ordering on four free-form capability measurements. This evidence compares complete recipes and does not isolate the contribution of an individual data source.

Protocol and Aggregation

The main comparison follows the OLMo evaluation protocol in OLMo-Core. Appendix E and Table 13 specify the per-task settings, including:

• GSM8K: 8-shot, one completion per problem, pass@1.
• MATH-500: 4-shot, 32 completions per problem, pass@1.
• HumanEval and MBPP: 3-shot, 32 completions per problem, execution pass@1.

The sample count is the number of generated completions per problem, not the inference batch size. Overall AVG is the unweighted mean of 26 constituent benchmark scores, excluding the aggregate MMLU row and domain averages. The ten-benchmark BPB AVG is calculated separately and is excluded from Overall AVG. See the evaluation configuration for the corresponding release assets and complete runtime settings.

Architecture

NCP-ArchPreview processes text through three modules:

1. A 16-layer Token Encoder produces contextual token states. Mean pooling over each group of four states forms a continuous concept representation.
2. An 8-layer Concept Module predicts the next concept. Product quantization defines the concept vocabulary using 32 codebooks. Predictions are differentiable weighted combinations of codewords.
3. A 16-layer Token Decoder receives token states and causally aligned concept predictions, then produces the next-token distribution.

Intra-module residual connections mix states across depths. Cross-module residual connections connect Encoder to Concept Module, Encoder to Decoder, and Concept Module to Decoder. Concept feedback is shifted and repeated at token resolution to preserve causality.

tokens -> Token Encoder -> mean pooling -> Concept Module -> concept prediction
                 |                                                |
                 +--------------> Token Decoder <-----------------+
                                       |
                                next-token logits

Property	Configuration
Hugging Face architecture	NCPOlmo3ForCausalLM
Total parameters	Approximately 8.94B
Encoder / Concept Module / Decoder	16 / 8 / 16 causal Transformer layers
Hidden size	4,096
FFN intermediate size	11,008
Attention heads / KV groups	32 / 32
Attention head dimension	128
Vocabulary size	100,278
Maximum training context	8,192 tokens
Token-level attention	4,096-token local window; full attention every fourth layer
Position encoding	RoPE, base 500,000
Activation / normalization	SwiGLU / RMSNorm; layer-wise QK RMSNorm
Concept compression	4 token states per concept
Product quantization	32 codebooks, each with 128 codewords of dimension 128
Parameter precision	BF16

Architecture details follow Sections 2-3 and Appendix B, Table 10, of the technical report.

Training and Data

Stage 2 continues the Stage 1 checkpoint with the Dolma 3 Dolmino data curriculum described in Section 4.1. The underlying data family is documented in the Dolma 3 Dolmino pool. The pool describes available source data; the exact sampling mixture belongs to each Stage 2 recipe.

The Encoder, Concept Module, Decoder, and learned concept vocabulary continue to train jointly using NTP, NCP, and VQ objectives. The report describes Muon for matrix-valued parameters and AdamW for other parameter groups, with a cosine learning-rate schedule.

Quick Start

The checkpoint includes custom Transformers model code. This example uses a single prompt on one CUDA GPU with sufficient memory for the BF16 weights, activations, and cache.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage2_v1"
device = "cuda"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).eval().to(device)

prompt = "The role of hierarchical representations in language modeling is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        use_cache=True,
        pad_token_id=tokenizer.eos_token_id,
    )

continuation = output_ids[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(continuation, skip_special_tokens=True))

Use plain completion or few-shot prompts for this base model. The example is a loading and generation example; benchmark reproduction requires the prompts, sampling configuration, and scorers described in the report. For reproducible runs, pin the model and tokenizer to the same Hub commit with revision. See the inference guide for supported runtime versions and optimized serving.

Using Stage 2 V1 with NCPFlash

The separate NCPFlash repository contains a roughly 1.1B-parameter DFlash2-style drafter paired with Stage 2 V1. It uses Target hidden states and a causal concept signal to propose blocks for speculative verification.

Section 5.3 reports an increase in mean accepted length from 5.933 to 6.180 (+4.17%) relative to the matched drafter without concept conditioning. This measures tokens committed per verification round. End-to-end throughput requires a separate measurement with the paired models and serving runtime. See the NCPFlash inference guide for use of the two checkpoints together.

Intended Use and Limitations

The Stage 2 releases support research on continued pretraining, latent-space modeling, data-recipe selection, benchmark evaluation, and further adaptation. Stage 2 V1 also serves as the Target for the paired NCPFlash drafter.

• These are base models, without conversational alignment. They may produce inaccurate, biased, or harmful content.
• A lower training loss or a higher version number is not a reliable capability ranking. Select a recipe using evaluations relevant to the intended task.
• The reported maximum training context is 8,192 tokens. Longer-context capability requires separate evaluation.
• Results from one variant or evaluation setup should not be attributed to another. The VQ adaptation and 3B-scale MTP studies in the report concern separate experimental checkpoints.

Citation

@techreport{ncpteam2026archpreview,
  title = {{NCP-ArchPreview} Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction},
  author = {{The NCP Team}},
  institution = {Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University},
  year = {2026},
  month = sep,
  note = {Technical report dated September 4, 2026}
}

License

The model weights are released under the Apache License 2.0.

Acknowledgements

We thank the OLMo and Dolma teams and the contributors to the open datasets, training libraries, and evaluation tools used in this work.