ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage2_DFlash2_NCPFlash

NCPFlash - Stage 2 DFlash2 + NCP

Model collection | Target model | Technical report | Inference guide | Training code (coming soon)

NCPFlash is a concept-conditioned, block-parallel draft model for NCP-ArchPreview Stage 2 V1, developed by The NCP Team at Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University. It combines a DFlash2-style drafter with information from the Target model's learned concept space, allowing a single draft forward pass to propose a block of future tokens for Target verification.

This repository contains the approximately 1.1B-parameter drafter. The 8.9B in its repository name identifies the paired Target model's scale. NCPFlash requires that Target's tokenizer, token embeddings, output head, hidden states, and causal concept signal to perform speculative decoding.

Component	Repository
Draft model	NCP_ArchPreview_dolma3_8.9B_Stage2_DFlash2_NCPFlash
Paired Target	NCP_ArchPreview_dolma3_8.9B_Stage2_v1

Highlights

• Block-parallel proposals. The report evaluates a horizon of 16 draft tokens per proposal.
• Features from several Target depths. Each draft layer learns its own weighted combination of five Target hidden representations.
• Causal concept conditioning. A normalized concept signal from the last fully completed Target chunk is added through learned gates at every draft layer. This adds only 0.04M parameters, with no additional Target-model computation in the report's comparison.
• Longer accepted sequences. Mean accepted length increases from 5.933 to 6.180 (+4.17%) over the matched baseline, with the largest relative gain on HumanEval (+7.59%).

Architecture

The drafter combines three components:

1. DFlash2-style local mixing and path selection. A two-tap dynamic convolution models dependencies between adjacent proposal positions. A lightweight path selector chooses a coherent sequence from the top-16 token candidates at each position.
2. DFlare-style layer-wise Target fusion. Every draft layer learns a different mixture of five Target hidden states, exposing information from multiple representation depths.
3. NCP concept feedback. A per-layer RMSNorm and a tanh-gated residual inject the causal concept representation into each proposal position. Gates start at zero during training.

verified prefix -> NCP-ArchPreview Stage 2 v1
                         |              |
                   hidden states   causal concept signal
                         |              |
                         +-> NCPFlash <-+
                                |
                       block of draft proposals
                                |
                    exact Target verification
                                |
                  committed tokens -> next round

Property	Configuration
Hugging Face architecture	ConceptLMDFlashModel
Auto class	AutoModel
Drafter parameters	Approximately 1.1B
Draft Transformer layers	5
Draft / Target hidden size	4,096 / 4,096
FFN intermediate size	11,008
Attention heads	32
Vocabulary size	100,278; shared with the Target tokenizer
Maximum training sequence length	8,192 tokens
Proposal horizon / exported block size	16
Local mixer	Dynamic convolution, kernel size 2
Proposal selection	Path selector, top-16 candidates per position
Target feature layer IDs	[1, 4, 7, 10, 13]
Draft initialization layer IDs	Target decoder layers [0, 4, 8, 12, 15]
Target fusion	Learned separately for each draft layer
Concept conditioning	Causal residual; Target chunk size 4
Added concept-conditioning parameters	40,960, approximately 0.04M
Position encoding	RoPE, base 500,000
Parameter precision	BF16
Exported attention backend	PyTorch FlexAttention

The implementation settings are recorded in the published configuration and model code. The checkpoint stores the drafter's own parameters and relies on the paired Target for token embeddings and the language-model output head.

Training

The Baseline and Baseline + Concept variants in the report share the same online Target-distillation pipeline, training data order, random seed, and training budget. The controlled change is the addition of the causal concept-conditioning path.

Training setting	Value
Target	NCP-ArchPreview Stage 2 v1
Training approach	Online Target distillation
Fixed training subset	5B tokens
Epochs over the subset	10
Sequence length	8,192
Training anchors per sequence	512
Global batch size	512

Here, online distillation means that Target representations are computed during training on fixed corpus sequences. It does not imply that the drafter generates a new on-policy corpus. The drafter's training subset is distinct from the Target's large-scale pretraining budget.

Evaluation

Table 9 compares the matched baseline drafter with the concept-conditioned variant using exact speculative verification and a 16-token proposal horizon.

Benchmark	Baseline	Baseline + Concept (NCPFlash)	Relative gain
GSM8K	6.351	6.537	+2.93%
MATH	6.105	6.240	+2.22%
HumanEval	5.432	5.845	+7.59%
MBPP	5.844	6.099	+4.37%
Macro average	5.933	6.180	+4.17%

What Mean Accepted Length Measures

The reported metric is mean accepted length (MAL):

MAL = total tokens committed across verification rounds / number of rounds

A round's committed count includes a Target correction or bonus token when present. Thus, MAL measures committed tokens per round, rather than only the number of draft tokens accepted. The initial Target token produced during prefill is excluded under the release's evaluation convention. The macro average is the unweighted mean of the four benchmark MAL values.

The baseline already includes the DFlash2-style mixer and path selector and the DFlare-style Target-feature fusion. The reported gain isolates adding concept conditioning to that baseline; it is not a comparison with Target-only autoregressive throughput.

Evaluation Scope

The published checkpoint evaluation notes specify a matched evaluation set with common Target, prompts, sampling, and BF16 execution. Cases where either compared drafter produces a repeated-output loop or reaches the generation limit without valid stopping are excluded. The table therefore describes that matched set, rather than an unrestricted evaluation of every generation.

The 4.17% improvement is a relative MAL gain. It is not a 4.17% increase in task accuracy or a measured end-to-end speedup. Throughput depends on Target and draft cost, verification, batch size, prompt length, and runtime. See the evaluation protocol and results for the complete reproduction setup.

Exact speculative verification can preserve the Target distribution when the verifier and Target-state updates are correct. The report's exact result applies to its evaluated implementation. Other backends, caching strategies, or approximate verification modes require their own correctness and quality evaluation.

Loading the Models

The draft export uses AutoModel and custom Transformers code. Its FlexAttention implementation requires a compatible PyTorch/CUDA environment. The following example loads both checkpoints and the Target tokenizer on one GPU with sufficient memory.

import torch
from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer

target_id = "ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage2_v1"
draft_id = (
    "ArchSpace-Collection/"
    "NCP_ArchPreview_dolma3_8.9B_Stage2_DFlash2_NCPFlash"
)
device = "cuda"

tokenizer = AutoTokenizer.from_pretrained(target_id)
target = AutoModelForCausalLM.from_pretrained(
    target_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).eval().to(device)
drafter = AutoModel.from_pretrained(
    draft_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).eval().to(device)

Loading these modules is the first step. To generate text with NCPFlash, the speculative-decoding runtime must extract the required Target features, construct proposals, verify them, and update both models' state. Calling target.generate() alone uses the Target's ordinary generation path; it does not connect the drafter automatically. The draft module does not expose a standalone text-generation interface.

Use the NCPFlash inference guide for the complete generation entry point and supported dependency versions. For reproducible runs, pin the Target, its tokenizer, and the drafter to their respective Hub commit revisions.

Draft forward interface for runtime integration

The released ConceptLMDFlashModel.forward consumes Target features and embeddings supplied by the runtime:

Argument	Shape
aux_hidden_states	[batch, sequence, 5, 4096]
anchor_embeddings	[batch, anchors, 4096]
mask_embedding	[4096]
anchor_positions	[batch, anchors]
sequence_lengths	[batch]
hlm_hidden_states	[batch, anchors, 4096]

hlm_hidden_states carries the causally available concept signal. The runtime must preserve the export's feature ordering and select concepts only from completed Target chunks. See the published model code for the full signature.

Intended Use and Limitations

NCPFlash supports research on block-parallel speculative decoding, the use of concept representations for drafting, and inference systems for the paired NCP-ArchPreview Target.

• The release is paired with Stage 2 V1. Matching tensor dimensions alone does not establish compatibility with V2, V3, or another Target.
• Total inference memory includes both models, the Target cache, and cached hidden representations; the drafter's parameter count alone understates it.
• The reported training sequence length is 8,192 tokens. Long-context and different-batch behavior require separate evaluation.
• The Target is a pretrained base model. Speculative decoding does not add instruction tuning, alignment, or protection against inaccurate, biased, or harmful output.
• Higher MAL does not guarantee lower latency for every workload. Report throughput and correctness together for the actual serving configuration.

Citation

@techreport{ncpteam2026archpreview,
  title = {{NCP-ArchPreview} Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction},
  author = {{The NCP Team}},
  institution = {Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University},
  year = {2026},
  month = sep,
  note = {Technical report dated September 4, 2026}
}

License

The model weights are released under the Apache License 2.0.

Acknowledgements

This work builds on OLMo 3, DFlash2, DFlare, and the speculative-decoding research ecosystem. The report follows the DSpark convention for mean accepted length. Related references are listed in the technical report.