Noema 1.5 2B

Noema 1.5 2B is an open-weight, local-first language model optimized for broad knowledge retention, precise instruction following, multi-turn constraint persistence, short-form code generation, and efficient reasoning.

It is the most balanced release in the Noema 2B lineage.

Compared with the previous Noema release, Noema 1.5 improves pooled knowledge, MMLU-Pro development, HumanEval+, strict IFEval, verified mathematics, and novel constraint following. Against stock Qwen3.5-2B in final evaluation, it improves strict IFEval by 7.02 points, complete three-turn Multi-IF success by 3.11 points, HumanEval+ by 3.05 points, and reasoning efficiency by 13.5%, while passing a preregistered full MMLU-Pro knowledge-retention gate.

No retrieval, external tools, test-time answer repair, or external model calls were used in the reported benchmark results.

Highlights

  • Knowledge recovery: +1.42 points on the 1,900-item pooled knowledge development composite over Noema v2.
  • MMLU-Pro recovery: +2.86 points over v2 on the matched development panel.
  • Stronger code: HumanEval+ increased from 50.00% to 53.66% over v2.
  • Better instructions: strict IFEval increased from 70.24% to 73.01% over v2.
  • Better multi-turn behavior: complete three-turn Multi-IF success was 21.1% higher than stock Qwen3.5-2B.
  • Efficient reasoning: 13.5% fewer completion tokens than stock Qwen3.5-2B on the final thinking panel.
  • Broad knowledge retained: passed the preregistered MMLU-Pro non-inferiority gate across 11,332 untouched questions.
  • Open-weight deployment: downloadable weights built for private and resource-conscious inference.

Release-over-release results

The table below compares Noema 1.5 with the exact previous Noema checkpoint from which training began.

Benchmark Noema v2 Noema 1.5 2B Delta
Pooled knowledge composite 61.84 63.26 +1.42 pp
MMLU-Pro development 50.14 53.00 +2.86 pp
HumanEval+ pass@1 50.00 53.66 +3.66 pp
IFEval prompt strict 70.24 73.01 +2.77 pp
Verified math development 78.00 92.00 +14.00 pp
Novel constraints prompt strict 45.83 51.67 +5.83 pp

These are paired development and selection results. They demonstrate release-over-release progress but are not untouched final-lockbox estimates.

Final benchmark results

The final evaluation compared the frozen Noema 1.5 candidate with the exact stock Qwen3.5-2B foundation using identical prompts, generation settings, and graders.

Benchmark Samples Stock Qwen3.5-2B Noema 1.5 2B Delta
MMLU-Pro 11,332 53.88 53.17 -0.71 pp
GPQA-Diamond 198 45.45 40.40 -5.05 pp
HumanEval+ pass@1 164 50.61 53.66 +3.05 pp
IFEval prompt strict 541 65.06 72.09 +7.02 pp
IFBench prompt loose 300 31.00 27.33 -3.67 pp
Multi-IF mean per turn 4,501 conversations 33.49 35.84 +2.36 pp
Multi-IF all three turns 4,501 conversations 14.77 17.88 +3.11 pp
Thinking native accuracy 300 67.33 68.67 +1.33 pp
Thinking mean tokens 300 7,968.95 6,894.59 -13.48%

Statistical interpretation

  • MMLU-Pro: passed the preregistered non-inferiority test. The one-sided 95% lower bound was -1.438 points against a -1.5-point margin.
  • IFEval: +7.02 points, statistically significant (p = 0.000475).
  • Multi-IF: +3.11 points on complete three-turn success, statistically significant (p = 4.44e-8).
  • HumanEval+ and thinking: favorable point estimates that were not statistically resolved.
  • GPQA-Diamond and IFBench: negative point estimates and active areas for future research.

MMLU-Pro was the strict confirmatory endpoint. The final evaluation report is available in the Noema training repository.

Model details

Property Value
Public name Noema 1.5 2B
Internal research candidate noema-v3-lite-l4-step-0150-bf16
Architecture Qwen3.5 hybrid Gated DeltaNet and gated attention
Parameters Approximately 2B
Layers 24
Hidden dimension 2,048
Vocabulary 248,320 tokens
Training method Restoration-aware on-policy distillation
Weight update Rank-128 all-projection LoRA, merged for release
Native backbone context Up to 262,144 tokens
Independently validated context Up to 24,576 tokens
Primary evaluated language English
Release status Open-weight; not open source
MTP Disabled

Noema 1.5 is the text-generation trunk. Multimodal capabilities were not trained or evaluated in this release.

Training

Noema 1.5 began from the exact previous Noema 2B release and used a restoration-aware on-policy distillation program:

  • 150 optimizer steps;
  • 19,200 on-policy trajectories;
  • 7,982,979 valid completion tokens;
  • 32 unique prompts per step;
  • four student rollouts per prompt;
  • rank-128 LoRA with alpha 256;
  • Gated DeltaNet, ordinary-attention, and MLP projections targeted;
  • tied embedding and output matrix kept frozen.

Seventy percent of assignments used Qwen3.5-9B as the capability scorer. Thirty percent used untouched Qwen3.5-2B as a restoration scorer to protect general behavior.

The training mixture contained:

  • 52.5% broad knowledge;
  • 17.5% mathematics and code;
  • 22.5% general-chat restoration;
  • 7.5% novel composed constraints.

An experimental secondary constraint-RL stage was tested, failed its first mandatory gate, and was rejected. It did not modify the released checkpoint.

Intended uses

Noema 1.5 is well suited to:

  • local and privacy-sensitive assistants;
  • structured and format-constrained generation;
  • multi-turn conversations;
  • short Python program synthesis;
  • educational and lightweight reasoning tasks;
  • offline applications;
  • memory- and compute-constrained deployments.

For knowledge-intensive applications, use retrieval or authoritative tools and verify important factual claims.

Usage

Noema 1.5 requires a transformers version with qwen3_5_text support. The release was built and validated with Transformers 5.12.1.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "NoemaAI-labs/Noema-1.5-2B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {
        "role": "user",
        "content": "Write a Python function that merges two sorted lists.",
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=1024,
    do_sample=False,
)

response = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)

print(response)

Non-thinking mode is recommended for concise instruction following, structured generation, and code. The packaged defaults are greedy (do_sample=false) and stop at <|im_end|>.

For harder reasoning tasks, use enable_thinking=True, do_sample=True, temperature=1.0, top_p=0.95, and top_k=20. Deployments should still enforce output limits and repeated-span detection.

Evaluation methodology

The final evaluation used:

  • frozen model and tokenizer revisions;
  • identical prompts and settings across both arms;
  • BF16 inference on NVIDIA L40S hardware;
  • vLLM 0.19.1;
  • pinned official graders;
  • paired bootstrap intervals and McNemar tests;
  • deterministic keys and exact-count verification;
  • no score inspection before both arms were complete.

Limitations

  • The release-over-release v2 comparison uses matched development suites, not a third untouched final arm.
  • Broad MMLU-Pro knowledge was statistically retained against stock but not improved in the final point estimate.
  • GPQA-Diamond and IFBench produced lower final point estimates.
  • The release is based on one primary training seed and does not establish that on-policy distillation beats matched supervised training.
  • Evaluation was primarily English and text-only.
  • Native 262K context support comes from the Qwen3.5 architecture; Noema independently evaluated contexts only up to 24,576 tokens.
  • Thinking mode can still produce long or repetitive responses.
  • The model can hallucinate and should not be treated as the sole authority for high-stakes decisions.

Release status

Noema 1.5 2B is an open-weight release, not an open-source release. The weights are publicly downloadable, but no open-source license is granted with this repository at this time.

Citation

@misc{noema15_2b_2026,
  title        = {Noema 1.5 2B: Restoration-Aware On-Policy Distillation for Local AI},
  author       = {NoemaAI Labs},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/NoemaAI-labs/Noema-1.5-2B}}
}
Downloads last month
329
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for NoemaAI-labs/Noema-1.5-2B

Finetuned
Qwen/Qwen3.5-2B
Finetuned
(1)
this model
Finetunes
1 model
Quantizations
4 models