nuri-base-moe-it

Instruction-tuned research checkpoint with a trained length head and masked-refinement generation.

Nuri combines a bidirectional input encoder, a recurrent latent core and a bidirectional masked readout. It has no autoregressive decoder. Latent recurrence is an architectural feature; a general reasoning benefit has not been established.

Research release: Korean encoder features are useful on the measured small classification task. Semantic embeddings trail stronger references, and generated explanations remain repetitive, factually unreliable and unsuitable for general chat. See measured evaluation and limitations.

Nuri collection · Project repository

Model family

Model	Parameters	Purpose
nuri-base	302.3M	Pretraining; explicit generation canvas length
nuri-base-it	302.3M	Instruction tuning; predicted generation length
nuri-base-moe	527.4M total /302.5M active	Pretraining; explicit generation canvas length
nuri-base-moe-it	527.4M total /302.5M active	Instruction tuning; predicted generation length
nuri-base-embed	302.3M	Contrastive sentence embeddings
nuri-tokenizer	65,536 tokens	Shared byte-level BPE tokenizer

Use

Install Python 3.10+, PyTorch, Transformers, safetensors and tokenizers. The examples are checked with Transformers 5.18.0; use a version supporting the dtype argument. These checkpoints use custom Python inference code through trust_remote_code=True; pin a reviewed Hub revision for reproducible deployment.

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "alice-noa-chan/nuri-base-moe-it"
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForMaskedLM.from_pretrained(
    repo_id, trust_remote_code=True, dtype=torch.float32,
).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(repo_id)
text = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain why it rains in two sentences."}],
    tokenize=False, add_generation_prompt=True,
)
batch = tokenizer(text, add_special_tokens=False, truncation=True,
                  max_length=1024, return_tensors="pt").to(device)
with torch.inference_mode():
    result = model.generate_refine(**batch, T=4, S=4, max_len=128)
print(tokenizer.decode(result.tokens[0], skip_special_tokens=True))

The example uses FP32 parameters, including on CPU. BF16 weights are stored on the Hub. Inference throughput depends on device, dtype, input length and canvas size. No unmeasured throughput is claimed. T controls recurrent depth; S controls output refinement. Standard autoregressive generate() does not apply. Base checkpoints require an explicit canvas_len; only IT has a trained automatic length head. A loaded sequence-classification head is new and must be fitted on downstream labels.

Measured performance

Model	KO topic: test accuracy %	EN sentiment: validation accuracy %	EN inference: validation accuracy %
nuri-base	85.16	77.73	54.30
nuri-base-it	77.73	80.08	54.69
nuri-base-moe	85.16	74.61	53.52
nuri-base-moe-it	79.30	71.09	51.95
nuri-base-embed	85.94	77.34	57.03
KLUE RoBERTa Base	84.38	-	-
BERT Base uncased	-	84.77	53.12
ELECTRA Base	-	88.67	53.12
RoBERTa Base	-	85.94	58.20
DeBERTa Base	-	85.94	58.59
ModernBERT Base	-	78.52	56.25

English columns are consulted validation scores; Korean is a separate 256-row completion test using the already-selected frozen-feature classifier. They are distinct assessment panels, not interchangeable scores. Frozen probes use 1,024 fitting rows and 256 validation rows. The compared model checkpoints trained for different amounts. Full protocol, semantic similarity, reference models and generation diagnostics are in EVALUATION.md; numeric aggregates are in evaluation.json.

Model	KO extraction: 40 examples %	EN extraction: 40 examples %	Combined: 80 examples %
Nuri Dense IT	37.50	50.00	43.75
Nuri MoE IT	25.00	27.50	26.25
Qwen2.5 0.5B Instruct	0.00	85.00	42.50
Qwen3 0.6B (non-thinking)	90.00	100.00	95.00
SmolLM2 360M Instruct	0.00	60.00	30.00

Architecture and training

Property	Value
Parameters	527.4M total /302.5M active per token
Encoder /readout layers	14 /2
Hidden size /heads	1,024 /16
Recurrent latent slots	32; shared block; inference T=4
Refinement rounds	S=4
Input /answer limit	1,024 /256 tokens including EOS
Vocabulary	65,536
Stage	it
Own-stage processed input/output positions	91,869,632
Reported own-stage GPU time	0.99 hours, one NVIDIA A100 80GB PCIe

IT and Embed inherit their Base model; own-stage counts and hours exclude Base. Processed positions count actual non-padding input/output work, not unique data or masked prediction targets. Embed metadata describes its selected best checkpoint, not the later final training state. Base used MLM/span denoising; IT used instruction/reasoning/span denoising; Embed used contrastive pairs and Matryoshka prefixes. The trained run predates the sampler retention correction described in EVALUATION.md. Training data sources are not disclosed; no raw data or optimizer/resume states are included.

Limitations

The evidence is a bounded task assessment, not a clean broad benchmark. Dense/MoE amounts differ. The model can repeat phrases, output incorrect facts and calculations, and mispredict answer length. More T or S is not guaranteed to improve answers. The 30%/50% full-mask continuation experiments were rejected; these files retain the original evaluated weights. No safety assessment or production-readiness claim is made.

License

MIT. See LICENSE.