Nebula-1-preview

Nebula-1-preview is a preview of Nebula-1, a Japanese and English base language model of about 122M parameters (Llama-style decoder, 24 layers, d_model 576, vocabulary 64,000), trained from scratch on about 2B tokens by Nebou Rocket. This checkpoint is the release.

This is arm A0 of our pre-registered ablation: next-token prediction only, without D1 (self-definition anchoring).

The pre-registered decision rule did not adopt D1: A1 did not beat the baseline on rare-token validation loss by more than the seed noise. This baseline recipe is therefore what Nebula-1 ships.

D1 verdict (pre-registered rule): NOT ADOPTED. On rare-token validation loss, A0 = 5.2458, A1 = 6.1644, A2 = 5.2015 (seed noise 0.0613). The full rule and numbers are in eval/ablation-result.json.

• [FAIL] A1 beats A0 on rare loss by more than the noise: A0.rare - A1.rare = -0.9186 > noise 0.0613
• [FAIL] A1 beats A2 on rare loss by more than the noise: A2.rare - A1.rare = -0.9629 > noise 0.0613
• [PASS] English benchmark average is not lower than A0 by more than 0.5 points: A1.avg_en = 30.31 >= A0.avg_en - 0.5 = 29.69
• [PASS] Japanese benchmark average is not lower than A0 by more than 0.5 points: A1.avg_ja = 32.04 >= A0.avg_ja - 0.5 = 31.54

Related repositories

All four arms of the pre-registered ablation are published, so the comparison can be reproduced and inspected.

Repository	Arm	What it is
LoNebula/Nebula-1-preview	A0	This is arm A0 of our pre-registered ablation: next-token prediction only, without D1 (self-definition anchoring).
LoNebula/Nebula-1-ablation-a1	A1	This is arm A1 of our pre-registered ablation: it was trained with D1 (self-definition anchoring) on top of A0. During training the model reads Wiktionary definitions of rare tokens, and an auxiliary loss pulls those tokens' embeddings toward the model's own pooled hidden state over the definition.
LoNebula/Nebula-1-ablation-a2	A2	This is arm A2 of our pre-registered ablation: A0 plus the same Wiktionary definitions, inserted as plain text documents (<word>: <definition>) at the same total token cost as A1's definition reads. It is the control that separates the D1 mechanism from simply reading dictionary text.
LoNebula/Nebula-1-ablation-a0-seed1235	A0'	This is arm A0' of our pre-registered ablation: the A0 baseline trained again with seed 1235 (initialization and data order), so that the difference between A0 and A0' could be used as the seed-noise level of the decision rule.

Benchmarks

Every number below was measured by us with lm-evaluation-harness 0.4.13, under one fixed setting (same seed, dtype and batch size for every model). We did not copy any official number. Comparison models are public base models of 1B parameters or fewer. Values are percentages; the best value per column is bold.

Model	Params	MMLU	ARC-e	ARC-c	HellaSwag	WinoGrande	PIQA	LAMBADA	GSM8K	MMLU-ja	JComQA	JNLI	XWinograd-ja	Belebele-ja	MGSM-ja	Avg (en)	Avg (ja)
Nebula-1-preview	122M	25.0	38.4	23.6	27.5	51.0	58.2	17.7	0.2	24.8	42.9	45.5	54.7	22.7	1.6	30.2	32.0
HuggingFaceTB/SmolLM2-135M	135M	25.2	58.4	30.0	43.0	53.0	68.3	42.9	1.4	26.4	23.1	44.7	51.6	22.9	1.2	40.3	28.3
facebook/MobileLLM-R1-140M-base	140M	23.8	50.0	24.7	34.1	50.7	62.9	31.2	19.0	25.3	22.0	49.9	52.2	21.4	1.6	37.1	28.7
llm-jp/llm-jp-3-150m	152M	25.8	38.3	24.7	31.6	50.0	60.9	28.2	0.2	25.9	56.7	46.7	55.0	24.2	1.2	32.5	35.0
google/gemma-3-270m	268M	26.5	57.2	28.1	41.4	53.6	67.8	42.9	1.3	27.3	46.1	38.4	55.2	23.1	1.2	39.8	31.9
LiquidAI/LFM2-350M	354M	43.1	66.7	39.6	49.0	56.7	69.0	39.8	34.0	36.8	54.6	36.8	54.8	39.8	0.0	49.8	37.1
facebook/MobileLLM-R1-360M-base	359M	26.5	61.2	34.4	44.1	55.2	69.0	39.4	41.6	24.0	29.2	55.0	53.4	27.6	0.0	46.4	31.5
HuggingFaceTB/SmolLM2-360M	362M	25.1	68.1	38.1	56.2	59.7	72.0	53.2	4.2	26.0	26.6	49.3	51.5	28.2	2.0	47.1	30.6
EleutherAI/pythia-410m	405M	26.6	45.2	24.3	40.4	53.0	67.0	47.5	1.2	26.3	26.1	37.4	52.8	23.1	0.8	38.2	27.8
llm-jp/llm-jp-3-440m	447M	24.8	45.1	24.3	42.5	52.0	66.7	39.5	1.3	24.9	59.8	40.3	58.3	22.1	0.8	37.0	34.4
Qwen/Qwen2.5-0.5B	494M	47.6	58.8	31.9	52.1	56.7	70.0	52.0	34.1	37.5	35.8	39.8	58.3	33.4	0.0	50.4	34.1
LiquidAI/LFM2-700M	742M	49.4	71.0	46.9	57.5	59.1	71.0	45.1	46.4	42.6	65.1	29.8	58.9	44.2	12.4	55.8	42.2
sbintuitions/sarashina2.2-0.5b	793M	27.3	37.0	26.7	35.8	54.8	58.1	35.1	19.4	27.8	72.7	30.3	66.0	27.3	23.6	36.8	41.3

Not measured: Qwen/Qwen3-0.6B-Base, facebook/MobileLLM-R1-950M-base, llm-jp/llm-jp-3-980m, google/gemma-3-1b-pt

On the English average (30.2) this model ranks 13 of 13. On the Japanese average (32.0) it ranks 7 of 13.

Task	Language	lm-eval task	Shots	Metric
MMLU	en	mmlu	5	acc
ARC-e	en	arc_easy	0	acc_norm
ARC-c	en	arc_challenge	0	acc_norm
HellaSwag	en	hellaswag	0	acc_norm
WinoGrande	en	winogrande	0	acc
PIQA	en	piqa	0	acc_norm
LAMBADA	en	lambada_openai	0	acc
GSM8K	en	gsm8k	5	exact_match
MMLU-ja	ja	global_mmlu_full_ja	5	acc
JComQA	ja	nebula_jcommonsenseqa	3	acc
JNLI	ja	nebula_jnli	3	acc
XWinograd-ja	ja	ja_leaderboard_xwinograd	0	acc
Belebele-ja	ja	belebele_jpn_Jpan	0	acc
MGSM-ja	ja	ja_leaderboard_mgsm	5	acc

Avg (en) and Avg (ja) are plain means of the task scores of that language. Per-task results are in eval/.

Limitations

• This is a small preview model: about 122M parameters and 2B training tokens.
• It is weaker than models trained on trillions of tokens, as the table shows.
• It is a base model. It is not instruction-tuned and does not follow instructions or chat.
• It may produce false, biased or harmful text. Do not rely on it for facts or for any decision that matters.

Training data

All text was filtered, tokenized with nebula-64k and checked against the evaluation suite (see below).

Source	Language	License	Share of tokens
HuggingFaceFW/fineweb-edu (sample-10BT)	en	ODC-BY-1.0	40%
HuggingFaceTB/finemath (finemath-4plus)	en	ODC-BY-1.0	5%
wikimedia/wikipedia (20231101.en)	en	CC-BY-SA-3.0	5%
hotchpotch/fineweb-2-edu-japanese (sample_10BT)	ja	ODC-BY-1.0	40%
wikimedia/wikipedia (20231101.ja)	ja	CC-BY-SA-3.0	10%

• FineWeb-Edu, FineMath and FineWeb-2-Edu-Japanese are released under ODC-BY 1.0; credit goes to their authors. They are filtered CommonCrawl snapshots, and the CommonCrawl terms of use apply to the underlying pages.
• Wikipedia text is used under CC BY-SA 3.0. Credit: Wikipedia contributors. Whether model weights count as an adaptation of share-alike text is legally unsettled.
• Wiktionary definitions (extracted by kaikki.org, CC BY-SA 4.0) were used only by the D1 arms of our ablation, not by this model. Credit: Wiktionary contributors.

Decontamination. Text is normalized (NFKC, lowercase, no whitespace, punctuation or control characters). A training document is dropped if any 50-character window of it matches a window of the evaluation documents, so an overlap of 75 normalized characters or more is always caught.

About D1 (self-definition anchoring)

D1 reads a rare token's dictionary definition with the model that is being trained, pools the hidden states, stops the gradient, and pulls the token's embedding rows toward that vector. In our searches, run on 16 October 2026, we did not find the same method. This is not proof that nobody has done it. The nearest work is DefinitionEMB (2408.01308, applied after pretraining with a separate encoder) and Token Distillation (2505.20133, applied after pretraining to initialize new tokens).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("LoNebula/Nebula-1-preview")
model = AutoModelForCausalLM.from_pretrained("LoNebula/Nebula-1-preview")
inputs = tok("The capital of Japan is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

The beginning-of-sequence and end-of-sequence token is <|endoftext|> (id 0). The context length is 2048.