ChiTako/unofficial-photon-repro-ja-250m-v3

Unofficial. This is an independent, hobby reimplementation of the PHOTON architecture described in arXiv:2512.20687, trained from scratch by a private individual. It is not released by, affiliated with, or endorsed by the paper's authors or any organisation, and no weights, data or code from any official PHOTON release were used. Treat it as a research artefact, not a product.

Correction to the perplexity numbers (2026-07-28)

Earlier revisions of this card reported Wikipedia perplexity measured by streaming wikimedia/wikipedia from record 0 — which is where prepare_data.py also starts building the training shards. Those numbers were measured on training data and are withdrawn.

The table below uses articles past the ones training consumed (--ppl-skip-ja 700000 --ppl-skip-en 400000).

Held-out Wikipedia, 409,600 tokens each, identical protocol for all runs:

model	data	ja-wiki	en-wiki
unofficial-photon-repro-ja-250m-v1	Wikipedia only, 200M tok	60.32	65.91
unofficial-photon-repro-ja-250m-v3	Wikipedia only, 200M tok	57.12	62.17
unofficial-photon-repro-ja-250m-v4	Wikipedia only, 200M tok	52.11	59.52
unofficial-photon-repro-ja-250m-v3-900m	full mixture, 900M tok	67.60	69.37

Do not rank these models by the Wikipedia column. v1-v4 trained on Wikipedia; v3-900m did not (5% of its mixture). On 青空文庫, which none of them trained on, v3-900m scores 237.10 against v4's 365.49 — the ranking reverses. See docs/findings.md sections 1b-1d in the repository.

PHOTON-JP small v3

A Japanese language model on the PHOTON hierarchical autoregressive architecture (arXiv:2512.20687), with every transformer stack replaced by a fine-grained Mixture-of-Experts using auxiliary-loss-free load balancing.

Tokens are folded into multi-resolution latent units instead of being scanned one at a time:

tokens   t1 t2 t3 t4 | t5 t6 t7 t8 | ...      <- level-1 decoder, every token
           \  \  /  /
level-1      u1      |     u2      | ...      <- once per 4 tokens
              \_____/____________/
level-2            m1                          <- once per 16 tokens

A level-l encoder runs once per C<=l tokens, so its cost is amortised. That is the whole point: capacity sits at the top of the hierarchy where it is cheapest per token.

Size

total parameters	0.250 B
active per token	0.101 B
FLOP-equivalent dense size	0.068 B
forward FLOPs / token	0.129 GFLOP
hierarchy	L=2, C≤L=16 tokens per top-level unit
context	1024 tokens
vocabulary	99,584 (llm-jp-tokenizer v3)
KV cache, HierGen	0.95 KiB/token
KV cache, RecGen	0.20 KiB/token

Training

tokens seen	0.20 B
steps	1,520
tokens / parameter	0.78
hardware	1x NVIDIA GB10 (DGX Spark), 5 h
throughput	11.0K tokens/s
optimiser	Muon (2-D weights) + AdamW (embeddings, norms, router)
schedule	WSD, 1-sqrt cooldown
final train CE	4.0494 (ppl 57.36)

Evaluation

benchmark	result
ppl/wiki_ja	57.1191
ppl/wiki_en	62.1680

Data

Japanese Wikipedia (190M tokens) and English Wikipedia (64M tokens), tokenised with llm-jp/llm-jp-3-1.8b, sampled 75:25.

Honest limitations

This model is under-trained by design of the budget, not by accident. At 0.78 tokens per parameter it is far below the ~20 that Chinchilla-optimal training implies, and further still below what an inference-efficient model would normally get. It produces fluent Japanese surface form -- correct particles, natural kana/kanji mixing, sentence-final forms -- while being largely incoherent semantically. Treat it as a demonstration that the architecture trains, not as a usable assistant.

Samples

市内人口 

2021都市（市中心部）に所在し、産業統計によると、人口は2015

富士山は今は標高が高いため、砂浜からは石灰岩が頻繁に定されている。

日本では、海水浴客の減少や温泉の普及など量が多く、地元では宿泊客減少に伴うリゾート施設の老朽化し（高山市観光交流センター）や、今日では観光客の

======================================================================
人工知能とは別の認識能力を持つため、人工知能と呼ばれる機械的系などの構造を持つ。
 生物がヒトでも生成している遺伝子を人工的に物質、非核生物を含む生物にとって究極の不親類の遺伝子系の合成分子にほぼ同等の機能を持つ生物細胞の存在を

夏目漱石の代表作は『女の怪談』（明治18年） Brunswick and Other Stories（1950）が出版された（「狩野" in an effort of cutbacks of action, and his life to die, a taxonomical survey of his own property.)

今日、日中y and

水の沸点は1.08 m3である。

石灰岩を-dollar milesout southwest of the river may have a wide range of the surrounding area. It is a valley, and is the most important transportation system a few other canals. The other hand of the water supply is called the Pen and

Generation protocols

• HierGen keeps encoder state at every level. Exact: it reproduces the training-time distribution (verified to 2e-4 in the test suite).
• RecGen keeps only the top-level KV cache and feeds the decoder cascade its own reconstructions, cutting cache by 4.8x. Exact when recursive consistency holds, which is what L_token + alpha * L_rec optimises for.

Usage

from photon_jp.model.config import PhotonConfig
from photon_jp.model.photon import PhotonForCausalLM
from photon_jp.model.loading import load_state_dict_compat
from photon_jp.infer.generate import PhotonGenerator, GenerationConfig
from safetensors.torch import load_file

cfg = PhotonConfig.load("model_config.json")
model = PhotonForCausalLM(cfg)
load_state_dict_compat(model, load_file("model.safetensors"))

gen = PhotonGenerator(model.cuda().eval(), "cuda")
out = gen.generate(input_ids, GenerationConfig(mode="recgen", max_new_tokens=256))

Accounting

==============================================================================
PHOTON-JP parameter report
==============================================================================
vocab=99,584  D0=512  L=2  C_<=L=16  ctx=1024

stack                    total        active   amort     amort.act
------------------------------------------------------------------------------
L1.encoder              31.8M        12.1M       4         3.0M
L1.decoder              24.9M         7.8M       1         7.8M
L2.encoder             127.1M        21.0M      16         1.3M
L2.decoder               8.6M         4.5M       4         1.1M
------------------------------------------------------------------------------
embedding               51.0M
lm_head                  0.0M
chunk/convert            4.2M
mtp                      2.7M
==============================================================================
TOTAL          : 0.250 B   (0.199 B non-emb)
ACTIVE / token : 0.101 B   (0.050 B non-emb)
AMORTISED      : 0.068 B  (FLOP-equivalent dense size)
sparsity       : 2.49x

fwd FLOPs/token @ ctx=1024: 0.13 GFLOP (matmul 0.13, attn 0.001)
KV cache mode    : MLA latent (weight-absorbed)
KV cache  HierGen: 0.945 KiB/token   (other mode: 2.312)
KV cache  RecGen : 0.195 KiB/token   (other mode: 1.562)
  -> 0.9 MiB for a full 1024-token context (HierGen, per sequence)
==============================================================================

Citation

@article{ichikawa2025photon,
  title  = {PHOTON: Hierarchical Autoregressive Modeling for Lightspeed and
            Memory-Efficient Language Generation},
  author = {Ichikawa, Yuma and Takagi, Naoya and Nakagawa, Takumi and
            Kanazawa, Yuzi and Sakai, Akira},
  journal= {arXiv preprint arXiv:2512.20687},
  year   = {2025}
}