Yomi-Distill (yomi-distill-ja)

An attempt to put Beam's fixes to Japanese TTS readings into a 1.8B model, trained on Beam's corrections. Pre-registered and scored once: it did not beat OpenJTalk.

Japanese text-to-speech first turns text into a katakana reading with a dictionary-based converter such as OpenJTalk, which misreads numbers, words whose reading depends on context, and proper nouns. Part 1 showed that Beam-1.0-501B-A23B (Reflection AI) fixes many of these errors when it is given OpenJTalk's reading. A 501B model is too large to sit inside a speech pipeline, so this model, llm-jp-3-1.8b fine-tuned on Beam's corrections, tries to do the same job at run time without Beam.

• Demo: https://huggingface.co/spaces/ayousanz/yomi-distill
• Teaching data, test sets and results: https://huggingface.co/datasets/ayousanz/beam-yomi-teacher
• Code: https://github.com/ayutaz/beam-ja-yomi (space_distill/, scripts/, yomibench/)
• Pre-registration: prereg.md in the dataset (written and committed before training, commit 0ab8d1f in the author's working repository; its change log lists every later edit)

How Beam is used

• As the teacher, not the answer key. Beam read 7,551 Japanese Wikipedia sentences twice at reasoning_effort=low: once on its own and once correcting OpenJTalk's reading. Only the 5,515 sentences where the two answers agreed (73%) were kept, and the student learned the corrected reading. In Part 1, when these two ways of asking agreed, the reading was right 97% of the time (a post-hoc finding).
• Never at run time. The student reads (sentence, OpenJTalk's reading) and writes the corrected reading. The demo holds no API key.
• Never as gold. No score below uses Beam's readings as the correct answer. The main yardstick is the 300 sentences of Part 1, whose readings a person approved. The 196 new Wikipedia sentences (dev 97, used to choose the model; test 99, scored once) have readings written by Claude and cross-checked by a second, blind Claude reading, not by a person. None of them are in the teaching data (Beam was run on test only for the comparison below).

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import pyopenjtalk  # pyopenjtalk-plus

tok = AutoTokenizer.from_pretrained("ayousanz/yomi-distill-ja")
model = AutoModelForCausalLM.from_pretrained("ayousanz/yomi-distill-ja", dtype=torch.bfloat16).cuda().eval()

text = "このカレーは辛い"
draft = pyopenjtalk.g2p(text, kana=True)
prompt = f"文: {text}\n下書き: {draft}\n読み: "
ids = torch.tensor([[tok.bos_token_id] + tok(prompt, add_special_tokens=False)["input_ids"]]).cuda()
out = model.generate(input_ids=ids, max_new_tokens=160, do_sample=False, eos_token_id=tok.eos_token_id)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Use the output only if it is non-empty, contains only katakana and ー, was not cut off, and its length is between 0.8 and 1.25 times the draft's; otherwise keep OpenJTalk's reading. This gate was part of the frozen setup and is included in every number below (yomibench/student.py in the code repository).

Results

Scored once, after the model was frozen (frozen.json holds the file hashes). Exact match with 95% bootstrap intervals, against the human-approved readings of Part 1 and the Claude-written readings of test.

system	Part 1: 300 sentences, human-approved	test: 99 sentences, Claude's readings
OpenJTalk (the draft)	74.7% (69.7-79.7)	74.7% (65.7-82.8)
student (this model)	74.7% (69.7-79.7): fixed 10, broke 10	75.8% (66.7-83.8): fixed 3, broke 2
Beam, correcting the draft (beam_fix, for reference)	86.0% (82.0-89.7)	89.9% (83.8-96.0)

The student did not beat OpenJTalk. On dev it reached 80.4% against OpenJTalk's 78.4%, short of the pre-registered bar (+3 points, at most 3% broken); as pre-registered, it was frozen and scored anyway.

• It mostly copies the draft: its reading equals OpenJTalk's on 270 of the 300 sentences. Only 18% of its training rows (1,013 of 5,515) contain a correction, which may be why (not tested).
• Place names are the exception: 36% -> 52% on Part 1's 50 place-name sentences (beam_fix: 50%). All 10 fixes are one habit: reading 町 as チョウ instead of OpenJTalk's マチ (it also changed 坂城町, correctly サカキマチ in OpenJTalk, to サカキチョウ). On the other 250 sentences it fixed none and broke 8 (82.4% -> 79.2%), for example 本居宣長 and 与謝野晶子; it fixed no homograph.
• A by-product holds up: on the new test sentences (Claude's readings), when Beam's two answers agreed (67 of 99), the reading was right 65 times (97.0%), as in Part 1's post-hoc analysis (96.9%).

Training

• Base: llm-jp/llm-jp-3-1.8b (LoRA r=16 on the q/k/v/o projections, merged), 2 epochs, learning rate 2e-4, batch 16 with 2 accumulation steps, seed 0, loss on the reading only. Chosen among the five pre-registered candidates (llm-jp-3-440m full fine-tuning for 1-3 epochs, llm-jp-3-1.8b LoRA for 1-2) on the 97 dev sentences only.
• Data: 5,515 (sentence, OpenJTalk reading, Beam reading) rows from two shards of Japanese Wikipedia (wikimedia/wikipedia, 20231101.ja). Sentences close to any test sentence were removed first (an exact match after normalization, or a shared 5-gram with two or more kanji or digits): 25,815 against the 300 and 4,416 against the 200.
• Teacher cost: about $18.52 of Hugging Face Inference Providers credit (Baseten, $0.40 / $1.20 per million input / output tokens; estimated from the token counts of all 14,272 responses, including calls for sentences that did not end with both answers) for 6,916 sentences, and Reflection AI's free Beta-plan daily quota for the first 635. 1,023 more sentences failed (server errors, rate limits) and were not retried.
• Hardware: one RTX 4070 Ti SUPER (16 GB), 28 minutes.

Limitations

• The training targets are Beam's readings. Where Beam is wrong and its two answers agree, the student learns the error.
• OpenJTalk is used as shipped with piper-plus (pyopenjtalk-plus[onnxruntime]==0.4.1.post9); a different front end gives different drafts.
• Only the sequence of sounds is evaluated, not pitch accent.
• The new dev and test readings were written by Claude and checked only against an independent Claude reading (they agreed on 193 of 200 before settling; 4 unusable sentences were dropped). No person checked them, so the test numbers measure agreement with an AI's readings.
• Place names stay hard: neither Beam nor the student reliably knows when 町 is チョウ.

License and credits

• Model: Apache 2.0, as the base model llm-jp/llm-jp-3-1.8b (LLM-jp).
• Teacher: Beam-1.0-501B-A23B by Reflection AI (Apache 2.0; its model card says Reflection does not prohibit using its outputs to train other models, checked 2026-10-25).
• Training sentences: Japanese Wikipedia, CC BY-SA 3.0.
• Reading converter: pyopenjtalk-plus (OpenJTalk), as used by piper-plus.