hanja-wsd-large — 한글 → 국한혼용 변환기 (candidate scorer)

Converts plain-Hangul Korean into mixed Hangul–Hanja script by choosing, for every Sino-Korean word, the correct Hanja spelling from a dictionary candidate set using sentence context. This repository holds the XLM-R large candidate scorer (run7: 4M training records + modern annotations) together with the lexicon, priors and the conversion code. It is the better choice for modern text; for 1920–60s text and for speed use LinkinShan/hanja-wsd-base, which saw all 21.5M records.

정부는 발전소 건설과 경제 발전을 위해 공사를 시작했다.
政府는 發電所 建設과 經濟 發展을 위해 工事를 始作했다.

부인은 그 사실을 부인했다.
夫人은 그 事實을 否認했다.

Write-ups: 中文 · 한국어 · 국한혼용 (auto-converted by this model)

How it works

1. Candidate spans. Kiwi tokenises the sentence; every 2–8 syllable substring aligned to token boundaries (and covering only noun-like tokens) is looked up in data/lexicon.tsv (2.2M Hangul forms, 140k ambiguous). Each hit is a candidate span with its list of possible Hanja.
2. Scoring. The model is a cross-encoder: input is the sentence with the target span marked by 【】, paired with one candidate Hanja string; a linear head on the <s> state gives a score. Softmax over the span's candidates gives $p(h \mid x)$. Because the candidate is plain text, the encoder can score Hanja strings it never saw in training (this is why a multilingual encoder was used: XLM-R's tokenizer leaves only 0.54 % of dictionary Hanja as <unk>; KLUE-RoBERTa, which does not know Hanja, scores below the frequency baseline).
3. Convert or not. A conversion prior $p_{\text{sino}}(w)=n_{漢}/(n_{漢}+n_{한}+1)$ from 1920–1962 mixed-script newspapers gates spans (native words ≈ 0); dynamic programming then picks non-overlapping spans maximising $\sum (b-a)(p_{\max}-\tau)$.

Results

Test sets are held out by file (OKHC) or by page (wiki/namuwiki); the lexicon and all counts were built from training files only.

Record-level accuracy (gold Hanja among ≤12 candidates; the scorer's own task):

test set	spans	most-frequent-Hanja	claude-sonnet-4-6	base (run6)	large (run7, this repo)	large, 2M + modern (run5)
OKHC 1920–62, hard forms (MFS ≤ 0.9)	7,263	0.708	0.920	0.947	0.908	0.892
OKHC 1920–62, all ambiguous	20,000	0.907	—	0.968	0.941	0.928
Korean Wikipedia (bracket annotations)	20,000	0.401	0.886*	0.772	0.788	0.817
Namuwiki (bracket annotations)	12,073	0.369	—	0.757	0.789	0.783

* LLM figures are on 100–150-item samples of the same candidate sets. run5 (not released) is listed because it is the best checkpoint on Wikipedia; more historical data (run7) helped the hard subset and hurt modern text slightly.

End-to-end conversion (1,000 held-out OKHC sentences, 8,333 gold spans, --preset historical):

checkpoint	P	R	F1	wrong Hanja
base (run6)	0.766	0.866	0.813	≈2 %
large (run7)	0.763	0.845	0.802	≈2 %
large, 2M + modern (run5)	0.810	0.802	0.806	2.4 %

Precision on the newspaper set is a lower bound: over half of the "spurious" spans are words the corpus usually writes in Hanja that the original author happened to leave in Hangul. On modern text (Wikipedia, --preset modern) span recall is 0.74; precision cannot be measured against bracket annotations. Remaining errors on modern text are mostly person and place names (金泳三 vs 金永三), which context cannot resolve.

Training

encoder	FacebookAI/xlm-roberta-large, add_pooling_layer=False, linear head on <s>, gradient checkpointing
objective	listwise cross-entropy over a record's candidates (gold + up to 11 negatives)
data	4M records sampled from the 21.5M generated from 1920–1962 mixed-script newspapers (OKHC newslibrary), plus 0.57M records from Wikipedia / Namuwiki / public-domain literature bracket annotations, oversampled ×2
hardware	8 × V100-16GB, DDP, 32 records/step, 156,160 steps (one epoch), ~21 h
optimiser	AdamW, lr 2e-5, 500 warm-up steps, linear decay, fp16 autocast, grad-clip 1.0
dev	5,000 ambiguous records from a held-out newspaper file; best 0.953

Data size mattered more than model size: base went 0.815 → 0.892 → 0.947 on the hard subset at 2M → 8M → 21.5M records, overtaking large at 2M (0.872) and 4M (0.908). Modern annotations did nothing for historical words and lifted Wikipedia from 0.53 to 0.77–0.82.

Training data is derived from the Open Korean Historical Corpus (newspapers 1920–1962, public domain; corpus CC BY-NC 4.0), Korean Wikipedia (CC BY-SA), Namuwiki dump (CC BY-NC-SA 2.0) and Wiktionary/kaikki (CC BY-SA). The NC-SA terms carry over to these weights.

Usage

pip install torch transformers sentencepiece kiwipiepy huggingface_hub
hf download LinkinShan/hanja-wsd-large --local-dir hanja-wsd-large
cd hanja-wsd-large
python src/convert.py --ckpt . --lexicon data/lexicon.tsv --hangul-counts data/hangul_counts.tsv \
    --preset modern "그녀는 아침마다 화장을 한다." "시신을 화장하였다."

--preset historical (gate 0.3, τ 0.7) suits 1920–60s text; --preset modern (gate 0.05, τ 0.5) suits current text. --stdin reads one sentence per line and emits JSON with every chosen span's candidates and probabilities. src/serve.py starts a small HTTP API + web page. CPU: ~320 ms/sentence for large; V100: ~25 ms.

Loading the scorer directly:

import torch
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("LinkinShan/hanja-wsd-large")
enc = AutoModel.from_pretrained("LinkinShan/hanja-wsd-large", add_pooling_layer=False)
head = torch.nn.Linear(enc.config.hidden_size, 1)
head.load_state_dict(torch.load("head.pt", map_location="cpu"))   # from the repo
ctx = "그녀는 아침마다 【화장】을 한다."
x = tok([ctx]*3, ["化粧", "火葬", "畫匠"], return_tensors="pt", padding=True)
scores = head(enc(**x).last_hidden_state[:, 0]).squeeze(-1)
print(scores.softmax(-1))   # 化粧 ≈ 0.98

Limitations

• Trained on 1920–1962 newspaper language. Modern coinages, loanwords in Hanja and current proper nouns are under-represented; the modern-text preset compensates only partly.
• Person and place names are chosen by frequency, not knowledge.
• The lexicon decides what can be converted; words absent from it (recall 90.6 % on newspapers, 81 % on Wikipedia) stay in Hangul unless the character-level fallback is on.
• No human evaluation yet; all numbers are against automatically derived gold.

Citation and attribution

Paper: submitted to arXiv (cs.CL), DOI to be added on announcement. Redistribution of these weights, the lexicon or derived outputs must credit Qifeng Xu (ORCID 0009-0009-1584-1519) and link to this repository or https://shan.ink (CC BY-NC-SA 4.0 attribution clause).

@misc{hanja-wsd-2026,
  title  = {hanja-wsd: converting plain-Hangul Korean to mixed script with a candidate-scoring cross-encoder},
  author = {Qifeng Xu},
  note   = {ORCID: 0009-0009-1584-1519},
  year   = {2026},
  url    = {https://huggingface.co/LinkinShan/hanja-wsd-large}
}