EvidenceLink-DART-Reranker-KO

🇰🇷 한국어 카드: MODEL_CARD.ko.md

PASS_V24 (G1–G5 passed) — the verdict of one pre-registered experiment (prereg_E_v2_4), not a general claim. Sealed test = 648 correction filings from 79 issuers never seen in training. Gold links come from DART's own correction chain, not from human review. Natural "no match" cases were too few (25 < 30), so gate G6 is UNDERPOWERED and generalization of the NONE decision to natural cases is not established.

A Korean cross-encoder that links a DART correction filing for a convertible-bond issuance decision ([기재정정] 주요사항보고서(전환사채권발행결정)) back to the original filing (the right bond series/round) it amends, or answers NONE when the original is not in the candidate set. It is a full fine-tune of BAAI/bge-reranker-v2-m3 trained only on public OpenDART filings.

The hard part is not finding the issuer. It is choosing among several filings of the same issuer (different series, same day, near-identical tables) and refusing to link when none of them is right.

At a glance

Task	correction filing → one of K candidate originals, or NONE
Scope	DART convertible-bond issuance decisions (전환사채권발행결정) and their corrections, filed 2022–2026
Sealed test (CLEAN, n=499)	Top-1 0.978 vs BM25 0.7375 · rule T0 0.7174
NONE precision / recall (CLEAN)	1.0 / 0.9308 (BM25 0.5548 / 0.6615)
Same-issuer wrong-link rate (CLEAN)	0.0246 (BM25 0.1387 · T0 0.1432)
Paired issuer-cluster bootstrap (10,000)	Top-1 vs BM25 +0.2387, 95% CI [0.1645, 0.3091]
Decision rule	argmax score over candidates; NONE if the top logit < −8.4375 (frozen on dev before the sealed run)
Weights	model.safetensors sha256 a402de3dc80fa8d45b42551ef381f063ec997fdba7d5d6e31462108fd48f6c19
Training data	OpenDART public filings only. No AI-Hub data. No filing text is shipped here.

What it does

Output	Meaning
pred = <candidate id>	the candidate original filing this correction amends (highest score, score ≥ threshold)
pred = None	NONE: no candidate scored above the frozen threshold
scores	raw logit per candidate (pointwise; not calibrated probabilities)

Quick start

pip install torch transformers huggingface_hub
hf download ThakiCloud/EvidenceLink-DART-Reranker-KO --local-dir evidencelink
cd evidencelink
python evidencelink.py --selftest                       # decoder + card format checks (no model)
python evidencelink.py --model . --example examples/synthetic_query.json

from evidencelink import load, query_card, candidate_card, link, dart_zip_to_text

tok, model = load(".")                                   # fp32, CPU or GPU
q = query_card(corp_name, correction_text)               # text from dart_zip_to_text(<OpenDART document.xml zip>)
cands = [{"id": rcept_no, "card": candidate_card(corp_name, rcept_dt, text)}
         for rcept_no, rcept_dt, text in originals]      # order: receipt number descending
print(link(tok, model, q, cands))                        # {"pred": ... or None, "top_score": ..., "scores": {...}}

Use the input format in evidencelink.py. The model was trained on a specific card: a [요약] summary line (series, principal, conversion price, coupon, yield, refixing floor) followed by the masked filing text (first-submission date and 14-digit receipt numbers are replaced by [MASK]/[RCEPT]). Raw filing text gives different scores, and the frozen threshold no longer applies.

Decode OpenDART documents with the included decoder (dart_zip_to_text). Filings before mid-2022 are cp949. Reading them as UTF-8 with errors="replace" silently destroys the fields the model relies on (see Limitations).

examples/synthetic_query.json is a fictional issuer generated from a template. It is not a DART filing.

What is in this package

File	Role
model.safetensors	fine-tuned weights (fp32), sha256 a402de3dc80fa8d4…
config.json, tokenizer.json, tokenizer_config.json	model config and tokenizer (unchanged from the base)
evidencelink.py	decoder v2, field extraction, card builder, scoring, NONE rule
examples/synthetic_query.json	synthetic query + 3 candidates
LICENSE, NOTICE	Apache-2.0, attribution

How the candidate set is built (evaluation protocol)

For each correction filing: up to 10 original filings of the same issuer received within the previous 3 years (most recent first), plus up to 3 filings of other issuers received within ±10 days (hard negatives). Candidates are ordered by receipt number, descending. For 0.9499 of CLEAN queries the true original falls inside this window; the remaining 25 queries are the natural NONE cases (original filed more than 3 years earlier). In another 105 CLEAN queries the true original was removed on purpose to create NONE cases (NONE_SYNTH below).

Sealed test results (sealed4, prereg_E_v2_4)

648 correction-filing queries from 79 issuers. Issuers in sealed4 were excluded from training and dev. CLEAN (n=499; 369 linked, 130 NONE) drops queries whose card shows the receipt date of any candidate, so a model cannot shortcut through a visible date. CLEAN is the primary set for all gates.

CLEAN (primary, n=499)

Arm	Top-1	Linked acc	NONE precision	NONE recall	Same-issuer wrong-link rate
This model (H3, seed 0)	0.978	0.9946	1.0	0.9308	0.0246
BM25 (card vs card, NONE threshold on dev)	0.7375	0.7642	0.5548	0.6615	0.1387
T0 rule (field-overlap count, NONE threshold on dev)	0.7174	0.6856	0.5899	0.8077	0.1432
bge-m3 bi-encoder, zero-shot	0.5431	0.5339	0.3289	0.5692	0.1723

All arms were measured by us, in the same sealed job, on the same candidate sets.

Confidence intervals (issuer-cluster paired bootstrap, 10,000 resamples, seed 1109)

Difference in Top-1	Mean	95% CI
This model − BM25, CLEAN	+0.2387	[0.1645, 0.3091]
This model − T0, CLEAN	+0.2573	[0.1635, 0.3497]
This model − BM25, ALL	+0.2263	[0.1616, 0.2907]
This model − T0, ALL	+0.2633	[0.1747, 0.3499]

ALL (n=648) Top-1: this model 0.9722 · BM25 0.7454 · T0 0.7068.

Pre-registered gates

Gate	Condition (CLEAN)	Result
G1	Top-1 vs BM25, bootstrap CI lower bound > 0	pass
G2	same-issuer wrong-link rate ≤ 0.05 and below BM25 and T0	pass (0.0246)
G3	NONE precision and recall ≥ 0.80	pass (1.0 / 0.9308)
G4	linked accuracy ≥ 0.90	pass (0.9946)
G5	ALL Top-1 ≥ BM25	pass (0.9722 vs 0.7454)
G6	natural-NONE recall ≥ 0.80, evaluated only if n ≥ 30	UNDERPOWERED (n=25) — excluded from the verdict

By filing year (CLEAN Top-1)

Year	n	This model	BM25	T0
2022	100	0.97	0.57	0.68
2023	67	0.9552	0.7463	0.7612
2024	140	0.9857	0.8143	0.7286
2025	109	0.9817	0.7798	0.7064
2026	83	0.988	0.747	0.7229

NONE by stratum (CLEAN)

Stratum	n	This model	BM25	T0
NONE_SYNTH (true original removed from the set)	105	0.9333	0.6	0.7714
NONE_NATURAL (true original outside the 3-year window)	25	0.92	0.92	0.96

On the 25 natural NONE cases the model is not better than the rule baseline (T0 0.96). With n=25 none of these numbers separates the arms.

Limitations — do not overstate

• Verdict scope. PASS_V24 is the verdict of prereg_E_v2_4 on sealed4 only. It does not certify other filing types, other years, or customer documents (contracts, spreadsheets).
• G6 UNDERPOWERED. Only 25 natural NONE cases exist in sealed4 (target ≥ 30). Most NONE cases (105 of 130 in CLEAN) were built by removing the true original from the candidate set. Whether the NONE decision generalizes to natural "no original in range" cases is not established.
• Gold is not human-reviewed. The gold original comes from DART's correction chain (the filing family DART itself links when a correction is submitted). It is rule/chain based. Training labels are weaker still: a rule that matches the first-submission date written in the correction (that date is masked in the model input). On CLEAN queries where the weak rule decides, it agrees with gold 0.8868 of the time.
• Dev has almost no 2022–2023 data. Dev = 93 queries from 11 issuers (2022: 0, 2023: 1, 2024: 26, 2025: 31, 2026: 35). Epoch and the NONE threshold were selected on that dev. The 2022–2023 sealed numbers above are therefore out-of-distribution for model selection.
• Truncation. All checked pairs (4,000 train, 432 dev) exceed 1024 tokens and are truncated (longest_first). The [요약] summary line is placed first so it survives; filing text deep in the form is not seen.
• Encoding dependency (decoder v2). OpenDART documents before mid-2022 are cp949. An earlier pipeline decoded them as UTF-8 with replacement characters, which damaged 1,427 cached documents and collapsed field extraction for 2022–2023 (the cause of the previous NO_GO on sealed3). With decoder v2, query series/principal extraction on sealed4 is 0.948–0.992 per year. If you feed text decoded another way, expect lower accuracy.
• sealed3 re-score is a report-only diagnostic. Re-scoring the earlier H2 model (not this checkpoint) on the already-exposed sealed3 set after the decoder fix raised its CLEAN Top-1 from 0.6571 to 0.9455 on the same 385 queries. That set was exposed before, so it is evidence about the input defect, not a pass criterion.
• Candidate window. Only originals filed within 3 years before the correction (10 most recent of the same issuer) are candidates. An older original can never be linked; the correct output is then NONE.
• Seed. Two seeds were trained; seed 1 tied on dev (0.9085) and seed 0 was chosen by the pre-registered tie rule before sealed evaluation. Seed 1 was not evaluated on sealed4.

Training settings (from pre-registration)

Base	BAAI/bge-reranker-v2-m3, full fine-tune, pointwise BCE
Hyperparameters	max_len 1024, batch 16, 2 epochs (best epoch 1), lr 2e-5, positive weight 3
Negatives	all other candidates in the set (same-issuer other series + other-issuer near-date)
Train	1,430 queries, 205 issuers, 2022–2026
Dev	93 queries, 11 issuers; selection = macro average of per-year Top-1 (0.9085)
Sealed4	648 queries, 79 issuers, evaluated once

Data source

OpenDART (Financial Supervisory Service, Korea) public disclosure filings: convertible-bond issuance decisions and their corrections, accessed through the OpenDART API. This repository contains no filing text, extracted fields, queries, candidate sets, or labels. No AI-Hub data was used.

License

Apache-2.0, matching the base model BAAI/bge-reranker-v2-m3 (Apache-2.0). See LICENSE and NOTICE.