Rip-Shadw/chronotrace-report-ner

ChronoTrace report-row NER

Tags each printed row of a lab report word by word: test name (TEST), result (VALUE), unit (UNIT), reference range (RANGE) and H/L flag (FLAG), as BIO tags. It is the extraction step of ChronoTrace, a doctor-facing tool that puts a chronic patient's lab results from different labs into one time series.

Important limits

• Trained on synthetic reports only. No real patient data was used. All metrics below are on synthetic reports.
• Real-report accuracy is not yet measured. It is pending evaluation on real reports (ChronoTrace step 9). Do not assume the synthetic numbers carry over.
• Extraction only. It reads what is printed. It does not interpret results, decide what is abnormal, or recommend anything; trend flags in ChronoTrace are computed by separate code.
• Not a medical device. Not for diagnosis or treatment decisions. Extracted values must be checked against the source report.

Data

Rows come from synthetic Indian-style lab reports rendered to PDF and read back with pdfplumber (ChronoTrace step 2): 26,607 train rows and 3,069 validation rows from 8 layout families. The test set (6,586 rows) uses 2 layout families never seen in training: flag_emphasis, inline_unit.

Results (synthetic data)

Entity F1 (seqeval, strict IOB2). "Rules" is rules (generator vocabulary): a hand-written tagger that knows every test name, unit and method string the synthetic generator uses. On synthetic data it is therefore an upper bound, not a realistic competitor. The comparison that matters is on real reports, which will contain names and units outside that vocabulary (ChronoTrace step 9, pending).

	val model	val rules	held-out test model	held-out test rules
TEST	1.0000	1.0000	1.0000	1.0000
VALUE	1.0000	1.0000	1.0000	1.0000
UNIT	1.0000	1.0000	1.0000	1.0000
RANGE	1.0000	1.0000	1.0000	1.0000
FLAG	1.0000	0.9779	1.0000	1.0000
overall	1.0000	0.9989	1.0000	1.0000
exact value accuracy	1.0000	1.0000	1.0000	1.0000

FLAG recall on the held-out families:

family	model	rules	model before flag fix
flag_emphasis	1.0000	1.0000	0.9074
inline_unit	1.0000	1.0000	0.4083

The held-out score is not fully blind. A first model (same hyperparameters) was trained on data where each layout family printed flags in one fixed position. Error analysis on the held-out families showed it missed flags printed right after the value or unit. The generator was then changed to vary flag position in every family, and the model was retrained with unchanged hyperparameters (twice; see checkpoint selection below). Before that fix, held-out test overall F1 was 0.9905 and FLAG F1 0.8792 (rules: 1.0000). All numbers are in step3_metrics.json in the ChronoTrace repository, the pre-fix ones under before_flag_fix.

Checkpoint selection was also changed after test results had been printed. The first retrain after the flag fix tied at val entity F1 1.0000 in every epoch, so the "higher F1 only" rule kept epoch 1, and that checkpoint's held-out test results were printed once. Selection was then changed to break F1 ties by lower validation loss, and the model was retrained with the same hyperparameters and seed. The change was prompted by the validation tie, not by the test numbers, but it came after they had been seen, so the reported test score is not fully blind on this point either.

CPU latency: 18.18 ms per row (mean over 500 rows, batch 1, 4 threads, on a laptop CPU; it varies with machine load).

Training

distilbert-base-uncased, 3 epochs, lr 5e-05, batch 32, max length 64 sub-tokens, weight decay 0.01, fp16, seed 42. Best epoch (3) chosen on validation entity F1. Trained on NVIDIA GeForce RTX 3050 6GB Laptop GPU in 242 s.

Use

from transformers import AutoTokenizer, AutoModelForTokenClassification
tok = AutoTokenizer.from_pretrained("Rip-Shadw/chronotrace-report-ner")
model = AutoModelForTokenClassification.from_pretrained("Rip-Shadw/chronotrace-report-ner")
words = ["Serum", "Creatinine", "1.11", "mg/dL", "0.7", "-", "1.3"]
enc = tok(words, is_split_into_words=True, return_tensors="pt")
pred = model(**enc).logits.argmax(-1)[0]
# each word's tag is the prediction at its first sub-token (enc.word_ids())