UMCU/CardioNER.eu_XLMR

Training and inference using CardioNER.

Settings

• Optimizer: AdamW
• Learning rate: 2e-5, with class-weights
• Decay rate: 1e-4
• batch-size: 16
• 10-fold stratified CV, with 10 epochs for the statistics below
• Architecture: EuroBERT + 3-layer dense head.
• Output: multilabel with probas for DISEASE, MEDICATION, PROCEDURE, SYMPTOM.
• Chunking: centered around the span of interest

Note: for inference we use poetry run python -m cardioner.main --inferency_only.. CardioNER with --pipe=dt4h.

Performance on internal hold-outs

Use with caution/not standalone: performance may be considerably less on external datasets.

Combined

Setting	Averaging	Precision	Recall	F1
Strict	Micro	0.731 ± 0.008	0.710 ± 0.007	0.721 ± 0.007
Strict	Macro	0.760 ± 0.007	0.735 ± 0.008	0.749 ± 0.007
Relaxed	Micro	0.875 ± 0.005	0.849 ± 0.008	0.864 ± 0.005
Relaxed	Macro	0.885 ± 0.007	0.860 ± 0.007	0.872 ± 0.007

Strict

Category	Precision (mean ± std)	Recall (mean ± std)	F1 (mean ± std)
DISEASE	0.721 ± 0.010	0.705 ± 0.007	0.713 ± 0.009
MEDICATION	0.884 ± 0.011	0.847 ± 0.015	0.862 ± 0.012
PROCEDURE	0.754 ± 0.011	0.711 ± 0.010	0.730 ± 0.010
SYMPTOM	0.685 ± 0.008	0.680 ± 0.007	0.683 ± 0.009

Relaxed

Category	Precision (mean ± std)	Recall (mean ± std)	F1 (mean ± std)
DISEASE	0.877 ± 0.009	0.856 ± 0.011	0.868 ± 0.008
MEDICATION	0.934 ± 0.011	0.895 ± 0.016	0.915 ± 0.012
PROCEDURE	0.885 ± 0.008	0.835 ± 0.008	0.858 ± 0.007
SYMPTOM	0.850 ± 0.009	0.845 ± 0.007	0.845 ± 0.007