HIDDEN-RAD:
Hidden Causality Inclusionin Radiology Report Generation
Key-Sun Choi♢ , Young-Gyun Hahm
J.H. Hahn ♢ , S.Y. Lee♡ , Y.J. Kang
♢ Konyang Univ., Teddysum Inc., ♡ Konyang Univ. Hospital

NTCIR-18 Pilot Task

Hidden-RAD: Purpose with example

2To Recover the Hidden Causality why such Impression andFindings are drawn in the Radiology Report
Radiography (option)

Radiology report

Task input
Task Output

Impression:
Pneumothorax

Finding:
The pneumothorax in this case may be attributed to a combination of factors,
including trauma and anatomical location. The right pneumothorax observed at theT8-11 thoracic spine level in the right pleural space indicates a localized issue in theupper to middle region of the right lung.

Hidden causality:
The lack of symmetry in the apical, upper, middle, and lower zones suggests anasymmetric distribution of air in the pleural space, further confirming the presence ofpneumothorax.

3

Hidden-RAD: dataset overview•Data collection: Decision of Steps in Reading Process of real doctors.
• Hidden-RAD dataset: generated report from the collected dataRadiography

Reading process
Q1. First impression

Hidden-RADDataset
Finding

Task Input
Q2. Anatomical location

Impression

Q3.Thoracic spine levels

Hidden causality

GroundTruth

Q4. Final impression
Q5. Confirmation
checklist for Q4

confirmation checklist obtained
from the MD qualification exam
textbook

The reading process was annotated with questions and answers to reduce costs, and GPT-4 was used to generate the dataset into a report format.

Task Definition

4

Recovering report by discovering hidden (missing) causality in radiology
reports.
Task Input

How to discover andrecover hidden causalityof radiology reports.

A report expressing the causalityof an impression that the doctor
could not express.

Image (option)
Radiology report(including the findingand impression)
Additional data available(KB, etc.) (option)

Method(task participants)

Task Output
Radiology report toinclude/add
Hidden Causality••
Evaluation••
Ground Truth•(Hidden-RADdataset)

A report made from a checklist
obtained from a doctor'squalification exam.

BLEUROUGEMETEORBERT Score
And GPT-basedevaluation(TBD)

Confirmation checklist: ABCDE approach• 32 checklists,• Chest x-ray review is a key competency for medical students,junior doctors and other allied health professionals.
• Using A, B, C, D, E is a helpful and systematic method for chest
x-ray review:• A: airways – 5 checklist• B: breathing (the lungs and pleural spaces) - 11 checklist• C: circulation (cardiomediastinal contour) - 5 checklist• D: disability (bones - especially fractures) - 6 checklist• E: everything else, e.g. pneumoperitoneum - 5 checklist

5Terminology and Definition 1/2•

6Radiology Report––A detailed report that describes the results of an imaging test.
A radiology report includes information about the type of imaging test that wasdone and how it was done. (A brief medical history of the person having the test,
including any symptoms or known diseases and why the test was needed)

Interpretation (or “reading”)– Radiologic interpretation is based upon the understanding of disease
processes and the behaviour of diseases in a speciﬁc anatomic region.
– Combined with knowledge of the specific radiological characteristics ofvarious lesions, the main features of the lesion can be identified andcontribute substantially to the diagnosis.

•1.

2.
3.
4.
5.

[NCI Dictionary, National Cancer Institution (NCI) in National Institution of Health (NIH), https://www.cancer.gov/publications/dictionaries/cancerterms/def/radiology-report ][Hartung, M. P., Bickle, I. C., Gaillard, F., & Kanne, J. P. (2020). How to create a great radiology report. Radiographics, 40(6), 1658-1670.][Reiner, B. I., Knight, N., & Siegel, E. L. (2007). Radiology reporting, past, present, and future: the radiologist’s perspective. Journal of the American College ofRadiology, 4(5), 313-319.][Koong, Bernard. “The basic principles of radiological interpretation.” Australian dental journal 57 Suppl 1 (2012): 33-9 .][Baines, Elizabeth A. and Andrew Holloway. “Principles of radiological interpretation.” (2013).]

Terminology and Definition 2/2

7•

CAD (Computer-aided diagnosis)– CAD is a system that assist doctors in interpreting medical images
produce a lot of information that needs to be analyzed and evaluatedin a short time.

•“Hidden Causality”
In this task, Required conditions for the final impression as
determined by the doctor
In this study, hidden causality was limited to 28 checks in the chest
radiography interpretation procedure and does not imply all causal
relationships in the interpretation.
The hidden causality was manually annotated by doctors.

Task summary

8

Task background• If the hidden cause is not sufficiently included in the report, it may cause errors that would worsen
the patient's illness.
• Errors according to :• Limitations of the perceptual capacity of the human eye and brain to perform interpretation• Differences in the scope and quality of background knowledge

Task Definition• Recovering report by discovering hidden (missing) causality in radiology reports.
• Extraction of mechanical features from images to resolve perceptual errors and learning ofqualification test lists to resolve knowledge gaps.

Dataset (Hidden-RAD dataset)• “Hidden-RAD dataset” is a set of radiology reports including Findings, Impressions, and Hidden
causality.
• The current data was augmented based on MIMIC-CXR and automatically generated in report format
through GPT-4.
1.
2.
3.

Waite, S., Scott, J., Gale, B., Fuchs, T., Kolla, S., & Reede, D. (2017). Interpretive error in radiology. American Journal of Roentgenology, 208(4), 739-749.
Brady, A. P. (2017). Error and discrepancy in radiology: inevitable or avoidable?. Insights into imaging, 8, 171-182.
Pow, R. E., Mello‐Thoms, C., & Brennan, P. (2016). Evaluation of the effect of double reporting on test accuracy in screening and diagnostic im aging studies: a review of the evidence. Journal of
medical imaging and radiation oncology, 60(3), 306-314.

9Comparison• No dataset containing hidden causality for lesions.
• This dataset was augmented based on MIMIC-CXR.
Resource

Dataset nameReleas
e yearSize

Hidden-RAD dataset
(Our)

2024

PadChest

X-Ray

ReportImpression

Hiddencausality

300

O

O

O

O

2020

206,222OOOXMIMIC-CXR

2019

371,920

O

O

O

X

CheXpert

2019

224,316OX

OX

National Institutes ofHealth Repository

2017

112,120

O

X

O

X

PLCO

2012

185,421

O

X

O

X

X-Ray: Chest radiography, Report: A doctor's report on chest radiographs, Impression: classification of patients written by
doctors in text form, Hidden causality: Interpretation of reading procedures in radiological images.

Expected result (Contribution)

10• Explainable CAD technology.
• Reducing diagnostic error• Hidden causality is included in the report through systematicinterpretation of the images. This is a complete scan of theradiograph and contributes to reducing the number of diagnosticerrors.

1.

2.
3.
4.

Kok, E. M., Jarodzka, H., de Bruin, A. B., BinAmir, H. A., Robben, S. G., & van Merriënboer, J. J. (2016). Systematic viewing in radiology: seeing more, missingless?. Advances in Health Sciences Education, 21, 189-205.
Waite, S., Scott, J., Gale, B., Fuchs, T., Kolla, S., & Reede, D. (2017). Interpretive error in radiology. American Journal of Roentgenology, 208(4), 739-749.
Brady, A. P. (2017). Error and discrepancy in radiology: inevitable or avoidable?. Insights into imaging, 8, 171-182.
Pow, R. E., Mello‐Thoms, C., & Brennan, P. (2016). Evaluation of the effect of double reporting on test accuracy in screening and diagnostic im aging studies: areview of the evidence. Journal of medical imaging and radiation oncology, 60(3), 306-314.

11Data release plan (2024-2026)

Data for User-Adaptable Explanation Report Generation of Medical Interpretation and Diagnosis2026.11

Hidden-RADDatalicense

Knowledge
Graph

CC
BY-NC-SA

KG
enriched

CC
BY-NC-SA

KG
enriched

PhysioNet
1.5.0

CC
BY-NC-SA

KG for QA+ patientinstance

PhysioNet
1.5.0

CC
BY-NC-SA

Image
Hidden Interpretation
sourcelicenseDiscovery Data>3000

Medical expert
Crowdsourcing

ReportGeneration Task

QA
Task

Integrated
interpretation task

2025.11>2000

knowledge+LLM

2024.11
HIDDENRADChest-2
2024.5HIDDENRADChest-1

1000
MIMIC CXR
base+sync

300
MIMIC CXR
basecrowdwork
log withreport ref.

KG+
generative

crowdwork
log w/oreport ref.

generative
GPT basedclassification

(The initial) Early experimental results

12• Use the same input for tasks• Perform analysis through GPT-V
Metric

Model

BLEU1
GPT-V 25.41

BLEU2
11.32

BLEU3
7.11

ROUG
E-1
21.71

ROUG
E-2
7.14

ROUG
E-I
1.69

METEOR0.19

BERTScore
0.34

Important dates• Kick-off Event: March 29, 2024
• 1st-Training Set Release: May 31, 2024 / Quantity: 300
cases• 2nd-Training Set Release: Jun 30, 2024 / Quantity: 300
cases• Test Set Release: Dec 8, 2024 / Quantity: 400 cases• System Submission Deadline: Jan 8, 2025
• Evaluation Results Release: Feb 1, 2025
• Submission Due of Participant Papers (draft): March 1, 2025
• Camera-Ready Participant Paper Due: May 1, 2025
• NTCIR-18 Conference: Jun 10-13, 202513