Visual and Domain Knowledge for Professional-level Graph-of-Thought Medical Reasoning
Authors: RINA BAO
Shilong Dong
Zhenfang Chen
Sheng He
Ellen Grant
Yangming Ou
Venue: ICML 2025 spotlightposter (2025)
Keywords: Medical
Medical Reasoning
Primary area: applications->health_medicine

Abstract: Medical Visual Question Answering (MVQA) requires AI models to answer questions related to medical images, offering significant potential to assist medical professionals in evaluating and diagnosing diseases, thereby improving early interventions. However, existing MVQA datasets primarily focus on basic questions regarding visual perception and pattern recognition, without addressing the more complex questions that are critical in clinical diagnosis and decision-making. This paper introduces a new benchmark designed for professional-level medical reasoning, simulating the decision-making process. We achieve this by collecting MRI and clinical data related to Hypoxic-Ischemic Encephalopathy, enriched with expert annotations and insights. Building on this data, we generate clinical question-answer pairs and MRI interpretations to enable comprehensive diagnosis, interpretation, and prediction of neurocognitive outcomes. Our evaluation of current large vision-language models (LVLMs) shows limited performance on this benchmark, highlighting both the challenges of the task and the importance of this benchmark for advancing medical AI. Furthermore, we propose a novel ``Clinical Graph of Thoughts" model, which integrates domain-specific medical knowledge and clinical reasoning processes with the interpretive abilities of LVLMs. The model demonstrates promising results, achieving around 15\% absolute gain on the most important neurocognitive outcome task, while the benchmark still reveals substantial opportunities for further research innovation.

Visual and Domain Knowledge for Professional-level Graph-of- trapuhhf-Thought Medical Reasoning
Rina Bao
Shilong Dong²
Zhenfang Chen
Sheng He
Patricia Ellen Grant
Yangming Ou'
Abstract
LVLM works?
Clinicians' interest?
Medical Visual Question Answering (MVQA) re- quires models to answer questions related to medical images, offering significant potential assist medical professionals in evaluating and di- agnosing diseases, thereby improving early in- terventions. However, existing MVQA datasets primarily focus on basic questions regarding vi- sual perception and pattern recognition, without addressing the more complex questions that are critical in clinical diagnosis and decision-making This paper introduces new benchmark designed for professional level medical reasoning, simu- lating the decision-makinn process. We achieve this by collecting MRI and clinical data related to Hypoxie-Ischemic Encephalopathy, enriched with expert annotations and insights. Building on this data, we generate clinical question-answe answer pairs and MRI interpretations enable compre- hensive diagnosis, interpretation, and prediction of neurocognitive outcomes. Our evaluation of current large vision- language models (LVLMs) shows limited performance on this benchmark, highlighting both the challenges the task and the importance of this benchmark for advancing medical AI. Furthermore, we propose novel "Clinical Graph of Thoughts" model, which in- tegrates domain-specifie medical knowledge and clinical reasoning processes with the interpretive abilities of LVLMs. The model demonstrates promising results, achieving around 15% absolute gain on the most important neurocognitive out- come task, while the benchmark still reveals sub- stantial opportunities for further research innova- tion. Project page: https: : ggihhbt..hh...ggggg com/ 13-resea www.eeeaarcceeeeeeee wwwrrch/HHHE. HEEHHTE-RRRR Hea-Reason ng
Knowledge?
Question Type?
(a)
Visual
X
ModailnyCrgann?
2-year
X
Neurocag
(b)
Visual Clinical
Glinieat Graph Thoughts
Easy cilntcut--mctevontt
Clinical intermediate knowlodge task
Complex cilnncat-imperratttt
Figure I. Differences between existing MVQA and the pro- posed dataset (b). Compared to existing methods that focus on simple, clinically irrelevant tasks (green eircle), our approach targets more complex, clinically important tasks that align with clinicians' interests tasks that traditional LVLMs fail to address. To tackle this challenge, we propose Clinical Graph of Thought Model, a model that decomposes these complex, clinically signifi- cant tasks (red circle) into graph of simpler clinical and interme- diate knowledge tasks (purple circle), leveraging both visual and clinical knowledge.
1. Introduction
Large Vision- Models (LVLMs) have achieved significant success in areas like natural image and video analysis (Bordes et al., 2024; OpenAI, 2023; 2024) and natural language processing (NLP) (Ouyang et al., 2022: Touvron al., 2023). These models leverage advancements in neural networks, large-scale datasets, and computational resources, enabling capabilities such automated image captioning and complex visual question answering. How- ever, the medical field, LVLMs are still their early stages of development (Hu et al., 2024; He et al., 2023), with research progressing but not yet matching the success seen in non-medical domains. key challenge lies in the limited availability of suitable medical data, hindering their broader application.
BBoston Children's Hospital and Harvard Medical School, Boston, USA 'New York University 'MIT-IBM Watson AI Lab. Correspondence to: Zhenfang Chen <hiercasoning @aaingg@@@ammmmmmmmmm.. comil Yangming Ou <yangming.ov
seiiddens.harvard.cdd
Many medical datasets for LVLMs (Alkhaldi et al., 2024;
Li et al., 2024; Moor et al., 2023; Saab et al., 2024) are organized in a visual vides natural inputs for LVLMS and offers a flexible way
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
format, which pro-

---

Visual and Domain Knowledge for Professional-level Graph-of-Thoughh Medical Reasoning
to test their capabilities across different aspects of medi- cal data. However, most of these datasets are labeled by general annotators. As shown in Fig. 1, they focus gen- eral gueestion-answer pairs (e.g., "Q: What is the organ? A: Brain"), "): primarily assessing basic pattern recognition and object classification. It remains uncertain how effectively these datasets can aid Al models in understanding complex MRI tasks that clinical experts prioritize, such as "What is the predicted two-year neurocognitive outcome for this individual?" " and enhancing clinical workflows (see Fig- ure 3 (B)). This paper introduces benchmark designed to challenge LVLMs' performance in professional- ical reasoning. focusing on questions relevant to medical professionals to support diagnossis and prognosis.
as shown in Fig. 2. We defined six tasks: Lesion Grad- ing, Lesion Anatomy, Lesion in Rare Regions, Neurocogni- five Outcome Prediction, and MRI Interpretation Summary, Each task targets distinct aspects of clinical significance. Specifically, Lesion Grading assesses HIE lesion sever- ity; Lesion Anatomy identifies the anatomical locations of lesions; Lesion in Rare Locations determines whether injuries are present in typical or atypical regions: MRI In- jury Score generates overall brain injury severity score, widely used as a biomarker for predicting adverse two-year outcomes and is implemented in many HIE clinical trials worldwide (Laptook et al., 201 Shankaran et al., 2017b); Neurocognitive Outcome Prediction forecasts two-year neurocognitive outcomes, crucial since 30%-50% of HIE patients experience adverse outcomes (Graham et al., 2008; Lee et al., 2013; Weiss al., 2019); and MRI Interpreta- tion Summary provides brief, structured MRI summaries from neonatal radiologists and neonatologists' views, high- lighting key findings on neonatal brain injury. Totally, our benchmark comprises 749 professional question-answe answer pairs and 133 MRI interpretation summaries derived from unique MRI images of 133 individuals.
med-
Developing such a professional-level medical reasoning benchmark poses challenges. Medical data is often scarce and subject to strict sharing and usage limitations. Addi- tionally, labeling this data is resource-intensive and costly, requiring medical professionals like radiologists and neona- tologists to carefully examine MRI images, consult with patient families, and compose diagnostic reports. To build this benchmark, we compiled a decade's worth of MRI im- ages and clinical expert-verified interpretations related to Hypoxic-Ischenin Encephalopathy (HIE, a neonatal brain dysfunction that occurs in 1-5/1000 neonates) from 133 individuals, in collaboration with experts in pediatric neu- roradiology, nconatal MRI interpretation, neonatology, and neonatal neurology. This interdisciplinary team brings criti- cal expertise in managing neonates with HIE, resulting in high-quality benchmark closely aligned with real-world neonatal care and long-term neurodevelopmental outcomes.
We evaluated a range of state-of-the-ar general and medical LVLMs our benchmark, finding while these models per- form well on general datasets focusing on visual perception and pattern recognition, they demonstrate limitations on our benchmark, which requires both visual perception and specialized medical knowledge for reasoning.
To address this gap, we propose the Clinical Graph of Thought Model (CGoT), which emulates the diagnostic pro- cess through clinical mhowledge-guidee prompting. This approach not only enhances model per- formance but also improves transparency. Additionally, CGoT incorporates domain-specific clinical knowledge as input both visually and textually to strengthen the predic- tive power of LVLMs. Experimental results demonstrate a significant improvement, with performance gains exceeding over 15% across tasks, although further progress is needed.
graph-of-thoughee
Using this raw annotated data, we generated question- answer pairs along with a comprehensive MRI interpre- tation summary that reflects expert reasoning, enabling us to evaluate LVLMs' capacity to comprehend these ques- tions, identify correct answers, make accurate decisions, and predict future neurocognitive outcomes. We defined the questions with input from medical experts and obtained the answers by parsing clinical reports.
In summary, the key contributions of this paper are: (I) the creation of the novel HlE-RRassoing benchmark, which replicates clinical decision- makion workflows to evaluate L.V.Mss in professional- medical reasoning, represent- ing the first medical reasoning benchmark that combines clinical visual perception with professional-level medical knowledge; (2) a comprehensive evaluation of state-of-the- art general and medical LVLMs, revealing their limitations in medical domain knowledge; and (3) the introduction of CGoT, a clinically guided model that mimics the clinical decision-making process, integrating medical expertise with LVLMs to enhance decision-making support.
After acquiring raw MRI reports from experts, we formu- lated tasks that address the priorities of medical profession- als, as opposed to general pattern recognition questions in existing benchmarks. We modeled clinical decision-making workflows and designed six tasks, as shown in Fig. 2. Each task was meticulously validated by an expert neonatal radi- ologist or neonatologist, depending on the task, with over 30 years of experience, ensuring clinical relevance, signifi- cance, and accuracy for model training. We prompted large language models to generate answers by parsing medical reports and manually verified the responses for accuracy.
This approach establishes benchmark that mirrors es- sential components of professional-Ivvel MRI assessment
2

---

Visual and Domain Knowledge for Professional-Leve Graph-of-Thoughh Medical Reasoning
Table 1. Comparison of Medical VQA Datasets, HIF-Reasoning uniquely integrates climician-laheled, profession-level knowledge and
requires Al models to predict future outcomes, crucial for clinical early interventions.
General Medical
Disease-Speciffe Knowledge
Future Prediction
Data
Clinical Workflow
Dataset
Knowicdge
Source
Related
VQA-Med (Ben Abacha et al., 2019)
Weak
x
Medical Database
Low
OmniMedVQA (Hu et al., 2024)
Weak
X
Medical Database
Low
VQA-RAD (Lau al., 2018)
Weak
Medical Students & Fellows
Medium
PathVQA (He al., 2020)
Weak
X
Textbooks
Low
SLAKE (Liu et al., 2021)
Weak
X
Medical Database
Low
HIE- Rei--nean
Strong
Clinical Report
High
2. Related Work
tion, Anatomy Identification, Disease Diagnosis, Lesion grading, and Other Biological Attributes. SLAKE (Liu et al., 2021) contains only two types of questions about organ and disease on medical images. VQA-RAD (Lau et al., 2018) focuses on visual attributes of medical images, such as modality, organ, color, size of objects, counting, etc. Path VQA (He et al., 2020) supports specific questions on pathology images.
Large Vision Language Models for VQA. LLMs provide a promising performance in natural language processing and it has been extended to moltiimodallity// LVLMs, integrating the textural information from texts and visual information from images. Most famous foundation models start to support the texts and images, such as GPT4o (OpenAI, 2024) and Gem- ini (Saab et al., 2024) which accept as input any combination of text, audio, image, and video and generate any combina- tion of text, audio, and image outputs. These models are trained on general vision- -aiiun-e tasks that may or may not include medical data. Medical vision-language tasks require strong domain knowledge and professional skills. Thus, efforts have been made to integrate medical specific knowledge into foundation models by finetuning the general LVLMs, such as the Mld-Flamingo (Moor et al., 2023), LLava-Med (Li et al., 2024), and MiniGPT-Med (Alkhaldi et al., 2024). Med-Flamingo is a multimodal model adapted to the medical domain by pre-training on paired and in- terleaved medical image-text data from publications and textbooks (Moor et al., 2023). LLava-Med extends mul- timodal instruction-tuning to the biomedical domain as a biomedical multimodal conversational assistant (Li et al., 2024). MiniGPT-Med (Alkhaldi et al., 2024) is a vision- language model derived from large-scale language models and tailored for medical applications which shows superior performance in VQA benchmarks on medical imaging. In this paper, we evaluate models on medical disease by ask- ing complex and clinical-imper important questions that require specialized knowledge from medical experts.
These datasets are designed to answer the question from the image content without requiring deep professional medical knowledge, domain inference, or clinical-workfllw- related knowledge. Questions about general information on medical images such as the modality, organ, and disease information are easy and Fig. ((a) and Table 1). which are not ones. Other types of questions, such as the complex and clinical-important task (see Fig. 1(b) and Table I) are the core questions for clinicians but failed for the traditional LVLMs. To fill this gap, we proposed a template bench- mark, named -laa-neaonnng, focusing on answering the cliiiciansinterrsseedd questions using both general medical knowledge and strong disease-speeific knowledge for future outcome prediction, which cannot be performed on other datasets (Table 1).
workfloowirrelevan (see
Prompt Engineering in Large Language Models. L.LMs face limitations in complex reasoning tasks that require specialized knowledge (Sun et al., 2023). These limita- tions can be addressed using prompting techniques, such as think-of-graph (ToG) (Sun et al.,, 2023) and chain-of-thought (CoT) (Wei et al., 2022), which encourage LLMs to gener- ate step-by-ster solutions for complex tasks. ToG treats the LLM as an agent to interactively explore related entities and relations on knowledge graphs (Sun et al., 2023), while CoT breaks down multi-step problems into intermediate steps. These prompt engineering techniques are primarily used for VQA on text inputs. In this paper, we extend this ap- proach by introducing a clinical graph-of-thought prompting method for medical VQA, answering complex and clinical- important tasks by decomposing it into graph of thoughts with easy clinical and intermediate knowledge tasks (see Fig. and Fig. 3).
Benehmark Datasets for VQA in Medical Images. Bench- mark datasets are important to evaluate the performance of AI models for visual question answering and few medical benchmarks have been developed, such as VQA- med (Ben Abacha et al., 2019), OmiMed-VQA (Hu et al., 2024), SLAKE (Liu et al., 2021), VQA-RAD (Lau et al., 2018) and Path3000 (He et al., 2020). The VQA- med (Ben Abacha et al., 2019) is designed to answer four categories of clinical questions: Modality, Plane, Organ System, and Abnormality. OmiMed-VOR (Hu et al., 2024) provides five different question types: Modality Recogni-
3

---

Visual and Domain Knowledge for Professional-eewe Graph-of-Thoughh Medical Reasoning
HIE- Rea-oingg
Task 1. Lesion Grading
Task 4. MRI Injury Score
Q: What is HIE lesion percentage this MRI2 (Open)
What injury score MRIP (MC)
A: The MIR shows that 6, 93% of brain
A: B: 3.33 D:4
volume injured
Q What is the severity level of the HIE
lesions? (MC)
Task 5. Neurocognitive Outcome
C:3 (5%-500 D: (+50%)
(+1%)
(1%-5%)
3D MRI Volume
9: predicted neurscognitive outcome for this patient et years? (MC)
Task 2. Lesion Anatomy
A: (Adveregt B. (Normal)
9:
resions affected by HIE in this MHIP (Open)
Task 6. MRI Interpretation
A. Paraiital Lateral GM left, Parietal WM left, Occipital left,
HIE injury
Q Summarize MRI findings. (Ogen)
A The MRI shows that 6, 5.93% of the brain
volume is injured. The affected regions include Parietal
Task 3.
Lesion in Rare Locations
WM left,
Occipital left, ... Additionally, HIEE lesion regions that are not typical for HIE injury include Occipital Medial GM off, Frontal Opercular GM Left, Occipital Lateral GM Left.... MRI injury soore for
Are there recions that are uncommoniy. intured in HIEZ (Open) A: Occipital Medial Left, Frontal Opercular Occipital Lateral GM Left...
Figure 2. HIF- Rel-onnag Dataset and Task Overview. This figure illustrates the HIF-RReasoning dataset structure, comprising six tasks designed to assess reasoning capabilities for HIE MRI interpretation and outcome prediction using MRI data. This dataset supports the development of reasoning models by providing both open open-ended (open) and multiple-choice (MC) questions, encouraging comprehensive understanding of HIE MRI and crucial aspects of MRI towards prognosis.
3. HIE-Reasoning Dataset
sion maps across the HIE cohort. This atlas highlights brain regions most commonly affected by HIE, aligning with clinical knowledge on areas frequently impacted by HIE injuries (Shankaran et al., 2012; Weiss al., 2019). 22.5% threshold, recommended by an experienced radiol- ogist, was applied to generate commonly affected regions. The brain regions most frequently impacted by HIE include the basal ganglia, internal capsules, thalamus, perirolandic cortex/subcortical white matter, temporal lobes, cerebral white matter, brainstem, and vermis. For each case, primary injury regions (i.e., commonly affected areas) identified, with uncommon injury sites noted when affected regions fall outside the typical injury profile.
We define series of tasks, illustrated in Fig. 2, for LVLMs to perform professional-level clinical reasoning.
Task 1. Lesion Grading. This task quantifies brain injury by estimating the percentage of brain volume affected by HIE lesions and assessing the lesion severity extents. This task outputs lesion volume percentage and severity. Accord- ingly, we defined two specific tasks as in Fig. 2: one to evaluate lesion percentage and another to categorize lesion severity into four levels: level 0 (< 1% brain injury), level (1%-5%), level (5%-50%). and level 3 (> 50%). Manual lesion annotations served as the ground truth (Bao et al., 2025b) and were used generate the ground-trume answers.
Task 4. MRI Injury Score. This task outputs over- all injury score of MRI, providing a standardized measure of injury severity to guide treatment and predict outcomes. Expert scoring of neonatal MRI is currently employed in clinical trials predict 18- to 22-month outcomes (Lap- took al., 2017b; Shankaran al., 2017b). The Na- tional Institute of Child Health and Human Development (NICHD) Neonatal Research Network (NRN) injury scoring system (Shankaran et al., 2012; 2015) is the most widely used MRI injury score for assessing brain injury severity in HIE. To facilitate machine learning modeling, we consoli- dated the original 6-1evel score into 4 levels, instructed by radiologist, as certain distinctions within the 6 levels were clinically ambiguous and difficult to quantify for machine learning purposes.
Task 2. Lesion Anatomy. This task identifies specific brain regions affected by lesions. The brain is divided into 62 regions of interest (ROIs) based on standard anatomical ref- erences (Doshi et al., 2016; Morton et al., 2020), and the task outputs the lesioned regions among these ROIs. Such task is critical for accurately predicting functional impair- ments, assessing injury severity, and informing prognosis in conditions like HIE. For example, lesions in the basal ganglia or thalamus are strongly associated with adverse neurocognitive outcomes.
Task 3: Lesions in Rare Locations. This task identifies lesions caused by HIE and categorizes affected regions as common or uncommon, helping to determine if additional attention is needed for the patient. Specifically, (Bao al., 2025b) provides a lesion atlas generated by aggregating lesion masks from each patient, resulting in statistical le-
Task S. Ywo-Year Neurocognitive Outcome. This task pre- diets the patient's 's neurocognitive outcome at two years, aid- ing clinicians anticipating long-term impacts and planning

---

Visual and Domain Knowledge for Professional-evel Graph-of-Thought Medical Reasoning
appropriate interventions. Specifically, this task outputs binary label where the two-year neurocognitive outcome is categorized as either normal (0) or adverse (1), based clinical criteria and NRN recommendations (Laptook et al., 201 7b; Shankaran et al., 201 7b).. An outcome is classified as adverse if any of the following conditions are met: a Bayley- III cognitive score below 85 in any domain, a GMFCS level between 2 and 5, blindness, hearing impairment, or if the patient is deceased in two years. Otherwise, the outcome is considered normal (Laptook et al., 2017b; Shankaran et al., 2017b; Bao & Ou, 2024).
4. Proposed Clinical Graph of Thought Model
A straightforward approach to addressing the HIE- Reasoning task is to directly input MRI images into an LVLM to generate predicted outcome as shown in Fig. 3 (A). However, without integrated clinical reasoning and domain-specifee knowledge, the LVLM's performance is often no better than random chance due to its lack of essen- tial interpretative capabilities as shown in our experimental analysis in Sec 5. To overcome this limitation, we propose Clinical Graph of Thought Model (CGoT), which incorpo- rates clinical knowledge into the LVLM, guiding it through clinician like diagnostic process, as depicted in Fig. 3 (B)- (C). This framework yields more reliable neurocognitive outcome predictions. This novel approach demonstrates how structured clinical reasoning, when coupled with ad- vanced AI, can elevate medical image interpretation to new levels of precision and reliability.
Task 6. MRI Interpretation Summary. The initial Tasks to cover the key aspects of MRI interpretation: estimating lesion volume, identifying lesion locations, detecting atypi- cal lesion locations, and rating injury severity. These tasks structured based on a neonatal MRI summary template recommended by radiologists, enabling the generation of comprehensive MRI interpretations for patients.
As detailed in Fig. 3 (C), CGoT systematically incorporates insights from previous clinical evaluations, mimicking real- world diagnosis workflows. The framework works with two innovations, (1) clinical graph- structurd prompting for medical reasoning and (2) visual and professional-level clinical knowledge as input.
Dataset Details. The HIE- Reasoning is the first publicly available HIE dataset that integrates MRIs, clinical infor- mation, neurocognitive outcomes, and includes question- answer (QA) pairs along with comprehensive MRI inter- pretation summaries. It was retrospectively collected from Massachusetts General Hospital (MGH) between 2001 and 2018. The dataset includes high-quality MRIs acquired within the first 0- 14 days after birth from neonates with HIF, totaling 133 MRI scans. Expert lesion annotations are avail- able for all cases, provided by one clinical fellow with over 3 years of experience and three additional neuroradiologists with over 5, 5, and 30 years of experience, respectively. Neu- rocognitive outcome data was retrospectively gathered from follow-ups ups conducted by more than four neonatologists and neurologists, assessing outcomes at least 18-22 months post- birth. In total, the dataset comprises 749 question-anse -antwer pairs and 133 MRI comprehensive interpretation summary.
4.1. Clinical Graph of Thought for Reasoning
In practice, a neonatologist would interpret a radiologist's report to assess brain injury severity. At the core of CGoT is a structured "reasoning graph of thought" that mirrors clini- cians' step-by-step clinical diagnosis and decision- process. Similarly, CGoT leverages the knowledge gained from each preceding task to guide the LVLM, enabling it to address complex questions by progressively refining its understanding in a stepwise manner.
CGoT consists of six key tasks that replicate the domain- specific expertise of medical specialists. These tasks include lesion grading (informed by radiologists and neonatologists), anatomical localization (radiologist-driven)) injury scoring (led by neonatologists), and two-year outcome prediction (integrating insights from radiologists, neonatologists. and neurologists). Each task builds upon the previous one, cre- ating a reasoning pipeline that aligns with the sequential clinical evaluation process. Finally, the outputs of these tasks are integrated and structured to form the MRI inter- pretation summary by MRI summary template. Fig. 3 (B) depicts CGoT 's reasoning graph, which integrates critical diagnostic elements and emulates the specialized expertise of clinical roles, forming an interactive, layered diagnostic approach. For example, in the lesion anatomy task, CGoT, after receiving clinical guidelines, enables the LVLM to emulate a radiologist's 's role, identifying regions affected by HIE with statements like, "I can identify HIEE-njured brain regions in this patient."
Justification for Small Sample Size. While this sample size may appear modest for contemporary machine learn- ing applications, several factors underscore its significance. First, HIE diagnosis and treatment demand specialized ex- pertise available only at select tertiary care centers, making large-scale data collection inherently challenging. Second, this benchmark emerges from decades of clinical collabo- ration across multiple specialties' efforts, establishing its unique value in neonatal care research. The dataset's 's com- prehensive nature, depth of clinical knowledge, combining diverse clinical indicators and expert annotations, compen- sates for its size limitations and provides a foundational resource for advancing medical Al in this critical domain.
5

---

Visual and Domain Knowledge for Professional-leve Graph-of-Thouge Medical Reasoning
A. LVLM Only
Neurocognitive Outcome
CER
B. LVLM +Clinical Reasoning
klnttly polont!
Radiologist
sewar by of thi tty
the iterid boar ingitie
con poed Zoear aunostsee
MRI injury Score
Neoratologist
Lesion Grading
Anetomy
Neurocogniivve Outcome
Lesion Rate Location
can detemmereeeeeeeee there traay rus ruse locat ona
Neurologist
C. CGoT
Task Lesion Grading
(1%-5%
2-year neurocogniltive
Visual knowledge
Clinical
Visual knowledge
Textual Clinical Knowledge
injury score
Visual knowladge
Textual Clinical Knowledg
+-2 has high probability for lesion regions
Injury
clinicl trisis
Relation biomarkers
the severity losions?
Previous answer: lesion anatomy
Previous answer. lesion grading
Previous answer: injury
Previous answer: lesion grading
A: Acterre
Q: What is MRI injury score on this MRI?
Provious lesion
Task 2.
Lesion Anstomy
Visual knowledge
Q: What the predicted neurocognitive outcome for this patient at years?
Textual Clinical Knowledgo
His typ qured bra ons affected
brain this MA1?
A: (Cray
matter injury)
Visual knowledge
Rowe
oelectioo
A: Vermis Thalamus
Task 3. Lesion in rare location
Visual knowledge
Textual Ciinical Knowledge
Textual Clinical knowladge
Brain ROIs
A: Dcciphal
Meta clinical knowledge (share across patients)
Previous answer: lesion anstomy
Are there regions that are uncommonly injured HIE?
Individual clinical knowledge (patient specific)
Figure 3. (A) The LVI.M-olly approach, which directly inputs MRTs for outcome prediction, lacks structured clinical reasoning and thus performs poorly in outcome accuracy. (B) The proposed CGoT incorporates clinical reasoning steps that mimic clinician's 's diagnostic workflow. (C) CGoT organizes tasks into sequential, dinician-innppreed diagnostic flow, making reliable neurocognitive outcome predictions by progressively integrating visual and textual clinical knowledge across interrelated diagnostic steps.
4.2. Clinical Knowledge Representation
Raw MRI. Apparent diffusion coefficient (ADC) maps serve as the primary imaging modality for identifying HIE abnormalities by radiologists (Douglas-Esoe) Escobar & Weiss, 2015; Wei et al., 2019; Liauw al., 2009) and are used as visual knowledge input for the LVLM.
CGoT relies on two primary knowledge sources for each task: visual knowledge and textual clinical knowledge, Vi- sual knowledge includes raw MRI data and derived visual features, such as lesion probability and brain anatomical re- gion information. Textual clinical knowledge is divided into essential clinical contextual knowledge for answer- ing the current question and individual clinical knowledge, which is obtained from the previous task's 's output. This approach allows CGoT to sequentially build understanding 's question by leveraging preceding an- swers, thereby simulating a clinician's 's reasoning process. Below we dive into the knowledge detail.
Lesion Probability. Neuroradiologist detect acute brain injury from HIE by identifying regions with abnormally low ADC values, which indicate reduced water diffusion due to ischemic necrosis (Weiss et al., 2019). However, distinguish- ing these regions from normal variation is challenging due to variability ADC values across different brain regions (Ou et al., 2017: Sotardi et al., 2021). To address this, Bao al. (Bao et al., 2025b) introduced ZDD maps to normalize ADC values across brain voxels. By applying threshold of -2 on ZADC, a lesion prediction map is generated, offering performance comparable to more complex deep learning methods for lesion detection (Bao et al., 2025a;b). Details
and tackle each task's
4.2. VISUAL KNOWLEDGE
There three types of visual knowledge: raw MRI, lesion probability, and patient brain anatomy.
6

---

Visual and Domain Knowledge for Professional-evel Graph-of-Thougge Medical Reasoning
of this information are provided in the appendix.
Evaluation Metries. We design task-specific evaluation metrics in Hll--Reasoning to comprehensively assess model performance as summarized in Table 2. For Lesion Grading task, we use accuracy and Mean Absolute Error (MAE) to assess models' prediction performance of severity levels and lesion percentages, respectively. The Fl Score is ap- plied to measure the classification and retrieval quality of model-predicted lesion regions compared to ground truth in tasks such as Lesion Anatomy Identification and Lesion in Rare Locations. To evaluate predictions for MRI Injury Scores, we use accuracy, while for the two-year Outcome task, we utilize the average inter-class accuracy (defined as the mean accuracy across all output categories) to address imbalanced and biased ground truth label distribution. Fi- nally, the ROUGE-L Score (Lin, 2004), capturing content overlap and fluency by comparing generated answers to ex- pert references, commonly in medical summarization tasks with LLM (Tang et al., 2023), is employed to measure the quality of the MRI interpretation summary task.
Brain Anatomy. Knowledge of patient-specific brain anatomy is essential for accurate interpretation. Brain re- gions are identified using a standard set of predefined 62 ROIs. To enable the LVLM to understand brain anatomy, a normalized brain structure map with these 62 ROIs in standard brain space was transformed to each patient's 's in- dividual space using the DRAMMS tool (Ou et al., 201 I). This patient-specific anatomy map provides the LVLM with the same anatomical knowledge a radiologist would use, enhancing its ability to interpret brain region localization during tasks. Details of the ROIs in the standard atlas space are provided in the appendix.
4.2.2. TEXTUAL CLINICAL KNOWLEDGE
Answering expert-level questions in HIE diagnosis requires advanced clinical knowledge and patient-speciff insights. We define two types of textual clinical knowledge: meta clinical knowledge and individual clinical knowledge.
Meta Clinical Knowledge. Meta knowledge includes gen- eral disease-related insights, such as brain anatomy, lesion distributions, relationships between MRI biomarkers and outcomes, and knowledge from radiology, neonatology, and neurology. This shared knowledge across patients provides essential context for HIE diagnostic tasks. A summary of the relevant meta knowledge is provided, with further details available in the appendix.
5.2. Result Analysis
Performance of Different Baseline LVLMs. As shown in Table 2, when original MRI image slices and task descrip- tions are directly input into the LVLMs, their performance is either comparable to or worse than random guessing. This indicates a lack of sufficient clinical knowledge provide accurate answers. Med-Flamingo fails to address hallucina- tion issues in the tasks Lesion Anatomy and Lesion in Rare Locations, generating repetitive and meaningless responses; consequently, its performance is not reported for these tasks. Additionally, GPT-4o and GPT-4o-mini fail to answer cer- tain questions due to their conservative alignment, which rejects responses to questions with high uncertainty. For comprehensive evaluation, we aggregate their performance metrics as GPT4o-Series. 'hese results highlight the fact that, despite being state-of-the-ar models, these LVLMs do not inherently acquire or retain professional- medical knowledge during training. This reinforces the importance of our proposed Reasonng for evaluating LVL.Ms' ca- pabilities in professional-level medical reasoning.
Individual Clinical Knowledge. Individual knowledge is patient-specific and essential for personalized diagnostic tasks. Derived dynamically through the task's reasoning process, it allows the model to incorporate personalized clinical insights. As shown in Figure 3, meta and individ- ual knowledge components are used selectively, ensuring tailored responses for each task. Additional details about task-specific knowledge are in the appendix.
5. Experiments
We conduct comprehensive experiments to demonstrate the effectiveness of the proposed benchmark, HIE- Reasonnnn,ng, and model, Clinical Graph of Thought Model.
Effectiveness of CGoT. We also compare our CGoT with the corresponding baselines in Table 2. The results demon- strate that CGoT significantly outperforms the baselines, par- ticularly in predicting the critical two-year outcome, which is the primary task of HlE-Reasoninn and supports medi- cal experts in HIE prognosis. In addition to the two-year outcome, we observe substantial performance gap be- tween CGoT and the naive direct-output baselines across tasks such as Lesion Grading, Lesion Anatomy, and MRI Injury Score. This underscores the value of incorporating additional clinical knowledge into LVLMs. Overall, these findings highlight the effectiveness of CGoT in mimicking the diagnostic and clinical decision making process, thereby enhancing professional medical reasoning.
5.1. Implementation Details
Model Details. We begin by performing zero-shot eval- uations on six representative large vision-language mod- els (LVLMss): three general-purpose LVLMs (Gemini- 1.5-Flash (Team, 2024), GPT4o-Mini (OpenAI, 2024), and GPT4o (OpenAI, 2024)) and three medical LVLMs (MiniGPT4-Med (Alkhaldi et al., 2024), LLava-Med (Li et al., 2024), and Med- Flamilann (Moor et al., 2023)). All settings and hyperparameters are configured according to the specifications of the released versions. We provide more model details and all clinical prompts in the appendix.
7

---

Visual and Domain Knowledge for Professional-eevs Graph-of- Thouuht Medical Reasoning
Table Performance comparison of various models on Hif--Reasnning benchmark. Performance comparison HIFE- Reasonnng
benchmark reveals that directly feeding the MRI images and task descriptions into LVLMs yields poor performance. In contrast, our CGoT demonstrates enhanced reasoning capabilities, significantly aiding medical decision makinn.
Lesion
Lesion
Lesion in Rare Locations
MRI Injury
Neurocognitive
Interpretation
Grading
Anatomy
Score
Outcome
Summary
Model
Ace (1)
MAE (1)
Fl Score (f)
FI Score (1)
(c)
Inter Class Acc (1)
ROUGE-L (t)
GPT4o-Series
33.83%
0.1085
14.81%
11.47%
30.08%
52.16%
37.03%
Gemini-1.5-Fian
24.81% %
0.1977
30.17%
22.53%
30.83%
56.60%
41.47%
MiniGPT4-Medd7B
12.03%
0.3785
8.00%
4.62%
34.59%
53.95%
41.09%
LLava-Med-7B
30.83%
0,1468
24.28%
22.52%
11.28%
49.50%
42.05%
Med-Flamingo-77B
18.05%
0.4651
21.05%
50.00%
36.90%
PooT-GPT4o-SSee Series
56.25%
0.0715
34.25%
33.04% 41.47% %
51.13%
61.11%
44. 14%
1.5-Flash Flash
62.41 %
0.0703
43.57 %
49.62%
71.73% %
53.68 %
Patient 010
Q: What are HIEE lasion percentage in this MRI?
Coot: : The Hill shows 59% irnin volume injuned.
The Mill shown w 10.00%% the volume injured.
Q: What the smverity level the HIE lesions?
CGoT: Level 3 (S00-)
Level (516-509))
Q: What brain reglons am affected by HIE
OooT: The affocted legions lndl Right Bosal Gangli,
GT: The afrected righenes inslude iight Resall Conglia Left thelammss Fight thelemma..
ADC
Loft thamm Right thalomus,
Are there restons that are uncommonty Intured HIEZ
coal Ci
Cingurate
Liiiimbic Ding Right...
mbbi Cingrrate for injuy reglons.
tysicill
injury score of MA1?
CDoT: :2:
QT:2
Q: the predicted neurocognitive outcome for this patient 2years2
AxpetstAno HIE Lesions
(dlvense).
(haverse).
MAI Interpretstion Summary by CGoT
MRI Interpretstion Summary from Ground
The shows
brain volume injured affected regions indlude loft
brain volume Inured. The affected regins include basal, ganglia, right basal gangla, thalomus, right thalsmus, ... Adotionnlly, HIE lesion regions that are not typical for HE Injuy include the left and ngght umbic cinguiate gray matter.... The MPI injury score forthis scan 1s
shows that 22
bosal sanglo, right basal, ganglia, thoiamus, right thalomus, Additionally, HIE losion
regions that are not typical for HIE inury include the left and rihht limicc cingulate gray
matter.... The Mill muuy score for scon
Figure 4. Typical qualitative result of CGoT. CGoT can generate clinically relevant intermediate outputs at each diagnostic step, such as lesion regions and MRI injury scores, which directly inform the final ywo-year neurocognitive outcome prediction and provide key biomarkers in natural language format.
in natural language.
Table 3. Ablation of Clinical knowledge and graph of thought. It shows that models incorporating both clinical knowledge and graph of reasoning improve prediction accuracy.
5.3. Ablation Study
In this subsection, we conduct series of ablation studies to assess the contribution of each component in CGoT and address the following research questions: Q1: Does the inclusion of clinical knowledge in CGoT improve perfor- mance? Q2: Does the clinical graph of reasoning in CGoT enhance answer accuracy? Q3: Are all the reasoning tasks for neurocognitive outcome prediction in CGoT essential for achieving optimal performance? Q4: Is CGoT robust to minor inaccuracies in its intermediate clinical tasks?
Clinical
Graph
Neurocognitive Outcome Prediction
Knowledge
of Thoughts
71.73%
X
52.43% (L 19.30%)
X
51.89% (1 19. 99%
54.44% (4 17.29%
Qualitative Result of CGoT. The graph-of-thought. rea- soning approach in CGoT offers greater transparency and provides interpretable intermediate steps leading to the fi- nal two-year outcome prediction. A qualitative example shown in Fig. 4. CGoT can progressively generate clin- ically relevant information at each step of the diagnostic process (e.g.. "the affected regions include the right busal ganglia and left thalamus" and "MRI injury score is 2" in Fig. 4). These intermediate outputs are directly relevant to the clinical question (e.g., What is the two-year neurocogni- tive outcome for this patient?) and provide key biomarkers
As shown in Table 3, models without clinical knowledge as input and without graph of reasoning perform worse. The combination of both clinical knowledge and the rea- soning graph yields the highest prediction accuracy, under- scoring the importance of structured, clinically -informedd approach in neurocognitive outcome prediction. The re- sults also demonstrate that the clinical graph of reasoning in CGoT significantly improves answer accuracy. The largest accuracy drop occurs when the reasoning graph is absent, highlighting its critical role in the model's reasoning process
8

---

Visual and Domain Knowledge for Professional-evvel Graph-of-Thought Medical Reasoning
der real conditions, further supporting the practical utility of our clinically grounded, stepwise reasoning framework CGoT (answering Q4).
Table Ablation of CGoT with various combinations of reasoning tasks. The results show that including all tasks, particularly the MRI Injury Score, crucial for optimal performance.
Lesion Grading
Lesion Anatomy
MRI Injury Score
Neurocognitive Outcome Prediction
6. Conclusion
We present a novel benchmark and framework for advanc- ing Medical VQA, focusing on professional-Ivee medical reasoning for neonatal brain injury. We introduce HIE- Reasoning, the first benchmark to combine clinical visual perception with specialized medical knowledge, specifically for the prediction of neurocognitive outcomes in HIE. We also evaluate state-of-the-art LVLMs, revealing their limita- tions in handling complex medical data. To address this gap. we propose CGoT, clinical knowledge-guided model that enhances diagnostic accuracy by mimicking clinical reason- ing processes. Our experiments demonstrate a significant performance improvement, underscoring the potential for further innovation in medical AI for professional reasoning.
71.73%
x
50.94% (L 20.79%)
70.12% (4 1.61%)
x
V
61.11% (1 10.62%)
(answering Q1 and Q2).
As shown in Table 4, the results indicate that including all reasoning tasks in CGoT is necessary for achieving opti- mal performance. The model's 's accuracy drops significantly when any task, especially the MRI Injury Score, is excluded. This was expected, as the MRI Injury Score is a key MRI biomarker for predicting neurocognitive outcomes at two years in many clinical trials worldwide, making it a crucial piece of clinical knowledge for outcome prediction. Further- more, this suggests that holistic approach, incorporating Lesion Grading, Lesion Anatomy, and the MRI Injury Score, is essential for high-quality neurocognitive predictions, As a medical AI model, it is clear that each task provides unique and complementary information, and omitting any one of these tasks disrupts the comprehensive reasoning process required for reliable outcome prediction (answering Q3).
Impact Statement
This paper introduces benchmark for evaluating LVLMs in professional-level medical reasoning, with a focus on neonatal Hypoxie-Ischemic Encephalopathy diagnosis and outcome prediction. The broader impact of this work lies in its potential to improve Al-driven clinical decision sup- port, thereby assisting radiologists and neonatologists in diagnosing and prognosticating neonatal brain injury more efficiently and effectively. By structuring complex medical reasoning tasks through clinical knowledge-guided graph-of- thought prompting, our approach enhances interpretability and reliability, which are critical for real-world adoption in healthcare. However, as with any AI-driven medical tool, ethical considerations, such as potential biases in training data, the risk of over-reliance on automated predictions, and the necessity of human oversight, must be carefully addressed. Future research should focus model generaliz- ability across diverse populations, sites and clinical settings to ensure equitable benefits.
Table 5. CGoT robustness to perturbations in intermediate predic- tions.
Perturbation Ratio (%)
Neurocognitive Outcome Prediction (%)
71.73
10
67.83
20
66.22
30
62.72
Although CGoT relies on prerequisite reasoning tasks, it remains robust to minor inaccuracies in these intermedi- ate steps. As shown in the second row of Table 4 (with- out MRI Injury Score: 50.94%; with MRI Injury Score: 71.73%), removing the MRI Injury Score from the reason- ing chain results in a substantial drop in outcome prediction accuracy, highlighting the critical role of prerequisite tasks in the model's performance. As demonstrated Table 5, CGoT maintains robustness when faced with small errors in intermediate clinical reasoning tasks. Specifically, when applied to the predicted MRI Injury Score in 10%-30% of test cases, the model exhibits only a gradual decline in performance. This indicates that CGoT is resilient to the types of variability and uncertainty commonly encountered in real-world clinical settings. Such robustness is essential for ensuring reliable predictions un-
A Acknowledgement
This work was funded, in part, by the Harvard Medi- cal School and Boston Children's Hospital. This work was also funded by Thrasher Research Fund Early Career Awards 02402, NIH R2INS121735, R61NS126792, and R03HD104891.
+1-level perturbations
References
Alkhaldi, A., Alnajim, R., Alabdullatef, L., Alyahya, R., Chen, J., Zhu, Alsinan, A., and Elhoseiny, M. Minigpt- med: Large language model as general interface for radiology diagnosis. arXiv preprint urXiv:2407.04100

---

Visual and Domain Knowledge for Professional-evee Graph-of-Thought Medical Reasoning
2024.
Graham, E. M., Ruis, K. A., Hartman, A. L.., Northington, F. J., and Fox, H. E. A systematic review of the role of intrapartum hypoxia-ischemia in the causation of neonatal encephalopathy. American Journal of Obstetrics and Gynecology, 199(6):877-555, 2008.
Azzopardi, D. V., Strohm, B., Edwards, A. D., Dyet, L., Halliday, H. I.,, Juszczak, E., Kapellou, O., Levene, M.,
Marlow, N., Porter, E., et al. Moderate hypothermia to treat perinatal asphyxial encephalopathy. New England Journal of Medicine, (61) 1358, 2009.
He, S., Bao, R., Li, J., Stout, J., Bjornerud, A., Grant, P. E., and Ou, Y. Computer-vision benchmark segment- anything model (sam) in medical images: Accuracy in 12 datasets. arXiv preprint arXiv:2304. 09324, 2023.
Bao, R. and Ou, Y. Boston neonatal brain injury data for hypoxic ischemic encephalopathy (bonbid-hie): li. 2-year neurocognitive outcome and nicu outcome. arXiv preprint arXiv:2411. 1.03556, 2024.
He, X., Zhang, Y., Mou, L., Xing, E., and Xie, P. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 10286, 2020.
Bao, R., Grant, E., Kirkpatrick, A., Wachs, J., and Ou, Y. Al for Brain Lesion Detection and Trauma Video Action Recognition: First BONBID-IIIE Lesion Segmen- tation Challenge and First Trauma Thompson Challenge, Held in Conjunction with MICCAI 2023, Vancouver, BC, Canada, October 16 and 12, 2023, Proceedings, volume 14567. Springer Nature, 2025a.
Hu, Y., Li, T., Lu, Q. Shao, W., He, J., Qiao, Y., and Luo, P. Omnimedvqa: new large- scale comprehensive evaluation benchmark for medical Ivlm. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 221 70-22218,, 2024.
Laptook, A. R., Shankaran, S., Tyson, J. E.,, Munoz, B.,
Bao, R., Song, Y., Bates, S. V., Weiss, R. J., Foster, A. N., Jaimes, C., Sotardi, S., Zhang, Y., Hirschtick, R. L., Grant, P. E., et al. Boston neonatal brain injury data for hypoxic ischemic encephalopathy (bonbid-hie): I. mri and lesion labeling. Scientific Data, 12(1):53, 2025b.
Bell, E. F. Goldberg, R. N., Parikh, N. A., Ambalavanan, N., Pedroza, C., Pappas, A., et al. Effect of therapeutic hypothermia initiated after hours of age on death or disability among newborns with hypoxic-ischemie en- cephalopathy: a randomized clinical trial. JAMA, 318 (16):1550-156) 2017a.
Ben Abacha, A., Hasan, S. A., Datla, V. V., Demner- Fushman, D., and Müller, H. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 Septem- ber, 2019.
Laptook, A. R., Shankaran, S., Tyson, J. E., Munoz, B.,
Bell, E. F., Goldberg, R. N., Parikh, N. A., Ambalavanan, N., Pedroza, C., Pappas, A., et al. Effect of therapeutic hypothermia initiated after 6 hours of age death or disability among newborns with hypoxic-ischemic en- cephalopathy: randomized clinical trial. JAMA, 318 (15):155- 2017b.
Bordes, F., Pang, R. Y., Ajay, A., Li, A. C., Bardes, A., Petryk, S., Mañas, O., Lin, Z., Mahmoud, A., Jayaraman, B., et al. An introduction to assionllanguag modeling. arXiv preprint arXiv:2405.1 17247, 2024.
Lau, J. J., Gayen, S., Ben Abacha, A., and Demner- Fushman, D. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Doshi, J., Erus, G., Ou, Y., Resnick, S. M., Gur, R. C., Gur, R. E., Satterthwaite, T. D., Furth, S., Davatzikos, C., Initiative, A. N., et al. Muse: Multi-atlas region segmen- tation utilizing ensembles of registration algorithms and parameters, and locally optimal atlas selection. Neuroim- age, 127:186-199 195, 2016.
Lee, A. C., Kozuki, N., Blencowe, H., Vos, T., Bahalim, A., Darmstadt, G. L., Niermeyer, S., Ellis, M., Robertson, N. J., Cousens, S., al. Intrapartum-rellan related neonatal encephalopathy incidence and impairment at regional and global levels for 2010 with trends from 1990. Pediatric Research, 74(1):50-77 00005,0-222000 2013.
Douglas-Escobar, Escobar, M. and Weiss, M. D. Hypoxic- Hypoxie-ischemic encephalopathy: a review for the clinician. JAMA Pedi- atrics, 169(4))397-403 2015.
ischemic
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang. J., Naumann, T., Poon, H., and Gao, J. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024.
Edwards, A. D., Brocklehurst, P., Gunn, A. J., Halliday, H., Juszezak, E., Levene, M., Strohm, B., Thoresen, M., Whitelaw, A., and Azzopardi, D. Neurological outcomes at 18 months of age after moderate hypothermia for peri- natal hypoxic ischaemic encephalopathy: synthesis and meta-analy analysis of trial data, British Medical Journal, 340,
Liauw, L., van Wezel-Meijler G., Veen, S., Van Buchem, M., and van der Grond, J. Do apparent diffusion coef- ficient measurements predict outcome in children with
2010.
10

---

Visual and Domain Knowledge for Professional-evel Graph-of-Though Medical Reasoning
neonatal hypoxic-ischemic encephalopathy? American Journal of Neuroradiology, (0(2):264-270, 2009.
Pinto, A. L.,, Ou, Y., Sahin, M., and Grant, P. E. Quantita- tive apparent diffusion coefficient mapping may predict seizure onset in children with sturge-weber syndrome. Pediatric Neurology, 84:32-38. 1832-38,220 2018.
Lin, C.-Y. Rouge:
A package for automatic evaluation of summaries. In Annual Meeting of the Association for Computational Linguistics, 2004. URL https::////a.. /api semant www.nntisccllaaa.. ar. . http://opusstts 964287.
Saab, K., Tu, T., Weng. W.-H., Tanno, R., Stutz, D., Wul- czyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404. 18416, 2024.
Liu, B., Zhan, L.-M.,, Xu, I.., Ma, L., Yang, Y., and Wu, X.- M. Slake: A semantically-labeie knowledge-enhaneed dataset for medical visual question answering. In 2021 IEFE 18th International Symposium on Biomedical Imag- ing (ISBI), pp. 1650-1654, IEEE, 2021.
Shankaran, S., Laptook, A. R., Ehrenkranz, R. A., Tyson, J. E., McDonald, S. A., Donovan, E. F., Fanaroff, A. A., Poole, W. K., Wright, L. L., Higgins, R. D., et al. Whole- body hypothermia for neonates with hypoxic-ischenee encephalopathy. New England Journal of Medicine, 353 (15):1774-1884, 2005.
Moor, M., Huang, Q., Wu, S., Yasunaga, M., Dalmia, Y., Leskovec, J., Zakka, C., Reis, E. P., and Rajpurkar, P. Med-
Med-flamingee flamingo: multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp. 353-367. PMLR, 2023.
Shankaran, S., Barnes, P. D., Hintz, S. R., Laptook, A. R., Tattera-Baxter, K. M., McDonald, S. A., Ehrenkranz, R. A., Walsh, M. C., Tyson, J. E., Donovan, E. F., et al. Brain injury following trial of hypothermia for neonatal bypoxic-ischaemic encephalopathy. Archives of Disease Childhood-Ferd and Neonatal Edition, (7(6). F404, 2012.
Morton, S. U., Vyas, R., Gagoski, B., Vu, C., Litt, J., Larsen, R. J., Kuchan, M. J., Lasekan, J. B., Sutton, B. P., Grant, P. E., et al. Maternal dietary intake of omega-3 fatty acids correlates positively with regional brain volumes in 1-month-old term infants. Cerebral Corlex, 30(4): 2057-2069, 2020.
Shankaran, S., McDonald, S. A., Laptook, A. R., Hintz, S. R., Barnes, P. D., Das, A., Pappas, A., Higgins, R. D., Ehrenkranz, R. A., Goldberg, R. N., et al. Neonatal magnetic resonance imaging pattern of brain injury as a biomarker of childhood outcomes following a trial of hy- pothermia for neonatal hypoxic- encephalopathy. The Journal of pediatrics, 167(5):987-999) 993, 2015.
Murphy, K., van der Aa, N. E., Negro, S., Groenendaal, F., de Vries, L. S., Viergever, M. A., Boylan, G. B., Benders, J., J., and Rgum, I. Automatic quantification of ischemic injury diffuuson-wee diffusion-weighted mri of neonatal hypoxic ischemic encephalopathy. Neurolmage: Clinical, 14:222- 232, 2017.
Shankaran, S., Laptook, A. R., Pappas, A., McDonald, S. A., Das, A., Tyson, J. E., Poindexter, B. B., Schibler, K., Bell, E. F., Heyne, R. J., et al. Effect of depth and duration of cooling on death or disability at age 18 months among neonates with hypoxic-ischemic encephalopathy: a ran- domized clinical trial. JAMA, 3 318(1):57761) 2017a.
OpenAl. Gpt-4o system card, 2024. URL https::// arxiv. orgo hbtpa/a./222222221
OpenAL, R. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023.
Ou, Y. Sotiras, A., Paragios, N., and Davatzikos, C. Dramms: Deformable registration via attribute matching and mutual-saliency weighting. Medical image analysis, 15(4):622-639, 2011.
Shankaran, S., Laptook, A. R., Pappas, A., McDonald, S. A., Das, A., Tyson, J. F., Poindexter, B. B., Schibler, K., Bell, E. F., Heyne, R. J., et al. Effect of depth and duration of cooling on death or disability at age 18 months among neonates with hypoxic-ischemic encephalopathy: a ran- domized clinical trial. JAMA, 318(1):577-1) 2017b.
Ou, Y., Zöllei, L., Retzepi, K., Castro, V., Bates, S. V., Pieper, S., Andriole, K. P., Murphy, S. N., Gollub, R. L., and Grant, P. F. Using clinically acquired mri to construct age-specific ade atlases: Quantifying spatiotemporal ade changes from birth 6-year old. Human Brain Mapping, 3866)3305--3088, 2017.
Sotardi, S., Gollub, R. L., Bates, S. V., Weiss, R., Murphy, S. N., Grant, P. E., and Ou, Y. Voxelwise and regional brain apparent diffusion coefficient changes on mri from birth to 6 years of age. Radiology, 298(2):415, 2021.
Ouyang, I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Sun, J., Xu, C., Tang, L.,, Wang, S., Lin, C., Gong, Y., Ni, L. M.. Shum, H.-Y., and Guo, J. Think-on-graphi Deep and responsible reasoning of large language model on knowledge graph, 2024. URL https://arxiv. org/abs/2307, 7697, 2023.

---

Visual and Domain Knowledge for Professional-level Graph-of-Thouee Medical Reasoning
Tang, L.,, Sun, Z., Idnay, B., Nestor, G., Soroush, A., Elias, P. A., Xu, Z., Ding, Y., Durrett, G., Rousseau, J. F., al. Evaluating large language models medical evidence summarization. NPJ digital medicine, 6(1):158, 2023.
Team, G. Gemini 1.5:
Unlocking multimodal understanding across millions of tokens of context, 2024. URL https:
//arxiv. orgg abs/ 403 05530.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-.,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13371, 2023.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le. Q. V., Zhou, D., et al. thain-of-tthough prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
Wei, R., Wang. C., He, F., Hong, L.. Zhang. J., Bao, W., Meng. F., and Luo, B. Prediction of poor outcome af- ter hypoxic-ischemic brain injury by diffusion- imaging: A systematic review and meta-analysis. Plos One, 14(12)e00266295, 2019.
Weiss, Bates, V., Song, Y. Zhang. Y., Herzberg, E. M., Chen, Y.-C., Gong, M., Chien, I., Zhang, L.,, Mur- phy, S. N., et al. Mining multi-site clinical data to develop machine learning mri biomarkers: application to neonatal hypoxic ischemic encephalopathy. Journal of Transla- tional Medicine 17(1):1-16, 2019.
12

---

Visual and Domain Knowledge for Professional-Ivvl Graph-of-Thougg Medical Reasoning
Appendix
In this appendix, we back up our claims by supplementing the main paper with details and examples. We provide our dataset structure Section A, visual knowledge details in Section B, Clinical Graph of Thought Model (CGoT) details in Section C, model details in Section and more ablation studies Section E, respectively.
A. HIE-Reasoning Benchmark Structure
A.1. Benchmark Data Structure
MGHNICU 000 101_QA.json
Data
MGHNICU_001
MIGHNICULO01_AADCsss..mo
HIE Reasoning
MGHNICU_00X htcpr/LoO___NLiiiiirrrrttttonon
Meta Knowledge
Task 1.son, Task 2. json, Task 3. json, Task 4.json, Task 5. json, Task 6.json
(CGHNICU_001_Lesionnnnie
Individual Knowledge
MGHNICU _001
NNGHNICU_001_ROI..m
Figure Reasoning dataset structure. The Hll--Reasoning dataset is organized into three components: Data, Meta Knowledge, and Individual Knowledge.
This work was approved by Institutional Review Boards (IRBs) at Boston Children's Hospital and IRB at Massachusetts General Hospital. Figure 5 illustrates the hierarchical data structure for the HIE- AlEE-Rasonning framework, which is designed to integrate multimodal data and domain-specific reasoning for Hypoxic-Ischeme
Encephalopathy (HIE) research. The structure organized into three primary components: Data, Meta Knowledge, and Individual Knowledge. Alongside the Data components, the meta knowledge and individual knowledge this dataset provide expert knowledge complement this dataset, facilitating further research and fostering advancements in this domain.
The Data component comprises patient-level files, such MGHNICU_001, which encapsulate core data assets, includ- ing raw MRI files (e.g., MGANTCU 001 ADC ss.rit i gz for apparent diffusion coefficient maps) and task-specific reasoning files. These include our proposed Question- interpretation
pairs (e.g., MGHNICU_001.00 json) and MRI The Meta Knowledge component contains task- specific meta knowledge files (Task1. jaon through Task6. json, as shown Table 7), enabling structured rea- soning processes for all six tasks. Lastly, the Individual Knowledge component includes patient-specific files like MGHNICU_0O) Lesionesoooooob nii gz for probabilistic lesion mapping and MGHNICU_001. _OI mii.gz gz for brain anatomy labeling. This structured data organization supports comprehensive reasoning workflows, bridging raw data, domain-specific meta knowledge, and task-driven insights to enable automated analysis in HIE MRI interpretation and outcome prediction tasks.
A.2. Benchmark Distribution
Figure 6 illustrates the label distributions for three key aspects: lesion grading, MRI injury scores, and neurocognitive outcomes. Table 6 provides a detailed breakdown of six critical reasoning tasks (c.g., lesion grading, anatomical assessment, and rare location identification) the HIE- Reasonngg benchmark. This benchmark emphasizes a comprehensive evaluation of lesion severity, anatomical analysis, injury scoring, and neurocognitive outcome prediction, advancing automated reasoning for clinical HIE diagnosis and prognosis.
13

---

Visual and Domain Knowledge for Professional-leve Graph-of-Thought Medical Reasoning
Lesion Grading Distribution
MRI Injury Score Distribution
Outcome Distribution
1.2%
5.3%
(0%5,1%) the injured
6.8%
(15,55% the broin injured
9%
(5%.10%) brain injured
100% 20%) the brain injured
50%% brein injured
62.4%
Score
Score
No mamal
550%.100 100%] the brain injured
15
Figure 6. Distribution of benchmark lahels.
Table 6. The Hif--Reasoning benchmark includes six critical reasoning tasks related to neonatal HIE in MRI analysis. It consists of six question-anse answer (QA) pairs focusing on specific reasoning aspects.
Tasks
QA Pairs
QI: Lesion Grading
133
Q2: Lesion Anatomy
133
Q3: Lesion Rare Locations
133
Q5: MRI Injury Score
133
Q5: Neurocognitive Outcome Prediction
84
Q6: Interpretation Summary
133
Total MRI: 133
749
B. Visual Knowledge
B.1. Raw MRI: ADC
In the clinical flow of HIE, apparent diffusion coefficient (ADC) maps are pivotal imaging modalities, supplemented by structural MRIs, for detecting HllErrelated abnormalities (Douglas-Escobar & Weiss, 2015; Wei al., 2019; Liauw et al.. 2009). Neuroradiologists rely ADC maps identify acute brain injuries associated with HIE by locating regions with abnormally low ADC values. These low values indicate restricted water diffusion, hallmark of ischemic necrosis caused by hypoxic-icchemic insult within the first week after birth (Weiss et al., 2019). In alignment with these clinical insights, HlE-RRasoning includes patient-specific ADC maps as part of the MRI data, as illustrated in Figure 7 (A).
B.2. Derived Lesion Probability
Distinguishing abnormally low ADC values from normal regional variations remains challenging, even for experienced neuroradiologists, as normal ADC values vary across brain regions (Ou et al., 2017; Sotardi et al., 2021). ). To address this issue, (Bao et al., 2025b) introduced ZADC maps, which normalize ADC values by quantifying how many standard deviations a patient's 's ADC value a given voxel deviates from the mean normal ADC value at that anatomical location (Pinto al., 2018). These Z ADC maps, residing in the patient's raw ADC image space, enable consistent comparisons across brain regions.
As highlighted by (Bao al., 2025b), ZAdC values are effective for segmenting HlE--elated lesions, with thresholding-based segmentation achieving Dice score comparable to machine learning algorithms (Murphy et al., 2017). Applying threshold of 2, the most straightforward value for Z AADC maps, yielded the highest Dice score (0.54 + 0.28). Accordingly, we used this threshold to generate lesion prediction masks for each patient, as shown in Figure (B) (ZADC) and Figure (C) (lesion mask from thresholded ZADC, and lesions are colored by red).
14

---

Visual and Domain Knowledge for Professional-Levd Graph-of- Thought Medical Reasoning
A. Raw MRI: ADC
B. ZADC
1000
C. ZADC Computed Lesion Map
D. Brain Anatomy
Figure 7. Example of visual knowledge from patient 10. (A) Raw ADC maps from MRI, representing apparent diffusion coefficient values used to identify hypoxic-ischemic injury. (B) ZADC maps, which normalize ADC values to quantify deviations from regional means healthy neonatal ADCs, highlighting abnormalities. (C) Lesion prediction map generated by applying threshold of -2 to the ZADC map, with lesions visualized red. (D) Patient-speciffc brain anatomy map, showcasing ROIs mapped from standard brain anatomical space to individual patient space. Different colors represent different brain ROI. Together, these visual knowledge facilitate tasks in Reasoning.
B.3. Brain Anatomy
The brain is divided into 62 regions of interest (ROIs) based on standard anatomical references (Doshi et al., 2016; Morton et al., 2020), which are crucial for accurate interpretation of patient-specific MRI interpretation. Figure 7 (D) illustrates the brain ROIs mapped into patient space, generated by transforming the standard brain anatomy ROIs into the patient-specific space using the DRAMMS tool (Ou et al., 2011). 1). Figure illustrates the predefined 62 ROIs standard brain structure space used in clinical practice. Although the generated ROIs may not achieve 100% accuracy, they represent the most
15

---

Visual and Domain Knowledge for Professional-eet Shaph-of-Though Medical Reasoning
reliable brain anatomy that can be produced with maximal information and minimal effort.
GMULL
32.Artaror imb
Left muda
137.ümblc Medial Temporal fight
101 Parital Lateral GM hight
102, Semppral Supratemporal
calieur
$1.toteeiri imm left
Fruntal Labera GM
138,limbic Medial Temperal da
100, Periical Lateral GM
Supratemporal lett
Gi.r Vetrall
96,PUC
Frontal Laber Left
163,0pciptaal intertor
1.0 Prrveaal Medial GM Right
66, M.Tempprn,
Ventral
left
Foottal Medal
16000coipital
Left
1.08 Medial
7.7emport, jeft
Frontal Medaal GMl
120 Lateral Right
Parattaa
ventnilee
st.riftt cembellurs
40. hippocampu
Frontal Opercuaar GM
125 decipital Lateral Left
Paretaal lett
ventricte
3LLeft cerebetirn
13l amygtais
Fonnta Operoueor GM
decipita Medial oM Right
Tempors interior
congio
hipoccoppus
Frontal WN Right
109.0ccipital om
Temporal
cangla
inferior
montall
sight
thalamus
Left intenar
Limbic Cinglate Right de
Occipital WM
Temporal GH
#5.CSF
suleef thalamas
100,Aight
Figure 8. Brain Anatomy. Standard brain structure map is divided into 62 regions of interest (ROIs) based on predefined anatomical references. Each ROI assigned unique label and color visualization, encompassing major anatomical areas such as gray matter (GM), white matter (WM), ventrieles, and basal ganglia. These ROIs provide a detailed knowledge for region-specific analysis and lesion localization HlE-Reeaoning tasks.
C. CGoT Knowledge Details
In this section, we provide detailed explanation of how professional-level clinical knowledge is used to guide the model in correctly addressing each node task, ultimately forming the CGoT pipeline.
Table Clinical knowledge proposed in CGoT for HlE-RReasoning tasks summarized in the table. Meta Knowledge encompasses essential clinical knowledge for HIR clinical reasoning, while Individual Knowledge refers to clinical knowledge tailored to cach individual patient for each task.
Task
Meta Knowledge
Individual Knowledge
Lesion definition
Lesion Grading
Lesion detection
Lesion grading levels
Lesion detection
Lesion Anatomy
Brain ROI regions
Lesion anatomy
Lesion in Rare Location
HIE commonly affected regions
Lesion anatomy
Lesion grading
MRI Injury Score
Definition of MRI injury score
Lesion anatomy
Lesion grading
Relationship between
Neurocognitive Outcome
Lesion anatomy
MRI biomarkers and outcome
MRI injury score
Lesion grading
Lesion anatomy
MRI Interpretation
MRI interpretation template
Lesion in rare locations
MRT injury score
16

---

Visual and Domain Knowledge for Professional-evel Graph-of-Thoughe Medical Reasoning
Table 7 summarizes the various types of clinical knowledge utilized across the proposed in CGoT in Hll-RRasoning tasks. These include lesion definitions, lesion grading levels, lesion detection, brain ROI regions, and lesion anatomy (as illustrated in the previous section). In the following sections, we further illustrate the brain regions commonly affected by HIE, the definition of the MRI injury score, and the relationship between MRI biomarkers and neurocognitive outcomes.
C.I. HIE Commonly Affected Brain Regions
HIE commonly affected brain regions
Left Basal Ganglia, Right Basal Ganglia
Left Thalamus, Right thalamus Brainstem, Vermis, Corpus Callosum Frontal WM Left, Frontal WM Right Parietal WM Left, Parietal WM Right, Temporal WM Left, Temporal WM Right
Anterior Limb IC Left, Anterior Limb IC Right
PLIC Left, PLIC Right
Parietal Lateral GM Left, Parietal Lateral GM Right Parietal Medial GM Left, Parietal Medial GM Right
0.00
0.05
Figure 9. The left panel displays the lesion frequency atlas for HIE, illustrating the statistical distribution of lesions across brain regions based cohort data. Higher frequencies represented by warmer colors (yellow red), indicating most commonly affected by HllE-related brain injury. The right panel lists the high-prohability brain regions associated with HIE, including basal ganglia, thalamus, brainstem, corpus callosum, anterior limb of the internal capsule (IC), and etc. These regions were identified based clinical thresholds and expert validation.
To address Task 4, which focuses on lesions in rare locations, it is essential to first identify the regions commonly affected by HIE. According to the study (Bao et al., 2025b). the lesion atlas provided by (Bao et al., 2025b) (left panel) visualizes the distribution of HIE lesions. Based on recommendations from a neonatal radiologist, a threshold of 22.5% was applied to identify high- probabiiityy regions of HIE lesions. The resulting list of regions (right panel) was further validated by a national radiologist, confirming these areas as high-probability brain injury regions associated with HIE.
C.2. MRI Injury Score
A. Key Regions the NICHD-NRN scoring criteria
NRN MRI scoring criteria
Score
MRI Finding
ALIC (Anterior Limb of the Internal Capsule)
0
No injury detected
Basal Ganglia
1A
Minimal injury BGT, ALIC,
injury
1B
More extensive cerebral injury without BGT, APLC, PLIC, or WS injury
BGT
Thalamus
PLIC (Posterior Limb of the Internal Capsule)
2A
BGT, ALIC, PLIC, or WS injury without other cerebral Injury
2B
Any BGT, ALIC, PLIC or WS injury with other cerebral
Carebral hemisphere devastation
Figure 10. NRN MRI biomarker system. (A) Key regions in the NICHD-NRY scoring criteria, highlighting the basal ganglia, thalamus (BGT), anterior limb of the internal capsule (ALIC), and posterior limb of the internal capsule (PLIC) critical areas for assessing HlE-related brain injury. (B) The NRN MRI scoring criteria used for severity grading, ranging from 0 (no injury detected) to 3 (cerebral hemisphere devastation). Intermediate scores (1A, IB, 2A, 2B) deseribe varying levels of cerebral and subcortical injuries involving BGT, ALIC, PLIC, or watershed (WS) regions, with increasing severity correlating with adverse neurocognitive outcomes.
To address Task 5 of predicting neurocognitive outcomes, it is crucial to incorporate MRI biomarkers from clinical trials for HIE. The Neonatal Research Network (NRN) score, developed by the National Institute of Child Health and Human Development (NICHD), is the most widely used method in clinical trials to assess the severity of brain injury and predict y-ar neurocognitive outcomes in infants with HIE (Shankaran et al., 2015; 2012). This scoring system relies on neuroradiological experts evaluating the severity of brain injuries based on neonatal brain MRI. As shown in Figure 10, the
17

---

Visual and Domain Knowledge for Professional-evve Graph-of-Thoughh Medical Reasoning
NRN MRI score is based the anatomic locations and extent of HIE lesions visualized brain MRIs. The score ranges across six levels (0, la, 1b, 2a, 2b, and 3), where 0 represents the absence of HIE injury, and 3 indicates the most extensive injury within the brain. In our task, we simplify the scoring by grouping la and lb into single score of and 2a and 2b into a single score of 2, as recommended by neonatal radiologists. This adjustment accounts for the ambiguity and lack of distinct quantitative boundaries between these subcategories and relies on clinical expert judgment.
C.3. MRI Biomarkers for Neurocognitive Outcome
Predicting 2-year neurocognitive outcomes in HIE remains significant clinical challenge (Bao & Ou, 2024; Weiss et al., 2019; Bao et al., 2025b; Laptook et al., 2017a; Shankaran et al., 2017a). Medical experts continue to explore more accurate MRI biomarkers to improve outcome prediction (Shankaran et al., 2005; Edwards et al., 2010; Azzopardi et al., 2009; Laptook et al., 2017a; Shankaran et al., 2017a). Current clinical consensus indicates that the higher the MRI injury score, the more severe the brain injury, with the MRI injury score serving biomarker strongly associated with adverse 2-year neurocognitive outcomes. A score of 3 is nearly 100% predictive of adverse outcomes. Additionally, injuries to critical regions such as the vermis, cerebellum, brainstem, and hippocampus are linked to adverse neurocognitive outcomes, Patients with lower MRI injury scores or fewer lesion regions are likely to have normal neurocognitive outcomes after years. In this context, the answers provided by Task 1 (lesion grading). Task 2 (lesion anatomy), and Task 4 (MRI injury score) align closely with the consensus knowledge and deliver essential information to aid in neurocognitive outcome prediction.
C.4. Knowledge Prompts
Knowledge Example for Task 1. Lesion Grading
Meta Knowledge:
[Input Description] Suppose you are an expert in detecting Neonatal Brain Injury for Hypoxic Ischemic Encephalopathy, and you are allowed to use any necessary information on the Internet to answer questions. I will give you a set of MRI scanning slices of neonatal brains, these slices are marked with corresponding slice labels, like "Slice 10" and "Slice 11". The label means the slice depth of this slice, for example, "Slice 11" is in the middle layer between "Slice 10" and "Slice 12".
[Lesion Definition] will give you a pair of images (actually two images) marked with the same "Slice xx" label. The one with the title "Slice x" is the original ADC value of MRI scanning, while the one with the title "ZADC Slice x" is the ZADC value visualization of the gray-scale sean processed by a lesion detection algorithm, where the highly possible abnormal (lesion) region pixel is marked with red (indicating their ZADC values are less than -2). If there is red pixel, there should be lesion in this MRI. You should make comprehensive judgments based both your domain knowledge and the ZADC visualization.
[Lesion Grading Levels] The lesion percentage is defined as the area with ZADC value less than -2 divided by the area with ADC value greater than This will help you better answer the following questions. You need to judge the lesion level of the brain MRI slices by the following rules:
If the lesion region percentage < 0.01, answer with " "level1, <lesion region percentage>", If 0.01 lesion region percentage S 0.05, answer with "leve12, <lesion region percentage>", If 0.05 < lesion region percentage I 0.5, answer with "1evel3, <lesion region percentage>", If 0.5 < lesion region percentage < 1.0, answer with "1eve14, <lesion region percentage>". The output format should be like "level4, 4, 7344".
Individual Knowledge:
[Visual Input] Refer to Figure 7 (A) and (C).
Question & Answer:
[Multiple Choices] What is the severity level of the HIE lesions?
[Open Ending] What HIE lesion percentage in this MRI?
[Task 1 Answer] level 3, 0.2206.
18

---

Visual and Domain Knowledge for Professional-evel Traph-of-Though Medical Reasoning
Knowledge Example for Task 2. Lesion Anatomy
Meta Knowledge:
[Input Description] Suppose you are an expert in detecting Neonatal Brain Injury for Hypoxic Ischemic Encephalopathy, and you are allowed to use any necessary information on the Internet to answer questions. I will now provide you with a series of MRT scanning slices and some pre-processed slices as visual input. The titles these images include slice sequence labels, such "s*Sl 10" and "Slice 11". These labels indi- cate the depth of the slice; for example, "Slice 11" represents the layer between "Slice 10" and "Slice 12", These images can be grouped into sets of three based on the same slice label. For instance, the images titled "Slice 10", "ZADC Slice 10", and "ROI Slice 10" form one group, indicating that they all correspond to the same scanning depth. These three images, based on their titles, represent the following:
"Slice 10": This is the original MRI scanning ADC value at depth 10, visualized in grayscale.
"ZADC Slice 10": This is the gray-scale visualization of the ZADC values processed by a lesion detection algorithm. In this image, pixels with high probability of being abnormal (lesions) are marked in red, indi- cating that their ZADC values are less than -2. If no red pixels are present, it suggests that this MRI sean contains no lesions. It is also possible that the individual has no lesions but has a few areas marked in red. "ROI Slice 10": This represents different ROI areas of the brain appearing at this scanning depth, with each area highlighted in a different color. The color-to-ROI mapping is provided in the legend on the right side of the image. Note that the ROI regions appearing in slices of different depths are not exactly the same, as only the ROI regions present at a particular depth are displayed. However, for the same cross-slice ROI region, the color used remains consistent across slices.
[Brain ROI List] ID and Region Name Relationship:
95 corpus callosum, 62 Right Ventral DC, 61 Left Ventral DC, 71 vermis, 39 Right cerebellum,
38 Left cerebellum, 30 Right Basal Ganglia, 23 Left Basal Ganglia, 60 Right thalamus, 59 Left thalamus,
92 Anterior limb IC right, Anterior limb IC left, 94 PLIC right, PLIC left, 32 Right amygdala,
48 Right hippocampus, 31 Left amygdala, 47 Left hippocampus, 105 Right Inferior GM, 104 Left Inferior GM,
103 Right insula, 102 Left insula. 121 Frontal Lateral GM Right, 120 Frontal Lateral GM Left,
125 Frontal Medial GM Right, 124 Frontal Medial GM Left, 113 Frontal Opercular GM Right,
112 Frontal Opercular GM Left, 82 Frontal WM Right, 81 Frontal WM Left, 101 Limbic Cingulate GM Right,
100 Limbic Cingulate GM Left, 117 Limbic Medial Temporal GM Right, 116 Limbic Medial Temporal GM
Left, 161 Occipital Inferior GM Right, 160 Occipital Inferior GM Left. 129 Occipital Lateral GM Right, 128 Occipital Lateral GM Left, 109 Occipital Medial GM Right, 108 Occipital Medial GM Left,
84 Occipital WM Right, 83 Occipital WM Left, 107 Parietal Lateral GM Right, 106 Parietal Lateral GM Left,
149 Parietal Medial GM Right, 148 Parietal Medial GM Left, 86 Parietal WM right, 85 Parietal WM left,
123 Temporal Inferior GM Right, 122 Temporal Inferior GM left, 133 Temporal Lateral GM Right,
132 Temporal Lateral GM Left, 181 Temporal Supratemporal GM Right, 180 Temporal Supratemporal GM
left, 88 Temporal_wm. right, 87 Temporal.wm. leffff, 4 3rd ventricle, 11 4th ventricle, Right ventricle, 49 Left ventricle, 35 Brainstem, 46 CSF.
You need choose the names of the ROIs from the above 62 ROI list that contain lesions in this case and output them along with their IDs in the format like: 4.3rd ventricle, 123 Te.peoprrll Inferior GM Right, 84_ 84_Occipital WM Right, imb_iimbic Medial Temporal GM Left.
Individual Knowledge:
[Visual Input] Refer to Figure 7 (A), (C) and (D).
Question & Answer:
[Question] What brain regions are affected by HIE in this MRI? [Task 2 Answer] The affected regions include Right Basal Ganglia, Left thalamus, Right thalamus,
19

---

Visual and Domain Knowledge for Professional-evel Haaph-of-Thoughhe Medical Reasoning
Knowledge Example for Task 3. Lesion in rare Locations
Meta Knowledge:
[HIE Common Injured Regions] Refer to Figure 9.
Individual Knowledge:
[Task 2 AnswerJ Your previous answer for the lesion anatomy task is Right Basal Ganglia, Left thalamus, Right thalamus,
Question & Answer:
[Open Question] Are there regions that are uncommonly injured in HIE? (Task 3 Answer] Limbic Cingulate GM Left, Limbic Cingulate GM Right.... are not typical for common HIE injury regions.
Knowledge Example for Task 4. MRI Injury Score
Meta Knowledge:
[Input Description] Refer to [Input Description] in knowledge for Task 2.
[Lesion Definition] Refer to [Lesion Definition] in knowledge for Task
[Lesion Grading Levels] Refer to [Lesion Grading Levels] in knowledge for Task 1.
[Brain ROI List] Refer [Brain ROI List] in knowledge for Task
[Definition of MRI Injury Score] We have introduced new diagnostic metric called MRI Injury Score. This metric consists of four levels: Score 0, Score 1, Score 2, and Score 3. Each score level determined by the injury regions within the ROIs in a given case and the severity of the injury in certain regions.
Score 0: Defined as no injury detected in this case.
Score 1: Defined either the following a) b) situation occurs:
a). Minimal cerebral injury without BGT region, ALIC region PLIC region, or detected
WS (watershed) injury.
b). More extensive cerebral injury without BGT region, ALIC region PLIC region, detected WS (watershed) injury.
NOTE: BGT region (including left BGT and right- BGT), ALIC region (including
left_LIC and right.ALIC), PLIC region (including left_PLCC and rLgC)
Score 2: Defined either the following a) b) situation occurs:
a). Any BGT region, ALIC region, PLIC region, WS injury detected without other cerebral injury.
b). Any BGT region, ALIC region, PLIC region, WS injury detected with other cerebral injury.
NOTE: BGT region (including left BGT and BGT), ALIC region (including left ALIC and ALgC,, PLIC region (including left PLIC and rLCC..
Score 3: Defined as cerebral hemisphere devastation.
Individual Knowledge:
[Visual Input] Refer to Figure 7 (A), (C) and (D).
[Task 1 Answer] Your previous answer for the lesion grading task is level 0.2206.
[Task 2 Answer] Your previous answer for the lesion anatomy task is Right Basal Ganglia, Left thalamus, Right thalamus,
Question & Answer:
[Question] What is the MRI injury score of this MRI? [Task 4 Answer] Score 2.
20

---

Visual and Domain Knowledge for Professional-evvel Graph-of-Thought Medical Reasoning
Knowledge Example for Task 5. Neurocognitive Outcome
Meta Knowledge:
[Input Description] Refer to [Input Description] knowledge for Task 2.
[Lesion Definition] Refer to [Lesion Definition] in knowledge for Task 1.
[Lesion Grading Levels] Refer to [Lesion Grading Levels] in knowledge for Task 1.
[Brain ROI List] Refer [Brain ROI List] in knowledge for Task
(Definition of MRI Injury Score] Refer to [Definition of MRI Injury Score] in knowledge for Task (Relationship between MRI Injury Biomarkers and Outcome] The higher the MRI injury score, the more severe the brain injury. The MRI injury score is biomarker strongly associated with adverse HIE 2-year neurocognitive outcomes. If the score is 3, then towards 100% adverse outcome. Injuries to the vermis, cere- bellum, brainstem, and hippocampus are also related to adverse neurocognitive outcomes. If an individual is a current patient, it doesn't t necessarily mean he/she will still be patient in years. Patients with lower MRI injury scores, or fewer lesion regions are likely to become normal in 2 years.
Individual Knowledge:
[Visual Input] Refer Figure (A), (C) and (D).
[Task 1 Answer] Your previous answer for the lesion grading task is level 0.2206.
(Task 2 Answer] Your previous answer for the lesion anatomy task is Right Basal Ganglia, Left thalamus, Right thalamus,
(Task 4 Answer] Your previous answer for the MRI injury score task is Score 2.
Question & Answer:
[Question] What is the predicted neurocognitive outcome for this patient at 2 years?
(Task 5 Answer] (Adverse).
Knowledge Example for Task 6. MRI Interpretation
Meta Knowledge:
[MRI Interpretation Template]
The MRI shows <Task Answer> of the brain volume injured. The affected regions include <Task Answer>. For uncommon areas, <Task Answer> are not typical for common HIE injury regions.
The MRI injury score for this MRI is <Task 4 Answer>,
Individual Knowledge:
[Task 1 Answer] 22.06%
[Task 2 Answer] Right Basal Ganglia, Left thalamus, Right thalamus,
[Task 3 Answer] Limbic Cingulate GM Left, Limbic Cingulate GM Right,
[Task 4 Answer] Score 2
Caption:
The MRI shows 22.06% of the brain volume injured.
The affected regions include Right Basal Ganglia, Left thalamus, Right thalamus, For uncommon areas, Limbic Cingulate GM Left, Limbic Cingulate GM Right, are not typical for common HIE injury regions.
The MRI injury score for this MRI is Score 2.
21

---

Visual and Domain Knowledge for Professional-evel Graph-of-Thouge Medical Reasoning
D. Model Details
In this section, we provide additional details about the baselines used in the experimental section. The baselines are directly accessed using public APIs or implemented using publicly available source code.
D.I. Gereral-Purpose LVLMS
As detailed in Section C, for the three general-purpose LVLMs, we arranged selected MRI slices into a sequence as visual input, paired with specifically designed prompts, and integrated input from previous steps where applicable form the input context for the proposed CGoT pipeline. The Gemini-1.5-Flaa (Team, 2024) responded effectively to our prompts, whereas GPT4o-Mini (OpenAI, 2024) and GPT4o (OpenAI, 2024) declined to answer certain questions with high uncertainty. GPT4o faild to respond to all five tasks introduced in Figure 2 of the main paper when directly feeding the MRI images and task descriptions. Consequently, we used the quantitative results from GPT4o-Mini to represent the performance of GPT4o-Series models on these tasks. Similarly, after adapting the GPT4o model into CGoT-GPT40, it still refused to answer lesion grading, lesion anatomy, and Neurocognitive Outcome tasks based on the answers of previous tasks generated by CGoT-GPT4o-Miin Thus, we reported CGoT-GPT4o-Ming evaluation results for the first three tasks and CGoT-GPT4o's performance for the remaining tasks. This fusion approach provided comprehensive and effective evaluation of the capabilities of the (GoT-GPP4o-S3 models.
tasks, but it successfully addressed the MRI injury score and
D.2. Medical-Purpose LVLMs
For the three medical Flamingo (Moor et al., 2023) wwe selected implementations directly from the respective papers where these models were implemented or introduced. To ensure a fair comparison, we used the 7B parameter versions of each model, as this configuration provides a balanced trade-off between model performance and computational resource requirements. In line with their instruction- instruction-fnetuning approach, we combined all selected MRI slices into a single image, annotated with titles indicating their scanning depth, and paired them with task briefings and evaluation questions. However, as demonstrated in the results, these models performed only at or even below the random-guee gugosssguess level, with Med- Med-Flamingo even failing to generate valid outputs for certain tasks.
MinGGPT-MMd (Alkhaldi et al., 2024)², LLava-Med (Li al., 2024)², and Med-
Since the models are primarily trained on 2D general- -purpose or medical images, while brain MRI scanning data inherently represents 3D structural format where slices at different depths are interrelated, we propose to unroll 3D volumes into sequential 2D slices in order, serving as the visual input for the LVLMs. However, for the three mentioned medical LVLMs, since their fine-tuning or instruction instruction-tuning data mainly consists of single image-text pairs, they fail to provide satisfactory zero-shot responses when queried about 2D-slice sequence input. Therefore, we arrange the series of slices sequentially as subplots within unified image and annotate subplots with corresponding scanning depth information ensure input compatibility with these models.
E. More A Ablation Studies
Table 8. Ablation study on three types of visual inputs.
Lesion Grading
Lesion Anatomy
Lesion in Rare Locations
MRI Injury
Neurocognitive Outcome Inter Class Acc (t)
Interpretation
Score
Summary ROUGE-L. (?)
Raw ADC
ZADC Brain Anatomy
Acc (t)
MAE ())
Score (t)
Fl Score (t)
Acc (f)
62.41%
0.0703
43.57%
41.47%
49.62%
71.73%
53.68%
58.64%
0.0849
42.78%
41.22%
47.37%
68.11%
51.07%
46.62%
0.1152
26.96%
25.10%
30.83%
60.44%
34.55%
62.41 %
0.0703
39.90%
37.98%
39.85%
64.05%
47.97%
'Code available at: http://hhhabbbcooo//ggg///////ld Matt:o/ooWWLobMMiiiMooodalaattet
"Code available at: http://gihaacccommmmiccco/////at
3 *Code available at: htpp://gthhhcccommmmppaaaooooeed
22

---

Visual and Domain Knowledge for Professional-leve Graph-of-Thought Medical Reasoning
Table 9. Ablation study different thresholds of ZADC
Lesion
Lesion
Lesion in Rare Locations
MRI Injury
Neurocognitive
Interpretation
ZADO
Grading
Anatomy
Score
Outcome
Summary
Acc (f)
MAE (4)
Fl Score (1)
FI Score (1)
Acc (t)
Inter Class Acc (1)
ROUGE-L (
ZADC < ~1.8
58.64%
0.0782
43.04%
41.42%
47. 14.14%
68. 68.23%
51.42%
ZADC < 222
60.90%
0.0867
42.58%
38.19%
37.59%
70.93%
50.67%
ZADO < -2.
62.41%
0.0703
43.57%
41.47%
49.62%
71.73%
53.68%
E.1. Effectiveness of visual knowledge
Each visual knowledge type (ADC, ZADC, brain anatomy) has complementary roles. Raw ADC provides signal details, ZADO identifies abnormal regions, and brain anatomy supply anatomical priors. Removing any component degrades performance, but removing ADD has the crucial effect, as provides the probability of abnormal brain regions-cruciall for MRI injury interpretation.
E.2. Robustness to varying ADD threshold values
Justification of And thresholding on -2: The threshold of -2 aligns with clinical understanding and prior studies for HIE abnormal regions probabilities (Weiss et al., 2019; Bao et al., 2025b), indicating ADC values below two standard deviations from the normal atlas-often interpreted as abnormally reduced diffusion in neonatal HIE ADC maps-and serves as a marker for potential brain injury regions.
Across ZadC threshold variations, CGo'T outperforms baselines, demonstrating robust and effective performance. The drop at ADD < 2.2 in MRI injury stems from overly strict threshold that misses mild injuries. NRN includes mild cases (0 and 1), which are often excluded by this threshold, leading to missed low-gren low-grade injuries. ZADC < -2 better captures these signals, yielding optimal performance.
23


Reviews:
[{"review_id": "om3D0Y3CSV", "reviewer": "Reviewer_fHCe", "summary": "The paper introduces HIE-Reasoning, a professional-level medical visual question answering (MVQA) benchmark focused on neonatal Hypoxic-Ischemic Encephalopathy (HIE). The authors propose the Clinical Graph-of-Thought Model (CGoT), which integrates visual and clinical domain knowledge into a structured reasoning framework, significantly enhancing the interpretability and accuracy of clinical decisions and prognoses.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments. We address your concern below within 5000 characters.\n### Q1. Computational Complexity\n\n| Model | Input Tokens | Output Tokens | Inference Time | API Cost |\n|-|-|-|-|-|\n| Gemini|10,580| 446| 6.50s| $0.00080  |\n| CGoT-Gemini| 36,256 | 1,146 | 21.09s | $0.00272|\n| MedAgent-Gemini| 354,801| 10,497| 210.75s| $0.02976|\n| MdAgent-Gemini| 236,193| 9,228| 142.75s| $0.02048|\n\n*Table 1-1: Case-wise computational analysis on the Gemini-1.5-flash backbone.*\n\nCGoT introduces modest overhead compared to Gemini-1.5-Flash but achieves 15% accuracy improvement with clinical transparency. CGoT significantly reduces computational cost compared to SOTA agents (MedAgent, MdAgent) through structured reasoning that mirrors clinical workflows. We will add this new analysis in the later version.\n\n### Q2. Dependency on Prerequisite Tasks\n\nPrerequisite tasks before final decisions align with standard clinical practice. Our approach enhances both transparency and effectiveness.\n\n1. **Intermediate steps are necessary and model is robust to minor errors of these intermediate steps:**\n\n   **Intermediate step is necessary**:\n\n    *Table 2-1: Removing MRI injury scoring from the reasoning chain leads to a sharp performance drop in outcome prediction.* \n    | MRI Injury Score | 2-year Outcome |\n    |------------------|---------------|\n    | ✘  | 50.94 |\n    | ✔  | 71.73 |\n\n    **Robust to small errors of intermediate steps**: We applied random ±1-level perturbations (across 4 severity levels)  to MRI injury score in varying percentages of cases (0–30%). Results show graceful degradation, indicating robustness to small inaccuracies.\n\n    *Table 2-2. CGoT is robust to minor errors of intermediate steps.*\n    | Perturbation Ratio (%) | 2-year Outcome (%) |\n    |------|-----|\n    | 0   | 71.73 |\n    | 10 | 67.83 |\n    | 20 | 66.22 |\n    | 30 | 62.72 |\n\n2. **Clinically Grounded:** CGoT mimic clinical diagnostic workflows by modeling the stepwise reasoning clinicians use. Its intermediate tasks are clinically meaningful and enhance interpretability. Step-by-step inference in chain-of-thought reasoning [A] and inference scaling [B] has been proved to enhance interpretability and improve performance on other complex reasoning tasks. Table 4 (main paper) shows incorporating such intermediate tasks improves the final outcome prediction compared to relying solely on end-to-end black-box models (Table 2, main paper).\n\n### Q3. Comparisons with MDAgent and MedAgent\n\n| Model    | Lesion Grading (%)| Lesion Anatomy (%) | Rare Lesion Locations (%) | MRI Injury Score (%) | Neurocognitive Outcome (%) | Interpretation Summary (%) |\n|---|---|---|----|-----|------|-----|\n| MDAgent| 28.57, 0.1508| 42.61 | 38.29 |47.85 | 48.81| 51.22|\n| MedAgent| 30.95, 0.1674| 41.88| 37.57| 45.80| 54.32| 49.17|\n| **CGoT** | **62.41, 0.0703** | **43.57** | **41.47** | **49.62**|**71.73**| **53.68** |\n\n*Table 3-1: Performance comparisons.*\n\nCGoT outperforms both baselines by incorporating structured clinical reasoning with domain-specific visual and textual knowledge, enabling superior performance on complex clinical tasks.\n\n### Q4. Question Types\n\n|Total QA Pairs|Open-Ended|Multiple Choice |\n|--|---|---|\n| 749 | 399| 350|\n\n*Table 4-1: Question distribution.*\n\n- **Open-ended:** Lesion percentage, lesion anatomy, rare lesion localization, MRI interpretation summaries.\n- **Multiple-choice:** Lesion severity ratings, MRI injury scoring, outcome classification.\n\n### Q5. Evaluation Metric\n\n(1) ROUGE-L, capturing content overlap and fluency by comparing generated answers to expert references, commonly in medical summarization tasks with LLM [C]. (2) F1 score for questions related to brain regions to assess correctness and completeness of region injured (Sec.5).\n\n### Q6. Identify Clinical Knowledge Relevance\n\nClinical experts (radiologists, neonatologists, and neurologists) curated, identified, and validated all relevant clinical knowledge during dataset construction to align with real-world clinical reasoning.\n\n### Q7. Analysis of Components in Table 3 (main paper):\nRetaining only GoT or only clinical knowledge performs worse than removing both due to misaligned or incomplete reasoning. GoT without clinical context may propagate irrelevant patterns, while clinical knowledge without GoT ignores task dependencies. When both are removed, the model defaults to a purely end-to-end approach, which avoids the confusion caused by partially constrained reasoning. This highlights the importance of combining knowledge and clinical reasoning.\n\n### Q8. Paper Revision\nWe will carefully review and revise the paper.\n\n[A]. W., J., et al. \"Chain-of-thought prompting elicits reasoning in large language models.\" NeurIPS 2022.\n\n[B]. G., D., et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv 2025.\n\n[C]. T., L., et al. \"Evaluating large language models on medical evidence summarization.\" NPJ digital medicine 6.1 (2023): 158.", "claims_and_evidence": "See Strengths And Weaknesses part", "methods_and_evaluation_criteria": "See Strengths And Weaknesses part", "theoretical_claims": "See Strengths And Weaknesses part", "experimental_designs_or_analyses": "See Strengths And Weaknesses part", "relation_to_broader_scientific_literature": "See Strengths And Weaknesses part", "essential_references_not_discussed": "See Strengths And Weaknesses part", "other_strengths_and_weaknesses": "**Strengths**\n\n1. This paper provides a new HIE-Reasoning benchmark tailored for professional-level medical reasoning, which consists of six MRI-related tasks. \n2. This paper introduces a CGoT framework, which integrates clinical knowledge and clinical reasoning steps with LVLMs, enhancing both model performance and transparency.\n3. The paper is well-structured and easy to follow.\n\n**Weaknesses**\n\n1. This paper lacks an analysis of the computational complexity, efficiency, and resource requirements of the proposed CGoT compared to SOTA baselines, including the inference time, memory usage, and API costs. This would be helpful in assessing its scalability and practical applicability.  \n2. The performance of CGoT on more complex tasks (e.g., predicting 2-year neurocognitive outcomes) is heavily dependent on the results of previous tasks (e.g., MRI injury scores) (shown in Table 4). In other words, solving these more complex tasks requires first addressing related prerequisite tasks, which reduces efficiency and limits practical applicability.\n3. The paper lacks comparisons with important baselines. The proposed CGoT incorporates clinical knowledge into the LVLM, guiding it to addressing more complex medical questions via a clinician-like diagnostic process. However, existing methods such as MDAgent [1] and MedAgents [2] also emulate real-world medical decision-making processes for multimodal medical reasoning tasks using LLMs and LVLMs. These should be included as strong baselines to better validate the effectiveness of CGoT.\n4. The proposed HIE-Reasoning benchmark requires a more detailed description, particularly regarding the distribution of open-ended and multiple-choice questions.\n\n**Reference**\n\n[1] MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making, NeurIPS, 2024.\n\n[2] MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning, ACL, 2024.", "other_comments_or_suggestions": "1. Several typos are present in the paper. For example, in Line 048 and Line 090, \"Fig. 1\" and  \"Fig. 2\" should be revised to \"Figure 1\" and Figure 2\". In Line 124, \"proposed\" should be revised to \"propose\". In Line 255, \"Sec\" should be revised to \"Section\". In Line 207, \"MRI Interpretation Summary.\" should be revised to \"Task 6. MRI Interpretation Summary.\".\n\n2. \"MRI\" needs to be introduced with its full name at the beginning of the paper.", "questions_for_authors": "1. How to evaluate the model performance on open-ended questions in the HIE-Reasoning benchmark?\n2. How to identify the clinical knowledge (i.e., visual and textual knowledge) most relevant to each specific question? Was this knowledge preprocessed at the beginning?\n3. In Table 3, why does retaining only GoT or clinical knowledge lead to worse performance compared to removing both simultaneously?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cm476UdbEL", "reviewer": "Reviewer_H6vg", "summary": "The authors introduce the HIE-Reasoning dataset and a Clinical Graph of Thought (CGoT) model for professional-level Medical Visual Question Answering (MVQA) focused on Hypoxic-Ischemic Encephalopathy (HIE). The dataset, built from a decade of MRI and clinical data, includes 749 expert-annotated question-answer pairs and aims to simulate complex clinical reasoning. CGoT integrates visual and textual clinical knowledge into LVLMs, outperforming baselines by ~15% on neurocognitive outcome prediction. Evaluations reveal limitations in existing LVLMs for such tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer's valuable suggestions and address concerns below within 5000 characters.\n\n### Q1. Fig. 1 lacks examples of \\\"clinically irrelevant\\\" questions.\n\n\\\"Clinically irrelevant questions\\\" include general or superficial queries about images, such as modality (\\\"What type of scan is this?\\\") or organ identification (\\\"What organ is shown?\\\"). Although valid in general VQA, **these questions lack direct clinical relevance or diagnostic utility**.\n\n### Q2. How does CGoT avoid hallucination pitfalls like Med-Flemingo?\n\n- **Structured Reasoning Pathway:** CGoT decomposes tasks into clinically meaningful subtasks aligned with expert workflows, **increasing interpretability and traceability**.\n- **Clinical Knowledge Grounding:** Reasoning steps explicitly use specialist-curated clinical knowledge (e.g., ADC maps, injury scores), **anchoring predictions in verifiable evidence**.\n- **Modular Task and Output Verification:** Each subtask undergoes ground-truth validation, reducing hallucinations via fine-grained feedback. Intermediate outputs (injury scores, consistency signals) allowing both users and the model itself to validate predictions and detect inconsistencies early in the pipeline.\n\n### Q3. Ablation studies on visual knowledge components (ADC vs. ZADC).\n\n**Table 3-1. Ablation study on visual knowledge components**\n| Raw ADC | ZADC | Brain Anatomy | Lesion Grading | Lesion Anatomy | Rare Locations | MRI Injury Score | Neuro Outcome | Interpretation Summary |\n|-|-|-|-|-|-|-|-|-|\n|✔️|✔️|✔️| 62.41%, 0.0703 | 43.57%| 41.47%|49.62%| 71.73%| 53.68%|\n|| ✔️| ✔️| 58.64%, 0.0849 | 42.78%| 41.22%| 47.37%| 68.11%| 51.07%|\n|✔️|| ✔️| 46.62%, 0.1152 | 26.96%| 25.10%| 30.83%| 60.44%| 34.55%|\n|✔️| ✔️| | 62.41%, 0.0703 | 39.90%| 37.98% | 39.85%| 64.05%| 47.97%|\n\nEach visual knowledge type (ADC, ZADC, brain anatomy) has complementary roles. Raw ADC provides signal details, ZADC identifies abnormal regions, and brain anatomy supply anatomical priors. Removing any component degrades performance, but removing ZADC has the crucial effect, as it provides the probability of abnormal brain regions—crucial for MRI injury interpretation.\n\n### Q4. Justification and sensitivity of ZADC (–2).\n\n- **Justification (z < –2):** The threshold of –2 aligns with clinical conventions and prior studies for HIE abnormal regions probabilities [A], indicating ADC values below two standard deviations from the normal atlas—often interpreted as abnormally reduced diffusion in neonatal HIE ADC maps—and serves as a marker for potential brain injury regions.\n\n**Table 3-2. Ablation study on varying ZADC threshold values**\n| Threshold | Lesion Grading | Lesion Anatomy | Rare Locations | MRI Injury Score | Neuro Outcome | Interpretation Summary |\n|--|--|--|--|--|--|--|\n| z<–1.8| 58.64%, 0.0782|43.04%| 41.42%| 47.14%| 68.23%| 51.42%|\n| z<–2.2| 60.90%, 0.0867|42.58%| 38.19%| 37.59%| 70.93%| 50.67%|\n| z<–2| 62.41%, 0.0703|43.57%| 41.47%| 49.62%| 71.73%| 53.68%|\n\nAcross ZADC threshold variations, CGoT outperforms baselines, demonstrating robust and effective performance. The drop at z < −2.2 in MRI injury stems from an overly strict threshold that misses mild injuries. NRN includes mild cases (0 and 1), which are often excluded by this threshold, leading to missed low-grade injuries. z < −2 better captures these signals, yielding optimal performance.\n\n### Q5. Alternative reasoning structures.\n\n**Clinically Grounded.** CGoT is predefined and grounded in well-established diagnostic and prognostic workflows used in neonatal care. Its structure captures a real clinical reasoning pipeline [A,B], where each edge and reasoning step reflects realworld clinical logic. Modifying or removing these edges would compromise both interpretability and clinical validity.\n\n**Quantitative Analysis.** Table 4 (main paper) demonstrates omitting intermediate tasks significantly reduces neurocognitive outcome prediction accuracy, confirming all subtasks are essential. Alternative structures (e.g., bypassing injury scoring or directly predicting outcomes from lesions) resulted in lower accuracy and clinical implausibility (Table 4). Intermediate reasoning nodes and current reasoning structures are necessary; the model is robust and tolerant to minor errors in these steps, as shown in response to Reviewer #fHCe (Tables 2-1 and 2-2). Future explorations of adaptive structures must maintain clinical interpretability. Ablation studies on node features (Tables 3-1, 3-2) confirm that all types of knowledge inputs contribute meaningfully to performance, reinforcing the thoughtful and necessary design of CGoT.\n\n[A] Mining multi-site clinical data to develop machine learning MRI biomarkers: application to neonatal hypoxic ischemic encephalopathy. 2019.\n\n[B] NICHD Magnetic Resonance Brain Imaging Score in Term Infants With HIE: A Secondary Analysis of a Randomized Clinical Trial.\n### Q6. Add related evaluation work [1,2].\nWe will add and discuss the new related work in the later version.", "claims_and_evidence": "- The HIE-Reasoning dataset is a pioneering effort, shifting MVQA from basic perception to clinically relevant reasoning, with tasks like neurocognitive outcome prediction.", "methods_and_evaluation_criteria": "- CGoT’s graph-of-thought approach, mimicking clinical workflows, creatively decomposes complex tasks into manageable steps, enhancing interpretability and performance.  \n- Fig. 1 contrasts existing MVQA with HIE-Reasoning but lacks examples of “clinically irrelevant” questions for clarity.  \n- Sec. 5.2: Med-Flamingo’s failure is noted, but no discussion of why or how CGoT avoids similar pitfalls (e.g., hallucination).", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- The dataset’s decade-long curation and expert validation ensure high clinical fidelity, while CGoT’s significant gains (e.g., 71.73% vs. 56.60% on outcome prediction) validate its efficacy.\n- Ablation studies (Tables 3, 4) lack depth—e.g., no analysis of visual knowledge components (ADC vs. $Z_{ADC}$) or alternative reasoning structures beyond task omission. How robust is CGoT to variations in graph design? \n- Sec. 4.2.1: How are $Z_{ADC}$ thresholds (-2) justified beyond citing Bao et al. (2023)? Sensitivity to this choice is unexplored.", "relation_to_broader_scientific_literature": "- Good to include recent work, but miss some related evaluation work [1,2].\n\n[1] Yan Q, He X, Yue X, et al. Worse than random? An embarrassingly simple probing evaluation of large multimodal models in medical VQA[J]. arXiv preprint arXiv:2405.20421, 2024.\n\n[2] Xia P, Chen Z, Tian J, et al. Cares: A comprehensive benchmark of trustworthiness in medical vision language models[J]. Advances in Neural Information Processing Systems, 2024, 37: 140334-140365.", "essential_references_not_discussed": "As said in Relation To Broader Scientific Literature", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KZGt0SrMjt", "reviewer": "Reviewer_Hbac", "summary": "The paper introduces a novel dataset specifically designed for professional-level medical reasoning in medical visual question answering. It leverages a decade-long collection of MRI and clinical data related to Hypoxic-Ischemic Encephalopathy (HIE), enriched with expert annotations and insights. The authors generate clinical question-answer pairs and MRI interpretations to facilitate comprehensive diagnosis and neurocognitive outcome prediction. Furthermore, the paper proposes the innovative Clinical Graph of Thoughts (CGoT) model that integrates domain-specific medical knowledge with LVLMs. The reported results, including a 15% absolute gain on key neurocognitive outcome tasks, underscore the dataset’s potential and the model’s promising performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer's constructive feedback; we have addressed your concerns below.\n\n>Q1. One potential concern is that the paper mentions the dataset is sourced from Massachusetts General Hospital, which might conflict with the anonymity requirements of a double-blind review process. This detail could inadvertently reveal the institution behind the dataset, thereby compromising the anonymity of the submission. Aside from this, the paper is strong in its methodological design and clinical relevance. The combination of long-term data collection, expert annotations, and integrated clinical reasoning within the model is highly commendable. Addressing the anonymity issue explicitly would help mitigate any ethical or procedural concerns.\n\nThanks for recognizing the value of our dataset and method. We further address the anonymity issue here.\nWhile the dataset used in this study originates from Massachusetts General Hospital (MGH) and was collected and shared under appropriate IRB approvals, **this does not imply that the research was conducted by or on behalf of MGH, but just provide that we can collect the HIE specific data from one of the best hospitals in the world, with the most professional clinical knowledge of HIE on the world.** The use of the MGH dataset was independent of any institutional affiliation and does not reveal the authors' identities.\n\nMoreover, before submission, we have carefully reviewed the manuscript to ensure that any potentially identifying references to the authors' institutions have been removed or anonymized to comply fully with double-blind review requirements. We will restore the appropriate acknowledgments and dataset provenance upon acceptance, as per standard publishing practice. \n\n>Q2. Figure 3 could be improved as the current design allows for some overlap between images and text, which hampers clarity. A clearer layout that avoids any overlap will improve both readability and the overall presentation of the results. The authors might consider revising the figure with better spacing or annotations to ensure that all visual elements are distinct and easy to interpret. Enhancing the visual quality here would further support the clarity of the paper's contribution. Overall, this is a minor presentation issue that, once fixed, will polish an otherwise well-structured paper.\n\nWe thank the reviewer for the thoughtful suggestion regarding Figure 3. We agree that improving the layout and removing the overlap between images and text will enhance the clarity and readability of the figure. In the revised version, we will update Figure 3 to ensure that all visual elements are clearly separated, with improved spacing and annotations. We appreciate the reviewer’s feedback on this presentation.\n\n\n>Q3. The paper mentions that domain prompts are defined by doctors with varying years of experience, specifically distinguishing between low-experience and high-experience physicians. Could you elaborate on whether this difference in clinical expertise leads to significant performance variations or impacts the model’s outputs? How have you ensured that the variation in clinical experience among the doctors does not bias the dataset or the evaluation of the Clinical Graph of Thoughts (CGoT) model?\n\nThe initial annotations were performed by a junior yet experienced fellow and subsequently reviewed through consensus checks with senior experts. This process resulted in a single, unified annotation set. By consolidating annotations through expert agreement and consensus, we aimed to minimize inter-reader variability and reduce potential bias introduced by individual experience levels in both the dataset and the evaluation of the CGoT model. We will include more detailes of the annotation protocol in the revised manuscript to enhance transparency.", "claims_and_evidence": "The results presented in the paper convincingly support the conclusions drawn by the authors. The experimental findings are robust and clearly demonstrate that the proposed approach leads to significant improvements over baseline methods. Detailed evaluations on the dataset illustrate that the clinical reasoning and predictions are well-grounded in the data. The evidence provided aligns with the claims, ensuring that the novel contributions are both meaningful and reproducible. Overall, the alignment between claims and supporting experimental outcomes is strong and well-articulated.", "methods_and_evaluation_criteria": "The methods and evaluation criteria used in this paper are well-suited for the problem of clinical diagnostic reasoning. The paper describes a thorough experimental setup and uses appropriate metrics to assess performance, ensuring that the impact of the proposed methods can be reliably measured. The benchmark dataset, constructed from real clinical data, provides a realistic and challenging testbed for evaluating diagnosis and prognosis tasks. The inclusion of both visual and clinical data in the evaluation strengthens the overall design. In summary, the approach is methodologically sound and the evaluation criteria are fitting for the intended application.", "theoretical_claims": "The paper does not involve formal theoretical proofs or derivations, which is appropriate given its focus on clinical application and experimental demonstration.", "experimental_designs_or_analyses": "The experimental design is well-constructed and thoroughly detailed, providing confidence in the validity of the analyses performed. The paper clearly explains the procedure for data collection, annotation, and the subsequent generation of clinical question-answer pairs. The evaluation is comprehensive, with proper consideration given to both visual interpretation and neurocognitive outcome prediction tasks. Each experiment is designed to test specific aspects of the proposed Clinical Graph of Thoughts (CGoT) model, and the results are transparently presented. Overall, the experimental design and analysis section is robust and leaves little room for ambiguity regarding the scientific claims.", "relation_to_broader_scientific_literature": "The paper makes a significant contribution by providing a benchmark that is closely aligned with real-world clinical applications. By incorporating a dataset sourced from relevant clinical settings and focusing on professional-level medical reasoning, the work fills an important gap in the literature on medical visual question answering. It relates well to prior studies in clinical diagnosis while also expanding the scope by integrating both imaging and clinical data for neurocognitive outcome prediction. The proposed CGoT model builds upon existing concepts in LVLMs and adapts them with domain-specific insights, further positioning the work within the broader scientific discourse. Overall, the paper successfully bridges the gap between academic research and clinical practice.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "One potential concern is that the paper mentions the dataset is sourced from Massachusetts General Hospital, which might conflict with the anonymity requirements of a double-blind review process. This detail could inadvertently reveal the institution behind the dataset, thereby compromising the anonymity of the submission. Aside from this, the paper is strong in its methodological design and clinical relevance. The combination of long-term data collection, expert annotations, and integrated clinical reasoning within the model is highly commendable. Addressing the anonymity issue explicitly would help mitigate any ethical or procedural concerns.", "other_comments_or_suggestions": "Figure 3 could be improved as the current design allows for some overlap between images and text, which hampers clarity. A clearer layout that avoids any overlap will improve both readability and the overall presentation of the results. The authors might consider revising the figure with better spacing or annotations to ensure that all visual elements are distinct and easy to interpret. Enhancing the visual quality here would further support the clarity of the paper's contribution. Overall, this is a minor presentation issue that, once fixed, will polish an otherwise well-structured paper.", "questions_for_authors": "The paper mentions that domain prompts are defined by doctors with varying years of experience, specifically distinguishing between low-experience and high-experience physicians. Could you elaborate on whether this difference in clinical expertise leads to significant performance variations or impacts the model’s outputs?\nHow have you ensured that the variation in clinical experience among the doctors does not bias the dataset or the evaluation of the Clinical Graph of Thoughts (CGoT) model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents HIE-Reasoning, a professional-level benchmark for Medical Visual Question Answering (MVQA) focused on neonatal Hypoxic-Ischemic Encephalopathy (HIE). The authors contribute a highly valuable and unique dataset, curated over a decade and consisting of 749 expert-annotated question–answer pairs that simulate complex clinical reasoning tasks. The benchmark provides a realistic and challenging setting for evaluating Large Vision-Language Models (LVLMs) in clinical decision-making.

To address this challenge, the authors propose CGoT, a reasoning model that integrates clinical workflows and structured visual–textual knowledge. CGoT achieves ~15% improvement in neurocognitive outcome prediction over existing baselines, highlighting its practical effectiveness.

All reviewers commend the significant effort in building this pioneering dataset, the thoughtful design of CGoT to mirror actual medical practice, and the well-constructed evaluation protocol. The paper is a notable example of bridging the gap between academic AI research and clinical application, offering both a realistic benchmark and a working system that addresses real-world diagnostic needs.

Overall, this work is strong in contribution, clearly written, and would make a valuable addition to ICML.
Decision: Accept (spotlight poster)