AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation
Authors: Qingqiu Li
Zihang Cui
Seongsu Bae
Jilan Xu
Runtian Yuan
Yuejie Zhang
Rui Feng
Quanli Shen
Xiaobo Zhang
Shang Gao
Junjun He
Shujun Wang
Venue: NeurIPS 2025 poster ()
Keywords: Medical Large Multimodal Models; Chain of Thought; Anatomical Ontology; Chest X-rays; Medical Visual Question Answering
Primary area: machine_learning_for_sciences

Abstract: Chest X-rays (CXRs) are the most frequently performed imaging examinations in clinical settings. Recent advancements in Medical Large Multimodal Models (MLMMs) have enabled automated CXR interpretation, improving diagnostic accuracy and efficiency. However, despite their strong visual understanding, current MLMMs still face two major challenges: (1) insufficient region-level understanding and interaction, and (2) limited accuracy and interpretability due to single-step prediction. In this paper, we address these challenges by empowering MLMMs with anatomy-centric reasoning capabilities to enhance their interactivity and explainability. Specifically, we propose an Anatomical Ontology-Guided Reasoning (AOR) framework that accommodates both textual and optional visual prompts, centered on region-level information to enable multimodal multi-step reasoning. We also develop AOR-Instruction, a large instruction dataset for MLMMs training, under the guidance of expert physicians. Our experiments demonstrate AOR's superior performance in both Visual Question Answering (VQA) and report generation tasks. Code and data are available at: https://github.com/Liqq1/AOR.

AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation
Qingqiu Li Zihang Cui², Seongsu Bae Jilan Xu', Runtian Yuan', Yuejie Zhang 1.6
Rui Feng 1.5 Quanli Shen 4 Xiaobo Zhang Shang Gao Junjun He Shujun Wang" 7,8
College of Computer Science and Artificial Intelligence,
Shanghai Key Laboratory of Intelligent Information Processing, Fudan University
School of Computer Science and Technology. Xidian University
Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology
National Children's Medical Center, Children's ' Hospital of Fudan University
School of Information Technology, Deakin University
Shanghai Artificial Intelligence Laboratory
7 Department of Biomedical Engineering, The Hong Kong Polytechnic University
Research Institute for Smart Ageing, The Hong Kong Polytechnic University
fyjzhang, www.geinnnfuudeeeeeee @udda edu. en, zhangriaaboc30701163. com
Abstract
Chest X-rays (CXRs) are the most frequently performed imaging examinations in clinical settings. Recent advancements Medical Large Multimodal Models (MLMMs) have enabled automated CXR interpretation, improving diagnostic ac- curacy and efficiency. However, despite their strong visual understanding, current MLMMs still face two major challenges: (1) insufficient region-level understanding and interaction, and (2) limited accuracy and interpretability due single-step prediction. In this paper, we address these challenges by empowering MLMMs with anetory-centrie reasoning capabilities to enhance their interactivity and ex- plainability. Specifically, we propose an Anatomical Ontology -uideed Reasoning (AOR) framework that accommodates both textual and optional visual prompts, centered on region-level information to enable multimodal multi-step reasoning. We also develop LOR-Instruction a large instruction dataset for MLMMs training, under the guidance of expert physicians. Our experiments demonstrate AOR's 's supe- rior performance both Visual Question Answering (VQA) and report generation tasks. Code and data are available at: https::// thub. cow Lig/g//QRq
1 Introduction
Chest X-rays (CXRs), with over 2 billion examinations annually 22, are essential for diagnosing and monitoring thoracic diseases 34 However, their interpretation time-consuming and expertise- dependent, leading to diagnostic delays, variability, and errors [ o especially under increasing clinical workloads and radiologist shortages. These challenges highlight the urgent need for accurate and efficient automatic CXR interpretation systems support modern medical practice.
Early task-specific models for CXR interpretation focused disease classification 32 detec- tion (28), or report generation [8], but often lacked generalization and interpretability. Recently, the advancement of Large Multimodal Models (LMMs) [23] (2 has emerged as a promising and scalable solution for CXR interpretation. Notable examples include LLa [18] and CheXagent (9]
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pn
Single Stop
Our with Region fevel Understunding and sutti-stap Reasoning.
mediestinum/
Observed the
Consicering ozesenation,
No.
medaetinum
[o o.0, o 0.73
the
chvoor lod junction
In G, 0.
night
therels dight strlem
meciedirum i o,0 2.17
(o e.o
Tierdfore, there a
(0.4, 0.5 0.550.5.022 o.5).
So the answeris
l: the eregion- enlorged??
Single-Step Answer
Our AOR with Region-level Understonding and Multi-step Reasoning.
The cardlohhradc
The region in cardic snduccstoouette lo 2o..0.d
the cardar sihouetra
and
catt monomore
than salls
maeian horizental diameter of thorax (o al
Figure I: Previous image-level MLMMs (shown in light red) make incorrect predictions or fail to predict due to (1) insufficient region-level perception and (2) reliance single-step reasoning. In contrast, our AOR model (shown in light green) delivers explainable and accurate answers by (1) emphasizing region-level understanding and (2) employing multi-step reasoning.
While these existing Medical LMMs (MLMMs) have shown remarkable capabilittes in both visual understanding and language modeling, they still face two key challenges that significantly limit their effectiveness in medical imaging applications:
(1) Insufficient Region-Leve Understanding and Interaction Radiologists interpret CXRs by scanning the full image and then examining specific anatomical regions for abnormalities. To emulate this process, models must capture fine-grained visual details, spatial relationships, and anatomical hierarchies. However, current image-level MLMMs [18] 36) struggle detect subtle but clinically relevant findings. Additionally, as shown in Fig. region-level interaction is crucial: not only for radiologists revisiting suspicious areas but also for non-expenn non-experts who rely on models interpret visual cues without medical terminology. Existing MLMMs lack such capabilities, limiting their real-world applicability. Thus, it is imperative to highlight region-level perception overcome these obstacles.
(2) Limited Accuracy and Interpretability due to Single-Step Prediction Without Reasoning Medical imaging often involves overlapping symptoms and diverse disease manifestations, demanding multi-step reasoning for accurate diagnosis. Models must jointly consider lesion location, appearance, and clinical context in relation to the question. However, current MLMMs typically make single-step predictions without explicit reasoning, resulting in misinterpretations 10., hallucinations [29]. and alignment. A key bottleneck is the lack of high-quality, clinically grounded instruction data to support multi-step reasoning, given the specialized knowledge and low error tolerance required. To address this, constructing a clinically credible Chain-of -houuhhhht (CoT) dataset is essential for improving both accuracy and interpretability in MLMMs.
poor tampoom-lesion-diseee
In this paper, we introduce the Anatomical Ontology-Guided Reasoning (AOR) framework with three-stage training strategy. AOR centers on the anatomical regions relevant to the given question, incorporating their positional and representational information to enable multimodal, multi-step reasoning. Then, to address the shortage of high-quality multimodal reasoning datasets for MLMMs, we develop the Ins-Instrttion dataset under the guidance of three expert physicians. This dataset consists of two subsets: AOR-VQA and AOR-RG. Specifically, for AOR-VQA. we construct 2,812 12 CoT templates based three anatomical ontologies to provide precise CoT answers for 290k Visual Question Answering (VQA) samples. For AOR RG. 133k CXR- pairs are used for full image report generation, and raw reports are further decomposed into fine- tne-grained descriptions, yielding 399k strictly aligned region-sentene pairs for interpretable region report generation.
By empowering Medical LMMs with imatomy-centric reasoning capabilities, we introduce new paradigm toward interactive and explainable medical imaging analysis for CXR interpretation. The contributions of this work summarized as follows.
We propose an Anatomical Ontology Ontology-Guided Guided Reasoning (AOR) framework, which supports both textual and optional visual prompts (i.e., region-of-intered (ROI) crops) as input, centered on region-level information to enable multimodal, multi-step reasoning.
We develop a large instruction dataset named AnR-Instraction by leveraging anatomical regions and their ontologies. It consists of two parts: AOR-VQA for VQA and RRR-RG for full image and region report generation, containing 290k and 532k data pairs, respectively.

---

Extensive experiments demonstrate the superiority of AOR, which outperforms the second-best MLMM by an average of 6.81% % on the VQA and 5.27% on report generation, underscoring the crucial role of region perception and reasoning capabilities in supporting clinical decision-making.
2 Related Works
Medical Large Multimodal Models With the success of LLMs 1
37 3
researchers are
enhancing these models by incorporating visual understanding capabilities, leading to the emergence LMMS 23 (22). In the medical domain, numerous LMM-based studies have also arisen. Prominent models such as VA-MMe [8] and Med-Flamingo [27] first perform image-text feature alignment using paired medical data, followed by meticulously designed instruction tuning. Although these models exhibit strong visual understanding, they are primarily limited to image-level tasks like report generation and medical visual question answering. They do not explicitly learn region-level features during the training process, which constrains their region-level perception.
To achieve more fine-grained image understanding, recent research has integrated region-level data into the training of LMMs. Shikra 7 directly quantizes bounding boxes into coordinates (numerical representations of positions). Subsequently, GPT4Rol 42 and RegionGPT [ extract region features from the original images and include them as part of the input token sequences, allowing the models to comprehend region representations and enabling them to process visual prompts. However, in the medical domain, research on region-level LMMs is still limited. BiRD [22 aims to equip MLMMs with grounding and referring capabilities through multi-task learning while maintaining their conversational ability. MAIRA-2 [4] focuses on enhancing LMMs for grounded report generation tasks. Both methods locate specific regions using textual coordinates and rely on single-step diagnoses, lacking the comprehensive perception and reasoning to fully leverage these regions.
Region-Leevl Medical LMMs
CoT in Medical LMMs Chain-of-Thought (CoT) prompting guides LLMs through intermediate reasoning steps to solve complex tasks [39 [6]. Recent efforts have introduced CoT into LMMs to enhance visual reasoning. For example, SoM 41 integrates supplementary visual cues (e.g.. segmentation maps). while VoCoT (20) and Visual-CoT 33 build instruction datasets for object- centric reasoning. However, CoT integration in medical LMMs remains limited. MedCoT 24 uses Gemini-Pro [35] for CoT generation, but lacks medical domain rigor. MedVLM-RI [30] applies reinforcement learning for reasoning without reference answers, though it is restricted to multiple- choice VQA. X- X-Reasoner (25) further shows that without carly-stage CoT supervision. RL alone yields suboptimal reasoning, underscoring the need for domain-specific CoT guidance in MLMMS.
3 Method
3.1 Model Overview
As illustrated in Fig. 2 AOR mainly consists of three components: (i) an image encoder Z, responsible for extracting image features; (ii) a region encoder R. deployed to extract multi-scale region features from image features; and (iii) a large language model LCM is designed to jointly model image, region, and text for reasoning after projecting image and region features into the linguistic space.
3.2 Model Development
Fig. (b) shows our three-stage training procedure for AOR. We progressively enable AOR to perform anatomy-centrie recognition, detection, reasoning, and report generation. All three training stages use cross-entropy loss for auto-regressive language modeling.
Stage 1: Anatomical Region Recognition The first stage aims to align region features with linguistic embeddings, enabling the model to recognize each anatomical region in CXR. During this stage, only the region encoder R and the region projection fú are kept trainable. For each image /, we use the anatomical bounding boxes = {(c,, nr) ) provided by Chest ImaGenome Dataset [40]. Here, d¹ E IRA, n² and N₂, represent the coordinate, region name, and the number of anatomical regions in T. The image / is first encoded into the feature maps { N; where N, is the number
of feature maps. Inspired by GPT4Rol 42), we design the region encoder which constructs
hierarchical feature pyramid from four selected layers of the image encoder. According to el.
3

---

(8)
Find
the traches [coor]
Observe the carina (coor]
Stages
(b)
Image Encoder
Region
Prgastion
Projction/
LLM
Large Language Model
Stages
Encoder
Region
Prjection +
Coor Nome
Namre Coor
Image
Find
the trachea [coor]
Region Encoder
Observe the carina (coor)
Region Encoder
Stageg
Encoder
Region
Projection
A
Projetionn,
LLM
VIDA Report
Figure 2: (a) Overview of AOR framework, which flexibly accommodates both textual and optional visual prompts (i.e., region-of affinterest (ROI) crops) as input, centered on region-level information to enable multimodal multi-step reasoning: and (b) Three-stage training procedure for AOR.
RolAlign is applied generate 14x 14 feature map from the fused hierarchical feature, followed by a pooling layer to embed multi-scale region features r². Image projection fp and region projection fp are used to connect 2N, and r into the linguistic space. Finally, the LCM integrates the projected
visual features and the text instruction embedding to recognize the current anatomical region and output its name n²:
n 4.
(1)
Stage 2: Anatomical Region Grounding In the second stage, the model is trained to localize anatomical regions, laying the foundation for subsequent reasoning tasks. Since the generation of coordinates requires overview of the entire image and the generative capability of the LLM, keep both the LLM and image projection modules trainable. Two types of tasks are considered: (I) To prevent catastrophic forgetting and align the format closer to the reasoning tasks, the model revisits the amatomical region recognition of Stage with some adjustments, i.c., concatenating the coordinates to the region feature. Here, use bbox [Eyinn Vmin Tmar: |mar as object coordinates, where and y are normalized between 0 and relative to the image size. The LLM reads the projected visual features, text instruction embedding and textual coordinates embedding ed to predict the region name n²: nd CCM((f((x))) f(("))
(2)
(2) Given region's 's name n², the model grounds the corresponding coordinates d²:
é = (C.)
Stage 3: Instruction Tuning Based on the pre-trained model, this stage fine-tunes the model using AOR-Instruction (detailed in Section
three tasks:
(1) Medical Visual Question Answering: AOR is capable handling questions that require both global and local clues, and is flexible enough to accept both textual and optional visual prompts as input. Based on the given prompt, the model centers on anatomical regions to generate logically reasoned answers. During reasoning, each region is represented in a triplet format: (region name) (coordinates) (ROI visual representation). e.g., "sve [0.27, 0.08, 0.92, 0.81] rsve". Once the end of the coordinates token "I" is generated, the region encoder R. is activated to obtain the (ROI visual representation) based on the coordinates between "(" and "I", which is formulated as follows:
ans' = -CC.(()((E))),, [n², f,,(
(2) Full Image Report Generation: Given CXR, AOR generates a comprehensive report describing the entire image:
report = L.M.(f,
(5)
For a CXR, users provide textual and optional visual prompts specifying the anatomical regions of interest. AOR generates report sentence & specifically related the designated region r'.
(3) Region Report Generation:
s LCM
(6)

---

(8) The Construction AOR-VQA
(b) The Construction of AOR-RG
Ontology & (0b)) Objj
idea
Region Region-Sentence Alignment
OML
Study
cardlopulmonary process.
The lungs are clear any focal
opecibies conceming infection.
Heart size within normel limits. No pleuraleffusion effusion pneumothorax.
unidy Aljnn
Mi
ObL Chdl
Oetsllgy oby
Obj Pauver
ontology
Ontology (Attr-Attr)
Chest ImaGenome
RadGraph
Oity
AttrE
Attra
Chill
Altra
Anetomical Region bbox Generation
Sentence
Attr
detalig
Parse
Ceriigg: ADD ASY
Arty
ADDL ANDo
Attrx. Parort
of betra
Left lung
leftlung ling is any focel oparitiis
conterring for infection. No plaural effusion pneumothorae in the left lang.
Ontology (Attr-Obj)
Example: cardiac silhouette abnormal
Right lung: scute figt und process. The right lung is clear any focal spacitiss concerning for infection. No pleural effusion pneumothoros the luno.
cavoatriaa
junction
and device the heart Check the complications
Observe city Child calculate CTR Observe
right
Cardiacs silhouette: acure beart process Healt size within no normal limits..
Altra
Steps
Stepo
Step3
Figure 3: Overview of AOR- Instruction, which consists of two sub-datasets: (a) The construction of AOR-V VQA: Anatomical ontologies design CoT construction - Sample expansion and (b) The construction of AOR-RG: Strict alignment between anatomical region and report sentence.
Instruction Data
Currently, there a shortage of high-quality multimodal reasoning datasets for training Medical LMMs, leading models that lack fine- fineggriined understanding and reasoning capabilities. To bridge this gap, we construct the AOR-Instruction dataset by leveraging anatomical regions and their ontologies. This dataset, enriched with explainable region-level visual information, helps LMMs achieve a clearer understanding of image content. The AOR-Instruction dataset consists of two components: AOR-VQA for Visual Question Answering (VQA) and AOR-RG for full image and region report generation, containing 290k and 532k data pairs, respectively.
4.1 AOR-VQA
AOR-VQA is primarily designed to enhance the model's capabilities in medical VQA. We use SOMIC-CXR-VEE VQA [3] comprehensive dataset with (Image, Question, Answer) samples, as our primary data source. It includes both global-level questions that evaluate the overall findings of CXR (e.g.. "Do you notice any abnormalities?",, and local-level questions that focus on specific anatomical regions (e.g., "Can hyperaeration be detected in the left lung?").
The entire construction process is conducted under the guidance of expert physicians two board- certified radiologists and one clinician with 24, 18, and 27 years of experience, respectively. We meticulously design and refine three types of anatomical ontologies. Based on these ontologies, we construct Chain of Thought (CoT) for each sample. Finally, the expanded information is attached to each sample, resulting in structure that includes (Image, Question, Region Box, CoT Answer). The details follows:
4.1.1 Anatomical Ontologies Design
We define anatomical regions in the CXR as objects, each associated with several attributes selected from pool of 68 attributes across five categories. Fig. (a) illustrates our anatomical ontologics.
Ontology 1: Hierarchical Relationships Between Objects In visual perception, humans organize content into hierarchical structures to understand the phrt- relationships within images, thereby obtaining the answers they seek. Fortunately, such whot- hierarchical relationships clearly exist between anatomical regions. As illustrated in Fig. (a), Obj Porent (e.g., mediastinum) includes two related objects: Obj_child1 (e.g., upper mediastinum) and Obj_child2 (e.g., cardiac silhouette). By leveraging this hierarchical relationship. the model can engage in reasoning by shifting focus from the whole to the parts and then comprehensively considering the whole based on the parts. Therefore, explore and organize the hierarchical relationships of 38 anatomical regions.
5

---

Ontology 2: Causal Relationships Between Attributes During the progression of specific at- tributes, multiple conditions can dynamically interrelate and influence one another. Additionally, attributes categorized as anatomical findings are typically the imaging manifestations of attributes classified as diseases. Thus, we can construct causal relationships between different attributes. As (e.g.. atelectasis) within an Obj must also be present. Leveraging these causal relationships enables the model to better understand the associations between attributes and utilize other attributes to complete the reasoning process when encountering attributes with unclear or difficult-to-detrrmiin visual manifestations. Consequently, the causal relationships of 68 attributes are constructed.
shown in Fig. 3] (a), if Attr atr_Child (e.g., lobar/segmental collapse) exists, then
Ontology 3: Restrictive Relationships Between Objects and Attributes Finally, we consider the restrictive relationships between objects and attributes. As shown in Fig. 3 (a), only certain attributes appear within specific objects. For example, fractures can never occur the cardiac silhouette, while pleural effusion most commonly affects the costophrenic angle. By utilizing such restrictive relationships, the model can eliminate certain scenarios, enabling more rational and effortless reasoning. Therefore, the restrictive relationships between 38 objects and 68 attributes are organized. Refer to Appendix A.2.I for more details about the three Ontologies.
4.1.2 CoT construction
Integrating the aforementioned three ontologies, as illustrated Fig. (a), we organize an ideal case for each image (represented as Study id node) in the source dataset, establishing connections between all objects and attributes for each sample. For global-based questions, we do not expand the answers, enabling the model to make fine-grained observations while also developing global summarization skills. For local-based questions, we construct a rigorous and comprehensive CoT for each question based on the ideal case, following the steps below.
For the queried object, we first identify its sub-objects (if it is already the smallest anatomical structure unit, this step is skipped). Considering the question: "Is the cardiac silhouette abnormal?", we identify the right atrium and the cavoatrial junction as the sub-objects of the cardiac silhouette for focused analysis.
Step 1: Identify Sub-objects (Using Ontology 1)
Step 2: Consider All Possible Attributes (Using Ontology 2) If the question targets a specific attribute, analysis and reasoning are conducted solely for that attribute. However, the question concerns all abnormalities of the queried object, all possible scenarios must be taken into account. Continuing with the example from Step 1, for abnormalities in the cardiac silhouette contour, this primarily includes the presence of tubes devices, changes in the size of the cardiac silhouette, and the development of other associated complications.
Once have identified the attributes to be discussed, we focus our observation and reasoning on their primary associated objects sub-objects. Continuing with the example above: for the presence of tubes or devices, particularly observe the right atrium and the cavoatrial junction; for measurements, i.e., cardiac silhouette size, place cardiac silhouette within the global context and compare the entire thorax; for the development of complications, after detecting an enlarged cardiac silhouette, consider other features, i.e., pulmonary translucency, further assess the presence of lung opacity.
Step 3: Associate the Relevant Objects and Attributes (Using Ontology 3)
the size of
Following the three steps outlined above, we can construct a CoT answer for any combination of objects and attributes (or categories and abnormalities). A total of (68 + = 2,812 types of CoT answers are constructed, all of which are reviewed and refined by three expert physicians.
Furthermore, source data includes complex combinatorial questions, such as those involving conjunc- tion and disjunction, which are commonly seen in real-world application scenarios. Therefore, we decouple such data by extracting the involved sub-questions to perform the aforementioned CoT and finally conduct an additional logical inference based on the answers to the sub-questions.
4.1.3 Sample Expansion
All samples in the dataset are expanded from (Image, Question, Answer) to (Image, Question, Region Box, CoT Answer) according to the above rules. For more details, please refer to Appendix A.2.

---

Table Comparison of methods on SIMIC-CXR-TEE VQA-RAD, and CheXpert. "_" means Med-Flaminge and MedVLM-RI lack training code and thus cannot be fine-tuned on "*" means CheXagent lacks training code, but its instruction data include MIMIC-CXR-VE VQA, so we report zero-shot results. Results in gray are excluded from the comparison since CheXagent data include VQA-RAD. Bold numbers mark the best result each column.
MIMIC-CXR-VO
VQA-RAD
CheXpert
Method
Res.
verify
choose
query
closed
open
closed
open
General-domain LMM
LLaVA [2
224
75.97
56.07
58.87
43.84
21.09
27.72
34.50
LLaVA-1.5 22
3362
75.25
58.70
56.10
44.74
13.54
27.72
33.88
GPT4RoI 42
224
77.16
56.37
60.54
43.84
17.64
52.19
32.65
VoCoT 220
448
76.17
48.79
60.70
35.62
21.38
49.60
40.89
Medical-domain LMM
LLaVA-Med 18
224
75.71
58.31
60.37
61.64
22.36
54.95
42.08
Med-Flamingo
27
224
28.77
23.89
40.99
39.80
XrayGPT
(36
224
60.00
40.97
24.07
43.84
22.51
60.59
30.45
CheXagent
9
448²
75.02*
33,49*
48.49*
68.49
24.94
62.28
32.14
MedVI.M-RI
30
Dyn
52.05
22.07
60.89
33.55
AOR(Ours)-t
336"
80.48
71.96
65.05
63.01
28.19
71.58
53.85
AOR(Ours)-r/t
336²
80.68
70.16
65.43
57.53
24.99
74.06
45.35
4.2 AOR-RG
Full Image Report Generation We directly utilize the image-report pairs provided in MIMIC- CXR [15], make of frontal images, and include findings and impressions in the report.
Region Report Generation To perform fine-grained region report generation, we need to decom- pose raw report data into fine-grained descriptions for each organ mentioned in medical scans. As shown in Fig. 3 (b). we utilize the bounding boxes provided by Chest ImaGenome Dataset and parse the text using RadGraph [14], employing the rules proposed in ASG [19] to achieve strict alignment between the two. Additionally, we optimize the alignment method to handle cases where two different anatomical regions appear in the same short sentence, introducing new rules to split such sentences into two separate ones. This approach constructs region-sentence pairs for each image-report pair.
5 Experiments
5.1 Experiment Settings
Implementation Details We initialize the image encoder with CLIP- [1 and the language model with LLaVA-1.5 (22). The image resolution set to 336x336. We use AdamW as our optimizer, with a learning rate of 2 x 10. Experiments are conducted using 4 NVIDIA A100 GPUs.
Dataset 1) For the VQA task, we evaluate on the test sets of MIMIC-CXR VQA B. VQA-RAD [T7] and CheXpert [3.. MIMIC-CXR-VOL contains 500 images and 13,793 QA pairs the test dataset. CheXpert contains 191,229 frontal chest radiographs, and we use the expert-labeled validation set as the test data, which contains 202 images and 1,212 QA pairs. VQA-RAD includes 315 images and 3,515 QA pairs across the head, chest, and abdomen. We filter it to include only chest X-ray images and their corresponding question-answer pairs, resulting in 69 images and 102 QA pairs in the test dataset. 2) For the full image report generation task, dataset of chest radiographs with free-text radiology reports, to evaluate the model's performance. The test dataset contains 500 images and their corresponding reports. 3) For the region report generation task, we use the same 500 images and sample 3 anatomical regions per image for evaluation.
MIMIC-XXR large publicly available
Evaluation Metrics For the VQA task, regarding MIMIC-CXR-VOL VQA, its questions can be catego- rized into three primary semantic types: For the "verify" questions, which include yes/no questions, we report the accuracy; for the "choose" questions, which involve selection from provided options, we also report the accuracy; for the "query" questions, where the answers are in the form of list, we report the Fj score (micro). For CheXpert and VQA-RAD following [18], for closed-end
7

---

Table 2: Comparison of report generation meth- ods on MIMIC-CXR, "1" means Res. is 336".
Table 3: Comparison of the different CoTT repre- sentations.
MIMIC-CXR
ID
coor
region
CoT
verify
choose
query
Method
R-1.
BERTScore F,CheXhert
X
76.83
61.07
58.77
2
X
X
v
80.69
67.62
63.37
Full image report generation
3
V
X
v
79.14
69.54
63.89
LLaVA-Med
13.49
75.93
0.40
4
V
v
80.68
70.16
65.43
Med-Flaminn -la-mnaa
5.21
71.26
13.61
XrayGPT
24.02
83.18
26.71
CheXagent
24.09
83.52
37.54
I.LaV--Ram V
24.22
83.67
46.20
AOR(Ours)-t
25.37
83.92
46.53
AOR(Ours)=rht
25.38
83,95
48.28
Region report generation
LLaVA-Med
9.47
71.51
16.70
Med-Flan Med-Flamingo
12.98
75.24
14.66
XrayGPT 15.80
80.19
19.96
CheXagent
20.79
81.57
33.02
Figure 4: Impact of anatomical region shifts on model predictions. Solid lines represent the orig- inal visual prompt bboxes, dashed lines represent bboxes enlarged by 10% of the original size.
LLaVA-Ras Radt
18.71
81.80
31.40
AOR(Ours)-t 35.11
84.54
36.65
AOR(Ours)-rit
35.62
84.76
36.89
questions with a single correct answer, we report accuracy; for open-end questions, we use recall to evaluate the model's responses. For the report generation task, we selected ROUGE-L (R-L) [21], BERTScore [43], and Fj CheXbert (26) as evaluation metrics to compare model performance at the word level, semantic level, and clinical efficacy level.
5.2 Quantitative Comparison
Performance on MIMIC-CXR-VQA We fine-tune models on MIMIC-CXR-VQA For fair comparison, introduce AOR-t, which uses the same training data baseline methods, where all questions are in textual form. AOR-r/t represents a setting where questions are presented in both textual and visual formats during training and inference, enabling multimodal interaction. This reflects a more general and practical usage scenario, requiring the model to dynamically parse and reason heterogeneous input modalities. Notably, regardless of how the questions are presented, both AOR-t and AOR-r/t leverage region-level visual information to enhance reasoning throughout the answering process. As revealed in Table I AOR-t outperforms the second-best method by average of 6. 6.98%, especially on the more complex "choose" and "query" types, highlighting the advantage of reasoning centered on anatomical regions.
Zero-shot Transfer to VQA-RAD and CheXpert We evaluate AOR's generalization ability on unseen data distributions, i.e., VQA-RAD and CheXpert. As shown in Table AOR-t obtains superior performance on both datasets. Moreover, AOR is capable of reasoning and providing logical answers rather than simply responding with "yys"" or "no".
Performance on MIMIC-CXR Table that AOR outperforms all previous methods in both full-image and region-based report generation. Notably, it addresses the gap in generating report sentences for specific regions, which remains limitation in previous MLMMs.
5.3 Ablation Studies and Discussions
Comparison of CoT Representations As shown in Table 3 we demonstrate the effectiveness of the AOR representation by comparing it with three alternative CoT representations for the VQA task. ID-1 uses the original answer without any CoT reasoning, resulting in the lowest accuracy, especially on complex question types. ID-2 introduces text-based CoT that outlines object-level reasoning steps, which significantly improves performance and highlights the value of explicit multi- step reasoning. ID-3 adds textual coordinates. enabling the model to gain spatial awareness by first identifying key anatomical regions. Building this, ID-4, our proposed AOR representation, further integrates cropped, multi-scale visual features from each ROI. This enhancement allows the model to leverage direct visual cues during reasoning and achieves the best overall performance.

---

Table 4: Comparison of AOR's referring and grounding capabilitiees under different training strategies.
Strategy
VQA
Referring Acc
Grounding
Stage
Stage 2
verify
choose
query
R@0.3
R@0.5
R@0.7
X
79.53
67.96
62.33
98.03
98.48
96.16
88.79
X
80.53
70.05
63.96
98.30
98.31
96.54
89.40
80.68
70.16
65.43
98.58
98.56
96.62
90.40
Medical Visual Question Answering
Question: Can both enteric tube and IJ line be identified within the mediastinum?
Answer: Yes
AOR: Find mediastinum [0.32, 0,12, 0.63, 0.60].
enteric tube, observe upper mediustinum [0. (0.36,
0.57,0.36), the cardiac silbouette [0.32, 0.37, 0.63, 0,60), and the mediastiumm [0.32, 0.12. 0.63, 0.60). Considering all observations, there enteric tube in the mediastinum (0.32, 0.12, 0.63, 0.60). For the LI line, observe the sve [0 36, 0.25, 0.48, 0.36], cavestrial junction 10,32, 0.37, 0.44, 0.43), right atrium [0.32. 0.43, 0.56]. mediastinum (0.32,0.12,0..)) 12,0.63,0.60). Considering all observations, there is IJ line in the mediastinum [0.32, 0.12, 0.63, 0.60). So the answer is yes.
Full Image Report Generation & Region Report Generation
GroundTruth:: AP portable upright the chest. tube appears stable position tip above carina. NG tube courses into the left upper abdomen. Mild bosal ateloctasis noted. Otherwisc lungs are clear. Cardiomed- iastinal silhouctte is stable. No acuto bony abmeemalitios. Tubes positioned appropriatoly. Mild bibasilar atelectasis.
from the carina. An enteric tube tip within the stomach. Heart size is normal. Mediustinal and hilar contours are unremariaale. Pulmonary vasculature is not engurged. Patchy opacities are noted the lung bases, potentially atolectasis. pleural effasson pacumothorax present. acute osseous ahnormality is detected. Standard positioning of the endotracheal and enteric tubes.
4OR:
Endotracheal tube tip terminates upprosimately
Question: Please generate corresponding radiology report based on region Chest X-ray.
AOR: cardiomediastina silhowcttc is within normal limits.
GroundTruth: cardiomediastinal silhouctic is stable.
Figure 5: Qualitative analysis of AOR on VQA task and report generation task.
Referring and Grounding Capabilities We analyze the model's referring and grounding capabili- ties to verify the rationality of the CoT process. In Table when all three training stages are involved, the model achieves a 98.58% referring accuracy and 90.40% R @00.7, ensuring both the accuracy and explainability of the answers. When Stage 2 is removed, the recall drops accordingly, indicating that the grounding task in Stage 2 lays a solid foundation for reasoning. Similarly, without Stage I, the performance of referring is affected, demonstrating the effecttivnness of our three-stage training.
The Impact of Anatomical Region Shifts on Model Predictions In practical applications, con- sidering that radiologists may not provide perfectly accurate region prompts and patients might offer bboxes that deviate from standard anatomical regions, we explore the impact of bbox shifts on prediction accuracy. As shown in Fig. 4 we apply shifts to the bbox in the horizontal and vertical directions during model inference (d) r- w. d2 = r- ht,, where r is a random number between O and percentage E 0.1 {0..... 5). and and h are the width and height of the image. When p E [0.1, 0.2). the model's performance is barely affected. However, downward trend emerges when p exceeds 0.3. This indicates that for shifts that do not impact the bbox class prediction, AOR is robust enough to produce correct predictions. Conversely, when the shift becomes larger, the model might misclassify the category during the first step of reasoning (e.g., shifted left lung might be misclassified as the mediastinum). thereby affecting subsequent accuracy. This also indirectly demonstrates that our model performs step-by-step logical analysis of anatomical regions during reasoning. As shown by the dashed lines in the Fig. [ we explore the impact moderately enlarging the bboxes in the visual prompt (by 10% of the original size). This brought certain degree of performance gain, but there remains a performance gap compared to using perfectly accurate bboxes.
Scalability of CoT Construction We generate CoT answers for each sample in VQA-RAD to verify the generalizability of our construction method to other CXR datasets. The con- struction is accomplished using a two-stage approach, Keyword Mapping and Sample Expansion. Please refer Appendix B.5 for more details. As shown in Table 5 the model fine-tuned with CoT answers achieves a performance gain of 4.78%
Table 5: Performance gain from CoT answers on VQA-RAD.
Epoch
Epochs Closed Open
Closed Open
who CoT
66.38
32.56
83.62
59.33 65.23
w/ CoT
68.97
37.16
89.66

---

5.4 Qualitative Analysis
Fig. illustrates AOR's capabilitiis in medical grounded chat and referential dialogue. For the VQA task, AOR is capable of generating correct and logically reasoned answers. For the report generation task, due to the incorporation of fine-graing aeeeggained anatomical regions, AOR demonstrates a stronger grasp of details, such as ET tube, NG tube, and basal atelectasis. Moreover, it can generate corresponding report sentences for specified regions. Please refer to Appendix C for more cases.
6 Conclusion
In this paper, we empower MLMMs with anatomy-centric reasoning capabilities, by (1) proposing the AOR framework, which centers on the anatomical regions relevant to the given question, and integrating the regions' positional and representational information to conduct multimodal multi- step reasoning; (2) developing the medical CoT dataset AOR- Instruction, which provides tailored CoT answers for each VQA sample and strictly aligned region-sentence pairs for report generation. Experiments demonstrate the superiority of AOR over prior MLMMs in visual question answering, report generation, referring, and grounding. revealing its potential in clinical practice.
Acknowledgements
This work was supported by Shanghai Natural Science Foundation (No. 252R 1401028), and the Science and Technology Commission of Shanghai Municipality (No. 235 13511100 1106022).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rXii:23033.08774, 2023.
[2] Yasmeena Akhter, Richa Singh, and Mayank Vatsa. Ai-based radiodiagnosis using chest x-rays: A review. Frontiers in big data, 6:1 120989, 2023.
[3] Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric Chang, Tackeun Kim, et al. Ehrxqa: A multi-modal question answering dataset for electronic health records with chest x-ray images. Advances in Neural Information Processing Systems, 36, 2024.
[4] Shruthi Bannur, Kenza Bouzid, Daniel C Castro, Anton Schwaighofer, Anja Thieme, Sam Bond-Taylor, Maximilian Ilse, Fernando Pérez-García, Valentina Salvatelli, Harshita Sharma, et al. Maira-2: Grounded radiology report generation. arXiv preprint XrXiv:2406.0444 2024.
[5] Adrian P Brady. Error and discrepancy in radiology: inevitable or avoidable? Insights into imaging, 8:171-182, 2017.
[6] Michael Bruno, Eric A Walker, and Hani H Abujudeh. Understanding and confronting our mis- takes: the epidemiology of error in radiology and strategies for error reduction. Radiographics, 35(6):1663 (116):6668-6766 1676, 2015.
[7] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.15195 2023.
[8] Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer. arXiv preprint arXiv:2010. 16056, 2020.
[9] Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. Chexagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401. 122008 2024.
[10] Bowen Gu, Rishi J Desai, Kueiyu Joshua Lin, and Jie Yang. Probabilistic medical predictions of large language models. npj Digital Medicine, 7(1):367, 2024.
10

---

[11] Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13796-13806, 2024.
[12] Xiaoshuang Huang, Haifeng Huang, Lingdong Shen, Yehui Yang, Fangxin Shang, Junwei Liu, and Jia Liu. A refer-and-groume multimodal large language model for biomedicine. In International Conference on Medical Image Compuling and Compuler-Assisted Intervention, pages 399-409. Springer, 2024.
[13] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ileus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAl conference artificial intelligence, volume 33, pages 590-597, 2019.
[14] Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang. Matthew P Lungren, Andrew Ng. et al. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106. /4463, 2021.
[15] Alistair Johnson, Matt Lungren, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. Mimic-cxt-jpg-chest radiographs with structured labels. PhysioNet, 2019.
[16] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-222111 22213, 2022.
[17] Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
[18] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and and-viison assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024.
[19] Qingqiu Li, Xiaohan Yan, Jilan Xu, Runtian Yuan, Yuejie Zhang, Rui Feng, Quanli Shen, Xiaobo Zhang, and Shujun Wang. Anatomical structure- gureee medical vision-language pre- training. In International Conference on Medical Image Computing and Computer-Asiisted Intervention, pages 80-90. Springer, 2024.
[20] Zejun Li, Ruipu Luo, Jiwen Zhang, Minghui Qiu, and Zhongyu Wei. Vocot: Unleashing visually grounded multi-step reasoning in large multi-modal models. arXiv preprint 2rXiv:2405.1691 16919, 2024.
[21] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pages 74-81, 2004.
[22] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296 26306, 2024.
[23] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[24] Jiaxiang Liu, Yuan Wang, Jiawei Du Joey Tianyi Zhou, and Zuozhu Liu. Medcot: Medical chain of thought via hierarchical expert. arXiv preprint arXiv:2412.13736 2024.
[25] Qianchu Liu, Sheng Zhang, Guanghui Qin, Timothy Ossowski, Yu Gu, Ying Jin, Sid Kiblawi, Sam Preston, Mu Wei, Paul Vozila, et al. X-reasoner: Towards generalizable reasoning across modalities and domains. arXiv preprins arXiv:2505.0398) 2025.
[26] Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, and Dan Jurafsky. Improving fac- tual completeness and consistency of image-to-text radiology report generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5288-5304 2021.
11

---

[27] Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia. Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: multimodal medical few-shot learner. In Machine Learning for Health (MLAH), pages 353-367. PMLR, 2023.
[28] Ha Q Nguyen, Khanh Lam, Linh T Le, Hieu H Pham, Dat Q Tran, Dung B Nguyen, Dung D Le, Chi M Pham, Hang TT Tong, Diep H Dinh, et al. Vindr-exr: An open dataset of chest x-rays with radiologist's 's annotations. Scientific Data, 9(1):429, 2022.
[29] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Med-halt: Medical domain hallucination test for large language models. arXiv preprint arXiv:2307.15343 2023.
[30] Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvim-rl: Incentivizing medical reasoning capability of -anguage models (vlms) via reinforcement learning. arXiv preprint arXiv:2502.19634 196344 2025.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 9748-8763. PMLR, 2021.
[32] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint 1711.027 2017.
[33] Hao Shao, Shengju Qian, Han Xiao, Guanglu Song. Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[34] Paul Suetens. Fundamentals of medical imaging. Cambridge university press, 2017.
[35] Gemini Team, Rohan Anil, Sehastian Borgeaud, Jean- Saan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 11805, 2023.
[36] Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radiographs summarization using medical lision-language models. arXiv preprint vrXiv:2306.07971, 2023.
[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efffciint foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[38] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.00 09288, 2023.
[39] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of of oh-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24887, 2022.
[40] Joy T Wu, Nkechinyere N Agu, Ismini Lourentzou, Arjun Sharma, Joseph A Paguio, Jasper Yao, Edward C Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, et al. Chest imagenome dataset for clinical reasoning. arXiv preprint arXiv:2108.003 2021.
[41] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. 1441.
[42] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest arXiv preprint arXhn:2307.03600 2023.
12

---

[43] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1944006675, 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In Abstract and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Appendix D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

This work focuses on model design and data construction, and does not include theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Section 5 and Appendix B
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We provide the code and scripts for reproducing all the experimental results in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
public/guidess https///oooosuubssssoooooooooo/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp cc/public/guu /etggudddsse/eebbbinninn ssiiisolccyyyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information on the computer resources in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
This study conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
In Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We indicate the licenses public datasets in the Appendix A.I and acknowl- edge the referenced code in the anonymous link.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the documentation along with the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: In Section 3
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: /ttparrrpp..cc//ooofeeeeeee///// for what should or should not be described.
20

---

Appendix
A. Data Details
22
A. Data Source
22
A.2. Instruction Data Construction
23
A.2.1. AOR-VQA
23
A.2.2. AOR-RG
31
B. Experimental Details
32
B.1. Prompt Details
32
B 3.2. Training Details
33
3. Evaluation Details
33
B.4. Result Details
34
B.4.1. Intermediate Results of Stage
34
B.4.2. Intermediate Results of Stage
34
B.4.3. Results Details of Stage
34
B.5. Scalability of CoT Construction
36
C. Case Study
37
C.I. Visual Question Answering (VQA) on MIMIC-CXR-VE VQA
37
C.2. Visual Question Answering (VQA) VQA-RAD and CheXpert
38
C.3. Full-Image Report Generation on MIMIC-CXR
39
C.4. Region-Level Report Generation MIMIC-CXR
40
D. Limitations and Future Work
41
Impact Statement
21

---

A Data Details
A.1
Data Source
Table A1: Summary of datasets and tasks across different stages.
Stages
Tasks
Source
Size
Stage 1
Anatomical Region Recognition
Chest ImaGenome
133k x 3
Anatomical Region Recognition
Chest ImaGenome
133k x 3
Stage 2
Anatomical Region Grounding
Chest ImaGenome
133k x 3
Visual Question Answer
MIMIC- VQA
290k
Stage 3
Full Image Report Generation
MIMIC-CXR
133k
Region Report Generation
MIMIC-XXR
133k X 3
We present data details in TableAA] We utilize the dataset under the PhysioNet license, and the following is a detailed introduction to it:
A large publicty available dataset of chest radiographs with free-text radiology reports. The dataset contains 377,110 images corresponding to 227,835 radiographic studies performed the Beth Israel Deaconess Medical Center in Boston, MA.
MIMIC-CXR
Chest ImaGenome Datase Chest ImaGenome dataset constructs fine- une-grained annotations top of MIMIC-CXR The annotations for each CXR are structured as an anatomy-centerd scene graph. Based this scene graph, pairs of anatomical regions and report sentences can be derived. The dataset size consistent with that of MIMIC-CXR.
MIMIC-CXR-VOA MIMIC-CXR VQA is created based on the fine- one-grained annotations of the Chest ImaGenome dataset, establishing a medical visual question answering (VQA) benchmark. It includes a variety questions, such as those about the entire image and those focused on one or multiple anatomical regions. The questions are diverse and challenging.
Since the three data sources are interrelated (MIMIC-CK CAMIC--XR-AE - Chest ImaGenome Dataset-> MIMIC- CXR-VQA), MIMIC-CX-- -XQ''sss data selection and split strategy are adopted across all three to prevent data leakage, with 133k images for training. 500 images for testing. In Stage 1 and Stage 2, we use the anatomical bounding boxes provided by the Chest ImaGenome dataset, randomly selecting three anatomical regions per image for training. Similarly, in Stage 3, during region report generation, three anatomical regions randomly selected per image for generating corresponding report sentences. Additionally, as shown in Fig. A1 we analyze the distribution of different types of questions MIMIC-CXR- VQA.
Plane (0.7%%
Gender (0.77%
Combinatorial 37.08%
Non- Combinatorial
(62 92%)
Query
Global-tewe 99%)
Gender (0 (0.77%)
Cheose
Prane (0.77%)
Al Aavermate
lorat-level (71011%)
Poo Poleeen snic
2206 70%
Plano (0.7%)
(0 (7377)
Figure A1: Distribution of question types in MIMIC-CXR-VOL VQA.
"MIMIC-XX" Available at http://yyyiiottoonmmmmmmmmmmnnnn
"Chest ImaGenome Dataset: Available at http://hhyyyonccccoooooooooiiiee .0.0/
MIMICC HHIIIC-CX--I EXR-VQA- Available at http://yyyyaaeeeeeooocmmmmmmeeee
22

---

A.2 Instruction Data Construction
The Ant-Instruction consists of two parts: AOR-VQA and AOR-RG.
A.2.1 AOR-VQA
Table A2 displays the objects, attributes, and categories used in AOR- VQA, including 36 objects and 68 attributes (across categories). We adopt the same values as in MIMIC-CXR- VQA and adjust the capitalization of the specialized terms IJ line and PICC.
Table A2: The objects, attributes, and categories used in AOR-VQA.
Values
object
abdomen, aortic arch, cardiac silhouette, carina, cavoatrial junction, left apical zone, left breast, left chest wall, left clavicle, left costophrenic angle, left hemidiaphragm, left hilar structures, left lower lung zone, left lung, left mid lung zone, left shoulder, left upper lung zone, mediastinum, neck, right apical zone, right atrium, right breast, right chest wall, right clavicle, right costophrenic angle, right hemidiaphragm, right hilar structures, right lower lung zone, right lung, right mid lung zone, right shoulder, right upper lung zone, spine, svc, trachea, upper mediastinum
attribute
(anatomical finding) lung opacity, airspace opacity, consolidation, infiltration, at- electasis, lincar/patchy atelectasis, lobar/segmental collapse, pulmonary edema/- hazy opacity, vascular congestion, vascular redistribution, increased reticular mark- ings/ild pattern, pleural effusion, costophrenic angle blunting, aleural/parenchymal scarring, enlarged cardiac silhouette, mediastinal displacement, mediastinal widen- ing, enlarged hilum, tortuous aorta, vascular calcification, pneumomediastinum, pneumothorax, hydropneumothhrax, lung lesion, mass/nodule (not otherwise spec- ified), multiple masses/nodules, calcified nodule, superior mediastinal mass/en- largement, rib fracture, clavicle fracture, spinal fracture, hyperaeration, cyst/bullae, clevated hemidiaphragm, aub-diahhragmatic air, subcutancous air, hernia, scol- iosis, spinal degenerative changes, shoulder osteoarthritis, bone Icsion, (disease) pneumonia, fluid overload/heart failure, copd/emphysema, granulomatous disease, interstitial lung disease, goiter, lung cancer, aspiration, alveolar hemorrhage, peri- cardial effusion, (device) cabg grafts, prosthetic valve, cardiac pacer and wires, (technical assessment) low lung volumes, rotated, breast/nipple shadows, (tubes and lines) chest tube. mediastinal drain, endotracheal tube, tracheostomy tube, PICC, I line, chest port, subclavian line, swan-ganz catheter, intra-aortic balloon pump, enteric tube
category
anatomical finding, device, disease, technical assessment, tubes and lines
Fig. illustrates the step-by-step construction process of AOR-VQA, which consists of ontologies design, chain-of-thought (CoT) construction, and sample expansion enrich dataset annotations.
Step
Step 2
Step 3
Ontologies Design
CoT Construction
X
Sample Expansion
Ontology 1:
Step 1:
(Image, Question, Answer)
Hierarchical relationships
Identify sub-objects
Ontology
Step
Causal relationships
Consider all possible attributes
(Image, Question, Region Box, CoT Answer)
Ontology 3:
Step 3:
Restrictive relationships
Associate objects and attributes
Figure A2: Overview of the AOR-V VQA construction process.
23

---

Step 1: Ontologies Design Fig. A3 illustrates the hierarchical relationships between objects (Ontology that we utilize, where the green boxes represent objects existing in the silver/gold datasets of the Chest Imagenome Dataset, and the gray boxes represent the objects we used during the construction process.
Ontology
(Obj-Obj) (b)
[M] other soft téonne
[M] majorstmecture
[S] subanslemyy
[S] neek?
[S] shoulder
[S] chest wall
[S] abdiomen
[S] anti
[L] subanatomy
objects existed in silverigold dataset
hemridixphram
right chouldeer
right
[S] arm
shouller
major airways
hilar structares
[S] left bemidlaphraam
[S] right hamidiaphmyn
[S] left breast"
[S] right
[i] tracheo
[S] left hilar stroctures
[S] right
hile structures
[G] carina
[M] bones
[M] lungs
[S] spine
[S] rib
[S] clavtole
[L] left lung
[L] left lung
[S] left claviole
[S] right clavicle
[-] left opical
apper lung
apper
right upper
[M] mediastinum
ling
[L] right lung
[S] crilii::
Sillewette
upper mediastinum
left costophrenic angle
Si costophrenic
costophrenic
[L.] right striums
[1.] cavcortiaal junction
[C]] ave
[S] nortic arch
Figure A3: Ontology 1: Hierarchical relationships between objects.
Table A3 and Table A4 display the causal relationships between attributes (Ontology 2) and the
restrictive relationships between objects and attributes (Ontology 3).
Table A3: Ontology 2: Causal relationships between attributes (Child - Parent(s))
Attribute (Child)
Attribute(s) (Parent)
airspace opacity
lung opacity
alveolar hemorrhage
lung opacity
aspiration
lung opacity
atelectasis
lung opacity
bronchiectasis
lung opacity
calcified nodule
lung opacity, lung lesion
consolidation
lung opacity
costophrenic angle blunting
lung opacity, pleural effusion
fluid overload/heart failure
lung opacity
granulomatous disease
lung opacity
hydropneumothorax
lung opacity, pneumothorax, pleural effusion
increased reticular markings/ild pattern
lung opacity
infiltration
lung opacity
interstitial lung disease
lung opacity
linear/patchy atelectasis
atelectasis, lung opacity
lobar/segmental collapse
atelectasis, lung opacity
lung cancer
lung opacity
lung lesion
lung opacity
mass/nodule (not otherwise specified)
lung opacity, lung lesion
multiple masses/nodules
lung opacity, lung lesion
pleural effusion
lung opacity
pleural/parenchymal scarring
lung opacity
Continued on next page

---

Table A3 -ontinued from previous page
Child Attribute
Parent Attribute(s)
pneumonia
lung opacity
pulmonary edema/hazy opacity
lung opacity
vascular congestion
lung opacity
Table A4: Ontology 3: Restrictive relationships between objects (Parents) and attributes (Children).
Object (Parent) cardiac silhouette
Attributes (Children)
cabg grafts, aortic graft/repair, prosthetic valve, cardiac pacer and wires, fluid overload/heart failure, pericardial effusion, lung opacity, medi- astinal displacement, preumomediastinum. enlarged cardiac silhouette, vascular calcification, hernia, low lung volumes, rotated, enteric tube, mediastinal drain, pigtail catheter, chest port, IJ line, picc, subclavian line, swan-ganz catheter
mediastinum
cabg grafts, aortic graft/repai,, prosthetic valve, cardiac pacer and wires, lung cancer, fluid overload/heart failure, pericardial effusion, goiter, calcified nodule, lung opacity, lung lesion, mediastinal displacement, me- diastinal widening, pneumomediastinum, superior mediastinal mass/en- largement, enlarged cardiac silhouette, tortuous aorta, vascular calcifica- tion, hermia, low lung volumes, rotated, endotracheal tube, tracheostomy tube, enteric tube, mediastinal drain, pigtail catheter, chest port, IJ line, intra-aortic balloon pump, picc, subclavian line, swan-gan wwan-ganz catheter aortic gratV/rppar, mediastinal widening, tortuous aorta, vascular calcifi- cation, intra-aortic balloon pump
aortic arch upper mediastinum
aortic graft/repair, cardiac pacer and wires, lung cancer, goiter, calcified nodule, lung opacity, lung lesion, mediastinal displacement, mediasti- nal widening, pneumomediasinum, superior mediastinal mass/cnlarge- ment, tortuous aorta, vascular caleification, rotated, endotracheal tube, tracheostomy tube, enteric tube, mediastinal drain, chest port, IJ line, intra-aortic balloon pump, pice, subclavian line, swan-ganz catheter cardiac pacer and wires, chest port, IJ line, picc, subclavian line, swan- ganz catheter
cavoatrial junction
arms (left/right)
cardiac pacer and wires, bone lesion, subcutaneous air, skin fold, pice cardiac pacer and wires, bone lesion, rib fracture, subcutancous air, breast/nipple shadows, skin fold, chest tube, pigtail catheter, chest port cardiac pacer and wires, bone lesion, clavicle fracture, shoulder os- teoarthritis, rotated, endotracheal tube, tracheostomy tube, chest port, pice, subclavian line
chest wall (left/ighh)
clavicle (left/right)
right atrium
cardiac pacer and wires, chest port, IJ line, picc, subclavian line, swan- ganz catheter
svc
cardiac pacer and wires, chest port, IJ line, picc, subclavian line, swan- ganz catheter
lower lung zone (left-
alveolar hemorrhage, aspiration, copd/emphysema, granulomatous dis- ease, interstitial lung disease, pneumonia, lung cancer, fluid overload/- heart failure, airspace opacity, calcified nodule, consolidation, cyst/bul- lae, hyperaeration, increased reticular markings/ild pattern, infiltration, atelectasis, linear/patchy atelectasis. lobar/segmental collapse, lung opac- ity, lung lesion, mass/nodule (not otherwise specified), multiple mass- es/nodules, pulmonary edema/hazy opacity, hydropneumothorax, pleu- ral/parenchymal scarring, pneumothorax, vascular congestion, vascular redistribution. bronchiectasis, pleural effusion, breast/nipple shadows, low lung volumes, chest tube, pigtail catheter
/right)
Continued on next page
25

---

Table A4 Continued from previous page Attributes (Children)
Object (Parent)
lung (left/right)
alveolar hemorrhage, aspiration, copd/emphysema, granulomatous dis- ease, interstitial lung disease, pneumonia, lung cancer, fluid overload/- heart failure, airspace opacity, calcified nodule, consolidation, cyst/bul- lae, hyperaeration, increased reticular markings/ild pattern, infiltration, atelectasis, linear/patchy atelectasis, lobar/segmental collapse, lung opac- ity, lung lesion, mass/nodule (not otherwise specified), multiple masses/n- odules, pulmonary edema/hazy opacity, enlarged hilum, hydropneumothh orax, pleural/parenchhmall scarring, pneumothorax, vascular congestion, vascular redistribution, bronchiectasis, pneumomediastinum, vascular calcification, costophrenic angle blunting, pleural effusion, breast/nipple shadows, low lung volumes, rotated, endotracheal tube, chest tube, pig- tail catheter, chest port, subclavian line, swan-ganz catheter
mid lung zone (left-
alveolar hemorrhage, aspiration, copd/emphysema, granulomatous dis- ease, interstitial lung disease, pneumonia, lung cancer, fluid overload/- heart failure, airspace opacity, calcified nodule, consolidation, cyst/bul- lae, hyperaeration, increased reticular markings/ild pattern, infiltration, atelectasis, linear/patchy atelectasis, lobar/segmental collapse, lung opac- ity, lung lesion, mass/nodule (not otherwise specified), multiple mass- es/nodules, pulmonary edema/hazy opacity, hydrroneeumothorax, pleu- ral/parenchymal scarring, pneumothorax, vascular congestion, vascular redistribution, bronchiectasis, pleural effusion, low lung volumes, chest tube, pigtail catheter
/right)
upper lung zone (left-
alveolar hemorrhage, aspiration, copd/emphysema, granulomatous dis- ease, interstitial lung disease, pneumonia, lung cancer, fluid overload/- heart failure, airspace opacity, caleified nodule, consolidation, cyst/bul- lae, hyperaeration, increased reticular markings/ild pattern, infiltration, atelectasis, linear/patchy atelectasis. lobar/segmental collapse, lung opac- ity, lung lesion, mass/nodule (not otherwise specified), multiple mass- es/nodules, pulmonary edema/hazy opacity, hydropneumothorux, pleu- ral/parenchymal scarring, pneumothorax, vascular congestion, vascular redistribution, bronchiectasis, pleural effusion, low lung volumes, chest tube, pigtail catheter, chest port, subclavian line
/right)
apical zone (left- /right)
aspiration, granulomatous disease, pneumonia, lung cancer, calcified nodule, cyst/bullae, lung opacity, lung lesion, mass/nodule (not other- wise specified), multiple masses/nodules, hydropneumothorax, pleural/- parenchymal scarring, pneumothorax, pleural effusion, chest tube, pigtail catheter
hilar structures (left- /right)
aspiration, copd/emphysema, granulomatous disease, pneumonia, lung cancer, fluid overload/heart failure, airspace opacity, calcified nodule, consolidation, cyst/bullae, infiltration, atelectasis, linear/patchy atelecta- sis, lung opacity, lung lesion, mass/nodule (not otherwise specified), mul- tiple masses/nodules,, pulmonary edema/hazy opacity, enlarged hilum, pleural/parenchymal scarring, vascular congestion, vascular redistribu- tion, bronchiectasis, pneumomediastinum, vascular calcification, endo- tracheal tube, swan-ganz catheter
hemidiaphragm (left- /right)
copd/emphysema, hyperaeration, elevated hemidiaphragm, diaphrag- matic eventration (benign), hernia, sub-diaphragmatic air, enteric tube lung cancer, fluid overload/heart failure, airspace opacity, calcified nod- ule, consolidation, infiltration, atelectasis, linear/patchy atelectasis, lung opacity, lung lesion, mass/nodule (not otherwise specified), multiple masses/nodules, pulmonary edema/hazy opacity, hydropneumothorax, pleural/parenchynal scarring, pneumothorax, costophrenic angle blunt- ing, pleural effusion, pigtail catheter
costophrenic
angle
(left/right)
neck
goiter, subcutaneous air, endotracheal tube, tracheostomy tube, enteric tube, IJ line, awan-ganz, catheter
trachea
goiter, mediastinal displacement, endotracheal tube, tracheostomy tube Continued next page
26

---

Table A4 Continued from previous page Attributes (Children)
Object (Parent)
shoulder (left/right)
bone lesion, shoulder osteoarthritis, skin fold, picc
spine
bone lesion, scoliosis, spinal degenerative changes, spinal fracture, tated
abdomen
sub-diaprrammatic air, enteric tube, intra-aortic balloon pump, swan- ganz catheter
breast (left/right) carina
breast/nipple shadows
endotracheal tube, tracheostomy tube
Step 2: CoT Construction As shown in Fig, A4 provide tree diagram to illustrate the CoT construction process. ! First, determine whether the current problem contains conjunctions and disjunctions.
does, split into two non-combinatorial samples, analyze each separately, and then combine the results. 2 For non-combinatorial samples, assess whether they are global-based or local-based. If they are global-based, directly assign the CoT answer. Otherwise, proceed to step 3. For local-based questions, determine whether the involved object is the smallest unit (indicated by [L] in Fig. A3). If it is, directly assign the CoT answer; otherwise, use Ontology to identify the sub-object.
For the current problem, if it focuses on specific attribute, construct the corresponding CoT answer in conjunction with Ontology 3 under the guidance of expert doctors. If it focuses on category and abnormality, it is necessary subdivide into sub-aturiie attributes to complete the construction of the CoT answer.
sample
Combinatorial
Non-combinatorial
Non-combinatorial
Non-combinatorial
local
global
objectl
object
cot
objectl.object ooject2
category
abnormality
attribute
attributel attributee2
category 1,categor22
cot ans
Figure A4: The COT construction process, illustrating the hierarchical decision-making steps from sample classification to CoT answer generation.
27

---

Step 3: Sample Expansion from (Image, Question, Answer) to (Image, Question, Region Box, CoT answer)
The above is the complete construction process of AQR-VQA. Below, we provide examples for different objects. Here, <cot_ bboo_ pox_nnnm>in the examples will be replaced with the corresponding bounding box coordinates in the instruction data.
cardiac silhouette As shown in Table A5 explain "enlarged cardiac silhouette" and "lung opacity" together: For enlarged cardiac silhouette, reasoning is performed by measuring and calculat- ing the cardiothoracic ratio. For lung opacity, it also involves findings related to the enlarged cardiac silhouette. However, beyond this, further assessment of the cardiac shape and lung lucency is required. For example, the cardiac silhouette maintains its regular shape, suggesting fluid overload/heart failure. Additionally, decreased lung lucency further indicates the presence of lung opacity.
We group "low lung volumes" and "rotated" together as well, as they share common finding: the cardiac silhouette is not located exactly in the center of the entire thorax. Therefore, further observa- tion of clavicle symmetry is needed to make the final judgment. If the clavicles are symmetrical, it suggests low lung volumes. Conversely, if asymmetry is observed, it indicates rotated, as the entire body appears asymmetrical in this case.
Table A5: Examples of CoT for cardiac silhouette.
attributes
CoT
enlarged cardiac silhouette
Find the cardiac silhouette <cot bbox 1>. Mcasure the maximal horizontal diameter of the cardiac silhouette <cot maximal horizontal diameter of the thorax _boxx 1>and the ma__mmergee_bl 1>. The cardiothoracie ratio more than 50%, indicating there is en- larged cardiac silhouette in the cardiac silhouette <cot boot__box____ 1>. Find the cardiac silhouette <cot_ <cot_bbox___________ee_ee 1>. Measure the maximal horizontal diameter of the cardiac silhouette ccootbbox________oooooo maximal horizontal diameter of the thorax <cot_merge___e sese__bbx_____ 1>. Observe the cardiac silhouette's 's <cot_ poo__bbx_IIshhape lung lucency <cot merge_ __be__bbo___________e__ee_ 1>. Considering all observations, there is lung opacity in the cardiac silhouette <cot _box 1>. So the answer is yes.
lung opacity
the
Observe the
low lung volumes
Find the cardiac silhouette ccot_bbox 1>. Observe the position of the cardiac silhouette <cot_bbox__ 1>within the entire thorax <cot ccco__mrge__a__ Lts__merr_eeo________ 1>. The cardiac silhouette <cot_ _oot_bbo______ 1>is displaced, and the left clavicle <cot_bbox_____ _b__x_ooxx_nddd right clavicle <cot_bbox abbae_ symmetrical. Therefore, there are low lung vol- umes in the cardiac silhouette <cot_ <cot_bbox_l>.
rotated
Find the cardiac silhouette coot_bbox__s 1>. Observe the position of the cardiac silhouette <cot_ _box__ox___ 1>within the entire thorax <cot merge bbox 1>. The cardiac silhouette <cot bbox 1>is displaced, and the left clavicle <cot <cot_bbox_______ <cot bbox _____bbooarrrr___e 17arre asymmetric. Therefore, there is rotated in the cardiac silhouette <cot_ bat__box_____ 1>.
right clavicle
mediastinum For the mediastinum, Table A6 presents three examples related tubes and lines to demonstrate the reasonable and clinically relevant reasoning process of AOR-VQA.
For the IJ line, it typically passes through the SVC and the cavoatrial junction (the recommended tip position for the IJ line). Here, we also emphasize observing the right atrium to ensure the catheter's position is correct and does not extend beyond the cavoatrial junction. Unlike the IJ line, the enteric tube is a medical device inserted through the nose or mouth into the digestive tract. Since the enteric tube traverses almost the entire chest X-ray, we cannot focus solely the sve and cavoatrial junction. Instead, we observe from the upper mediastinum to the cardiac silhouette. Finally, the endotracheal tube is flexible plastic tube inserted into the trachea through the mouth nose maintain an open airway. Thus, we primarily focus on the upper mediastinum for evaluation.
28

---

These reasoning methods for different tubes and lines enable the model to understand each tube/line's position and function effectively. This capability is reflected in our report generation performance (please refer to Fig. 5 in the main manuscript).
Table A6: Examples of CoT for mediastinum.
attributes line
CoT
Find the mediastinum <cot <cot_bbox __. Observe the sve <cot ccot_bbox___ 7>, the cavoatrial junction <cot
<cot_bbox bbox__boooo_____ 2s the right atrium <cot_ <cot_bbox bbox_ _3, and the mediastinum <cot_bbox_44 Considering all observations, there is IJ line in the mediastinum ccot_bbox 4>. So the answer is yes.
enteric tube
Find the mediastinum <cot _box @box_44>.> Observe the upper mediastinum <cot bbox_ 5>, the cardiac silhouette <cot bbox 1>, and the mediastinum <cot bbox_ 4>. Considering all observations, there is enteric tube in the mediastinum <cot bbox_4o 42>
<cot_bbox___
endotracheal tube
Find the mediastinum <cot_ [ar__box_444 4>. Observe the upper mediastinum <cot _bo__bbx_____ 5>, and there is endotracheal tube in the upper mediastinum <cot_ <cot_bbox boox_boo____ 5>. So the answer is yes.
left lung The left lung is a relatively large region, making direct judgment challenging. Thanks to Ontology 3, for specific attributes, we can focus certain sub-objects, As shown in A7 for breast/nipple shadows, we only need to locate the left upper lung zone; for pleural effusion, we specifically observe the left costophrenic angle.
Table A7: Examples of CoT for left lung.
attributes
CoT
breast/nipple shadows
Find the left lung <cot_ <cot_bbox___ _xx___ Observe the left upper lung zone <cot boox_bbxx___ 9>, and there breast/nipple shadows in the left upper lung zone <cot coot_bbox___ _obx_99>.o0 So the answer is yes.
pleural effusion
Find the left lung <cot_ crot_bbox_$s_ 8>. Observe the left upper lung zone <cot scot_bbox__e bbo the left mid lung zone ccot_bbox__lg 10>, the left lower lung zone <cot_ coot_bbox___ 11>, and the left lung <cot <cot_bbox bbob_bbooo__________________ee 8>, Pay special attention to the left costophrenic angle <cot <cot_bbox 20> Considering all observations, there is pleural effusion in the left lung xcot_bbox____ 8>
Table A8 presents examples of compositional questions. Compositional questions are broken down into sub-questions, reasoned step by step, and finally concluded with judgment.
Table A8: Examples of compositional questions. Question: Are there indications of both vascular calcification and mediastinal displacement in the upper mediastinum?
CoT answer: Find the upper mediastinum scot_bbox_5x For the vaseular calcification, observe the aortic arch scot_bbox_6x and there is vascular calcification in the aortic arch <cot_bbox b___bbox__>>____ po__6>, Therefore, there is vascular calcification in the upper mediastinum <cot_bbox
For the mediasti-
nal displacement, observe the position of the upper mediastinum <cot_ bbox 55sithiinnnn the entire thorax <cot merge bbox 1>. The upper mediastinum is mediastinal displacement in the upper mediastinum <cot bbox 5>, So the answer is yes.
bbox 5>is displaced, indicating there
Question: Concerning the left lung, which anatomical finding is involved, enlarged hilum or cyst/bullae?
CoT answer: Find the left lung ccot_bbox__t For the enlarged hilum, observe the left hilar structures <cot_ bbox 18>, and there is enlarged hilum in the left hilar structures <cot, bbox 18>, Therefore, there is no enlarged hilum in the left lung <cot bbox_ 8>. For the cyst/bullae, observe the left upper lung zone <cot bbox 9>, the left mid lung zone <cot bbox. 10>, the left lower lung zone <cot <cot_bbot bbox_bboooso_c 11>, and the left lung <cot_ coot_bbox_8s_ 8> Considering all observations, there is cyst/bullae in the left lung <cot bbox 8>, So the answer is cyst/bullae.
29

---

As shown in Fig. A5 of our multimodal CoT data.
we provide 8 examples from the AOR- VVQ to facilitate a better understanding
Opes the upper mediastioum present mediestinal widening?
Determine the presence mediast tnall widening by measuring the mediastinal- sesiiastinal-hhrreee retio.
Meazure the maximal
medlastinll-thoate ratio more then 33 indicating there widening the upper mediastinum [o 0:3). So the answer ye
diameter the upper mediastinum
Find the uppper mediastinum [0.4,
(o-d, o.6, o and maximel horizontal diameter of the thorax 3.8.0.9l.
incumamediastinum the upper mediastinum
Facus promiment sigrs preumomeddiastmumm lucent shedows.
Observe the left edge regions ous, oig and the right edge regions 9.4] the upper mediastinum (o.4, 6-3, o.5).
Find upper
mediastiuum
There is lucent shadows which indicate pneumo secummmeciastiumm astinum. So answer
[o.4, 6.2, o.6,
Q: Is there IJ line the <region>?
The line pesses through SVC and cavoetrial junction, ultimetely reeching the upper portion the right right atrium. Examine each these anatomical regions sequence. PICC, lI line, and Subciavion line follow simllrr paths.
Obsarved the 9.al,
Considering ring all obeservation, right atrium [o.d, 0.5, o.6, o.6) Therefore, there Uline mediastinum (o 0.5) So the answer yes.
The region is the mediestinum (0 0.5, 0.6)
cavoatrial cavoatrialjunction junction
9.4
right atrium 0.5. o.5,o.5) o.6).
Notel Althouh completely diffreett distinct ooTa answers must be provided.
are
purposes anatomical regions they traverse
mediastinal drain the <regions??
Observe the upper mediastinum (o-s, 6j, cardiac silhovette [o-4, o.g, 0.71, the mediestinum 0.6).
Considering
The region is the metinestiuum (0.4, 6.1,
obsenvations, there mediestinal
mediastinum (0:4, 0.1, p.9: 0.6), So yes.
special attention to areas where mass| nodule commonly appear: the left hilar structures.
nodule in the left lung?
"isserve the left upper lung
(ois: ou,, 0.9 olal the left mid
Considering obyervations, there is mass/hodule
lung zone (o.5 0.3, n.9, ois), the left lung zone [olS,
Find the left lung [o.5, o.g, 0.8).
0,71, and the left lung (o.5
the left lung (o.s. 0.4, o.8], So the answer yes.
left hilar special attention 0.4).
(o.5,
Pay special attention to areas www.rp effusioo commonly appean: e the left costophrenk angle. lung (o.s, 0.7, 0.31, left
lung present with pleural effusio??
Obierve the
Considering abservations, there mare!
lung zone (o.5, 0.3, 0.7, 0.41, the left lower lung zone (o.5, 0-4, o.7, 0.7l end the left lung [o.5, 0.2,
left lung (o.5, o.), o.6].
nodule in the left lung yes.
6), special attention the left costophrenic angle
o.6, 03, 0.7).
Low lung volumes vs rotated: Both condatios can lead to asymmetry between the left and right lungs, it's important further examine the symmetry of the left and right clavitles. the clavicies ies are symmetriccl, the cause is likely low lung volumes.
displey low lung
Observe e the left
Therofore, there
Find the lung [o.5, 0.8,
o.2, 9.4,
low lung volumes the left lung (o.6, o.o.
asymmetncc. Observe
clavicle (o.s o.g. o.3) and the cluicil (o. a.d, D-4, 0.s,, and they symmetrical.
o.8, o yes.
O: Are there Indications the left lung?
between and right further examine the symmetry the left and right clavicles, Il the clavidees symmetrical, causeis lieely low valumes.
lung volumes rotated: conditions
lungy?
Observe
0., o.8)
right lung (ols,
Find the left lung
o 0.4, 0.4.a.0l they are asymmetric Observe
Therefore, there is rotate in the left [o yes.
to.s, 0.2,
clevidle (o 0.0. 0.8 and
the right davicee (o.o, and asymmetric.
Figure A5: Samples from
30

---

A.2.2 AOR-RG
For region report generation in AOR-RG, we adopt ASG's strict alignment approach and further optimize it. The details are as follows: First, for a given chest X-ray, we extract anatomical regions using the bounding box (bbox) coordinates from Chest ImaGenome Dataset, i.e., the objects listed Table A2 Next, we utilize RadGraph to parse the corresponding paired reports. As shown in Table A9 the anatomical regions defined in RadGraph differ from those in Chest ImaGenome Dataset. To address this, we apply alignment rules provided by ASG ensure consistency. As illustrated in Fig. (b) of the main manuscript, this aligmment strategy guarantees one-to-one correspondence between anatomical regions and sentences. This prevents scenarios where, for instance, merely detecting one side of the lung would lead to incomplete statement like "The lung is..." without specifying the side.
Table A9: Objects in RadGraph.
Objects in RadGraph
trachea, left hilar, right hilar, hilar unspec, left pleural, right pleural, pleural unspec, heart size, heart border, left diaphragm, right diaphragm, diaphragm unspec, retrocardiac, lower left lobe, upper left lobe, lower right lobe middle right lobe, upper right lobe, left lower lung, left mid lung, left upper lung left apical lung, left lung unspec, right lower lung, right mid lung, right upper lung right apical lung, right lung unspec, lung apices, lung bases, left costophrenic right costophrenic, costophrenic un- spee, cardiophrenic sulcus, mediastinal, spine clavicle, rib, stomach, right atrium, right ventricle, aorta, sve, interstitium, parenchymal, cavoatrial junction, cardiopulmonary, pulmonary, lung volumes.
Additionally, we further optimize the alignment method to handle cases where two different anatomi- cal regions appear in the same short sentence, e.g., "Mediastinal and hilar contours are unremarkable. Using the previously mentioned method alone, this sentence would not be split to correspond sepa- rately to the mediastinal region and hilar region. To address this, we introduce a new splitting rule: short sentence describes two different anatomical regions simultancously. it is split into two separate sentences. Thus, the original sentence transformed into "Mediastinal is unremarkable." and "Hilar contours unremarkable." Based on this, we further refine the alignment by distinguishing left/right and specific anatomical regions.
31

---

B Experiment Details
B.1 Prompt Details
We provide the detailed prompts for the staee training, along with their inputs and outputs. Here, [anatomical regions list] includes the objects listed in [A2, and the responses highlighted red indicate those for which loss needs to be calculated during training.
Stage I: Anatomical Region Recognition
<image>In the conversation below, you are required to answer the category name based on what you in the imagery inside particular region. will give you only region each time. Categories containing (anatomical regions list).
<region> left lung <region> mediastinum
Stage 2:
Anatomical Region Grounding
<image>In the conversation below, you are required to answer the category name based on the given region. The region provided coordinate form and imagery form. The coordinate form is [1,, values correspond the top left æ, top left y, bottom right z, and bottom right y. Categories containing (anatomical regions list).
Task 1:
31, 22 y2l. with floating point number from 0 to 1. These
[0.50, 0.14, 0.89, 0.82] <region>left lung [0.33, 0.28, 0.75, 0.80] <region>mediastiimm
Task 2: <image>In the conversation below, you are required to locate the corresponding region of the given category name in the image, and output its coordinates in the form of [x), M1, I2, y2], with floating point number from 0 to 1. These values correspond to the top left æ, top left y, bottom right æ, and bottom right y. will give you only one category name each time.
left lung [0.50, 0. 14, 0.89, 0.82] mediastinum [0.33, 0.28, 0.75, 0.80]
Stage 3: Instruction Tuning
Task 1: <inage>provides an overview of the picture.
Question cot answer
Task <image>provides an overview the picture. Please generate radiology report based on this Chest X-ray.
Full image report
Task <image>provides an overview the picture. Please generate a corresponding radiology report based on this region of the Chest X-ray.
cregion>region report sentence
32

---

B.2 Training Details
We supplement the hyperparameters for each stage in Fig. B10
Table B10: Detailed training hyperparameters for AOR.
Configuration
Stage 1
Stage
Stage 3
Visual Encoder
CLIP VIT-L/14
CLIP ViT-L/14
CLIP ViT-L/I4
Model Init
VA-15-
Stage
Stage 2 128 2 x 10
Global batch size
128
128
Learning rate
2 x 10
2 10
Weight decay
0
0
Resolution
336
336
336
Training time
6h
9h
(-28h; r/t-43h
Epochs
2
2
Optimizer
AdamW
AdamW
AdamW
More Details About Region Encoder R
Following GPT4Rol
42
we introduce a region encoder
R to provide fine-graing greeggraddd,, multi-scale visual representations. The process includes:
(1) Feature Extraction:
Four feature maps 2, 17,20 23) extracted from the image
encoder and rescaled via bilinear interpolation to progressively larger scales ( T4: 14 14 1 forming multi-level feature pyramid {px)
4
(2) Pyramid Shuffle Fusion: At each pyramid level k, features from adjacent levels are resized to match the resolution level k; concatenated with current feature map, and passed through convolutional layers to generate the fused representation Pa-
Conv Concat (Resize(pe-1). Pk. Resize(pu+1))). k == 1,2,3,4.
(3) Region Feature Aggregation: Region features are extracted from {px 1i-121 via RolAlign, fused by convolutions, and pooled into a unified region representation for downstream reasoning tasks.
B.3 Evaluation Details
ROUGE-L ROUGE-L measures the length of the longest common subsequence (LCS) shared by the candidate and reference texts. It computes precision, recall, and F; based the LCS length. We use the Fi score of ROUGE-L to compare the generated radiology report with the ground truth report. For implementation. we use the evaluate library: https:///giii /ttp//gtttttttt/// com/huggingfacee Saaluatte/txee/maai//eettttt
BERTScore BERTScore neural metric that leverages pre-trained BERT model evaluate the semantic similarity between text pairs. It computes pairwise cosine similarities between the contextu- alized token embeddings of the candidate and reference texts, then aggregates these similarities into precision, recall, and Fj scores. In this work, we report the BERTScore F, measure to assess how closely the generated radiology reports match the ground truth reports. For implementation, the evaluate library: btttp:://githn com/ wwwm//uggingfaa/eeeeeeeeeeeeeeee htt.g//aeee/lllaattttee//e/a//// bertscore
F, CheXbert Chehbbert is computed using CheXbert, a Transformer- model trained to identify 14 chest X-ray abnormalities from radiology report. To evaluate the quality of generated reports, F, Xherrrt measures the F2 score between Cherthet's predicted labels on the generated report and those the ground truth report. Following previous work, the calculation focuses five specific observations: atelectasis, cardiomegaly, consolidation, edema, and pleural effusion. For implementation, use the ficheberrt library: https:://// //tg:////ttttttt/ wwwp:ooee//iiiiimm/e.eeeeee/ee
33

---

B.4 Result Details
B.4.1
Intermediate Results of Stage
As shown in Table BIT after completing Stage training, we conducted an initial evaluation of the model's anatomical region recognition capability. The average accuracy of anatomical region recognition is 98.37%, laying a foundation for subsequent training.
Table B11: Intermediate results of Stage 1.
Anatomical Region
Acc
Anatomical Region
Acc
left costophrenic angle
100.00
right hemidiaphragm
97.39
right clavicle
98.59
upper mediastinum
99.20
trachea
96.60
left hilar structures
98.80
abdomen
99.80
svc
98.80
right hilar structures
97.79
left mid lung zone
99.40
carina
99.60
cardiac silhouette
100.00
right atrium
99.00
left upper lung zone
97.58
left lung
99.60
left apical zone
96.98
left hemidiaphragm
93.60
right lower lung zone
97.20
left lower lung zone
95.20
right lung
100.00
mediastinum
99.60
spine
99.80
left clavicle
97.99
right costophrenic angle
100.00
right apical zone right mid lung zone
95.79
right upper lung zone
99.00
98.80
aortic arch
98.80
cavoatrial junction
97.80
B.4.2 Intermediate Results of Stage 2
Table B12
presents the intermediate results of Stage 2, validating the model's grounding capability. Additionally. ly, using ][ to represent coordinates achieved better results compared to the special token "<coor> </coor>".
Table B12: Intermediate results of Stage 2.
format
R@0.3
R@0.5
R@0.7
[1 <coor>
97.08
92.05
79.20
97.05
91.88
78.37
B.4.3
Results Details of Stage 3
Table B13 B
B15 display the subclass results of AOR and other comparative methods on MIMIC- CXR-VQA, helping us analyze which tasks AOR excels in. As discussed in the main manuscript, AOR demonstrates significant advantage in handling complex questions, such as choose and query types. By systematically analyzing multiple attributes involved in a question, the model is better equipped to perform logical reasoning and provide the correct answer. Additionally, for size-related questions, AOR improves answer accuracy by comparing the proportion of the heart or upper mediastinum relative to the entire thoracic cavity, leading to more precise evaluations.
34

---

Table B13: Detailed results of comparative experiments (verify).
verify size
Method
Res
presence
abnormality
plane
gender
total
general-domuine
LMM
LLaVA
2242
75.91
77.02
75.32
95.45
50.76
75.97
LLaVA-1.5
336²
74.66
74.96
73.62
96.97
85.61
75.25
GPT4Rol
224
76.02
77.59
77.31
95.45
93.18
77.16
VoCoT
4482
76.48
76,95
73.62
70.45
76.52
76.17
Medical-domain
LMM
LLaVA-Mee
224
75.32
75.53
74.47
94.70
78.79
75.71
Med-Flamingo
224
XrayGPT
224
61.21
57.94
58.87
53.03
53.03
60.00
CheXagent
448²
76.65
76.31
70.07
50.00
56.06
75.02
MedVLM-RI
Dyn
AOR(Ours)-t 336"
79.48
78.51
84.97
99.24
93.18
80.48
AOR(Ours)-rit
3362
79.02
79.01
93.76
98.49
93.94
81.17
Table B14: Detailed results comparative experiments (choose).
choose anatomy
Method
Res
attribute
abnormality
plane
gender
total
General-domane
LMM
LLaVA
224
52.72
51.63
49.66
93.18
60.61
56.07
LLaVA-1.5
3362
50.64
53.12
51.34
95,45
90.91
58.70
GPT4Rol
224
53.69
40.06
46.31
96.21
94.70
56.47
VoCoT
4482
43.91
37.39
36.91
93.18
83.33
48.79
Medical-domain
LMM
LLaVA-Med
224
50.32
51.04
52.01
95.45
91.67
58.31
Med-Flamingo
224
XrayGPT
224
29.97
40.95
43.62
78.79
49.24
40.97
CheXagent
448"
27.72
24.33
21.81
91.67
52.27
33.49
MedVLM-RI
Dyn
AOR(Ours)-t 336"
66.51
69.44
64.43
98,49
94.70
71.96
AOR(Ours)-r/t
3362
63.62
69.73
60.40
98.48
96.21
70.19
Table B15: Detailed results of comparative experiments (query).
Method
Res
query
attribute
abnormality
anatomy
plane
gender
total
General-domaie
LMM
LLaVA
2242
60.70
56.20
64.79
86.07
55.30
58.87
LLaVA-1.5
336
60.79
53.77
62.43
96.72
87.12
56.10
GPT4RoI
224
60.40
57.14
67.32
98.36
90.15
60.54
VoCoT
4482
61.22
55.38
62.80
97.54
81.82
60.70
Medical-domain
LMM
LLaVA-Med
224
61.48
57.19
65.98
97.54
90.15
60.37
Med-Flamingo
2242
XrayGPT
224
44.06
34.18
37.77
23.50
50.00
24.07
CheXagent
448"
37.42
34.09
46.55
68.85
28.03
48,49
MedVLM-RI
Dyn
AOR(Ours)-
336
66.71
62.69
69.43
98.36
91.67
65.05
AOR(Ours)-r/t
3362
66.95
62.75
71.18
100
93.18
65.49
35

---

B.5 Scalability of CoT Construction
Using VQA-RAD as an example, we generate CoT answers for each sample as follows: Keyword Mapping We use LLMs (GPT-4o) to extract relevant terms from each question and map them to our predefined 68 attributes, categories, and 38 objects. Unmapped terms are reviewed manually (though rare, as explained later). Sample Expansion Once mapped, we generate CoT answers based on our 2,812 predefined CoT types. Key findings:
Performance Gain Defining 2,812 CoT types enables LLMs to generate CoT answers for any CXR dataset. The LLMs can easily map attributes/categories and objects (c.g., the base of the right~>lower right lung) and proceed to construct CoT answer. As shown in Table incorporating CoT answers improves performance by 3.60% after I epoch and 5.97% after 3 epochs of fine-tuning, demonstrating the generalizability of our approach to new datasets.
Data Quality Enhancement We examine failure cases in mapping and identify two main points: (a) Non-specific anatomical regions, e.g., "the lungs" cover both sides. We resolve this by splitting the region into left and right sides for individual analysis and then aggregating the results. (b) Unmapped attributes, i.e., cardiomyopathy and aortopulmonary window, could not be mapped. Radiologist consultation confirmed these are challenging to assess via a single X-ray, as heart size and shape alone are insufficient. Similarly, the aortopulmonary window overlaps with adjacent structures chest X-rays. These findings validate the accuracy and clinical reliability of our 68 predefined attributes and help refine low-quality or non-specific samples in existing datasets.
36

---

Case Study
C.1
Case Study: Visual Question Answering (VQA) on MIMIC-CXR-VOA
Medical Visual Question Answer
Verify
Question: Does the cardiac silhouette display technical ussessments or devices?
Answer: Yes
LLaVA-Med:
XrayGPT: X
CheXagent:
0.82]. For technical assessment, observe position cardiac silhouette [0.33, 0,50, 0,80, 0.82] within the entire thorax 10.15,0 0..90.07..911...92 The cardinc silhouette
Find the cardiac
0.3330..0.0.8000.8.. is not displaced, indicating there no technical assessment in cartiac silhouctte (0.33,0.50,008000.33) For the device, observe the cavontrial junction [0.33,0.50,0 5.1.0.1111111 the right atrium [0.33.0.61...3) 0.500. and the cardiac silbouctte 333,00.000080000..0 Considering all observations. there is device in cardiac silbouette [0.33,0.50,0.800 80,0...
answer is yes.
Choose
Question: Which anatomical findings, enlarged hilum or mass/noduls (not otherwise spocified), is associated with left lung? Answers enlanged hilum
LLaVA-Med: nome x
Gru mone x
CheXagemr: Massinodule (not otherwise specified)
x
4OR: Find the left lung 1..07.010 88.0088..701 For the enlarged hilum, observe the left hilar structures
(0,47,0.30,0.65 0,049,,, and there no oalarged hilum in the left hilar structures 37,00.0,..65,0.999 Therefore, there otherwise specified), observe left upper lung zone 00.47.0.1..0. 0.799..331. the left mid lung zone 10.48 (1.33 33.0 (..46,, the left lower ling zone
cularged hilum
lung 88.00.00..88880....
[0.50.0.46.0 88.00.888...... 20), and the left lung 1...0.10,,.88,0000000 Pay special attention to hilar structures [0.47,0.30.0.65. 0..49) Considering all observations, there is mussinadule (not otherwise specified) in the left lung (0.47.0. 10.0088...0 So the answer calarged bilum.
Query
Question: Please common diseases found both the <region and <rogion 2>,
Answer: fluid heart failure
MaV:AMMed:
XrayGPT: -
CheXagont:
AOR:
The first region lelt lung (0.51.0.15 5.0.8444.... 84.0.72). Observe the left upper lung zone [0.51,011 66.00.2.033). the left mid lung 10.52,0.38.0 38.00.444.......00 the left lower lung
0.52.0.50.0 84,0.72), and the left lung
(0.51.0.15.0.84.....2 Considering all observations, discase in the left lung (0.51,0.15,0.844 15,00.0.. includes fluid everlond heart fiilure and pacumonia.
the rieht upner lung zone [D, 1,,00.1,,,,44,.010 383, the right mid lung zone (0. 10.13 10.13,,.388 0.38.0.42.0.511 11, the right lower lung zone
second region
lung [0. 10.1110.16.0
0.744.
[0.11.0.51.0444] 1...5.....1..0000 74j, and the right lung (0.11.0.16,0.. 0,106,044,00744 Considering all observations, discase in the right lung (0.11,0016.044440774) includes fluid overloud heart failure, So the answer fluid overlead heart failure.
Figure C6: Cases of VQA on MIMIC-CXR VQA: Leveraging region-level understanding and multi- step reasoning, AOR achieves superior performance.
37

---

C.2
Case Study: Visual Question Answering (VQA) on VQA-RAD and CheXpert
Medical Visual Question Answer
VQ4-RAD (Closed)
Question: Is the heart wider than more than half the width of the thorax? Answer: No.
4OR: Find heart [0.43, 0.40, 0.75, 0.65). Measure the maximal horizontal diameter of the cardiae silhouctte [0.43, 0.40, 0.75, 0.65] the maximal borizontal diameter the thorax 0,08, 0,85, 0.78). The eardiothoracic ratio no more than 50% there is no enlarged cardiac silhouctte in the cardise silhonetre [0.43, 0.40, 0.75, 0.65]. So answer is
VQ4-RAD (Open)
Question: Which lungs are hyperinflated? Answer: Bilateral lungs.
lung right lung.
CheXpert (Cloxed)
Question: Does the X-ma reveal any lung cancer? Answer:
CheXpert (Open)
Question: What abnoemalitics you
Enlarged cardinc silhouctte and odema.
AOR: Enlarged cardiac silbouette and edema.
Figure C7: Cases of VQA on VQA-RAD and CheXpert: For local-level questions, even on unseen VQA-RAD data, AOR can perform reasoning and provide correct answers. For global-level questions, AOR is also capable of synthesizing information to make final judgment.
38

---

C.3
Case Study: Full-Image Report Generation on MIMIC-CXR
Image Report
GroundTrulk: AP upright and lateral views of the chest were provided. A Port-A-Cath resides over the right chest wall with catheter tip in expected region of mid SVC. Lung volumes are low. No effusion or pncumothoras. No convincing evidence for pneumonia CHF. The cardiomadiastinal silhouctte norma. Bony structures intact, free air below the right bemidiaphragm. No acute findings.
4OR: Right-sided Porl-A-Cath tip torminates in the mid SVC. Heart size is normal. Mediastinal and bilar contours are unremarkable. Pulmonary vasculature is normal, Lang volumes are low: No focal consolidation, pleural effissonn or abnormality visualized. No eardiopulmonnary abnormality.
pncumothorax present. No
LLatA-MMd:: The chest X-ra appears to be normal, which means that there are visible signs of ahuormalities or issues the lungs, beart, or surrounding stractures. This positive finding, it suggests that the paticut's
Med-flamingo:
patient a history of cough and lever. The patient supine position. The cardiac silhoucttc is normal. The lungs are clear. The trachea is central. The hila are normal. The lung fields are clear. The mediastinum normal. The ribs are nommal. The bony thorax nommal.
radiograph a 55-year-old
GreyGPT: The chest shows normal heart size, unsemarkable mediastinal and hilar contours, and clear lungs without any focal cardiopulmonaary process.
consolidation. There plzural cifusion or pacumethorax. The impression there
CheNagent: In comparison with the study there is little change and no evidence of acute candiopulmonary disease No pneumonia. vascular congeation, or pleural effusion. Central cotbeter remains in place.
GroundTruth Truth: Heart size is normal. The mediastinal and halar centours are normal. The pulmonary vasculature is normal. Lungs
pleural preumathorux seen. are no acute osseous abmornalities. Moderate degenerative changes demonstrated within the imaged thoracic spine. No acute cardiepulmonary abnormality.
lungs
cuntiomediastima
within normal limits, acute osseous abnormaliries,
No acute cardiopulnonary process.
M.VVA-MMd: The chest X-my appears to be normal, which means that there are no visible signs of ahnormalities or ismes in the lungs, heart, surrounding structures.
Med-flominga: This is frontal chest radiograph of 55-year-old male patient with history of cough and fever. The paticnt is in supine position. The cardinc silhouctte normal. The lungs clear, The trachaa central. The hila nommal The lung fields clear. mediastinum normal. bones normal
GrayGPT: The chest x-my shows clear lungs with no signs of consolidation, effusion, or edema. The cardiomedinstiall silhouctte is normal, and there are no acute osseous ahnormalities. The impression is that there no acuse cardiopulmonary process.
CheXagent: Heart
normal, The mediastinal hilar pleural effusion or preumothorax seen.
normal. The pulmonary vasculature nommal. Longs elear.
Figure C8: Cases of full-image report generation on MIMIC-CXR: AOR demonstrates strong detail- capturing capabilities.
39

---

C.4
Case Study: Region-Level Report Generation on MIMIC-CXR
Region Report Generation
Please generatie a corresponding radiology report based on left lawerr lang some the Chert GroundTruth: There mild stelectasis.
AOR:
There is mild left basilar atcloctasis.
The image chest X-my of 32-year-old male, taken one month after a motoreycle accident. The left lower zone shows homogeneous opacity, which
of increased density the lung.
Med-flamingo: left
zone.
upper zone
middle
zone.
The chest x-my shows nomnal heart size, with normal mediastinal and hilar contours, The pulmonary vasculature is also normal, and the lungs are clear. There are no signs of pleural effusion or preumothorax. The impression that there is no acute candiopulmonary abnommlity..
XrTyGPT:
CheXagent: In comparison with the study of there little change evidence cardiopulmonary discasc. No poeamooni, congestion, or pleural effusion.
Pleuse generate a corresponingg radiology report brsed on sve of the Ches X-ray GroundTruth: The picc now in the mid svc. The picc has been pulled back und within the mid
AOR: Right pice line tip the level of mid sve.
visible signs d'aabemmaliiics Med-flamingo: The patient 60-year-old male with history of hyperiension and dinbetes. He currently being treated for right lower lebe pucumonia. The patient is currently ou a beta-blockar for his hypeteasion..
LLa KA-Medd: chest X-ray appears normal, which the lungs. beart, or sumounding structures.
XruyGPT: The chest x-ray shows normal heart size, normal mediastinal and hilar contours, and clear lungs. There evidence of ploural elfusion or pacumothorax. The impression that
cardiopulnoeary process.
CheNagent: In comparison with the study the patient has taken a much better inspiration. There is no evidence of'acute pneumoniaa. vascular congestion,, or pleural effusion.
Figure C9: Cases of region report generation on MIMIC-CXR: AOR can generate accurate, grounded captions.
40

---

D Limitations and Future Work
Frontal and Lateral Images AOR only utilizes frontal images for experiments, while lateral images also play a significant role clinical practice. However, it is worth noting that the of lateral images is far more complex than simply encoding and concatenating them with frontal images. Lateral images inherently provide rich CoT reasoning. For example, shown in Fig. C8 when only the frontal image is provided as input, all medical LMMs fail to detect spine degenerative change, which further validates the importance of lateral images certain cases. Similarly, when pulmonary abnormalities obscure the hilum in frontal images, lateral images are essential for further observation and confirmation. Leveraging lateral images will be an important direction for our future work.
Currently, AOR only analyzes static CXRs. However, real-world scenarios, diseases progress time, and CXRs often presented sequence captured at different time points. Therefore, the ability to understand temporal images is also crucial, which we hope to explore further in future work.
Temporal Images
Extension to More Modalities AOR focuses on CXR interpretation, providing anatomical ontology-guiin -guided reasoning capabilities. Based on this, discuss the potential of extending AOR to additional modalities: (1) Extension to modalities with clearly defined anatomical regions. For modalities such as chest CT, abdominal CT, and brain MRI, which have clearly defined anatomical regions, we can use these existing divisions as anchors to construct corresponding CoT answers, thereby mimicking radiologists' reasoning processes in clinical decision- making. (2) Extension to modalities without explicitly defined anatomical regions. We also explore structures such as bones teeth, which do not have conventional anatomical zoning. We consulted certified radiologists design reasoning flows based on their unique structural features. For example, in bone structures, reasoning be organized around specific vertebrae ribs; in tooth structures, it can be decomposed into crown, root, jaw, condyle, airway, etc.
E Impact Statement
For our research, we utilize the source dataset under the PhysioNet license, ensuring compliance with the required credentials and permissions, and eliminating the risk of privacy infringement. By empowering Medical LMMs with anatomy-centrie reasoning capabilities, we offer new paradigm for interactive and explainable LMMs medical imaging analysis. Experiments demonstrate AOR's superior performance in both VQA and report generation tasks, revealing its potential in supporting clinical decision-makinn making.
41


Reviews:
[{"review_id": "iAR1yYlVDw", "reviewer": "Reviewer_CaAU", "summary": "To tackle the problem of insufficient region-level understanding and interaction, as well as limited accuracy and interpretability due to single-step prediction in MLMMs for chest X-ray interpretation, this paper proposes an AOR framework. The AOR framework enhances MLMMs with anatomy-centric reasoning capabilities, enabling multimodal, multi-step reasoning through region-level information. Additionally, the paper develops the AOR-Instruction dataset, which includes anatomical regions and their ontologies, to support training MLMMs for improved accuracy and interpretability in  VQA and report generation tasks.", "questions": "Please carefully address Cons 1. My evaluation score would drop if more recent methods is not included and compared.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n1.The paper introduces a novel AOR framework that enhances MLMMs with anatomy-centric reasoning for improved medical imaging analysis.\n2.The methodology and experimental setup are clearly described. Illustrations in supplementary are well put.\n3.The authors validate their approach using extensive datasets, demonstrating the effectiveness and robustness of the AOR framework.\n\nCons:\n1.My main concern is the baselines are not up-to-date. The work below, published in 1st April this year, proposed LLaVa-Rad that brings dramatic improvements over LLaVa-Med. The reported performances may surpass the current work. Though the current method has merits of its own, it’s more complete to include LLaVa-Rad in baselines. (Zambrano Chaves J M, Huang S C, Xu Y, et al. A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings[J]. Nature Communications, 2025, 16(1): 3108.)\n\nMinors:\n1.Line 6, 7: “Insufficient” and \"Limited\" should not be capitalized?\n2.Line 35: A comma is needed before \"but\" to separate the clauses.", "was_revised": true, "final_justification": "My concerns have been resoloved and I will keep my score.", "rebuttal": "We are grateful for your valuable feedback and suggestions. We hope that the following responses address your concerns.\n\n\n### **W1: Include more up-to-date baselines.**\n\nIn response to the reviewer’s concern, we have additionally compared three representative up-to-date baselines on VQA, full image report generation, and region report generation tasks:\n\n- **Qwen2.5-VL** [1], a recent general-domain large multimodal model,\n- **MedRegA** [2], a region-aware bilingual medical large multimodal model, and\n- **LLaVA-Rad** [3], a model specifically designed for radiology report generation.\n\n**VQA Performance**: As shown in Table 1, our model outperforms Qwen2.5-VL by an average of 8.06% and MedRegA by 4.50% on the VQA task. These improvements demonstrate the strength of our carefully designed multimodal reasoning framework, particularly in lesion recognition and localization.\n\n> ***Note***:\n> - LLaVA-Rad was exclusively trained for the report generation task and lacks VQA capabilities; therefore, it is not included in this comparison. \n> - Due to the large size of MedRegA (40B), we are currently unable to provide its fine-tuned results on the MIMIC-CXR-VQA dataset within a short timeframe. We will include these results in the revised version.\n\nTable 1. Performance on VQA. Results in *italics* indicate that the training data of MedRegA includes VQA-RAD.\n| Method     | Res | Params |           | MIMIC-CXR-VQA |       |   |   | VQA-RAD |           |   |   | CheXpert |       |\n|------------|:---:|:------:|:-----------:|:-------------:|:-------:|---|---|---------|-----------|---|---|----------|-------|\n|            |     |        | verify    |     choose    | query |   |   | closed  | open      |   |   | closed   | open  |\n| Qwen2.5-VL | 336 |   7B   | 75.18     |     56.66     | 61.46 |   |   | 61.64   | 23.22     |   |   | 61.78    | 37.20 |\n| MedRegA    | 448 |   40B  | -         |       -       | -     |   |   | ***66.46***   | *27.82*     |   |   | 71.49    | 32.85 |\n| Ours       | 336 |   7B   | **80.48** |   **71.96**   | **65.05** |   |   | 63.01   | **28.19** |   |   | **71.58**    | **53.85** |\n\n**Full Image Report Generation**: As shown in Table 2, our method achieves an average performance gain of **12.18%** over Qwen2.5-VL, **4.47%** over MedRegA, and **0.58%** over LLaVA-Rad, when evaluating fairly at the same input resolution of 336. The performance of our model in R-L and BERTScore even comes close to that of the LLaVA-Rad model using a 518 resolution, and increasing the resolution of our model would be a promising future direction.\n\n>***Note***: The reported metrics differ slightly from those in the original LLaVA-Rad paper, mainly due to two reasons: (1) LLaVA-Rad evaluates on the full test split of MIMIC-CXR, containing 2461 image-report pairs, whereas our evaluation is based on 500 image-report pairs from the MIMIC-CXR-VQA test set which is a gold standard dataset manually validated and corrected by clinicians; (2) LLaVA-Rad focuses on generating the \"findings\" section, while our ground truth includes both the \"findings\" and the \"impression\" sections, the latter being an  essential component of a complete radiology report. These factors contribute to the observed discrepancy in reported performance. \n\nTable 2. Performance on full image report generation.\n| Method      | Res | Params |  R-L  | BERTScore | F1CheXbert |\n|-------------|:---:|:------:|:-----:|:--------:|:----------:|\n| Qwen2.5-VL | 336 |   7B   |  12.80 |   76.34  |    30.15   |\n| MedRegA     | 448 |   40B  | 21.78 |   83.29  |    37.33   |\n| LLaVA-Rad   | 518 |   7B   | **25.68** |   **84.94**  |    **50.52**   |\n| LLaVA-Rad   | 336 |   7B   | 24.22 |   83.67  |    46.20   |\n| Ours        | 336 |   7B   | 25.37 |   83.92  |    46.53   |\n\n**Region Report Generation**: As shown in Table 3, our model demonstrates strong interpretability and fine-grained capability in generating region-specific descriptions in response to user-provided textual and visual prompts. Compared to existing  baselines, it achieves substantial improvements in this task.\n\nTable 3. Performance on region report generation.\n| Method      | Res | Params |    R-L    |  BERTScore | F1CheXbert |\n|-------------|:---:|:------:|:---------:|:---------:|:----------:|\n| Qwen2.5-VL | 336 |   7B   |   11.07   |   78.81   |    12.26   |\n| MedRegA     | 448 |   40B  |   16.93   |   80.85   |    26.33   |\n| LLaVA-Rad   | 518 |   7B   |   18.77   |   81.90   |    33.51   |\n| LLaVA-Rad   | 336 |   7B   |   18.71   |   81.80   |    31.40   |\n| Ours        | 336 |   7B   | **35.11** | **84.54** |  **36.65** |\n\nThese up-to-date baselines will be incorporated into the revised manuscript to strengthen the evaluation. \n\nWe appreciate your attention to detail. “Insufficient” and “limited” will be decapitalized as suggested. We will also add the appropriate comma before “but” on Line 35.\n\n---\n\n[1] Bai, S., Chen, K., Liu, X., et al. *Qwen2.5-VL Technical Report*. arXiv preprint, 2025.\n\n[2] Wang L, Wang H, Yang H, et al. *Interpretable bilingual multimodal large language model for diverse biomedical tasks.* ICLR 2025.\n\n[3] Chaves, J. M. Z., Huang, S. C., Xu, Y., et al. *Towards a Clinically Accessible Radiology Foundation Model: Open-Access and Lightweight, with Automated Evaluation*. Nature Communications, 2025."}, {"review_id": "QimIkYlSe0", "reviewer": "Reviewer_3vmJ", "summary": "This paper focuses on enhancing the interpretability and accuracy of medical large multimodal models (MLMMs) in chest X-ray (CXR) interpretation by introducing an anatomy-centric reasoning framework named AOR (Anatomical Ontology-guided Reasoning). The authors identify two key limitations of existing MLMMs (i.e., insufficient region-level understanding and the lack of multi-step reasoning) and address them via a three-stage training framework that incorporates anatomical ontology and Chain-of-Thought (CoT) prompting. To support this, they construct a large-scale instruction dataset, AOR-Instruction, including both visual question answering (AOR-VQA) and report generation (AOR-RG) tasks with fine-grained region–sentence alignments. Experiments across multiple benchmarks demonstrate that AOR achieves promising performance on both VQA and report generation tasks.", "questions": "Please refer to the weaknesses above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The proposed AOR framework integrates anatomical ontologies and region-level reasoning into medical multimodal models. This design closely mimics the diagnostic process of radiologists and enhances both interpretability and clinical relevance.\n2. The authors construct a large-scale and clinically grounded instruction dataset (AOR-Instruction), including both visual question answering (AOR-VQA) and region-level report generation (AOR-RG) components. The dataset is guided by experts and offers fine-grained region–sentence alignment, making it a valuable resource for training and benchmarking future medical MLMMs.\n\nWeaknesses\n1. The paper lacks descriptions for the loss functions used in the three training stages, making it difficult to evaluate how each stage is optimized or how learning objectives are balanced.\n2. From the provided code, it appears that the three-stage training is conducted separately, which may lead to error accumulation across stages. Since each stage is trained in isolation, suboptimal performance in earlier stages (e.g., incorrect region recognition) can propagate to downstream tasks like grounding and reasoning, potentially affecting overall performance and robustness. It remains unclear how the proposed framework mitigates this issue or ensures stability across stages.\n3. In Table 1, AOR(Ours)-t refers to the model trained with text-only prompts, while AOR(Ours)-r/t uses both region (visual) and text prompts. However, it is unclear why in some cases the performance of AOR(Ours)-r/t is worse than that of AOR(Ours)-t. Intuitively, incorporating region-level visual information should provide additional cues for reasoning and improve performance. The paper does not provide an explanation for this performance drop.\n4. In Table 3, the performance differences among the variants (especially between ID 2, 3, and 4) are relatively small and inconsistent across question types. Given these marginal gains, it is difficult to assess the true effectiveness of each representation strategy. Providing additional results on other datasets or tasks would help better demonstrate the advantages of the proposed CoT design.\n5. In Table 5, Stages 1 and 2 are designed to improve region recognition and grounding, respectively. However, their impact on the downstream referring and grounding performance appears limited (i.e., adding these stages leads to only marginal gains) and in some cases, even slight drops (R@0.3: 98.48 -> 98.31). This raises questions about the effectiveness of the pretraining stages and whether their benefits consistently transfer to the final reasoning tasks. Further analysis or ablation would help clarify their contribution.\n6. It would be helpful to clarify whether the proposed dataset will be publicly released, as it is also claimed as one of the key contributions of the paper.", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns, and I will be keeping my score.", "rebuttal": "Thank you for your valuable comments. Below are our responses to your specific points and suggestions.\n\n### **W1 & W2 & W5: More Details and ablation studies on our training strategy.**\n\n**Loss functions**: All three training stages use cross-entropy loss for auto-regressive language modeling.\n\n- Stage 1: The model is given an image and region features as input and trained to generate the corresponding anatomical [region name] (Loss calculation).\n  \n- Stage 2 - Task 1: Given an image, region coordinates, and region features, the model is trained to predict the [region name] (Loss calculation).\n  \n- Stage 2 - Task 2: The model is provided with an image and a region name as input and is trained to generate the corresponding [bounding box coordinates] (Loss calculation).\n  \n- Stage 3: The model is fine-tuned on instruction-based tasks using the AOR-Instruction dataset, with outputs such as [CoT answers or reports] (Loss calculation).\n\nWe will include a more detailed description of these loss functions in the revised manuscript and refer readers to Appendix B.1 for further clarification.\n\n**Three-stage training paradigm and knowledge retention**: Our training framework follows a progressive learning strategy, similar to the mainstream training paradigms of current LMMs involving referring or grounding (e.g., GPT4RoI). Specifically:\n\n- Stage 1 improves anatomical region recognition.\n- Stage 2 jointly optimizes anatomical region recognition and localization.\n- Stage 3 benefits from the prior stages and further enhances the model's region recognition and localization capabilities while optimizing reasoning.\n\nAlthough the stages are not jointly optimized, knowledge is preserved via model checkpoints and parameter initialization. This mitigates error accumulation, as the pretrained representations are progressively refined rather than discarded or overwritten.\n\n**Effectiveness of Our Training Strategy**:  As shown in Table 5 of the manuscript, the inclusion of Stage 1 and Stage 2 leads to an average improvement of 2.15% on the VQA task. \n\nTo further validate the effectiveness of our training strategy, we conduct an ablation study on the VQA-RAD dataset under the zero-shot setting. As shown in Table 1, VQA accuracy increased by 3.74%, while referring and grounding accuracies improved by an average of 5.46% and 4.31%, respectively. The inclusion of Stage 1 and Stage 2 enhances the model’s stability and robustness and yields relatively more significant gains on the out-of-distribution (OOD) test set.\n\n\n**Further analysis of the Referring and Grounding results can be conducted from the following two perspectives**:\n- Compared to MIMIC-CXR-VQA, both Referring and Grounding performance metrics decline on the VQA-RAD dataset. This is primarily because VQA-RAD serves as an out-of-distribution (OOD) dataset, where some questions involve anatomical regions not included in our predefined set of 29 zones. These regions, such as the aortopulmonary window (an anatomical gap between the aortic arch and pulmonary artery), are rarely encountered in routine clinical practice and are more clearly visualized in chest CT or contrast-enhanced imaging, rather than on standard chest X-rays, where they are not a primary focus.\n\n- On VQA-RAD, the inclusion of Stage 1 and Stage 2 training leads to more noticeable gains. This can be attributed to the fact that the training data used in Stage 3 is skewed toward anatomically frequent and diagnostically dominant regions in chest X-rays, such as the lungs and heart. In contrast, VQA-RAD often includes regions that appeared less frequently in our Stage 3 training data, e.g.,  the trachea. According to our statistics from a collaborating hospital, the incidence of tracheal abnormalities in chest X-rays from 2018 to 2022 was less than 0.01%. In Stage 1 and Stage 2, however, the distribution of anatomical regions is balanced, allowing the model to learn to better recognize and localize underrepresented regions.\n\nTable 1. Effectiveness of our training strategy on VQA-RAD (zero-shot setting).\n| **Strategy** |             | **VQA-RAD** |           |   |   | **Referring** |   |   |           | **Grounding** |           |\n|:--------------:|:-------------:|-------------|-----------|---|---|---------------|---|---|:---------:|:-------------:|:---------:|\n| **Stage 1**  | **Stage 2** | **closed**  | **open**  |   |   | **Acc**       |   |   | **R@0.3** |   **R@0.5**   | **R@0.7** |\n| **✗**        | **✗**       | 54.00       | 21.04     |   |   | 87.27         |   |   |   82.14   |     76.79     |   69.64   |\n| **✓**        | **✗**       | 56.16       | 22.53     |   |   | 89.09         |   |   |   84.21   |     78.18     |   70.90   |\n| **✓**        | **✓**       | **57.53**   | **24.99** |   |   | **92.73**     |   |   | **86.79** |   **81.13**   | **73.58** |\n\nThese results confirm that the early-stage training benefits generalization and performance in later reasoning tasks. We will revise the manuscript to clarify these findings and their implications.\n\n### **W3: Performance Comparison Between AOR-t and AOR-r/t**.\n\nThank you for pointing out this. The setting of AOR(Ours)-r/t is designed to demonstrate our model’s flexibility in handling **multimodal input formats** during inference, specifically, both textual and visual prompts (e.g., bounding boxes drawn directly on the image). While this reflects a more general and practical usage scenario, it also introduces additional complexity, as the model must dynamically parse and reason over **heterogeneous input modalities**. The slight performance drop observed in AOR-r/t is not indicative of a failure but rather a result of the increased input diversity and interpretive burden placed on the model. We will clarify this distinction and the implications in the revised manuscript.\n\nIn addition, it’s important to note that both AOR-t and AOR-r/t utilize region-level visual information to enhance reasoning, consistently reasoning with regions throughout the whole process. For more details, please refer to Figure C6 in Appendix.\n\n\n### **W4: Analysis and Additional Results on Different CoT Representations**.\n\n**Inconsistency Across Question Types**: The observed performance \"inconsistency\" is primarily due to differences in **question difficulty** across types. As described in Section 5.1 of the manuscript:\n\n- **Verify-type questions** require only a binary yes/no response. Even without fully understanding the question, the model has a 50% chance of answering correctly by guessing.\n- **Choose-type questions** are evaluated strictly: selecting one incorrect option or omitting a correct one results in the entire answer being marked incorrect. These questions require the model to reason about multiple lesions and anatomical regions.\n- **Query-type questions** also demand precise generation of comprehensive answers. A correct response necessitates full and accurate understanding of the image and question.\n\nTherefore, the benefits of coherent, logically structured reasoning are more evident in the more complex **choose** and **query** tasks.\n\n**Performance Across Model Variants (IDs)**: We elaborate from the following three perspectives:\n\n- **Further analysis of Table 3 in the manuscript**: **ID1 → ID2 (Effect of CoT)**: As shown in Table 3 of the manuscript, incorporating our CoT strategy significantly improves performance across all question types. This early gain establishes a strong baseline, making subsequent improvements more challenging. **ID2 → ID4 (Effect of Spatial and Visual Cues)**: Despite the strong baseline, introducing spatial and visual cues in later stages still yields additional performance gains, with a 2.64% improvement on choose questions and a 2.03% improvement on query questions. This highlights the added value of spatial and visual information to fine-grained reasoning tasks.\n\n- **Zero-shot Evaluation on VQA-RAD**: We conduct an additional ablation on the VQA-RAD dataset under the zero-shot setting. As shown in Table 2, the inclusion of spatial and visual cues improves accuracy by 5.18% on closed-type questions and 5.79% on open-type questions. These gains further confirm the model’s enhanced reasoning and generalization under unseen distributions.\n\n- **Interpretability and Practical Value**: Finally, we emphasize that our final model (ID-4) not only achieves the highest answer accuracy, but also offers enhanced interpretability and practical value by visualizing the approximate lesion regions attended to during reasoning, providing valuable insights in clinical scenarios.\n\nTable 2. Comparison of different CoT representations on VQA-RAD. \n| Exp_ID |  coor | region |  CoT  |   closed  |    open   |\n|:------:|:-----:|:------:|:-----:|:---------:|:---------:|\n|    1   | **✗** |  **✗** | **✗** |   47.74   |   16.90   |\n|    2   | **✗** |  **✗** | **✓** |   52.05   |   19.20   |\n|    3   | **✓** |  **✗** | **✓** |   54.50   |   21.91   |\n|    4   | **✓** |  **✓** | **✓** | **57.53** | **24.99** |\n\n### **W6: Dataset Release Plan.**\nWe confirm that the proposed dataset **will be publicly released**. We are currently organizing the files and documentation and will upload the dataset to **PhysioNet** as soon as possible."}, {"review_id": "oaOhBIf17V", "reviewer": "Reviewer_SRND", "summary": "This paper introduces the Anatomical Ontology-Guided Reasoning (AOR) framework, which leverages multi-modal region-level anatomical correlation for step-by-step reasoning. Two datasets with multi-modal CoT answers are constructed to train AOR for both question-answering and report-generation. The proposed AOR shows strong results across several benchmarks.", "questions": "1. Could other models except the AOR model, such as LLava-med, gain performance enhancement by supervised fine-tuning with CoT data from these datasets?\n\n2. Is it necessary to employ an independent encoder to extract local information? Additionally, the author should offer more details about the region encoder.\n\n3. Will the trained AOR model generate inaccurate reasoning steps, such as incorrect identification of anatomical regions and their attributes?", "limitations": "The proposed method only focuses on Chest X-Ray; the generalization capability across different tissues and imaging modalities of the proposed data generation pipeline and AOR architecture is unknown.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- This paper proposes a new medical dataset with CoT reasoning via exploring objects' relationships. For medical VLLMs, it is hard to generate CoT reasoning by distilling a stronger reasoning model, as there are no models that are strong enough to consistently provide reliable reasoning.\n- The proposed data generation pipeline is sound and effective. Performing step-by-step reasoning by analyzing the correlation across anatomical regions facilitates the generation of more interpretable answers for medical QA and report generation tasks.\n- The paper is well-written; the author provided a detailed description of the proposed methodology. Extensive experiments are conducted to demonstrate the effectiveness of the AOR model.\n\n\n**Weaknesses:**\n- Lacks evaluating how the newly introduced AOR-QA and AOR-RG datasets impact the performance of other medical multi-modal models.\n- Lacks the ablation on the region encoder; it is unclear whether the performance improvement in the AOR model is caused by higher data quality or a specifically designed model architecture.", "was_revised": true, "final_justification": "After reading the authors' rebuttal, most of my concern have been resoloved. I will keep my positive score.", "rebuttal": "Thank you for your recognition and valuable suggestions. We hope the following point-by-point responses address your concerns.\n### **W1 & Q1: Effectiveness of AOR-VQA and AOR-RG in enhancing other LMMs.**\nTo evaluate the impact of our proposed  AOR-VQA and AOR-RG datasets on other multimodal models, we fine-tuned **LLaVA-Med** and **GPT4RoI** using these datasets. \n\nAs shown in Table 1 and Table 2: \n- **ID-1** refers to models trained on MIMIC-CXR-VQA (w/o CoT) and MIMIC-CXR (report generation with full images only). \n- **ID-2** denotes models fine-tuned with  our AOR-VQA dataset (w/ CoT) and AOR-RG (including both full-image and region-level report generation). \n\nThe results are as follows: \n- **LLaVA-Med** achieves a +3.20% improvement on VQA and +5.81% on report generation. \n- **GPT4RoI** gains +4.62% and +6.69%, respectively. \n\nThese results indicate the effectiveness of AOR-VQA and AOR-RG for other LMMs.\n\nTable 1. Performance on the VQA task.\n| Model     | Exp_ID |              | MIMIC-CXR-VQA |              |   |   | VQA-RAD      |              |   |   | CheXpert     |              |\n|-----------|:--------:|--------------|---------------|--------------|---|---|--------------|--------------|---|---|--------------|--------------|\n|           |        | verify       | choose        | query        |   |   | closed       | open         |   |   | closed       | open         |\n| LLaVA-Med | 1      | 76.34        | 57.58         | 60.76        |   |   | 48.09        | 23.05        |   |   | 58.02        | 39.50        |\n|           | 2      | **77.57**(+1.23) | **66.97**(+0.39)  | **62.17**(+1.41) |   |   | **50.56**(+1.47) | **26.22**(+3.17) |   |   | **60.79**(+2.77) | **42.48**(+2.98) |\n| GPT4ROI   | 1      | 76.17        | 56.34         | 60.39        |   |   | 45.59        | 16.49        |   |   | 54.55        | 35.66        |\n|           | 2      | **78.37**(+2.20)x | **69.14**(+12.8)  | **62.99**(+2.60) |   |   | **49.63**(+4.04) | **19.94**(+3.45) |   |   | **59.41**(+4.86) | **38.07**(+2.41) |\n\nTable 2. Performance on full image report generation.\n| Method    | Exp_ID | R-L          |   |   | BERTScore    |   |   | F1CheXbert   |\n|-----------|:------:|--------------|---|---|--------------|---|---|--------------|\n| LLaVA-Med |    1   | 23.68        |   |   | 83.13        |   |   | 37.60        |\n|           |    2   | **25.89**(+2.21) |   |   | **83.29**(+0.16) |   |   | **40.51**(+2.91) |\n| GPT4ROI   |    1   | 23.76        |   |   | 83.23        |   |   | 39.45        |\n|           |    2   | **25.13**(+1.37) |   |   | **83.86**(+0.63) |   |   | **44.11**(+4.66) |\n\nTable 3. Performance on region report generation.\n| Method    | Exp_ID | R-L           |   |   | BERTScore    |   |   | F1CheXbert    |\n|-----------|:------:|---------------|---|---|--------------|---|---|---------------|\n| LLaVA-Med |    1   | 17.24         |   |   | 80.76        |   |   | 24.07         |\n|           |    2   | **34.26**(+17.02) |   |   | **82.76**(+2.00) |   |   | **34.62**(+10.55) |\n| GPT4ROI   |    1   | 13.50         |   |   | 78.58        |   |   | 26.66         |\n|           |    2   | **33.82**(+20.32) |   |   | **82.96**(+4.38) |   |   | **35.42**(+8.78)  |/\n\n### **W2 & Q2: More details and ablation study on the region encoder.**\n**More details.** Our region encoder is designed to provide fine-grained, multi-scale visual representations. The process includes:\n\n1. **Feature Extraction**: Four feature maps $z_j(j = 14, 17, 20, 23)$ are extracted from the image encoder and rescaled via bilinear interpolation to progressively larger scales $(\\frac{1}{14}, \\frac{2}{14}, \\frac{4}{14}, \\frac{8}{14})$, forming a multi-level feature pyramid {$p_k$}$_{k=1}^4$.\n2. **Pyramid Shuffle Fusion**: At each pyramid level $k$, features from adjacent levels are resized to match the resolution of level $k$, concatenated with the current feature map, and passed through convolutional layers to generate the fused representation $\\hat{p}_k$.\n$$\n\\hat{p}_k=\\mathrm{Conv}(\\mathrm{Concat}(\\mathrm{Resize}(p\\_{k-1}), \\ p\\_k ,\\ \\mathrm{Resize}(p\\_{k+1}))), \\ k = 1, 2, 3, 4\n$$\n3. **Region Feature Aggregation**: Region features are extracted from {$\\hat p_k$}$_{k=1}^4$ via RoIAlign, fused by convolutions, and pooled into a unified region representation for downstream reasoning tasks.\n\n\n**Ablation study.** As shown in Table 3 (on MIMIC-CXR-VQA), removing the region encoder (Only RoIAlign) results in a performance drop of 1.06%. Simply cropping region features using RoIAlign from a frozen image encoder leads to inaccurate region embeddings, highlighting the advantage of our multi-scale, trainable region encoder. In future work, we plan to explore more efficient image and region representations to further amplify the benefits brought by fine-grained visual cues.\n\nTable 3. Region encoder ablation on MIMIC-CXR-VQA.\n| Condition          |        |   VQA  |       |   |   | Referring |   |   |       | Grounding |       |\n|--------------------|:------:|:------:|:-----:|---|---|-----------|---|---|:-----:|:---------:|:-----:|\n|                    | verify | choose | query |   |   | Acc       |   |   | R@0.3 |   R@0.5   | R@0.7 |\n| Only RoIAlign      |  79.78 |  69.76 | 63.56 |   |   | 98.43     |   |   | 98.51 |   96.46   | 89.51 |\n| Our region encoder |  **80.68** |  **70.16** | **65.43** |   |   | **98.58**     |   |   | **98.56** |   **96. 62**  | **90.40** |\n\n### **Q3: Evaluation of reasoning quality.**\n\nTable 4 in the manuscript presents the referring and grounding performance of AOR. Specifically: \n\n- Referring evaluates the model's ability to correctly identify anatomical regions given visual prompts. \n- Grounding measures its localization precision during reasoning. \n\nAOR attains a referring accuracy of 98.58% and a grounding R@0.7 of 90.40%, highlighting its strong capability in **visual-text alignment**.\n\nTo further assess the quality of **text-level reasoning**, we randomly sample 100 test cases and evaluate the outputs by both GPT-4o and a radiologist. Each instance (ground truth and prediction) is assessed on the following dimensions (scored 1–5):\n\n- **Accuracy**: Correct identification of attributes or categories.\n- **Completeness**: Consideration of all attributes and categories relevant to the given question.\n- **Consistency**: Logical coherence and alignment between intermediate steps and final conclusions.\n\nAs shown in Table 4, AOR consistently outperforms **LLaVA-o1** and **MedVLM-R1**, validating the **factuality and coherence** of its reasoning process. While absolute scores differ slightly between GPT-4o and the radiologist, the evaluation **trends are consistent**, supporting the feasibility of using LLMs as scalable proxies for expert judgment. This comparative evaluation will be added to the revised manuscript.\n\nTable 4. The score of AOR's reasoning steps.\n| Judger      | Model     | Accuracy | Completeness | Consistency |\n|-------------|-----------|:--------:|:------------:|:-----------:|\n| GPT-4o      | LLaVA-o1  |   2.52   |     3.03     |     3.86    |\n|             | MedVLM-R1 |   2.31   |     2.59     |     3.74    |\n|             | AOR(Ours) |   **3.65**   |     **4.13**     |     **4.20**    |\n| Radiologist | LLaVA-o1  |   2.68   |     2.97     |     4.11    |\n|             | MedVLM-R1 |   2.27   |     2.76     |     3.82    |\n|             | AOR(Ours) |   **3.83**   |    **4.49**     |     **4.55**    |\n\n### **Limitations: Generalization across modalities and anatomical structures**\n**Different imaging modalities**: Taking 3D chest CT as an example, we curate a chest CT dataset under the guidance of radiologists, defining **130+ anatomical regions**. 2,070 CT scans are processed and split (80% training / 20% test). We adapt our model by replacing the 2D image encoder with a 3D variant, and evaluate its performance on report generation. \n\nAs shown in Table 5, even with limited training data, our method outperforms baselines without anatomical decomposition by an average of 8.66%, demonstrating the potential generalizability of our anatomy-centric reasoning strategy to 3D modalities.\n\nTable 5. Performance  on CT region report generation.\n| Condition                    | R-L   | BERTScore | F1CheXbert |\n|------------------------------|:-------:|:-----------:|:------------:|\n| w/o anatomical decomposition | 8.09  | 68.33     | 13.94      |\n| w anatomical decomposition   | **16.45** | **79.18**     | **20.72**     |\n\n**Different tissues and Anatomical structures**: Through collaboration with radiologists, we find that the AOR paradigm can generalize to other anatomical structures as follows: \n- For organs with clearly-defined anatomical regions (e.g., brain lobes in MRI), reasoning flows can be designed around established anatomical ontologies (e.g., frontal, parietal, temporal, occipital lobes). \n- For  structures without conventional anatomical zoning, such as oral structures, reasoning can still follow structural decomposition, using components like vertebrae, ribs, crowns, roots, jaws, condyles, and airways as spatial anchors.\n\nThese findings suggest that the proposed data generation pipeline and AOR architecture are adaptable to a **broad range of tissues and imaging modalities**."}, {"review_id": "9ZDerjYvIJ", "reviewer": "Reviewer_wKs5", "summary": "This paper presents a framework called AOR (Anatomical Ontology-Guided Reasoning), which aims to improve the reasoning and interpretability of large multimodal models for chest X-ray analysis. The authors focus on two key limitations of current models: the lack of region-level understanding and the reliance on single-step prediction. To tackle this, they propose a three-stage training approach that includes anatomical region recognition, grounding, and instruction tuning. A major component of their work is the construction of AOR-Instruction, a large-scale dataset developed under expert guidance, which incorporates structured reasoning based on anatomical ontologies. The method supports both textual and optional visual prompts, and is evaluated on several tasks including VQA and report generation. Experimental results show clear improvements over existing medical vision-language models, especially in cases requiring step-by-step reasoning or regional grounding.", "questions": "1. It would be helpful if the authors could clarify whether any manual review or quality check was conducted on the generated AOR-VQA or AOR-RG samples, especially given the scale of the dataset. Additionally, is there any plan to release the dataset (or a portion of it) to facilitate future research?\n2. Since the backbone of the proposed model is a general-purpose LLM, one would expect it to retain some degree of open-domain or out-of-distribution (OOD) reasoning capability. I wonder if the authors have considered evaluating the model on more global or metadata-related questions (e.g., patient age, gender, image view) that are present in MIMIC-CXR-VQA. Such evaluation could help assess whether the model preserves generalization ability beyond the region-level tasks it was primarily trained for.\n3. In Table 3, it is interesting to observe that the setting using CoT alone (without coordinate or region features) already achieves the best performance on the “verify” question type, nearly matching or slightly outperforming the full AOR setting. Could the authors comment on why CoT alone is particularly effective for this category, and whether this suggests that “verify” questions benefit more from structured textual reasoning than from spatial or visual cues?", "limitations": "While the paper addresses some limitations, one important concern is not discussed. The proposed use of region-level reasoning is arguably more suited to 3D imaging modalities, where anatomical regions are better defined and spatially consistent. In 2D chest X-rays, the projection inherently compresses anatomical structures (e.g., the heart), and region definitions may not fully capture the relevant semantic or clinical context. As a result, some region-level supervision might be misaligned with true anatomy, and the benefit of region guidance may be overestimated in this setting. It would be helpful for the authors to reflect on this limitation and discuss whether their approach could generalize or even perform better in volumetric imaging tasks like CT or MRI.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper proposes a structured and well-motivated approach that introduces region-level information in a staged manner, which aligns with how radiologists interpret medical images.\n\n2. The authors construct a large instruction dataset based on anatomical ontologies and Chain-of-Thought reasoning, which required domain expertise and could benefit the broader medical AI community.\n\n3. Experiments are thorough and cover both visual question answering and report generation tasks, with consistent improvements over strong baselines.\n\n4. The method enhances interpretability through region-aware reasoning, which is especially valuable in high-stakes domains like healthcare.\n\nWeakness\n\n1. While the model is built upon a general-purpose LLM, the paper does not explore its generalization ability to out-of-distribution question types such as age or gender, which are not relevant and present in the dataset.\n\n2. The ablation study in Table 3 shows that CoT alone achieves nearly the best result on “verify” questions, but the paper lacks discussion on why coordinate and region features seem to offer limited additional benefit in this case.", "was_revised": true, "final_justification": "After reading the authors' rebuttal, most of my concern have been resoloved. I will keep my positive score.", "rebuttal": "Thank you for recognizing our work and the valuable comments. We hope that the following point-by-point responses address your concerns.\n\n### **Q1: Quality Control and Release Plan for AOR-VQA and AOR-RG**.\n\nYes, we performed manual review for quality control during the dataset construction process. Details for each dataset are provided below:\n\n**AOR-VQA**: From ontology design to CoT construction, the entire process was guided by three expert physicians. We constructed **2,812 CoT types**, derived from combinations of question levels, anatomical regions, and findings. Each CoT type was thoroughly reviewed by at least two physicians, with ~1,900 types reviewed per physician. Disagreements were resolved through collective discussion.  \n  These reviewed CoT types were then expanded to individual QA samples through logical composition. Since this step involves deterministic transformations, it theoretically introduces no additional risk of error. To further ensure data quality, we randomly sampled **1% of the ~290,000 QA pairs post-generation for manual review.** These samples were evenly distributed among three physicians.\n\n**AOR-RG**: We constructed AOR-RG based on the rules proposed in ASG [1], which stated that their anatomical region–sentence alignment paradigm was developed under the guidance of radiologists. Building on this foundation, we optimized the alignment method to handle cases where two different anatomical regions appear in the same short sentence, introducing new rules to split such sentences into two separate ones. Similarly, we randomly sampled **0.5% of the ~399,000 region–sentence pairs for manual review** by three physicians to ensure that no recurring or quality-compromising errors were present.\n\n**Dataset Release Plan**: We will publicly release the AOR-VQA and AOR-RG datasets and are in the process of preparing the files and documentation for upload to PhysioNet.\n\n### **W1 & Q2: Generalization to Global and Out-of-Distribution Questions**\n\n**Our Design Choice: Focus on Higher-Priority CXR Tasks.** Our study was intentionally designed to focus on clinically relevant, fine-grained reasoning tasks that are of high priority in real-world radiology, such as localizing lesions, interpreting complex findings, and generating detailed report sections. In contrast, we assigned a lower clinical priority to meta-level information (e.g., patient age or gender) that can be readily obtained from hospital systems, and thus did not incorporate detailed reasoning chains for them in our primary task design. This focus allows our model, AOR, to excel at its core purpose.\n\n**Performance on Global Questions in AOR-QA.** Despite this focus on fine-grained tasks, our model is still highly capable of handling the global or metadata-related questions present in our dataset. AOR-QA includes all question types from MIMIC-CXR-VQA, and as shown in Fig. A1 (Appendix), 'Plane' (image view) and 'Gender' questions are indeed part of it. Our model achieves accuracies of ~95% on these question types (Tables B13-B15), demonstrating strong performance on the non-regional tasks available in the dataset.\n\n**Further Evaluation on OOD Reasoning.** To more directly address the important point about out-of-distribution (OOD) generalization, we conducted further tests. We evaluated AOR on a subset of VQA-RAD questions, which are entirely OOD with respect to our training data (e.g., knowledge-based questions unrelated to the image). AOR achieves 42.86% accuracy on closed questions and 10.00% on open ones. While these results are preliminary, they provide a valuable benchmark.\n\n**Future Research Direction.** These findings reinforce our view that extending the model’s capabilities to open-domain and OOD tasks is a valuable next step. As this work lays the foundation for anatomy-oriented reasoning, we see a clear path forward. We believe that using stronger base models (e.g., Qwen2.5-VL) and leveraging retrieval-augmented techniques are promising directions to improve AOR on these tasks. We particularly aim to explore efficient learning from small samples, a critical challenge in the medical domain.\n\n### **W2 & Q3: Further discussion on the Comparison of different CoT representations.**\n\n**Further discussion**: Among the three types of questions, verify questions are relatively easier. The binary nature of verify questions (i.e., yes/no) compresses the model’s output space, making it more difficult to demonstrate performance improvements once a high accuracy level is reached (e.g., our baseline ID 2 achieves 80.69% on verify). In contrast, choose and query questions impose higher demands on the model’s reasoning correctness. Compared to ID 2 (CoT-only), ID 4 (full AOR) shows performance gains of **+2.64%** and **+2.03%** on choose and query questions respectively, highlighting the importance of spatial and visual cues.\n\n**Additional zero-shot evaluation on VQA-RAD**: To further support this observation, we conducted an ablation study on the VQA-RAD dataset under a zero-shot setting. As shown in Table 1, incorporating spatial and visual cues yields improvements of **5.48%** on closed-form (verify-like) questions. This demonstrates that such cues enhance the model’s comprehension and reasoning abilities on previously unseen data distributions.\n\nTable 1. Comparison of CoT representations on VQA-RAD.\n| Exp_ID |  coor | region |  CoT  |   closed  |    open   |\n|:------:|:-----:|:------:|:-----:|:---------:|:---------:|\n|    1   | **✗** |  **✗** | **✗** |   47.74   |   16.90   |\n|    2   | **✗** |  **✗** | **✓** |   52.05   |   19.20   |\n|    3   | **✓** |  **✗** | **✓** |   54.50   |   21.91   |\n|    4   | **✓** |  **✓** | **✓** | **57.53** | **24.99** |\n\n### **Limitations:  generalization to 3D imaging modalities.**\n\n**Effectiveness on 2D chest X-rays**: We acknowledge that 2D chest X-rays are subject to certain degrees of projection-induced compression. However, chest X-rays remain the first-line imaging tool  in routine clinical practice due to their low cost, minimal radiation exposure, and broad accessibility. Our anatomical region definitions are based on widely used clinical zoning standards and have been reviewed and validated by experienced radiologists, ensuring the **semantic and diagnostic utility** of the defined regions . Therefore, despite the limitations of  projection, region-level information in 2D chest X-rays still holds substantial clinical value and research relevance. \n\n**Extension to 3D chest CT**: We fully agree that 3D imaging modalities offer richer anatomical information and provide a more consistent spatial context. Through our practical experience, we found this to be a highly meaningful yet very challenging task. For example, in 2D chest X-rays, dividing the heart into two or three subregions is usually sufficient to support most reasoning needs. However, chest CT, by its nature, contains much richer anatomical information. The heart in CT images can be subdivided into eight distinct sub-anatomical regions, e.g., the left and right atria, ventricles, myocardium, and so on. Extending AOR  to such modalities is an important direction of our ongoing work. \n\nTo this end, we have curated **a 3D chest CT dataset with over 130 anatomical regions**, compared to 29 in the 2D X-ray setting. The dataset includes 2,070 CT scans, split 80/20 into training and test sets.We replaced the 2D image encoder with a 3D-compatible variant, and evaluated region-level report generation.\n\nAs shown in Table 2, our method **achieves an average performance improvement of 8.66%** over baselines lacking anatomical decomposition, even with limited data. These results affirm the potential of the AOR framework to generalize effectively to volumetric modalities.\n\nTable 2. Performance of CT region report generation.\n| ID                  | R-L   | BERTScore | F1CheXbert |\n|------------------------------|:-------:|:-----------:|:------------:|\n| 1 | 8.09  | 68.33     | 13.94      |\n| 2   | **16.45** | **79.18**     | **20.72**      |\n\n**Extension to other more modalities**: Beyond chest CT, we are exploring generalization to other modalities and regions within the oral cavity. In cases like dental imaging, where conventional anatomical zoning is not well established, we collaborate with radiologists to design structure-aware reasoning flows by incorporating clinically important components commonly referenced in dental practice, such as crowns, roots, jaws, mandibular canals, and alveolar processes.\n\nThis broader discussion will be added under the Limitations and Future Work section in the revised manuscript.\n\n---\n[1] Li Q, Yan X, Xu J, et al. *Anatomical structure-guided medical vision-language pre-training.* MICCAI 2024."}]

Author rebuttal:

Meta-review: The paper presents AOR (Anatomical Ontology-Guided Reasoning), a new framework that enhances the reasoning capabilities of large multimodal models when interpreting chest X-rays. AOR mimics how radiologists approach diagnoses by reasoning anatomically, and it is supported by a new dataset, AOR-Instruction, crafted from expert input to promote step-by-step clinical reasoning. The approach aims to improve accuracy, interpretability, and generalizability across multiple tasks, including visual question answering and report generation.

Initial reviewer concerns focused on missing comparisons to recent baselines like LLaVA-Rad, limited training detail, and uncertainty about the added value of some model components. The authors responded thoroughly, introducing new experiments, results, and analyses to show AOR’s improvements across a wide range of settings. These included new baselines, validation on 3D CT data, and a detailed component analysis supported by human and GPT-4o evaluations.

After the rebuttal, all reviewers affirmed their positive scores and noted that their concerns were resolved. The final ratings are one Accept, and three Borderline Accepts. Considering the paper's positive scores and contributions to interpretable and robust AI in medical imaging, the area chair recommends Accept (Poster).
Decision: Accept (poster)