Title: AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Medical Large Multimodal Models; Chain of Thought; Anatomical Ontology; Chest X-rays; Medical Visual Question Answering
Abstract: Chest X-rays (CXRs) are the most frequently performed imaging examinations in clinical settings. Recent advancements in Medical Large Multimodal Models (MLMMs) have enabled automated CXR interpretation, improving diagnostic accuracy and efficiency. However, despite their strong visual understanding, current MLMMs still face two major challenges: (1) insufficient region-level understanding and interaction, and (2) limited accuracy and interpretability due to single-step prediction. In this paper, we address these challenges by empowering MLMMs with anatomy-centric reasoning capabilities to enhance their interactivity and explainability. Specifically, we propose an Anatomical Ontology-Guided Reasoning (AOR) framework that accommodates both textual and optional visual prompts, centered on region-level information to enable multimodal multi-step reasoning. We also develop AOR-Instruction, a large instruction dataset for MLMMs training, under the guidance of expert physicians. Our experiments demonstrate AOR's superior performance in both Visual Question Answering (VQA) and report generation tasks. Code and data are available at: https://github.com/Liqq1/AOR.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework called AOR (Anatomical Ontology-Guided Reasoning), which aims to improve the reasoning and interpretability of large multimodal models for chest X-ray analysis. The authors focus on two key limitations of current models: the lack of region-level understanding and the reliance on single-step prediction. To tackle this, they propose a three-stage training approach that includes anatomical region recognition, grounding, and instruction tuning. A major component of their work is the construction of AOR-Instruction, a large-scale dataset developed under expert guidance, which incorporates structured reasoning based on anatomical ontologies. The method supports both textual and optional visual prompts, and is evaluated on several tasks including VQA and report generation. Experimental results show clear improvements over existing medical vision-language models, especially in cases requiring step-by-step reasoning or regional grounding.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper proposes a structured and well-motivated approach that introduces region-level information in a staged manner, which aligns with how radiologists interpret medical images.\n\n2. The authors construct a large instruction dataset based on anatomical ontologies and Chain-of-Thought reasoning, which required domain expertise and could benefit the broader medical AI community.\n\n3. Experiments are thorough and cover both visual question answering and report generation tasks, with consistent improvements over strong baselines.\n\n4. The method enhances interpretability through region-aware reasoning, which is especially valuable in high-stakes domains like healthcare.\n\nWeakness\n\n1. While the model is built upon a general-purpose LLM, the paper does not explore its generalization ability to out-of-distribution question types such as age or gender, which are not relevant and present in the dataset.\n\n2. The ablation study in Table 3 shows that CoT alone achieves nearly the best result on “verify” questions, but the paper lacks discussion on why coordinate and region features seem to offer limited additional benefit in this case.\n\nQUESTIONS:\n1. It would be helpful if the authors could clarify whether any manual review or quality check was conducted on the generated AOR-VQA or AOR-RG samples, especially given the scale of the dataset. Additionally, is there any plan to release the dataset (or a portion of it) to facilitate future research?\n2. Since the backbone of the proposed model is a general-purpose LLM, one would expect it to retain some degree of open-domain or out-of-distribution (OOD) reasoning capability. I wonder if the authors have considered evaluating the model on more global or metadata-related questions (e.g., patient age, gender, image view) that are present in MIMIC-CXR-VQA. Such evaluation could help assess whether the model preserves generalization ability beyond the region-level tasks it was primarily trained for.\n3. In Table 3, it is interesting to observe that the setting using CoT alone (without coordinate or region features) already achieves the best performance on the “verify” question type, nearly matching or slightly outperforming the full AOR setting. Could the authors comment on why CoT alone is particularly effective for this category, and whether this suggests that “verify” questions benefit more from structured textual reasoning than from spatial or visual cues?\n\nLIMITATIONS:\nWhile the paper addresses some limitations, one important concern is not discussed. The proposed use of region-level reasoning is arguably more suited to 3D imaging modalities, where anatomical regions are better defined and spatially consistent. In 2D chest X-rays, the projection inherently compresses anatomical structures (e.g., the heart), and region definitions may not fully capture the relevant semantic or clinical context. As a result, some region-level supervision might be misaligned with true anatomy, and the benefit of region guidance may be overestimated in this setting. It would be helpful for the authors to reflect on this limitation and discuss whether their approach could generalize or even perform better in volumetric imaging tasks like CT or MRI.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Anatomical Ontology-Guided Reasoning (AOR) framework, which leverages multi-modal region-level anatomical correlation for step-by-step reasoning. Two datasets with multi-modal CoT answers are constructed to train AOR for both question-answering and report-generation. The proposed AOR shows strong results across several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- This paper proposes a new medical dataset with CoT reasoning via exploring objects' relationships. For medical VLLMs, it is hard to generate CoT reasoning by distilling a stronger reasoning model, as there are no models that are strong enough to consistently provide reliable reasoning.\n- The proposed data generation pipeline is sound and effective. Performing step-by-step reasoning by analyzing the correlation across anatomical regions facilitates the generation of more interpretable answers for medical QA and report generation tasks.\n- The paper is well-written; the author provided a detailed description of the proposed methodology. Extensive experiments are conducted to demonstrate the effectiveness of the AOR model.\n\n\n**Weaknesses:**\n- Lacks evaluating how the newly introduced AOR-QA and AOR-RG datasets impact the performance of other medical multi-modal models.\n- Lacks the ablation on the region encoder; it is unclear whether the performance improvement in the AOR model is caused by higher data quality or a specifically designed model architecture.\n\nQUESTIONS:\n1. Could other models except the AOR model, such as LLava-med, gain performance enhancement by supervised fine-tuning with CoT data from these datasets?\n\n2. Is it necessary to employ an independent encoder to extract local information? Additionally, the author should offer more details about the region encoder.\n\n3. Will the trained AOR model generate inaccurate reasoning steps, such as incorrect identification of anatomical regions and their attributes?\n\nLIMITATIONS:\nThe proposed method only focuses on Chest X-Ray; the generalization capability across different tissues and imaging modalities of the proposed data generation pipeline and AOR architecture is unknown.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on enhancing the interpretability and accuracy of medical large multimodal models (MLMMs) in chest X-ray (CXR) interpretation by introducing an anatomy-centric reasoning framework named AOR (Anatomical Ontology-guided Reasoning). The authors identify two key limitations of existing MLMMs (i.e., insufficient region-level understanding and the lack of multi-step reasoning) and address them via a three-stage training framework that incorporates anatomical ontology and Chain-of-Thought (CoT) prompting. To support this, they construct a large-scale instruction dataset, AOR-Instruction, including both visual question answering (AOR-VQA) and report generation (AOR-RG) tasks with fine-grained region–sentence alignments. Experiments across multiple benchmarks demonstrate that AOR achieves promising performance on both VQA and report generation tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The proposed AOR framework integrates anatomical ontologies and region-level reasoning into medical multimodal models. This design closely mimics the diagnostic process of radiologists and enhances both interpretability and clinical relevance.\n2. The authors construct a large-scale and clinically grounded instruction dataset (AOR-Instruction), including both visual question answering (AOR-VQA) and region-level report generation (AOR-RG) components. The dataset is guided by experts and offers fine-grained region–sentence alignment, making it a valuable resource for training and benchmarking future medical MLMMs.\n\nWeaknesses\n1. The paper lacks descriptions for the loss functions used in the three training stages, making it difficult to evaluate how each stage is optimized or how learning objectives are balanced.\n2. From the provided code, it appears that the three-stage training is conducted separately, which may lead to error accumulation across stages. Since each stage is trained in isolation, suboptimal performance in earlier stages (e.g., incorrect region recognition) can propagate to downstream tasks like grounding and reasoning, potentially affecting overall performance and robustness. It remains unclear how the proposed framework mitigates this issue or ensures stability across stages.\n3. In Table 1, AOR(Ours)-t refers to the model trained with text-only prompts, while AOR(Ours)-r/t uses both region (visual) and text prompts. However, it is unclear why in some cases the performance of AOR(Ours)-r/t is worse than that of AOR(Ours)-t. Intuitively, incorporating region-level visual information should provide additional cues for reasoning and improve performance. The paper does not provide an explanation for this performance drop.\n4. In Table 3, the performance differences among the variants (especially between ID 2, 3, and 4) are relatively small and inconsistent across question types. Given these marginal gains, it is difficult to assess the true effectiveness of each representation strategy. Providing additional results on other datasets or tasks would help better demonstrate the advantages of the proposed CoT design.\n5. In Table 5, Stages 1 and 2 are designed to improve region recognition and grounding, respectively. However, their impact on the downstream referring and grounding performance appears limited (i.e., adding these stages leads to only marginal gains) and in some cases, even slight drops (R@0.3: 98.48 -> 98.31). This raises questions about the effectiveness of the pretraining stages and whether their benefits consistently transfer to the final reasoning tasks. Further analysis or ablation would help clarify their contribution.\n6. It would be helpful to clarify whether the proposed dataset will be publicly released, as it is also claimed as one of the key contributions of the paper.\n\nQUESTIONS:\nPlease refer to the weaknesses above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo tackle the problem of insufficient region-level understanding and interaction, as well as limited accuracy and interpretability due to single-step prediction in MLMMs for chest X-ray interpretation, this paper proposes an AOR framework. The AOR framework enhances MLMMs with anatomy-centric reasoning capabilities, enabling multimodal, multi-step reasoning through region-level information. Additionally, the paper develops the AOR-Instruction dataset, which includes anatomical regions and their ontologies, to support training MLMMs for improved accuracy and interpretability in  VQA and report generation tasks.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n1.The paper introduces a novel AOR framework that enhances MLMMs with anatomy-centric reasoning for improved medical imaging analysis.\n2.The methodology and experimental setup are clearly described. Illustrations in supplementary are well put.\n3.The authors validate their approach using extensive datasets, demonstrating the effectiveness and robustness of the AOR framework.\n\nCons:\n1.My main concern is the baselines are not up-to-date. The work below, published in 1st April this year, proposed LLaVa-Rad that brings dramatic improvements over LLaVa-Med. The reported performances may surpass the current work. Though the current method has merits of its own, it’s more complete to include LLaVa-Rad in baselines. (Zambrano Chaves J M, Huang S C, Xu Y, et al. A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings[J]. Nature Communications, 2025, 16(1): 3108.)\n\nMinors:\n1.Line 6, 7: “Insufficient” and \"Limited\" should not be capitalized?\n2.Line 35: A comma is needed before \"but\" to separate the clauses.\n\nQUESTIONS:\nPlease carefully address Cons 1. My evaluation score would drop if more recent methods is not included and compared.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}