Title: ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-modal LLM

multi-modal learning

representation learning

healthcare
Abstract: State-of-the-art medical multi-modal LLMs (med-MLLMs), such as LLaVA-Med and BioMedGPT, primarily depend on scaling model size and data volume, with training driven largely by autoregressive objectives. However, we reveal that this approach can lead to weak vision-language alignment, making these models overly dependent on costly instruction-following data. To address this, we introduce ExGra-Med, a novel multi-graph alignment framework that jointly aligns images, instruction responses, and extended captions in the latent space, advancing semantic grounding and cross-modal coherence. To scale to large LLMs (e.g., LLaMa-7B), we develop an efficient end-to-end training scheme using black-box gradient estimation, enabling fast and scalable optimization. Empirically, ExGra-Med matches LLaVA-Med’s performance using just 10\% of pre-training data, achieving a 20.13\% gain on VQA-RAD and approaching full-data performance. It also outperforms strong baselines like BioMedGPT and RadFM on visual chatbot and zero-shot classification tasks, demonstrating its promise for efficient, high-quality vision-language integration in medical AI.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting medical multimodal LLMs rely heavily on large-scale data and autoregressive training, which often results in poor vision-language alignment and high dependence on instruction-tuning data. The authors propose ExGra-Med, a novel multi-graph alignment framework that jointly aligns images, responses, and extended captions to enhance semantic grounding and cross-modal coherence. ExGra-Med achieves competitive or superior performance to state-of-the-art models like LLaVA-Med and BioMedGPT using significantly less training data, showing strong efficiency and scalability for medical vision-language tasks.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. The paper identifies a critical limitation in existing autoregressive medical MLLMs by revealing their heavy reliance on large-scale instruction-following data, highlighting a key inefficiency in current pre-training paradigms.\n\n2. It introduces a novel multi-graph alignment objective that effectively captures triplet-level semantic relationships among images, instruction responses, and enriched captions, enhancing cross-modal understanding.\n\n3. An efficient training solver is proposed for large LLMs, supported by theoretical insights into geodesic distances and shortest paths in multimodal graph space.\n\n4. The proposed model, EXGRA-MED, achieves competitive or superior performance to state-of-the-art med-MLLMs using only a fraction of the data, demonstrating strong efficiency, scalability, and generalization across diverse medical vision-language tasks.\n\nCons:\n\n1. Does EXGRA-MED use full fine-tuning? How does it perform with parameter-efficient fine-tuning methods such as LoRA, and could this further improve efficiency?\n\n2. The proposed method appears to be domain-agnostic---could it also be effective in the general domain beyond the medical field?\n\n3. It would be helpful to cite a recent survey on MLLMs to provide readers with a comprehensive background and better contextualize your contribution within the broader MLLM research landscape [1].\n\n[1] \"MM-LLMs: Recent Advances in MultiModal Large Language Models.\" Findings of the Association for Computational Linguistics ACL 2024. 2024.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes EXGRA-MED, a training-efficient framework for medical vision-language models that aligns images, instructions, and enriched captions through a multi-graph alignment strategy. By combining this with black-box optimization, the model achieves strong performance with significantly less instruction-following data, outperforming baselines across multiple medical VQA and classification tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper introduces a multi-graph alignment framework that directly trains LLMs with triplet correlations across images, instruction responses, and extended captions, improving vision-language coherence without requiring large datasets\n- EXGRA-MED demonstrates strong performance using only 10% of the instruction-following data, reaching or exceeding LLaVA-Med's full-data performance on multiple benchmarks.\n- Employs Implicit Maximum Likelihood Estimation to scale combinatorial graph alignment efficiently to LLaMA-7B-sized models, a notable technical achievement. \n- Covers three major MedVQA datasets, chatbot benchmarks, and zero-shot classification on 23 datasets, showing consistent improvements over several baselines.\n\nWeaknesses\n- While LLaVA-Med and LLaMA-7B are used as backbones, the method’s generalization to other base models (e.g., Qwen) remains underexplored. \n- The enriched caption generation pipeline depends on proprietary GPT-4 APIs. This reliance may hinder reproducibility or deployment in restricted environments. \n- The graph alignment formulation, though elegant, is mathematically intensive and may benefit from simplified descriptions or more intuitive illustrations for accessibility.\n\nQUESTIONS:\n- Could the authors provide quantitative results when training EXGRA-MED with smaller frozen LLMs to evaluate downstream performance trade-offs?\n\n- How robust is EXGRA-MED to inaccurate or noisy extended captions, especially if GPT-4 is replaced with lower-quality generators?\n\n- Is the triplet graph structure essential, or could similar benefits be achieved using just pairs (e.g., image and extended caption)?\n\n- Can the authors release a toolkit or provide open-source code for the black-box optimization with IMLE to support reproducibility?\n\nLIMITATIONS:\nNo. Authors should be rewarded rather than punished for being up front about the limitations of their work and any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces EXGRA-MED, a novel multi-graph alignment framework for medical vision-language models that addresses the data-hungry nature of autoregressive training in medical multimodal LLMs. The authors identify that models like LLaVA-Med suffer significant performance drops when trained on limited instruction-following data, even after fine-tuning. To address this, EXGRA-MED constructs three modality-specific graphs representing visual features, original instruction responses, and GPT-4 generated extended captions, then jointly aligns them using a barycenter graph approach with black-box gradient estimation for scalable training.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The paper clearly demonstrates the data-hungry nature of autoregressive training in medical MLLMs through systematic experiments showing dramatic performance drops (e.g., 72.64% to 52.39% on VQA-RAD) when using only 10% of pre-training data.\n2. The novel fine-tuning with multi-graph alignment\n\n## Weaknesses\n1. Training efficiency analysis. According to section 4.1 the actual training time of ExGra-med is 0.5 hour longer than the original LLaVA-Med. How much additional cost (e.g. GNN training) will the multi-graph alignment training will bring should be clarified.\n\nQUESTIONS:\n1. You mention the ExGra-Med's language model backbone is LLaMA-7B, but which LLaMA version exactly? From what I know, LLaVA-med uses mistral-7b-v0.2 as the language model backbone. Usually better language model backbone will result in better downsteam performance, will that cause unfair comparison? \n2. Please try using the term \"pre-training\" for stage 1 and \"fine-tuning\" on stage 2 properly, according to the original LLaVA paper. On line 281, I assume you want to say \"complete the whole fine-tuning process compared to the LLaVa-Med\".\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ExGRA-Med, a multi-graph alignment framework designed to improve vision-language alignment in medical multimodal large language models. The authors address a significant limitation of existing approaches (e.g., LLaVA-Med, BioMedGPT) that suffer from weak cross-modal alignment due to their heavy reliance on autoregressive training objectives.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed multi-graph alignment mechanism effectively tackles the vision-language alignment problem by jointly aligning images, instruction responses, and extended captions in the latent space.\n2. Demonstrates impressive data efficiency, achieving comparable performance to LLaVA-Med using only 10% of the pre-training data.\n3. Shows substantial improvement (20.13% gain) on VQA-RAD benchmark, indicating better medical visual question answering capabilities.\n4. The black-box gradient estimation approach enables scalable training for large LLMs.\n\nWeakness:\n1. The proposed multi-graph alignment framework introduces significant architectural complexity that makes the paper difficult to follow and understand. The intricate design may hinder reproducibility and practical adoption due to unclear implementation details and a lack of intuitive explanations. \n2. The proposed approach extends VQA capabilities primarily through simple conversational formats as shown in Figure 2, limiting applicability to complex medical reasoning scenarios. The evaluation lacks assessment on more challenging conversational AI tasks that require sophisticated multi-turn dialogues or clinical decision-making processes. \n3. The experimental evaluation is constrained to relatively simple benchmarks (VQA-RAD, SLAKE, PathVQA) and primarily compares against LLaVA-Med, an earlier generation model. The absence of comparisons with recent state-of-the-art medical multimodal models limits our understanding of the method's true competitive standing.\n\nQUESTIONS:\n1. The proposed method uses GPT-4o-powered extended context instruction-following data for multi-graph alignment. How does this compare with methods that directly optimize instruction-following data using GPT-4o?  Direct comparison with GPT-4o optimization methods would strengthen contribution claims.\n2. Current medical VQA datasets are typically more complex than the Figure 2 examples. Can your method maintain effectiveness on sophisticated medical datasets involving complex clinical reasoning? Demonstrating effectiveness in complex medical scenarios would enhance practical significance.\n3. Please incorporate more challenging benchmarks such as MMMU Health & Medicine, MedXpertQA, and OmnimedVQA to better assess model capabilities. Results on advanced benchmarks would provide stronger evidence of superiority.\nLimitations\n4. The complex alignment architecture requires more thorough validation beyond simple benchmarks. Effectiveness has not been sufficiently demonstrated across diverse scenarios.\n5. Unclear whether performance gains persist with stronger base models or comprehensive VQA corpora. Robustness to state-of-the-art foundation models needs validation.\n\nLIMITATIONS:\n1. The complex alignment architecture requires more thorough validation beyond simple benchmarks. Effectiveness has not been sufficiently demonstrated across diverse scenarios.\n2. Unclear whether performance gains persist with stronger base models or comprehensive VQA corpora. Robustness to state-of-the-art foundation models needs validation.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}