LoGra-Med: Long-Context Multi-Graph Alignment for Medical Visual-Language Models

Authors: Duy Minh Ho Nguyen, Nghiem Tuong Diep, Trung Quoc Nguyen, Bao Hoang Le, Tai Nguyen, Anh-Tien Nguyen, TrungTin Nguyen, Nhat Ho, Pengtao Xie, Roger Wattenhofer, James Zou, Daniel Sonntag, Mathias Niepert
Primary area: foundation or frontier models, including LLMs
Keywords: multi-modal LLM, AI for Healthcare, multi-modal learning

Abstract:
State-of-the-art medical multi-modal large language models (med-MLLM), such as LLAVA-MED or BIOMEDGPT, leverage instruction-following data in their pre-training stages. However, those models primarily focus on scaling the model size and data volume to boost performance while mainly relying on the autoregressive learning objectives. Surprisingly, we reveal that such learning schemes might result in a weak alignment between vision and language modalities, making these models highly reliant on extensive pre-training datasets — a significant challenge in medical domains due to the expensive and time-consuming nature of curating high-quality instruction-following instances. We address this challenge with a new multi-graph alignment algorithm, namely LOGRA-MED, which enforces triplet correlations on the latent embedding space among image modalities, conversation-based descriptions, and extended contextual captions. Owing to this technique, the model is encouraged to capture the semantic meaning of the context, handle linguistic variability where the captions or questions may differ from training instances, and learn cross-modal associations, linking visual elements with various textual interpretations. To scale our algorithm to the med-MLLM setting, we also design an efficient end-to-end learning scheme based on advanced black-box gradient-estimation techniques that permit fast forward and backward steps through the LLM model (LLaMa 7B). Empirical results show
that we can match the performance of LLAVA-Med pre-trained on 600K image-text pairs from PMC-15M for Medical VQA tasks and significantly outperform it when trained on only 10% of the data. For instance, on VQA-RAD, we exceed LLAVA-Med (both trained on 10%) by 20.13% and achieve near parity with the 100% pre-training setting (72.52% vs. 72.64%). Additionally, we also surpass other SOTA pre-training methods and med-MLLM such as BIOMEDGPT on visual chatbot or RADFM on zero-shot image classification with VQA, showcasing the power of multi-graph alignment in improving vision-language integration for medical-MLLM.

Decision: reject
Reviewer scores: 6, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730012965668

thread_type:
official_review

<thread_object>
creator: reviewer_cex3

<content>
summary:
This paper introduces LOGRA-MED, a novel approach addressing the data efficiency challenge in medical multi-modal large language models (med-MLLM) through multi-graph alignment. The work reveals that current autoregressive training methods like LLAVA-Med require extensive instruction-following data, showing a significant performance drop from 72.64% to 52.39% on VQA-RAD when reducing training data to 10%. To address this, LOGRA-MED enforces triplet correlations in the latent space among image features, instruction data, and GPT-4 generated extended contextual captions using a structure-aware graph matching framework. The method efficiently handles the combinatorial complexity of multi-graph alignment through an implicit maximum likelihood estimation approach with black-box gradient computation.

strengths:
The technical innovation of LOGRA-MED lies in its formulation of vision-language alignment as a multi-graph matching problem, generalizing beyond traditional pairwise contrastive learning approaches. The theoretical foundation is particularly strong, proving both metric and geodesic properties of the graph alignment distance. The implementation cleverly circumvents the NP-hard nature of multi-graph matching using Lagrange decomposition and efficient heuristic solvers. The empirical results are comprehensive, demonstrating that with only 10% of training data, LOGRA-MED achieves 72.52% accuracy on VQA-RAD, nearly matching LLAVA-Med's 72.64% with full data. The method's effectiveness extends across various tasks including medical visual chatbot capabilities (44.82% overall score) and zero-shot classification on 23 datasets spanning microscopy, CT, and CXR modalities.

weaknesses:
The reliance on GPT-4 for generating extended contexts raises questions about potential biases and the method's generalizability. While the paper demonstrates improved performance, the computational complexity analysis could be more thorough - the reported one-hour additional training time compared to LLAVA-Med merits deeper discussion given the sophisticated graph matching operations. The variable performance across different medical domains (particularly in PathVQA and some zero-shot classification tasks) suggests potential limitations in handling diverse medical contexts. The ablation studies, while informative, could benefit from exploring different graph construction strategies beyond k-nearest neighbors and analyzing the impact of various distance metrics in the graph matching formulation.

questions:
The sensitivity of LOGRA-MED to the quality and consistency of GPT-4 generated contexts remains unclear？ 

how would the method perform with simpler context extension approaches or with different language models? 

The paper could elaborate on the interplay between the traditional autoregressive loss and the graph alignment objective, particularly regarding their relative contributions to the final performance？

The choice of hyperparameters in the graph construction phase, especially the selection of k in k-nearest neighbors, appears critical but lacks detailed justification？
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655367481

thread_type:
official_review

<thread_object>
creator: reviewer_9zpv

<content>
summary:
In this paper, the authors introduced a self-supervised multi-graph alignment objective strategy for pre-training medical MLLM. Specifically, the proposed method aligns the triplet consisting of the input image, its instruction data, and its extended long context generated by GPT. The graphs of images, the instruction data, and the long context are trained to align with the barycenter graph. The authors perform experiments on multiple downstream tasks, and the proposed method outperforms baseline methods. The proposed method achieved better performance with the limited amount of pre-training dataset.

strengths:
1. The empirical results demonstrate that the proposed approach can reduce the demand for pre-training data for MLLM alignment, outperforming previous approaches.

2. The authors conducted a set of ablation studies, which demonstrated the contribution of each component.

3. The visualization in the manuscripts helped me to better understand the proposed approach and experiments.

weaknesses:
1. From Table 5, it seems that the generated long context data plays an important role in LoGra-Med. Without it, the performance is slightly lower than the baseline LLaVA-Med. Since LLaVA-Med doesn't use the generated long context data, these results cast doubt on the contribution of the graph alignment algorithm. For a fair comparison to validate that the performance gain is not mainly due to the extra data, the author could also pre-train LLaVA-Med with the long context data and compare the performance.

2. The proposed method's performance gain may come from the extra computing in training (the authors mention that the training time is longer than the baseline method by 1 hour) rather than the method itself. For a fair comparison, the authors may pre-train the LLaVA-Med baseline method using some self-supervised objectives (e.g., InfoNCE) or additional long context data for 1 hour.

3. The authors need to mention some of the important hyperparameters used. For example, what's the batch size B? I would assume the larger the batch size, the larger the graph. And what's the value of k used in the k-NN algorithm for building edges?

questions:
1. Can you provide the hyperparameters used in the paper, including the batch size B and the value of k used in the k-NN?

2. Are there any experiment results presented in the paper that can prove the performance gain compared to LLaVA-Med is due to the graph alignment algorithm itself rather than additional data or extra computing?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706588587

thread_type:
official_review

<thread_object>
creator: reviewer_jqrh

<content>
summary:
This work propose a method that aiming for decrease the data needed for medical Multi-Modal large langauge model pre-training by enforcing triplet correlations on the latent embedding among image, descriptions, and captions. How to efficiently pre-train large models with limited data in medical domain is a quite important topic, and thus the motivation is good enough. However, the reason behind applying multi-gprah alignment is not well-explained. And this process required retrain the LLM model, which is more costful compared other methods.

strengths:
1. This model propose a method to convert auto-gressive finetuning to Graph-alignment task, a fresh perspective. This seems interesting but lack of motivation.
2. The results showing that the proposed method is definetly increase the performance on different tasks with limited datasize. This is quite important and might showing a new way except data curation.
3. Some of the detail are not well-explained in the paper, please refer to the weakness and quesiton section. I will consider adjust my scores based on the authors response.

weaknesses:
1. The motivation of adding graph alignment task in the pre-training stage is not explained.
2. This paper also cite the work of MedTrinity(Xie et al.2024), but you didn't include their results in your comparison tables for your experiments. Their model's performance on some dataset are better than yours, you should include them to give readers a fair comparision. You may argue that their training data is larger than yours. But your method also introduce augmented description data generated by GPT-4.
3. SInce you introduced more data by extending the short answer to long answers. So, when you finetuning the LLavaMed basedline with 10% data, do you also include the extended version answer?

questions:
1. Why extending the short answer to longer-context, why would that beneficial the pre-training? 
2. Have you run any qualification test on the GPT generated text?
3. Would you mind to clarify, when you mentioned "10% data", you mean, "10% datasize for the stage2 instruction-tuning stage, but the datasize for the stage1 pre-trianing is still fullzie". Is that correct?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708731233

thread_type:
official_review

<thread_object>
creator: reviewer_kirs

<content>
summary:
The paper proposes a multi-graph matching objective to enhance the vision-language alignment in generative VLMs for medical tasks. The authors motivate the need for such an auxiliary learning objective from a training efficiency point of view, showing that the next token prediction loss function demands a large amount of visual instruction tuning data that's not easy to acquire. To reduce the sample complexity, this paper proposed their method of treating visual features, text features, and augmented text features (based on a more verbose version of the ground truth answer) as graphs and letting the VLM learn to match between the triplets. Since solving a multi-domain graph alignment problem is computationally expensive, the authors propose a more scalable solution by separately aligning each domains with a barycenter graph. The experiments in the paper show that multi-graph alignment significantly improves upon LlaVA-Med, at both low pretraining data (10%) and full data regimes across a wide range of tasks.

strengths:
I like how the paper motivates the necessity of the proposed algorithm from the perspective of training efficiency, which is quite refreshing. The experiments also validate that the proposed algorithm indeed results in better learning outcomes at low data regimes and even does so when using all the pretraining data. 

The presentation of the core methodology flows very well, with tight logical connections. The gradient-based method to by-pass the high time complexity of the solver also seems reasonable.

The paper also seems to be quite comprehensive in terms of baselines. The ablation studies are satisfactory, especially the one without “long-context”.

The experiment details in the supplemental material is very comprehensive, including important details such as system prompt and evaluation protocols.

weaknesses:
I have concerns regarding the core methodology. 

1. The paper seems to have skipped simpler solutions that might as well worked and jumped right into a complex proposal. In section 3.4, the authors argue the necessity of their scalable multi-graph alignment algorithm by saying pairwise graph alignment requires K choose 2 pairs. But in this specific case, K is 3 and K choose 2 is really only 3 pairs. From my point of view, this really should be the method to be tried first before moving on to more complex formulations. 
2. I am too sure what leads to LoGRA’s success due to how the visual & text representations are obtained. In its essence, the representations from both modalities are sets with varying number of elements. For images, this will be different number of patches due to different resolutions; for text, it’s whatever the number of tokens the ground truth answer corresponds to. The authors are simply taking a naive average over the elements. I really doubt the average of word embeddings / visual tokens will end up giving any meaningful representations. I would like to see some analysis on this design choice, for example comparing the distribution of cosine similarity (for example) between positive and negative pairs, so that I know the proposed objective is really forcing the alignment instead of picking up some other signals. 
3. As far as I know, most medical datasets are highly imbalanced, but the paper seems only to report the overall accuracy instead of per-disease macro scores. This is a bit concerning since if all the alignment algorithm does is to guild the model to favor the majority class, we will get a model with high accuracy but no practical use. I think for classification tasks, macro F1 scores should be reported. For VQA tasks, per-disease scores should be reported for datasets that include a class-label for the image-question-answer pair.
4. I think the word “long-context” is misused: from the examples, it looks like the so called “long-context” texts are no more than a few hundred words. Based on the context of the paper, the authors really meant a more verbose / extended rewrite of the original ground truth answers as an augmentation for the triplet learning. A better terminology should be used to distinguish this paper from works dealing with long-context tasks, which usually consumes at least tens of thousands of tokens for each query.

questions:
1. I’m actually a bit confused on why the graph perspective is needed here. It seems the paper is solving the exactly the same problem as optimal transport, but all these optimal papers do not talk about graphs? 
2. I wonder, without using the verbose version, how’s this paper different from the PLOT paper, since it’s essentially solving a matching problem between 2 domains?
3. I find counterintuitive that DCI provides mixed results. What’s the explanation here? To be honest, I would be totally fine if those “+DCI” are not there. But if the authors decide to add these experiments, then they need to explain why utilizing multi-scale features are harmful for some cases.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732360617188

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We sincerely thank the reviewers for their insightful comments and thorough feedback. The **reviewers found our LoGra-Med algorithm, inspired by the efficiency needs of multi-modal LLM training and implemented through multi-graph alignment, to be innovative** (Reviewer Kirs and Reviewer cEx3). The reviewers also concurred that our use of black-box gradient estimations and efficient heuristic solvers to **address the high time complexity of NP-hard multi-graph matching is well-founded** (Reviewer Kirs and Reviewer cEx3) and supported by robust **theoretical results on metric and geodesic properties** (Reviewer cEx3). Additionally, all reviewers acknowledged the **strength of our comprehensive experiments, spanning three popular VQA datasets, a medical chatbot, and zero-shot classification on 23 datasets**. They also appreciated the well-conducted ablation studies, which demonstrated that (i) LoGra-Med, using only 10% of pre-training data, achieves comparable performance to LLAVA-Med trained with 100% of the data (on VQA tasks), and (ii) outperforms several state-of-the-art multi-modal LLMs.

The reviewers raised concerns, which we addressed in the individual responses below. We summarize the highlights of our responses where additional experiments are provided:

- **Test with simpler solutions, including two graph alignments/using a pair-wise graph alignment solver rather than solving through graph barycenter**:  We provide additional experiments for two settings: (i) performing contrastive learning between two graphs (images and their original captions) and (ii) addressing pairwise graph alignment instead of using graph barycenters in case of three graph alignments. In summary, we observe that (i) achieves lower performance compared to LoGra-Med with three-graph alignment, while (ii) matches the speed of our solver but results in reduced overall performance.


- **Validating using average pooling token features**: We conducted a new experiment by pre-training on 70% of the data and utilizing the trained model to compute distances for positive and negative pairs across 1000 randomly selected unseen samples. The results suggest that the average pooling feature remains an effective approach, as it establishes a clear margin of separation between the two distributions.


- **Evaluating the contribution of long-context vs. multi-graph alignment by pre-training LLava-med baseline with long-context and longer training time:** We ran extra experiments to further validate our approach: (i) comparing the performance of LoGra-Med and the LLaVa-Med baseline, both pre-trained on 40% of the extended context data, and (ii) repeating the same setup as (i), but allowing LLaVa-Med to train for a longer duration than LoGra-Med. The results consistently demonstrate that LoGra-Med outperforms LLaVa-Med in both scenarios by significant margins, underscoring the effectiveness of our multi-graph alignment framework.


- **Validating quality of contexts generated by GPT-4**: We organized a new user study in which four general practitioners, who are our collaborators and are working at a public medical hospital in Vietnam, evaluated the quality of extended captions generated by GPT-4. Given the limited time available during the rebuttal phase, the study was restricted to 1,000 samples, evenly distributed across five data modalities: CT, X-ray, MRI, histology, and others. Till now, we have completed statistical results for three domains, as presented in Figure X, that confirm that the captions generated by GPT-4 are generally consistent with established medical knowledge.


- **Test with different LLM models (simpler and other models) to create extended contexts:**  We present results for LoGra-Med using a more straightforward paraphrase approach (previously included in the ablation studies) alongside new results obtained with the Gemini LLM model for 10% and 40% pre-training scenarios. The performance of the models on two VQA tasks demonstrates that LoGra-Med maintains its effectiveness even when integrated with the Gemini LLM model.

- **Others: clarifying setting of 10% pre-training data, number of K-nearest neighbors in graph construction steps, and interpolating between auto-regressive and graph-alignment in the pre-training phase:**  We provide detailed clarifications regarding the pre-training data in the response below and conducted four additional experiments: (i) varying the number of neighbors in graph construction using KNN with K = 3 and K = 7, and (ii) exploring the trade-off between auto-regressive modeling and multi-graph alignment using different coefficient values. The outcome results validate our choices on K = 5 and coefficient = 1.

We summarize new experiments ran during the rebuttal phase in this link:

https://anonymous.4open.science/r/logra-med-386F/readme.md
</content>
</thread_object>

</thread>


Metareview: