LLM-guided spatio-temporal disease progression modelling

Authors: Tiantian He, An Zhao, Elinor Thompson, Anna Schroder, Ahmed Abdulaal, Frederik Barkhof, Daniel C. Alexander
Primary area: applications to neuroscience & cognitive science
Keywords: disease progression modelling, LLM, spatio temporal modelling

Abstract:
Understanding the interactions between biomarkers across brain regions during disease progression is essential for unravelling the mechanisms underlying neurodegenerative disease. For example, in Alzheimer's Disease (AD) and other neurodegenerative conditions, there are typically two kinds of methods to contract disease trajectory. Existing mechanistic models describe how variables interact with each other spatiotemporally within a dynamical system driven by an underlying biological substrate often based on brain connectivity. However, such methods typically grossly oversimplify the complex relationship between brain connectivity and brain pathology appearance and propagation. Meanwhile, pure data-driven approaches for inferring these relationships from time series face challenges with convergence, identifiability, and interpretability. We present a novel framework that bridges this gap by using Large Language Models (LLMs) as expert guides to learn disease progression from irregular longitudinal patient data.  Our method simultaneously optimizes two components: 1) estimating the temporal positioning of patient data along a common disease trajectory and 2) discovering the graph structure that captures spatiotemporal relationships between brain regions. By leveraging multiple LLMs as domain experts, our approach achieves faster convergence, improved stability, and better interpretability compared to existing methods. When applied to modelling tau-pathology propagation in the brain, our framework demonstrates superior prediction accuracy while revealing additional disease-driving factors beyond traditional connectivity measures. This work represents the first application of LLM-guided graph learning for modelling neurodegenerative disease progression in the brain from cross-sectional and short longitudinal imaging data.

Decision: reject
Reviewer scores: 3, 1, 5, 6

Reviews and discussion:
<thread>
creation date:
1730354003543

thread_type:
official_review

<thread_object>
creator: reviewer_pz9o

<content>
summary:
The authors propose a novel LLM-guided spatio-temporal framework for modeling regional variable changes during disease progression. The key innovation of this paper lies in utilizing a large language model (LLM) to enhance graph initialization and regularization in both mechanistic and data-driven settings, demonstrating improvements in accuracy and convergence. The models were evaluated on regional tau-pathology propagation across the brain.

strengths:
1. The paper is well-written and engaging. The idea of leveraging LLM knowledge for graph initialization is particularly intriguing. Given the recent advancements in LLM, effectively incorporating this knowledge into fundamental clinical research is crucial, and the authors demonstrate this integration in a good way. 
2. The authors provide solid background information that helps readers easily grasp the context of their work. 
3. Sufficient detail is offered on critical components, including prompting strategies, clear mathematical definitions, and experimental details, allowing readers to understand the underlying processes. 
4. The evaluation of the model through critical edge numbers is both interesting and effective, showcasing the advantages of LLM-based graphs. 
5. The authors conduct an extensive analysis, covering prompt ablation, synthetic data, different contexts (both data-driven and mechanistic), and model convergence.

weaknesses:
1. I may have missed it, but the calculation of the metric "SSE" lacks a clear definition, making it challenging to assess its scale effectively. 
2. The experiments appear to be conducted on a single data split, without cross-validation or random splitting. Given the relatively small variances reported in Table 2, it’s difficult to evaluate the significance of differences and reproducibility. Incorporating additional splits and reporting standard deviations would enhance the findings. 
3. The representation of faster convergence (A.3), particularly between the second and third figures, could be clearer. Adjusting the scale might help elucidate this aspect. 
4. While the authors emphasize the importance of identifiability and interpretability, their discussion primarily revolves around reducing the number of edges in the graph. There is limited exploration or discussions on how the derived graphs enhance understanding of regional interactions and differences during disease progression compared to other graphs.
5. In figure 4, descreasing the critical edge number throgu “7-factor” prompt is not very clear to me. It seems that the other factors provided by LLM does not lead to much advantages. 
6. It appears that the model assumes a single typical progression path of phenotypes per disease, potentially overlooking disease heterogeneity. Recent research has demonstrated significant heterogeneity in disease progression, particularly in brain diseases, including amyloid and tau spreads.

questions:
1. In Section A.3, why is the summation limited to four types of brain connectivities rather than five? 
2. For clarification, when comparing Model 1 and Model 2, are the authors directly comparing the performance of these two methods, or are they comparing the graphs used in these models within the context of Method 3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730426926954

thread_type:
official_review

<thread_object>
creator: reviewer_yr4j

<content>
summary:
This paper introduces a novel spatio-temporal framework guided by large language models (LLMs) to model long-term disease progression on brain graphs, utilizing structure discovery from longitudinal, population-level patient data. A key innovation lies in addressing the unknown temporal alignment of observations by embedding the method within a dual-optimization framework. This framework simultaneously estimates the disease progression model—capturing common biomarker trajectories across a population—and performs structure discovery, learning a graph that represents the spatiotemporal interactions among biomarkers. The proposed approach achieves superior prediction accuracy, faster and more stable convergence, and enhances the interpretability of the model.

strengths:
This paper explores the use of Large Language Models (LLMs) as expert guides to enhance graph inference by integrating expert knowledge.

weaknesses:
1. There are a lot of typos, such as "To address these limitations, we **consder** use Large Language Models (LLMs) as
expert guides to enforce the graph inference with expert **knowlegdge**" and so on.
2. In Eq. (5), why **p**  is calculated from the **99th** percentile of the tau distribution at each region?
3. In Eq. (8), what does $D_{LLM}$ represent? Could you clarify the statement, "This not only accounts for the diffusion process along the white matter bundles but also considers additional factors related to tau accumulation from the knowledge base of various LLMs"? From Equations (7-9), I am not seeing how structural connectivity is addressed (i.e., the diffusion process along the white matter bundles).
4. In Eq. (10),  what does $X$ stand for? what's the relationship between this section and the previous sections? I can't follow this.
5. What's the dataset you use? How about the performance on other diseases? What's the sample size? How many time point for each subject do you use?
6. Insufficient experiments and lack of comparison of many disease progression modeling models.
7. This method claims to estimate regional tau-pathology propagation over the brain. Can authors provide the related propagation pattern in the brain over time?

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605386851

thread_type:
official_review

<thread_object>
creator: reviewer_kk5d

<content>
summary:
This work presented a proof-of-concept approach to model Alzheimer's disease (AD) using LLM. The idea is interesting. However, the feasibility and power of LLM in understanding the etiology of AD is not clearly demonstrated. Our current understanding on disease mechanism is still elusive in the AD field. Not sure how the proposed approach could advance our current understanding on AD. 

Another main concern is the data sample size. Neuroimaging data from ADNI is used in this work. There is only couple hundreds (<300) subjects having both modalities (structural connectome and pathology data). It is not clear how to address the issue small sample size.

Also, the data pre-processsing is not clear. Which brain atlas is used? Based on the node number, looks like the Desikan-Killiany Atlas. Please note this atlas does not include sub-cortical regions, means it is not a perfect brain parcellation for studying AD.

Result interpolation lacks biological underpinnings and supporting evidence from current literatures in AD.

strengths:
First try of LLM in understanding disease progression.

weaknesses:
Not clear how this approach distinct to other knowledge graph approaches. 
Lack of a killer application supporting the motivation why we need invent the new wheel for AD.
Figures are not very helpful to understand the method.

questions:
What is the essential motivation of employing LLM in modeling AD progression?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699388164

thread_type:
official_review

<thread_object>
creator: reviewer_erut

<content>
summary:
This paper presents a novel disease progression model for neurodegenerative diseases that leverages a mixture of large language models (LLMs) to infer brain region interactions for predicting temporal dynamics. The approach uses prompts to query LLMs about potential interactions between brain regions, which are then used to construct a graph that informs disease progression

strengths:
### Strengths

1. **Innovative Use of LLMs in Disease Progression Modeling (DPM)**:
The paper introduces a creative and potentially impactful approach by leveraging large language models (LLMs) for graph-based disease progression modeling  in neurodegenerative diseases.

weaknesses:
### Weaknesses 

1. **Vague Language and Disorganized Manuscript**:  
   The paper uses terms like "causal interactions" without justification. The use of graph learning alone does not establish causation, as this typically requires specific assumptions or experimental design elements, such as interventions or causal inference techniques. Without these, the inferred relationships are associative rather than causal. Also,  the term "long-term" is not clearly defined, despite its critical importance in temporal modeling of disease progression. The lack of a precise timeframe (e.g., years or decades) makes it difficult to understand the model's intended scope and applicability. A clearer definition of "long-term" would strengthen the paper. Additionally,  there are instances where equations or concepts are referenced before they are defined, disrupting the logical flow and making it difficult for readers to follow. For instance, equation 9 is mentioned in the text (Section 2.2) before it is officially introduced (Section 2.5.1). Finally, there are several typos throughout the manuscript. Minor notice, but important, evaluation measures like SSE should be introduced first and then utilized with abbreviation for more clarity.

2. **Lack of Comparisons with Baselines and Limited Qualitative Results**:  
   Although the authors cite several relevant baseline methods, such as Bellot et al. (2021), they do not provide empirical comparisons. Without benchmarking against these established approaches, it is difficult to assess the advantages of the proposed method. 
Also, there are no qualitative results with predicted trajectories that would showcase that the method works in several test examples.
Additionally, in section 3.3 and 3.4 of the results authors use only the Claude3.5 LLM. However, their proposed method is the mixture of the LLMs and not only a single LLM (Claude3.5). This inconsistency, weakens result's coherence  as well as the validity of the proposed method. 

3. **Questionable Prompt Strategy for LLMs**:  
   The approach relies on querying LLMs with prompts for each brain region to infer relationships, which raises several concerns:
   - **Scientific Validity**: General-purpose LLMs lack domain-specific scientific knowledge and are prone to hallucinations. This means that the generated weights or connections could be inaccurate or even fabricated.
   - **Scalability**: The strategy of creating prompts for each brain region does not scale well for fine grained ROI brain representations, making it computationally intensive and potentially inconsistent.
   - **Stability and Consistency**: Since LLM responses can vary with prompt phrasing, it’s unclear if these outputs are reliable across prompts. An analysis of prompt sensitivity or consistency checks would be necessary to validate this approach.

4. **Limited Longitudinal Data for Temporal Modeling**:  
   With only 254 samples from 185 subjects, it appears that most subjects have only a single acquisition (baseline) and one follow-up. This data limitation raises doubts about the model’s ability to learn meaningful temporal dynamics. The authors do not address how the model captures progression trends with such limited longitudinal data, leaving the temporal component of the model questionable.

5. **Limited Discussion of Practical Application and Real-World Feasibility**:  
    The paper lacks a detailed discussion of how the proposed Disease Progression Model could be applied in real clinical or research settings. The reliance on prompts and general-purpose LLMs may hinder broader adoption, as access to specific LLMs, prompt engineering, and technical resources may not be feasible in clinical environments. Addressing these practical challenges and offering guidance for real-world application would enhance the paper's relevance.

questions:
### Questions

1. **Handling of Repeated Measures**:  
   - "Authors mention that they leverage longitudinal population-data. How do you handle repeated measures in the training process?"

2. **Ratio Specification for Mixing LLMs**:  
   - "How do you specify the ratio (0.865:0.135) for mixing the two LLMs? Could you provide a rationale or any empirical basis for choosing this specific ratio?"

3. **Cross-Validation for Robustness**:  
   - "A simple splitting of the 255 subjects into train-test-validation is not enough. A fold cross-validation (train-test-validation) splitting multiple times would produce more powerful results. Could you comment on the reason for choosing a single split rather than cross-validation?"
</content>
</thread_object>

</thread>
<thread>
creation date:
1733071638026

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summarized responses to all the reviewers. Looking forward to further replies

comment:
We thank all reviewers for their time, effort, and valuable feedback, and we are looking forward to the follow-up replies to our responses. We appreciate the recognition of the novelty and innovation of our approach, particularly the use of Large Language Models as an expert to enhance graph learning for disease progression modelling (erUT, YR4J, Pz9o). We also value the positive feedback on the clarity of our background information, clear mathematical framework, and comprehensive analysis, covering prompt ablation, synthetic data, different contexts (both data-driven and mechanistic), and model convergence (Pz9o).
In response to the constructive critiques, we have made several improvements to the manuscript, addressing concerns across clarity, methodology, and experimental rigour. Key updates include:

**1.	Small Dataset Concerns (Kk5d, YR4J, Pz9o):**
- Contextualized our sample size within field standards, citing comparable recent studies using similar or smaller cohorts
- Explained how our framework addresses sparse data challenges through:

   * Simple parameterisation in mechanistic models
   * LLM-derived constraints reducing parameter count
   * The construction of long-term trajectories from irregular, sparse and limited snapshots is the precise contribution of our proposed framework to address such a problem.

**2.	Experiments for Graph Validity and Model Comparison:**
 - Included 3-fold cross-validation results in Tables 1 and 2. (erUT, Pz9o).
- Clarified the experiments had been included in our initial submission, including the comparison with Bellot et al. (2021) in Table 2, validating the LLM graph by comparing it with the existing biological graphs (Appendix A.3) and other baseline models of graph inference for time series (Appendix A.1) (erUT). 
- Added more results using the mixture of LLM in Table 2  (erUT).
- Emphasized how LLM constraints improve graph stability and reduce overfitting in mechanistic and data-driven models. Added Figure 3, showing that the initial methods without expertise provide different graph inference outcomes. (Kk5d, Pz9o).

**3.	Clinical Application:**
- Clarified the clinical applicability of the proposed framework, particularly in disease staging and mechanistic insights into tau propagation. Figures 9 and 10 were added to describe the disease staging and quantitative visualization of predicted disease patterns in Appendix A.5, with a discussion on practical applications in disease staging and diagnosis.
- Referenced recent reviews highlighting improving connectome accuracy and understanding as a key challenge and explained how we improve graph inference and interpretation using LLMs. (erUT, Kk5d).

**4.	Clarified the Motivation for using LLM (Kk5d)**
- Cited recent publications validating LLM applications in brain research and LLMs' unique capabilities in integrating multi-modal brain data and analysing brain-behaviour relationships and network dynamics.
- Explained how expertise from LLMs addresses limitations of both mechanistic models (combining multi-modal connectomes to obtain high accuracy and fewer learnable parameters) and data-driven approaches (enhancing both identifiability and interpretability with faster and smoother convergence)
- Emphasized that our goal is to improve the graph component in existing models, which has been a known limitation waiting to be solved, mentioned by several nature neuroscience review papers.
- Explained the advantages of LLM compared with traditional knowledge graph

**5.	Manuscript Clarity:**
- Enhanced logical flow and definitions in Tables 1 and 2 (erUT, YR4J).
- Added detailed explanations and corrected typos and unclear notation (YR4J, Pz9o).

We hope these enhancements address the reviewers’ feedback and further highlight the rigor and applicability of our work.  We thank the reviewers again for their constructive insights and the AC for considering our submission. Any further discussion from the reviewers would be highly appreciated.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers raise significant concerns about the scientific validity and clinical applicability of the proposed LLM-guided framework for modeling Alzheimer's disease progression. Key issues include the lack of rigorous evidence supporting the biological relevance of LLMs, small datasets, insufficient experiments, limited baseline comparisons, and the inability to address disease heterogeneity. While the rebuttal provided some clarifications, it failed to resolve doubts about the reliability of the LLM-inferred graphs and their clinical feasibility.