Joint Adaptation of Uni-modal Foundation Models for Multi-modal Alzheimer's Disease Diagnosis
Authors: Wentao Gu
Yuquan Li
XINYANG JIANG
Zilong Wang
Dongsheng Li
Zehui Li
Zijian Dong
Cairong Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: Artificial Intelligence for sciences; Alzheimer's disease; multi-modal diagnosis; Foundation Models
Primary area: applications to neuroscience & cognitive science
TL;DR: We propose a multi-modal framework for Alzheimer’s Disease diagnosis that facilitates interaction among four modalities and their foundation models.

Abstract: Alzheimer’s Disease (AD) is a progressive neurodegenerative disorder and a leading cause of dementia worldwide. Accurate diagnosis requires integrating diverse patient data modalities. With the rapid advancement of foundation models in neurobiology and medicine, integrating foundation models from various modalities has emerged as a promising yet underexplored direction for multi-modal AD diagnosis. A central challenge is enabling effective interaction among these models without disrupting the robust, modality-specific representations learned from large-scale pretraining. To address this, we propose a novel multi-modal framework for AD diagnosis that enables joint interaction among uni-modal foundation models through modality-anchored interaction. In this framework, one modality and its corresponding foundation model are designated as an anchor, while the remaining modalities serve as auxiliary sources of complementary information. To preserve the pre-trained representation space of the anchor model, we propose modality-aware Q-formers that selectively map auxiliary modality features into the anchor model’s feature space, enabling the anchor model to jointly process its own features together with the seamlessly integrated auxiliary features. We evaluate our method on AD diagnosis and progression prediction across four modalities: sMRI, fMRI, clinical records, and genetic data. Our framework consistently outperforms prior methods in two modality settings, and further demonstrates strong generalization to external datasets and other neurodegenerative diseases such as Parkinson’s disease.

Published as a conference paper at ICLR 2026
JOINT ADAPTATION OF UNI-MODAL FOUNDATION MODELS FOR MULTI-MODAL ALZHEIMER'S 'S DISEASE DIAGNOSIS
Wentao Cu Yuquan Li
Xinyang Jiang2
Zilong Wang"
Dongsheng Li
Zehui Li
Zijian Dong
Cairong Zhao
School of Computer Science and Technology. Tongji University
Microsoft Research Asia
3 National University of Singapore
ABSTRACT
Alzheimer's Disease (AD) progressive neurodegenerative disorder and a leading cause of dementia worldwide. Accurate diagnosis requires integrating diverse patient data modalities. With the rapid advancement of foundation models in neurobiology and medicine, integrating foundation models from various modalities has emerged a promising yet underexplored direction for multi-modal AD diagnosis. A central challenge is enabling effective interaction among these models without disrupting the robust, modality-specific representations learned from large- scale pretraining. To address this, we propose a novel multi-modal framework for AD diagnosis that enables joint interaction among uni uni-modal foundation models through -anchorr interaction. In this framework, one modality and its corresponding foundation model are designated as an anchor, while the remaining modalities serve as auxiliary sources of complementary information. To preserve the pre-trained representation space the anchor model, we propose modality- aware Q-formers that selectively map auxiliary modality features into the anchor model's feature space, enabling the anchor model to jointly process its own features together with the seamlessly integrated auxiliary features. We evaluate method on AD diagnosis and progression prediction across four modalities: sMRI, fMRI, clinical records, and genetic data. Our framework consistently outperforms prior methods in two modality settings, and further demonstrates strong generalization to external datasets and other neurodegenerative diseases such as Parkinson's 's disease.
INTRODUCTION
Alzheimer's disease (AD) (Li et al. 2022b Cahill 2020 Ning et al. 2021 Association 2019 Dong jet al. 2024) one of the most common causes dementia worldwide, leading to a progressive decline in cognitive function that significantly interferes with daily activities (Xue al. 2024 Qiu ct al. 2022) among the elderly. With the growing aging population, there is a pressing need for more precise early detection and progression prediction of Alzheimer's 's discase.
While deep learning has advanced AD analysis, most existing studies focus on developing models on uni-modal data (Liu et al. 2024). such medical imaging (Wang al. 2023) or cognitive
assessment scores (Fang et al.] 2024). However, Alzheimer's 's disease is a complex neurodegeneratiie disorder involving diverse and interacting pathological mechanisms, reflected across multiple data modalities. Structural MRI (sMRI) highlights brain atrophy (Ferreira & Busatto (011)), functional MRI (IMRI) captures neural activity (Dong et al. (2024). non- non-imaging data (including demographics and clinical assessments) reflect overall patient status (Li et al.) 2024a). and genetic data reveal hereditary risks (Beebe- Wang et al., 2021). Recent AD guidelines from NIA-AA also emphasize the necessity of integrating biomarkers from multiple modalities, as each provides complementary
"quual contribution. Work done during Wentao's internship at MSRA.
Corresponning Authors. Email: xinyangjiang mierosoft.com whwncairong@torngg wooggi...iooniii.....n

---

Published as a conference paper at ICLR 2026
and valuable insights (Jack Jr et al., 2024). Therefore, integrating complementary information from multiple modalities is essential for a comprehensive understanding and accurate prediction of AD.
While conventional multi-modal approaches have predominantly relied training models from scratch (Xue et al. 2024 Qiu et al. 2022; Feng et al. (023a), deep learning is undergoing a paradigm shift toward leveraging large-scale, pre-trained foundation models for downstream tasks adaptation. This approach is particularly advantageous in medical domains like AD, where labeled data are scarce and more efficient and robust learning method is required. A range of uni-modal foundation models in neurobiology and medicine show strong potential for enhancing AD diagnosis, such as BrainMVP (Rui et al. 2025) and Brain-JEP: (Dong et al. 2024) for brain imaging, large language models for clinical records, and gene foundation models (Dalla- Torre et al. 2025) for genomics, all demonstrating strong performance in their respective domains.
Despite the availability of powerful uni-modal foundation models, integrating them into unified multi-modal AD diagnosis framework remains a significant challenge. The core difficulty lies in effectively enabling meaningful interaction among foundation models, which requires aligning their feature spaces and integrating their outputs to leverage complementary information. Since each foundation model is pre-trained to capture distinct, modality-specific features, their representations are inherently heterogencous and well-structured. Naively aligning or merging these spaces may compromise their integrity and reduce their effectiveness.
As result, our goal is to strike a balance between enabling sufficient interaction among foundation models and preserving the integrity of their pre-trained feature spaces. To address this, propose a unified multi-modal framework for Alzheimer's 's Disease diagnosis based on modality-anchored foundation model interaction. Specifically, we designate one modality's 's foundation model as an anchor and freeze most of its parameters to preserve its feature space, while projecting auxiliary modalities' features extracted by other foundation models into this space for cross moosl interaction. This alignment is achieved by our Modality- Q-formers (Tong et al. 2024 Zong et al., 2024] Alayrac et al., 2022, Liu al., 2023a), which use learned queries to selectively extract relevant information from the auxiliary modalities and project it to anchor model feature space, enabling the anchor foundation model to jointly process them with the anchor features. Modality-anchored interaction is applied to each modality in turn, and final predictions are aggregated, allowing to retain the strengths of each foundation model while enabling effective multi-modal integration.
In experiments, the proposed method is evaluated on AD diagnosis and progression prediction tasks involving the four most common data modalities (i.c., sMRI, fMRI, clinical records, and genetic data). By integrating four uni-modal foundation models, method achieves state-of toofthhhaatttt performance under both modality-comple et al. 2005). We further evaluate our approach on the external OASIS (LaMontagne et al.) 2019) and PPMI (Marek et al., 2011) datasets, where it achieves state-of-the-ar performance and shows strong generalization to both out-of-distribution AD diagnosis tasks and other neurodegenerative diseases.
and modality-incompler
scenarios from the ADNI dataset (Mueller
2 RELATED WORKS
Multi-modal Fusion Methods for AD Diagnosis. Alzheimer's Disease (AD) is a complex neuro disorder, and its accurate diagnosis requires multi-modal data integration. Early efforts focus on neuroimaging data combination. Modalities such as MRI, PET, IMRI, and DTI are integrated via methods including shared representations (Ning et al. 202T). GCNs (Song et al., 2022). and 3D networks (Qiu et al. 2024). Subsequent studies integrated non-imaging data, for instance, by using LLMs or deep learning to combine MRI with cognitive scores (Hett et al.) [02T |Fng et al.) 2023b)
Qiu et al. 2022. Chen & Hong, 2024, Xue et al., 2024 Methodologies also evolved to address data challenges like missing modalities (Liu et al. 2023c) or limited labeled samples (Feng et al.. 2023ab. While prior methods often relied a limited subset of modalities, our framework is the first to incorporate all three major types of AD-related data: genetic, neuroimaging. and clinical. Our broader modality coverage, combined with the adaptation of foundation models, enables more comprehensive understanding of AD pathology and improves diagnostic accuracy.
Adaptation Methods on Foundation Models. The development of foundation models has sig- nificantly impacted healthcare by enabling powerful uni-modal data analysis. In medical imaging.
diverse foundation models are utilized for imaging analysis (Wang al. 2023 Rui et al. 2025
Caro
2

---

Published as a conference paper at ICLR 2026
al.
2023 Dong et al. 2024). and shows impressive performance in downstream tasks. In genomics,
foundation models pre-trained on DNA sequences (Dalla- Torre et al. 2025 Nguyen et al. 2023
Zhou et al. 2023 have shown considerable success in cross-species genomic modeling and analysis.
In clinical records, previous efforts (Ben Shoham & Rappoport 2024, Singhal et al. 2023 Li et al. (0224a) either focus scaling predictors attempting adapt LLMS clinical predictions. While powerful uni-modal foundation models exist, how to effectively leverage them remains underexplored. Efforts (Zhang et al. 2023b) like M4Survive (Lee et al.) 2025) try integrate medical foundation models using symmetric late-fusion, which may hinder deep inter-modal interactions. In contrast, our work targets more effective interaction across various uni-modal foundation models.
Q-formers in Multi-modal Pretrained Models Prior studies primarily used query transformers
(q-formers) Li et al. (2022a) or connectors Liu et al. (023a) to project non-text modalities, such as images, video, or audio, into the text embedding space of large language models (LL.Ms) for multimodal alignment. For example, BLIP-2 al. (2023) and MiniGPT-4[Zhu et al. (2023) use query transformer to extract features from image patches and output query embeddings that the LLM consumes. InstructBLIP Dai et al. (2023) extend this approach to fuse images, video, and audio, with separate query transformers for each modality projecting their features into the LLM text space. Similarly, speech-, video-, and visual models like EmoQ Yang & Mak (2025), Video-LLaMAA Zhang et al. (2023a), and MMS-LLaMA Yeo et al. 2025) employ query transformer to compress their own modality embeddings into textual representations for LLM processing. Differing from those prior works, where query transformers project each modality exclusively into the text embedding space of LLMs, our moddlityy-nnhhove the anchor, with the remaining three modalities serving as auxiliary modalities. Consequently, our Q-former is designed to be more general, capable of projecting into any of the four modality spaces when designated as the anchor, rather than being restricted to text.
interaction sequentially treats each modality as
METHOD
In this section, we present the methodology of our proposed multi-modal framework for Alzheimer's Disease diagnosis. The overall pipeline to train a multi-modal AD diagnosis model contains two stages. The first stage, uni-modal foundation model adaptation described in Section 3.1 individu- ally fine-tunes each foundation model on its respective modality data to extract highly expressive, modality-specific features for the AD diagnosis tasks. In the second stage, given the uni-modal AD diagnosis models obtained in the first stage, a Modality-anchored Foundation Models Interaction Strategy (elaborated in section (322) adopted to enable interaction among uni-modal models without compromising the integrity of feature space from each model. This is achieved by aligning the feature of auxiliary modalities to the primary modality feature space with Modality-aware Q-formers (elaborated in section 3.3). designed to learn set of learnable queries to extract relevant information from the auxiliary modalities. Finally, our method gives the final AD diagnosis predictions by combining the outputs of the fine-tuned poodality-specifff foundation model.
3.1 UNI-MODAL FOUNDATION MODEL ADAPTATION
In the first stage, the primary goal is to adapt each uni-modal foundation model to AD diagnosis using limited labeled data from its corresponding modality, leveraging the model's inherent strong representations.
Problem formulation Our objective is predict an individual's Alzheimer's disease status or prodromal progression patterns using multi-modal inputs. In this paper, we focus on four data modalities denoted as m M {s, f, c, g). where s and f refer to neuroimaging data sMRI and fMRI, с refers to clinical records and g refers to genetic data. As depicted in stage 1 in Fig I given for modality m, where x:"" is the input sample, y/ is the corresponding diagnosis label, Nm, is the size of datasct for modality m. All uni und-modal foundation models used for AD diagnosis are based on transformer architectures. Each model (., (..)))))) is attached with a linear classification head that takes the output class token from the transformer predicts the diagnostic label and is fine-tuned individually using the standard cross-entropy loss:
training dataset Dm {x . }r) Nm
Nm
Lm =
LCe (Fm (x: pm), yi)
(1)
Nm 1=1
3

---

Published as a conference paper at ICLR 2026
Modality Adapton
b) Stage Modalig-anchoodd Interaction
Toxt F-andaatin
Test
Model
Litena
MMddilityy Aware QFFommer
Gento Frunndike
Toleatic
Mode
BB
NT
Dres Atarki
XMRI Fountation
Model
Drein
Auilaa
Coss Altirtin
IMRI
TMRR Foumbalion
Ma Engcal Mlmnee
Briin- JUPA
Model
AD diagnosis framework. a) In the first stage, the uni-modal foundation model is finetuned to AD diagnosis using limited labeled data from its corresponding modality. b) In the second stage, four uni-modal AD diagnosis models from the first stage are further finetuned to enable cross-modality interactions by modality-anchored interaction. Modality-aware Q-former (Auxiliary Q-Former stage 2) a munsformer-based connector that selectively projects the features from the auxiliary modality to the feature space of the anchor model.
Figure 1: The overall pipeline of the proposed
Next, we detail the process of adapting each foundation model and its corresponding uni-modal data for the AD diagnosis task, including the choice of foundation model and the preprocessing steps applied to each input modality, as illustrated Stage of Fig
Neuro-image Modalities Neural-image modalities include structural MRI and functional MRI data. For structural MRI, adopt BrainMVP (Rui et al. 22025) large-scale medical imaging foundation model trained on volumetric data with rich anatomical priors. For functional MRI, we employ Brain-JEPP (Dong et al. 2024), which learns generalizable representations of brain dynamics through predictive learning across IMRI datasets.
Gene Modality For genetic data, we adopt the NT Transformer (Dalla-Tore al. 2025 a genomic foundation model pre-trained on nucleotide sequences to support tasks such disease association and phenotype prediction. The genetic training dataset is constructed with whole genome sequencing (WGS) studies for AD, which provides base-pair level coverage of the entire genome, allowing for comprehensive assessment individual genetic variation. We provide full details variant selection for genetic input and sequence representation constructing pipeline provided in the Appendix A.2
Textual Modality The textual modality encompasses subjects' comprehensive clinical records, including demographic information, cognitive assessments, neuropsychiatric symptoms, functional abilities, and medical history. To model this modality, employ LLaMA-22 (Touvron al. 2023), large language model capable capturing latent patterns in both textual and tabular clinical features. We adopt a text template strategy convert clinical records into description and task prompts. The converting pipeline is detailed in the Appendix A.
3.2 MODALITY-ANCHOOE FOUNDATION MODELS INTERACTION
As shown in Fig 1 in the second stage of the multi-modal training pipeline, fine-tune the four uni-modal AD diagnosis models from the previous stage introduced in section 3.1 to enable foundation model interaction.
Instead of allowing all uni-modal models to interact equally, the modality-anchored anchored interaction designates modality the anchor modality along with its corresponding anchor model obtained
4

---

Published as a conference paper at ICLR 2026
from stage 1. For specific anchor modality m, its auxiliary modalities refer to all remaining modalities whose features will be aligned to the anchor modality's 's feature space, denoted as M' {m E MMm m). Given set of input signals, including anchor modality input xth and auxiliary modality inputs (X m E }. we first extract features from the auxiliary inputs using their corresponding uni-modal AD diagnosis models obtained from Stage 1. The extracted features from the auxiliary modalities are then aligned to the feature space of the anchor model F' using the proposed modality-aware Q-former in section 3.3 Specifically, the aligned auxiliary representation Ha is computed as follows:
Hd Qformer (Concat((F((( (mmx))))))))))))))
(2)
where Concat(-) denotes the concatenation of the input features vectors, Qformer(-) refers to the modality-awan Q-former elaborated in the next suctsection 3.3 Then the interaction is achieved by feeding the aligned auxiliary modality tokens H into the anchor model Fm alongside with the anchor modality input produce the diagnosis prediction. Finally, the anchor model Fa. is further fine-tuned with a standard cross-entropy loss:
Non
P
Loe (Fa (Concat(X*) HH))). (H))))
N
Building on the modality-anchoree interaction described above, we fine-tune each uni-modal model Fm from Stage 1 by designating it the anchor modality. while treating the remaining modalities as auxiliary inputs. To better preserve the original feature space of the anchor model, we apply LoRA (Hu al., (022) fine-tuning, where only small subset of parameters is updated. The final diagnostic prediction is obtained by aggregating the outputs from all fine-tuned models.
3.3 MODAIJIT--WARE Q-FORMERS
As shown in Equation 2 allow effective interaction between the anchor model and the auxiliary models, a transformer-bast based connector is proposed to selectively project features from the auxiliary modality to the feature space of the anchor model, called modality Q-former. As illustrated in [ig[ our modality-aware Q-former incorporates two types of information, namely uni-modal and cross modal information.
Uni-modal Q-formers
Modality-aware Q-former first extracts the uni-modality information from a specific auxiliary modality m E .M'. Specifically, we create a set of learnable tokens to serve as uni- uni-modality queries, denoted as Xug N Given the auxiliary features extracted from the corresponding auxiliary model Fn, we first project them to the same dimension as the anchor modality:
zm Linear (F RL"
Then, the learnable uni-modal queries interact with the projected features through cross- cross-attention layer, which further projects the auxiliary modality features into the anchor feature space and extracts information relevant to the anchor modality from the auxiliary one m:
X m CrossAttn(Q X Zm ZT
(5)
The resulting output Xm modality m.
R NNC are features containing uni-modi information from auxiliary
Cross-modal Q-former Besides uni-modal information, further propose set of cross-modal queries Xeg R.N N.N xC that enables feature interaction among all auxiliary modalities. Specifically, the
cross-modal queries interact with all the output tokens of uni-modal Q-formers (X*m/m M') with a cross-attention layer to capture cross-modality correlations among different auxiliary modalities, resulting in the cross-modality auxiliary features denoted as Xc:
X CrossAttn(Q Xeqx K ==,V=22 =zz),
(6)
where
7 Concat (
(7)
5

---

Published as a conference paper at ICLR 2026
Table I: Results of three AD prediction tasks across three ADNI cohorts. Experiments conducted under the Modality-compiet setting, which includes only individuals with all four data modalities available. (C: Clinical records, F: IMRI, S: SMRI, G:Genetic data) The best results are in bold.
Modality
Method
NC
sMCI vs. PMC1
ACC
SPE
SEN
ACC
SPE
SEN
ACC
SPE
Uni-Modality
0.557
C
LLaMA 2
0.793
0.854
0.640
0.814
0.879
0.687
0.721
0.809
0.574
F
Brain-JoPA
0.777
0.838
0.542
0.807
0.857
0.576
0.714
0.723
0.522
F
BrainLM
0.768
0.809
0.537
0.781
0.841
0.575
0.705
0.735
0.509
$
BrainMVP
0.724
0.819
0.589
0.730
0.832
0.669
0.703
0.774
0.601
S
SamMed3D
0.714
0.807
0.597
0.714
0.814
0.675
0.689
0.758
0.607
S
Swin-UNETR
0.609
0.628
0.495
0.612
0.724
0.579
0.521
0.595
0.503
S
M
0.647
0.665
0.538
0.671
0.778
0.609
0.622
0.666
0.591
6
NT-Human
0.694
0.775
0.751
0.857
0,492
0.652
0.719
0.424
G
SEI
0.483
0,500
0.462
0.568
0.680
0.491
0.415
0.657
0.342
DNA-Ber12
MultiMoodlity
C+G+14S
MaSurvive
0.827
0.865
0.568
0.804
0.879
0.657
0.746
0.840
0.557
C+G+R+S
LateFusion
0.818
0.894
0.433
0.798
0.867
0.581
0.714
0.782
0.582
C+G+F4S
Ours
0.871
0.921
0.700
0.846
0.902
0.707
0.763
0.876
0.617
Table Results of three AD prediction tasks under the Modality-incompple setting. * denotes the use of pretrained weights from the original paper for evaluation.
Modality
Method
NC MCT
NC vs. AD
sMCI PMCT
ACC
SEN
SPE
ACC
SPE
SEN
bi-Modality
S+C S+C S+C S+C
Neommns
0.945
0.932
0.947
0,928
0.939
0.911
0.773
0.766
0.698
Al-dingnosis
0.924
0.947
0.938
0.910
0.920
0.890
0.766
0.839
0.574
Al-diagnosis
0.950
0.937
0.955
0.924
0.939
0.895
0.825
0.849
0,740
SMART
0.932
0.943
0.877
0.917
0.944
0.891
0.810
0.832
0.768
Multi-Modality
C+G+T4S
M4Survive
0.926
0.921
0.931
0.911
0.936
0.851
0.812
0.879
0.652
C+G+F4S
LateFusion
0.881
0.927
0.861
0.899
0.912
0.879
0.801
0.871
0.693
C+G+F4S
Ours
0.979
0.957
0.963
0.945
0.960
0.931
0.846
0.901
0.711
Finally, the cross-modal auxiliary feature X and set of uni-modal auxiliary features {X m M'} are concatenated to obtain the final output of the modality Q-former:
Hº Concat ({Sm (8'"))MM...... ..EEM..)))) E BANNxC.
(8)
4 EXPERIMENTS
4.1 DATASETS
We leverage the ADNI (Mueller et al. 2005) dataset evaluate our method AD diagnosis and progression prediction task. Alzheimer's 's Disease Neuroimaging Initiative (ADNI) offers the most comprehensive set of modalities, including structural and functional MRI (sMRI and fMRI), genetic data, and textual and tabular clinical records. ADNI includes participants across three main diagnostic categories: normal controls (NC), mild cognitive impairment (MCI), and Alzheimer's Disease (AD).
We use two datasets for externa evaluation: PPMI (Marek et al. 201 dataset focuses Parkinson's 's disease, providing the same set of modalities as ADNI. It includes subjects across three diagnostic categories: normal controls (NC), mild cognitive impairment (MCI), and Parkinson's 's disease (PD). OASIS-3 (LaMontagne et al., 2019) is a multi-modal dataset providing sMRI, fMRI, and clinical records, but unlike ADNI, it lacks genetic data. It comprises subjects diagnosed normal controls (NC) and Alzheimer's 's disease (AD). Further details on the three datasets and preprocessing pipelines for the four modalities are provided in the Appendix A

---

Published as a conference paper at ICLR 2026
Table 3: Results of PD prediction tasks on PPMI.
Table 4: AD prediction task on OASIS.
Modality
Method
NC vs. MCI
|
NC vs. PD
pPD vs sPD
Modality
Method
NC vs. AD
ACC
AUC
|
ACC
AUC
ACC
AUC
ACC
AUC
C
LLaMA 2
0.857
0.839
0.913
0,909
0.694
0.681
C
LLaMA 2
0.697
0.650
Brain-JaPA
0.808
0.797
0.669
Bruin-
0.681
0.667
S
BrainMVP
0.781
0.783
0.898
0,882
0.647
0.650
S
BrainMVP
0.655
0.621
NT-Human
0.633
0.629
0.745
0.750
0.615
0.603
G
NI-Human
0.491
0.489
S+C
Noomms
0.867
0.854
0.919
0,922
0.707
0.711
S+C
Ncomms
0.662
0.636
S+C
SMART
0.892
0.888
0.940
0,943
0.751
S+C S+C
SMART
0.701
0.679
S+C
Al diagnosis
0.909
0.905
0.934
0,948
0.731
0.739
Al diugmusis
0.705
0.688
C+G+F48
M4Survive
0.889
0.910
0.954
0,968
0.752
0.753
C+G+F+S C+G+F+ C+G+F48
M4Survive
0.722
0.640
C+G+F4S C+G+F+S
LateFusion
0.860
0.846
0.940
0.951
0.707
0.719
LateFusion
0.694
0.648
Ours
0.927
0.944
0.967
0.969
0.769
0.773
Ours
0.722
0.699
4.2 EXPERIMENTAL SETTINGS
In this study, we focus on two types of multi-modal diagnostic tasks: Alzheimer's disease (AD) diagnosis and prediction of prodromal progression, evaluated under both modality-complee and moddllty-incomplee settings.
AD prediction. For AD diagnosis evaluation, we follow established practices (Ning et al. 2021 Song et al., [202 Hett et al., 202T, Feng al. 2023a) by assessing performance two binary classification tasks: normal controls (NC) vs. Alzheimer's Disease (AD) and NC vs. mild cognitive impairment (MCI). All models are evaluated under identical conditions ensure fair comparison using standard performance metrics: accuracy (ACC), specificity (SPE), and sensitivity (SEN).
Prodromal progression prediction. We extend the AD diagnosis task into more challenging task of predicting AD progression (Rahim et al., 2023 El-Sappagh et al.) 2020) by distinguishing between stable MCI (sMCI) and progressive MCI (pMCI). Both sMCI and pMCI patients are initially diagnosed with MCI at baseline, but the cognitive condition of sMCI group remained stable and did not convert to AD within 36 months after their first visit. In contrast, the pMCI group progressed to a clinical diagnosis of AD during the same 36-month follow-up up period. (Ning et al. (221) This task demands subtle discrimination between individuals with different temporal outcomes.
Modality setting. To comprehensively assess the robustness of our framework, we evaluate its performance on all three tasks under two different modality settings: Modality-complete setting includes only individuals with all four data modalities available. Under this strict requirement, from the ADNII, ADNI2, and ADNI3 cohorts, we obtained 414 CN, 68 AD, and 273 MCI samples for two AD prediction tasks, and 182 sMCI and 44 pMCI samples for the progression prediction task. -nncoplleei setting includes individuals with at least one available modality, better reflecting real-world clinical scenarios and enabling fuller data utilization. We collected 898 CN, 416 AD, and 986 MCI samples for AD prediction, as well as 220 sMCI and 81 pMCI samples for sMCI vs. pMCI.
Cross-domain generalization setting. To assess generalization ability, we conduct dfitt--diittibutiinn (OOD) evaluation under modality compleee setting. Our framework is trained on NC vs. AD data from ADNI. For OOD testing, we use 120 NC and 42 AD subjects from OASIS-3, all of which are unseen during training and with all three modalities (sMRI, IMRI, and clinical records) available. Predictions are obtained by combining the outputs of the three corresponding foundation models trained on ADNI. To further demonstrate the adaptability of our method to other brain diseases, we train our framework on PPMI dataset and evaluate on Parkinson's 's disease (PD) diagnosis and progression prediction. Similar to AD, PD prediction assesses the performance on NC vs. PD and NC: vs. MCI tasks, and prodromal progression prediction of PD follows the same protocol to distinguish between stable PD (sPD) and progressive PD (pPD). Under modality-complete setting, obtained 743 NC, 329 PD, and 143 MCI samples for NC vs. PD and NC vs. MCI classification, as well as 225 sPD and 104 pPD samples for sPD vs. pPD classification.
Implementation Details. In our framework, each modality is processed by a dedicated pre-trained foundation model: LLaMA2-13B 13B (Touvron et al. 2023) for clinical records, Brain-JEPs (Dong et al.) 2024) for fMRI, BrainMVP (Rui et al.] (2025) for sMRI, and NT-500M (Dalla Torre et al. (025) for genetic data. Full details of models, data split, and hyper parameters are in the Appendix
7

---

Published as a conference paper at ICLR 2026
NC MCI Prediction
NC AD Prediction
SMCI PMCI Prediction
0.98
1.99%
0.94
2.88%
0.84
1.51%
0.92
0.03
Perforannc
Performance
Reformanc
3.082
0.02
0.90
0.90
0.88
0.80
3.59%
9.64%
ACC
10.9%
0.08
0.79
-8- AUC
-9-
AUC.
-8-
AUC
Ours
Q-former
Proj
late Fusion
Ours
Q-former
Proj
late Fusion
Ours
Q-former Proj late Fusion
Figure 2: Analysis of Anchor and Auxiliary modality fusion methods for AD Diagnosis on ADNI
Neuroimage unimodal
Non-imaging
Clinical TextualTexxtul
Neuroimage bimodal
Multi- modal
0.50
unimodal
bimodal
0.00
0.75
0.70
0.65
00
P-A
PoRO
6-W
64900
Cul
GMO
C10-00
Ci0-40
Cro.wo
Cin-40
$40-00
and wet
PAGMO
CrGcsin-40
Cxo- Hei
S
Figure 3: Overall performance trend with different modality combinations. C: Clinical records;
F: fMRI; S: sMRI; G: Genetic data. Red striped bars indicate performance where one modality is removed from a given multi-modal combination. Figure Details are in the Appendix D.7
4.3 MAIN RESULTS
Baselines. We compare our proposed method against representative uni-modality (Rigatti 2017 7
Touvron et al. 2023; Dong et al. 2024; Caro et al., 2024| Rui et al., 2025 Wang et al. 2023
Tang et al. 2022; Liu et al. 2023b Nguyen et al. 2023 Chen et al. 2022 Zhou et al.
2024p.
bi-modality Xue et al. 2024 Chen & Hong 2024 Qiu et al. 2022). and multi mudii-mmdality models (Lee
et al. 2025 on diagnosis and progression prediction tasks. Baselines' details are in the Appendix D.I
AD classification Results. We compare our method with SOTA baselines under both modality- complete and incomplete settings on ADNI, with results presented in Table and 2 Overall, our method outperforms all baselines in overall accuracy across all three AD classification tasks under two settings. Compared conventional Multi-Modal methods including M4Survive and Late Fusion, our Modality-anchored Interaction ensures the anchor modality feature space to be preserved while integrating sufficient auxiliary features into the anchor representation. Therefore, our joint interaction and adoption framework is robust to both limited data availability and missing modalities. Modality- anchored interaction between anchor and auxiliary modalities preserves well-structured feature space of each pre-trained model and avoids representation degradation caused by missing modality inputs. For each modality, we select the most recent and representative foundation models and evaluate their uni-modal performance on the ADNI dataset in Table [ Among the basclines, LLaMA2 (clinical records), BrainMVP (sMRI), BrainJePA (IMRI), and NT-Human (genetic data) achieve the highest overall performance for their respective modalities.
Cross-Domale Domain Generalization. Table 3 demonstrates that our method generalizes effectively beyond AD, significantly outperforming both uni- and multi-modal baselines in Parkinson's 's disease diagnosis on the PPMI dataset. Similarly, Table 4sowss that when transferring a model trained on ADNI to the dittr--iiittibttooon dataset like OASIS-3, our approach achieves sate-of-the-ars AUC performance.
8

---

Published as a conference paper at ICLR 2026
Table Fusion baselines comparison un- Table 6: Ablation study on choice of foundation models der modality-incomplee setting on ADNI. for each modality on ADNI.
Methods
NC
Modality
Foundation Medels
NC MCI
N.WAAD
SMCC PMC1
Fusion Banelines
ACC
ATC
ACC
AUC
Sammal
0,91
0008
00002
4.021
Feuture Concatonation
0.894
0.855
0.833
0.846
DNA-Ben
0,980
Linear Fusion
0.881
0.851
0,899
0.811
0019
0003
4,016
Self Seli-Altenion Fusion
0
0.917
0,901
0.561
Branl
0,58
Modalit-anee anchored interaction (Chrs)
pro-trained
0.969
0.944
Clinical records
Modl ModJemau
Selerges or 2025
0,577
0331
40014
0313
4.811
BraisMVP
NT
Liama2
Subnetwerk Visualization
1.0
0.6
0.4
0.0
Aachor (S)
PMRI Anchar Genetic (G)
(I)
(d)
Anchor G
Attemboe
Antention
DMM
M6
CW
DWd
Der
Dale
DaN
Natwork
Netwoka
Networks
Natworks
Figure 4: Attention maps of fMRI images across three anchor modalities and their models. Attention across seven brain networks is computed for NC/MCI classification.
These results highlight both the adaptability of our framework across different neurodegenerative diseases and its strong generalization ability to unseen AD datasets.
4.4 ABLATION STUDY
Effectiveness of key components. To assess the effectiveness of our proposed modality-aware Q-former, we replace it with two alternative fusion strategies in Fig. 2 (1) Linear Projection (Proj), where features from each modality are projected to a shared space and concatenated; (2) Q-Former, where features from each modality interact only with their own learnable query tokens via attention and are then concatenated. While both Proj and Q-Former surpass the late-fusion baseline, their ability to align heterogeneous modalities is limited. Our proposed modality-aware Q-Former consistently outperforms those baselines, demonstrating the effectiveness of our cross-modal query design in extracting synergistic information while preserving the structured representations of each modality,
we compare our modality-anchored interaction with three fusion baselines using the same pre-trained foundation models. Feature Concatenation: concatenate features from all modalities. Linear Fusion / Self-Attention: project features from each modality into a shared space using linear self-attention layer. Our input-level interaction method achieves superior performance and shows richer modality integration than output-level fusion.
Comparison with fusion baselines. In Table 5
Performance trend with different modality combinations. Fig presents the performance trend from uni-modal to bi-modal and full multi-modas Combining modalities generally improves perfor- mance (denoted by red striped bars), with clinical records and IMRI showing the most significant performance gains. Using all four modalities (C+G+S+F) achieves the highest performance, showing the complementary nature of cross-moot modal information and the effectiveness of our method to enable sufficient interaction among foundation models.
Number of queries. Shows an ablation study on the number of queries. When set to 0, the model degrades to late-fusion baseline. As the number increases, the ACC performance two tasks

---

Published as a conference paper at ICLR 2026
improves the ADNI dataset. The performance with 16 queries suggests that sufficient modal interaction has been achieved at this point.
Ablation foundation model selection. To vali- date our foundation model choices, we conduct abla- tion studies by replacing the foundation model of modality at a time while keeping the others fixed. Ac- curacy comparisons Table show that selected models consistently achieve better performance, con- firming their suitability for our framework.
0.871
0.941
CN
0.908
0.985
353
(31
Visualization and Interpretability. We visualize the attention maps of IMRI images in the NC/MCI classification task. In Stage 2, each anchor modal- ity model takes fMRI representations identify which IMRI regions are most attended to, we compute the average attention weights that anchor models' embeddings assign to each brain parcel using the Attention-Rollow method (Abnar) & Zuidemaa (020). With the Schaefer functional atlas (Schaefer et al., 2018), the brain network is partitioned into seven subnetworks. We then average the ROI attention values within each subnetwork and normalize them to derive the network-level attention distribution. As shown in Fig. 4 when serving as the anchor modality, sMRI predominantly attends to the DMN, LN, and CN; genetic data emphasizes the DAN, LN, and SMN; and clinical records highlight the DMN, SAN, and CN. Our findings show that modality-anchore on its most discriminative fMRI regions while facilitating complementary information exchange across modalities. Averaged across the three anchor modalities (Fig. d).. the attention emphasizes the roles of DMN, CN, SAN, and LN in cognitive impairment, in line with prior research (Talwar et al. 2021 Sheline & Raichle 2013
32
0.889
Figure 5: Query numbers ablation.
input. To
interaction enables each anchor modality to selectively focus
Brier et al. 2014). (More visualization in Appendix D.5)
5 CONCLUSION
In this paper, we propose unified multi-moda, framework for Alzheimer's ' Disease diagnosis that leverages pre-trained uni-modal models with modality-anchored interaction and modality Q- Former to enable early and effective anchor and auxiliary modalities interaction. Broad evaluations on ADNI, PPMI, and OASIS-3 demonstrate the strong performance and generalization of our framework, underscoring the potential of foundation model adaptation for medical multi-modal learning.
ACKNOWLEDGMENT
This work was supported by National Natural Science Fund of China (No. U25A20527, 62473286). This work was also supported by Shanghai Municipal Science and Technology Major Project (No. 2025SHZDZX025G10))
REPRODUCIBILITY STATEMENT
To support replication, we provide implementation details of our method in Appendix B including settings, data splits, and training hyper parameters for experiments and foundation models across four modalities. The training parameters of modality-aware Q-former are in Appendix D.8 The implementation details of compared methods are in Appendix D.I The preprocessing pipelines for each modality deseribed in Appendix A. Together, these details reflect practical considerations for real-world deployment.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. No human or animal experiments were conducted. All datasets were publicly available and used in accordance with their respective guidelines, ensuring privacy protection. We are committed to maintaining ethical standards and fostering responsible Al use.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
Han-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning, Advances neural information processing systems, 35:2371 2022.
Alzheimer's Association. 2019 alzheimer's disease facts and figures. Alzheimer's & dementia, 15(3):
321-387, 2019.
Madison Bailey, Zlatomira G Ilchovska, Akram A Hosseini, and Je Young Jung. Impact of apolipopro- tein e E4 in alzheimer's s disease: aneaa-analysis of voxel-based morphometry studies. Journal of clinical neurology (Seoul, Korea), 20(5):469, 2024.
Nicasia Beebe- Wang, Safiye Celik, Ethan Weinberger, Pascal Sturmfels, Philip L De Jager, Sara Mostafavi, and Su-In Lee. Unified ai framework to uncover deep interrelationships between gene expression and alzheimer's disease neuropathologies. Nature Communications, 12(1))5369, 2021.
Ofir Ben Shoham and Nadav Rappoport. Cpllm: Clinical prediction with large language models.
PLOS Digital Health, 3(12)xe0000680 2024.
Matthew Brier, Jewell B Thomas, and Beau M Ances. Network dysfunction alzheimer's 's disease:
refining the disconnection hypothesis. Brain connectivity, 4(5):299-311, 2014.
Suzanne Cahill. Who's s global action plan on the public health response to dementia: some challenges and opportunities. Aging & Mental Health, 24(2):197-199, 2020.
Josue Ortega Caro, Antonio H de O Fonseca, Christopher Averill, Syed A Rizvi, Matteo Rosati, James L Cross, Prateek Mittal, Emanuele Zappala, Daniel Levine, Rahul M Dhodapkar, al. BrainIm: A foundation model for brain activity recordings. bioRxiv, pp. 2023-09, 2023.
Josue Ortega Caro, Antonio Henrique de Oliveira Fonseca, Syed A Rizvi, Matteo Rosati, Christopher Averill, James L Cross, Prateek Mittal, Emanuele Zappala, Rahul Madhav Dhodapkar, Chadi Abdallah, and David van Dijk. BrainLM: A foundation model for brain activity recordings. In The Twelfth International Conference on Learning Representations, 2024. URL https
//openreview. n forummid-Rw11222RR.
Kathleen M Chen, Aaron K Wong, Olga G Troyanskaya, and Jian Zhou. A sequence-based global map of regulatory activity for deciphering human genetics. Nature genetics, 54(7):940 949, 2022.
Qiuhui Chen and Yi Hong. Smart: Selghteighted multimodal fusion for diagnostics of neurodegen- erative disorders. Proceedings of the 32nd ACM International Conference on Multimedia, pp- 4426 4435, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- -prrpose vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250-49267, 2023.
Hugo Dorla-Torre, Liam Gonzalez, Javier Revilza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski. Francesco Oteri, Christian Dallago. Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, et al. Nucleotide transformer: building and evaluating robust foundation models for human genomies. Nature Methods, 22(2):287-297, 2025
Zijian Dong, Ruilin Li, Yilei Wu, Thuan Tinh Nguyen, Joanna Chong, Fang Ji, Nathanael Tong, Christopher Chen, and Juan Helen Zhou. Brain-jepa: Brain dynamics foundation model with gradient positioning and spatiotemporal masking. Advances in Neural Information Processing Systems, 37:86048-86077 2024.
11

---

Published as a conference paper at ICLR 2026
Shaker El-Sappagh, Tamer Abuhmed, SM Riazul Islam, and Kyung Sup Kwak. Multimodal multitask deep learning model for alzheimer's ' disease progression detection based on time series data. Neurocomputing, 412:197-2155 2020.
Oscar Esteban, Christopher J Markiewicz, Ross W Blair, Craig A Moodie, A Ilkay Isik, Asier Erramuzpe. James Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, et al. fmriprep: robust preprocessing pipeline for functional mri. Nature methods, 16(1):111-116, 2019.
Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large language models (Ilms) on tabular data: Prediction, generation, and understanding-a survey, arXiv preprint urXiv:240217944, 2024.
Yingjie Feng. Wei Chen, Xianfeng Gu, Xiaoyin Xu, and Min Zhang. Multi-modal semi-supervised evidential recycle framework for alzheimer's disease classification. International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 130-140. Springer, 2023a.
Yingjie Feng, Xiaoyin Xu, Yueting Zhuang, and Min Zhang. Large language models improve alzheimer's disease diagnosis using multi-modality data. In 2023 IEEE International Conference on Medical Artificial Intelligence (MedAI), pp. 61-66. IEEE, 2023b.
Luiz Kobuti Ferreira and Geraldo F Busatto. Neuroimaging in alzheimer's disease: current role in clinical practice and potential future applications. Clinics, 66:19-24, 2011.
Kilian Hett, Vinh- Thong Ta, Ipek Oguz, José V Manjón, Pierrick Coupé. Alzheimer's Disease Neu- roimaging Initiative, et al. Multi-scale graph-based grading for alzheimer's ' disease prediction. Medical image analysis, 67:101850, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Clifford R Jack Jr, Scott Andrews, Thomas G Beach, Teresa Buracchio, Billy Dunn, Ana Graf, Oskar Hansson, Carole Ho, William Jagust, Eric McDade, et al. Revised criteria for diagnosis and staging of alzheimer's 's disease: Alzheimer's association workgroup. Alzheimer's & Dementia, 20 (8):5143-5166 2024.
DS Knopman, ST DeKosky, JL Cummings, H Chui, Corey-Bloom, N Relkin, GW Small, B Miller, and JC Stevens. Appendix b: Practice parameter: Diagnosis of dementia (an evidence-based review): Report of the quality standards subcommittee of the american academy neurology. CONTINUUM Lifelong Learning in Neurology, 13(2):210-25) 221, 2007.
Pamela J LaMontagne, Tammie LS Benzinger, John C Morris, Sarah Keefe, Russ Horbeek, Chengjie Xiong, Elizabeth Grant, Jason Hassenstab, Krista Moulder, Andrei G Vlassenko, al. Oasis-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. medrxiv, pp. 2019-12, 2019.
Ho Hin Lee, Alberto Santamaria- Pang, Jameson Merkov, Matthew Lungren, and Ivan Tarapov. Multi-modal mamba modeling for survival prediction (m4survive): Adapting joint foundation model representations. arXiv preprint arXiv:2503. 15.50,00000, 2025.
Bolun Li, Jie Shi, Boris Gutman, Leslie Baxter, Paul M Thompson, Richard J Caselli, Yalin Wang. and Alzheimer's Disease Neuroimaging Initiative. Influence of apoe genotype on hippocampal atrophy over time-an n= 1925 surface-based adni study. PloS one, 11(4):e0152901, 2016.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12880- PMLR, 2022a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference machine learning, pp. 19742. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Tianhao Li, Sandesh Shetty, Advaith Kamath, Ajay Jaiswal, Xiaoqian Jiang, Ying Ding, and Yejin Kim. Cancergpt for few shot drug pair synergy prediction using large pretrained language models. NPJ Digital Medicine, 7(1):40, 2024a.
Xue Li, Xiaojin Feng, Xiaodong Sun, Ningning Hou, Fang Han, and Yongping Liu. Global, regional, and national burden of alzheimer's disease and other dementias, 1990-2019. Frontiers in aging neuroscience, 14:937486, 2022b.
Zchui Li, Vallijah Subasri, Guy-Bart Stan, Yiren Zhao, and Bo Wang. Gv-rep: A large-scale dataset for genetic variant representation learning. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-33916, 2023a.
Hong Liu, Dong Wei, Donghuan Lu, Jinghan Sun, Liansheng Wang, and Yefeng Zheng. M3ae: Multimodal representation learning for brain tumor segmentation with missing modalities. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 1657-1665, 2023b..
Linfeng Liu, Siyu Liu, Lu Zhang, Xuan Vinh To, Fatima Nasrallah, and Shekhar S Chandra. Cascaded multi-modal mixing transformers for alzheimer's disease classification with incomplete data. Neurolmage, 277:120267, 2023c.
Qiao Liu, Wanwen Zeng. Hongtu Zhu, Lexin Li, Wing Hung Wong, and Alzheimer's Disease Neu- roimaging Initiative. Leveraging genomic large language models to enhance causal genotype- brain-clinical pathways in alzheimer's discase. medRxiv, pp. 2024-10, 2024.
Kenneth Marek, Danna Jennings, Shirley Lasch, Andrew Siderowf, Caroline Tanner, Tanya Simuni, Chris Coffey, Karl Kieburtz, Emily Flagg, Sohini Chowdhury, et al. The parkinson progression marker initiative (ppmi). Progress in neurobiology, 95(4):629-635, 2011.
Susanne G Mueller, Michael W Weiner, Leon Thal, Ronald C Petersen, Clifford Jack, William Jagust, John Q Trojanowski, Arthur W Toga, and Laurel Beckett. The alzheimer's disease neuroimaging initiative. Neuroimaging Clinics, 15())88698877, 2005.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36:43177-44200 2023.
Zhenyuan Ning, Qing Xiao, Qianjin Feng, Wufan Chen, and Yu Zhang. Relation-induced multi-modal shared representation learning for alzheimer's ' disease diagnosis. IEEE Transactions Medical Imaging, 4036):1632-16666 2021.
Shangran Qiu, Matthew I Miller, Prajakta S Joshi, Joyce C Lee, Chonghua Xue, Yunruo Ni, Yuwei Wang, Ileana De Anda-Duran, Phillip H Hwang, Justin A Cramer, et al. Multimodal deep learning for alzheimer's 's disease dementia assessment. Nature communications, 13(1):3404, 2022.
Zifeng Qiu, Peng Yang, Chunlun Xiao, Shuqiang Wang, Xiaohua Xiao, Jing Qin, Chuan-Min Ming Liu, Tianfu Wang, and Baiying Lei. 3d multimodal fusion network with disease-induced joint learning for early alzheimer's disease diagnosis. IEEE Transactions on Medical Imaging, 2024.
Nasir Rahim, Shaker El-Sappagh, Sajid Ali, Khan Muhammad, Javier Del Ser, and Tamer Abuhmed. Prediction of alzheimer's 's progression based on multimodal deep-learning-basee fusion and visual explainability of time-series data. Information Fusion, 92:363-388, 2023.
Steven Rigatti. Random forest. Journal of Insurance Medicine, 47(1):31 2017.
Shaohao Rui, Lingzhi Chen, Zhenyu Tang, Lilong Wang, Mianxin Liu, Shaoting Zhang, and Xiaosong Wang. Multi-modi modal vision pre- pre-training for medical image analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5164-5174, 2025.
13

---

Published as a conference paper at ICLR 2026
Andrew J Saykin, Li Shen, Tatiana M Foroud, Steven G Potkin, Shanker Swaminathan, Sungeun Kim, Shannon L Risacher, Kwangsik Nho, Matthew J Huentelman, David W Craig, et al. Alzheimer's 's dis- case neuroimaging initiative biomarkers as quantitative phenotypes: Genetics core aims, progress, and plans. Alzheimer's 's & dementia, 6(3):265-275, 2010.
Alexander Schaefer, Ru Kong, Evan M Gordon, Timothy O Laumann, Xi-Nian Zuo, Avram J Holmes, Simon B Eickhoff, and BT Thomas Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri. Cerebral cortex, 2809):309531144, 2018.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprini arXiv:2507.0522 2025.
Yvette I Sheline and Mareus Raichle. Resting state functional connectivity in preclinical alzheimer's disease. Biological psychiatry, (4(5):340-347, 2013.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172-188 2023.
Xuegang Song, Feng Zhou, Alejandro F Frangi. Jiuwen Cao, Xiaohua Xiao, Yi Lei, Tianfu Wang, and Baiying Lei. Multicenter and multichannel pooling gen for early ad diagnosis based on dual-modality fused brain network. IEEE Transactions on Medical Imaging, 42(2):354-367, 2022.
Puneet Talwar, Suman Kushwaha, Monali Chaturvedi, and Vidur Mahajan. Systematic review of different neuroimaging correlates in mild cognitive impairment and alzheimer's disease. Clinical neuroradiology, 31 31(4:9553-677 2021.
Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2022.
Ye Tian, Daniel S Margulies, Michael Breakspear, and Andrew Zalesky, Topographic organization of the human subcortex unveiled with functional connectivity gradients. Nature neuroscience, 23(11): 2020.
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang. Manoj Middepogu, Ziteng Wang, et al. Cambrian-1; A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 37:87310- 10-85366886 2024.
Hugo Touvron, Thibaut Lavril Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971,
Haoyu Wang, Sizheng Guo, Jin Ye, Zhongying Deng, Junlong Cheng, Tianbin Li, Jianpin Chen, Yanzhou Su, Ziyan Huang, Yiqing Shen, et al. Sam-med3d: towards general- purposs segmentation models for volumetric medical images. arXiv preprint arXiv:2310.1516/1 2023.
Chonghua Xue, Sahana Kowshik, Diala Lteif, Shreyas Puducheri, Varuna Jasodanand, Olivia Zhou, Anika S Walia, Osman B Guney, J Diana Zhang, Serena T Pham, et al. Ai-based differential diagnosis of dementia etiologies on multimodal data. Nature Medicine, 30(10):2977-2998 2024.
Yiqing Yang and Man-Wai Mak. Emoq: Speech emotion recognition via speech-aware q-former and large language model. arXiv preprint arXiv:2509. /5775, 2025.
Jeong Hun Yeo, Hyeongseop Rha, Se Jin Park, and Yong Man Ro. Mms-llama: Efficient llm- based audio-visual speech recognition with minimal multimodal speech tokens. arXiv preprint arXiv:2503.1 1315, 2025.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuma audio-visual language model for video understanding. arXiv preprint arXiv:2306.000 02858, 2023a.
14

---

Published as a conference paper at ICLR 2026
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915, 2023b.
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert-2: Effi- cient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15000 2023.
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V Davuluri, and Han Liu. DNABERT-2: Efficient foundation model and benchmark for multi-species genomes. The Twelfth International Conference Learning Representations, 2024. URL http:ttts///pperee////// net forum?
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context, arXiv preprint arXiv:2404. 13046, 2024.
15

---

Published as a conference paper at ICLR 2026
a. Clinical Textual Data
gender
LM recall
Trail
race
Male
White
78
femelee 19/15
29.0
white
Convert tabular input to natural prompt
LLM
Your task to determine whether patient is likely to have Alzheimer's is disease based on their descnosiidesccipppiinn descriptions provided below. Clinicel Record Input. Diagnosis
Prompt
Token
Converted input
The putient 84-yeur-oldf 48fernale with 18 years of education. The patient's is race white. The MMSE score in 28 reported family history cagnitive impairment.
NO
04
Figure 6: Tabular Data Prompt Construction Pipeline.
A DATA CONSOLIDATION AND AUGMENTATION
A.1 CLINICAL RECORDS PROMPT TEMPLATE.
To LLM for clinical tabular records, the tabular input and classification task must be transformed into natural text. 6ig give a instance of tabular data.
Tabular Data. Our tabular data includes of clinical features covering multiple main cate- gories: demographic information (e.g., age, gender, education, ethnicity, and race), genetic risk factors (APOE genotype). cognitive assessment scores (including MMSE, CDR, MoCA, and other neuropsychological tests), neuropsychiatric symptoms (from item-level NPI-Q responses). functional ability metrics (from the Functional Activities Questionnaire), medical history and lifestyle indicators (e.g., cardiovascular conditions, psychiatric disorders, smoking and alcohol use), and imaging-related parameters such as scanner strength.
Task prompt
To serialize structured tabular data into natural language prompts, adopt text templating strategy that systematically enumerates each feature along with its corresponding value in natural language form. The detailed prompt and prefix follows:
Listing 1: Task Prompt.
Task prompt (AD vs. NC) Your task is to determine whether a patient is likely to have an Alzheimer's disease based on their diagnosis descriptions provided below.
Task prompt (MCT vs. NC) = Your task is determine whether a patient is likely to have mild cognitive impairment based on their diagnosis descriptions provided below.
Task prompt (sMCI vs. PMCI) '''
Your task is to determine whether a patient with mild cognitive impairment is likely to remain stable or progress to Alzheimer' disease based their diagnosis descriptions provided below.
16

---

Published as a conference paper at ICLR 2026
Description Prompt. To generate the description prompt, for categorical and numerical attributes, we combine the name and its values into a sentence. And for binary features, we include the feature name in the prompt only if the value is True to avoid generating unnecessary and false information.
Listing 2: Description Prompt.
Description Prompt '''
The patient is (age) -yerr-old (gender) with (education) years of education. Their ethnicity is coded as (hispanic), and their race is (race). The APCE status is (apce). The MMSE score is { mmse), and the CDR score is (cdr) with a sum of boxes score of (cdrSum). MRI Tesla strength is recorded as (Tesia).
Cognitive test results include Trail A ((trailA)), Trail B (f trailB]), immediate recall ((lm_im)), LM delayed recall (f Im_del)), Boston naming test ((boston)), Animal fluency (f
animall), Digit span backward ((digitB)), Digit Span Backward Longest (fdigitBL)), Digit span forward ((digitF)), Digit Span Forward Longest ([digitFl)). Neuropsychiatric inventery (NPI-
2) results indicate symptoms of (Miio//ooeratee/eevee) + symptom list), or: Neuropsychiatric inventory (MPI-Q) results indicate reported symptoms. Functional assessment (FAQ) shows difficulties in [faq_BILLS
BILLS (label), faq_TAXES TAXES (label), ..., faq TRAVEL (iabel)). Medical history includes Cardiovascular events (Yes), Psychiatric disorders (Yes), Other depression-relatees conditions (Yes) GDS score: (gds).
MoCA score: (moca)
...
A.2 GENOMIC DATASETS CONSTRUCTION.
|lllstraats the genomic datasets construction process. Firstly, we construct a genetic dataset with whole genome sequencing (WGS) studies for Alzheimer's disease (AD) Mueller et al. 2005),, which provides base- pair level coverage of the entire genome, allowing for compre- hensive assessment of individual genetic variation. To enable downstream machine learning models to process genetic data within their input length limitations. for ADNI, we select two major genes, APOE and TOMM40, as the primary sources of genetic input. For PPMI, we select RIMS2 and TMEM108. These genes have been identified as being most strongly associated with AD or PD susceptibility Saykin et al. (2010). For each individual, up to 80 genetic variants were selected from two major genomic regions. Then a sequence extractor is used to convert those genetic variants records by locating the position of variants the human reference genome.
Construction Fig 7
Usage Following Li et (0024b), for each variant, we located its position in the GRCh38/hg38 human reference genome and reconstructed the corresponding reference (ref) and alternate (alt) sequences. Variants located
configs present in the reference genome were filtered out. These sequences were then concatenated separately, namely refs with refs and alts with alts, to construct pair of representation of ref and alt sequences per individual, capturing AD-relevant genomic features for downstream prediction tasks.
Genetic Variants Record Formulation Following this construction process in Fig the minimum unit of ADNI gene dataset is individual variants record, which is an (I, y) pair. Here, x = (ref, alt), and y denotes the individual's diagnostic label.
A.3 MRI PRE-PROCESS PIPELINE.
We adopted two distinct pre-processing pipelines for structural and functional MRI data respectively.
The pre-procssingg pipeline for sMRL. We use the TI-weighted MRI scan closest to each subject's baseline visit. To ensure consistency and quality, exclusively selected 3T MRT scans, they
17

---

Published as a conference paper at ICLR 2026
TOMM40
APOE
Extraction
Filter
Subject Variants Records
Chsttonosition.
Ref.
Alt.
ADNI WGS Data
chrl 1::44 1941
G
T
chr19:44008684
C
T
Sequence Extractor
Subject Records
chr99:44904126
A
T
chr19:44899045
C
G
Extraction
Subject Labels
MCI/ Normal Control
Predict
AD Normal Control
Finetune Genomic Model
Stable MCI/ / Progression MCI
Figure 7: Genetic Dataset Construction and Usage. The workflow of gene dataset from ADNI dataset. Genetic variant records first extracted from selected APOE and TOMM40 gene for each individual. These records are paired with diagnostic labels of individual derived from the ADNI dataset. A sequence extractor then organizes the variants into fixed-length sequences, which serve model inputs for downstream genetic foundation models.
Table Multimodal modalities and subject distributions across datasets.
Datasets
Type
Modalities
#NC
#MCI
#AD
#PD
ADNI
NC/MCI/AD NC/AD
C+G+F+S
898
986
416
OASIS
C+F+S
120
43
PPMI
NC/MCL/PD
C+G+F+S
743
143
329
provide higher signal-to-nn to-seose ratio, better spatial resolution, and improved image quality compared to 1.5T scans, which benefits downstream learning. Pre- reepprocsssing structural MRI specifically involves resampling all 3D structural T1 Te-weighted (TIw) MRI volumes to a standardized spatial resolution of 128 x 128 x 128, accordance with the input requirements of the SAM-Med3D Wang et al.) (2023) pretrained model.
The pre-processing pipeline for fMRI. Following the preprocessing pipeline of Brain-JEPA Dong al. (2024). functional MRI data were preprocessed using the fMRIPrep pipeline Esteban al. (2019). which incorporates nigh-precision anatomical reference via Tlw MRI images. The pipeline included skull stripping, cortical surface reconstruction, slice-timing correction, followed by co- registration of fMRI time series to the corresponding anatomical scans. Spatial normalization to the MNI152 standard space was subsequently performed using nonlinear transformation. Following normalization, the fMRI time series were parcellated into 450 regions of interest (ROIs), with cortical regions delineated by the Schaefer atlas Schaefer et al. (2018) and subcortical regions defined using the Tian-Scal III atlas Tian et al. (020)). To enhance inter-subject comparability and mitigate matrr-participant variability, robust scaling was applicd to each ROI by subtracting the across-subject median and dividing by the interquartile range Caro et al. (2023)
18

---

Published as a conference paper at ICLR 2026
A.4 DATASETS DETAILS.
As illutstrated in Table 7 three public multi modal datasets are evaluated in our study, considering four medical modalities, including two imaging modalities, genetic data, and clinical records.
ADNI. We evaluate our method using the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset Mueller al. (2005), which is the publicly available dataset that offers the most comprehensive set of modalities, including structural and functional MRI (sMRI and IMRI), genetic data, and textual and tabular clinical records. ADNI includes participants across three main diagnostic categories: normal controls (NC), mild cognitive impairment (MCI), and Alzheimer's ' Disease (AD).
PPMI. PPMI is a public dataset Marek et al. (2011) focused on Parkinson's 's disease, providing the same set of modalities as ADNI. It includes subjects across three diagnostic categories: normal controls (NC), mild cognitive impairment (MCT), and Parkinson's 's disease (PD).
OASIS-3. The OASIS-3 dataset LaMontagne et al. (0199) is multimodal neuroimaging and clinical record resource that provides sMRI, IMRI, and textual and tabular clinical information, but lacks the genetic data available in ADNI. It comprises subjects diagnosed as normal controls (NC) and Alzheimer's disease (AD).
B PRETRAINED FEATURE EXTRACTION IMPLEMENTATION DETAILS
B.1 SETTINGS OF EXPERIMENTS,
All experiments were conducted on NVIDIA A100 80GB GPUs, with a total of 32 GPUs used. Each compute worker was equipped with 64 CPU cores and 512 GB of RAM. Training under the modality- complete setting typically took around 1.5 hours per epoch. We adopt 16 as the number of queries in our proposed Modality-aware Q-former. Both Stage 1 (uni-modal foundation model adaptation) and Stage 2 (Modality-anchonl anchored interaction), illustrated in Fig [ are trained in supervised setting, The ground truth labels are determined based on established clinical diagnostic criteria Mueller et al. (005)). For the three datasets, we performed strict de-duplication of subjects using their unique IDs across all phases to ensure that each subject appears only once in the entire dataset. This guarantees that there is no overlap of subject data between the training, validation, and test sets, effectively eliminating the risk of data leakage. The dataset is partitioned into 60% for training, 20% for validation, and 20% for testing. We conducted test experiments using 5-fold cross- volidaaalio For each fold, we record the performance on the test set. This resulted in five paired performance values, one for each fold. We then conducted statistical significance test through paired t-test on these fold-wise results to assess whether the performance difference was statistically significant. The resulting p-value below 0.01 (p= = 0.005) confirms that our performance improvements reported in the main results over the second-best aproach have been confirmed to be statistically significant. Our approach treats each modality as the anchor one in turn. As a result, in modality-incomplere setting, when one modality is missing, the remaining modalities can still serve as anchor inputs, and their features are extracted using the corresponding foundation models without disruption. In practice, when a modality is absent, we simply omit passing data to its associated Q-Former. For the Q-Formers of the remaining modalities, we set the query tokens corresponding to the missing modality to zero and apply an attention mask to prevent the model from attending to it.
B.2 SETTINGS AND HYPER PARAMETERS OF FOUNDATION MODELS.
Each modality is processed by a dedicated pre-trained foundation model: LLaMA2- 13B Touvron et al.. (2023) for textual data, Dalla- Torre et al. (2025) for genetic data, Brain-JEPA Dong et al. (2024 for
1MRI, and BrainMVP Rui et al. (2025) for sMRI. Training details of each model is presented below.
Textual feature Extraction. The LLaMA2 model Touvron et al. 20233 in Fig 8 (Left) was leveraged to extract textual features from clinical records into a latent space representation. LLaMA is an auto-regressive language model based on optimized transformer architecture. The fine-tuned variants are aligned with human preferences through a two-stage process: supervised fine-tuning (SFT) on curated catructtionfollowiin data, followed by reinforcement learning with human feedback (RLHF) to further optimize for helpfulness, factuality, and safety. We utilize LLaMA2- 12b-HB pre- traind weights. For ILaMA2-13BB use the Adam W optimizer with learning rate of 2e-4. The
19

---

Published as a conference paper at ICLR 2026
Softmax
0.710.29
Linear
RMS Norm
Downstream Prediction
Prediction Head
Feed Forward SwIGLU
Embedding Token
RMS Norm
x N
Self-Attention (Grouped Multi-Query Attention with KV Cache)
LoRA Weights
Q
S
Nucleotide Transformer
RMS Norm
Embeddings
Input Sequence
Task Prompt Description Prompt
LLaMA
Nucleotide Transformer
Figure Architecture of LLaMA 2 (left) and NT-Human500 (right).
model is trained for total of 6 epochs with batch size of 8. We apply LoRA Hu al. (2022) using the Hugging Face peft library with the following configuration: rank 32, a = 32, dropoul 0.1, and bias adaptation for classification.
Genomic Feature Extraction. The Nuclcotide Transformer Dalla-Torre et al. (2025) in Fig
(right), was leveraged extract genomic features from ADNI Gene Dataset constructed in Fig The Nucleotide Transformers are foundational language models pre-trained on DNA sequences from whole genomes. Unlike traditional approaches that rely solely on single reference genome, our models incorporate genetic information from over 3,200 diverse human genomes and 850 genomes across various species, including both model and non-m mondell organisms. Through comprehensive evaluations, we demonstrate that these large-scale models significantly outperform existing methods predicting molecular phenotypes with high accuracy. We utilize Srreetiiderransffomme---e-e human-ref pre-traind weights, 500M parameters transformer pre-trained on the human reference genome. For NT-500M- model is trained for epochs with batch size of 4. We apply LoRA Hu et al. (2022) using the Hugging Face peft library with the following configuration: rank 32, a 32, dropout 0.1, and no bias adaptation for classification.
we use the Adam optimizer with a learning rate of 5e-4. The
Neuro-imaging Feature Extraction BrainJEPA Dong et al. (2024) was employed to extract latent representations from fMRI scans, leveraging pretraining on large-scale 3D brain MRI data. We adopt ep00-ep300 weights, the pre-trained model weights resulting from training on the UKB dataset. The target encoder from this model serves the feature extractor for fMRI data. We also adopted BrainMVP Rui et al. (2025) for sMRI representation. Its fully 3D architecture includes a BrainMVP image encoder and BrainM VP decoder, built with UniFormer backbone capture spatial features
20

---

Published as a conference paper at ICLR 2026
Table 8: Effect of Connectors. The ACC and AUC performance across three AD diagnosis and
progression prediction tasks on ADNI dataset in
settings is reported.
NC vs. MCI
NC vs. AD
sMCI
PMCI
Components
ACC
AUC
ACC
AUC
ACC
AUC
Fusion
0.881
0.851
Proj Q-former
0.961
0.957
0.911
0.904
0.825
0.791
0.960
0.960
0.915
0.910
0.830
0.810
Modality- Q-former
0.979
0.969
0.945
0.944
0.846
0.818
effectively. We utilize 16k mpMRI weights. For Brain-JEPA, training is conducted for 200 epochs with batch size of 8, using A with an initial learning rate of 5e-5, For BrainMVP, we apply AdamW with learning rate of 8e-4 and train the model for 200 epochs with batch size of 4.
C IMPLEMENTATION DETAILS OF Q-FORMER.
C.I DETAILS OF MODALITY-AWARE Q-FORMERS
To enable interaction between the representations from the anchor and auxiliary modalities, we propose to employ set of learnable queries to explicitly project the auxiliary features into the representation space of the anchor one. As shown Eq [] to allow effective interaction between the anchor model and the auxiliary models, transformer-baseee connector is proposed to selectively project features from the auxiliary modality to the feature space of the anchor model, called modality- aware Q-former. As illustrated in Fig I our modality-aware Q-former incorporates two types of information, namely uni-modal and cross modal information. Specifically, we create a set of learnable queries X R N.N.GC. xC x E IR with number of learnable query tokens Ng. serving as the query:
X
(9)
Number of learnable query tokens. We set the number of learnable query tokens N₂ 16 to strike balance between leveraging auxiliary modality information and maintaining stable training of the anchor modality. As illustrated in Fig increasing N,, allows the model to incorporate more in- formation from auxiliary modalities, enhancing cross-modal interactions and semantic representation. Conversely, reducing Ng limits the contribution of auxiliary modalities; when N₂ approaches zero, the model effectively degrades to a late fusion strategy, where only uni-modal representations from the anchor modality are used without cross- cross-modal guidance.
Uni-modal Q-formers Modality aware Q-former first extracts the uni-modality information from a specific auxiliary modality m .M'. Specifically, we create set of learnable tokens to serve as uni-modalig queries, denoted as Xug E NN,x Given the auxiliary features extracted from the corresponding auxiliary model Fm, we first project them to the same dimension as the anchor modality:
tTh LineerrFF.(X"" E L'
(10)
Then, the learnable uni-modal queries interact with the projected features through cross-attention layer, which further projects the auxiliary modality features into the anchor modality feature space and extracts information relevant to the anchor modality from the auxiliary one m:
m CrossAttn(4 = ug. K = 7 V 7T))
(11)
The resulting output Xm E XC are features containing uni-modal information from auxiliary modality m.
Cross-modal Q-former Besides uni-modal information, we further propose set of cross- mooal queries X R. XC that enables feature interaction among all auxiliary modalities. Specifically, the
cross-modal queries interact with all the output tokens of uni-modal Q-formers (Xmmmm M') with
21

---

Published as a conference paper at ICLR 2026
cross-attention layer capture cross-modality correlations among different auxiliary modalities, resulting in the cross-modality auxiliary features denoted as X
X CrossAttm = X K V=,V== V²"V==27 77),
(12)
where
z = Concat( (oncaa([X*)) ({8T) ImEMe).
(13)
Finally, the cross-modal auxiliary feature Xe and of uni-modal auxiliary features {X I 3' are concatenated to obtain the final output of the modality-awe Q-former:
H Concat (X"))EM MEM, X*) E RR2N,CC
(14)
Q-former and Linear dimensions For anchor modality E M = s, c, and its uni-modal learnable query tokens X E R. and cross-modal query tokens Xog E RNx the dimen- sionality On for each modality is defined as follows:
512,
if m = B
768,
if f
C.
(15)
5120,
if C if m = g
1280,
C.2 EFFECTIVENESS OF CROSS-MODALITY Q-FORMER.
Compared with conventional fusion methods such as Late-Fusion, which combine uni-modal pre- dictions at the output level, our Q-Former design enables earlier and more effective modal interactions at the level anchor token embeddings and auxiliary features, allowing more effective integration of complementary information from all modalities. To isolate the effect of modality interaction mechanisms and assess the effectiveness of our proposed Modality- Q-former, we consider setting where all four modalities (C/F/S/G) are available, and apply four types of interaction strategies on top of pretrained uni-modal foundation models from stage 1. Specifically, we apply four types of fusion strategies: (1) Linear Projection (Proj), where features from each modality are projected to shared space with simple linear modules and concatenated; (2) Late- sate-Fusion, where each modality is independently processed and final predictions are aggregated; (3) Q-Former, where concatenated auxiliary modality embeddings serve as keys and values in the cross-attention layer, and learnable query tokens attend them, enabling interaction with the anchor modality. (3) Modality- aware Q Q-Foreer which introduces aatention-based query tokens to adaptively gather relevant features from auxiliary modality.
For fair comparisons, we freeze the uni-modal pretrained models from stage I and only train the fusion modules and anchor models. Results are reported in Table g under scenarios. While both Projection and Late-Fusion benefit from pre-trained features, their ability to align heterogeneous modalities is limited. Projection performs better than Late-Fusion on general classification accuracy. Our Q-Former consistently outperforms both baselines across all tasks. This demonstrates the effectiveness of our cross-modal query design in extracting synergistic information while preserving the structured representations of each modality.
D EXTENDED EXPERIMENTAL ANALYSIS
D.1 IMPLEMENTATION DETAILS OF COMPARED BASELINES.
We compare our approach with SOTA multi-modal baselines. M4Survive Lee et al. (0225). Ncomms Qiu al. (2022). Smart Chen & Hong (0024), and Aldiagnosis Xue et al. (2024) are SOTA multi-modall models to independently process and generate modality-specifee embeddings. Then the embeddings are mapped into a modallmy feature space and processed by a Mamba adapter to perform interaction for downstream prediction. M4Survive adopts LLaMA, BrainMVP, BrainJEPA, and
framework. M4Survive leverages various Radiology-Pathology pre-trained
22

---

Published as a conference paper at ICLR 2026
Table 9: Comparison to Uni-modal baselines. The performance across three AD diagnosis and
progression prediction tasks in modality-complett settings is reported.
Modality
Method
NC AD ACC (%) SPE (%) SEN (S) (%) SEN (%) ACC (PE(%) SEN (%)
NC MCI
sMCI vs. PMCI
C
Foreat Rigatti 2017
0.709
0.724
0.612
0.745
0.738
0.557
0.696
0.736
0.602
LLaMA 2 Touvion
0.793
0.640
0.687
0.574
F
Brain-JePA Dong el
2024
0.777
0.838
0.542
0.807
0.857
0.576
0.714
0.723
0.522
F
BrainLM Caro
2024
0,768
0,809
0.537
0,781
0.841
0.575
0.705
0.735
0.509
S
BrainM Kun
20.35
0.724
0.819
0.589
0.730
0.832
0.669
0.703
0.730
0.640
S
SamMed3D Wang el al.
2023
0.714
0.807
0.597
0.714
0.814
0.675
0.689
0.718
0.647
Swin-UNETH Lang
2072
0.609
0.628
0.495
0.612
0.724
0.579
0.521
0.595
0.503
S M M Liu et 20236
AB
0.647
0.665
0.538
0.671
0.778
0.609
0.622
0.666
0.591
G NT-Human Nguven ofal.
2203
0.694
0.775
0.521
0.751
0.857
0.492
0.652
0.719
0.424
G SEI Chen
0.483
0.500
0.462
0.568
0.680
0.491
0.415
0.657
0.342
G DNA Benz 7loi Ci 2t.. 2024
0.709
0.724
0.612
0.746
0.840
0.557
0.659
0.813
0,460
Table 10: Results of three AD prediction tasks across the ADNI-1, ADNI-2, and ADNI-3 cohorts. Experiments are conducted under the Modality-Incompll setting, indicating the presence of partial clinical textual records, functional MRI, structural MRI, and genetic data.
NC MCI
NC vs. AD
PMCI vs. sMCI
Methods
ACC
AUC
ACC
AUC
ACC
AUC
Feature Concatenation
0.894
0.885
0.833
0.846
0.771
0.750
Linear Classifier
0.881
0,851
0.899
0.851
0.805
0.783
Self etf--Attetion Fusion
0.921
0.917
0.901
0.861
0.785
0.751
0.961
0.969
0.945
0.944
0.825
0.846
NT as mopelity-specific encoders for text, sMRI, fMRI, and genomic modalities to adapt to our AD analysis tasks. After fine-tuning the pretrained encoders our diagnosis tasks, We freeze the modality-spedee speciifc foundation models and train the adapter network with a batch size of 16, learning rate of 0.0003, over 30 epochs. Experiments employing MLP and transformer are executed NVIDIA A 1000 GPU.
Ncomms introduces a deep learning framework that comprises three components: (1) an MRI-only CNN model, (2) non-imaging models based on traditional machine learning classifiers, and (3) hybrid fusion model that integrates imaging and non-imaging data by combining CNN with CatBoost classifier for final disease diagnosis. All models in Ncomms were optimized with AdamW.
The CNN for MRI data was trained with learning rate of 0.001 for 101 epochs. The CatBoost regression models for non-iman con-imaging data were trained with dropout rate of 0.5, a batch size of 32, and learning rate of 0.001. The fusion CatBoost classifier, integrating imaging and non-imaging inputs, was trained with the same hyperparameters as the non-imaging models (dropout 0.5, batch size 32, learning rate 0.001).
Aldiagnosis proposes multi-modal ML framework to process diverse array of clinical textual data as well as multi-modal neuro imaging data to perform disease diagnosis. Model training was performed using mini-batch strategy with the AdamW optimizer, employing an initial learning rate of 0.001 over 256 total epochs. A cosine annealing learning rate scheduler with warm restarts was used to facilitate convergence, with the first restart occurring at epoch 64 and each subsequent restart period doubled relative to the previous one. The hyperparameters were empirically set as follows: e -..55, x= 0.005, and / 0.0005.
SMART consists of dual disual-textual branches, including an image encoder for MRI data and a text encoder for clinical records. A gated attention transformer serves the fusion module, integrating features from both branches for joint representation learning and performing AD diagnosis. The model is trained with the Adam optimizer using learning rate of 3e-4, batch size of 32, for 300 epochs. The hyperparameters are set as a = 3 = 1 and temperature 0.05.
D.2 MORE UNIMODAL COMPARED BASELINES.
Under the modality-comppl setting, we evaluate several strong uni-modee baselines, including Random Forest Rigatti (207)) and ILLMMA 2 Touvron et al. (2023) for clinical records, Brain-
23

---

Published as a conference paper at ICLR 2026
Prediction
NC Prediction
AD diagnosis Prediction
0.971
0.941
-C NC vs. MCI
0.94
-
NC vs. AD
0.956
0.935
Accuneyy
Rccureyy
Accuracy
0.5
0.92
0.7
0.913
0.903
0.881
ACC
0.90
0.899
ACC
16
8
16 Train from pre-trained
Train from scratch
Figure 9: Analysis of how query number in modality-awarr Q-former and foundation pre preitriind weights affect AD diagnosis performance.
JEPA Dong et al. (2024) and LrainLM Caro et al. (2024) for fMRI, BrainMVP Rui et al. (2025) SAM-Med Wang et al. )2023). Swin-UNETT Tang et al. (2022), and M Liu ct al. (023b) for sMRI, well as NT-
Nguyen al. (2023) SEI Chen al. (202)). and DNA- Bert2 Zhou et al. (024) for genomic data. Table 9 summarizes the performance of uni-modal baselines across three distinet AD-related tasks. Among all modality-specific models, LLaMA achieves the highest overall performance, highlighting the rich and discriminative nature of clinical textual data Alzheimer's 's disease analysis. LLaMA, NT. BrainMVP, and BrainJePA consistently outperform other foundation models within their respective modalities, confirming their suitability for integration into our framework.
D.3 COMPARISON WITH FUSION BASELINES.
Table 10 compares our modallty-anchored interaction with three fusion baselines, all built on the same pre-trained foundation models. The baselines include (i) feature concatenation. which directly merges modality features, and (ii) linear or self-attentio fusion, which projects modality features into shared space. Our input-level modality-anchoree interaction consistently outperforms these baselines, demonstrating that performing fusion at the input level enables richer cross-modal integration than output-level fusion.
D.4 EFFECT OF TRAIN FROM PRETRAINED WEIGHTS.
Fig. 9 (rightmost subfigure) compares training from scratch with training from pre-trained weights on two AD diagnosis tasks on ADNI dataset. The results consistently show that initializing from pre-trained weights yields better performance than training from scratch, highlighting the benefit of leveraging prior knowledge encoded foundation models.
D.5 VISUALIZATION OF ATTENTION MAPS ON TEXTUAL RECORDS AND SMRI IMAGE
To evaluate interpretability, we compare attention maps produced by our method against those from late-fusion baseline. Fig 10 (a) presents the attention weight distribution over clinical text descriptions. Results indicate that our model better captures features in longer sequences. For instance, in the sentence 'The APOE status is 0,' known AD biomarker Knopman et al. (207)) the baseline assigns little attention, whereas our model effectively identifies task-relevant words. Fig TO (b) compares the sMRI attention maps between highest attention weights are highlighted in red (late-fusion) and blue (ours). The baseline late fusion method overlooks some critical medical imaging biomarkers, whereas method more accurately attends to these key subregions.
method and the late-fusion bascline. Image patches with the
We further visualize the AD patients' attention maps of sMRI images and clinical records across different anchor modalities and their models in Fig IT In Stage 2, each anchor modality model takes sMRI or clinical record representations input. To reveal which regions are most attended,
24

---

Published as a conference paper at ICLR 2026
(ADNI) AD subject
LateFusion
Ours
LateFusion
potientis B4-ysnn romais years education Their othnicity coded os and their necw ses, APCI status B. MNSi scoee ans score Bil with nosee-acore 0.0. MR Teala strength recorded us 1.5. Cugiitte heal results nolude (81.3) 12), Trail (51.0), immediate recall (1B)) LH dolayed rocad/88. Boston testnggest(27..0 0), Animal sueney (22) Medical history includes Family hrotory engn impaimmnt
Ours
oatient is Ri-yese-oll fermel with 18 years eduipaiin Their echnicity cocled WK
and Jil nace with The AFCE status ia The HMSE score 28 and the CDR acore 0.2
with sum boows soon O.
strenghh is recorded
resulis include Trail STG Trail S1.. LM Mimimddiite moill
(E))
delayed recall
Soston nomnin we 127.0)
fuonny (22)... Medicol history noludes Family histors
coug allve impaiement (Nax)
CDS acore:
(a) Textual Records
(b) sMRI Image
Figure 10: Comparison of attention maps between the late-fusion baseline and our method on clinical textual records and sMRI images of an AD patient from the ADNI dataset. A darker color indicates a higher attention on text. Image patches with the highest attention weights are highlighted in red (late-fusion) and blue (ours).
we compute the attention weights of input data assigned by the model embeddings. As shown in Fig T1, for sMRI data, the fMRI foundation model (BrainlePA) attends to critical medical subregions, while the gene (NT) and clinical text (LLaMA) foundation models cover broader areas. For clinical records, both the gene (NT) and IMRI (BrainJePA) models show broad attention to demographic information, cognitive scores, neuropsychiatric symptoms, medical history, and lifestyle factors, whereas the sMRI foundation model (BrainMVP) focuses mainly on demographic information and neuropsychiatric symptoms. These findings show that our modality -anchore interaction enables each anchor modality selectively attend the text and sMRI regions most discriminative for its own semantic representation, and promotes meaningful ful interactions between modalities, allowing them to complement each other and produce more discriminative multi-modal representations.
D.6 MULTI-MODAL BIOMARKER ASSOCIATIONS VIA Q-FORMER
We visualize the attention maps of AD patients' sMRI scans across different machor-modalit models trained in Stage 2 (BrainMVP and NT-Transformer), highlighting the brain regions that each model attends to most during prediction, as shown in Fig [2] Specifically, the NT-Transformer trained with genetic features (including APOE 4 status) can highlight hippocampal and medial temporal lobe regions on sMRI, as shown in Fig 12 b. This pattern aligns closely with well-established AD neurobiology: extensive prior studies [i et al. (016)); (Bailey et al. (2024) have shown that APOE 4 is strongly associated with hippocampal and parahippocampal atrophy. Therefore, the fact that our gene-anchored model also attends to hippocampal regions on sMRI indicates that the q-former successfully captures meaningful cross- moddll biomarker relationships. This provides experimental evidence that our model learns biologically grounded correspondences between different modalities.
D.7 FURTHER EXPLANATION OF FIGURE 3
In this section, we provide a detailed analysis of Fig. 3 focusing on ablation experiments conducted under the modality-complete setting with various modality combinations. The combinations are grouped into five categories: Neuroimage Unimodal, Non-imaging Unimodal, Clinical Textual Bimodal, Neuroimage Bimodal, and Modal. In the Neuroimage Unimodal group, sMRI and fMRI are evaluated independently to assess their individual predictive capacity. The Non- Non-imaging Unimodal group includes clinical textual records and genomic data, also evaluated separately. For the
25

---

Published as a conference paper at ICLR 2026
AND eodko
Thik we beclere 4000
Subject I
praniline
sidiad with
Subject
NT transformer Clinical Records
Subject 3
Subject 4
askind tooke lett paou diaac ac uneed (ved)
BrainJePA Clinical Records
Subject 5
Fomeur that
Malicl
Subject
Buch ua Dug coneect Dug conol Legun Saany ocands the Evenn here
BrainJePA
NT transformer
Llama 2
BrainMVP Clinical Records
Figure 11:
Attention maps of sMRI images and clinical text records across anchor modalities and their models, computed for NC/AD classification on the ADNI dataset.
(a) BrainJoPA
(9) NT-Tranafommr
Figure 12: Multi-Modal Biomarker Associations via Q-Former Mechanisms.
Clinical Textual Bimodal group, clinical textual data is paired in turn with each of the other three modalities to assess complementary effects. In the Neuroimage Bimodal group, sMRI and fMRI are each combined with one of the remaining modalities to evaluate how neuroimaging enhances different types of information. Finally, the Multi-modal group incorporates all four modalities to assess the full potential of multi-modal integration. Striped red bars indicate ablation settings where specific
26

---

Published as a conference paper at ICLR 2026
Table 11: Ablation studies clinical assessment scores (MMSE, MoCA, and CDR) evaluate their contributions to our framework under the modality-complete setting of the ADNI dataset.
NC vs. MCI
NC vs. AD
pMCI vs. sMCI
Methods
ACC
AUC
ACC
AUC
ACC
AUC
Forest (only scures)
0.703
0.667
Random Forest
0.709 0.007
0.711 + 0.020
0.745 0.007
0.768 + 0.009
0.696 + 0.010
0.670 0.012
Ours Scores
0.871 0.012
+ 0.010
0.846 0.015
0.854
0.763 0.022
0.786 0.027
Ours Scores
0.850 0.055
0.840 a: 0.103
0.819 0.105
0.822 a 0.054
0.726 0.078
0,751 = 0.027
Table 12: Parameter complexity of modality-aware Q-Former with increasing number of modalities.
Number of Modalities
Modalities
Q-Former Parameters (M)
Clinical Recunis
9.32
2
Clinical Records + Gene
26.52
3
Clinical Records Gene sMRI
45.56
4
Clinical Records + Gene + sMRI IMRI
63.82
modalities are removed from bi-modal multi-modal combination to assess its combination contribution. For instance, in the Clinical Textual bimodal group. the red striped bars represent the removal of clinical data. In the Neuroimage bimodal group, the striped bars reflect the exclusion of neuroimaging data. When sMRI and fMRI are combined, the striped bars correspond to removing sMRI to evaluate the standalone impact of fMRI. In the Multi-modal group, the striped bars indicate the exclusion the G,S and modalities.
D.8 SCALABILITY ANALYSIS OF THE MODALITY-AWARE Q-FORMER
As shown in Table 12 compared the incorporated uni-modal foundation models, which range in size from 500M parameters INT-Transformer) to 7B parameters (LLaMA), the Q-Former is relatively lightweight module with only 60M parameters(additonns cost compared late fusion baseline). As such, it does not introduce much computational overhead to the overall framework. Furthermore, the Q-Former's parameter count increases with the number of modalities, shown in the table below. For instance, starts with 9.32M parameters for the Clinical records modality alone and increases 63.82M when four modalities are included. This demonstrates that the additional computational cost introduced by Q-Former remains manageable more modalities are integrated.
D.9 ABLATION STUDIES ON CLINICAL ASSESSMENT SCORES.
We conduct an ablation studies on clinical assessment scores, and results in Table [1 show that clinical assessment scores play important role in model performance. Our ablation experiments confirm that removing these scores leads to measurable performance drop. However, the model still achieves strong results even without them, indicating that it effectively leverages broader textual information beyond just the structured assessment scores. Furthermore, our model achieves stronger result andd outperforms the Random Forest baseline significantly, indicating that our method leverages rich textual information beyond clinical scores.
E
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We use LLMs solely for checking grammar and polishing writing.
27


Reviews:
[{"review_id": "p2UVRJF7qh", "reviewer": "Reviewer_bkBU", "summary": "The paper proposes a multi-modal framework for Alzheimer’s Disease (AD) diagnosis that integrates several uni-modal foundation models (for sMRI, fMRI, clinical records, and genetic data). The core innovation is the modality-anchored interaction mechanism, where one modality serves as an anchor, and others act as auxiliary inputs. To align heterogeneous feature spaces, the authors introduce modality-aware Q-formers, transformer-based connectors that selectively map auxiliary features into the anchor modality’s feature space. The method is evaluated on ADNI, OASIS-3, and PPMI datasets for AD and Parkinson’s disease, showing state-of-the-art accuracy and generalization under both modality-complete and modality-incomplete scenarios.", "questions": "1. Q-former Configuration: How sensitive are the results to the number of queries and attention heads in the modality-aware Q-former? Could a lighter version achieve comparable performance?\n2. Scalability: Can the approach be generalized to more than four modalities or to non-neuroimaging domains?\n3. Interpretability: Can the method mechanisms be linked to specific neurobiological or genetic markers for clinical interpretability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Innovative framework: The modality-anchored interaction and modality-aware Q-former are novel and conceptually strong for integrating pre-trained models across heterogeneous medical modalities.\n2. Strong performance: The proposed method consistently outperforms baselines across tasks, achieving superior accuracy, specificity, and sensitivity in AD diagnosis and progression prediction.\n3. Comprehensive evaluation: Results are validated on multiple datasets (ADNI, OASIS-3, PPMI), demonstrating solid generalization to out-of-distribution data and to a different disease.\n4. Thorough experimentation: Includes modality-complete/incomplete settings, ablation on foundation model choice, number of queries, and visualization of attention maps for interpretability.", "weaknesses": "1. Limited interpretability for clinicians: While attention visualization is provided, the interpretability of the fused multimodal decision process remains abstract; more clinical linkage to specific biomarkers would be valuable.\n2. Complexity and scalability: The two-stage fine-tuning pipeline increases computational demands; practical feasibility in clinical deployment is not analyzed.\n3. Dependence on large pre-trained models: The framework assumes availability of powerful foundation models, which may not always be accessible or easy to fine-tune with limited resources.\n4. Limited cross-modal interpretive analysis: The paper would benefit from a deeper analysis of why certain modality combinations improve results (e.g., specific complementarity between clinical and imaging data).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HTD0JyMJ1j", "reviewer": "Reviewer_x1Jq", "summary": "This paper proposes to use multiple Q-formers to align various medical modalities for Alzheimer's disease prediction. The authors specifically use train these Q-formers by taking one modality as an anchor, and aligning the other modalities to the latent space of the anchored modality.", "questions": "1) Figure 2: What is the difference between the authors' method and a Q-former?\n2) The unimodal performance for the M4Survive and LateFusion models is sometimes worse than the unimodal performance, which begs the question: How fair are the comparisons with the baselines? Do the authors use the same unimodal foundation models and fine-tuning approaches? For example, in the authors' model they use fine-tuned unimodal foundation models, is this also the case for the multimodal baselines?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The quantitative results are quite convincing, the authors compare the method across three different datasets, and they perform various ablation studies.", "weaknesses": "1) The authors focus a lot on classification performance, but there are many other tasks that are possible within these datasets. Specifically, to further strengthen their results the authors can show that their model also performs better when predicting biomarkers, age, etc. The classification tasks, although important, are not the only way to evaluate these models.\n2) The approach is not that technically novel. There are many papers that align modalities using Q-formers, and it is therefore unclear to me what the technical novelty is in this paper beyond a new application area. A paper that mostly focuses on a new application area is fine, but the application area is quite limited (the authors restrict themselves to Alzheimer's disease instead of broadly neurological disorders), and although results are clearly better in the tables, it is unclear how significant these results are.\n3) This brings me to the third point, which is that the authors do not seem to use cross-validation when evaluating their model(s), and do not compute confidence intervals or standard deviations for their results. Especially in neuroimaging, where the exact training and test subsets can have a large impact on the results, it is important to perform experiments over multiple training and test sets, and even initialization seeds, to ensure results are repeatable.\n4) The authors mention modality-aware Q-formers in the introduction, but do not discuss the referenced papers in depth in the related work section. In general, I think the authors can do a much better job at placing their work into the context of current work, especially given how large the field of multimodal foundation models is. Make sure to highlight exactly how your model and use of Q-formers is different, and why this is leading to performance improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2mQMDRX6iL", "reviewer": "Reviewer_gHU6", "summary": "This paper presents a novel multi-modal framework for diagnosing Alzheimer's Disease (AD) through the joint adaptation of uni-modal foundation models. The central concept is the utilization of a \"modality-anchored interaction\" strategy. In this strategy, the foundation model of one modality serves as an anchor, and features from other auxiliary modalities are projected into its feature space via a proposed \"Modality-aware Q-former\". This enables interaction among modalities while maintaining the pre-trained representations of the anchor model. The framework is evaluated in the context of AD diagnosis and progression prediction, leveraging four modalities: sMRI, fMRI, clinical records, and genetic data, and it showcases superior performance compared to previous approaches.", "questions": "1.  Could you provide more details on the computational cost of the proposed framework, both during training and inference?\n2.  Have you experimented with other methods for aggregating the predictions from the different anchor models? For example, a weighted average based on the confidence of each model.\n3.  How sensitive is the model to the number of learnable queries in the Q-formers?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The proposed modality-anchored interaction framework for integrating uni-modal foundation models is a novel and interesting approach.\n- The paper addresses the important problem of multi-modal AD diagnosis, which has significant clinical relevance. The use of foundation models is a promising direction.\n- The method shows strong performance on AD diagnosis and progression prediction, outperforming several baselines. The generalization experiments on external datasets are a key strength.", "weaknesses": "- The proposed framework is quite complex, involving multiple foundation models, Q-formers, and a two-stage training process. This might make it difficult to reproduce and apply in practice.\n- While the paper provides some analysis, more in-depth ablation studies could further clarify the contribution of each component (e.g., the cross-modal Q-former, the LoRA fine-tuning).\n- The paper states that each modality is used as an anchor in turn, and the final prediction is an aggregation. It would be interesting to see an analysis of how the choice of anchor modality affects performance. Is there an optimal anchor?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZVaweTFsEe", "reviewer": "Reviewer_okfp", "summary": "This paper proposes a multi-modal framework for AD diagnosis that enables joint interaction among uni-modal foundation models through modality-anchored interaction. The experimental results show the effectiveness of the proposed framework.", "questions": "(1) There are several existing works about Modality-aware Q-formers (Tong et al., 2024; Zong et al., 2024; Alayrac et al., 2022; Liu et al., 2023a). Thus, it is not clear about the difference between the proposed model with these existing works. \n\n(2) This paper should compare the proposed model with more state-of-the-art AD diagnosis methods.\n\n(3) The authors should clearly summarize the contributions of this paper.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) This paper proposes modality-aware Q-formers that selectively map auxiliary modality features into the anchor model’s feature space, enabling the anchor model to jointly process its own features together with the seamlessly integrated auxiliary features.\n\n (2) The proposed method is evaluated on AD diagnosis and progression prediction tasks involving the four most common data modalities, and experimental results validate the effectiveness of the proposed framework.", "weaknesses": "(1) This paper designates one modality’s foundation model as an anchor and freezes most of its parameters to preserve its feature space, while projecting auxiliary modalities’ features extracted by other foundation models into this space for cross-modal interaction. This technical novelty is very limited.\n\n (2) There are several existing works about Modality-aware Q-formers (Tong et al., 2024; Zong et al., 2024; Alayrac et al., 2022; Liu et al., 2023a). Thus, it is not clear about the difference between the proposed model with these existing works. \n\n(3) This paper should compare the proposed model with more state-of-the-art AD diagnosis methods.\n\n (4) The authors should clearly summarize the contributions of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)