SMART: Self-Weighted Multimodal Fusion for Diagnostics of Neurodegenerative Disorders

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 15:38:35

Abstract: Multimodal medical data, such as brain scans and non-imaging clinical records like demographics and neuropsychology examinations, play an important role in diagnosing neurodegenerative disorders, e.g., Alzheimer's disease (AD) and Parkinson's disease (PD). However, the disease-relevant information is overwhelmed by the high-dimensional image scans and the massive non-imaging data, making it a challenging task to fuse multimodal medical inputs efficiently. Recent multimodal learning methods adopt deep encoders to extract features and simple concatenation or alignment techniques for feature fusion, which suffer the representation degeneration issue due to the vast irrelevant information. To address this challenge, we propose a deep self-weighted multimodal relevance weighting approach, which leverages clustering-based constrastive learning and eliminates the intra- and inter-modal irrelevancy. The learned relevance score is integrated as a gate with a multimodal attention transformer to provide an improved fusion for the final diagnosis. Our proposed model, called SMART (Self-weighted Multimodal Attention-and-Relevance gated Transformer), is extensively evaluated on three public AD/PD datasets and achieves state-of-the-art (SOTA) performance in the diagnostics of neurodegenerative disorders. Our source code will be available.

Main paper: 
 introduction: 
 Neurodegenerative disorders [19] , e.g., Alzheimer's disease (AD) [9, 49] , Parkinson's disease (PD) [8, 26] , affect millions of people worldwide; unfortunately, they are currently incurable. However, early diagnosis of neurodegenerative diseases is crucial for intervention and provides people affected by dementia the opportunity to access early treatments and make plans for future care. In the clinical Figure 1 : Conceptual comparison between prior state-of-theart methods (top) [5, 60, 64] and our approach (bottom) for the multimodal diagnostics of neurodegenerative disorders.diagnosis of neurodegenerative disorders, medical data commonly exhibits a multi-modal nature, including medical image scans like brain MRIs and non-imaging clinical information, e.g., demographics, serological tests, neuropsychology examinations, etc. The abundance of high-dimensional medical scans and vast clinical data overwhelms the disease-related information. Therefore, how to distinguish and fuse task-discriminative information among these multimodal clinical data is essential but challenging for an accurate computer-aided diagnosis of neurodegenerative disorders.In the early stage, machine learning-based methods require manual selection of disease-related features for diagnosis [40, 47, 56] ; however, the related expert knowledge is too limited to support an accurate selection of diagnostic features. Along with the popularity of deep learning (DL) approaches, automatic feature selection has become a top choice, and most existing methods completely depend on DL techniques to handle the feature selection and fusion task among massive information. As illustrated in Fig. 1 , a typical approach adopted by most researchers [15, 28, 64] is extracting features for imaging and non-imaging data separately and then concatenating them for diagnostic classification. Since task-related and irrelevant information is intermingled, it is exhausting and overwhelming to fuse so much information for diagnosis. Also, due to the space discrepancy of multimodal features and the demonstrated effectiveness of recent vision language pre-training (VLP) models, e.g., CLIP [45] , ALBEF [32] , BLIP [31] , a couple of recent works, e.g., MedCLIP [55] , Alifuse [5] , align imaging features to the text feature space of the non-imaging data and fuse them using a pre-trained large language model or the cross-attention techniques. However, VLP-based models prefer paired image and text data, such as medical scans and reports [58] , where texts describe the associated medical images. Differently, the non-imaging data for clinical diagnosis is partially paired with medical scans and provides complementary diagnostic information about a subject. More importantly, the alignment between imaging and non-imaging happens at the image level, it lacks the fine-grained features that help fuse partially-paired data and contribute to the disease diagnosis.Considering the existence of a large amount of irrelevant information for disease diagnosis in both imaging and non-imaging data, in this paper, we explore the possibility of automatically highlighting and fusing the multimodal features that have a strong correlation with the target disease, as shown at the bottom of Fig. 1 . The disease-irrelevant information exists at both intra-modal and inter-modal levels. To eliminate this irrelevancy issue, we propose a Self-supervised Multimodal Relevance Weighting (SMRW) module, which hierarchically clusters relevant information and learns a relevance score vector to weight within and among multimodal data. Guided by the learned weights, the multimodal feature fusion becomes relatively easy via a transformer with attention mechanisms. Hence, we propose a Multimodal Relevance-gated Attention Transformer (MRAT), which involves intra-modal and inter-modal relevance scores in calculating self-attention and image-to-text and text-to-image cross-attention, resulting in an efficient relevancegated fusion for the diagnostic prediction. Figure 2 depicts the overview architecture of our proposed model, i.e., Self-weighted Multimodal Attention-and-Relevance gated Transformer (SMART), which is applied to the diagnostics of neurodegenerative disorders.Overall, the contributions of this paper are summarized below:(1) We propose a novel framework SMART for multimodal neurodegenerative disorder diagnosis. Extensive experiments on three public benchmark datasets for neurodegenerative disorders like AD and PD demonstrate the superiority of our approach over ten baselines, including previous SOTA methods.(2) We propose a self-weighted multimodal representation learning technique SMRW, which adopts a self-supervised two-level contrastive learning to automatically cluster and weight relevant information at both intra-modal and inter-modal levels. A follow-up relevancy-gated attention module allows an efficient multimodal feature fusion for the final prediction.(3) Thanks to the relevancy score learned by SMRW, our model is explainable to some extent while having a high diagnostic accuracy. Also, our model is theoretically designed for multiple modalities, which could include more modalities like audio to fully leverage all possible medical information. 
related work 2.1 vision-language model (vlm): 
 Vision-language representation learning aims to jointly encode vision and language in a fusion model, which has been demonstrated to learn uni-modal and multi-modal representations with superior performance on downstream tasks [32, 45] . CLIP [45] and ALIGN [24] are dual-encoder models that are pre-trained with contrastive learning objectives on image-text pairs. They learn strong image and text unimodal representations with simple multimodal alignment, which is not enough to handle tasks that require complex reasoning. ALBEF [32] , CoCa [61] , and BLIP [31] promote a deeper interaction between images and text using a deep fusion encoder with cross-modal attention. These models achieve better performance for vision-language classification by learning efficient multi-modal representations. Medical image-text representation learning is investigated based on contrastive learning as well, such as MedCLIP [55] , CheXZero [52] , and MedViLL [38] . However, these methods prefer modeling on paired image and text data, such as medical scans and reports.In medical diagnosis, fully leveraging medical data collected from multiple modalities becomes a popular choice, since multimodal machine learning models can leverage more information and more easily identify patterns of diseases, compared to using a single modality [2, 7, 12, 35, 51, 59] . A straightforward way for multimodal learning is to concatenate features and feed them into a classifier such as SVM or MLP for prediction [14, 21, 40, 43, 48] . To improve the model performance, feature selection methods are applied to reduce the feature dimension [18, 47, 63, 65] . Compared to machine learning methods, deep learning methods are feasible to capture hierarchical representations and achieve better performances. Kim et al. [28] propose a heterogeneous graph learning method to fuse the multimodal medical data. Zhou et al. [64] introduces the IRENE model based on Transformers, which fused representations among modality-specific low-level embeddings for diagnosis. Also, multimodal data like images and genomes have been used in diagnostics of breast cancer [15] . However, most of these studies potentially ignore the heterogeneity between modalities due to a lack of ability to fully explore the task relevance in intra-and-inter modalities, resulting in sub-optimal results.Contrastive learning [17, 41, 44] pulls positive pairs closer and pushes negative pairs farther contrastively to obtain discriminative features. In addition to the single-modality representation learning, contrastive methods for multiple modalities are also widely explored. Multi-modality data often contain two or more modalities and they naturally form multiple inputs. The common methods [3, 24, 27, 32, 45, 50, 62] leverage the cross-modal contrastive matching to align two different modalities and learn the intermodality correspondence. However, since the semantic consistency among intra-modal and inter-modal is not guaranteed, it is challenging to capture the useful information in multimodal data, while considering the side effects of irrelevant information. For example, Yang et al. [60] leverages the maximization of mutual information to conduct consistency learning across different views and aims to achieve a provable sufficient representation. Xu et al. [57] observe the conflict between consistency and reconstruction objectives in encoder-decoder frameworks and propose to learn multi-level features for multiple modalities. Li et al. [34] propose leveraging domain-specific medical knowledge as guiding signals to perform multi-level contrastive learning. Although satisfactory results are achieved in many cases, the multimodal representation degeneration is still not well considered and addressed. Recently, new perspectives of deep clustering have been offered by the development of contrastive clustering [33] . In this paper, we propose a cluster-based self-weighted intra-modal and inter-modal contrastive learning to address the irrelevancy issue in multimodal medical diagnosis. 
methodology: 
 Figure 2 presents the architecture framework of our SMART network. The input to our model is the multi-modality (e.g., brain structure MRI and clinical transcribed text in our case). Our model includes three components: the feature encoding, the Self-supervised Multimodal Relevance Weighting (SMRW) module, and the Multimodal Relevance-gated Attention Transformer (MRAT) module.We extract unimodal features for brain structure MRI and textdescribed clinical data with independent encoders to obtain their hidden representations. Following [64] , we use ViT [11] as the feature encoder for each patch of brain structure MRIs and BERT [10] as the feature encoder for each sentence of the textual description of non-imaging clinical data. Specifically, we denote the generated image and text features as I ∈ R 𝑆 𝐼 ×𝐷 𝐼 and T ∈ R 𝑆 𝑇 ×𝐷 𝑇 , where 𝑆 𝐼 indicates the number of patches of a medical image and 𝑆 𝑇 indicates the number of sentences of the clinical text, respectively. Features of different modalities are then projected into a common embedding space with the same dimension using a linear fully connected layer.In multi-modality learning, variations in data quality within and across modalities are common, posing a risk of representation degradation when merging data or features. Specifically, this degeneration makes the representation of task-relevant data mediocre, potentially leading to the loss of crucial discriminative information, as observed in Fig. 3 . We take a multi-modal dataset ADNI [42] as an example. We first evaluate the representation quality of containing semantics of different categories of clinical information except for image scans, including SC (Subject Characteristics), MH (Medical History), NS (Neuropsychology Scores), BA (Blood Analysis), GN (Genetic), and All of them. Concatenating all categories of clinical semantic information performs worse than simply using NS in AD diagnosis. Similar phenomenon is observed in learning multimodal representation using images and text.To mitigate such representation degeneration issues in multimodal learning, we seek a technique that upgrades the task-relevant information while downgrading the task-irrelevant one, so that all intra-modal and inter-modal relevant information can be highlighted for further fusion. Hence, as shown in Fig. 2 (c) we propose a Self-supervised Multimodal Relevance Weighting (SMRW) module based on cluster contrastive learning, which learns distinct representations for intra-and inter-modal relevant information.X = {X 𝑚 ∈ R 𝑁 ×𝑆 𝑚 ×𝐷 𝑚 } 𝑀𝑚=1 with 𝑀 modalities (e.g. 𝑀 = 2, image and text modalities in our case) and N instances (e.g. the number of patients in our case); 𝑆 𝑚 denotes the total items (e.g. image patches or text sentences) of the 𝑚-th modality, 𝐷 𝑚 denotes the feature dimension of the 𝑚-th modality. X 𝑚,𝑠 = [x 𝑚,𝑠 1 , x 𝑚,𝑠 2 , ..., x 𝑚,𝑠 𝑁 ] denotes the instance collection of x 𝑚,𝑠 𝑖 from the 𝑚-th modality's 𝑠-th sub-items of the 𝑖-th subject. We designate the encoder E 𝑚 and the decoder D 𝑚 for the specific 𝑚-th modality. Encoder E 𝑚 projects the raw data into modality-specific feature space viaEQUATIONresulting in Z 𝑚,𝑠 = [z 𝑚,𝑠 1 , z 𝑚,𝑠 2 , ..., z 𝑚,𝑠 𝑁 ] ∈ R 𝑁 ×𝐷 𝑚 .Inspired by [16] , we employ a decoder D 𝑚 to reconstruct z 𝑚,𝑠 𝑖 , which ensures learning sufficient features from the raw data for clustering those relevant information, i.e., x𝑚,𝑠𝑖 = D 𝑚 z 𝑚,𝑠 𝑖 = D 𝑚 E 𝑚 x 𝑚,𝑠 𝑖 .(EQUATIONThe reconstruction loss involving all modalities is defined as:EQUATIONTo extract discriminative features from data of each modality, we use an adaptive weighting technique to obtain the intra-modal mean feature Z𝑚 ∈ R 𝑁 ×𝐷 𝑚 , where Z𝑚= [z 𝑚 1 , z𝑚 2 , • • • , z𝑚 𝑁 ]𝑖 is defined as:EQUATIONHere, 𝑤 𝑚,𝑠 denotes the 𝑠-th item's weight of the 𝑚-th modality and 𝑆 𝑚 𝑠=1 𝑤 𝑚,𝑠 = 1. The weights are optimized during training, reflecting the importance of different items within a single modality.To optimize the above weights, we adopt a two-layer linear MLP F 𝑚 intra to map the modal-specific features and the mean intra-modal feature to a 𝐾-dimension space. Here, 𝐾 is the number of clusters. Then, a softmax layer for normalization is attached to obtain the probability of each feature belonging to each cluster. That is, we obtain the probability matrix for each intra-modal item of each subject, i.e., 𝐶 𝑚,𝑠 = [𝑐 = [c 𝑚 1 , c𝑚 2 , • • • , c𝑚 𝑁 ] ∈ R 𝑁 ×𝐾 . Specifically, 𝑐 𝑚,𝑠𝑖,𝑗 denotes the probability of the 𝑠-th item of the 𝑖-th subject from the 𝑚-th modality that belongs to the cluster 𝑗. Since the intra-modal items from the same cluster should have similar features, cluster contrastive learning decreases the distance between item pairs from the same cluster while increasing the distance between those pairs from distinct clusters. Therefore, {c 𝑚 𝑖,𝑗 , 𝑐 Based on this metric, we define the intra-modal cluster contrastive loss asEQUATIONHere, 𝜏 1 denotes the temperature coefficient. The total intra-modal loss is:EQUATION3.2.2 Inter-modal Contrastive Learning. Similarly, we apply intermodal contrastive learning to learn modality consistency across multiple modalities. We also use the above adaptive weighting technique to obtain the inter-modal mean featuresZ = [ z1 , z2 , • • • , z𝑁 ] ∈ R 𝑁 ×𝐷, where each z𝑖 is computed as:EQUATIONHere, 𝑤 𝑚 is the weight of the 𝑚-th modality and Transformers [53] have demonstrated their superior advantages in modeling different modalities, e.g., visual, language, and audio, on various multimodal tasks, e.g., visual question answering [1, 30, 39] , vision-language pre-training [4, 5, 55] . Therefore, we adopt the transformer architecture to fuse our multimodal weighted features, as shown in Fig 2(b ). After obtaining image and text features and their relevance scores from our SMRW module, we fed them into a feature enhancer for cross-modality feature fusion. The feature enhancer includes two feature enhancer layers, i.e., an image-totext cross-attention and a text-to-image cross-attention for feature fusion. These modules help align features of different modalities, which are fed into a feed-forward network (FFN) layer, respectively. After such deep interaction between image and text features, we concatenate and pass them into a self-attention layer for modality alignment and fusion and finally output the diagnostic prediction. Different from the commonly-used self-and cross-attention layers, we propose the relevance-gated attention operations to fully leverage the relevance scores learned during the SMRW module. Specifically, our attention block is defined as: AttentionBlock(𝑄, 𝐾, 𝑉 , 𝑅) = LN(𝑄 + FFN(𝑄 + Attn(𝑄, 𝐾, 𝑉 , 𝑅))), (10) where 𝑄, 𝐾, 𝑉 represents the query, key, and value features, 𝑅 is the relevance score obtained from SMRW, LN(•) represents a layer normalization, and the feed-forward network, FFN(•), is a sequence of layer normalization, linear transformation, GELU activation, and another linear transformation. Lastly, the attention mechanism, Attn(𝑄, 𝐾, 𝑉 , 𝑅), is calculated as follows:𝑄 ′ = Linear(𝑄), 𝑅 ′ = LN(Linear(𝑅)), 𝐾 ′ = Linear(𝐾 ⊕ 𝑅 ′ ), 𝑉 ′ = Linear(𝑉 ), Attn(𝑄, 𝐾, 𝑉 , 𝑅) = Softmax 𝑄 ′ 𝐾 ′⊤ 𝑉 ′ . (11)Here, ⊕ denotes addition along the feature dimension, and Linear(•) represents a linear transformation.We employ three loss functions to train our model, including one classification loss and two self-supervised relevance weighting losses, i.e., the intra-and inter-modal contrastive learning losses. In the training process, the classification loss is constructed by a cross-entropy function:EQUATIONwhere 𝑝 𝑐 𝑖 is the predicted classification score of each subject for each class 𝑐, while 𝑦 𝑐 𝑖 is the corresponding one-hot ground-truth label. The overall loss is defined below:EQUATIONwhere L intra and L inter represent the intra-modal contrastive loss given in Eq. ( 7) and the inter-modal one in Eq. ( 9), respectively; 𝛼 and 𝛽 are hyper-parameters balancing these three terms. 
experiments 4.1 datasets: 
 Three public multimodal datasets are evaluated in this study, with considering both medical image scans (e.g., structure MRIs) and non-imaging clinical information (e.g., demographics, serological tests, neuropsychology examinations). The detailed statistics of these datasets are summarized in Table 1 .ADNI [42] . The ADNI dataset is a longitudinal and multi-site multimodal neuroimaging dataset. We collect 1042 subjects for evaluation, including 342 normal controls (NC), 351 with mild cognitive impairment (MCI), and 349 patients with Alzheimer's disease (AD) from ADNI1, ADNI2, ADNIGO, and ADNI3. We excluded followup scans and included a single baseline scan of the structure MRI and non-imaging clinical information per subject. These subjects are divided into three groups (i.e., AD, MCI, and NC), and MCI is considered to be a significant stage for the preclinical AD diagnosis.AIBL [13] . A total of 858 subjects are included in this dataset, where 609 NCs, 144 MCIs, and 105 AD subjects were recruited from the same technical infrastructure as the ADNI. AIBL is also a longitudinal dataset and we process it in the same way with ADNI.PPMI [37] . The PPMI dataset includes 247 NCs and 352 PDs. The patients were diagnosed at baseline, and the NCs were healthy at their first examination. Notably, the PPMI dataset is also a longitudinal database, and each participant has multiple scans. We also included As shown in Table 1 , we collect over a hundred pieces of nonimaging clinical data for each patient in the ADNI dataset. For instance, there are 20 medical histories for each patient. Clinical text data are semantically prepared before encoding for all datasets. Clinical variable terms are collected and cleaned using the ADNI Data Dictionary or PPMI Data Dictionary, and then semantically aggregated to the appropriate level of granularity to ensure a less sparse dataset. For example, if a patient's HMT3=4.39, a corresponding sentence is written as "Red blood cell count is 4.39". If a patient's MH4CARD=1, then it is described as "has a cardiovascular medical history". We argue that its contained digital numbers and categories are more meaningful in the text context. Therefore, we create a text representation for each data sample using the information contained in the tables of raw data. Regarding the collected T1weighted structural MRI (sMRI) scans, we apply the same data pre-processing to normalize and standardize them from a multiinstitutional database. We resize the images to have the same voxel spacing (i.e., 1.75mm × 1.75mm × 1.75mm) and the same volume size (i.e., 128 × 128 × 128), then normalize the image intensities of all the voxels using the zero-mean unit-variance method.The experimental framework is implemented on the Pytorch platform and executed on a 24GB NVIDIA Geforce RTX 3090 Linux server. The image patch size is set to 16 × 16 × 16. The encoder and decoder in SMRW are implemented using fully connected layers with the encoder dimensions of input-500-1000-1000-512. The decoder is symmetric with the encoder. We set hyper-parameters 𝛼 and 𝑏𝑒𝑡𝑎 both to 1. Adam optimizer is adopted with a learning rate of 3e-4 and a batch size of 32, trained for 300 epochs. For all experiments, we evaluated the performance in terms of two metrics, i.e., accuracy (ACC) and area under curve (AUC).Baselines. In this study, we compare our proposed SMART with baseline deep learning approaches and well-estimated transformerbased methods. These methods include: (1) the image-only group. We choose four recent baselines that are commonly used, including Net50 [20] , Med3D [6] , VisionTransformer [11] , M3T [23] . ( 2)The text-only group. We choose BERT [10] and RoBERTa [36] as our text-only baseline. (3) the multi-modal group. We have four baselines, i.e., GIT [54] , Preciver [22] , IRENE [64] and Alifuse [5] . They are recent transformer-based models that fuse multi-modal information for classification. IRENE and Alifuse are previous SOTA methods designed for multimodal medical diagnosis.Comparison Results. Table 2 and Table 3 report the comparison results of our SMART with ten baseline methods on three datasets We conduct ablation studies on ADNI and AIBL to evaluate the crucial factors in our proposed SMART, as shown in Table 4 . Here we provide some implementation details. (a) "w/o L res intra ": we remove our intra-modal reconstruction loss in the SMRW module in the training process. (b) "w/o L intra ": we remove our intra-modal constrastive learning loss in the training process. (c) "w/o L inter ": we remove our inter-modal constrastive learning loss in the training process. (d) "w/o SMRW": we remove the SMRW module and replace the task-relevance score with a randomly initialized learnable tensor, which has the same shape as the value features 𝑉 . (e) "w/o MRAT": we remove the guidance of relevance in MRAT by replacing the task-relevance score with a zero tensor, which has the same shape as 𝑉 as well. (f) "-I": we only train the visual modality in SMRW and remove its textual branch by replacing the textual feature with zero tensor. (g) "-T": we perform a similar operation as (f), except that we use the prediction of the textual branch during inference. (h) "-S": our standard model with both images and text. 4 , we observe that the prediction accuracy improves significantly when trained with self-supervised multimodal relevance weighting, verifying the effectiveness of our self-weighted task relevance technique. Taking one step further, on the ADNI dataset we achieve ∼15% performance improvement on ACC by using SMRW, and ∼14% performance improvement on AUC; while on AIBL, these two numbers are about 14%, 11%. This indicates that our SMRW module can alleviate the information irrelevancy efficiently especially when facing a large amount of data. We further perform ablation studies on loss functions in SMRW, which are reported in rows of (a,b,c), intra-modal contrastive learning is more helpful than inter-modal contrastive learning. Also, it suggests that to prevent the model from crashing, reconstruction regularization is crucial on smaller datasets with varying data quality. 4 , there are great performance drops of ∼4% on ACC of ADNI and 3% on ACC of AIBL, if the relevance-gated attention is removed. This indicates that the guidance of the relevance score is essential for a more accurate diagnosis and reduces the difficulty of fusing high-dimensional multimodal data.Benefits of Multimodal Learning. Comparing the rows (f,g) with (h) in Table 4 , there are also dramatic drops of 11%, 12% on ACC and AUC of ADNI if the text modality is removed, and 1.5%, 2% drops on ACC and AUC when the image modality is removed. This ablation study demonstrates that the multi-modal design outperforms the unimodal one, showing the necessity of using both imaging and non-imaging clinical data for neurodegenerative disorder diagnosis. Compared to the imaging data, the non-imaging data is more effective in classifying AD. Thus, it is worth to discover and amplify the crucial clues within the non-imaging clinical text data.To Pretrain or Not? Since our SMRW module is self-supervised, one potential attempt would be using the self-supervised loss in SMRW to pre-train. For instance, we first pre-train the SMRW module for 100 epochs, which is followed by another 200 epochs using the overall loss functions. As shown in Figure 4 , the earlier stage performance improvement is stable with pre-training SMRW. Interestingly, in the ADNI experiment, the later adjustment in the optimization process weakens the model's performance. This indicates that suitable pretraining is a good choice while excessive adjustment using supervised signals would discard some valid information learned in the early stage of training.Figure 5 shows the visual comparison between SMART and the baseline, i.e., our method witout the SMRW module. The image patches and text sentences with the top five highest scores are selected for comparison. We observe that the baseline model misses to pay great attention on some important biomarkers [25, 29, 46] . On the contrary, our model demonstrates better locations of those key image subregions and text sentences.To further demonstrate the interpretability of our model, we compare SMART with the previous SOTA method Alifuse. Figure 6 shows the attention weight distribution over the text description of the non-imaging clinical data. We discover that our model can capture the feature of a longer sequence. In the sentence "ApoE4 gene detection of apolipoprotein", which is a biomarker of Alzheimer's disease [29] , Alifuse exerts little attention, while our model does capture task relevance between the meaning of the words and the neurodegenerative disease. Our method demonstrates its ability to handle a long text description to some extent. This is essential in the diagnostics of neurodegenerative disorders since a lot of nonimaging clinical information is desired to be explored to identify the unknown yet potential biomarkers for diagnosis. 
conclusion: 
 In this paper, we propose a novel diagnosis framework SMART that fuses multimodal medical data via a self-supervised weighting approach and augments transformer attention techniques to achieve a more accurate diagnostic classification. We demonstrate the effectiveness of our model in classifying both AD and PD, which outperforms ten baselines and achieves the SOTA classification accuracy. Currently, we only consider brain MRI scans; in future work, more image modalities like PET and even others like audio will be included as well for more accurate diagnosis. Another extension to our current model is the incorporation of longitudinal data. Because degeneration over age is an essential characteristic of neurodegenerative disorders, multimodal fusion on longitudinal data will be our future work as well. 
 
 Appendix:  


Decision: The paper introduces SMART, a deep self-weighted multimodal correlation weighting method that uses cluster-based constraint learning to eliminate intra- and intermodal uncorrelations. This method is designed to improve the fusion of multimodal data for diagnosing neurodegenerative disorders, specifically Alzheimer’s disease (AD) and Parkinson’s disease (PD). The proposed approach is validated on three datasets, demonstrating its effectiveness and generalization capabilities.
The paper presents a well-structured and technically sound approach to multimodal medical data classification, addressing a significant challenge in diagnosing neurodegenerative disorders. Despite some areas needing further explanation and justification, the method's validation on multiple datasets and the comprehensive experiments provide strong evidence of its effectiveness. The paper's contributions are valuable to the field and warrant acceptance as a poster.

Reviews: review: paper_summary: paper_summary: This work introduces SMART, a deep self-weighted multimodal correlation weighting method that utilizes cluster-based constraint learning to eliminate intra- and intermodal uncorrelations. The learned correlation scores are integrated as gates with multimodal attention converters to provide improved fusion for the final diagnosis and validated on three datasets (AD/PD).
strength_weakness: strength_weakness: 1.The multimodal problem is widespread in the field of multimodel medical data classification. However, previous work in this field has not dealt with it properly. This work provides an excellent formulation of the problem for describing this study and proposes valuable solutions that can be used to inform subsequent research.
2.The paper is very well organized. The research question is fully described.
3.Validation was carried out on three datasets, which is important to demonstrate its generalization.
limitations: limitations: 1. One weakness of the paper is that some architectures, experimental designs, and parameter choices are not explained and not proven. For example, the choice of cosine as the similarity function, the effect of two-layer MLP on intra-modal mean feature, etc
2. Again, while listing the hyperparameters used in the model can help improve repeatability, this alone is not enough because their choices are irrational. A search in the main paper shows no mention of hyperparameter tuning.
3.In Table 1, the AIBL dataset has the least clinical data with class-imbalance, but best results are achieved in Tables 2 and 3, and the reasons for this phenomenon need to be explained.
4.Other missing but required elements include: How is the entire framework trained? In particular, self-supervised modules and the whole networks. We know that clinical data is often insufficient, what is the training strategy for this work?
5. In the experiment, the results of the mean ± standard deviation are more convincing.
paper_summary: paper_summary: The paper discusses the challenges and solutions in diagnosing neurodegenerative disorders, such as Alzheimer's disease (AD) and Parkinson's disease (PD), using multimodal medical data. This data includes brain scans and non-imaging clinical records like demographics and neuropsychology examinations. The main issue in fusing these multimodal inputs lies in the overwhelming amount of irrelevant information contained within the high-dimensional data.
strength_weakness: strength_weakness: 1. Novelty and Theoretical Approach
The paper introduces a novel method called SMART (Self-weighted Multimodal Attention-and-Relevance gated Transformer), which leverages deep self-weighted multimodal relevance weighting combined with clustering-based contrastive learning. This innovative approach effectively addresses the challenge of representation degeneration in high-dimensional medical data by filtering out irrelevant information, ensuring a more accurate and robust feature fusion process.

2. Technical Correctness and Adequate Evaluation
The proposed methodology is technically sound, with a clear and detailed description of the algorithms and model architecture. The SMART model is evaluated extensively on three public datasets for Alzheimer's disease (AD) and Parkinson's disease (PD), where it achieves state-of-the-art (SOTA) performance. Comparisons with existing methods provide strong evidence of its superior accuracy and robustness.

3. Clarity and Practical Applications
The paper is well-structured and clearly written, making complex concepts accessible. The practical implications are significant, particularly in improving the diagnosis of neurodegenerative disorders. By enhancing diagnostic accuracy, SMART has the potential to positively impact clinical decision-making and patient outcomes. Furthermore, the authors' commitment to sharing their source code promotes transparency and encourages further research in this domain.
limitations: limitations: Weakness: Lack of Discussion on Related Work <<Visual-Attribute Prompt Learning for Progressive Mild Cognitive Impairment Prediction>> in MICCAI 2023
1. Insufficient Literature Context
The paper does not adequately discuss related work in the area of visual-attribute prompt learning, particularly as it applies to progressive mild cognitive impairment (MCI) prediction. This omission leaves a gap in the understanding of how the proposed SMART model compares or relates to recent advancements in this specific domain. For instance, visual-attribute prompt learning has shown promise in dynamically integrating visual features with clinical attributes, which could provide valuable insights or complementary techniques for improving the diagnosis of neurodegenerative disorders.

By not including a discussion on visual-attribute prompt learning approaches, the authors miss an opportunity to compare and contrast their method with potentially relevant and recent methodologies. This comparison is crucial as it can highlight the unique contributions of the SMART model and provide a clearer understanding of its relative strengths and weaknesses. For example, examining how visual-attribute prompt learning handles feature fusion and relevance weighting could offer insights into possible enhancements or alternative strategies for the SMART framework.
paper_summary: paper_summary: The paper introduces SMART, a framework for diagnosing neurodegenerative disorders, outperforming ten baselines across three benchmark datasets. Key contributions include a novel self-weighted multimodal representation learning technique, SMRW, which enhances model explainability while maintaining high diagnostic accuracy. SMART is designed to incorporate multiple modalities, potentially improving diagnostic capabilities by leveraging diverse medical information sources.
strength_weakness: strength_weakness: One strength of this paper lies in its introduction of a SMART framework, which demonstrates superior performance over existing methods across multiple benchmark datasets. Additionally, the proposed self-weighted multimodal representation learning technique, SMRW, contributes to the model's interpretability while maintaining high diagnostic accuracy. Furthermore, the model's theoretical design allows for the incorporation of multiple modalities, potentially enhancing its diagnostic capabilities by leveraging a wide range of medical information sources.
limitations: limitations: 1. Why authors did not use more advanced metrics like precision, recall, specificity, f1 score, and so on included with AUC and accuracy metrics to evaluate their proposed approach performance?

2. Why authors did not use PSNR and SSIM metrics to compare reconstructed images with original images? In addition, this paper does not have a qualitative analysis regarding the reconstructed images which can able to compared with the original images.

3. the authors should discuss the impact of each modality employed in the study on enhancing robust performance in the results section.

4. The authors should conduct thorough performance comparisons with recent existing multimodal approaches to provide insights into the effectiveness of their proposed method. It would enhance the paper's contribution and provide valuable context for readers.

5. A detailed discussion is required because how complementary information from different modalities influenced the improvement of the learning model's performance is essential. Exploring the synergies between modalities and analyzing how they contribute to performance enhancement would provide valuable insights for researchers and practitioners in the field.

6. Regarding the writing of the method section lacks proper presentation according to these highly reputed conference standards, as there is no clear problem formulation provided to enhance the understanding of the paper better.
paper_summary: paper_summary: The paper introduces a novel approach called SMART that leverages a deep self-weighted multimodal relevance weighting technique to eliminate irrelevant information and improve fusion for accurate diagnosis. By integrating clustering-based contrastive learning and a multimodal attention transformer, the proposed model achieves state-of-the-art performance on three public datasets. The key contributions include the development of SMART, the introduction of the self-weighted multimodal representation learning technique SMRW, and the explainability and theoretical design for multiple modalities.
strength_weakness: strength_weakness: 1. The logic of this work is straightforward.
2. The motivation behind this work is clear.
3. The diagram of the framework is helpful to understand.
4. The methodology is reasonable.
5. The experimental section is comprehensive.
limitations: limitations: 1. This work claims that the proposed method allows efficient multimodal feature fusion; however, it does not provide evidence to support its efficiency. Why is the proposed method considered efficient? Could you please explain and provide supporting evidence?

2. The proposed framework lacks novelty to some extent. It is a combination of well-explored techniques. 

3. There is no source code provided, and the experiment results show that the proposed method almost outperforms all baseline methods in all cases, which is infrequent. I kindly request that you check the experiment setting to avoid unexpected mistakes like data leakage.

3. Some typo mistakes, e.g, 4.4 Experimental results, (3) 
"the multi-modal ..." should be "The multi-modal..."