Language-Guided Visual Prompt Compensation for Multi-Modal Remote Sensing Image Classification with Modality Absence

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 15:33:23

Abstract: Joint classification of multi-modal remote sensing images has achieved great success thanks to complementary advantages of multi-modal images. However, modality absence is a common dilemma in real world caused by imaging conditions, which leads to a breakdown of most classification methods that rely on complete modalities. Existing approaches either learn shared representations or train specific models for each absence case so that they commonly confront the difficulty of balancing the complementary advantages of the modalities and scalability of the absence case. In this paper, we propose a language-guided visual prompt compensation network (LVPCnet) to achieve joint classification in case of arbitrary modality absence using a unified model that simultaneously considers modality complementarity. It embeds missing modality-specific knowledge into visual prompts to guide the model in capturing complete modal information from available ones for classification. Specifically, a language-guided visual feature decoupling stage (LVFD-stage) is designed to extract shared and specific modal feature from multi-modal images, establishing a complementary representation model of complete modalities. Subsequently, an absence-aware visual prompt compensation stage (VPC-stage) is proposed to learn visual prompts containing missing modality-specific knowledge through cross-modal representation alignment, further guiding the complementary representation model to reconstruct modality-specific features for missing modalities from available ones based on the learned prompts. The proposed VPC-stage entails solely training visual prompts to perceive missing information without retraining the model, facilitating effective scalability to arbitrary modal missing scenarios. Systematic experiments conducted on three public datasets have validated the effectiveness of the proposed approach.

Main paper: 
 introduction: 
 Joint classification of multi-modal remote sensing images is an efficient technique that integrates information from various modalities to achieve precise classification of land unit elements [9] . It plays a crucial role in the earth observation tasks such as land analysis and utilization [12] , urban planning and management [3] , as well as environmental conservation and monitoring [28] . Recently, the researches of joint classification have shown great success, with their outstanding performance relying on the exploration of complementary advantage from complete modalities [7, 31, 42] . However, modality absence is a common dilemma [21, 38] in real-world scenarios due to sensor malfunctions or inconsistent satellite revisit period. This makes it challenging for traditional classification models to extract effective discriminative features from the limited modal data, resulting in a significant degradation in classification performance. Therefore, it becomes essential to develop joint classification method that can cope with modality absence.The existing methods to address the issue of modality absence in multi-modal classification can be categorized into three types: generation-based [36, 46] , transfer learning-based [33, 34] and modalityshared latent space learning methods [6, 10] . The generation-based methods restore missing modality images by synthesizing information from available modalities through generative network [2, 46] . Nevertheless, due to the instability of image generation, this may introduce considerable noise, which is harmful for classification [8] . Transfer learning-based methods typically transfer knowledge from a full-modal network to a network with modality absence through knowledge distillation [14] , thereby optimizing the classification boundary with modality absence. Whereas it is challenging to guide the network with modality absence to inherit complete modal information owing to significant heterogeneity among remote sensing images of different modalities, which may lead to sub-optimal performance [35] . Although promising results can be obtained, these methods require training a specific model for each missing scenario [37] , which undoubtedly introduces a significantly additional training parameters, severely limiting their scalability to application scenarios with arbitrary modality absence. To alleviate this limitation, modality-shared latent space learning methods aim to learn a unified model for various modal combinations [45] . It utilizes the latent commonality between modalities for classification, typically establishing shared subspace for all modalities and learning modalityinvariant features to mitigate the influence of modality gap [6] .However, the discriminative ability in feature representation of such methods is limited since they solely focus on modality-shared features [19] , neglecting modality-specific information, thus forfeiting the complementary advantages of multi-modality. These discussions motivated us to pose a research question: Can a unified model be constructed that simultaneously considers both modalityshared and modality-specific information, while remaining robust to arbitrary modality absence without incurring significantly additional training parameters?To address the above issue, we draw inspiration from prompt learning. The essence of prompt learning is to design prompts for downstream tasks, guiding pre-trained models to perform the anticipant tasks without modifying itself, where knowledge about the task is embedded as prompts in input tokens to help network understand the meaning of task. Inspired by this, we propose a language-guided visual prompt compensation network (LVPCnet) to achieve joint classification of multi-modal remote sensing images in case of modality absence. In this designed framework, the classification model can be guided to capture modality-specific information of missing modalities from available ones by learning a visual prompt that can perceive the missing modality knowledge, enabling the acquisition of complete modal information for classification. Concretely, it is achieved by a two-stage training process: language-driven visual feature decoupling stage (LVFD-stage) and absence-aware visual prompt compensation stage (VPC-Stage). The LVFD-stage decomposes multi-modal images into modal-shared and modality-specific representations through a shared encoder and multiple specific encoders, establishing a complementary feature representation framework. Unlike common decomposition methods, we employ modality attribute-associated language priors to guide the decoupling of multi-modal visual features under multidimensional visual-language alignment constraints. This approach leverages the rich semantic information provided by language to help the visual system better understand and interpret modal content. The proposed VPC-stage takes available modalities as input to the pre-trained feature representation framework of the LVFDstage, and integrates visual prompts with specific encoders for missing modalities and employ cross-modal representation alignment. This allows visual prompts to learn specific knowledge about missing modalities, thereby guiding these specific encoders to extract specific features of missing modalities from available ones.To summarize, the contributions of this work are as follows:• We propose a unified model LVPCnet for joint classification with arbitrary modal absence, which incorporates the modality complementarity through reconstructing the specific feature of missing modalities by learning visual prompts capable of perceiving missing modality-specific knowledge.• We design an language-driven visual feature decoupling stage (LVFD-stage) for multimodal image decoupling, where language priors are utilized to explicitly guide the model to capture modality-specific knowledge, facilitating subsequent visual prompts to adeptly acquire the specific knowledge associated with the absent modality.• We design absence-aware visual prompts for guiding the compensation of missing modality-specific features from the available ones, a process that only requires training the prompts without modifying original model, facilitating extension to arbitrary missing scenarios. 
related work 2.1 modality absence in multi-modal learning: 
 The issue of modality absence is common in multi-modal learning due to the limitations of imaging conditions, and several studies have emerged to provide solutions for overcoming modality absence [18, 29, 39] . Ma et al. [22] proposed the SMIL model, which applied a Bayesian meta-learning framework to learn the weighted sum of modal priors from complete modalities to reconstruct the features of missing modalities. Pande et al. [23] introduce an adversarial training-driven hallucination architecture that employs a cross-modal hallucination module based on C-GAN to generate discriminative features related to the missing modality from available modalities. MMIN [41] leverages a cascaded residual autoencoder for cross-modal imagination to learn joint multi-modal representations for classification. Wang et al. [30] proposed a learnable cross-modal knowledge distillation model for adaptive recognition of significant modalities and knowledge from them to assist other modalities in addressing modality deficiency from a cross-modal perspective.The concept of prompt learning was initially introduced in the field of natural language processing [43] . It adapts to various downstream tasks by modifying prompt instead of adjusting the pretrained language model. Presently, prompt learning have been incorporated into tasks related to computer vision [15, 25] . Coop [44] utilized learnable vectors in a continuous space to represent the prompt of context, while maintained fixed parameters for the entire CLIP pre-trained model. MaPLe [16] employed interactive prompt in both visual and language domains simultaneously to enhance the consistency of representations between vision and language.PromptFuse [20] utilized prompt vectors to align modalities, adapting to downstream multi-modal tasks in a modular and parameterefficient manner. These studies suggested that prompt learning can effectively adapt to various tasks in different input scenarios. This provides us with an idea of integrating prompt learning into multi-modal learning, and prompt learning can be applied to adapt to multi-modal learning in case of missing modalities. The overview of our framework is depicted in Figure 2 , a twostage framework is employed to direct prompts for the compensation of missing modality. To make visual prompts focus on learning modality-specific information, we propose a visual feature decoupling stage LVFD-stage to separate shared and specific information in multi-modal images. Considering the complexity of image distribution which makes it difficult for the prompts to fully learn the modality-specific representations, we utilize language priors to drive the decoupling representation of multi-modal images. The superiority of this strategy to explicitly guide the representation of image content instead of complex distribution can be attributed to the ability of language in capturing abstract concepts and descriptions of relationships in the images. Subsequently, an absenceaware visual prompts compensation stage VPC-stage is proposed to utilize visual prompts to guide the reconstruction of missing modality-specific features to complete the modal information. Specifically, LVFD-stage takes multi-modal remote sensing images 𝐷 as input. Under the constraint of multi-dimensional languagevisual contrastive alignment, each modality image is separately input into shared and respective modality-specific visual encoders to obtain modality-shared and modality-specific features, which are represented as {𝑐 𝑚 𝑖 , 𝑐 𝑚 𝑗 } and {𝑠 𝑚 𝑖 , 𝑠 𝑚 𝑗 }, respectively. To ensure the completeness of complementary feature representation in the absence of certain modalities, particular attention should be given to compensating for specific features 𝑠 𝑚 𝑖 or 𝑠 𝑚 𝑗 . To this end, the VPC-stage takes available modalities 𝑚 𝑖 (or 𝑚 𝑗 ) as input, while integrating visual prompts into the visual encoders specifically to missing modalities, aiming to learn the mapping from modality 𝑚 𝑖 to 𝑚 𝑗 in the modality-specific latent feature space by optimizing the prompts. This enables the derivation of the specific representation 𝑠 ′ 𝑚 𝑗 (or 𝑠 ′ 𝑚 𝑖 ) for modality 𝑚 𝑗 (or 𝑚 𝑖 ) based on 𝑚 𝑖 (or 𝑚 𝑗 ) when 𝑚 𝑗 (or 𝑚 𝑖 ) is missing. In this case, complete complementary features can be obtainable for classification, as follows: 
method 3.1 overview: 
 EQUATIONwhere 𝐸 𝑚 𝑗 𝑠𝑝 denotes the specific encoder of 𝑚 𝑗 modality, P represents the absence-aware visual prompt specific to 𝑚 𝑗 modality and ŷ denotes the predicted category labels. Due to the presence of feature information overlap in multi-modal features, directly recovering the features of missing modalities may lead to a greater emphasis on recovering over-lapping parts, potentially weakening the reconstruction effect of specific information. Therefore, it is necessary to decompose the complementary features of the multi-model data into modality-shared and modality-specific features, which facilitates targeted compensation for the missing specific features in subsequent steps. The traditional decoupling methods often merely separate features without providing guidance for expressing the content of modalities. Meanwhile, language can provide rich semantic information to assist the visual system to better understand and interpret images contents. To this end, language prior knowledge is introduced to guide the decoupling of multi-modal visual features. It is extracted from the pre-trained large language model, which is based on the land cover diversity of the multi-modal images, and can be decomposed into shared and specific language priors. Shared and specific language features are extracted from the priors through a language encoder, and the visual features of different modalities are aligned with their corresponding language features through contrastive learning to achieve feature decoupling. In order to extract effective language priors and guide visual representations, both language feature establishment and multi-dimensional visual-language alignment aspects are comprehensively considered.3.2.1 Language Feature Establishment. It is well known that language, as a comprehensive descriptor of land cover information, can reflect the representation forms of land cover characteristics in different visual modalities. Based on this, we establish language priors for each category of land cover, encompassing both shared and modality-specific information, which is achieved by providing modal attribute-related guidelines to a large language generation model. We consider the fact that different modal images share the same information in the semantic space as a basis for describing modality-shared attributes, utilizing the template "⟨𝑐𝑙𝑎𝑠𝑠; 𝑁 𝑎𝑚𝑒⟩" to expand complete shared language descriptions. For specific aspects, it depends on the information of land cover reflected by each modality. For example, for two data modalities, hyperspectral image (HSI) can better reflect spectral information, so language descriptions for land cover features regarding color and material can be derived from HSI. On the other hand, LiDAR can efficiently and accurately obtain elevation information of the ground compared to HSI. Therefore, height information serves as specific language descriptions for LiDAR. An example is shown in Figure 3 .After acquiring the language descriptions, a text encoder is employed to extract shared and specific language features from the corresponding language descriptions, which is constructed through the pre-trained transformer architecture of CLIP [24] that is widely used in language models. It utilizes lower-cased byte pair encoding (BPE) to obtain tokenized representations of the text, which are then passed through the the fixed-parameter transformer for encoding to extract modality-specific features denoted as F 𝑠𝑚 𝑖 𝑙 , F EQUATIONwhere In order to optimize shared and specific visual encoders for more comprehensive extraction of decoupled modality complementary information, a multi-dimensional visual-language alignment strategy has been proposed, which is implemented by aligning languageshared and language-specific features with modality-shared and modality-specific visual features respectively through contrastive learning between images and language pairs. Unlike traditional contrastive learning between individual language-image pairs, we assign the same language to all images of the same category, whether it's a shared or specific description. Treating all visual features of the same category with the same language as positive samples aims to maximize the similarity between their feature vectors, effectively reducing intra-class variance while widening the inter-class gap. For the shared visual features of modalities 𝑚 𝑖 and 𝑚 𝑗 , we align both of them with the shared language features, the loss takes the following form:F 𝑐𝑚 𝑖EQUATIONWhere L (F 𝑐 𝑙 ,F) and L (F 𝑐 𝑙 ,F) denote the align loss between the shared language and the images of modalities 𝑚 𝑖 and 𝑚 𝑗 , respectively. Taking L (F 𝑐 𝑙 ,F) as an example, and L (F 𝑐 𝑙 ,F) is computed in the same way. The image-to-text and text-to-image alignment losses are computed as:EQUATIONhere, for each embedding feature F 𝑐𝑚 𝑖 𝑣 and F 𝑐 𝑙 in minibatch, 𝑃 𝑣 (𝑛) and 𝐴 𝑣 (𝑛) are the sets of all positive and negative samples of visual features respectively, and 𝑃 𝑣 (𝑛) and 𝐴 𝑣 (𝑛) are their cardinality. Similarly, 𝑃 𝑙 (𝑛) and 𝐴 𝑙 (𝑛) represent the positive and negative sample sets for language features. The language and visual features belonging to the same category are put into 𝑃 𝑙 (𝑛) and 𝑃 𝑣 (𝑛), and the out-of-class features are put into 𝐴 𝑙 (𝑛) and 𝐴 𝑣 (𝑛). 𝜏 is a scalar temperature parameter. L 𝑠𝑝𝑒_𝑘 represents the alignment loss of the 𝑘-th modality-specific features, computed similar to L (F 𝑐 𝑙 ,F) .VPC-stage is designed to recover specific information of missing modalities. To minimize the introduction of additional parameters, we are inspired by the idea of prompt learning and design a visual prompt to learn specific knowledge of the missing modality. This prompt is then used to guide the model in the LVFD-stage stage to extract specific features of the missing modality from available ones. We achieve this by integrating visual prompts into specific encoders of the missing modality, taking available modalities as input to these encoders, and aligning the output features with the specific language features of the missing modality. During training, the only trainable parameters are the absence-aware visual prompts used to learn the missing modality features. We illustrate the compensation of specific features for modality 𝑚 𝑖 with missing modality 𝑚 𝑖 as an example.A dimension matching operation is employed to unify the dimensions of the available modal input X 𝑚 𝑗 ∈ R 𝐻 ×𝑊 ×𝐶 2 with the missing modal X 𝑚 𝑖 ∈ R 𝐻 ×𝑊 ×𝐶 1 , ensuring that it meets the input requirements of the specific encoder for modality 𝑚 𝑖 . The dimension matching is accomplished through a convolutional layer, which can be represented as:EQUATIONwhere X𝑚 𝑗 ∈ R 𝐻 ×𝑊 ×𝐶 1 represents the output after dimension matching. Then the X𝑚 𝑗 is divided into 𝑛 patches {I 𝑞 ∈ R 𝑝 ×𝑝 ×𝐶 1 |1 ≤ 𝑞 ≤ 𝑛}, 𝑝 × 𝑝 denotes the size of patches. Each patch is projected into 𝑑-dimensional latent space, as followsEQUATIONwhere 𝑃𝑟𝑜 𝑗 (•) represents the projection operation. To combine the available modality 𝑚 𝑗 with visual prompts to compensate for the specific features of modality 𝑚 𝑖 we effectively adapt the specific visual encoder of modality 𝑚 𝑖 with modality 𝑚 𝑗 through visual prompts. As mentioned before, the visual encoder is based on the ViT structure, which generally consists of a cascade of 𝑁 encoder layers (here 𝑁 = 4). We denote the patch embedding features of layer 𝑙 asE 𝑙 = {𝑒 𝑞 𝑙 ∈ R 𝑑 |1 ≤ 𝑙 ≤ 𝑁 , 1 ≤ 𝑞 ≤ 𝑛},where E 𝑙 ∈ R 𝑛×𝑑 . Then absence-aware prompts are introduced into the input space of each Transformer layer, which is attached to the embedding feature together with an extra learnable classification token to form an extension feature. For 𝑙-th layer 𝐿 𝑙 , the prompts are denoted as P 𝑖 ∈ R 𝑙 𝑝 ×𝑑 and randomly initialized, where 𝑙 𝑝 is the prompt length. Finally, its output feature is denoted as:EQUATIONwhere 𝑥 𝑐𝑙𝑠 0 ∈ R 𝑑 denotes classification token, (•; •) represents the concatenation operations along the dimension of sequence length. In order to enable the visual prompts to thoroughly learn the missing modality-specific information, we align the output features F 𝑚 𝑖 𝑣 with the specific language features of the missing modality 𝑚 𝑖 , which is formulated as:EQUATIONwhere F 𝑠𝑚 𝑖 𝑙 represents the specific language features, F 𝑚 𝑖 𝑣 is the compensated visual features.Stage1: Modality Feature Decoupling During the modality feature decoupling, a joint optimization objective is defined to extract complementary decomposed features from multi-modal data, which contains a combination of multiple contrastive loss and joint classification loss:EQUATION)where the hyperparameter 𝜆 1 , 𝜆 2 control the balance of multiple losses. The multiple contrastive loss is composed of both shared feature alignment loss and all modality-specific feature alignment losses:EQUATIONhere L 𝑠ℎ𝑎𝑟𝑒𝑑 and L 𝑠𝑝𝑒_𝑘 denotes the alignment loss for shared features and specific features of modality 𝑚 𝑘 , respectively. The classification loss L 𝑐𝑙𝑠 further optimizes the visual encoder, enhancing the extraction of more discriminative features. The computation formula is as follows:EQUATION)where 𝑁 is the number of classes.Stage2: Compensation of Specific Features This stage constructs cross-modal visual-language alignment loss L 𝑐𝑟𝑜𝑠𝑠 and classification loss L 𝑐𝑙𝑠 for feature compensation and classification,L 𝑠𝑡𝑎𝑔𝑒2 = 𝜆 3 L 𝑐𝑟𝑜𝑠𝑠 + 𝜆 4 L 𝑐𝑙𝑠 (12)here, L 𝑐𝑟𝑜𝑠𝑠 is the loss described in Section 3.3, 𝜆 3 and 𝜆 4 are hyperparameters. 
experiments 4.1 datasets description: 
 We conduct experiments on three publicly multi-modal datasets for performance evaluation. A brief description of these three datasets is as follows: 1) Houston2013 [4] This dataset is part of the 2013 IEEE GRSS Data Fusion Competition and contains hyperspectral (HS) and Li-DAR images depicting 15029 labeled samples of 15 categories.2) Trento [26] The dataset was taken in a rural area south of Trento and consists of HS and LiDAR data, with a total of 30214 labeled samples in 6 land cover categories. 3) Augsburg [1] This dataset originates from Augsburg, Germany, comprising three modalities of data: HS, SAR, and LiDAR. It encompasses 7 categories with a total of 78294 labeled samples.4.2.1 Evaluation Metrics and Implementation Details. Three evaluation metrics [32] are employed for quantitative analysis, including overall accuracy (OA), average accuracy (AA), and kappa coefficient.The proposed method is implemented on the PyTorch platform and trained on two NVIDIA GeForce 3090 GPUs using the Adam optimizer. The model is trained for 500 epochs in the LVFD-stage, followed by 300 epochs in the VPC-stage. The batch size is set as 1024. The learning rate is initially set to 1e-3, and updated by a CosineAnnealingLR strategy. All the comparison methods selected 40 samples for training.To demonstrate the effectiveness of the proposed method in the joint classification of multi-modal remote sensing images in case of modality absence, we set up three different experimental configurations: 1) Training and testing occur within single modality in the proposed method, with each model named after the modality it utilizes. For instance, "HS-net" refers to a model trained with HS images, while "LiDAR-net" indicates a model trained with LiDAR images. 2) State-of-the-art methods for joint classification in case of modality absence, including Cospace [13] , MDL-RS [11] and MSH-Net [35] . 3) State-of-the-art methods for joint classification with complete modalities: MFT [27] , Sal2RN [17] , HCT [40] and GLT-Net [5] . 1 show the performance comparison of OA, AA, and kappa on Houston2013 and Trento datasets under three types of comparison methods, respectively. Figures 4 and 5 show the classification maps of the comparison algorithms considered in the Houston and trento datasets, respectively. First, for scenarios without modality absence, it is evident that the model HS-LiDAR-Net trained with multi-modality outperform the HS-Net and LiDAR-Net trained with only uni-modality, which clearly demonstrates the complementary advantages of multi-modality. In the absence of HS or LiDAR modalities, the OA of HS-LiDAR-Net drops by 10.37% and 1.54% on the Trento dataset. On the Houston2013 dataset, the decrease is more pronounced, with declines of 16.81% and 1.97%. This indicates the ineffectiveness of applying traditional multi-modal models to the case of modal incompleteness. In contrast, the proposed LVPCnet significantly addresses this issue and yields superior performance to uni-modal models. Specifically, the OA of LVPCnet when LiDAR is missing outperforms the model trained with single modality by 0.76% and 1.04% on the Trento and Houston2013 datasets, respectively. Moreover, OA improved by 4.58% and 2.84% with the absence of HS images. Additionally, The LVPCnet outperforms the Cospace, MDL-RS and MSH-Net by 21.12%, 29.9% and 3.27% in terms of OA respectively when HS is missing and by 13.82%, 7.6% and 0.48% respectively when LiDAR is missing on the Trento dataset. Similarly, it also performs better on the Houston dataset. This indicates that the proposed method effectively utilizes language priors to extract and compensate for more discriminative specific features of the missing modality compared to other methods.We conduct experiments for HS and SAR modalities on the Augsburg dataset to further evaluate the generalization performance of LVPCnet. As shown in the right part of Table1, the LVPCnet demonstrates superior performance compared to joint classification with complete modalities. Moreover, it even outperforms partially multi-modal fusion classification models when certain modalities are missing. The proposed method with absence of SAR images achieves improvements of 4.65%, 2.33% and 0.26% compared to the multi-modal joint classification method MFT, HCT and GLT-Net. This indicates that the proposed method not only deal with missing modality but also demonstrates significant potential in joint classification. Figure 6 shows the classification maps.We conduct experiments on modal combinations of HS, LiDAR, and SAR images to evaluate the scalability of our method in the presence of multi-modal absence. As shown in Table 2 , we can observe that the proposed LVPCnet performs better than the conventional multimodal model under the setting of missing modalities, and also exceeds the uni-modal model in Table 1 during single-modal testing, which suggests that LVPCnet has superior robustness even in the case of missing multiple modalities. details through feature decoupling. To validate its effectiveness, a variant named as 'vf', is designed to directly extract features from multi-modal data without feature decoupling. As shown in the results from Figure 7 , LVPCnet achieves a more satisfactory performance than vf, which proves the effectiveness of feature decoupling.Prompts. This paper enhances the learning of specific features from missing modalities by introducing visual prompts, without the need to modify the model or introduce additional networks. To validate the effectiveness and efficiency of visual prompts, we design two variants (named 'vp1' and 'vp2') to compare the performance of the proposed method. vp1 remove visual prompts and vp2 replaces visual prompts with reconstruction network. The comparison results are shown in Table 3 , LVPCnet outperforms vp1 by a wide margin. As for vp2, there is a slight improvement in OA, but the accuracy improvement remained within 0.5%. The reason for this phenomenon is that the reconstruction network needs to retrain a network for each missing modality, introducing abundant training parameters as shown in Table 3 .Comparatively, the proposed method only trains the prompts without retraining the original network, and the parameter count of the prompts was only 0.6% of the reconstruction network. Therefore, visual prompts are trained for each missing modality significantly reduces computational complexity compared to a reconstruction network. 
conclusion: 
 In this paper, LVPCnet is proposed to address the issue of modality missing in joint classification of multi-modal remote sensing images by compensating for specific features of the missing modality. The network is designed with a two-stage process for extracting specific complementary information from each modality and learning crossmodal specific information. This facilitates the recovery of specific features of the missing modality from known modalities when dealing with modality absence. Specifically, LVPCnet utilizes language priors to drive visual decomposition to explore complementary representations of multi-modal data, reducing redundancy. Subsequently, by embedding visual prompts, the model is guided to learn specific features of the missing modality from the known modalities, enabling the acquisition of complete multi-modal complementary information for joint classification. Systematic experimental investigations have been conducted on three public datasets to validate the effectiveness of our method.  
 
 Appendix:  


Decision: The paper got four acceptances after rebuttal. An acceptance is OK for this paper. I suggest the authors improve the writing and have more discussion according to the reviews and rebuttals.

Reviews: review: paper_summary: paper_summary: The paper titled "Language-Guided Visual Prompt Compensation for Multi-Modal Remote Sensing Image Classification with Modality Absence" presents a novel approach to address the challenge of modality absence in the joint classification of multi-modal remote sensing images. The authors propose a Language-Guided Visual Prompt Compensation Network (LVPCnet) that leverages language priors to guide the model in capturing complete modal information from available modalities for classification, even when some modalities are missing. The method involves a two-stage training process: a language-driven visual feature decoupling stage and an absence-aware visual prompt compensation stage. The effectiveness of the proposed approach is validated through systematic experiments on three public datasets.
strength_weakness: strength_weakness: Novelty: The paper introduces a unique solution to the problem of modality absence in multi-modal remote sensing image classification by utilizing language-guided visual prompts. This approach is innovative and differs from existing methods that either generate missing modalities or transfer knowledge from complete to incomplete modal sets.

Theoretical Approach: The theoretical foundation of using language priors to guide the feature extraction process is sound and aligns with current trends in multi-modal learning.

Technical Correctness: The proposed LVPCnet is technically well-defined, with a clear explanation of the language-driven visual feature decoupling and absence-aware visual prompt compensation stages.

Adequate Evaluation: The paper provides a comprehensive evaluation of the proposed method on three public datasets, comparing it with state-of-the-art methods, which demonstrates the effectiveness of LVPCnet.

Clarity: The paper is well-structured and clearly presents the methodology, experimental setup, and results. The use of figures and tables enhances the clarity of the presentation.

Applications: The proposed method has potential applications in various remote sensing tasks such as land analysis, urban planning, and environmental monitoring, where modality absence is a common issue.
limitations: limitations: Lack of Comparative Analysis: While the paper compares the proposed method with several existing approaches, it lacks a deeper comparative analysis that might reveal the trade-offs involved in using language-guided prompts versus other techniques.

Generalization: The paper does not provide extensive experiments to show how well the proposed method generalizes across different types of modality absence scenarios or different datasets.

Computational Efficiency: The paper does not discuss the computational efficiency of the proposed method, which is crucial for practical applications, especially in the context of remote sensing where data volumes are typically large.

Limited Discussion on Prompt Learning: The paper introduces the concept of language-guided visual prompts but does not delve into the nuances of prompt engineering, such as the selection, design, and optimization of these prompts for different missing modality scenarios.
paper_summary: paper_summary: The article proposes a language-guided visual prompt compensation network (LVPCnet) that uses a unified model while taking into account modality complementarity for joint classification in the presence of arbitrary modality absence. It embeds missing modality-specific knowledge into visual cues to guide the model in capturing complete modality information for classification from the available modality information.
strength_weakness: strength_weakness: 1. a unified model LVPCnet for joint classification of arbitrary modal missing is proposed, which can cope with a wide range of modal missing situations and is highly practical.
2. the article explains the specific modules clearly, and explains the role of the modules superficially through diagrams and examples.
3. the article is very detailed in classifying and analyzing the existing methods, and summarizes the advantages and disadvantages of each class of methods
limitations: limitations: 1. There is a lack of explanation of X^(m_i ) in Eq. (1), and a specific explanation of the meaning should be given at the first appearance of the symbol.
2. In Table 2, in the absence of modes, the LVPCnet method does not perform better than the traditional multimodal model as claimed in the text, e.g., in the comparisons in rows 2, 3, and 6, the AA metrics of BASELINE are better than those of LVPCNet.
3. The specific category names are not uniform, Figures 2 and 3 show Healthy grass, but in Figure 4 it is Health grass.
paper_summary: paper_summary: This paper proposed LVPCnet to address the issue of modality missing in the joint classification of multi-modal remote sensing images by compensating for specific features of the missing modality. The network is designed with a two-stage process for extracting specific complementary information from each modality and learning crossmodal specific information. Specifically, a language-guided visual feature decoupling stage (LVFD-stage) is designed to extract shared modality-specific features from multimodal images to build a complementary representation model of the complete modality. Subsequently, a missing perceptual visual cue compensation stage (VPC-stage) is proposed to learn visual cues containing missing modality-specific knowledge through cross-modal representation alignment, which further guides the complementary representation model to reconstruct modality-specific features of the missing modality from the available modalities based on the learned cues.
strength_weakness: strength_weakness: 1, The method of using language-driven prompts proposed in this paper is innovative and inspiring and has achieved good results experimentally.

2, The paper is well organized and clear.

3, The two key stages are technically sound.
limitations: limitations: 1, This paper needs to discuss the differences and connections between their approach and existing methods in the related work section, in addition, there is a lack of related work on multimodal remote sensing image classification in the task context.

2, Lack of ablation experiments for hyperparameters.
paper_summary: paper_summary: This paper proposes a language-guided visual prompt compensation network to achieve joint classification in case of arbitrary modality absence. The network consists of a language-guided visual feature decoupling stage (LVFD stage) and an absence-aware visual prompt compensation stage (VPC stage). LVFD-stage extract shared and specific modal features. VPC-stage learn visual prompts containing missing modality-specific knowledge through cross-modal representation alignment.
strength_weakness: strength_weakness: 1. The article presents an innovative approach. It introduces the concept of utilizing language guidance to decompose modality information into shared and specific features, supplementing the specific information of the absent modality using visual prompts, which is a novel idea.
2. The article is well-structured, providing a comprehensive and explicit description of the research motivation and methodology design.
3. The proposed method exhibits strong performance. The unified network demonstrates the capability to handle the absence of any modality data relatively well compared to other methods.
limitations: limitations: 1.	The paper lacks ablation experiments for each hyperparameter.