# IFF-Net: Irregular Feature Fusion Network for Multimodal Remote Sensing Image Classification
## 1 Introduction
Recurrent neural networks (RNNs) can model sequences and efficiently extract spectral features in remote bands [31]. [PERSON] et al. [32] developed a hybrid framework that integrates a CNN and RNN to design a convolutional RNN model. This approach involves extracting spatial spectral features through a convolutional layer, followed by the extraction of contextual spectral features using a recurrent layer. However, the RNN has difficulty in sensing the spectrally significant changes between pixels, the RNN cannot train multi-sample networks at the same time, and the classification performance is constrained. Researchers have increasingly shown interest in transfer learning due to its capability to capture extended sequential relationships. Initially employed in natural language processing, it is now progressively utilized for RS classification. [PERSON] et al. [33] proposed a novel approach for spectral feature extraction by incorporating cross-layer transfer encoder (TE) modules, which effectively capture neighboring band information. However, the focus of their work lies solely on utilizing the transfer technique for processing spectral data. Although the spectral features can be successfully captured, it cannot utilize the spatial information. In their study, [PERSON] et al. [34] introduced a multimodal fusion transformer that combines HSI and LiDAR for joint classification. The approach involves utilizing LiDAR data as trainable tokens to facilitate feature learning alongside HSI tokens. However, this fusion process fails to effectively integrate the valuable information from both datasets, thereby limiting the accuracy of the classification results. Despite the advantages of the aforementioned approaches in terms of their effectiveness and performance in classification, they face challenges when it comes to integrating multimodal data. Specifically, the issue lies in handling unbalanced information and how this imbalance negatively impacts feature interaction. To address the aforementioned issues, this study introduces a novel approach called IFF-Net, which utilizes irregular feature fusion techniques to enhance the classification of multimodal remote sensing images. Specifically, the extraction of features is accomplished through the utilization of residual blocks that possess multiple weights in common and the learnable parameters of the BN layer are independently learned to enable irregular feature propagation. Additionally, a threshold is employed to evaluate the BN judgment factor. Moreover, if the current feature is considered redundant information, the corresponding channel feature will be substituted. In this way, the complementary propagation of multimodal feature information is realized, thus completing the fusion of irregular features. Furthermore, a module for enhancing the differentiation of spatial data is incorporated to standardize and calibrate features. The efficacy of this approach is showcased through the experimental findings presented in this study. Specifically, the key contributions of this paper can be succinctly summarized as follows:  
1. In this study, we propose a novel approach for classifying multimodal remote sensing data using irregular feature extraction. To enhance the model's generalization capability, we employ weight-sharing residual blocks in a cascaded manner for feature extraction and adjust the number of parallel branches to facilitate collaborative processing without introducing extra parameters. 2. By privatizing the judgment factors within the BN layer, we effectively substitute redundant information in the channel. This enables us to fully leverage the complementary advantages of multimodal remote sensing data, while also imposing sparse constraints on specific judgment factors to mitigate any undesired feature propagation. 3. A module for normalizing and calibrating features was developed to enhance information integration, optimize spatial information discrimination, and minimize redundancy.