# IFF-Net: Irregular Feature Fusion Network for Multimodal Remote Sensing Image Classification
## 2 Materials and Methods
### Feature Normalization and Calibration  
Reaching complementary benefits is made possible through complete information exchange among multimodal remote sensing data after the propagation of irregular features. Prior to classification, it is necessary to standardize and calibrate the multimodal remote sensing data features. To enhance the capabilities of feature recognition, we propose incorporating a module for feature normalization and calibration that establishes associations between global and local features and identifiable characteristics; Enhancing feature normalization and calibration via an enhanced self-attention mechanism. Refer to Figure 3 for an illustration of the feature normalization and calibration module. The improved self-attentive feature can be represented as \(SE(Q,K,V)\), which conducts query, key, and value operations on its \(Q,K,V\in R^{N\times C}\). In this case, \(N=H\times W\) represents the dimensions of a given area with height \(H\) multiplied by width \(W\), while C signifies the quantity of channels. Initially, we calculate the similarity between \(Q\) and \(K\), before determining a weighted average vector across \(V\) based on these similarity scores. Consequently, we can express this overall improved self-attention equation in a generalized form. \[c_{i  
Obviously, \(Q\), \(K\), and \(V\) are commonly linear transformations of the input features \(SE_{q}\). Hence, the improved self-attention mechanism solely focuses on the elemental relations within a single sample while disregarding the relationships among different samples. In this study, we propose an enhanced feature normalization and calibration module that incorporates a calibration module integrated with an improved self-attention module. The improved self-attention module employs a dataset consisting of two sets of self-learning parameters instead of (K, V) key-value pairs. These two datasets are used to store the relationships between data elements throughout the training process. For example, \(SE_{q}\) denotes an output value after the propagation of an irregular feature Q in the improved self-attention mechanism, which is then fed into the feature normalization and calibration module. The mathematical formula for this is  
\[c^{\prime}_{i,j}=\text{softmax}\bigg{(}norm(m_{i}g^{j}_{k})\bigg{)} \tag{13}\]  
\[\widetilde{v}^{\prime}_{i,j}=\sum_{j}c^{\prime}_{i,j}g^{j}_{v} \tag{14}\]  
In this model, we utilize two datasets, \(M_{k}\in R^{M\times C}\) and \(M_{v}\in R^{M\times C}\), which are linear learning parameters. These datasets contain rows \(M_{k}\) and \(M_{v}\), respectively, where \(c^{\prime}_{i,j}\) represents the similarity score between \(m_{i}\) and \(g^{j}_{k}\). The size of the convolutional kernel is denoted by \(M\). By sharing parameters across the training set, these storage units improve regularization effects while reducing learned parameters and linear complexity when M < N. Both the self-attention and improved self-attention module outputs are weighted averages for each query. To address this issue, a normalization and calibration module is added on top of the global attention module, as shown in Figure 4, to ensure accurate results. First, the linear layer averages the output characteristics of the improved self-attention module into two parts \([V_{1},V_{2}]\), and then, defines four convolution kernels \([K_{1},K_{2},K_{3},K_{4}]\), which have the shape \([C/2,C/2,k,k]\). Specifically, \([K_{1},K_{2},K_{3}]\) are used for feature calibration, which is calculated as  
\[V^{\prime}_{1}=\sigma(UP[(AvgPool(V_{1})*K_{1}]) \tag{15}\]  
\[Y_{1}=((V_{1}*K_{2})\cdot V^{\prime}_{1})*K_{3} \tag{16}\]  
where \(AvgPool(\cdot)\) is an average pooling operation with a convolution kernel of \(2\times 2\) and a step size of 2. \(UP(\cdot)\) refers to the utilization of bilinear interpolation for increasing the resolution to match the initial assessment. In this context, \(*\) denotes the convolution operation while \(\sigma\) represents the sigmoid function.