# IFF-Net: Irregular Feature Fusion Network for Multimodal Remote Sensing Image Classification
## 2 Materials and Methods
### Feature Extraction via Residual Blocks  
The integration of spatial and spectral information from both HSI and SAR data is advantageous for classification purposes. To represent the center pixel, we utilize spatial neighborhood information of size \(r\) that has been pixelated. The HSI block is denoted as \(\mathbf{X_{h}}\in\mathbf{R}^{r\times r\times c_{k}}\), while the SAR image block is denoted as \(\mathbf{X_{s}}\in\mathbf{R}^{r\times r\times c_{k}}\), \(c_{k}\) and \(c_{s}\) refer to the corresponding channel quantities. The sub-network of the model comprises a CNN layer and multiple residual blocks, followed by a BN layer and an activation layer. To ensure compatibility with multimodal data, the initial CNN layer performs dimension matching without sharing weights. Common patterns are then extracted from the multimodal remote sensing data by weight-sharing residual blocks, which is central to the reorganization and integration of irregular features. The fusion of features is accomplished through the utilization of supplementary modules that are linked to the convolutional layer. Figure 2 illustrates the residual block for irregular feature propagation, where the optimization of BN parameters is conducted independently. Therefore, the determination of irregular feature propagation is based on comparing the judgment factor values of the private BN layer with a threshold value. By employing weight sharing, the two modalities' features are endowed with a shared set of weights during feature extraction, effectively reducing the computational complexity from O(2\(r\)) to O(r). Simultaneously, the comparison between the judgment factor \(\gamma_{\textit{basic}}\) and threshold \(\theta\) in our model determines feature propagation across channels, leading to a space complexity reduction from O(2\(c\)) to O(\(c\)). This approach ensures that the model maintains its specificity. Thus, this approach reduces model complexity while preserving modal specificity. Unlike independent modal extraction, weight sharing of private BN layers can reduce the number of training parameters without compromising  
Figure 1: Showing the IFF-Net network architecture proposed in this paper.  
fusion when \(\gamma_{hisc}\to 0\), we effectively eliminate data redundancies. This calculation for performing feature fusion can be represented as follows:  
\[\delta_{h,c}=\left\{\begin{array}{l}\gamma_{hisc,h,c}\frac{x_{h,c}-h_{h,c}}{ \sqrt{\sigma_{h,c}+\tau}}+\alpha_{h,c},if\gamma_{hisc,h,c}>\theta\\ \gamma_{hisc,s,c}\frac{x_{c}-h_{h,c}}{\sqrt{\sigma_{h,c}+\tau}}+\alpha_{s,c}, otherwise\end{array}\right. \tag{10}\]  
In the HSI branch, \(h,c\) represents the BN parameter of the \(c^{\text{th}}\) channel, while in the SAR branch, \(s,c\) represents the BN parameter of the \(c^{\text{th}}\) channel. The feature \(x_{h,c}\) corresponds to the \(c^{\text{th}}\) channel in \(\mathbf{X_{h}}\), and \(x_{s,c}\) corresponds to the \(c^{\text{th}}\) channel in \(\mathbf{X_{s}}\). Here, a threshold value \(\theta=2\times 10^{-3}\) is used close to 0. If \(\gamma_{hisc,h,c}<\theta\), it indicates that the current feature of the \(c^{\text{th}}\) channel is redundant and should be replaced by another channel. This means that when a particular channel no longer contributes to subsequent predictions, it is substituted with feature propagation from a corresponding modality's channel. Applying regularization to the judgment factor \(\gamma_{hisc}\) can effectively extract the complementary information between modes and reduce the propagation of invalid irregular features.