# Convolutional Sparse Modular Fusion Algorithm for Non-Rigid Registration of Visible-Infrared Images
## 2 Mathematical Model  
Multimodal images, captured from different modalities, frequently exhibit comparable features within the same field of view. These shared attributes and differences can provide valuable prior knowledge for image registration and fusion. Let \(I^{x}\) and \(I^{y}\) denote two unaligned images from different modalities. Considering prior knowledge [14] potential interference in the images, each image feature is decomposed into three components: homogeneous features, isolated features, and noise. In this context, homogeneous features refer to characteristics shared among multiple modalities, such as large-scale scenes, objects, and sparse segmentation edges. These features inherently bridge inter-modality correlations,  
Figure 1: Examples of visible image and infrared image fusion, as well as typical fusion artifacts. thereby enabling smooth image registration. Conversely, isolated features represent modality-specific attributes, such as color information in visible images or thermal radiation information in infrared images. Integrating these features into fused images is a central challenge in image fusion research. Additionally, noise suppression enhances algorithmic robustness, ultimately yielding high-quality fused images. Assuming that the number of homogeneous features in \(I^{x}\) and \(I^{y}\) is the same, the model is formulated as follows:  
\[I^{x} = \sum_{i=1}^{i_{x}}w_{i}^{x}*f_{i}^{x}+\sum_{h=1}^{H}w_{h}^{x}*f_{ h}^{x}+s^{x},\] \[I^{y} = R_{(H,S)}\left(\sum_{i=1}^{i_{y}}w_{i}^{y}*f_{i}^{y}+\sum_{h=1}^{ H}w_{h}^{y}*f_{h}^{y}+s^{y}\right), \tag{1}\]  
where \(w_{i}^{n}\) and \(w_{h}^{n}\) are convolutional dictionary filters representing the homogeneous and isolated features of \(I^{n}\), and \(f_{i}^{n}\) and \(f_{h}^{n}\) are their corresponding sparse convolutional representations. Moreover, \(I^{n}\) signifies any image, and \(*\) denotes the convolution operation. \(R_{(H,S)}\) indicates the spatial transformation and resampling operation used for image registration. ICSF uses this framework to extract homogeneous and isolated features from \(I^{x}\) and \(I^{y}\), thanks to the modality generalizability of convolutional sparse coding. Thus, the optimization problem for multimodal image convolutional sparse feature decomposition is formulated as follows:  
\[\underset{f_{h}^{x},f_{h}^{y},f_{i}^{x},f_{i}^{y},f_{i}^{x},H,S}{ argmin}\frac{1}{2}\|I^{x}-\sum_{i=1}^{i_{x}}w_{i}^{x}*f_{i}^{x}-\sum_{h=1}^{H}w_{h}^{x}*f_ {h}^{x}-s^{x}\|_{2}^{2}\] \[+\frac{1}{2}\|I^{y}-R_{(H,S)}\left(\sum_{i=1}^{i_{y}}w_{i}^{y}*f_{ i}^{y}+\sum_{h=1}^{H}w_{h}^{y}*f_{h}^{y}+s^{y}\right)\|_{2}^{2}\] \[+\lambda_{h}^{x}\psi(f_{h}^{x})+\lambda_{h}^{y}\psi(f_{h}^{y})+ \lambda_{i}^{x}\psi(f_{i}^{x})+\lambda_{i}^{y}\psi(f_{i}^{y}), \tag{2}\]  
where \(\psi(\cdot)\) is the sparsity constraint for the convolutional sparse representations, and \(\lambda_{h}^{x}\), \(\lambda_{h}^{y}\), \(\lambda_{i}^{x}\), \(\lambda_{i}^{y}\) are regularization parameters. \(H\) and \(S\) signify the registration parameters used for image registration, which are fixed in this optimization problem. In addition, the four variables \(f_{h}^{x}\), \(f_{h}^{y}\), \(f_{i}^{x}\), and \(f_{i}^{y}\) must be optimized iteratively until convergence. We use a deep learning module to perform image feature decomposition to avoid the computational waste caused by the exhaustive search for mathematical solutions in traditional iterative algorithms. A two-step method is employed to decompose the source image into convolutional sparse features to simplify the feature extraction process.