A Novel State Space Model with Local Enhancement and State Sharing for Image Fusion

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-23 14:54:07

Abstract: In image fusion tasks, images from different sources possess distinct characteristics. This has driven the development of numerous methods to explore better ways of fusing them while preserving their respective characteristics.
Mamba, as a state space model, has emerged in the field of natural language processing. Recently, many studies have attempted to extend Mamba to vision tasks. However, due to the nature of images different from casual language sequences, the limited state capacity of Mamba weakens its ability to model image information. Additionally, the sequence modeling ability of Mamba is only capable of spatial information and cannot effectively capture the rich spectral information in images. Motivated by these challenges, we customize and improve the vision Mamba network designed for the image fusion task. Specifically, we propose the local-enhanced vision Mamba block, dubbed as LEVM. The LEVM block can improve local information perception of the network and simultaneously learn local and global spatial information. Furthermore, we propose the state sharing technique to enhance spatial details and integrate spatial and spectral information. Finally, the overall network is a multi-scale structure based on vision Mamba, called LE-Mamba. Extensive experiments show the proposed methods achieve state-of-the-art results on multispectral pansharpening and multispectral and hyperspectral image fusion datasets, and demonstrate the effectiveness of the proposed approach. Code will be made available.

Main paper: 
 introduction: 
 In the field of image fusion, there are two flourishing tasks that are applied in subsequent applications, i.e., multispectral pansharpening, and multispectral and hyperspectral image fusion. While multispectral pansharpening aims to fuse high-resolution panchromatic images and low-resolution multispectral images to yield highresolution multispectral images (HRMS), multispectral and hyperspectral image fusion involves high-resolution RGB images with low-resolution hyperspectral images to yield high-resolution hyperspectral images.Many deep-learning methods have been widely proposed for image fusion, such as convolution neural networks (CNNs) and vision Transformers. The CNN-based methods can extract and learn local feature information in image fusion. More importantly, vision Transformers have achieved excellent performance in image fusion [5, 38] . However, vision Transformers are limited by the quadratic spatial complexity of the Attention mechanism, making it challenging to process high-resolution images. Many works [14, 25, 40] aim to reduce the complexity of Transformers, but they often come with performance degradation, increased parameter count, and poor generalization capability.Recently, an alternative approach has shown promising results benefiting from efficiently modeling dependencies of sequence data based on the state space model (SSM). In recent work, [13, 28] introduces the structured state space sequence model as a general sequence model, called Mamba. Afterward, vision Mamba [24, 50] further verifies that it has great potential to be the next-generation backbone for vision foundation models. Compared with vision Transformers [10] , vision Mambas adopt sequential visual representation (i.e., image patches) and have subquadratic-time computation and near-linear memory complexity. However, Mamba faces two challenges: 1) needs for non-unidirectional modeling for 2D images in contrast to language sequence. 2) the issue of information stored in the state being lost when the sequence length increases, and 3) unable to characterize the spatial and spectral information of images.In addressing the first question, two pioneering works stand out: Vim [50] , which introduced a bi-directional vision Mamba block tailored for image processing, and VMamba [24] , which proposed a scanning mechanism for images to tackle the causality issue inherent in Mamba's unsuitability for vision tasks. As for the rest of the issues, there has been a lack of exploration of Mamba in the image fusion field. In this paper, we propose two bespoke techniques named local-enhanced vision Mamba (LEVM) block and state sharing to address issues 2) and 3), as its conception shown in Fig. 2 , respectively enhancing Mamba's perception of local information and fully exploiting spatial and spectral information in the SSM's state.The proposed method is more suitable for image fusion tasks with two advantages to the proposed methods. The first advantage is that the proposed LEVM block can represent local and global spatial information. Local information brings rich spatial details, while global information injects long-range pixel dependencies, thus enhancing the detail recovery in the fusion process. The second advantage is that the state sharing method allows information to be shared between layers in the adjacent flow and skip-connected flow. Furthermore, we have also proposed the spatial-spectral learning (S2L) of SSM for the state sharing method. The S2L maps the input image to the state space for spatial representation, and considers the state space as the basis for spectral representation, thereby achieving interaction between spatial and spectral information. To the best of our knowledge, there have been no improvements made to vision Mamba from the perspective of spatial and spectral learning specifically for image fusion tasks. Finally, we propose the LE-Mamba, which is constructed based on the U-Net [33] baseline with the inclusion of the LEVM block and the state sharing technique. The proposed LE-Mamba can achieve superior fusion performance.The contributions can be summarized as follows:1) We propose a local-enhanced vision mamba (LEVM) block for the existing vision Mamba architecture to address the issue of spatial information representation.2) The state sharing technique is designed for the proposed LEVM block, which customizes the image fusion tasks. Then, the state sharing can reduce information loss and enable simultaneous learning of spatial and spectral information within the state space model (SSM).3) The proposed LE-Mamba achieves state-of-the-art fusion performance for image fusion on four widely-used multispectral pansharpening and hyperspectral multispectral fusion datasets.The rest of the paper will be organized as follows: In Sect. 2, we review the related work on SSMs, then introduce deep learningbased methods on multispectral pansharpening, as well as multispectral and hyperspectral image fusion tasks. In Sect. 3, we revisit the background knowledge of SSM. Then, in Sect. 4, we elaborate the proposed method with the overall architecture, the localenhanced vision Mamba block, and the state sharing technique in Sect. 4.1, Sect. 4.2 and Sect. 4.3, respectively. Afterward, Sect. 4.4 analyzes the computational complexity of various common neural network operators. At last, we conduct extensive experiments and ablation studies on the proposed architecture to validate its effectiveness in Sect. 5. 
related work 2.1 state space models: 
 State space models (SSMs) were initially proposed for sequence-tosequence transformation tasks in natural language processing [12, 13, 28] , adept at handling long-range dependencies. In recent years, researchers have strived to address the computational and memory bottlenecks of SSMs in practice, enabling higher efficiency and simplicity when processing long sequences, such as structured state space (S4) [13] introduces parameterization for state space, while Mamba [12] incorporates high-efficient selection mechanism into the S4 based on hardware optimization.In vision tasks, S4ND method [32] was the first work to introduce SSMs into the vision field, demonstrating their potential to achieve performance on par with models like vision Transformer [10] . VMamba [24] and Vim [50] further applied the ideas of Mamba models to generic vision tasks, proposing mechanisms like bi-directional scanning to better capture image information. Due to the recent explosive growth in research on vision Mamba, we systematically summarize the most recent relevant works in the supplementary for a detailed background.For the multispectral pansharpening task, many CNN-based methods have emerged and obtained promising fused images, including DiCNN [15] , PanNet [45] , and FusionNet [4] . However, the local feature representation obtained by CNNs hinders better fusion results. For the multispectral and hyperspectral image fusion, there are also many outstanding DL-based works including InvFormer [49] and MiMO-SST [11] . However, these methods bear a significant computational burden due to the quadratic complexity of the Transformer. Despite many efforts to mitigate this issue [14, 25, 40] , these methods usually lead to a performance drop.Recently, SSMs have emerged with near-linear memory consumption and relatively low computational overhead, thus rapidly extending to various vision tasks. Most relative to our work, Pan-Mamba [16] adapted the bidirectional vision Mamba block [50] to the multispectral pansharpening task. However, it fails to realize the issue of state information loss and does not tailor state space representation to exploit the spatial and spectral domain, leading to suboptimal fusion performance. 
preliminary: 
 State space models (SSMs) are a class of sequence models inspired by linear systems, which aim to map a sequence 𝑥 (𝑡) ∈ R 𝐿 to 𝑦 (𝑡) ∈ R 𝐿 through the hidden space ℎ ′ (𝑡), ℎ(𝑡) ∈ R 𝑁 . A system matrics 𝑨 ∈ R 𝑁 ×𝑁 , 𝑩 ∈ R 𝑁 ×1 , and 𝑪 ∈ R 𝑁 ×1 represents the dynamics of the system:ℎ ′ (𝑡) = 𝑨ℎ(𝑡) + 𝑩𝑥 (𝑡), 𝑦 (𝑡) = 𝑪ℎ(𝑡).(EQUATIONIn practice, the above continuous system should be discretized using zero-order hold assumption, converting the matrics (𝑨, 𝑩) to the discretized forms by a timescale 𝚫 ∈ R > 0:EQUATIONThe discretized operator Δ maps 𝐴 and 𝐵 from Δ : R 𝑁 → R 𝐿×𝑁 (decretize continous system), where dimension 𝑁 is a higher dimensional latent state. This discretization forms a discretized model written as:EQUATION)Furthermore, in higher-dimensional state space, we can rewrite the above variables, that is input data 𝑥 : R 𝐿×𝐷 → R 𝐿×𝐷 ×𝑁 with 𝐷 channel. A timestep 𝑡 depends on the size of the sequence 𝐿 (i.e., 𝑥 𝑡 ∈ R 𝐷 ×𝑁 ). Then, at each timestep 𝑡, system matrices 𝑨 ∈ R 𝐷 ×𝑁 , 𝑩 ∈ R 𝐷 ×𝑁 are mapped into 𝑁 -dimension state space, and 𝑪 ∈ R 𝐷 ×𝑁 map ℎ 𝑡 back the original space. Thus, a hidden state is ℎ 𝑡 ∈ R 𝐷 ×𝑁 and the output feature is 𝑦 𝑡 ∈ R 𝐷 (i.e., 𝑦 ∈ R 𝐷 ×𝐿 ). According to the implementation of Mamba [12] , for efficient computation, the previous computation process can be formulated as the parallel convolution:EQUATIONwhere ⃝ * denotes convolution operation, and 𝐾 is the structured convolutional kernel. 
method: 
 In this section, we will first illustrate the overall network design of LE-Mamba in Sect. 4.1, including the network's input and output, as well as its structure. Subsequently, each local-enhanced vision Mamba (LEVM) block within the network will be delineated in detail in Sect. 4.2. Finally, the implementation details of the proposed state sharing technique are elucidated in Sect. 4.3. In Sect. 4.4, we analyze the complexity of the LEVM block.In some high-level tasks (e.g., classification [24, 50] and segmentation [22, 24, 29, 35] ), architectures similar to Meta-formers [46] are commonly used, where plain backbones can offer lower parameter counts, smaller computational loads, and lower latency. However, for the image fusion task, we opted for a multi-scale architecture akin to U-Net [34] , which has been proven effective [2] . The designed network is illustrated in Fig. 2(d) , featuring an encoderdecoder architecture. The input comprises LRMS and PAN images, while the output is fused images. The encoder and decoder are composed of multiple LEVM layers, with each LEVM layer comprising several LEVM blocks.For the LEVM block in the encoder, denoted as 𝑓 𝑒𝑛𝑐 (•), the input consists of the SSM hidden state 𝒉 𝑙 -1 from the previous block, LRMS, and PAN:EQUATIONwhere 𝑙 represents the 𝑙-th layer, and 𝒙 0 denotes LRMS in the first encoder layer (i.e., 𝑙 = 0), then outputs features and hidden states into the next layer, denoted as 𝒙 𝑙 and 𝒉 𝑙 , respectively. Different from the previous vision Mambas, the state 𝒉 𝑙 -1 is incorporated. Its rationale will be explained in Sect. 4.3.For the LEVM blocks in the decoder, their inputs are the concatenation of the output from the corresponding encoder at the same resolution and the output from the previous decoder layer, denoted as 𝑓 𝑑𝑒𝑐 (•). Similar to the LEVM block in the encoder, by incorporating the corresponding encoder's state 𝒉 𝑙 𝑒𝑛𝑐 , a better fusion performance can be achieved, which is akin to the encoder-decoder skip connections commonly employed in U-Nets. Our intuition is that the encoder's state contains abundant low-level information, which can complement the semantic information in the decoder's state space. This process can be formulated as:EQUATIONwhere 𝒙 𝑙 -1 and 𝒉 𝑙 -1 are input features and hidden states of the (𝑙 -1)-th layer in the decoder, respectively. Finally, the output of the last LEVM block is mapped back to the pixel space by a linear layer. Following the high-frequency learning strategy proposed in [4] , we add the LRMS to the network's output to obtain the fused image. To supervise training, we set the loss function as:EQUATION)where 𝐺𝑇 is the ground truth, 𝐿 denotes the total number of layers of the network, and L 𝑠𝑠𝑖𝑚 represents loss function based on the SSIM [41] . In practice, we set 𝜆 = 0.1 to balance the two losses.The current vision Mamba models [24, 50] directly extend the Mamba [12] , originally designed for language sequences, to 2D images by patchifying the images into image tokens. According to Eq. ( 3), each input 𝑥 𝑡 shares the same system matrices 𝑨, 𝑩, 𝑪 to extract global information. This inspires us to study the representation ability of the Mamba model. As shown in Fig. 2 (c), we design a local enhancement vision Mamba (LEVM) block to cope with local and global information. The LEVM block has a local and a global VMamba block (i.e., VMambaBlock function, see Fig. 2(b) ).In the local information representation, inspired by Swin Transformer, we partition the image into windows 𝑾 ∈ R 𝑝 ℎ ×𝑝 𝑤 ×ℎ×𝑤×𝐷 , then feed them into the local VMamba block to extract local information, where ℎ, 𝑤 denote the window size and there exists 𝑝 ℎ = 𝐻 /ℎ, 𝑝 𝑤 = 𝑊 /𝑤, which can be formulated as:EQUATION)EQUATIONIn the global information representation, we can gather the local windows to further represent features from the local VMamba block. Specifically, we utilize the WindowMerge function that merges these windows, yielding the entire images, i.e.,𝒙 𝑙 -1 = [𝑾 𝑙 -1 1 , 𝑾 𝑙 -1 2 , • • • , 𝑾 𝑙 -1 𝑝 ℎ ×𝑝 𝑤 ] ∈ R (𝑝 ℎ ×𝑝 𝑤 ) ×ℎ×𝑤×𝐷 .For the 𝑙-th layer, the global VMamba block can be formulated as follows:EQUATION)EQUATIONThe However, due to the computation limitations of the VMamba block, these SSMs are unable to extract spatial and spectral information separately, resulting in insufficient information fusion. Therefore, we introduce the state sharing technique to achieve interaction between spatial and spectral information. The state sharing technique will be stated in the next section.In this section, the state sharing technique considers layer-wise information representation and spatial-spectral learning (S2L) of SSM. Specifically, our LE-Mamba uses efficient SSM based on convolution representation to compute image tokens. Then, in a layer of SSM, 𝑨 and 𝑩 are linear transformations, and 𝑪 sums these states to generate output features. Therefore, the state has a layerwise information representation. Also, the layer-wise information becomes increasingly semantic from shallow to deep layers.Inspired by RevCol [1] , we aim to retain low-level information across LEVM layers. In contrast to RevCol, which employs deep supervision for each column, we leverage the hidden state of SSM for two information flows propagated across layers, which is a direct and convenient approach. As shown in Fig. 2(d) , we build an adjacent flow (denoted as a blue dotted line) and a skip-connected flow (denoted as a green dotted line) to transfer the current state into the next layer and the corresponding decoder layer, respectively. These two information flows are summarized as follows:(1) The adjacent flow: We can get the local and global hidden states from the LEVM at the (𝑙 -1)-th layer, i.e., 𝒉 𝑙 -1 𝑙𝑜𝑐𝑎𝑙 and 𝒉 𝑙 -1 𝑔𝑙𝑜𝑏𝑎𝑙 . Then 𝒉 𝑙 -1 𝑙𝑜𝑐𝑎𝑙 and 𝒉 𝑙 -1 𝑔𝑙𝑜𝑏𝑎𝑙 are fed into the next layer. (2) The skip-connected flow: Instead of the skip-connected method of U-Net, the (𝑙 -1)-th layer's global hidden state 𝒉 𝑙 -1 𝑔𝑙𝑜𝑏𝑎𝑙 is fed into the corresponding decoder layer.In Fig. 3 , we further show feature maps of two information flows containing semantic information and low-level details, respectively. Also, there are some ablation studies (see Tab. 4) to demonstrate the information of adjacent flow and skip-connected can improve fusion performance. Now, we improve the LEVM block with state sharing. Specifically, we perform a residual connection between the current input features and spatial-spectral learning of SSM (S2L). In S2L, two different linear blocks are conducted to project state space and input features, which can be formulated as follows:EQUATION)EQUATIONwhere the first linear projects 𝒉 𝑙 -1 along the channel dimension, and the second linear projects 𝒙 𝑙 -1 : R 𝐷 ×𝐿 → R 𝑁 ×𝐿 . Although the VMamba block is capable of capturing pixel dependencies well to achieve spatial fidelity, it does not fully exploit spectral information and overlooks the interaction between spatial and spectral information. Based on this, for the 𝑙-th layer, our S2L decomposes 𝒙 𝑙 -1 ∈ R 𝐷 ×𝐿 into state space 𝒉 𝑙 -1 ∈ R 𝐷 ×𝑁 and EQUATION)where the learnable parameter 𝛼 ∈ R 𝐷 is used to balance the information between the S2L and input features. In S2L, the state space is regarded as the basis for spectral representation, and the input image is mapped into the feature space with rich spatial information. Then, we use matrix multiplication to establish the interaction between spectral information and spatial information in the state space. In contrast to the SSM which focuses on processing spatial information, the S2L method takes into account the representation of hidden state in the spectral domain and the spatial characteristics of the image. This allows for extracting and learning two features that simultaneously possess spatial and spectral properties, making it more suitable for image fusion tasks. Finally, our state sharing technique is summarized in Algorithms 1 and 2. To the best of our knowledge, there haven't been improvements made to the vision Mamba that specifically addresses image fusion tasks. The proposed state sharing technique can be viewed as an independent contribution as it can adapted to any SSMs.In this section, we analyze the parameter counts and computational complexity of the proposed LEVM with the state sharing method. The parameter count of the local VMamba block and global VMamba block is both 𝑂 (5𝐷𝑁 ), respectively. The floating point operations (FLOPs) of these two blocks are 𝑂 (5𝐵𝐿 ′ 𝑁 + 2𝐿 ′ 𝐷) and 𝑂 (5𝐵𝐿𝑁 + 2𝐿𝐷), respectively. For the state sharing method, the parameter count is 𝑂 (𝐷 2 + 𝐷𝑁 ) and the FLOPs is 𝑂 (𝐷𝐿𝑁 ). The total parameter count is 𝑂 (10𝐷𝑁 + 𝐷 2 + 𝐷𝑁 ). The total FLOPs is 𝑂 (10𝐵𝐿𝑁 + 2𝐿𝐷 + 2𝐷𝑁 ). Then, the space requirement is 𝑂 (𝐵𝐿𝐷). Furthermore, we list these complexity metrics in Tab. 1. In addition, Fig. 1 illustrates that although our LE-Mamba has some increasing parameters and FLOPs than classical methods, we have near-linear memory consumption compared with convolution, self-attention, and attention of Swin Transformer. 
experiments: 
 This section presents extensive experiments on the proposed LE-Mamba, describing the used datasets, and implementation details, and comparing it with previous methods. We analyze the reducedresolution and full-resolution performances in the multispectral pansharpening task, as well as the performances in the multispectral and hyperspectral image fusion task. Ablation studies are conducted on the proposed LEVM and the state sharing technique to validate their effectiveness. ⊲ "CrossScan" refers to [24] .𝒙 𝑙 -1 : (B, K, D, L) ← CrossScan(𝒙 𝑙 -1 ); ⊲ State sharing. "groups" is the group of convolution.𝒉 𝑙 -1 : (B, K, D, N)←Linear(𝒉 𝑙 -1 , groups=𝐾); 6 𝒈: (B, K, N, L) ← Linear(𝒙 𝑙 -1 , groups=𝐾); ⊲ Spatial-spectral learning (S2L) of SSM.𝒙 𝑙 -1 ← 𝒙 𝑙 -1 + einsum('BKDN,BKNL→BKDL',𝒉 𝑙 -1 ,𝒈); ⊲ Parameterize 𝑨, 𝑩, 𝑪 and 𝚫. See Algo. 1.𝑨, 𝑩, 𝑪, 𝚫 ← ParamFn(𝒙 𝑙 -1 , Param 𝑨 , Param 𝚫 ); ⊲ 2D selective scan. Ref to [24] .𝒙 𝑙 , 𝒉 𝑙 ← SS2D(𝒙 𝑙 -1 , 𝑨, 𝑩, 𝑪, 𝚫); ⊲ "CrossMerge" refers to [24] . To evaluate the proposed LE-Mamba, we conduct experiments on two remote sensing multispectral pansharpening datasets: WV3 [20] 0.78±0.15 0.69±0.11 0.980±0.009 0.991±0.002 0.032±0.013 0.079±0.014 0.891±0.020 0.15M 0.54G Invformer [48] 0.83±0.14 0.70±0.11 0.977±0.012 0.980±0.002 0.059±0.026 0.110±0.015 0.838±0.024 2.77M 3.46G DCFNet [42] 0.89±0.16 0.81±0.14 0.973±0.010 0.985±0.002 0.023±0.012 0.066±0.010 0.912±0.012 2.77M 3.46G HMPNet [37] 0.80±0.14 0.56±0.10 0.981±0.030 0.993±0.003 0.080±0.050 0.115±0.012 0.815±0.049 1.09M 2.00G PanDiff [31] 0.89±0.12 0.75±0.10 0.979±0.010 0.989±0.002 0.027±0.020 0.073±0.010 0.903±0.021 45.33M 14.83G PanMamba [16] 0.68±0.12 0.64±0.10 0.982±0.008 0.985±0.006 0.016±0.008 0.045±0.009 0.939±0.010 0.48M 1.31G Proposed 0.60±0.11 0.52±0.09 0.987±0.007 0.994±0.001 0.018±0.009 0.027±0.008 0.955±0.011 0.74M 3.58G and GF2 datasets with super-resolved scale 4, and further on indoor hyperspectral-multispectral fusion datasets: CAVE and Harvard datasets with scale 8. More details on the datasets can be found in the supplementary.During the training process, we use the AdamW [27] optimizer and set the base learning rate to 1e-3, which is decreased to 1e-4 at 300 epochs, and further decreased to 1e-5 at 600 epochs. After that, the training continues till 1000 epochs for WV3 and GF2 datasets and 1600 epochs for CAVE and Harvard datasets. The weight decay is set to 1e-6. All experiments are conducted on two RTX 3090 GPUs. Network configurations on different datasets can be found in the supplementary.For the multispectral pansharpening datasets, we compare the proposed method with the recently SOTA traditional methods: MTF-GLP-FS [39] , BT-H [26] and LRTCFPan [43] , and DL-based methods: DiCNN [15] , LAGConv [20] , DCFNet [42] , HMPNet [37] , Pan-Diff [31] and PanMamba [16] . For the multispectral and hyperspectral datasets, we also choose some recent traditional methods including CSTF-FUS [21] , LTTR [8] , LTMR [7] and IR-TenSR [44] . Moreover, most competitive DL-based methods are compared: ResTFNet [23] , SSRNet [47] , HSRNet [18] , MogDCN [9] , Fusformer [17] , DHIF [19] , PSRT [6] , 3DT-Net [30] , DSPNet [36] , and MIMO-SST [11] .The fusion performance of LE-Mamba on the WV3 and GF2 datasets is provided in Tab. 2, indicating the SOTA performance of our method in reduced-resolution metrics. Traditional approaches lag significantly behind DL-based methods in terms of reduced-resolution metrics. Among DL-based methods, our approach demonstrates good fidelity for both spectral modality (i.e., SAM metric) and spatial modality (i.e., ERGAS metric). In terms of full-resolution metrics, LE-Mamba exhibits SOTA performance on the GF2 test set, indicating its good generalization capability. We present fusion images on the WV3 test set in Fig. 4 , showcasing LE-Mamba's minimal errors, particularly evident in the fusion of high-frequency components such as the edges of buildings and roads.Tab. 3 provides widely used numerical fusion metrics on CAVE and Harvard datasets. Our LE-Mamba outperforms all previous traditional and DL-based methods contributed by the multi-scale backbone and proposed LEVM equipped with state sharing technique. Error maps are depicted in Fig. 5 which shows that LE-Mamba owns fewer errors and better fusion performances. More visual results can be found in the supplementary. 14), we can observe that the state sharing technique is a residual connection that adds input feature 𝒙 𝑙 -1 with S2L (i.e., 𝒉 𝑙 -1 𝒈). To verify the effectiveness of using the S2L (whether the performance gain comes from residual connections and additional convolutional parameters), we design another residual variant that does not use the SSM state:EQUATIONWe validate its fusion performance on the WV3 dataset, as shown in Tab. 5. Our state sharing outperforms the residual variant. From this ablation, we can conclude that: using S2L to share state across different blocks can bring performance gain. To compare the proposed LEVM with previously common Attention mechanisms, we select common attention and its variants including traditional Attention [10] , PVT Attention [40] , Swin Attention [25] , and Linear Attention [14] . We replace the LEVM block with the corresponding Attention block (including FFN) and train them uniformly on the WV3 dataset until convergence. Their fusion performance is shown in Tab. 6. It can be seen that the traditional Attention performs the worst, which is because using Attention in the first layer of the network occupies an extremely large GPU memory, leading to the inability to train. Therefore, most methods based on traditional Attention perform patch embedding in the first layer to downsample the image size, which is disastrous for tasks like image fusion that require low-level information. The performance of Linear Attention is also relatively poor, which is caused by the approximation errors introduced by its approximation of the Attention mechanism. Next are PVT Attention and Swin Attention, both of which impose restrictions on Attention. PVT downsamples both the keys and values, resulting in information loss, while Swin limits the global information to the local regions. The proposed LEVM outperforms previous attention operations.  
 
 Appendix:  


Decision: This submission has been reviewed by four experts, the final ratings from whom are all positive after rebuttal (although one of them held a negative vote in initial phase, after rebuttal, the reviewer raised his/her score). Considering the quality of the submission, the comments from the reviewers, and the rebuttal from the authors, the paper can be accepted by the conference.

Reviews: review: paper_summary: paper_summary: Considering the characteristics of the image fusion task, this paper proposes a multi-scale network with local enhancement and state sharing. The network comprehensively preserves the details of the source image by extracting global and local information. State-sharing technology shares the state between adjacent and skip-connected flows, reducing information loss and learning spatial and spectral information simultaneously.
strength_weakness: strength_weakness: 1. Corresponding improvements have been made to address the shortcomings of Mamba in image fusion. Mamba blocks inevitably suffer information loss as the sequence length increases. The authors solve this problem by developing state-sharing technology, which applies the hidden state of SSM for two information flows propagated across layers. The authors constructed adjacent flow and skip connection flow transfer states to achieve competitive performance.
2. Extensive experiments have proven the superiority of LE-Mamba. From the qualitative and quantitative experimental results given in the paper, the performance of LE-Mamba has reached SOTA. In addition, the author conducted ablation experimental studies for his innovative design, and the results clearly demonstrate that the proposed technique is effective.
limitations: limitations: 1. For local-enhanced vision Mamba (LEVM), the authors are inspired by the Swin Transformer and partition the image into windows, then feed them into the network. So if the author has made corresponding improvements to the LEVM for image fusion tasks, is the LEVM just a migration application of Swin Transformer?

2. The author said in the introduction that one of the drawbacks of Mamba is that it is unable to characterize spatial and spectral information. Please explain what flaw in Mamba causes this situation. How does the S2L learning developed by the author achieve the characterization?

3. I wonder what the advantages of applying the state space model to image fusion are compared to general CNN.

4. The obvious advantage of Mamba is its efficiency. As can be seen from the authors' quantitative experiments, the efficiency of LE-Mamba is competitive. However, FLOPS is related to many factors and cannot fully prove the efficiency of the network. It is recommended to add time as a supplement to demonstrate the advantages of LE-Mamba.

5. When describing the advantages of his method, the author mentioned that the interaction of spatial and spectral information can be achieved. As can be seen from Figure 2(c), the author simply adds the two features and sends them into VMamba. Is this how the author implements interaction? If not, please add a detailed explanation in the paper.
paper_summary: paper_summary: This paper proposed a novel state space model with local-enhancement and state-sharing technique to enhance the performance of the vision Mamba on image fusion tasks. The proposed local-enhanced vision Mamba alleviates the state information loss issue of SSMs mentioned in the paper. Additionally, the proposed state-sharing technique can be applied to other SSMs to boost their performance. The conducted experiments on image fusion tasks, and the promising results demonstrate the effectiveness of their proposed methods.
strength_weakness: strength_weakness: (1) The proposed method reaches a new SOTA in image fusion task.
(2) This paper starts from the issue of insufficient SSM state capacity, and considers enhancing it to local-enhanced Mamba: assigning each window to a local Mamba for processing, and finally processing global information with another global Mamba block, which is innovative.
(3) The authors propose a state sharing technique, sharing the local and global states of each LEVM between adjacent and skipped blocks, solving the issue 3) considered in the paper.
limitations: limitations: (1) Akin to Swin transformer, the author should consider to shift the window to make the information communication in each window (although there still exists the global mamba block).
(2) The author should explain how to visualize the proposed adjacent flow and skip-connected flow.
(3) What does the $K\leftarrow 4$ in Algo.2 Line 3 means? 
typos: 1) "casual" in abstract, should it be "causal"?
paper_summary: paper_summary: This paper achieves new SOTA performance on image fusion task by lifting Mamba model with a new local-enhancement Mamba module and a state sharing technique. By comparing with previous commonly-used attention and convolution operators, the proposed LEVM shows promising multi-modal fusion performance and memory, FLOPs balance. Some illustrations of state sharing technique shows convinced motivation. Overall, I think this is a novel paper that improves vision mamba and helps multi-modal image fusion.
strength_weakness: strength_weakness: 1. This paper notes two issues that vision mamba models face. And the authors proposed two helpful improvements to mitigate these issues. The results and module ablation studies prove it.
2. Local-enhancement mamba module mitigate the information loss.
3. Clear illustrations of adjacent flow and skip-connected flow shows the effectiveness on feature map.
4. Relatively speaking, the proposed LEVM achieves a good balance between convolution and attention.
limitations: limitations: 1. The authors should make implementation of FFN clear, since it is also a prominent component of the proposed LEVM.
2. Why 3DT-Net owns 68.07G FLOPs, far more than other methods? Is this a typo? Correct me if I am wrong.
3. Since this paper is about SSM, the hyperspectral and multispectral image fusion benchmark lacks a SSM counterpart, the author should consider to include a SSM architecture to compare.

typos: (1) In Figure 2 (b), should the caption be "... of LEVM block"? (2) In Table 1, the "Space" indicates "GPU Memory"?
paper_summary: paper_summary: This paper proposes a local-enhanced Mamba network and an accompanying state sharing technique that achieves state-of-the-art results on remote sensing pansharpening and hyperspectral image fusion tasks. The ablation study also shows that incorporating the local-enhanced Mamba block performs better than other attention or convolutional operations. The paper also ablates the proposed state sharing technique, demonstrating its effectiveness.
strength_weakness: strength_weakness: 1. It's great that the paper has clear motivation, good writing, and positive illustrations explaining the role of each module.
2. The superior performance of the proposed LE-Mamba over other methods is a compelling result. Besides, the state-sharing technique proposed is novel and ingenious in addressing the limited state capacity issue faced by SSM-based networks for low-level tasks. The experimental validation of its effectiveness is a good point.
3. It is very commendable that authors compare different attention operations within the same network architecture, and analyse their tradeoffs in terms of memory consumption and FLOPs. The given systematic evaluation is valuable for understanding the pros and cons of different choices.
limitations: limitations: 1. PanDiff has such a large number of parameters, why does it perform poorly, and what is different between it and other methods?
2. What does "Space" in Tab.1 refer to? It should be memory consumption?
3. In Alg. 2 Line 5, why does Linear have groups, perhaps it is grouped convolution, this is not a technical issue, but the authors need to state it precisely.
4. In Sect. 2.2 Line 208, it writes "near-linear", but the caption in Fig.1 writes "linear". I think it should be corrected to also "near-linear" in the caption according to the plot.