Linearly-evolved Transformer for Pan-sharpening

Authors: junming  hou
zihan  cao
naishan  zheng
xuan  li
xiaoyu  chen
xinyang  liu
xiaofeng  cong
danfeng  hong
man 2024 zhou
  linearly-evolved
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 08:54:38

Abstract: Vision transformer family has dominated the satellite pan-sharpening field driven by the global-wise spatial information modeling mechanism from the core self-attention ingredient. The standard modeling rules within these promising pan-sharpening methods are to roughly stack the transformer variants in a cascaded manner. Despite the remarkable advancement, their success may be at the huge cost of model parameters and FLOPs, thus preventing its application over low-resource satellites. To address this challenge between favorable performance and expensive computation, we tailor an efficient linearly-evolved transformer variant and employ it to construct a lightweight pan-sharpening framework. In detail,  we deepen into the popular cascaded transformer modeling with cutting-edge methods and develop the alternative 1-order linearly-evolved transformer variant with the 1-dimensional linear convolution chain to achieve the same function. In this way, our proposed method is capable of benefiting the cascaded modeling rule while achieving favorable performance in the efficient manner. Extensive experiments over multiple satellite datasets suggest that our proposed method achieves competitive performance against other state-of-the-art with fewer computational resources.  Further, the consistently favorable performance has been verified over the hyper-spectral image fusion task. Our main focus is to provide an alternative global modeling framework with an efficient structure. The code is publicly available at \url{https://github.com/coder-JMHou/LFormer}.

Main paper: 
 introduction: 
 The proliferation of remote sensors has made explosive satellite imagery accessible across diverse domains such as military systems, environmental monitoring, and mapping services [7, 22] . Given the inherent physical constraints, satellites typically employ multi-spectral (MS) and panchromatic (PAN) sensors to capture the complementary information concurrently. Specifically, MS images exhibit superior spectral resolution but limited spatial resolution, whereas PAN images provide abundant spatial details but lack spectral resolution. Consequently, the fusion of MS and PAN images through Pan-sharpening techniques has garnered escalating interest from the image processing and remote sensing communities, enabling the generation of images with enhanced spectral and spatial resolutions.In recent years, convolutional neural networks (CNN) have made significant strides in the satellite pan-sharpening field, surpassing traditional optimization pan-sharpening methods by a substantial margin, thanks to their powerful learning capability. However, Figure 1 : The comparison of PSNR and computational overhead between our model and other cutting-edge techniques. Notably, the Parameters axis is depicted using a logarithmic scale with a base of 2 for clear illustration. It is evident that our method showcases the promising performance-efficiency balance compared to other approaches. the landscape has recently been disrupted by the emergence of the vision transformer family, which challenges the dominance of CNNs by leveraging global-wise spatial modeling based on dotproduct self-attention. Among the transformer-based methods, IN-Nformer [40] stands out as a representative approach, employing a multi-modal transformer to capture long-range cross-modality relationships and outperforming previous CNN-based methods. Since then, a multitude of explosive complex transformer variantsequipped pan-sharpening architectures have emerged and solidified their position at the forefront [2, 10, 13] . Notably, these promising transformer-based pan-sharpening architectures generally adhere to a cascaded stacking of transformer variants as a common modeling rule. As displayed in Figure 1 , despite their remarkable progress, the success of these approaches often comes at the expense of increased model parameters and floating-point operations (FLOPs), limiting their applicability in low-resource satellite scenarios. To tackle the aforementioned challenge of balancing high performance with substantial computation costs, we delve into the origins of the computation cost, identifying the dot-product self-attention mechanism as a major contributor. In our investigation, we delve into the underpinnings of self-attention and inquire whether an alternative 1-order modeling mechanism could replace the current transformer chain in a more computationally efficient manner. By exploring this avenue, we aim to find a solution that maintains performance while mitigating the resource-intensive nature of the transformer architecture.Building upon the aforementioned principle, we delve further into the widely adopted cascaded transformer modeling approach used in state-of-the-art methods and design a linearly-evolved transformer as illustrated in Figure 2 . This revelation leads us to develop an alternative 1-order linearly-evolved transformer variant using a chain of 1-dimensional linear convolutions. By leveraging this design, we construct a lightweight pan-sharpening framework that relies on a well-designed, simple yet efficient linearly-evolved transformer. This framework aims to strike a balance between computational efficiency and performance in pan-sharpening tasks. By adopting this design, our proposed method harnesses the advantages of the cascaded modeling rule while achieving impressive performance in a computationally efficient manner. Through extensive experimentation on various satellite datasets, we have demonstrated that our method delivers competitive performance compared to other state-of-the-art approaches while utilizing fewer computational resources. Our primary objective is to provide an alternative global modeling framework with an efficient structure, prioritizing both performance and resource efficiency. The work's contributions are summarized as follows:• We introduce a novel, lightweight, and efficient pan-sharpening framework that achieves competitive performance while reducing computation costs compared to state-of-the-art pansharpening methods.• We uncover the 1-order principle of self-attention and propose a linearly-evolved transformer chain that replaces the common modeling rule of N-cascaded transformer chains with a feasible approach utilizing 1-transformer and N-1 1dimensional convolutions to achieve the same function.• The proposed linearly-evolved transformer provides an effective alternative for global modeling, offering significant potential for designing efficient models. 
related works 2.1 pan-sharpening: 
 Existing pan-sharpening methods can be roughly divided into four types: component-substitution (CS)-, multi-resolution analysis (MRA)-, variational optimization (VO), and deep learning-based methods [22] . Among them, the first three categories are also classified as traditional methods. The fused results of CS-based approaches often exhibit significant spectral distortion [3, 14] , while the products from MRA-based methods suffer from spatial distortion despite they show superior spectral quality in comparison to CS-based approaches [1, 15, 24] . VO-based methods generate the image with desirable spatial-spectral preservation conditioned on the heavy computational burden [6, 32, 33] . Recently, deep learningbased methods have dominated the pan-sharpening field. The pioneering work only consists of three convolution layers, while achieving a competitive result compared with traditional methods [21] . Subsequently, Yang et al. propose the first deeper CNN for pan-sharpening [35] . Since then, more complicated network architectures have been designed for pan-sharpening, showing significant performance gains while leading to high computational and memory footprint [17, 26, 43] .Very recently, the vision transformer family has dominated the satellite pan-sharpening field. In pioneering works, however, researchers roughly employ the cascaded vision transformer designs The comparison between the prior cascaded self-attention designs within transformer and our proposed linearlyevolved mechanism. In this way, our linearly-evolved design is capable of inheriting the merits of a cascaded manner with the huge computation cost reduction. to the pan-sharpening problem [10, 23] , which ignore some taskrelated characteristics, e.g., the difference between input source images. Afterward, more task-specific transformers are designed for pan-sharpening. For example, Bandara et al.  
proposed method: 
 We first summarize the overall framework, and then revisit the modeling principle of self-attention within the traditional transformer and provide the details of the alternative linearly-evolved transformer chain, which is the core design of our work.Figure 4 outlines the overall architecture of the proposed method, which consists of two branches. Given an up-sampled MS image ℳ ∈ R H×W×c and PAN image 𝒫 ∈ R H×W×1 , the upper branch applies an input projection block to extract their shallow features. While the below one initially employs a high-pass filter to obtain their high-frequency details, denoted as ℳ ∈ R H×W×c and 𝒫 ∈ R H×W×1 , which are further projected into the feature space. Then, we conduct the cross-attention computation between MS and PAN features, yielding the long-range dependency feature representation, which is further interacted with the extracted highfrequency features. Next, we conduct several core building modules LFormer coupled with feature integration blocks to obtain the informative features, and then combine with the input ℳ ∈ R H×W×c to reconstruct a high-resolution MS image ℋ 𝑠 ∈ R H×W×c . Briefly, our method can be expressed as follows:EQUATIONwhere Φ(•) and Ψ(•) involve extracting the initial long-range features and the high-frequency detail information of the source images, respectively. LFormer {•} denotes the mapping function of the proposed linearly-evolved transformer chain.Revisiting the Traditional Multi-head Self-attention. Given an input feature 𝒳 ∈ R H×W×C flatten to HW × C, where H and W represent the height and width, respectively, while C is the channel number. Vision Transformer often applies the self-attention module to deal with the input feature, which can be mathematically formulated as follows:EQUATIONwhere Q, K and V ∈ R HW×C are embedded from 𝒳 , A ∈ R HW×HW denotes attention map calculated by Q, K. ⊗ means the matrix multiplication. 𝒴 is the output of the self-attention module. Furthermore, we can obtain the following expression:EQUATIONwhere q i ∈ Q, k j ∈ K, a i,j ∈ A. In terms of 𝒴 ∈ R HW×C , we can obtain the following formula:EQUATIONwhere 𝒪 1 (•) indicates the 1-order weights.Delving into the Modeling Rule of Cascaded Transformer Chain. Very recently, the vision transformer family has dominated the satellite pan-sharpening field driven by the global-wise spatial information modeling mechanism from the core self-attention ingredient. The common modeling rules within these promising pan-sharpening methods are to roughly stack the transformer variants in a cascaded manner, which can be formulated as follows:𝒳 → (Q 1 , K 1 , V 1 ) → (Q 2 , K 2 , V 2 ) • • • → (Q r , K r , V r ) • • • → (Q N , K N , V N ) (5) 𝒳 → 𝒪 1 (V 1 ) → 𝒴 1 → 𝒪 1 (V 2 ) → 𝒴 2 • • • → 𝒪 1 (V r ) → 𝒴 r • • • → 𝒪 1 (V N ) → 𝒴 NTaking the adjacent two steps of the above Markov's chain, for example, we can summarize it asEQUATIONWe give out the proof next. Similarly, standing on the output 𝒴 r , the r + 1 step performs the dot-product self-attention as(Q r+1 ,K r+1 , V r+1 ) ← (W r+1 q 𝒴 r , W r+1 k 𝒴 r , W r+1 v 𝒴 r ), 𝒴 r+1 = Softmax( Q r+1 K T r+1 √ d )V r+1 (7) = A r+1 ⊗ V r+1 ∝ 𝒪 1 (V r+1 ) ∝ 𝒪 1 (𝒴 r ),where A is the self-attention map, To emphasize, the A is independent as V. Based on the above principle, we can deduce asEQUATIONLinearly-evolved Transformer. Therefore, the above calculation can be simplified with any form of the 1-order weight function.Referring to the above calculating,EQUATIONand it can be modeled as follows:EQUATIONIn our study, we utilize the basic 1-dimensional convolutionC 1 i (i = 1, • • • , N -1)with the kernel size of 1 × 𝑘 to address 1-order functions, where * denotes the convolution operation. To emphasize, the complexity of the previous self-attention mechanism A is quadratic. In contrast, our 1-dimensional convolution design C 1 i exhibits linear complexity. It can be seen that our design reduces the complexity by 1 order of magnitude.As described in Equation 1, our proposed framework includes two fundamental components: the LFormer(•) branch and the feature integration branch. Flow of the LFormer Branch. In our proposed LFormer branch, we first project the PAN image 𝒫 and the up-sampled MS image ℳ into the feature space, denoted as ℱ 𝒫 and ℱ ℳ , through convolution layers with non-linear activation, formulated as follows:EQUATIONwhere Proj(•) denotes the convolution layers with reshape operation. Next, we conduct the cross-attention computation to capture the long-range dependency representation between PAN and MS modalities. Targeting at pan-sharpening, we take ℱ 𝒫 as query, while ℱ ℳ is used as key and value, written as following form:EQUATIONwhere A 1 denotes the calculated attention map, ℱ g 1 is the output of the cross-attention module. After that, we employ 𝑁 LFormer(•) modules to advance the global feature representation, in which the 1 × 𝑘 convolution layer is served as the 1-order linear weight function followed by the Softmax function to evolve the attention map. Besides, the value is also updated by injecting the integrated features at each stage. The whole procedure can be mathematically expressed as follows:EQUATIONwhere N is the number of the designed LFormer(•) module, Cat(•) denotes concatenation operation along the channel dimension, * denotes the convolution operation, C 1 i and A i represent the 1-order linear weight function and the evolved attention map with respect to the i-th LFormer(•) module, while V i is the corresponding value. ℱ g i is the output of the i-th LFormer(•) module and ℱ d i is detailed as below. Flow of the Feature Integration Branch. We combine the highfrequency information and the output of the LFormer(•) module at each stage to update the value. Specifically, we first employ the Sobel operator to extract the high-frequency components of MS and PAN, denoted as ℳ and 𝒫, and then adopt several convolution blocks similar to those of the LFormer(•) branch to project them into shallow features. Then, we integrate the output of the LFormer(•) module and the high-frequency information through several convolution layers to update the value. Mathematically, this process is formulated as follows:EQUATIONwhere Sobel(•) represents the Sobel operator, ℱ 𝑑 0 is the extracted high-frequency low-level features, FIB(•) denotes fusing global and detail formation that is further used to update the value.We adopt two loss terms, including the reconstruction loss ℒ 𝑟 and the structure loss ℒ 𝑠 , as following:EQUATION)where 𝛼 is the hyperparameter that is used to balance the overall performance and the structure details. Specifically, we choose a widely used L1 loss to calculate the reconstruction loss ℒ 𝑟 , while the structure loss ℒ 𝑠 is obtained through structural similarity (SSIM). They can be defined as follows:EQUATIONEQUATIONwhere ℋ 𝑠 and 𝒢𝒯 are the fused result and the matching ground truth, respectively. 
experiments: 
 Datasets and experimental settings are detailed in supplementary.Results on Reduced-resolution Scene. We perform reducedresolution assessment on WorldView-3 (WV3) and GaoFen-2 (GF2) datasets to quantitatively evaluate the similarity between the fused multispectral images and the ground truth images (original MS images). Table 1 presents the average performance of all compared pan-sharpening methods on WV3 and GF2 datasets, respectively, where our model achieves the best results across all metrics. Figure 5 and 6 display the qualitative comparisons of the error maps between our model and other cutting-edge methods over WV3 and GF2 datasets. It is clearly observed that our model presents a favorable outcome evidenced by its dark blue residual map.Results on Real-world Full-resolution Scene. To evaluate the generalization in real-world scenes, we conduct full-resolution evaluation. As ground truth images are not available, we rely on quality indexes without reference for performance assessment. Model trained on reduced-resolution data are applied to real scenes. exhibits the quantitative results of all compared pan-sharpening methods on GF2 datasets. Our proposed framework demonstrates superior performance again, showcasing its exceptional generalization capacity.We extend our model to the application of hyperspectral image super-resolution (HISR). This application shares similar degradation principles with the multispectral pan-sharpening task. We compare our model with several state-of-the-art methods using the widely used CAVE dataset. Table 3 showcases our model's superior performance, surpassing the compared methods by a significant margin. Figure 8 displays the residual maps between the fused images and the GT image, and their spectral response at a spatial location [400, 200] . It is apparent that the dark blue residual map further demonstrates the high similarity between the fused product of our model and the GT image. Moreover, the spectral response curve of our model closely aligns with the GT, indicating its desired spectral preservation capability.To further demonstrate the feature representation capabilities of our model, we provide the feature maps of different blocks as displayed in Figure 7 . It is clearly that the feature map becomes more distinguishable and provides rich detail information with the increase of the number of blocks, thereby proving the desirable expressiveness of our proposed key linearly-evolved transformer design. 
ablation study: 
 All ablation experiments are implemented on GF2 dataset. Effect of Attention Evolution. We use the proposed LFormer as the baseline and compare it with alternative methods by altering the approach for calculating the attention map. All comparison networks are trained using the same methodology. Specifically, we compare three configurations: Baseline: Conducting the crossattention in the first block, followed by leveraging our core linear evolution strategy to evolve the attention weights in the remaining blocks. Config.I: Performing cross-attention computation at each block. Config.II: Removing the linear evolution within the baseline, thereby directly sharing the cross-attention weights obtained from the first block with the remaining blocks. Table 4 presents the quantitative results of different models. Our proposed LFormer consistently achieves the best outcomes while significantly reducing the number of network parameters and FLOPs. It clearly illustrates the performance benefits of LFormer against its variants. In particular, the Config.I incorporates the repetitive and unnecessary self-attention computations, leading to increased model parameters and FLOPs while inferior performance.Effect of Kernel Size. To examine the function of the employed 1-D convolution kernel within the evolved process, we select several representative 1-dimensional convolution units with 1×1, 1×3, 1×5, and 1 × 7 kernel size. From the reported quantitative comparison in Table 5 over the proposed LFormer and its variants on the GF2 dataset, it can be deduced that with the kernel size increasing, the model performance tends to improve. While there is a slight decrease in performance when the kernel size increases to 1 × 7, probably due to the attention weights between different layers experiencing local fluctuation only. Therefore, we set the 1 × 5 kernel size as default.Extendibility. We further apply the proposed linear evolution paradigm to local attention mechanism to validate its scalability. Similar to the ablation experiment 1, we investigate three model configurations with different attention computation manners. Baseline: Conducting the window attention in the first block, followed by leveraging our core linear evolution strategy to evolve the attention weights in the remaining blocks. Config.I: Performing window attention computation at each block. Config.II: Removing the linear evolution within the baseline, thereby directly sharing the window attention weights obtained from the first block with the remaining blocks. As reported in Table 6 , the model configured with our linear evolution strategy yields the best results despite the slight increments in parameters and FLOPs compared to Config.II, showcasing its promising applicability.  
limitation: 
 We assess the effectiveness of our proposed framework in panchromatic and multispectral image fusion, as well as hyperspectral image fusion task. Additionally, we aim to test the scalability and versatility of the core linearly-evolved transformer in other low-resource image restoration tasks, such as efficient image super-resolution and Ultra-High-Definition tasks. To highlight, our primary objective is to offer an alternative global modeling framework with an efficient structure. 
conclusion: 
 We propose an efficient variant of the linearly-evolved transformer for lightweight pan-sharpening. By interpreting self-attention as a 1order linear weight function, we replace the N-cascaded transformer chain with a single transformer and N-1 convolutions. Leveraging this insight, we develop an alternative 1-order linearly-evolved transformer using 1-dimensional convolutions. Extensive experiments on multispectral and hyperspectral image sharpening tasks confirm the competitive performance of our method against stateof-the-art approaches. 
 
 Appendix:  


Decision: This work develops an efficient linearly-evolved transformer variant to construct a lightweight framework to address the challenge of high model parameters and FLOPs in satellite pan-sharpening using vision transformer family, achieving competitive performance with fewer resources and verified in hyper-spectral image fusion. Reviewers raised concerns about the novelty of the approach, unclear method description and insufficient experiments. With the authors' efforts in the rebuttal, most of the concerns are resolved. The authors are encouraged to improve the paper based on the reviewers' feedback.

Reviews: review: paper_summary: paper_summary: In this paper, a novel efficient variant of the linearly-evolved transformer is proposed for lightweight pan-sharpening. More specifically, the N-cascaded transformer chain is replaced by a single transformer and N-1 convolutions and the Soble operator is adopted by high-frequency information extraction.
strength_weakness: strength_weakness: 1. The performance of this work is good.
2. The motivation is somewhat novel. In this paper, they regard the cascaded transformer chain as the linearly-evolved transformer. From this perspective, designing a lightweight model is interesting.
3. The legend and experiments are clear.
limitations: limitations: 1. The article is readable, but somewhat difficult to understand, and a more organized presentation would be helpful. 
2. The clarify in “Proposed Method” section is not clear in detail. How about the details of “Proj” operation? “FIB” means the fusion of global and detail formation, but how to fusion? And where the Soble operator employ?  Maybe illustrated in framework figure 4 better. 
3. There are also some efficient linear attention modules based on ViT proposed in image classification, such as Flatten Transformer[1] in ICCV 2023. The differences between this method and these methods should be discussed, and the linear chains can be replaced by these linear attention?
[1]. Han, Dongchen, et al. "Flatten transformer: Vision transformer using focused linear attention." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
4. The experiments validation is lacking in some aspects and the effectiveness of methods in this paper can not be convinced. 
-In ablation study, the effectiveness of sobel operator and feature integration should be validated. 
-The FLOPs, Params, Latency metrics about efficiency is lack in table 1 and table 2.
paper_summary: paper_summary: This study focuses on the pan-sharpening problem and introduces a modified transformer model to alleviate the computational burden associated with the traditional transformer architecture. The proposed approach achieves efficient computation and competitive performance by introducing a 1-order linearly-evolved transformer variant with a 1-dimensional linear convolution chain. Experiments conducted on widely-used satellite images validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: The idea of linearly-evolved attention is interesting, and the experiments showcase the effectiveness of the proposed method.
limitations: limitations: 1. The paper suggests that the newly proposed linearly-evolved attention reduces computation costs. However, it only provides the FLOPs, and it's unclear how the computation complexity of the proposed attention compares to the O(N^2) complexity of traditional self-attention and the running speed of the method.
2. The paper adopts the fancy name linear-evolved attention, where its core operator is $1\times N$ convolutions. One-dimensional convolution has been used widely in the design of light-weight CNN networks, such as InceptionNet. The novelty of the proposed attention is questionable.
paper_summary: paper_summary: In this paper, the authors propose an efficient variant of the linearly-evolved transformer for lightweight pan-sharpening. By interpreting self-attention as a 1-order linear weight function, the authors replace the N-cascaded transformer chain with a single transformer and N-1 convolutions.
strength_weakness: strength_weakness: The authors delve into the popular cascaded transformer modeling with cutting-edge methods and develop an alternative 1-order linearly-evolved transformer variant with a 1-dimensional linear convolution chain to achieve the same function. This approach allows their proposed method to benefit from the cascaded modeling rule while achieving favorable performance in an efficient manner.
limitations: limitations: 1.The author should strengthen the article's motivation by conducting a detailed analysis of why current mainstream transformer models use a cascading approach to stack transformer blocks.
2.With rapid advancements in the field of Pan-sharpening, it is strongly advised that the authors compare their proposed method with more recent algorithms published within the last two years, as some of the comparison algorithms used in the study may be outdated.
3.Figure 3 should include a comparison of this article's approach with ViT.
4.The model's feature representation capability is illustrated in Figure 7. The author should elaborate on why replacing the n-cascade transformer chain with a single transformer and N-1 convolution can enhance the feature representation capability of the model.
paper_summary: paper_summary: For the pan-sharpening task, this work proposes an efficient linearly evolved transformer variant that replaces cascaded self-attention with the 1-dimensional linear convolution chain for computational efficiency.
strength_weakness: strength_weakness: The paper is sound, clear motivation, well-organized, and articulately presented, ensuring it is easily understandable.
limitations: limitations: - This paper focuses on lightweight structures, therefore, it is suggested that parameters and computations are listed in Table I for intuitive comparison.

- Since a 1-dimensional linear convolution chain is used instead of a cascaded SA. Therefore, Figure 7 should give a visualisation of the features in comparison with the transformer method.

- What is the role of extracting high-frequency boundary information? It is suggested to show its impact in ablation experiments.

- typo: L565, perfrm?