MappingFormer: Learning cross-modal feature mapping for visible-to-infrared image translation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 18:03:54

Abstract: Due to the limitations of infrared image acquisition conditions, many essential tasks currently rely on visible images as the main source of training data. However, single-modal data makes it difficult for downstream networks to show optimal performance. Therefore, converting the more easily obtainable visible images into infrared images emerges as an effective remedy to alleviate the critical shortage of infrared data. Yet current methods typically focus solely on transferring visible images to infrared style, while overlooking the crucial infrared thermal feature during cross-modal translation. To elevate the authenticity of cross-model translation at the feature level, this paper introduces a translation network based on frequency feature mapping and dual patches contrast, MappingFormer, which can achieve cross-modal image generation from visible to infrared. Specifically, the generator incorporates two branches: low-frequency feature mapping (LFM) and high-frequency feature refinement (HFR), both have embedded the Swin Transformer blocks. The LFM branch captures the fuzzy structural from visible images, while the HFR focuses on mapping edge and texture features. The extracted dual-branch frequency features undergo refinement and fusion through cross-attention mechanisms. Additionally, a dual contrast learning mechanism based on feature patch (DFPC) is designed to infer effective mappings between unaligned cross-modal data. Numerous experimental results prove the effectiveness of this method in cross-modal feature mapping and image generation from visible to infrared. This method holds significant potential for downstream tasks where infrared data is limited.

Main paper: 
 introduction: 
 In the domain of visual sensing technology, visible (VIS) image and infrared (IR) image constitute multimodal data that are frequently employed. VIS image provides rich textural details and geometric features, while IR image reveals the temperature distribution of objects and backgrounds [6, 27, 30] . However, due to the shooting limitations of infrared cameras or the absence of dependable and precise infrared simulation systems, many downstream applications struggle to collect adequate IR data to support model training, and cannot make downstream models perform well. These applications face a significant challenge of lacking IR image data [12] . Current research aimed at addressing IR data shortage mainly focuses on generating corresponding images from VIS. How to learn the mapping correlation between cross-modal data during training has become a crucial research question urgently needs to be solved [39, 41] .Currently, various solutions exist for cross-modal generation from VIS to IR images. One approach involves manually simulating the hot zone using physical models, but its efficiency and precision leave room for improvement [8, 25] . Alternatively, intelligent methods are widely used to embed the input VIS image into a latent feature space and subsequently reconstruct the corresponding IR image based on a nonlinear transformation relationship [9] . For example, using generative adversarial networks (GAN) [2, 14, 44] , unsupervised learning [21, 31] and self-supervised learning [10, 28] to transform the VIS domain into an IR style. While the above transformation methods can yield outputs resembling real IR images, they still overlook the conversion of the thermal features. Recent works based on contrastive learning [5, 11, 18, 23] aim to address these limitations, but a notable semantic gap persists between the VIS and IR domains, and these methods still require additional constraints. Moreover, numerous prevalent image translation methods focus more on the diversity of translation styles [1, 17] . When directly applied to VIS-to-IR conversion task, it is difficult to generate images that conform to infrared thermal features. Noting that the data collected in real-world scenarios are often unaligned, exhibiting differences in perspective or resolution, which increases the difficulty of cross-modal conversion.In this work, we propose a novel approach for VIS-to-IR image translation, exploring the correlations between multimodal images by mining the frequency feature, as shown in Figure 1 . Our motivation stems from the observation that despite VIS and IR features have modal specificity, they inherently contain shared representations to objects or features. Meanwhile, previous studies [16, 33, 46] have emphasized that mining frequency features and reconstructing images is maybe effective for image translation. Building upon thisVIS IRFigure 1 : Flow graph for classic style translation methods (left), and our MappingFormer (right). We explore the correlations between VIS and IR by mining the frequency feature, while classic method focusing on the style transfer.observation, we utilize cross-modal visual consistency and frequency priors to guide the translation between VIS and IR images. To achieve our goal, there are two issues we aim to address. The first is how to incorporate frequency features into the generative model efficiently. The self-attention and shifted window in Swin Transformer (Swin-T) [20] demonstrate remarkable perceptual ability and computational efficiency in feature extraction. Hence, integrating the Swin-T into the generator to process frequency domain information is worth investigating. Besides, the second issue is how to effectively impose constrains on cross-modal feature mapping. We postulate that incorporating the bidirectional contrastive learning into the constraints of feature mapping could enhance the quality of the asymmetric cross-modal translation.To ensure that IR images generated from VIS accurately reflect thermal laws, we introduce a network tailored from VIS to IR, MappingFormer. It comprises two generators and two discriminators, aiming to maximize the mutual information between input and output feature patches. First, the VIS image is decomposed into low and high-frequency components in the generator. Then, dual branches are devised for lowfrequency feature mapping (LFM) and high-frequency feature refinement (HFR) with embedded Swin-T. Cross-attention is employed to integrate the mapping information from both branches to generate more authentic infrared features. The LFM extracts fuzzy structural and color information, whereas HFR enhances edges and textures. Finally, by bidirectional contrast mechanism for feature patches, the mapping process of unaligned features is constrained. As such, our method diverges from general image translation methods, offering a specialized approach for cross-modal translation between VIS and IR images.In summary, our work makes the following contributions:• A cross-modal feature mapping network for VIS-to-IR (MappingFormer) is proposed, which ensures the translated results more related with the thermal features in infrared. To our knowledge, this is the first work that using frequency mapping to the VIS-to-IR translation. • Within the generator, two branches of low-frequency mapping and high-frequency refinement have been devised. And a dual contrast learning mechanism has been designed as the constrain in feature mapping process. • The experiment results clearly support the potential of our proposed method, confirming the applicability to cross-modal translation and downstream tasks. 
related work 2.1 visible to infrared translation: 
 Previous works have explored the translation from VIS to IR [2, 10, 13, 22, 28, 31, 38] or from IR to VIS [3, 15, 37, 40, 42] , which differs from image translation based on style transfer.Because IR images depict thermal textures, it is crucial to consider the mapping relationship between the two modal features when designing translation methods. Currently, many methods [14, 35, 44] rely on paired multimodal data for training to capture the thermal region features. However, this limits their generalization ability beyond the training dataset, making it challenging to achieve balanced results across different data domains, such as varying infrared wavelengths and imaging backgrounds. Additionally, some studies [19, 34] propose integrating thermal information into the model, utilizing segmentation or mask to highlight object features as salient areas of interest for the model. [7] uses infrared temperature information to guide the preservation of VIS details. The challenge with this method lies in accurately segmenting and simulating thermal region, as well as determining the appropriate weight of thermal features in output. Meanwhile, [13] proposes edge-guided method for multi-domain translation, emphasizing the retention of edge information and crucial details in generated IR images, but this focus sometimes leads to the neglect of structural features in vis images.Recently, image-to-image translation methods have rapid development. The majority of translation methods [5, 11, 16, 24, 29, 36, 45, 47] primarily focus on style transfer. Typically, they map images to latent spaces, perform specific conversions or comparisons, and then reconstruct into images. CUT [23] preserves the structural features of the source domain by employing patch contrast loss, whereas CycleGAN [47] incorporates cyclic consistency loss to ensure the structural coherence of the generated image. [4] explores the relationship between global features and instances through enforcing cross-granularity consistency. Nevertheless, convolution operations suffer from a limited receptive range, often resulting in blurring during high-resolution image translation. [24] utilizes autocorrelation regularization to facilitate zero-shot image translation based on diffusion models, but its generation domain is uncontrollable. Recent researches have focused on comprehending image features from the frequency and apply to image translation. [43] extracts the high-frequency band of images through discrete wavelet transform during knowledge distillation. [46] involves jointly learning the image features in both pixel and Fourier spectral spaces. [16, 33] utilize the Laplace pyramid to segregate images into high and low frequency components, translating contours in low-frequency and enhancing details with high-frequency. Although these methods have generated diversity, the feature differences in multimodal pose challenges for its direct application in VIS-to-IR translation. 
the proposed method 3.1 network architecture: 
 Given a VIS domain dataset {𝑉 } ∈ R 𝐻×𝑊 ×3 and an IR domain dataset {𝐼} ∈ R 𝐻×𝑊 ×1 under similar viewing angles, including VIS images 𝑣 ∈ 𝑉 and IR images 𝑖 ∈ 𝐼. Our goal is to learn the feature mapping that transforms the VIS domain into IR domain, i.e. 𝐺 : 𝑉 → 𝐼, and then produced the output image ̃︀ 𝑖 which adheres to the infrared thermal laws based on VIS image 𝑣. The translation process is denoted as follows:EQUATIONAs shown in Figure 2 , out proposed MappingFormer has two generators 𝐺𝑉 and 𝐺𝐼 , and two discriminators 𝐷𝑉 and 𝐷𝐼 . The generator 𝐺𝑉 learn the mapping from VIS to IR, whereas the discriminators 𝐷𝑉 assess authenticity. Upon input an image 𝑣 ∈ R 𝐻×𝑊 ×3 into the generator, the initial step is a Laplace decomposition to derive low-frequency 𝑣 𝑙 ∈ R 𝐻 2 × 𝑊 2 ×𝐶 and high-frequency components 𝑣 ℎ ∈ R 𝐻×𝑊 ×𝐶 . Then, both components are processed through the encoderdecoder, which embedded with the Swin-T blocks, getting a preliminary IR image generated. Meanwhile, identical steps are applied to real IR images 𝑖 ∈ R 𝐻×𝑊 ×1 to achieve corresponding outcomes ̃︀ 𝑣 ∈ R 𝐻×𝑊 ×3 . Perform feature patches encoding and space embedding on the dual mapping, followed by computing the corresponding mapping loss. Through network training, domain-transferred IR images ̃︀ 𝑖 are attained. Notably, while the training process need the involvement of both image modals, the inference relying solely on the generator 𝐺𝑉 to translation the VIS to IR.This section describes the feature mapping process within the generator. Inspired from [16, 33] , we use the Laplace image pyramid to decompose VIS image into low and high frequency. Then, the Swin-T blocks and encoder-decoder are used to extract and map the feature in two frequency components. Finally, the cross-attention in Swin-T is utilized to fusion the extracted features, resulting in generated images that more accurately reflect the real IR features, as shown in Figure 3 . Low-frequency feature mapping. The low-frequency component contains visual attributes like illumination and color within the VIS domain. During the translation to the IR domain, illuminations and color features are converted to grayscale values with varying luminosities, as well as the (𝑎) ∈ R 𝐻 2 𝑎 × 𝑊 2 𝑎 ×2 𝑎-1 𝐶 , 𝑎 = [1, 2, 3]. Then position encoding is added to reserve space information and express the positional relationships of features. Then, six Swin-T blocks are incorporated to efficiently extract the long-distance dependencies from the feature maps. After that, the feature maps are progressively decoded and concatenated, and the low-frequency components are reinstated to the feature maps with original resolution. Both the encoder and decoder are combining convolution with Leaky ReLU to adjust the feature channels, and the resolution of feature maps are adjusted by setting the step size.The low-frequency mapping branch based on Swin-T blocks efficiently integrates self-attention to the encoder-decoder. This enables the network to accurately capture more prominent feature regions within the low-frequency components. Specifically, the process for calculating low-frequency features in six Swin-T blocks is defined as follows:EQUATIONwhere (S)W-MSA and LN represent window self-attention computation and normalization layers, MLP represents multilayer perceptron, x𝑚 𝑙 and 𝑥 𝑚 𝑙 are the process variables of feature computation, representing the output of (S)W-MSA and MLP for the m-th block structure. Similar to previous work [20] , self-attention calculation is as follows:h w c   2 2 h w c   2 2 2 h w c   2 2 h w c   2 4 4 h w c   4 4 4 h w c   3 2 2 h w   2 2 h w c   2 4 4 h w c   4 8 8 h w c   8 8 8 h w c   4 8 8 h w c   2 2 h w c   … … … … Q K V W-MS(CEQUATIONwhere 𝑄, 𝐾, 𝑉 , and 𝑑 represent the query, key, value matrices, and their respective dimensions within the window. 𝐵 denotes the bias matrix. Furthermore, the low-frequency mapping branch has similarities to the network structure of U-Net [26] , where the encoder performs down-sampling, and the decoder handles up-sampling. Shallow-layer encoding captures more detailed visual features, whereas deep-layer encoding extracts richer local information. By concatenating these layers, efficient feature mapping is achieved in the low-frequency domain.High-frequency feature refinement. To reconstruct detailed information in the generated IR domain, such as texture and edges, we use high-frequency component for refinement and supplement. Given the component 𝑣 ℎ ∈ R 𝐻×𝑊 ×𝐶 of VIS domain, an encoder is used to perform down-sample, aligning its resolution with that of the low-frequency component. Subsequently, in line with the combined method of the Swin-T blocks described in the low-frequency, self-attention is calculated to identify salient regions within the high-frequency component. Following this, we perform cross-attention calculation based on the cross Swin-T for dual-branch features at a specific resolution 𝐻 2 × 𝑊 2 × 𝐶. In this process, the 𝐾 and 𝑉 values within the window originate from the low-frequency branch, while 𝑄 originates from the high-frequency branch, facilitating the fusion and refinement of feature sequences from different frequency domains. Finally, the output of crossattention undergoes decoding for up-sampling, resulting in the generation of a preliminary mapping image ̃︀ 𝑖 . The fusion of features can be considered as supplementing high-frequency features in low-frequency, thereby forming refined features and producing high-resolution translated images. The calculation of high and low frequency within the cross Swin-T can be described as follows:EQUATIONwhere (S)W-MCA represents crossing self-attention in window operation, t𝑛 ℎ and 𝑡 𝑛 ℎ are the output of (S)W-MCA and MLP in n-th block structure.Based on the CUT [23] , we introduce a dual contrastive learning mechanism based on feature patch. This serves as feedback and constraints on the generator's mapping process. The patch-based dual contrastive structure is shown in Figure 4 . We simultaneously apply this structure to bidirectional mapping, functioning as a dual contrast in MappingFormer. If a feature block is selected in the generated IR image, the module scouts for comparisons among multiple feature patches derived from the VIS input. It matches and associates the corresponding patches in the VIS image. Specifically, we initially use the patch embedding module to extract feature patches from both the input 𝑣 and output ̃︀ 𝑖 of generator 𝐺𝑉 , obtaining a stack of two feature patches. The patch embedding module consists of a four-layers convolutional encoder and a MLP projection head. Subsequently, when a query is chosen from the IR feature stack, the corresponding feature patches in the VIS are designated as positively correlated, whereas the non-corresponding ones are marked as negatively correlated. We map the query, positive, and negative samples onto K-dimensional vectors 𝑠, 𝑠 + ∈ R 𝐾 , 𝑠 -∈ R 𝑁 ×𝐾 respectively. To prevent spatial expansion or collapse, we normalize these vectors using L2 normalization, confining them to the unit sphere. Following this, we establish a (𝑁 + 1) directional classification problem and calculate the probability of positive samples surpassing negative ones. The mathematical definition is as follows:EQUATIONwhere 𝜏 is the scaling distance parameter and is set to a default of 0.07.The total loss of our method comprises the mapping contrastive loss ℒ𝑚𝑎𝑝𝑝𝑖𝑛𝑔, adversarial loss ℒ 𝑎𝑑𝑣 , and domain conversion loss ℒ 𝑑𝑜𝑚 . Mapping contrastive loss. We design a patch embeddingbased dual contrastive mechanism to perform patch encoding and spatial embedding for the mapping 𝐺𝑉 : 𝑉 → 𝐼 and 𝐺𝐼 : 𝐼 → 𝑉 . By selecting 𝐿 layers feature patches from the encoder and inputting them into a two-layer MLP projection head, we obtain a feature stack{𝑧 𝑙 } 𝐿 = {︀ 𝑀 𝐿𝑃 𝑙 𝑉 (𝐸𝑛𝑐 𝑙 𝑉 (𝑣)) }︀ 𝐿 ,where 𝐸𝑛𝑐 𝑙 is the encoded features of 𝑙-th layer. At this stage, every layer of features within the stack actually represents an image patch. The spatial positions of each layer can be denoted as 𝑝 ∈ {1, . . . , 𝑃 𝑙 }, where 𝑃 𝑙 is the number of spatial positions in each layer. Then we select a query, the corresponding positive sample patch is denoted as 𝑧 𝑝 𝑙 ∈ R 𝐶 𝑙 , and the remaining negative ones are denoted as 𝑧 𝑃 ∖𝑝 𝑙 ∈ R (𝑃 𝑙 -1)×𝐶 𝑙 , where 𝐶 𝑙 is the number of channels in each feature patch. Similarly, another feature stack {ẑ 𝑙 } 𝐿 = {︀ 𝑀 𝐿𝑃 𝑙 𝐼 (𝐸𝑛𝑐 𝑙 𝐼 (𝐺𝑉 (𝑥))) }︀ 𝐿 can be acquired.In order to match the positive sample patches between input and output, and evaluate the mapping effect of the generator, this section refers to the multi-layer contrast loss based on patch matching [23] as the mapping contrastive loss EQUATIONTotal loss. The total loss ℒ is calculated using the following combination:EQUATIONwhere 𝜆1, 𝜆2, and 𝜆3 are coefficients that respectively adjust the weight of mapping contrastive loss, adversarial loss, and domain conversion loss.  
experiments 4.1 datasets and details: 
 𝜆1 = 2, 𝜆2 = 1, 𝜆3 = 1.All experiments were conducted on a workstation with a 4090 GPU. We implemented and compared our method with advanced general image translation and VIS-to-IR translation methods, including CycleGAN [47] , CUT [23] , LPTN [17] , HnegSRC [11] , Swin-UNIT [16] , PUT [5] , and RGB-TIR [13] .Structural similarity (SSIM) is a metric to assess the likeness between generated and real images. The value closer to 1, the stronger resemblance between the generated and the real image. The peak signal-to-noise ratio (PSNR) quantifies the distortion between the generated and the real image. A higher PSNR indicates to lesser distortion in the generated image. The Frechet Inception Distance (FID) compares the feature distributions of real and generated images by computing depth features extracted using the Inception-V3 model. A lower FID value implies a closer distribution between the two images in the feature space. This paper normalized all the FID metrics. Furthermore, Section 4.6 assesses the performance of IR images translated by MappingFormer in downstream tasks, using precision, recall, and average precision (AP) under various intersection-over-union in object detection.The visualization of test results on three datasets is shown in Figure 5 , Figure 6 , and Figure 7 . Evidently, CUT performs well on the low-resolution traffic scenes in Figure 5 and Figure 6 . However, it causes ghosting shadow in the high-resolution scene shown in Figure 7 . We consider it might stem from inadequate constraints by the only one generator. Although CycleGAN and HnegSRC are capable of converting images from the VIS domain to IR, they often retain an excess of VIS features, thereby not adhering to the IR characteristics. PUT might introduce blurring, especially in Figure 6 and Figure 7 . Despite utilizing Laplace's high and low frequency components, LPTN still maintains the original color style and fails to translate images to the IR domain. RGB-TIR accurately highlights the heating area of the truck in Figure 6 , but its overall brightness is elevated and the generated texture features lack smoothness. Swin-UNIT's performance is suboptimal, presumably due to its two-stage approach involving global style translation and recurrent detail supplementation, which nonetheless falls short in capturing the essential thermal features of IR images. The visual results evident that our proposed method is superior in generating realistic IR thermal features while maximally preserving the intricate details in VIS images.Table 1 shows the quantitative evaluation of three datasets for VIS-to-IR translation. As evident from the table, our method achieved the best result in 7 indicators and achieved suboptimal in 1 indicator. Specifically, MappingFormer's SSIM metric on the AVIID-1, IRVI, and DroneCoast datasets were 0.86, 0.68 and 0.79, respectively. These results outperformed other advanced general image translation methods and VISto-IR translation methods, exceeding the suboptimal by 5%, 3% and 6%, respectively. The results strongly suggest that MappingFormer's frequency feature mapping, coupled with dual contrast learning, is suitable for VIS-to-IR cross-modal translation. Furthermore, Swin-UNIT shows promising performance, achieving suboptimal results in six metrics, which also proved the potential of applying frequency components to image generation.Effect of the network structure. We analyze the effect of various network components to the overall performance on AVIID-1 dataset. The results are shown in Table 2 . Initially, the baseline model, CUT, was trained using same settings. Afterward, we incrementally introduced LFM, HFR and DFPC to assess how these structures influence performance. Note that both the LFM and HFR branches incorporate Swin-T blocks. Model-1 and Model-2 integrated the LFM and HFR Model-4 + Cross Swin-T 0.858 0.547 branches, respectively, and the SSIM metric improved 4.5% and 5.8% compared to the baseline. Model-3 incorporates both LFM and HFR branches, yielding a substantial 10.5% SSIM improvement over the baseline. When all three modules are used concurrently, both SSIM and FID metrics exhibit a noteworthy enhancement. In conclusion, MappingFormer has achieved leading VIS-to-IR image translation effects. The design of the feature mapping branch is crucial, while the dual patch comparison also contributes meaningfully to the overall performance.Effect of the Swin transformer block. We explore the effect of embedding Swin-T blocks within feature mapping on AVIID-1 dataset. We incorporate block structures into one mapping branch while maintaining convolution structures in the other, and then contrast them with models that only rely on convolutional designs. Our findings indicate that embedding Swin-T blocks within the mapping branch yields higher quality images when compared to generators that are based solely on convolution structures. Note that each model fuses the outputs of high and low frequency branches, but only Model-5 uses cross Swin-T blocks during the fusion process, whereas other models rely solely on an accumulator for fusion. The detailed results are shown in Table 3 .This section aims to apply the outcomes generated by proposed MappingFormer to downstream object detection. For a quantitative evaluation, we trained and tested both our method and baseline network on the AVIID-1 dataset, obtaining images translated from the VIS domain to IR domain. We chose the latest Yolov9 [32] as our benchmark for object detection. To prevent data leakage, the IR images generated by two translation models serve as the training set for object detection respectively, while the unused real IR images constitute the test set. Detection results are calculated using the above training data. Additionally, we compare the detection 4 , and Figure 8 shows the detection visualization.Based on detection performance, directly training with VIS images and performing cross-domain detection yields poor results, lacking of generalization ability between the multimodal. Using IR images generated by MappingFormer as training data for detection achieves comparable performance compared to training directly with real IR data. Although our detection results do not surpass those trained on real IR images, this is reasonable because generative models can only simulate the distribution of real data to the best of their ability. Experiment results indicate that the feature distribution of the IR images generated by proposed method closely resembles that of real IR, making them effectively applicable to downstream tasks. 
conclusion: 
 In this paper, we propose a novel approach for image translation from VIS to IR. We have designed a specialized framework, MappingFormer, which integrates feature mapping and contrastive learning within a generator network for crossmodal image translation. Within the generator, two branches are designed: low-frequency feature mapping and highfrequency feature refinement, both of which are embedded with the Swin Transformer. Additionally, the dual contrastive structure based on feature patches serves as a constraint for mapping and generation. Experiment results indicate that the proposed method surpasses general image-to-image generation methods in both qualitative and quantitative evaluations, and outperforms the advanced VIS-to-IR translation methods. These experiments confirm the efficacy of feature mapping and dual contrastive learning. Future work could explore the temporal of video translation and unsupervised generation of multimodal images within VIS and IR domain.  
 
 Appendix:  


Decision: This paper initially got 2 accepts and 2 rejects, but all reviewers raised the scores after rebuttal. The authors have addressed the concerns from the reviewer.

Reviews: review: paper_summary: paper_summary: This paper presents an approach for image translation from VIS to IR. It designed a MappingFormer to integrate feature mapping and contrastive learning within a generator network for crossmodal image translation. Within the generator, two branches are designed: low-frequency feature mapping and high frequency feature refinement, both of which are embedded with the Swin Transformer. Additionally, the dual contrastive structure based on feature patches serves as a constraint for mapping and generation.
strength_weakness: strength_weakness: Experiment results indicate that the proposed method surpasses general image-to-image generation methods in both qualitative and quantitative evaluations. The paper also shows the extension results to object detection.
limitations: limitations: 1. How generalizable is the proposed VIS to IR conversion method? For example, in real-world scenarios, across datasets, etc.
2. What is the computational complexity and the number of parameters of the proposed method? It should be compared with existing methods.
3. I am not quite clear about what challenges in the image conversion task the proposed method addresses, as there are many cyclic conversion methods. What are its advantages compared to existing methods?
4. Can you explain the effectiveness of the proposed VIS to IR conversion method mathematically?
paper_summary: paper_summary: The authors introduce a translation network based on frequency feature mapping and dual patches contrast, Mapping Former, which can achieve cross-modal image generation from visible to infrared to alleviate the critical shortage of infrared data.  In this paper, a cross-modal feature mapping network is proposed for VIS-to-IR (MappingFormer) task. An important new idea is proposed, and its effectiveness is verified by experiments. However, further detailed comparative experiments, description of technical details and discussion of practical applications will enhance the integrity of the paper.
strength_weakness: strength_weakness: 1. MappingFormer improves the authenticity of cross-modal image translation through frequency feature mapping and two-patch contrast learning mechanism, which is a novel attempt in the field of visible light to infrared image conversion.
 
2. In the design of the network structure, they design a two-branch generator that includes low frequency feature mapping (LFM) and high frequency feature thinning (HFR). This structure helps to process different features of images separately and improve the translation quality. Swin Transformer blocks are embedded in the LFM and HFR branches, which helps to efficiently extract long distance dependent features and enhance model perception.
 
3. By contrast learning mechanism, the bidirectional contrast learning mechanism (DFPC) based on feature patches was designed to better learn the effective mapping between unaligned cross-modal data.
limitations: limitations: 1. In the contribution part, it is mentioned that it is the first work that applies frequency mapping to visible light to infrared conversion, but in the related work part, it is mentioned that recent studies mainly focus on understanding image features from frequency, which is not fully stated.
2.In the process of comparative experiments, experimental data show that the model has certain competitiveness, but it is not particularly outstanding. Can you further explain the performance of the model in terms of other evaluation indicators such as the number of parameters?
3.From the perspective of model complexity, the introduced double-branch structure and contrast learning mechanism may increase the complexity of the model, which may affect the training and reasoning efficiency of the model. Please add the explanation in this part.
4.In Figure 3, the W-MSA and W-MCA mentioned in this paper are not well shown. This is not intuitive enough for the reader. Please mark them in Figure 3.
5. In Extension to object detection section, only one method, CUT, is used for comparison. Vis-to-IR methods like Swin-UNIT, PUT, and RGB-TIR, are not mentioned in this experiment. 
6. In Equation 5, n is never mentioned in this section, which is ambiguous to the reader. Please provide an explanation of the meaning of this symbol after the introduction of the equation.
paper_summary: paper_summary: The paper addresses the challenge of limited infrared data availability by proposing a novel approach called MappingFormer for cross-modal image generation from visible to infrared. Current methods often fail to consider the crucial infrared thermal features during the conversion process, leading to suboptimal performance. MappingFormer leverages frequency feature mapping and dual patches contrast to enhance the authenticity of cross-modal translation at the feature level. The model consists of a generator with two branches - low-frequency feature mapping (LFM) and high-frequency feature refinement (HFR) - both utilizing Swin Transformer blocks. The LFM branch captures structural features from visible images, while the HFR branch focuses on edge and texture features. The refined features are fused through cross-attention mechanisms, and a dual contrast learning mechanism based on feature patches (DFPC) further enhances the alignment between unaligned cross-modal data. Experimental results demonstrate the effectiveness of MappingFormer in cross-modal feature mapping and image generation, showing promise for downstream tasks with limited infrared data availability.
strength_weakness: strength_weakness: 1. MappingFormer excels in preserving the authenticity of cross-modal translation at the feature level by incorporating frequency feature mapping, dual patches contrast, and Swin Transformer blocks. This enhances the accuracy and quality of generated infrared images from visible inputs.
2. The dual-branch generator design, with LFM and HFR branches focusing on different aspects of feature extraction, allows for a comprehensive representation of visible images that facilitates accurate mapping to infrared images. The fusion of these features through cross-attention mechanisms enhances the translation process.
3. The introduction of the dual contrast learning mechanism based on feature patches (DFPC) addresses the challenge of aligning unaligned cross-modal data, improving the effectiveness of feature mappings between visible and infrared images.
4. The paper provides comprehensive experimental results that demonstrate the effectiveness of MappingFormer in cross-modal feature mapping and image generation. This empirical evidence supports the potential of the proposed method for tasks requiring the translation of visible images to infrared in scenarios where infrared data is scarce.
limitations: limitations: 1. The practical applicability and scalability of MappingFormer in real-world scenarios with diverse infrared image acquisition conditions are not discussed.
2. While the proposed model architecture appears to be innovative and effective, the abstract does not address the computational complexity or resource requirements associated with implementing MappingFormer.
paper_summary: paper_summary: This article mainly introduces a cross-modal feature mapping method for visible images to infrared image conversion, namely MappingFormer, which achieves feature mapping by mining frequency features between image domains.The Swin Transformer is used to process high-frequency and low-frequency features separately, and the extracted dual-branch frequency features are refined and fused through a cross-attention mechanism. A dual-contrast learning mechanism (DFPC) based on feature blocks is designed to infer effective mappings between unaligned cross-modal data to improve the mapping effect.
strength_weakness: strength_weakness: 1. This article is clearly illustrated and written, making it easy for readers to understand the main ideas and structure.
2. The experimental results of this paper are sufficient to provide strong experimental evidence for the effectiveness of MappingFormer in implementing VI to IR.
3. Under the condition of limited infrared detectors, this paper proposes an effective remedy with certain practical value.
limitations: limitations: 1. This paper is just a combination of several existing proposed algorithm modules, with very few new proposed works. It lacks novelty and true ideological innovation.In my humble opinion,the main idea of the article seems to be a combination of LPTN and ROMA, with only the novel idea of decoupling and merging frequency-domain features using Swin-T. However, this single idea is not sufficient to justify the value of the entire article.
2. The article's design explanation for some structures is not complete, such as why only one encoder is used for high-frequency features. Although the shallow features have better original structures, how many layers are used is obtained through experimental testing or defined independently based on previous work experience. Please provide an appropriate explanation.