SIRLUT: Simulated Infrared Fusion Guided Image-adaptive 3D Lookup Tables for Lightweight Image Enhancement

Authors: kaijiang  li
hao  li
haining  li
peisen  wang
chunyi  guo
wenfeng  jiang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 02:53:04

Abstract: Researchers have applied 3D Lookup Tables (LUTs) in cameras, offering new possibilities for enhancing image quality and achieving various tonal effects. However, these approaches often overlook the non-uniformity of color distribution in the original images, which limits the performance of learnable LUTs. To address this issue, we introduce a lightweight end-to-end image enhancement method called  Simulated Infrared Fusion Guided Image-adaptive 3D Lookup Tables (SIRLUT). SIRLUT enhances the adaptability of 3D LUTs by reorganizing the color distribution of images through the integration of simulated infrared imagery. Specifically, SIRLUT consists of an efficient Simulated Infrared Fusion (SIF) module and a Simulated Infrared Guided (SIG) refinement module. The SIF module leverages a cross-modal channel attention mechanism to perceive global information and generate dynamic 3D LUTs,  while the SIG refinement module blends simulated infrared images to match image consistency features from both structural and color aspects, achieving local feature fusion. Experimental results demonstrate that SIRLUT outperforms state-of-the-art methods on different tasks by up to 0.88 $\sim$ 2.25dB while reducing the number of parameters. Code is available at \href{https://github.com/riversky2025/SIRLUT.git}{https://github.com/riversky2025/SIRLUT}.

Main paper: 
 introduction: 
 When taking photos with a camera, overexposure, underexposure, backlit conditions, and other shooting conditions may cause loss of highlight details, unclear shadow details, color distortion, or dark subjects. With the rapid development of deep learning, many algorithms [11] [12] [13] 24] based on deep neural networks have been applied to image enhancement tasks, particularly in addressing these issues, achieving satisfactory performance. Compared to traditional methods [1, 18, 27] , deep learning-based approaches improve performance while simplifying the processing pipeline. However, these methods utilize complex network inference for dense pixel mapping, which not only consumes high computational and storage resources but also fails to meet lightweight requirements and ensure efficient operations on camera hardware.Researchers utilize static 3D LUTs for color mapping and simplifying the inference process. Based on the RGB color space and trilinear interpolation, 3D LUTs achieve more accurate color adjustments, effectively improving the quality and effectiveness of image enhancement [7, 15, 34, 36, 40, 41] . Zeng et al. [38] first propose learnable 3D LUTs, achieving image enhancement by generating adaptive 3D LUTs based on image features. However, the uneven pixel distribution in the original image reduces the pixel utilization rate of the learnable 3D LUTs during the sampling process. Zhang et al. [41] propose an efficient hash-based 3D LUTs method called HashLUT. Building on the concept of hashing, they further develop a fast and lightweight image enhancement network. Nevertheless, Figure 1 : PSNR results v.s the size of parameters of different methods for image enhancement on MIT-Adobe FiveK dataset [2] . Our method achieves the optimal results with the smallest parameters.the problem of limited feature extraction ability of the learnable 3D LUTs model due to uneven pixel distribution in original images has not been solved.To balance the pixel value distributions of the original image, several methods increase feature components and adjust sampling methods to address the problem of uneven pixel distribution [5, 36, 37] . SepLUT [37] improves the utilization of 3D LUTs units by supplementing the ability of mixed color components with 1D LUTs separated from color components. AdaInt [36] improves the performance of 3D LUTs by introducing image adaptive sampling interval to learn non-uniform 3D LUTs layout. These methods achieve equalization of the pixel distribution of the original image features to a certain degree. The performance of previous state-ofthe-art methods is partially shown in Figure 1 .In order to fundamentally solve the problem of uneven pixel distribution in the original images, we introduce the simulated infrared modality into the process of color mapping learning for adaptive 3D LUTs, aiming to obtain a more uniform pixel distribution. As shown in Figure 2 , the pixel distribution of the original images is often concentrated due to the influence of the camera, which limits the effectiveness of the adaptive 3D LUTs learning ability and impairs network performance. In contrast, simulated infrared images reflect intensity information at the pixel level. Their pixel distribution is uniform, and their structural features can also serve as semantic-level features to enhance network performance. Specifically, we design a method called Simulated Infrared Fusion Guided Image-adaptive 3D Lookup Tables (SIRLUT). SIRLUT consists of a Simulated Infrared Fusion (SIF) dynamic 3D LUTs and a Simulated Infrared Guided (SIG) image refinement module. SIF utilizes a cross-modal channel attention mechanism to perceive global information and generate dynamic 3D LUTs. To address the problem of inconsistent local details in the image after enhancing with 3D LUTs, SIG refines the image from both structural and color perspectives using simulated infrared images. The combination of global adjustments and local refinement modules enables SIRLUT to achieve lightweight and efficient image enhancement.  
related work 2.1 image enhancement: 
 Deep learning-based image enhancement networks can be divided into two categories. The first category refers to pixel-to-pixel image enhancement networks based on the U-shape architecture [29, 35, 42] . These methods have large computational overheads and are difficult to embed in devices such as cameras. The second category [14, 24] combines neural networks with physical methods, using color mapping models to achieve lightweight and fast image enhancement. Typical color mapping models include affine transformations [8, 32] , curve-fitting functions [9, 17, 24] , and 3D Lookup Tables [36] [37] [38] . In recent years, researchers have proposed various LUT-based image enhancement methods [4, 7, 15, 34, 36, 40, 41] . and uses a CNN to predict the weights of multiple 3D LUTs. Subsequently, these weights are merged into an adaptive 3D LUT for image enhancement. However, the uniform sampling strategy used in this method [38] limits the expressive power of 3D LUTs. AdaInt [36] performs dense sampling in highly nonlinear color ranges and sparse sampling in nearly linear color ranges, enhancing the expressive power of 3D LUTs. Some researchers also optimize the lightweight nature of 3D LUTs themselves. HashLUT [41] utilizes an efficient hash-based form to adaptively learn HashLUT, handling hash conflicts to solve the memory consumption issue of 3D LUTs in image enhancement. However, the parameter memory consumption inside 3D LUTs is not significant, and few studies focus on optimizing the weight prediction parameters of 3D LUTs. We significantly optimize the network for predicting weights by combining simulated infrared modality to enrich pixel distribution information and cross-modal channel attention.In recent years, multimodal fusion has received extensive research attention in the field of image processing [22, 23, 28, [43] [44] [45] . Cong et al. [5] achieve high-resolution image harmonization by introducing semantic foreground and background information and using their structural information to guide pixel-level optimization. However, the information provided by semantic segmentation modality is limited and cannot provide effective pixel distribution and texture features. Some scholars [19, 23] use more detailed features provided by infrared images to perform image processing tasks. These methods fuse visible light images with infrared images, retaining the thermal radiation intensity features of infrared images and the fine details of visible light images, thereby producing clear highbrightness targets and rich image details. RTFNet [30] integrates visible light images and thermal infrared information, fully utilizing the advantages of thermal imaging cameras in various lighting conditions, and achieves accurate semantic segmentation tasks in complex urban scenes. Wang et al. [31] introduce a novel method to generate cross-modal paired images for RGB-IR Re-ID tasks, solving the differences between visible light images and infrared images and improving the accuracy of person re-identification. Therefore, infrared images are crucial for visual tasks. In image enhancement tasks, simulated infrared images have uniform pixel distribution and rich structural texture, eliminating noise introduced by strong infrared penetration compared to images captured by real infrared cameras. Our work introduces simulated infrared images to achieve a more efficient multi-modal 3D LUTs image enhancement method. 
method 3.1 overall architecture: 
 As shown in Figure 3 , SIRLUT consists of a learnable fusion of simulated infrared fusion 3D LUTs network and a simulated infraredguided refinement. The former comprises a weight prediction backbone network for cross-modal fusion features and a trilinear interpolation color mapping module for 3D LUTs, which captures contextual information through spatial transformation and cross-modal channel attention mechanism. The latter consists of a multi-modal structure extraction and a refinement module for local multi-modal feature fusion synergy. In this section, we will discuss 3D LUTs and trilinear interpolation, 3D LUTs with simulated infrared fusion, simulated infrared guided refinement, and loss function. . Given the value of D, the RGB color space is uniformly discretized into the grid G, and different 3D LUTs have different color conversion RGB outputs, which is the learnable parameter of the proposed method, and when D=33, the number of parameters for a single 3D LUTs is 107.8k(3𝐷 3 ). Color mapping via 3D LUTs is actually achieved through two basic operations: positioning and trilinear interpolation. Given the input RGB color 𝑟 (𝑖,𝑗,𝑘 ) , 𝑔 (𝑖,𝑗,𝑘 ) , 𝑏 (𝑖,𝑗,𝑘 ) , the first step is to index and locate the position coordinates (𝑖, 𝑗, 𝑘) of the RGB color in the grid G, The specific description is as follows:EQUATIONEQUATIONwhere 𝐶𝑜𝑙𝑜𝑟 𝑚𝑎𝑥 represents the maximum color value, and 𝑙 represents the sampling interval of the sampling grid G. After locating the input RGB color in grid G position, its adjacent 8 element points can be used to interpolate the color conversion output values 𝑟 ′ (𝑖,𝑗,𝑘 ) , 𝑔 ′ (𝑖,𝑗,𝑘 ) , 𝑏 ′ (𝑖,𝑗,𝑘 ) through trilinear interpolation. Assuming point P is located at a certain position in the RGB color grid G, the eight adjacent vertices of point P are {𝑃 0 , 𝑃 1 , ..., 𝑃 7 }. The formula for calculating trilinear interpolation is as follows:EQUATIONwhere 𝐺 (𝑃 𝐾 ) , 𝑘 ∈ [0, 7] represent the RGB color values of point𝑃 𝑘 in grid 𝐺. 𝑑 𝑟 , 𝑑 𝑔 , and 𝑑 𝑏 are the corresponding channel interpolation weight coefficients, respectively, and 𝐶 ′ (𝑖,𝑗,𝑘 ) represents the color conversion mapping result.Deep neural network blocks initially train the learnable simulated infrared fusion 3D LUTs on n-base 3D LUT blocks. Then, the dynamic LUT corresponding to the original image is rapidly inferred using n weights and n base LUTs. Finally, the dynamic LUTs are used to compute pixel values for the image on a per-pixel using trilinear interpolation. Weight prediction backbone network.To ensure the efficiency of the model, we propose a weight prediction backbone network, which incorporates a cross-modal channel attention mechanism to capture global information and generate dynamic 3D LUTs. This module allows the attention mechanism to operate at lower resolutions, effectively reducing computational complexity. We extract global features from the original image and simulated infrared image through three steps. First, we compress and transform image features using spatial-to-channel transformation. Next, we extract contextual features using the cross-modal channel attention mechanism. Finally, in the third step, we restore features using channel spatial transformation.First, we align the resolutions of the original image 𝐼 𝑣 and the simulated infrared image 𝐼 𝑟 by utilizing the resize and spatial information rearrangement mechanism, resulting in the corresponding features 𝐹 𝑣 and 𝐹 𝑟 . The specific description is as follows:EQUATIONwhere 𝜆 represents the conversion ratio used during the spatial transformation. Then, to reduce computational complexity and ensure the efficiency of the network feature extractor, we use a linear transformation with shared parameters to compress the number of features in 𝐹 𝑣 and 𝐹 𝑟 by a factor of 𝜇. The description is as follows:EQUATIONwhere 𝐹 𝑒 𝑣 and 𝐹 𝑒 𝑟 respectively represent the features after compression for 𝐹 𝑣 and 𝐹 𝑟 . Then, we design a multi-head attention mechanism guided by simulated infrared for fused feature extraction. We perform a preceding channel expansion for 𝐹 𝑒 𝑣 and 𝐹 𝑒 𝑟 through linear transformations for multi-head attention. The specific procedure is described as follows:EQUATIONwhere 𝑚 represents the number of heads in the multi-head attention.We obtain the final query 𝑄 𝑣 by performing self-attention on the features of the original image, and self-attention is applied to the features of the simulated infrared image to obtain 𝐾 𝑟 and 𝑉 𝑟 . Then, we conduct cross-channel attention feature extraction using 𝑄 𝑣 , 𝐾 𝑟 , and 𝑉 𝑟 . The calculation of attention is as follows:EQUATIONwhere 𝜉 is a learnable parameter. Then we extract features by sequentially applying self-attention to the original image and crossmodal channel attention. The cross-modal multi-head channel attention is described as follows:EQUATIONwhere 𝑚 represents the number of heads in the multi-head attention. Then, 𝑍 is obtained by aggregating the parameters using adaptive average pooling, resulting in 𝐹 𝑎𝑣𝑔𝑝𝑜𝑜𝑙 ∈ R 𝐶,2,2 . Subsequently, a linear transformation with a stride of 2 is applied to convert the last two dimensions into one-dimensional features. Finally, a fully connected layer is used to transform the squeezed image into the weight matrix 𝑊 ∈ R 𝑛 for the 3D LUTs.EQUATIONwhere 𝑛 represents the number of 3D LUTs base blocks. The length of the predicted weight vector 𝑊 is also 𝑛.We define n learnable LUT base blocks 𝐺 𝑛 ∈ R 𝐷 ×𝐷 ×𝐷 . The table lookup process for the LUT is described in Section 3.2. The LUT mapping for input image 𝐼 𝑙 is as follows:EQUATIONwhere 𝐼 𝑙 represents the image after mapping conversion.After the aforementioned processing by the SIF module, the enhanced image still fails to adequately capture local features. Therefore, we design the SIG module. As shown in Fig. 4 , the SIG module achieves the capture of local features by extracting structurally consistent features between the simulated infrared image and the original image, as well as performing local feature fusion. The specific description is as follows:We adopt shared parameter dilated convolution to learn the structural consistency between 𝐼 𝑣 and 𝐼 𝑟 , and utilize the fused features to refine the pixels. The extraction of structural consistency between 𝐼 𝑣 and 𝐼 𝑟 is described as follows:EQUATIONwhere 𝐺 𝑣 and 𝐺 𝑟 represent the features after structural alignment. Then, we perform single-pixel convolution on the concatenated features of 𝐺 𝑣 and 𝐺 𝑟 , followed by residual refinement with the activated and normalized 𝐼 𝑙 . Subsequently, we apply a single-pixel convolution with a kernel size of 1 to the fused features. Next, we apply the LeakyReLU activation function for non-linear processing of the outputs and normalize them using InstanceNorm2d. The specific description is as follows:EQUATIONwhere 𝐼 𝑒 represents the final enhanced image, and Φ denotes the cascaded use of LeakyReLU and InstanceNorm2d.The overall framework can be trained end-to-end. Our training loss function is defined as follows:EQUATIONwhere L 𝑟𝑒𝑐𝑜𝑛 denotes the mean squared error (MSE) loss used as a reconstruction loss. According to [38] , the smoothing term L 𝑠 and the monotonicity term L 𝑚 serve as regularization terms for the constraint of the look-up table (LUT). 
experiments 4.1 datasets and implementation: 
 Datasets for Photo Retouching and Tone Mapping. We evaluate the effectiveness of SIRLUT on the publicly available MIT-Adobe FiveK dataset [2] and the PPR10K dataset [20] . To verify our proposed SIRLUT, we expand the dataset using existing simulated infrared image generation models [6] . The MIT-Adobe FiveK dataset [2] [20, 36] . Our research involves photo retouching and tone mapping [39] . We utilize sRGB images and their corresponding simulated infrared images to evaluate the photo retouching task on the MIT-Adobe FiveK [2] and PPR10K datasets [20] . For the tone mapping task, we convert 16-bit CIE XYZ format images to 8-bit sRGB format and perform performance validation on the MIT-Adobe FiveK dataset [2] . We adopt evaluation standards previously used in research [38] , including PSNR, SSIM, and the L2-distance in CIE LAB color space (△𝐸 𝑎𝑏 ). On PPR10K [20] , we also include the human-centered evaluation metrics (denoted by the "HC" superscript).Implementation and Training Details. We conduct experiments on the RTX 3080Ti graphics processing unit using the Py-Torch environment [26] . For both datasets, we set the spatial transformation ratio 𝜆 to 4 and the feature compression factor 𝜇 to 8.During training, we employ the same data augmentation settings as in [20] , and we train the model with a batch size of 1. The training epochs for the MIT-Adobe FiveK and PPR10K datasets are set to 300 and 100 respectively. We use the Adam optimizer [16] , set the initial learning rate to 2 × 10 -4 , and gradually decrease it to 1 × 10 -5 using the cosine annealing strategy [21] .We compare our method with state-of-the-art photo enhancement methods. To verify the efficiency of the SIRLUT, we select a large number of methods based on 3D LUTs. For a fair comparison of methods based on 3D LUTs, we set the 𝑁 𝑠 of LUTs to 33 and the number of basic blocks 𝑀 to 3.Quantitative Comparison. We quantitatively compare stateof-the-art methods using the PSNR, SSIM, and △𝐸 𝑎𝑏 evaluation metrics. Table 1 displays the comparison results for photo retouching task on the MIT-Adobe FiveK dataset [2] . RSFNet [25] adoptsSepLUT RSFNet SIRLUT GroundTruth AdaInt Table 1 : Quantitative comparisons on the MIT-Adobe FiveK dataset [2] for photo retouching. "/" means the results are absent in the original paper. The best performance is indicated by the color red, while the color blue indicates the second-best performance. Please note that the visualization is optimized for color viewing.Param. PSNR↑ SSIM↑ △𝐸 𝑎𝑏 ↓ UPE [32] 927.1K 21.88 0.853 10.8 DPE [3] 3.4M 23.75 0.908 9.34 3D LUT [38] 593.5K 25.29 0.923 7.55 SA-3D LUT [33] 4.5M 25.50 / / SepLUT [37] 119.8K 25.47 0.921 7.54 HashLUT [41] 114.0K 25.50 0.926 7.46 3D-LUT+AdaInt [36] parallel region-specific filters, which may limit the model's ability to perform complex and fine adjustments to the image, resulting in poor image enhancement. SepLUT [37] improves the use of 3D LUTs, but in some cases, sub-transformations independent of components may lead to information loss or incompleteness, introducing errors. HashLUT [41] employs an approximation mapping method, which cannot accurately reflect the relationship between original data, and compared with traditional 3D LUTs, the nonlinear transformation effect is not accurate enough, resulting in limited enhancement effect. AdaInt [36] alleviates the problem of uneven pixel value distribution in the original image, but does not fundamentally supplement the additional pixel distribution information required. We supplement the pixel distribution information of the original image using simulated infrared image with uniformly distributed pixels and obvious semantic features. It is worth noting [20] . It can be found that the proposed method achieves better performance than other methods. The error maps are placed at the top-right of each image. It is best viewed on a screen. Table 3 : Quantitative comparison of tone mapping on the MIT-Adobe FiveK dataset [2] . The best performance is indicated by the color red, while the color blue indicates the second-best performance. Please note that the visualization is optimized for color viewing. [20] and tone mapping task on the MIT-Adobe FiveK dataset [2] . Qualitative Comparison. We conduct a qualitative analysis of state-of-the-art methods on the MIT-Adobe FiveK [2] and PPR10K datasets [20] . As shown in Figure 5 , our method produces more satisfactory results in the photo retouching task on the MIT-Adobe FiveK dataset compared to other methods. Specifically, in the first row, The performance of RSFNet appears unstable under overexposed or low-light conditions due to the adoption of parallel region-specific filters. Our method effectively addresses the problem of underexposure. In the second row, while SepLUT and AdaInt improve the unit utilization rate of 3D LUTs by separating color components and non-uniform interval sampling, they fail to fundamentally solve the problem of uneven pixel distribution in original images, resulting in discontinuous transitions in smooth areas. Our method better handles image shadows caused by uneven illumination. In the third row, RSFNet, SepLUT, and AdaInt exhibit poor handling of the reflection gloss of street lamps and building windows in close-up shots, demonstrating the superiority of our method in dealing with backlit shooting scenes. In the last row, the results obtained by the other methods are dark and lose many details, while our method preserves more texture details of buildings. As shown in Figure 6 , the results on PPR10K demonstrate the efficiency of our method in handling portraits with background blurring and depth of field effects, and we observe that our method retains more background details, even when the background of the portrait is complex.In this section, we conduct several ablation experiments on the photo retouching task on the MIT-Adobe FiveK dataset [2] . Simulated infrared images. To validate the effectiveness of simulated infrared images in SIRLUT, we gradually incorporate SIF and SIG into the 3D LUTs baseline on the MIT-Adobe FiveK dataset [2] . SIF refers to the fusion of simulated infrared modality during the LUT enhancement process, while SIG pertains to the introduction of simulated infrared modality during the refinement process. As shown in Table 4 , the first row displays results without using simulated infrared modality, the second and third rows show results with the introduction of SIF and SIG modules respectively, while the last row displays the results obtained from the joint use of SIG and SIF. It is evident that the introduction of only the SIF module significantly enhances the model's ability to extract global features and handle image details. While using only the SIG refinement module also improves the model's performance, the effect is not as significant as the SIF module. However, due to the complementary nature of SIF and SIG, SIRLUT exhibits significant performance improvements in both color feature mapping 3D LUTs network and image refinement.Simulated infrared fusion strategy. To evaluate the impact of the simulated infrared modality on the dynamic prediction weights of 3D LUTs, we conduct a comprehensive analysis of different fusion strategies on global feature extraction. We compare the influence of fusion methods based on linear transformation (CNN), spatial attention, and SIF on the model. Due to the small local receptive field of CNN, it cannot effectively capture global features. The spatial attention mechanism focuses on per-pixel feature extraction but often overlooks global context, resulting in high computational complexity and resource consumption. As shown in Table 5 , it can be observed that SIF performs the best in both algorithmic model and enhancement effects, with the smallest number of parameters. This is because the simulated infrared modality as a complement to pixel value distribution information can greatly enhance the unit utilization rate of 3D LUTs. Additionally, the dynamic weight prediction guided by cross-modal channel attention avoids longrange dependencies of receptive fields and a large number of matrix operations in the global feature extraction process.Spatial transformation ratio and feature compression factor. In the structure of our SIF module, we integrate a strategy that combines spatial dimension transformation with channel dimension compression. This approach aims to minimize the computational cost while improving the operational efficiency of the model. As shown in Figure 7 , spatial transformation plays an important role in adjusting the size of the model's parameters and improving its performance, compared to the adjustment factors of lambda (𝜆) and feature compression (𝜇). When 𝜆 is set to 8, an augmented spatial transformation ratio may impair the resolution of the channel attention mechanism in the context feature extraction process. Even with 𝜇 adjusted to 8, the performance does not reach the expected level. Nonetheless, setting 𝜆 and 𝜇 to 4 and 8, respectively, allows the model to achieve an optimal balance between parameter efficiency and performance. Employing this configuration, our method attains remarkable outcomes, where the PSNR reaches 27.25dB, SSIM is 0.942, and the △𝐸 𝑎𝑏 value stands at 6.19. 
conclusions: 
 In this paper, we propose a novel 3D LUT-based multi-modal image enhancement network called SIRLUT, which integrates simulated infrared fusion 3D LUTs and simulated infrared guided refinement to address the uneven pixel distribution in the original images. The SIF module achieves efficient 3D LUTs through spatial transformation and cross-modal channel attention mechanism. Meanwhile, the SIG module merges structurally consistent features from both simulated infrared images and the original images, incorporating local feature fusion for further refinement of the images. The combination of these two modules results in our method exhibiting exceptional performance and efficiency. Our proposed method not only achieves a significant visual enhancement effect but also is suitable for integration into both software and hardware devices such as cameras, mobile phones, and image processing software. 
 
 Appendix:  


Decision: This paper received two borderline rejects, one borderline accept, and one weak accept. Reviewers FoSn, gbDu and PCBx all acknowledged the novelty of this work. Though the experiments are not sufficent and more methods should be compared, meta reviewer believes the novelty weighs more than extensive comparisons. Based on the innovation of this work, meta reviewer made the decision.

Reviews: review: paper_summary: paper_summary: The paper introduces SIRLUT, an advanced method for image enhancement utilizing simulated infrared fusion and 3D Lookup Tables (LUTs). SIRLUT addresses the challenge of uneven pixel distribution in original images by integrating simulated infrared imagery to reorganize color distribution, enhancing the adaptability and performance of 3D LUTs. It consists of two main modules: the Simulated Infrared Fusion (SIF) module, which captures global information using cross-modal channel attention, and the Simulated Infrared Guided (SIG) refinement module, which improves local detail consistency. Experimental results demonstrate SIRLUT's superiority over state-of-the-art methods, achieving significant improvements in image quality with fewer parameters.
strength_weakness: strength_weakness: 1. This paper pioneeringly proposes the use of simulated infrared images to supplement pixel distribution information in 3D LUT-based image enhancement.
2. The SIF and SIG modules proposed in this paper efficiently integrate information from simulated infrared and original RGB images, enabling SIRLUT to achieve excellent results with a smaller number of model parameters, thereby enhancing the feasibility of deployment on camera devices.
3. The ablation experiments and comparative experiments in this paper are comprehensive, validating the effectiveness of each module in the proposed model and demonstrating its superior performance.
limitations: limitations: 1. Although the paper focuses on improving the performance of 3D LUT-based image enhancement methods by using models that integrate simulated infrared images, I believe that the comparative experiments should not be limited to 3D LUT-based methods. For example, other image enhancement methods utilizing simulated infrared images mentioned in the related work section, such as [19] and [23], should also be included.
2. Using a 3D LUT method that integrates simulated infrared images introduces an additional input from the simulated infrared image. Therefore, the time and space overhead required to obtain the simulated infrared image from the original image should not be overlooked. When comparing model parameters, inference latency, and memory consumption with other methods, the authors should comprehensively consider these factors.
3. A minor issue with this paper: What does the small image in the top right corner of each result image in Figures 5 and 6 represent? This small image covers the detail information of the original image. Could it be placed elsewhere? If it is not very important, it might be better to remove it.
4. Within the SIF module, it is recommended to conduct an ablation experiment: remove the self-attention mechanism for 𝐼𝑣 , allowing the upper branch to directly pass 𝐼𝑣  through Pix-U Block and Conv Block to obtain 𝑄𝑣, and then perform cross-attention with the lower branch. This experiment can help identify potentially redundant modules, thereby reducing the time and space complexity of the model's computation.


I think highly of your response and will consider increasing your rating based on your response.
paper_summary: paper_summary: - SIRLUT is a multimodal image enhancement network that utilizes simulated infrared modality to enhance 3D LUT. By leveraging pixel distribution information, it effectively extracts global features (via SIF) and improves spatial fidelity (via SIG), thereby enhancing the details in the processed images.
strength_weakness: strength_weakness: - By leveraging the pixel-level details information provided by the simulated infrared modality, SIRLUT enhances the perceptual quality of the images.
limitations: limitations: - The strength and weakness of the paper lie in the fact that the most significant performance improvement comes from using an infrared dataset generated by another method as an additional input for training. This contribution seems somewhat weak as a main contribution.
- It appears that the main improvement comes from the infrared image's uniformity compensating for the non-uniformity of the RAW data. However, the paper lacks an in-depth analysis of the distribution mismatch. I was curious about how the mismatch was utilized or overcome, but the paper merely states that SIG learned spatial fidelity. The role of pixel unshuffling, which might have had a normalizing effect, is not discussed at all.
- I spent quite some time trying to understand what "Linner-laysers" in Fig. 3 referred to. Both words were misspelled, making it hard to comprehend, which is critical for a figure explaining the main architecture.
- The reduction in parameters does not seem substantial enough to emphasize efficiency. It would be better to consider improvements in speed and memory efficiency as well.
paper_summary: paper_summary: This paper proposes SIRLUT, a method utilizing a simulated infrared image fusion strategy and 3D lookup table for lightweight image enhancement. Specifically, the authors designed the Simulated Infrared Fusion (SIF) module and the Simulated Infrared Guided (SIG) refinement module to fully exploit the additional information from simulated infrared images, extracting consistent features in structure and color. The enhanced images exhibit a wider pixel distribution, presenting better visual effects and achieving significant improvements in quantitative metrics. The authors also conducted ablation studies to demonstrate the effectiveness of the proposed modules.
strength_weakness: strength_weakness: 1. The paper innovatively proposes an image-adaptive 3D LUT generation method based on infrared image fusion and an infrared image-guided image reconstruction method. This method leverages information from simulated infrared images to complement the original visible light images, thereby enhancing image enhancement effects. In the simulated infrared image fusion module, the authors reasonably utilized the attention mechanism and provided detailed pipeline explanations.
2. The proposed new method achieved substantial improvements (up to 0.88~2.25dB) compared to the state-of-the-art methods, and the authors provided appropriate explanations for the principle of model performance improvement with simulated infrared images.
limitations: limitations: 1. There may be a lack of discussion regarding the practical application scenarios of the proposed method. For instance, previous methods (such as 3D-LUT and SepLUT) have demonstrated near real-time model efficiency, whereas this paper lacks a discussion on the operational efficiency after introducing more complex network modules. Additionally, the authors mentioned that the simulated infrared images were synthesized using an infrared image synthesis method based on visible light images, implying that additional infrared images need to be introduced when applying this method. It is worth further discussing whether this limitation will affect the practical application of the proposed method.
2. The details of the paper may need further refinement. For example, in Figures 5 and 6, there are some smaller sub-images in the upper right corner of the images, and the authors did not seem to explain the meaning of these sub-images, which may affect the understanding of researchers unfamiliar with this research field. Additionally, there are some typographical errors in the paper, such as "Linner-laysers" in Figure 3, which seems to be "Linear-layers" based on the context, and the paragraph heading "Quatitative Comparison" around line 683, which should be "Quantitative Comparison." The authors are advised to carefully review the manuscript to correct these errors.
paper_summary: paper_summary: This paper proposes a novel image enhancement method named Simulated Infrared Fusion Guided Image-adaptive 3D Lookup Tables (SIRLUT). The method addresses the challenges posed by non-uniform color distribution in images, which limits the performance of traditional 3D Lookup Tables (LUTs). SIRLUT introduces simulated infrared imagery to reorganize color distribution and enhance the adaptability of 3D LUTs. The method comprises two main components: the Simulated Infrared Fusion (SIF) module, which uses a cross-modal channel attention mechanism to generate dynamic 3D LUTs, and the Simulated Infrared Guided (SIG) refinement module, which blends simulated infrared images to achieve local feature fusion. The authors claim that SIRLUT outperforms state-of-the-art methods in various tasks, achieving significant performance improvements with fewer parameters.
strength_weakness: strength_weakness: 1. The introduction of simulated infrared imagery to enhance 3D LUTs for image processing is innovative and addresses the problem of non-uniform color distribution effectively.
2. The paper is well-organized, with clear explanations of the methodology, experimental setup, and results. 
3. Experimental results on publicly available datasets (MIT-Adobe FiveK and PPR10K) show that SIRLUT outperforms existing methods by a notable margin (up to 2.25dB in PSNR), while also reducing the parameter count, making it more efficient.
limitations: limitations: 1. While the overall approach is novel, the individual components such as 3D LUTs and cross-modal attention mechanisms are well-established in the literature. The novelty primarily lies in their combination and application. 
2. The presence of typos, Linner layers ->Linear layers in Figure 3.
3. Simulating infrared images is a key component of SIRLUT. This paper should provide some details on infrared image generation and simulate the impact of infrared images on network performance.
4. Although the method in the paper has a lower number of parameters, the running time of the network should also be provided to confirm the statement of efficiency and lightweight performance.
5. The paper lacks a description of the loss function.