HighlightRemover: Spatially Valid Pixel Learning for Image Specular Highlight Removal

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 06:19:08

Abstract: Recently, learning-based methods have made significant progress for image specular highlight removal. However, many of these approaches treat all the image pixels as spatially consistent, overlooking the negative impact of invalid pixels on feature reconstruction. This oversight often leads to undesirable outcomes, such as color distortion or residual highlights. In this paper, we propose a novel image specular highlight removal network called HighlightRNet, which utilizes valid pixels as references to reconstruct the highlight-free image. To achieve this, we introduce a context-aware fusion block (CFBlock) that aggregates information in four directions, effectively capturing global contextual information. Additionally, we introduce a location-aware feature transformation module (LFTModule) to adaptively learn the valid pixels for feature reconstruction, thereby avoiding information errors caused by invalid pixels. With these modules, our method can produce high-quality highlight-free results without color distortion and highlight residual. Furthermore, we develop a multiple light image-capturing system to construct a large-scale highlight dataset called NSH, which exhibits minimal misalignment in image pairs and minimal brightness variation in non-highlight regions. Experimental results on various datasets demonstrate the superiority of our method over state-of-the-art methods, both qualitatively and quantitatively.

Main paper: 
 introduction: 
 Specular highlights are natural occurrences when light strikes an object with smooth surface. However, the presence of continuous or discontinuous spots in specular highlight regions often leads to poor visibility and incoherent diffuse regions in images. This phenomenon significantly increases the complexity and difficulty of various vision tasks, including object detection [13] , semantic segmentation [3] , object tracking [6] , image segmentation [12] , and so on. Therefore, effectively removing specular highlights from Figure 1 : Image specular highlight removal. With the consistent feature manipulations, results of [27] and [4] may cause color distortion or highlight residual. In contrast, our method can produce more desirable result by utilizing valid pixels.images and recovering clear, highlight-free images is both important and challenging.Existing image specular highlight removal methods fall into two groups. Traditional methods [12, 24, 30, 31] leverage various constraints or assumptions to remove highlights from images but often demonstrate limited effectiveness. Recently, numerous learningbased specular highlight removal methods have been developed [4, 9, 10, 27] . They dig into the mapping relationship between highlight images and non-highlight images, aiming for enhanced performance. However, most of these methods uniformly process all pixels in the image, creating potential problems such as convolution of invalid pixels or deviation calculation of features. The main reason is that, the highlight regions with strong light spots are corrupted regions, and pixels in these region are invalid pixels for specular highlight removal. Simply mapping the features via consistent processing contains convolution of invalid pixels, resulting in mean and variance shifts in normalized features. It can result in invalid or biased recovery in highlight regions, leading to unsatisfactory results with highlight residual or color distortion, as shown in Figure 1(c ) Figure 1(d) .Moreover, the dataset has a crucial impact on the performance of learning-based models. Currently, there are only three benchmark datasets publicly available for specular highlight removal. However, these datasets still have quality defects. For example, SHIQ [4] and SSHR [5] are synthesized datasets. But the synthetic images still exhibit some statistical feature differences from the real images. On the other hand, PSD [27] is a real-world dataset, while the image pairs in this dataset suffer from obvious misalignment and brightness variations in non-highlight regions. The framework of the proposed HighlightRNet. We first use an encoder to extract features. Then, we introduce a context-aware fusion block (CFBlock) in the bottleneck layer to learn global contextual information. Next, we embed two location-aware feature transformation modules (LFTModule) into the decoder, aiding in the reconstruction of high-quality highlight removal results with a consistent appearance.To address the above challenges, we propose a novel image specular highlight removal network called HighlightRNet, which utilizes valid pixels in the image to reconstruct the highlight-free image. Figure 2 illustrates the framework of the proposed HighlightRNet, which is an encoder-decoder structure with a discriminator. Specifically, we introduce a context-aware fusion block (CFBlock) in the bottleneck module, which learns global contextual information in four directions and passes feature information from each pixel to the others. After several convolutions, the highlight region is gradually recovered, resulting in a distinct appearance from the original image. To this end, we propose a location-aware feature transformation module (LFTModule). Based on the spatial relationship of features, this module learns a spatial saliency map to demonstrate which are the valid pixels for specular highlight removal task. Thus, we can redecode the features using the valid pixels as references, avoiding information error caused by invalid pixels and promoting high-quality highlight-free results without color distortion and highlight residual, as shown in Figure 1(b) .Additionally, we construct a new large-scale real-world highlight dataset for specular highlight removal. To obtain high-quality highlight image pairs, we build a simple yet effective image-capturing system with multiple light sources. This multiple light source combination mechanism effectively avoids problems such as misalignment between image pairs and inconsistent brightness in nonhighlight regions. Our image-capturing system is portable and suitable for indoor and outdoor use.To sum up, our contributions are summarized as follows:• We propose a network called HighlightRNet to remove specular highlights in the image, which can recover a highquality highlight removal results without color distortion and highlight residual.• We introduce a context-aware fusion block to learn global contextual information and a spatial feature redecoding module to reconstruct the image features using valid pixels as references. • We construct a real-world highlight dataset without misalignment between image pairs and consistent brightness in non-highlight regions. Experimental results and evaluations demonstrate the superiority of our method over the state-of-the-art methods. 
related work: 
 Traditional methods for image specular highlight removal often rely on additional prior knowledge [8, 17, 26] . Shafer et al. [21] introduced a method to analyze standard color image to estimate the amount of interface (specular) and body (diffuse) reflection at each pixel. Klinker et al. [15] used the difference between the object color and highlight color to separate the color of every pixel into a matte component and a highlight component. Shen et al. [22] separated reflections in a color image based on the error analysis of chromaticity and the appropriate selection of body color for each pixel. Yang et al. [29] proposed a novel reflection components separation model based on H-S color space. Yang and Tang [30] formulated the highlight removal problem as an iterative bilateral filtering process. The method proposed by Kim et al. [12] was based on an observation that the dark channel usually provides an approximate highlight-free image. Shen and Zheng [23] considered color space to analyze the distribution of the diffuse and specular components and used this information for separation. Akashi [1] proposed a model-driven approach to improve the lighting normalization of face images. Zhang et al. [32] formulated highlight detection as a Non-negative Matrix Factorization (NMF) problem.With the development of deep learning, numerous learningbased methods have been proposed for image specular highlight removal, showing promising results using annotated training data. Lin et al. [16] proposed a fully-convolutional neural network (CNN), which automatically and consistently removes specular highlights from a single image by generating its diffuse component. Muhammad et al. [18] introduced Spec-Net, which took an intensity channel as input to remove high-intensity specularity from low chromaticity images. They also proposed Spec-CGAN, which input an RGB image to produce a diffuse image. Wu et al. [27] presented a novel GAN for specular highlight removal with the guidance of the detected specular reflection information. Fu et al. [4] developed a multi-task network for joint highlight detection and removal based on a new specular highlight image formation model. These methods can handle small sizes as well as weak highlights, but they still perform poorly for others and often carry for color or texture distortion. More recently, Fu et al. [5] proposed a three-stage specular highlight removal network, which first decomposed the input image into the albedo, shading, and specular residue components. Such treatment may causes the error accumulation and reduces the performance of the subsequent highlight removal due to intrinsic decomposition is also a difficult task. 
nsh dataset construction: 
 There are several image specular highlight datasets available, such as, SHIQ [4] , PSD [27] , and SSHR [5] . Table 1 summarizes the general information of the datasets. However, they still have some limitations:• SHIQ dataset: The highlight-free images in SHIQ are computationally synthesized, with feature differences from the real-world images. In addition, this dataset lacks images with highlights caused by color illumination. • PSD dataset: The variety of images is small and the background is simple. Some specular-free images have thin highlight residual. Also, the image pairs have misalignments and brightness variation in non-highlight regions. • SSHR dataset: The images are rendered in software to simulate real images that have simple textures . The backgrounds in the images are blank and filled with black color, and the visual effects are lacking in realism.In summary, the existing specular highlight datasets are still imperfect. To address this problem, we build an image-capturing system and construct a new and high-quality large-scale specular highlight dataset for image highlight removal. Our dataset is constructed on real scenes, and our image pairs have consistent brightness in non-highlight regions without misalignment.The common light source in the real world is natural light, which is unpolarized light. Existing techniques [19, 27] often use cross polarizers to capture specular highlight images. In a strict laboratory environment [30] , they convert a light source to linearly polarized light by adding a linear polarizer in front of the light source, as shown in Figure 3(a) . When linearly polarized light strikes an object, it produces linearly polarized specular reflection and unpolarized diffuse reflection [2, 19] . As these two different reflection lights [25] . Based on the dichromatic reflection model [20] , the observed image 𝐼 can be expressed as:EQUATION)where 𝜙 is a special angle between the two polarizers, as shown in Figure 3 (a). When capturing highlight images, we usually place a polarizer in front of both the camera and the light source. To prevent camera shake, we fix the polarizer in front of the camera, and rotate the polarizer in front of the light source to get the observed image. Wu et al. [27] use this strategy to construct PSD dataset. They capture the specular highlight image with 𝜙 = 0 and get the corresponding diffuse image with 𝜙 = 𝜋 2 :PolarizedEQUATION)As we know, objects are basically non-Lambertian. When the linearly polarized light source strikes the object, the object's surface is usually divided into the highlight regions and the non-highlight regions. While 𝜙 = 𝜋 2 , linear specular reflections are filtered out, resulting in significant brightness variations in non-highlight regions for the image pairs. As shown the first heat map in Figure 5 , the image pair from PSD has significant brightness variations in non-highlight regions. To solve this problem, we add the number of light sources to increase the diffuse reflection components, as shown in Figure 3 (b). The superposition of light reflection components is a very complex process, and here we view the process as a linear one. Assuming there are 𝑛 light sources in the environment, the observed image can indicate that,EQUATIONwhere 𝐼 𝑑 𝑖 and 𝐼 𝑠 𝑖 are the diffuse reflection component and the specular reflection component produced by the 𝑖-th light source, and 𝑖 ∈ {1, • • • , 𝑛}. 𝜙 𝑖 is the special angle between the two polarizers in front of the 𝑖-th light source and the camera. Assuming the 𝑛 light sources have the same intensity, the 𝑛 light sources have the same diffuse reflection component 𝐼 𝑑 and specular reflection component 𝐼 𝑠 . Thus, Eq. 3 can be rewritten as,EQUATIONTo obtain the image pair, we set 𝜙 𝑘 = 0 for the 𝑘-th light source, and the special angles of the other light sources are set to 𝜋 2 . Then, the image pair is that,EQUATIONGiven sufficient light sources, the diffuse reflection components tend to infinity, and the effect of specular reflections on non-highlight regions is relatively small. At this time, if the specular reflections are filtered out, the brightness in the non-highlight regions will not change significantly. As shown the heat maps in Figure 5 , our image pair remains unchanged from one another in non-highlight regions. Furthermore, we use a tripod to fix the camera and use a wireless trigger to control the captured process of the image, avoiding camera shake and misalignment in the image pair due to manual camera manipulation. To obtain a high-quality real-world dataset for image specular highlight removal, we built a simple yet effective image-capturing system, which consists of five light sources and a Conon 6D Mark II camera in a lighting-controlled environment, as shown in Figure 4(a, b ). Note that, our image-capturing system is a movable device. We can move it to the desired environment for image capture, both indoors and outdoors.Our image collection process mainly includes the following four steps: 1) we place the image-capturing device in the desired environment; 2) we fix a rotatable polarizer in front of both each light source and the camera; 3) we adjust the illumination direction and place an object in the intersection area of beams; 4) the image pair is captured by controlling the location of the light source and the polarizer. Specifically, according to Eq. 5, we first set all the light source's polarizers with 𝜙 = 𝜋/2 to obtain a diffuse image (highlight-free image). Then, we rotate the polarizer of one of the light sources with 𝜙 = 0 to obtain a specular highlight image.Repeating this process, we finally collect 30K image pairs from 3350 different scenes featuring a wide variety of materials that can easily produce highlights in daily life. Each image pair contains a highlight image and a corresponding highlight-free image. These images are divided into three parts: 22K pairs for training, 6K for testing, and 2K for validation. Figure 4 (c) presents some highlight and highlight-free image pairs in our NSH. 
proposed method: 
 We propose an image specular highlight removal network called HighlightRNet, which leverages valid pixels in the image to reconstruct the highlight-free image. To better recognize the valid pixels, we first introduce a context-aware fusion block (CFBlock) to learn global contextual information in four different directions. Then, we propose a location-aware feature transformation module (LFTModule) to reconstruct the features using valid pixels as referents. Our HighlightRNet is an encoder-decoder structure stacked with a discriminator, as shown in Figure 2 . The encoder employs a Con-vUnit and two ConvUnit+D-Conv to extract features from the image. Each ConvUnit comprises a convolution operation followed by a LRelu function, while D-Conv represents a dilated convolution with a LRelu function. The bottleneck module consists of three fusion blocks, and each fusion block applies a gated convolution and a D-Conv alongside a CFBlock. There is a residual connection between two neighboring fusion blocks. The decoder employs two ConvUnit+LFTModule layers, followed by a gated convolution, a residual module, and a ConvUnit to reconstruct the highlight-free images.Our discriminator is a binary classifier [11] to determine whether the predicted result is real or fake. It consists of six Conv+BN+ReLu layers and a fully connected layer. The final fully connected layer employs a sigmoid function to output the actual probability of the input image. The convolution operations typically operate in localized regions, which can limit the extraction of global contextual features. For tasks like specular highlight removal, these localization-based convolutions may not capture contextual associations over longer distances, leading to color or texture distortion in the results. To address this issue, we introduce a context-aware fusion block (CF-Block) to learn and fuse contextual information in four different directions, enabling more effective utilization of global information. Figure 6 illustrates the architecture of the proposed CFBlock. Initially, we segment the input features 𝐹 𝑖𝑛 ∈ R 𝐻 ×𝑊 ×𝐶 along the height dimension, where 𝐻 , 𝑊 and 𝐶 are height, width and number of channels, respectively. We then employ bidirectional LSTM (biLSTM) [7] to learn the segmented features leftward and rightward pixel-by-pixel, enabling each pixel to retain its left and right contexts. Following biLSTM processing, we recombine the learned features. Subsequently, the combined features are split along the width dimension, and we conduct upward and downward pixel-by-pixel learning on the segmented features using biLSTM. After that, we recombine the learned features to obtain a new feature map 𝐹 𝑐𝑜𝑛𝑡𝑒𝑥𝑡 . By alternately scanning horizontally and vertically, our CFBlock effectively fuses contextual features in four directions and pass them from each pixel to the others, facilitating the perception of global contextual information.Next, we apply a convolution to transform 𝐹 𝑐𝑜𝑛𝑡𝑒𝑥𝑡 to 𝐹 𝑚𝑖𝑑 ∈ R 𝐻 ×𝑊 ×𝑍 , and 𝑍 = 𝐻 × 𝑊 . Consequently, we can obtain a feature vector 𝑓 for each pixel. We perform a softmax operation to normalize 𝑓 along the channel dimension and obtain the contextual attention weights 𝜆, which is that:EQUATIONwhere 𝑖 ∈ {1, • • • , 𝑍 }, and 𝜆 ∈ R 𝑍 . Finally, we perform matrix multiplication of 𝐹 𝑖𝑛 and 𝜆 to construct the global contextual features 𝐹 𝑜𝑢𝑡 :EQUATION)Typically, the decoder utilizes all the computed features to reconstruct the highlight-free image. However, it's crucial to note that the regions with strong specular highlights are corrupted regions, and pixels in these regions are invalid pixels for specular highlight removal. Processing all features uniformly may introduces invalid or biased convolution of pixels, potentially leading to errors in feature computation and the generation of undesirable removal results, such as color distortion and highlight residual. Moreover, after several convolution operations, the specular highlights are gradually removed, as shown in Figure 7 . That indicates that the specular highlight regions dynamically change during the decoding process. The repaired contents, such as 𝑁 2 in Figure 7 (b), can also be considered as valid pixels for the restoration of the remaining highlighted regions.Based on the preceding analysis, we introduce a location-aware feature transformation module (LFTModule), which reconstructs features using the valid pixels as references. LFTModule utilizes the spatial relationship of input features to learn a spatial saliency map, which can be considered as the distribution and the intensity of the highlights at the current layer. Larger values in the spatial saliency map indicate stronger highlights at the current position. The stronger the highlight, the higher the probability that it is an invalid pixel. With the spatial saliency map, we can recognize which are the valid pixels for feature reconstruction. Thus, we can selectively manipulate the image features using the valid pixels as referents, avoiding information error caused by the consistent feature process and boosting satisfactory highlight-free results without color distortion and highlight residual. Figure 8 illustrates the pipeline of the proposed LFTModule. For the input feature 𝐹 𝑑𝑒 ∈ R 𝐻 ×𝑊 ×𝐶 , we first apply max-pooling and global average pooling along the channel axis to obtain efficient feature descriptor. We integrate the results of these two pooling operations and perform a convolution operation followed by a sigmoid function to compute a spatial saliency map 𝐴 ∈ R 𝐻 ×𝑊 ×1 . We can use the spatial saliency map 𝐴 to get the valid pixels through a threshold 𝑡. If 𝐴(ℎ, 𝑤) < 𝑡, we consider pixel (ℎ, 𝑤) is a valid pixel, and (ℎ, 𝑤) is an index of (𝐻,𝑊 ) axis; else, we consider this pixel to be in a strong highlight region and is an invalid pixel for specular highlight removal. In our experiments, we set 𝑡 = 0.5.Next, we utilize the valid pixels to normalize the input features and result in feature 𝑀 1 . Since the spatial saliency map 𝐴 contains global spatial information, we use convolution operation for 𝐴 to learn a global representation. We perform convolution operation on 𝐴 respectively to obtain two parameters 𝛾 and 𝛽. We use 𝛾 and 𝛽 as affine parameters to perform pixel-wise affine transformation on 𝑀 1 , obtaining the reconstructed features. With the affine transformation, our LFTModule promotes consistent-looking results of the highlight regions with the surrounding environment. The loss function for training our HightlightRNet contains three components: color consistency loss L 𝑐𝑜𝑙𝑜𝑟 , texture consistency loss L 𝑡𝑒𝑥𝑡𝑢𝑟𝑒 and adversarial loss L 𝑎𝑑𝑣 .Color consistency loss is used to suppress the color distortion during the reconstruction process. It is calculated using the mean squared errors (MSE) between the predicted highlight removal result 𝐼 𝑓 𝑟𝑒𝑒 and the ground-truth image 𝐼 𝑔𝑡 , as follows:L 𝑐𝑜𝑙𝑜𝑟 = ∥𝐼 𝑓 𝑟𝑒𝑒 -𝐼 𝑔𝑡 ∥ 2 2 .Texture consistency loss aims to preserve image structure using the gradient information in the image. It can prevent the generation of blurry results. Our texture consistency loss L 𝑡𝑒𝑥𝑡𝑢𝑟𝑒 is calculated as,EQUATIONwhere ∇ 𝑥 represents the gradient along the x-direction and ∇ 𝑦 represents the gradient along the y-direction.Adversarial loss. We employ relativistic average adversarial loss [11] to implement our adversarial loss L 𝑎𝑑𝑣 , which is described as:EQUATIONwhere 𝜎 is the sigmoid function and 𝐵𝐶𝐸 ( * ) measures the binary cross entropy. (𝑦 ′ , 𝑦) is set as (1, 0) for the generator and (0, 1) for the discriminator, respectively. 𝐷 is our discriminator. In summary, the total loss for our method is written as:EQUATIONwhere 𝜆 1 , 𝜆 2 and 𝜆 3 are the weighting parameters. In our experiments, we empirically set 𝜆 1 = 1.0, 𝜆 2 = 1, and 𝜆 3 = 0.01. 
experiments 5.1 implementation details: 
 Our network is implemented in PyTorch. We use the Adam optimizer [14] to train our HightlightRNet using an NVIDIA GeForce RTX 2080 Ti GPR for 80 epochs with a batch size of 8. The initial learning rate is set to 2 × 10 -4 and is decayed by a factor of 1/2 every 10 epochs until it reaches a value lower than 10 -5 .We evaluate the method on three datasets, including our NSH, SHIQ [4] and PSD [27] . We employ two commonly used metrics, including structural similarity index (SSIM) and peak signal-tonoise ratio (PSNR), to quantitatively evaluate the performance of our method.To verify the effectiveness of our method, we compare our method with three learning-based methods [4, 5, 27] and three traditional methods [23, 28, 31] . For a fair comparison, we directly use the codes provided by the authors with recommended parameter settings and retrain the learning-based methods on the same hardware. To train and evaluate method of Fu et al. [5] on the three datasets, we modify (e) Fu [4] (f) Wu [27] (g) Yang [31] (h) Shen [23] (i) Yamato [28] Figure 9 : Visual comparison of our method against state-of-the-art highlight removal methods. Compared with other results, our method produce satisfactory results without color distortion and highlight residual. 2 presents the quantitative comparisons on three datasets. From the table, we can observe that, our method achieves larger SSIM and PSNR scores on all datasets, indicating the better performance of our method compared to existing state-of-the-art methods.Visual Comparison. Figure 9 illuminates some visual highlight removal results to further demonstrate the effectiveness of our method. With inaccurate shadow detection results, Fu et al. [4] and Wu et al. [27] may produce undesirable results with highlight residual, as shown in Figure 9(e, f ). Without adequate information to guide, Fu et al. [5] may result in color distortion or incomplete removal of highlights, as shown in Figure 9(d) . Due to the lack of ability to capture high-level semantic information, the three traditional methods do not make good use of non-highlight pixels to restore the highlight regions, which usually result in color or texture distortion. For example, Yang et al. [30] suffer from severe artifacts such as black blocks and color distortion, as shown in Figure 9 (g). Shen et al. [23] often result in texture loss, as shown in Figure 9 (h). Yamamoto et al. [28] also suffer from black blocks and color distortion, as shown in Figure 9 (i). Comparatively, our To further verify the robustness and generalization ability of our HightlightRNet, Figure 11 presents some other highlight removal results for real-world natural images captured by smartphones or downloaded from the internet. As shown in Figure 11 , our method can effectively remove the highlights and obtain natural results with few artifacts. Moreover, although we focus on highlight removal, our HighlightRNet can also be applied to detect the highlight regions based on the predicted removal result, as shown in Figure 11 (c), which clearly distinguish the specular highlight regions.We performed a series of experiments to validate the effectiveness of our method and the superiority of our dataset.Effectiveness of the network. To demonstrate the effectiveness of our HightlightRNet, we compare our network with four variants to assess the impact of each component. The variants are (1) 𝑀 1 : remove CFBlock in HightlightRNet; (2) 𝑀 2 : replace LFTModule 3 summarizes the evaluated results on three datasets. From the table, we can observe: (1) our HightlightR-Net with all components gets the best results; (2) the proposed CFBlock and LFTModule can help improve the performance of the network, and the combination leads to the best performance; and (3) the loss functions L 𝑡𝑒𝑥𝑡𝑢𝑟𝑒 and L 𝑎𝑑𝑣 are necessary to ensure the high-quality highlight removal results. We also provide the visualization in Figure 10 , from which we can see that results produced by our HightlightRNet look more realistic with fewer artifacts.Superiority of NSH dataset. To validate the superiority of our NSH dataset, we train our HighlightRNet using four datasets: SHIQ, PSD, SSHR, and our NSH datasets. Table 4 summarizes the evaluation results on NSH dataset. It is evident from the table that the model trained with our NSH dataset outperforms the models trained with the other datasets, highlighting the superiority of our NSH dataset. 
conclusions: 
 In this paper, we propose a new network called HighlightRNet for image specular highlight removal, which utilize the valid pixels in non-highlight regions to reconstruct the highlight-free image. Particularly, we introduce a context-aware fusion block (CFBlock) to learn global contextual information in four directions. We also propose a location-aware feature transformation module (LFTModule) to adaptively learn the valid pixels for feature reconstruction, avoiding features error caused by the invalid pixels and promoting high-quality highlight-free results without color distortion and highlight residual. Experiments qualitatively and quantitatively demonstrate the superiority of our method over the state-of-the-art methods.  
 
 Appendix:  


Decision: All reviewers give positive ratings to this work. AC decides to accept this work. The authors are suggested to consider all suggestions in their  camera ready.

Reviews: review: paper_summary: paper_summary: The paper introduces a unique method, HighlightRNet, which uses valid pixels for reconstructing highlight-free images. This addresses a common problem in existing methods where invalid pixels can lead to color distortion and residual highlights.
strength_weakness: strength_weakness: (1)The integration of the context-aware fusion block (CFBlock) and location-aware feature transformation module (LFTModule) is innovative. These components enhance the ability to capture global contextual information and adaptively learn valid pixels for feature reconstruction.
(2)The creation of the NSH dataset using a multiple light image-capturing system is a significant contribution. This dataset mitigates issues like misalignment and brightness variations found in other datasets, providing a more reliable benchmark for evaluating highlight removal methods.
(3)The paper includes extensive experimental results demonstrating the superiority of HighlightRNet over state-of-the-art methods, providing both qualitative and quantitative evidence of its effectiveness.
limitations: limitations: (1)The proposed method, while effective, may be complex to implement and require significant computational resources, which could be a barrier to practical application.
(2)The method's performance in extremely diverse and uncontrolled environments is not extensively covered. Real-world applications often involve a wide variety of lighting conditions and object textures that could affect the method’s robustness.
(3)While the NSH dataset is a valuable addition, it would be beneficial to compare its performance on more diverse datasets and under different conditions to further validate its generalizability.
paper_summary: paper_summary: This paper implements a method for high-light removal and constructs a large-scale high-light dataset (NSH). In the network structure, a CFBlock is proposed to achieve long-distance information fusion in four directions based on biLSTM, and an LFTModule is introduced to dynamically learn features for image reconstruction. Additionally, a method for collecting high-light data in real-world scenarios is also proposed.
strength_weakness: strength_weakness: +Significant improvements were achieved on the self-constructed dataset (NSH) as well as other datasets SHIQ and PSD;

+The paper contributed a large-scale high-light dataset and provided a detailed explanation of its shooting principles, addressing the issues of brightness consistency and alignment that appeared in previous datasets.
limitations: limitations: -Lack of innovation in network structure design:

- Using cascaded biLSTM in CFBlock to obtain long-distance information—what advantages does this have compared to the Non-local module (or transformer module)? There is a lack of relevant theoretical explanations and experimental comparisons.
- The explanation of Normalization in LFTModule is not clear. Is it a variant of BatchNormalization?
- Extracting affine parameters directly from the spatial saliency map, which contains sparse information—what is the rationale behind this design? Why is it effective?

-The proposed data construction method cannot be used for large-scale (indoor/outdoor) data preparation.

-lack comparison on the SSHR dataset and some metrics shown in the table differ significantly from those presented in other works.
[Fu G, Zhang Q, Zhu L, et al. Towards High-Quality Specular Highlight Removal by Leveraging Large-Scale Synthetic Data[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.]

-lack experiments in more complex scenarios (e.g., indoor/outdoor).
paper_summary: paper_summary: The proposed method, HighlightRemover, introduces a network named HighlightRNet for the task of specular highlight removal in images.  This is achieved through two key innovative components: the Context-aware Fusion Block (CFBlock), designed to capture global contextual information, and the Location-aware Feature Transformation Module (LFTModule), which selectively utilizes valid pixels for feature reconstruction.  The method is underpinned by an encoder-decoder architecture with a discriminator, aimed at generating high-quality results without color distortion or highlight residual.
strength_weakness: strength_weakness: 1. This paper develops a new dataset, NSH, which is designed to minimize misalignment and brightness variation in non-highlight regions.  
2. The proposed method outperforms existing state-of-the-art techniques, both quantitatively and qualitatively comparison.
3. The authors have thoroughly evaluated their method, including comparisons with other methods and an ablation study demonstrating the contribution of different components of the proposed network.
limitations: limitations: 1. While the method performs well on the NSH dataset, it is unclear how well it generalizes to other datasets or real-world scenarios that were not part of the training process.
2.  Presenting and discussing failure cases could provide a more comprehensive view of the method's robustness.
paper_summary: paper_summary: The paper introduces HighlightRNet, a new network designed for efficient specular highlight removal in images. HighlightRNet leverages valid pixels from non-highlight areas for superior image reconstruction. It incorporates a Context-Aware Fusion Block (CFBlock) to gather global contextual information across four directions, enhancing the quality of the reconstructed images. The network also features a Location-Aware Feature Transformation Module (LFTModule), which accurately identifies and utilizes valid pixels, minimizing common issues like color distortion and residual highlights. Additionally, the paper presents a new large-scale highlight dataset, NSH, with minimal misalignment and brightness variation, which supports the network's effectiveness. Extensive testing on various datasets shows that HighlightRNet significantly outperforms existing methods, providing high-quality, highlight-free images both qualitatively and quantitatively.
strength_weakness: strength_weakness: 1. The author has contributed a high-quality, large-scale dataset of real-world object specular reflections, which has notably improved the effectiveness of the proposed method. This dataset is a valuable resource that will undoubtedly aid future research within this community.
2. The author introduces an innovative method and network architecture for specular reflection removal that performs exceptionally well on real-world datasets. The method has been rigorously compared with state-of-the-art techniques presented at leading conferences, showing notable improvements over previous approaches.
limitations: limitations: To ensure the comprehensive evaluation of the manuscript, I would like to inquire further regarding the methodology employed in setting the weights for the loss function. It has been noted that the author has based the weight assignment on empirical evidence. In the interest of scientific rigor, could you please clarify if there were any experiments conducted with alternative parameter settings? Additionally, it would be beneficial to have access to specific experimental results, if available, to better understand the impact of these different parameters on the overall performance of the proposed method. This information will greatly enhance the assessment of the robustness and applicability of the approach.