R2SFD: Improving Single Image Reflection Removal using Semantic Feature Dictionary

Authors: green  rosh
pawan  prasad
kaushik r mitra
lokesh r boregowda
  sfd
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 07:20:57

Abstract: Single image reflection removal is a severely ill-posed problem and it is very hard to separate the desirable transmission and undesirable reflection layers. Most of the existing single image reflection removal methods try to recover the transmission layer by exploiting cues that are extracted only from the given input image. However, there is abundant unutilized information in the form of millions of reflection free images available publicly. Even though this information is easily available, utilizing the same for effectively removing reflections is non-trivial. In this paper, we propose a novel method, termed R2SFD, for improving single image reflection removal using a Semantic Feature Dictionary (SFD) constructed from a database of reflection-free images. The SFD is constructed using a novel Reflection Aware Feature Extractor (RAFENet) that extracts features invariant to the presence of reflections. The SFD and the input image are then passed to another novel network termed SFDNet. This network first extracts RAFENet features from the reflection-corrupted input image, searches for similar features in the SFD, and transfers the semantic content to generate the final output. To further improve reflection removal, we also introduce a Large Scale Reflection Removal (LSRR) dataset consisting of 2650 image pairs comprising of a variety of real world reflection scenarios. The proposed method achieves superior results both qualitatively and quantitatively compared to the state of the art single image reflection removal methods on real public datasets as well as our LSRR dataset.We will release the dataset at https://github.com/ee19d005/r2sfd.

Main paper: 
 introduction: 
 Capturing images in the presence of obstructions such as glass typically results in unwanted reflections. An image I ∈ R 𝑚×𝑛 corrupted by reflections can be modeled as a combination of two layers -the desirable transmission layer B ∈ R 𝑚×𝑛 and the undesirable reflection layer R ∈ R 𝑚×𝑛 [41] . This can be represented as follows:EQUATIONwhere 𝐼 , 𝐼 𝑡 and 𝐼 𝑟 denotes the reflection corrupted image, the transmission layer and the reflection layer respectively. The blending factor 𝛼 denotes the strength of reflection, which is typically dependent on imaging conditions. The restoration of such an image involves layer separation which is severely ill-posed [54] . There have been many works in the past addressing the problem of reflection removal. Several earlier methods attempt to constrain the solution space using hand-crafted priors such as natural scene statistics [20] , ghosting cues [42] and gradient sharpness [54] . However, these hand-crafted priors often cater to specific kinds of reflections and is not generalizable to handle a wider range of in-the-wild reflections. More recently, deep learning based approaches have been observed to produce state-of-the-art results in reflection removal [3, 6, 8, 9, 12, 21, 25, 26, 34, 56, 58] . Even though deep learning methods have made significant strides, single image reflection removal still remains a significantly hard problem due to its ill-posed nature. The existing deep learning based reflection removal methods have tried to exploit cues that discriminate between transmission and reflection layers. These cues are either learned in a multi-stage pipeline [4, 6, 21, 56] or enforced using sophisticated loss functions [51, 59] . However, these cues are extracted using only the information present in the given input image. The dependence on reflectioncorrupted input images hamper the ability of the neural network to effectively recover the transmission layer, especially in regions with high reflections. To alleviate this challenge, we propose to utilize information from external reflection-free images with similar semantic content as that of the input image. We hypothesize that the higher level semantic information from reflection-free images can guide the network to learn a better representation of the underlying scene. An earlier method [50] proposed a non-deep learning based approach using non-local image priors extracted from external images. However, this method uses hand-crafted sparsity priors, which may not generalize well for in-the-wild reflections. To the best of our knowledge, there exists no deep learning based method for the extraction and utilization of reflection-free semantic content from a database of images for improving reflection removal.In this paper, we propose a novel deep learning based method that can automatically retrieve and utilize reflection-free semantic content from a large image database to effectively remove reflections (see Fig. 2 ). Given an input image, we first use a light-weight Coarse Semantic Search (CSS) module to select a set of similar images from a large database of reflection-free images. These images are then passed through a novel Reflection Aware Feature Extractor (RAFENet) to construct a semantic feature dictionary (SFD), consisting of image features. The RAFENet is trained using a novel Reflection Aware loss function to ensure that the extracted features are mostly invariant to reflections. The SFD along with the reflection-corrupted input image is then passed to a novel architecture, termed Semantic Feature Dictionary Network (SFDNet), to generate the final output. The proposed SFDNet is designed to utilize the information from the semantic feature dictionary to improve reflection removal from the input image. As shown in Fig. 1 , our method is able to remove reflections much better than the state-of-the-art methods, especially in regions with very high reflections.We also propose a new dataset for advancing the progress in reflection removal. The existing datasets for reflection removal are either small-scale (∼ 1000 image pairs) [17, 49] or consists of only reflection layer information [48] . To the best of our knowledge, our dataset, termed Large Scale Reflection Removal (LSRR) dataset, consisting of 2650 image pairs, is the largest dataset containing real paired data for reflection removal. The major contributions of this paper are as follows:(a) We propose a novel method for improving reflection removal by utilizing reflection-free semantic content from external image databases. To the best of our knowledge, this is the first deep learning based method that utilizes reflection-free image databases for improving single image reflection removal. (b) We also introduce a new dataset termed Large Scale Reflection Removal (LSRR) dataset consisting of 2650 image pairs. To the best of our knowledge, the proposed dataset is the largest paired real dataset for reflection removal. (c) The proposed method outperforms state-of-the-art methods in reflection removal both qualitatively and quantitatively on public real datasets as well as our LSRR dataset 
related works: 
 In this section, we provide an overview of the existing literature in reflection removal. Single Image Reflection Removal: Earlier works in reflection removal used hand-crafted priors such as natural image statistics [19, 20] , relative smoothness [24] , gradient priors [2, 39, 45, 50] , ghosting cues [42] [10] and depth-of-field differences [47] to remove reflections from images. However, the hand-crafted priors used in these methods often fail in several real-life situations, resulting in lack of generalizability for these methods. To address this issue, several deep learning based methods with novel network enhancements have been proposed in the recent past, which generate state-of-the-art results [3, 6, 8, 9, 12, 21, 25, 26, 34, 56, 58] . Several other methods proposed novel loss functions for reflection removal such as adversarial loss [59] , alignment invariant loss [51] and gradient loss [48] . Another method proposed a self-semantic segmentation guided approach using a multi-task network [26] . There have also been several works which proposed novel methods for synthetic data modeling such as [13, 52, 61] . There has also been methods exploring methods that are not fully supervised such as [14, 29, 37] . Multi Image Reflection Removal: Earlier works in multi-image reflection removal explored approaches using gradient sparsity interdependence [7] , SIFT flow [23] and video based methods [31, 43] . Lun et.al [27] proposed a deep learning based approach to remove reflections using optical-flow based alignment, while Prasad et.al [33] proposed a deep learning based approach to remove reflections A coarse semantic search is first performed on a database of images to find images similar to input. These images are then passed through the RAFENet to generate a Semantic Feature Dictionary (SFD). The input image and the SFD is then passed to SFDNet to generate the reflection-free output. from a burst of images. A method for reflection removal using a reference image provided by the user was proposed by [35] . Several other methods focused on reflection removal using images captured using multiple modalities such as multi-focal lengths [40] , dualpixel sensors [36] and different flash or exposure settings [16] [1] . Several other works such as [18] [22] [53] used polarized images to extract cues that helps in reflection removal. Reference Based Image Restoration: Reference based image restoration using deep learning have been studied in domains such as image super-resolution [11, 28, 32, 55, 60] and image deblurring [15, 38] . These methods try to match the features extracted from the input and reference images to find relevant semantic content. These methods mostly use pre-trained VGG features for this purpose [44] . Further, most of these methods require the user to explicitly provide a reference image along with the input image. Our method is different from these approaches in two ways. First, our method does not require any manual intervention for choosing any reference image and is fully automatic during inference. Second, our method uses a novel feature extraction module to extract reflection-aware features that are invariant to reflections unlike the reference based approaches that typically use VGG features for matching semantic information. 
proposed method: 
 In this section, we describe the proposed methodology in detail, including the network architecture design and the loss functions used.The objective of our method is to improve single image reflection removal using semantic cues from easily sourceable reflection-free images. We develop our algorithm based on the following design principles: a) Given an input image, the algorithm should be able to quickly retrieve a small subset of semantically similar images from a large database of reflection-free images ; b) The algorithm should also be capable of extracting features from the reflectioncorrupted input image and search for similar features from this subset of reflection-free images. Hence, this feature extractor should be reflection-aware, i.e, the features extracted from semantically similar images should be similar regardless of the presence of reflections ; c) The algorithm should then use these reflection-aware features to aid removal of reflections from the input image.An overview of the proposed method is shown in Fig. 2 . For a given input image, a coarse semantic search is first performed on a reflection-free image database to obtain a set of 𝑠 images semantically similar to the input image. The database comprises of ∼ 50,000 diverse images sourced from publicly available datasets [57] [60] , that are predominantly reflection-free. The set of 𝑠 images are then passed through the proposed Reflection Aware Feature Extractor (RAFENet) to generate a Semantic Feature Dictionary (SFD). The SFD contains features that are mostly invariant to the presence of reflections. The SFD, along with the input image is then passed to the proposed SFDNet to generate the reflection-free output image. The salient components of the proposed pipeline is detailed in the following subsections.We propose a lightweight Coarse Semantic Search (CSS) module that retrieves a set of 𝑠 images similar to the input image from the database of reflection-free images in an efficient manner. We use a pre-trained VGG network as an image descriptor to encapsulate the semantic information of an image into a vector. Given an input image, we first extract the VGG image descriptors. Next we use cosine similarity metric to compare the input image descriptor with the descriptors pre-computed from the reflection-free images. The images with top 10 similarity scores are then passed to the proposed pre-trained reflection-aware feature extractor to compute a Semantic Feature Dictionary (SFD). An example of the frames obtained from the CSS module is shown in Fig. 2 . It can be seen that the CSS module is able to find images with similar semantic or texture content as that of the provided input image. Please refer to the supplementary material for more examples of similar images from CSS module.While VGG based feature extractor can encapsulate the overall semantic content of an image, they are not designed to extract fine features from reflection-corrupted images consisting of superimposition of multiple image layers. The features extracted from the reflection-corrupted input image and reflection-free image database should be reflection-aware; i.e, similar features should be extracted from semantically similar images regardless of the presence of the reflections. This is essential to effectively search for features related to the transmission content from the semantic feature dictionary. We propose a novel reflection-aware feature extractor (RAFENet) to generate meaningful reflection aware features from the input image as well as from reflection-free images.Architecture. The proposed RAFENet consists of an encoder block and a Residual Convolution Block (RCB). As shown in Fig. 3 (a), the encoder block consists of 2 conv-batchnormrelu blocks of stride 1 (𝐶𝐵𝑅 1 ), followed by 3 conv-batchnorm-relu blocks of stride 2 (𝐶𝐵𝑅 2 ). The output from the RCB block is passed to a transmission branch, while the output feature map from the penultimate 𝐶𝐵𝑅 2 is passed to a reflection branch. We use separate branches so that features corresponding to reflection and transmission layers can be localized to their respective branches. This allows us to use features from the transmission branch as the required reflection-aware features. Both the transmission and reflection branches consists of further RCBs followed by decoder blocks. The decoder consists of 3 upsample-conv-batchnorm-relu blocks, followed by another conv-batchnorm-relu block and a convolutional layer with linear activation function. We use upsampling instead of deconvolution layers to avoid checkerboard artefacts. We also provide skip connections between the encoder block and decoder blocks. The output feature maps marked as 𝐹𝑒𝑎𝑡_1 and 𝐹𝑒𝑎𝑡_2 in Fig 3 (a ) are used as the reflection-aware features. These layers are chosen empirically so that the resultant receptive fields of the extracted features are of the appropriate size for effective semantic transfer between the images. function: transmission loss, cycle loss and siamese loss. Each of these components are detailed in this section. Transmission Loss (L 𝑡𝑟𝑎𝑛𝑠 ): This loss function is applied on the output from the transmission branch so that features specific to transmission layer are learned. It is defined as:EQUATIONHere, 𝐼 denotes the input image, 𝐼 𝑔 denotes the reflection-free ground truth image, and N 𝑡 (𝐼 ) denotes the outputs from the transmission branch. L 𝐼 2𝐼 (𝐼 1 , 𝐼 2 ) is an Image-to-Image loss function [34] defined as follows:L 𝐼 2𝐼 (𝐼 1 , 𝐼 2 ) = 0.6∥𝐼 1 -𝐼 2 ∥ 1 + 0.6∥𝐼 1 -𝐼 2 ∥ 2 + 0.4L 𝑔𝑟𝑎𝑑 (𝐼 1 , 𝐼 2 ) + 0.8L 𝑐𝑙 (𝐼 1 , 𝐼 2 ) (3)where L 𝑔𝑟𝑎𝑑 denotes 𝐿 1 in the gradient space and L 𝑐𝑙 denotes contextual loss, which is a feature space similarity loss introduced by [30] . Cycle Loss(L 𝑐𝑦𝑐𝑙𝑒 ): The sum of the outputs from the transmission and reflection branches should generate the original input image. This property can be used to implicitly constrain the output of the reflection branch using a cycle loss as follows:EQUATIONwhere N 𝑟 represents the reflection branch.Siamese Loss (L 𝑠𝑖𝑎𝑚 ): The objective of this loss function is to constrain the training of RAFENet such that 𝑓 𝑒𝑎𝑡 1 and 𝑓 𝑒𝑎𝑡 2 are similar for both reflection-corrupted input image (𝐼 ) and reflectionfree ground-truth image (𝐼 𝑔 ). To compute L 𝑠𝑖𝑎𝑚 , we perform two forward passes per iteration of the training. In the first forward pass, we provide 𝐼 as the input to the network to generate N 𝑡 (𝐼 ), 𝐹𝑒𝑎𝑡_1(𝐼 ) and 𝐹𝑒𝑎𝑡_2(𝐼 ). Here, N 𝑡 (𝐼 ) represents the output from the transmission branch and 𝐹𝑒𝑎𝑡_1(𝐼 ) and 𝐹𝑒𝑎𝑡_2(𝐼 ) represent the feature maps extracted from 𝐼 . In the second forward pass, we provide 𝐼 𝑔 as input to the the network and generates N 𝑡 (𝐼 𝑔 ), 𝐹𝑒𝑎𝑡_1(𝐼 𝑔 ) and 𝐹𝑒𝑎𝑡_2(𝐼 𝑔 ). The Siamese loss is then computed as follows:L 𝑠𝑖𝑎𝑚 = L 𝐼 2𝐼 (N 𝑡 (𝐼 ), N 𝑡 (𝐼 𝑔 )) + ∥ 𝑓 𝐹𝑒𝑎𝑡 _1 (𝐼 ) -𝑓 𝐹𝑒𝑎𝑡 _1 (𝐼 𝑔 )∥ 1 + ∥ 𝑓 𝐹𝑒𝑎𝑡 _2 (𝐼 ) -𝑓 𝐹𝑒𝑎𝑡 _2 (𝐼 𝑔 )∥ 1 (5)This loss function enforces the network to learn 𝑓 𝑒𝑎𝑡 1 and 𝑓 𝑒𝑎𝑡 2 in a manner invariant to the presence of reflections in the image.The overall reflection invariant loss function is then defined as:EQUATION3.4 SFDNet 3.4.1 Network Architecture. We propose Semantic Feature Dictionary Network (SFDNet), a novel network that utilizes the Semantic Feature Dictionary generated by our proposed RAFENet to enhance single image reflection removal. An overview of SFDNet is shown in Fig. 4 . The input image is first passed through the pre-trained RAFENet to generate reflection aware features. These features are then passed through a Feature Search (FS) module which returns the most similar matching features from the semantic feature dictionary. Next, we use a Semantic Transfer (ST) module [60] to fuse the matched features from the FS module with the reflection-aware features extracted from the input image. The resultant feature maps are then passed through Residual Convolutional Blocks (RCB), each consisting of two Conv-Batchnorm-Relu layers with residual connections. Finally, a decoder consisting of 3 upsample blocks and 2 convolutional blocks are used to generate the final output. We also provide skip connections between the feature extractor and decoder blocks. More details on FS and ST modules are discussed in this section. Feature Search (FS): Given a reflection aware feature extracted from the input image, the objective of FS module is to retrieve the most similar feature from the semantic feature dictionary. Trivially searching in the SFD is highly inefficient resulting in huge computational overheads. To this end, we propose a novel module for FS using lightweight differentiable operations. The proposed FS module can be plugged into any network architecture for efficient feature search. Let 𝑓 (𝐼 ) be the feature map of size (𝐻 × 𝑊 × 𝐶) extracted from the input image. We represent this feature map using the 2D matrix F consisting of 𝐻 .𝑊 rows and 𝐶 columns. Each row 𝑖 in this matrix (F [𝑖]) represents a 𝐶 dimensional feature vector extracted from the input image. We also define a matrix F 𝑠 𝑓 𝑑 of dimensions (𝑠.𝐻 .𝑊 × 𝐶) to represent the features in the SFD. Here, 𝑠 denotes the number of similar images obtained using coarse semantic search. For each feature in F , the FS module searches for the most similar feature in F 𝑠 𝑓 𝑑 using cosine similarity metric and generates a matched feature map (F 𝑚 ). We use matrix operations for efficient computations. First, we construct a cosine similarity matrix as follows:EQUATIONwhere F and F 𝑠 𝑓 𝑑 are norm vectors defined as follows:F[𝑖] = ∥F [𝑖] ∥ 2 (8)Here F and F 𝑠 𝑓 𝑑 are of dimensions (𝐻 .𝑊 × 1) and (𝑠.𝐻 .𝑊 × 1) respectively and F 𝑐𝑜𝑠 is a matrix of dimensionality (𝐻 .𝑊 × 𝑠.𝐻 .𝑊 ).[ 𝑗] in the cosine similarity matrix represents the cosine similarity of the 𝑖 𝑡ℎ input feature map with the 𝑗 𝑡ℎ feature map from the SFD. Next, an index matrix representing the indices of the most similar features in the SFD is constructed as follows:F 𝑖𝑛𝑑 [𝑖] = arg max 𝐹 𝑐𝑜𝑠 [𝑖](9) Here, F 𝑖𝑛𝑑 is a matrix of dimensions (𝐻 .𝑊 × 1). Each element F 𝑖𝑛𝑑 [𝑖] denotes the index of the most similar feature from F 𝑠 𝑓 𝑑 . The matched feature is then constructed as follows:EQUATION) F 𝑚 is a matrix of dimensions (𝐻 .𝑊 × 𝐶) consisting of the most similar reflection-aware feature for each input feature vector. Semantic Transfer (ST): We use a Semantic Transfer block similar to the one proposed in [60] to transfer the features from F 𝑚 to the subsequent features. A schematic of the ST module is shown in Fig. 4 . Specific implementation details of this module is provided in the supplementary material.We use image-to-image loss function defined in Eq. 3 to train the proposed SFDNet. 
large scale reflection removal dataset: 
 The efficacy of deep learning based image restoration is dependent on the availability of a large dataset of paired images, consisting of both reflection-corrupted and ground truth images. However, existing paired datasets for reflection removal [17, 49] consists of ∼ 1000 image pairs, which is not sufficient for generalization. Wan et.al [48] proposed a large scale Reflection Image Dataset (RID) consisting of 3250 images. However, this dataset consists only reflection layer images, and proposes to construct synthetic training dataset by combining the provided reflection layer images with reflection-free images from public datasets. However the inaccuracies in the mathematical models used in synthetic data creation often translates to lack of adaptability of these methods to real-life scenarios.To alleviate these issues, we propose a Large Scale Reflection Removal (LSRR) dataset for training and testing purposes. The proposed dataset consists of 2650 high resolution images with real reflections and their corresponding reflection-free ground truth images. The dataset was captured using a smartphone in various indoor and outdoor locations and in different lighting conditions. We use a portable glass panel to create images with and without reflections. The smartphone was affixed to a tripod so that the input image is aligned with the ground truth image. We also varied the angle of the portable glass with respect to the smartphone to simulate real-life capture conditions. Our dataset consists of images with strong as well as weak reflection components for better variability and generalization. To the best of our knowledge, the proposed dataset is the largest dataset consisting of images with real reflection and their corresponding aligned ground truths. We use 2550 image pairs for training and 100 images for benchmarking. Please refer to the supplementary material for some examples of image pairs from our dataset. 
training dataset.: 
 We use synthetic and real image data for training the proposed method. To create synthetic data for training, we make use of the datasets proposed in SRNTT [60] and PASCAL-VOC [5] . We use images from SRNTT dataset as the transmission layer, images from PASCAL-VOC as the reflection layer, and generate synthetic input images with reflections using the method proposed by [51] . We create a set of 11000 images pairs in this manner, of which, 1000 image sets are used for validation and the remaining for training. To train the proposed method using images with real reflections, we use the training set of the proposed LSRR Dataset consisting of 2550 image pairs. 5.1.2 Testing Datasets. We make use of the following datasets for evaluations: a) Berkeley-20 [59] ; b) Postcard, Wildscene and Solid Object datasets from SIR2 [46] ; c) Our LSRR test dataset; and d) ERR-DSLR50 [51] dataset. We use ERR-DSLR-50 dataset only for qualitative evaluations since this dataset consists of unaligned ground-truth images. We train the proposed RAFENet and SFDNet using PyTorch framework on a PC with NVIDIA Tesla V100 GPU and 32 GB RAM. Both RAFENet and SFDNet are trained till convergence using ADAM optimizer with an initial learning rate of 10 -5 and a weight decay of 0.025. Please refer to the supplementary material for more details on training and implementation.We compare the proposed method against state-of-the-art methods in reflection removal, BDN [56] , ERRNet [51] , IBCLN [21] , Dong et.al [4] , YTMT [9] , RAGNet [25] and SRNet [3] . For a fair comparison, we fine tune the methods with publicly available training codes (ERRNet [51] , IBCLN [21] and YTMT [9] ) using our proposed LSRR dataset. For the other methods we use the pre-trained models and inference codes provided by the authors for generating results.. We provide image comparisons against the aforementioned methods on public and our LSRR datasets in figures 5 and 6 respectively. We have shown results on challenging image sets consisting of strong reflections spread over a large area.It can be seen that the state-of-the-art methods fail to remove these reflections completely. Our method makes use of semantic information from the automatically extracted semantic feature dictionary to effectively remove reflections in these images. Hence, it can be seen that our method is able to remove reflections better than the state-of-the-art methods. Please refer to the supplementary material for more results.Results. We provide quantitative comparisons of our method against several state-of-the-art methods in reflection removal on multiple public datasets in Table 1 using PSNR and SSIM metrics. We use the notation -F to denote the methods fine tuned using our dataset. The proposed method is able to achieve better PSNR scores compared to the state-of-the-art methods on all the datasets. Our method also achieves the best SSIM scores on most of the datasets. Further, our method also achieves the best PSNR and SSIM scores averaged over all the public datasets. We also compare against a self-semantic guided method by Liu et.al [26] . This method proposes to extract a segmentation map from the input images, and then use it to guide the reflection removal. However, unlike our method, this method does not utilize external semantic information, resulting in an inferior performance. This method generated output images with a PSNR of 23.75 and an SSIM of 0.8895 averaged over Berkeley and SIR2 datasets. On the other hand, our method outperform [26] on both these metrics obtaining scores of 26.26 and 0.902 respectively, resulting in an improved PSNR of ∼ 2.5 dB. We also provide quantitative comparisons on the testing dataset of our proposed LSRR dataset in Table . 2. We compare our method against original (-O) and fine-tuned (-F) models of ERRNet and IBCLN methods. It can be seen that our method outperforms the existing methods on both PSNR and SSIM metrics on both public as well as our LSRR datasets. It can also be observed that the methods fine tuned using our training dataset (-F) significantly outperforms the original models (-O) on LSRR testing dataset. Moreover the fine tuned models are able to obtain comparable or better scores compared to original models on public datasets. This show the efficacy of the proposed LSRR dataset, in advancing further research in reflection removal.In this section, we provide ablation studies to justify the importance of the novel components introduced in our methodology. All the experiments are evaluated on SIR2-Wildscene dataset. We also provide more experimental studies in supplementary material, including details on network complexity, impact of SFD size, extension towards an interactive approach and a detailed analysis of reflection-awareness. Impact of Image Database: First, we analyze the impact of using image database of reflection-free images on the final output image. For this experiment, we evaluate the network without providing the SFD extracted from an external database of images. The result of this experiment is provided in row 1 of Table 3 . It can be seen that this experiment results in a PSNR drop of ∼1.3 dB compared to our proposed method which used image database (row 4 of Table 3 ). We also provide qualitative image comparisons for this experiment in Fig. 7 (top). It can be seen that the proposed method is able to remove reflections much better with the help of semantic content from external image database. This is evident especially in images with strong reflection components. Impact of Semantic Transfer: Next, we evaluate the impact of Semantic Transfer module (ST) by training a network without ST We also provide an ablation study to analyze the impact of the proposed Reflection Aware Feature Extractor (RAFE) on the output image quality. For this experiment, we used a pre-trained VGG as a feature extractor to extract features from the input image. We used the layers conv2_2, conv3_4 and conv4_2 of the VGG network for feature extraction. The extracted features were then passed through Feature Search and Semantic Transfer modules and the remaining layers of the SFDNet. We trained this network using the same training settings as that used for the original SFDNet. These results are summarized in row 3 of Table 3 . It can be seen that replacing the proposed RAFE feature extractor with VGG results in a PSNR degradation of ∼ 0.55 dB. We also provide qualitative results on SRNTT validation set in Fig. 7 (bottom). It can be seen that the results obtained from SFDNet using VGG feature extractor is unable to remove the reflections completely, resulting in an output image with lower visual quality. On the other hand, SFDNet trained using the proposed Reflection Invariant feature extractor is able to remove reflections much better. This experiment shows the importance of Reflection Aware Feature Extraction for effective transfer of related semantic content from the reflection-free database of images.Loss Component Analysis for RAFENet: In this section, we analyze the impact of the various components used in the proposed Reflection Aware loss function. For this experiment, we train the RAFENet using two different configurations: a) Only transmission loss (L 𝑡𝑟𝑎𝑛𝑠 ); and b) transmission loss and cycle loss (L 𝑐𝑦𝑐𝑙𝑒 ). The resultant feature extractor is then loaded into the SFDNet and trained for reflection removal. The final PSNR and SSIM scores of the outputs generated by these configurations are compared with that generated by the proposed method using all the three losses (L 𝑡𝑟𝑎𝑛𝑠 + L 𝑐𝑦𝑐𝑙𝑒 + L 𝑠𝑖𝑎𝑚 ) . These results are summarized in Table  
conclusion: 
 We present a novel methodology, termed 𝑅 2 𝑆𝐹 𝐷, to improve reflection removal using semantic information extracted from easily available reflection free images. Our methodology consists of two novel network architectures, termed RAFENet and SFDNet. RAFENet is designed to extract reflection aware features that are mostly invariant to the presence of reflections. The SFDNet is designed to utilize the reflection-aware features extracted from the image database to improve removal of reflections from the input image. We also introduce a new dataset for reflection removal, termed Large Scale Reflection Removal dataset, consisting of 2650 images with real reflections and their corresponding ground truth images.To the best of our knowledge, the proposed dataset is the largest available dataset for reflection removal consisting of real image pairs. The proposed method achieves superior results both qualitatively and quantitatively against state-of-the-art methods on public real datasets as well as our LSRR dataset. We also conduct extensive ablation studies to show the impact of the external image database, Semantic Transfer module and RAFENet on the final output image.We further analyze the impact of various components of the proposed loss Reflection Aware loss function. These experiments show that proposed novel components and the new dataset enable us to achieve state-of-the-art results in reflection removal 
 
 Appendix:  


Decision: Reviewers acknowledged the good results and novel use of an external database. The initial scores are all positive and final scores are  mixed, with both lowering scores and increasing scores happening. The main concerns from all reviewers are also from its novel design, that is the external database. While it shows effect on the good side, there are concerns with it. The reviewers questioned if it will also hurt in some cases, if the input data is outside the LSRR distribution, or if the semantic search fails, etc. The authors partially answered these questions but not all reviewers are fully satisfied, and there are lowering scores happening. After careful discussion and consideration, the ACs decided to accept this paper to encourage this kind of attempt. 

However, the ACs share the same concern with the reviewers about the potential negative side of using an external database. The authors are required to consider all the concerns raised by the reviewers and reveal the limitations by showcasing what the results will be if some part is not working properly as expected, or to what extent the method is robust. Revealing limitations is also part of research and will benefit the readers of this paper in this direction.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel method for single image reflection removal by constructing a semantic feature dictionary.  This dictionary is created using a reflection-aware feature extractor that captures features that remain consistent across images with and without reflections. To further enhance its effectiveness, this paper also introduces a new Large Scale Reflection Removal (LSRR) dataset.  The proposed method demonstrates superior performance in both visual quality and quantitative metrics when compared to state-of-the-art methods on public real-world datasets as well as the LSRR dataset.
strength_weakness: strength_weakness: 1. The proposed method introduces an innovative idea of using external reflection-free databases to enhance the reflection removal process, which could potentially lead to more accurate and effective results.
2. The introduction of the LSRR dataset is commendable as it provides a large-scale, high-quality benchmark for researchers to evaluate their methods, promoting the development of the field.
3. The proposed method outperforms state-of-the-art methods in reflection removal both qualitatively and quantitatively.
limitations: limitations: 1. There are some errors in this paper. For example, the SSIM of 'only Ltrans' has two decimal points in Table 4.
2. This paper does not compare the model complexity with other methods, but only compares the different components of its own model.
3. The testing time is not compared.
4. While the use of external reflection-free semantic content databases is innovative, it may be constrained by the size and diversity of available databases. Smaller or niche databases might not provide sufficient information to comprehensively improve reflection removal in single images. Additionally, the cost of acquiring and maintaining these databases could be substantial.
5. The LSRR (Large Scale Reflection Removal) dataset may suffer from data imbalance due to the variety of reflection conditions in real-world scenarios. If the training set only includes certain types of reflections, the model might perform poorly on unseen types. Addressing data imbalance is crucial for improving the model’s generalization capabilities.
6. The effectiveness of the proposed method is highly dependent on the quality of the external databases used. If the databases contain errors or inconsistencies, it could negatively impact the performance of the reflection removal algorithm.
paper_summary: paper_summary: The study presents "R2SFD," an innovative framework for effectively removing reflections from single images. Central to this approach is the utilization of a semantic dictionary, derived from a database of clean images, to assist in the accurate discrimination and subtraction of reflective components. The study also introduces a dataset designated for reflection removal tasks. The proposed method demonstrates notable improvements over existing models, showcasing its robustness through extensive tests on a variety of datasets.
strength_weakness: strength_weakness: 1. The deployment of a Semantic Feature Dictionary (SFD) presents a novel strategy for addressing the issue of reflections in images. By relying on semantic features extracted from a clean image database, the method introduces an innovative pathway to reflection removal that goes beyond conventional techniques.
2. The creation and utilization of the Large Scale Reflection Removal (LSRR) dataset represents a contribution to the research community.
3. The proposed R2SFD framework has demonstrated superior performance in comparison to existing techniques.
limitations: limitations: 1. The methods section is not clear enough, for example, line 403 “These layers are chosen empirically”, but there is no further explanation and no reference to related work.
2. The proposed method relies on a Semantic Feature Dictionary (SFD) built from a database of clean, reflection-free images. This dependence suggests that performance may be limited in cases where such clean images are not readily available or where the reflection properties differ significantly from those in the database.
3. The inclusion of multiple components like RAFENet and the SFDNet might increase the complexity of the model. Dealing with reflections in real-time scenarios may pose a challenge due to computational demands.
paper_summary: paper_summary: The paper presents a novel approach to the challenging problem of single image reflection removal. The proposed method uses a Semantic Feature Dictionary (SFD) constructed from a database of reflection-free images. To further enhance reflection removal, the authors introduce the Large Scale Reflection Removal (LSRR) dataset, consisting of 2650 image pairs from various real-world reflection scenarios. The method shows superior results both qualitatively and quantitatively compared to existing techniques.
strength_weakness: strength_weakness: 1. The introduction of a Semantic Feature Dictionary (SFD) and the innovative use of a Reflection Aware Feature Extractor (RAFENet) present a novel and promising approach to the problem of single image reflection removal.
2. The LSRR dataset  providing a diverse and representative set of reflection scenarios.
3. The paper addresses a practical problem with significant real-world applications, and the proposed method shows promise for improving the quality of images in various scenarios.
limitations: limitations: 1. The paper proposes a novel reflection removal method using a SFD and a Reflection Aware Feature Extractor (RAFENet). While the approach appears promising, some aspects lack detailed explanation. For instance, how does RAFENet ensure that the extracted features are invariant to reflections? Additionally, visualizations such as feature maps could be included to demonstrate the model's performance in handling reflections.
2. While the paper introduces a Reflection Aware Feature Extractor (RAFENet) as a key component of the proposed method, it does not provide specific details about the data format or representation of these Reflection Aware Features. This lack of information makes it difficult to understand how these features are utilized within the network and how they contribute to the reflection removal process.
3. The paper introduces a new Large-Scale Reflection Removal (LSRR) dataset, but it lacks detailed descriptions regarding the diversity and representativeness of this dataset. While it is mentioned that the dataset contains 2650 image pairs, there is insufficient information about the sources, types of scenes, and diversity of reflection scenarios.The authors should provide more comprehensive information about the LSRR dataset, such as the sources of the image pairs, the distribution of scenes, the intensity, and types of reflections. This would better demonstrate the diversity and representativeness of the dataset, enhancing its credibility and application potential.
4. The paper compares the proposed method with several existing reflection removal techniques. However, it appears that some of the compared methods on the proposed LSRR dataset are not the most recent or advanced in the field. The most recent research work compared seems to be for 2020? The authors should update their comparison to include more recent and advanced reflection removal methods published in the last couple of years. This will provide a more accurate and up-to-date evaluation of the proposed method's performance.
paper_summary: paper_summary: The R2SFD introduces an innovative approach to reflection removal by effectively leveraging a semantic feature dictionary (SFD) derived from reflection-free images. This represents a significant advancement over traditional methods that mainly rely on cues from corrupted input images. The paper also contributes a new dataset, Large Scale Reflection Removal (LSRR), which consists of 2650 image pairs that could be immensely beneficial for training and benchmarking in future studies.
strength_weakness: strength_weakness: 1. The integration of RAFENet and SFDNet to extract and utilize reflection-invariant features is both novel and technically sound.
2. The method outperforms state-of-the-art techniques in qualitative and quantitative evaluations across multiple public datasets and the newly introduced LSRR dataset.
limitations: limitations: 1. The method’s performance heavily relies on the availability of a large, semantically rich external database. How should the absence of certain semantic content in the dataset be addressed? Will this omission impact the effectiveness of the model?
2. How can the effectiveness of the reflection-free image database be ensured?
3. The paper lacks a comparative analysis of the computational complexity of the proposed method against existing methods.
4. The authors should consider incorporating a greater proportion of recent studies, particularly those published in 2023 and 2024, to ensure the paper's relevance and comprehensiveness.