Reversible Decoupling Network for Single Image Reflection Removal

## Abstract

Abstract Recent deep-learning-based approaches to single-image reflection removal have shown promising advances, primarily for two reasons: 1) the utilization of recognition-pretrained features as inputs, and 2) the design of dual-stream interaction networks. However, according to the Information Bottleneck principle, high-level semantic clues tend to be compressed or discarded during layer-by-layer propagation. Additionally, interactions in dual-stream networks follow a fixed pattern across different layers, limiting overall performance. To address these limitations, we propose a novel architecture called Reversible Decoupling Network (RDNet), which employs a reversible encoder to secure valuable information while flexibly decoupling transmission- and reflection-relevant features during the forward pass. Furthermore, we customize a transmission-rate-aware prompt generator to dynamically calibrate features, further boosting performance. Extensive experiments demonstrate the superiority of RDNet over existing SOTA methods on five widely-adopted benchmark datasets. Our code will be made publicly available.

## 1 Introduction

Figure: Figure 1: Quantitative comparison in PSNR between ours and previous SOTA methods, where we achieve new records on all 5 datasets. Note that the scale of each axis is normalized by its second-best value. The best and second-best PSNR are displayed for reference.
Refer to caption: extracted/5916962/radar3.png

Reflection is a common superimposition factor when photographing through transparent medium, such as glass. Under the circumstances, the captured image $I$ typically contains a mixture of transmission $T$ (the scene behind medium) and reflection $R$ (the reflected scene)  (Nayar et al., ), which can be simply expressed as $I=T+R$. The presence of reflections often hinders vital information in the transmission layer, impeding the performance of downstream computer vision tasks, such as stereo matching, optical flow, and depth estimation (Tsin et al., ; Yang et al., ; Costanzino et al., ). Thus, single image reflection removal/separation (SIRR) is desired to disentangle the transmission and reflection components from a single input image. However, this problem is severely ill-posed as infinitely many possible decompositions of $\hat{T}$ and $\hat{R}$ satisfy $I=\hat{T}+\hat{R}$. In other words, it is highly challenging to determine which combination is optimal if without effective priors or guidance on decomposition.

In recent years, learning-based approaches have made tremendous strides in this field (Zhang et al., ; Wei et al., ; Li et al., ; Hu & Guo, ; Zhong et al., ). A key consensus among these methods is to exploit hierarchical semantic representations through large-scale recognition-pretrained models, which serve as priors or regularizers during the decomposition. One pioneering deep-learning work (Zhang et al., ) leverages intermediate features from a pre-trained VGGNet (Simonyan & Zisserman, ) through the concept of hypercolumns to help differentiate between the transmission and reflection layers from mixtures. Originally from neuroscience, the term “hypercolumn” refers to a functional unit in the visual cortex that processes visual stimuli at multiple receptive-field sizes  (Hubel & Wiesel, ). This concept was first applied to object segmentation and localization by interpolating and stacking features extracted from different layers of a network (Hariharan et al., ). However, simply mapping stacked high-dimensional hierarchies into a group of much lower-dimensional features–as input for subsequent processes–inevitably leads to considerable semantic information loss.

Previous works with SOTA performance (Hu & Guo, ; ) suggest that, all information from the source image is valuable for the task. The two components can be optimized by exchanging information between them. For any feasible decomposition ($\hat{T}$, $\hat{R}$), the following relationship holds:

$$ $\quad\hat{T}:=T-Q,\quad\hat{R}:=R+Q\quad\text{s.t.}\quad I=\hat{T}+\hat{R},$ (1) $$

where $Q$ represents the information to exchange. Concretely, YTMT (Hu & Guo, ) and DSRNet (Hu & Guo, ) select $Q$ via activation functions and channel splitting, respectively. Though being effective, the information preservation is not fully guaranteed in their interaction designs, *i.e.*, the information bottleneck induced by linear layers in YTMT and the multiplicative reductions in the gating mechanism of DSRNet.

To avoid the above risk, reversible units (Gomez et al., ), which are designed to preserve information, may offer a viable solution. In particular, building coupled reversible units naturally fits the situation as follows:

$$ $\underline{\textit{forward process}}\left\{\begin{aligned} &\hat{T}_{2}:=\hat{ T}_{1}+\mathcal{F}(\hat{R}_{1})\\ &\hat{R}_{2}:=\hat{R}_{1}+\mathcal{G}(\hat{T}_{2})\\ \end{aligned}\right.;\quad\underline{\textit{reverse process}}\left\{\begin{ aligned} &\hat{T}_{1}:=\hat{T}_{2}-\mathcal{F}(\hat{R}_{1})\\ &\hat{R}_{1}:=\hat{R}_{2}-\mathcal{G}(\hat{T}_{2})\\ \end{aligned}\right.,$ (2) $$

where $\mathcal{F}(\cdot)$ and $\mathcal{G}(\cdot)$ can be any network modules, and the subscripts stand for the different versions of layer estimations before and after the reversible units, respectively. For simplicity, we also use $\hat{T}$ and $\hat{R}$ to represent the corresponding deep features, which is based on the understanding that if the features are sufficiently disentangled, mapping them back to the image space becomes an easy task.

Although the use of reversible modules can address the issue of information loss in feature interactions at the same scale, preserving multi-scale information during the feedforward process remains a challenge. Beyond the hypercolumn (Zhang et al., ) and the progressive hierarchy fusion (Hu & Guo, ), one intuitive scheme is to stack reversible modules at each scale to facilitate forward propagation while incorporating cross-scale connections to ensure effective multi-scale interaction and fusion. A straightforward approach aligning with this idea is MAXIM (Tu et al., ) (without consideration of information loss), which employs a fully connected mechanism across multi-scale hierarchies. Similar ideas can also be found in HRNet (Sun et al., ). However, operating on high-dimensional features is computationally expensive and memory-intensive.

Inspired by GLOM (Hinton, ), which employs a part-whole hierarchy to represent an image with multiple columns, and embodies both bottom-up and top-down interactions to mitigate the computational burden associated with fully connected layers,we integrate multi-scale feature processors into a single sub-network, referred to as a “column”. Further, we ensemble the columns in parallel and build interactions in both bottom-up and top-down manners. It is worth noting that, the scaled residual connections used in GLOM for same level interactions between adjacent columns can still cause information loss. To remedy this problem, we extend the residual connections by incorporating multi-level reversible connections, which upgrades the vanilla reversible unit (Gomez et al., ).

Compared with structural designs guided by information bottleneck principle (Tishby & Zaslavsky, ; Hu & Guo, ), our proposed framework learns disentangled representations (Desjardins et al., ; Bengio et al., ) by categorizing and recombining the original information, instead of merely selecting and discarding elements, based upon a solid foundation for its information-preserving module (reversible unit). Additionally, it retains multi-scale information and facilitates cross-scale interaction.
Besides, in real-world scenarios, the reflection pattern varies along with multiple factors, such as the refractive index of the transparent surface, color granularity, and viewing angle (Schechner et al., ). To enhance the robustness against variations in reflection strength, we further endow the model with an adaptive transmission-rate-aware prompt generator.

In light of these considerations, this paper proposes a network, called Reversible Decoupling Network (RDNet for short).
The major technical contributions of this work are twofold:

- •
We revisit the preservation and cross-level interaction problems of hierarchical semantic information during the single image reflection removal/separation. To address the challenges, we introduce a multi-column reversible encoder based on the part-whole hierarchy, complemented by a tailored hierarchy decoder. This design ensures a better retention of rich semantics, effectively mitigating the ill-posed nature of the SIRR task.
- •
To tackle the varied reflection parameters in real-world scenarios, we introduce an adaptive transmission-rate-aware prompt generator, which learns channel scaling factors from the dataset during training and leverages this knowledge as a prior when testing. It guides the decomposition network in selecting more accurate transmission-reflection ratios, significantly enhancing the model’s generalization capabilities.

Extensive experiments are conducted to verify the efficacy of our design, and reveal its superiority over other SOTA alternatives both qualitatively and quantitatively (see Fig. 1 for a brief summary). Notably, our approach also achieves robust generalization on in-the-wild cases, underscoring its practical value in real-world applications (shown by Fig. 4).

## 2 Related Work

### 2.1 Single Image Reflection Removal

Physical formulation.
In prevalent reflection removal frameworks (Levin & Weiss, ), an image $I$ is typically decomposed into transmission $T$ and reflection $R$ components, so as to $I=T+R$. However, in real-world scenarios, these two layers may be attenuated by factors such as diffusion and other environmental influences during superposition (Wan et al., ). To account for such complexities, an augmented modeling has been proposed: $I=\alpha T+\beta R$, where the coefficients $\alpha$ and $\beta$ provide adaptability to varying conditions (Wan et al., ; Yang et al., ). Nonetheless, the assumption of linear superimposition often breaks down, particularly in cases of overexposure (Wen et al., ). To address this concern, the concept of an alpha-matting map $W$ is incorporated, leading to a reformulation of the model as $I=W\circ T+\overline{W}\circ R$ with $\overline{W}=1-W$. While the adjustment improves the model’s flexibility, it also increases the complexity of the already ill-posed problem.

The above model struggles to encapsulate the diverse reflection phenomena, highlighting the challenge of developing a universal solution. Hu and Guo (Hu & Guo, ) offered a more comprehensive depiction of the superimposition process by introducing a residual term: $I=\tilde{T}+\tilde{R}+\phi(T,R)$, where $\tilde{T}$ and $\tilde{R}$ signify the altered transmission and reflection information within $I$ after superimposition and degradation, as captured by camera sensors. The term $\phi(T,R)$ denotes the residual information in the reconstruction, arising from factors such as attenuation and overexposure. However, current methods primarily use the above modelings to synthesize training data, expecting the generalizability to real-world data. But, they lack explicit estimation of the physical parameters involved. Furthermore, distance-based loss functions such as mean absolute error (MAE) and mean squared error (MSE) fail to account for global color and intensity shifts. Explicitly estimating the degradation rate of the projected image could improve performance. A more detailed explanation is provided in Sec. 3.2.

Deep-learning-based modeling.

Zhang et al. (Zhang et al., ) enhanced semantic awareness by leveraging hypercolumn features extracted from a pre-trained VGG-19 network (Hariharan et al., ), together with perceptual and adversarial losses. ERRNet (Wei et al., ) uses misaligned pairs as training data to take a step further. But it overlooks the reflection layer, potentially increasing ambiguity in transmission recovery. Li et al. (Li et al., ) proposed RAGNet, a two-stage network that initially estimates the reflection component and then uses it to guide transmission prediction. Recently, the YTMT strategy proposed in (Hu & Guo, ) treats both components equally through a dual-stream interactive network that restores both layers simultaneously. Yet, noticing the problem hidden in the physical formulation, their interaction module relies on a linear assumption, which may upper-bound its performance. Other methods, such as BDN (Yang et al., ) and IBCLN (Li et al., ) employ reflection models with scalar weights to iteratively estimate both components, ensuring that the reflection is not too faint. However, the interaction between the two components is ignored, sometimes leading to heavy ghosting effect in transmission and reflection.
Dong et al. (Dong et al., ) developed an iterative network that estimates a probabilistic reflection confidence map at each step. DSRNet (Hu & Guo, ) introduces a mutually gated interaction mechanism within a two-stage structural design. In the first stage, the network progressively fuses extracted hierarchical features, while the second stage focuses on further decomposing these features. However, the issue of information loss persists due to the multiplicative reductions in the gating mechanism. Additionally, the progressive hierarchical fusion, isolated in the first stage, does not fully ensure that the hierarchical information is preserved during the subsequent decomposition processes. Zhu et al. (Zhu et al., ) proposed a maximum reflection filter for estimating reflection locations and introduce a large dataset, but they similarly overlook interaction between the two layers. Our proposed RDNet addresses the drawbacks of existing approaches by incorporating reversible connections and a multi-column design.

### 2.2 Reversible Network

Reversible neural networks are designed to prevent information loss by enabling the recovery of original inputs from outputs, thereby maintaining data integrity. Deco and Brauer (Deco & Brauer, ) introduced a reversible architecture that guarantees data preservation through a residual design, which generates a lower triangular Jacobian matrix with unity diagonal elements. Building upon this concept, Dinh et al. (Dinh et al., ) developed the NICE framework, employing a non-linear bijective transformation between the data and a latent space. However, this design only allows volume-preserving mappings. Dinh et al. (Dinh et al., ) extended extended this idea by proposing a reversible transformation that does not require volume preservation. While Gomez et al. (Gomez et al., ) combined the concept of invertible networks with the ResNet architecture, ensuring that each layer’s activations can be derived from the subsequent layer’s activations. This manner enables backpropagation without storing the activations in memory, except for a few non-reversible layers.

Reversible Networks for Low-level Vision. Reversible CNNs have been effectively applied to various low-level tasks, including compression (Liu et al., ), enhancement (Zhu et al., ; Wang et al., ; Li et al., ) and restoration (Huang & Dragotti, ; Zhu et al., ; Yao et al., ). These solutions typically employ reversible networks as a shared encoder-decoder in a generative manner, where new textures are generated to supplement ost information during degradation. However, in the task of reflection removal, the target result (the transmission image) is mixed with the reflection rather than lost. This task requires precise decoupling of the input image components instead of generating new textures. To the best of our knowledge, our work is the first to design a reversible architecture specifically for reflection removal.

## 3 Methodology

In this section, we present the key components of the proposed RDNet, the overall structure of which is schematically depicted in Fig. 2. Specifically, it is composed of three primary modules: the multi-column reversible encoder (MCRE), transmission-rate-aware prompt generator (TAPG) and the hierarchy decoder (HDec). The Pretrained Hierarchy Extractor (PHE) captures semantically rich hierarchical representations from the input image and transmits them to each level of the first column in MCRE. Meanwhile, TAPG learns channel-level transmission-reflection ratio priors from the data, mapping these learned fundamental parameters into prompts that guide the MCRE network. Finally, each column in MCRE employs an HDec to encode the hierarchical information, providing effective side guidance (Qin et al., ). The decoded hierarchies from the last column yield the final results.

Figure: Figure 2: Overall structure of our RDNet, the input is fed in the transmission-rate-aware prompt generator, pretrained hierarchy extractor, and the column embedding. The output of the prompt generator will be transferred into the column network. After interactions between the columns, each column uses a separate decoder to obtain a pair of image layers.
Refer to caption: extracted/5916962/fig_main_arch_v2.jpg

### 3.1 Multi-scale Reversible Column Encoder

As shown in Fig. 2, our proposed Multi-Column Reversible Encoder (MCRE) employs an architecture that differs from end-to-end models (Zhang et al., ; Wei et al., ) by incorporating multiple sub-networks, each receiving column embeddings modulated by the Transmission-rate-Aware Prompt Generator (TAPG). The model is composed of a Column Embedding Layer and multiple columns that encode multi-scale information.

In MCRE, information propagation between columns is handled through two primary mechanisms: intra-level reversible connections (denoted by blue solid lines in the figure) that facilitate information preservation between columns at the same level, and inter-level connections (illustrated as red dashed lines) paired with Bidirectional Interaction Levels, enabling interactions across adjacent levels. This approach effectively decouples multi-scale features up to Level-3. As an exception, Level-4 lacks corresponding cross-level connections, conforming to the structure of the End Level. The initial column within MCRE accepts the hierarchical information extracted by the PHE, ensuring a semantic-rich representation. The subsequent multi-column reversible design ensures the lossless propagation of hierarchical information throughout the decomposition network.

Specifically, our column embedding layer employs a $7\times 7$ convolution layer with a stride of 2, producing $2\times 2$ overlapping patches $F_{-1}$ for subsequent processes.
For the $i$-th column ($i\in\{1,2,...,N\}$), each level feature $F_{j}^{i},j\in\{0,1,2\}$ receives information $F_{j-1}^{i}$ from the lower level of the current column and $F_{j+1}^{i-1}$ from a higher level of the previous one. The collected features are further fused with the signal $F_{j}^{i-1}$ of the current level. The operation described above for the level $j$ is expressed as:

$$ $\quad F_{j}^{i}=\omega(\theta(F_{j-1}^{i})+\delta(F_{j+1}^{i-1}))+\gamma F_{j} ^{i-1}.$ (3) $$

where $\omega$ denotes the network operation, while $\theta$ and $\delta$ represent downsampling and upsampling operations, respectively. The $\gamma$ term is a simple reversible operation. In our implementation, we utilize a learnable reversible channel-wise scaling as the reversible operation $\gamma$.  This connection is information lossless, as one can retrieve $F_{j}^{i-1}$ through the reverse operation:

$$ $\quad F_{j}^{i-1}=\gamma^{-1}\left[F_{j}^{i}-\omega(\theta(F_{j-1}^{i})+\delta (F_{j+1}^{i-1}))\right].$ (4) $$

Notably, for the first level of each column, we define $F_{-1}^{i}:=F_{-1}$. Moreover, since the last level does not receive any higher-level features, the $\delta(F_{j+1}^{i-1})$ term is hence discarded.

Hierarchy Decoder. Our hierarchy decoder integrates hierarchical codes from all scales to generate the final output. We leverage several Level Decoders (LD) to interpret the higher-dimensional hierarchies with smaller resolution into lower-dimensional ones at larger resolution. The up-sampling operator in an LD is implemented by pixel-shuffle (Shi et al., ), an information-consistent operator before and after the scaling. The up-sampled features are then fused with the information from the previous scale with multiplication modulation. Ultimately, the final LD produces the layer residuals ($\hat{T}_{res}$ and $\hat{R}_{res}$) through another pixel-shuffle up-sampling operation and are connected with the original input to obtain the layer decomposition $\hat{T}$ and $\hat{R}$.

### 3.2 Transmission-rate-Aware Prompt Generator

Previous methods for SIRR often exhibit limited generalization capabilities due to the inherent complexity and variability of optical factors in real-world reflective scenarios, compounded by the constraint of limited training data. This limitation can be observed in the real-world test samples we collected, as shown in Fig. 4. Meanwhile, in both real-world and synthesized data, color/intensity is often compromised due to the reflection overlaying the transmission, with the transmission $T$ itself being degraded by a transmission rate $a$. In image restoration tasks, the ground truths are typically clean images. But, linearly deviated input/result often occurs because of color/illumination shifts in the real-world scenarios. The phenomenon is further detailed in the appendix A.

To solve the aforementioned problems, we develop a transmission-rate estimator using a simplified version of the ConvNext model (Liu et al., ) pre-trained on ImageNet-1k (Deng et al., ). Given an input image $I\in\mathbb{R}^{3\times H\times W}$, our transmission-rate estimator predicts six parameters: $\alpha_{\{R,G,B\}},\beta_{\{R,G,B\}}$ such that $\|\alpha_{i}T+\beta_{i}-I\|_{2}$ is minimized for each $i\in\{R,G,B\}$.
When testing the input image using the six parameters generated by the prompt generator, we can obtain an average PSNR of 24.34dB across four benchmark datasets (Real20, Objects, Postcard, and Wild), surpassing the previous state-of-the-art method by Dong et al. (Dong et al., ). This result confirms the effectiveness of our estimated transmission rate.

Once the transmission rate factor $\alpha_{\{R,G,B\}},\beta_{\{R,G,B\}}$ is estimated, a three-layer MLP is used to generate prompts that guide the MRCE, resulting in a prompt $P\in\mathbb{R}^{C\times H\times W}$, where $C$ represents the output dimension of the patch embedding layer, set to 64 in our work. Subsequently, the prompt is used to modulate the intermediate features from the column embedding layer $F$ into $P\circ F$, which allows the network to dynamically adapt to the specific characteristics of each input image, thereby enhancing the accuracy of reflection removal.

### 3.3 Training Objective

Our model undergoes two training stages. In the first stage, we train the estimator for the transmission rate. Once this is complete, we fix the classifier and proceed to train the main model along with the prompt generator. This training scheme ensures that both the transmission-rate-aware prompt generator and the main model work harmoniously towards the task, resulting in a robust solution.

We employ content loss and perceptual loss for the task, evaluating each pair of images produced by each column using the following loss functions before aggregating them into the final outcome.

Content Loss. The content loss ensures consistency between the output images and the ground truth training data. In the image domain, we adopt the Mean Squared Error (MSE) loss. Following previous works (Hu & Guo, ; ), we further regularize the model by encouraging consistency between the output and ground truth in the gradient domain, which writes:

$$ $\displaystyle\mathcal{L}_{\text{cont}}:=c_{0}\|\hat{T}-T\|^{2}_{2}+c_{1}\|\hat {R}-R\|^{2}_{2}+c_{2}\|\nabla\hat{T}-\nabla T\|_{1},$ (5) $$

where $\|\cdot\|_{1}$ and $\|\cdot\|_{2}$ stand for the $\ell_{1}$ and $\ell_{2}$ norms, respectively. During the first stage of training, we set $c_{0}=1,c_{1}=0,c2=0$. In the second stage, these values are adjusted to $c_{0}=0.3$, $c_{1}=0.9$, $c_{3}=0.6$.

Perceptual Loss. To enhance the perceptual quality of images produced by our model, we minimize, we minimize the $\ell_{1}$ discrepancy between the features of predicted elements and the ground-truth references. This comparison is made at the ‘conv2_2’, ‘conv3_2’, ‘conv4_2’, and ‘conv5_2’ layers of a pre-trained VGG-19 network on the ImageNet dataset. Denoting the features at the $i$th layer as $\phi_{i}(\cdot)$, the perceptual loss is computed as:

$$ $\mathcal{L}_{\text{per}}:=\sum_{j}\omega_{j}\|\phi_{j}(\hat{T})-\phi_{j}(T)\|_ {1},$ (6) $$

where $\omega_{j}$ are weighting coefficients for each layer. The total loss turns out to be:

$$ $\mathcal{L}:=\mathcal{L}_{\text{cont}}+w\mathcal{L}_{\text{per}},$ (7) $$

where $w=0.01$ is empirically set.

## 4 Experimental Validation

### 4.1 Implementation Details

Our model is implemented in PyTorch (Paszke et al., ) and optimized with Adam optimizer (Kingma & Ba, ) on an RTX 3090 GPU for 20 epochs. The learning rate is initialized at $10^{-4}$, and remains fixed throughout the training phase, with a batch size of 2. The training dataset comprises both real and synthetic images. To align with previous works, we evaluate the performance of our model under two commonly used data settings: a) The setting from (Hu & Guo, ; Wei et al., ) and (Li et al., ), which consists of 90 real image pairs from (Zhang et al., ) and 7,643 synthesized pairs from the PASCAL VOC dataset (Everingham et al., ); and b) The setting from (Hu & Guo, ) and (Dong et al., ), which includes 200 additional real image pairs provided by (Li et al., ). For data synthesizing, we follow the pipeline and physical model from DSRNet (Hu & Guo, ), represented by $I=\alpha T+\beta R-T\circ R$. Slightly, we modify this approach by sampling individual $\alpha$ and $\beta$ for R, G, and B channels. This adjustment aims to prevent the transmission rate estimator from converging to a trivial solution. The parameters the PHE are initialized by a pretrained FocalNet (Yang et al., ).

**Table 1: Quantitative results of various methods on four real-world benchmark datasets. The best results are highlighted in bold, and the second-best results are underlined.**
|  | Methods | Real20 (20) | Objects (200) | Postcard (199) | Wild (55) | Average |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM |  |
| w/o Nat. | ERRNet | 22.89 | 0.803 | 24.87 | 0.896 | 22.04 | 0.876 | 24.25 | 0.853 | 23.53 | 0.879 |
| IBCLN | 21.86 | 0.762 | 24.87 | 0.893 | 23.39 | 0.875 | 24.71 | 0.886 | 24.10 | 0.879 |  |
| RAGNet | 22.95 | 0.793 | 26.15 | 0.903 | 23.67 | 0.879 | 25.53 | 0.880 | 24.90 | 0.886 |  |
| YTMT | 23.26 | 0.806 | 24.87 | 0.896 | 22.91 | 0.884 | 25.48 | 0.890 | 24.05 | 0.886 |  |
| DSRNet | 24.23 | 0.820 | 26.28 | 0.914 | 24.56 | 0.908 | 25.68 | 0.896 | 25.40 | 0.905 |  |
| Ours | 24.43 | 0.835 | 25.76 | 0.905 | 25.95 | 0.920 | 27.20 | 0.910 | 25.95 | 0.908 |  |
| w Nat. | Dong et al. | 23.34 | 0.812 | 24.36 | 0.898 | 23.72 | 0.903 | 25.73 | 0.902 | 24.21 | 0.897 |
| DSRNet | 23.91 | 0.818 | 26.74 | 0.920 | 24.83 | 0.911 | 26.11 | 0.906 | 25.75 | 0.910 |  |
| Zhu et al. | 21.83 | 0.801 | 26.67 | 0.931 | 24.04 | 0.903 | 26.49 | 0.915 | 25.34 | 0.912 |  |
| Ours | 25.58 | 0.846 | 26.78 | 0.921 | 26.33 | 0.922 | 27.70 | 0.915 | 26.65 | 0.917 |  |

**Table 2: Quantitative results on the Nature dataset. The competitors are all trained with the additional data from the Nature dataset**
|  | ERRNet | IBCLN | YTMT | DSRNet | Zhu et al. | Ours |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR | 22.18 | 23.57 | 23.85 | 25.22 | 26.04 | 26.21 |
| SSIM | 0.756 | 0.783 | 0.810 | 0.832 | 0.846 | 0.842 |

### 4.2 Performance Evaluation

For the comparison, we evaluate seven state-of-the-art methods: ERRNet (Wei et al., ), IBCLN (Li et al., ), RAGNet (Li et al., ), Dong et al. (Dong et al., ), YTMT (Hu & Guo, ), DSRNet (Hu & Guo, ), Zhu et al. (Zhu et al., ), on four real-world datasets, including Real20 (Zhang et al., ) and three subsets of the $\text{SIR}^{2}$ Datasets (Wan et al., ), for the Nature Dong et al. () dataset, we compare IBCLN, ERRNet, YTMT, DSRNet and Zhu et al..

Quantitative comparisons. The quantitative result is shown in Tab. 1. We directly employ the code and pre-trained weights publicly provided by their authors to obtain all the quantitative results.
To make a fair comparison, the methods with and without additional data from the Nature dataset are compared separately. Apparently, our methods show their superiority over other competitors on all testing datasets, only falling short on SSIM compared to Zhu et al. on the Objects dataset. Our methods achieved a promising boost, especially on the Real20 dataset, which contains hard cases collected in real-world conditions, meaning our method can better fit real-world conditions. The other three datasets contain a variety of scenes, illumination conditions, and glass thickness, meaning our method performs better in most conditions. The experimental result demonstrates that our proposed SIRS method can adapt to complicated situations and has a stronger generalization ability.

For a comprehensive comparison, we present the results obtained on the Nature dataset in Tab. 2, which comprises 20 real-world samples. Our method achieved the best PSNR and the second-best SSIM, with a marginal decrease of only 0.004 in SSIM. These results further underscore the superiority of our approach in real-world scenarios.

Figure: (a) Input
Refer to caption: extracted/5916962/fig/wild_c1/stitch_images/0input-4loca.png

Figure: (a) Input
Refer to caption: extracted/5916962/fig/ourreal2/stitch_images/0input-4loca.png

Qualitative comparisons. The qualitative comparison is shown in Fig. 3 and Fig. 4, with additional visual examples provided in the appendix. The first case in Fig. 3 illustrates a highly reflective object, which presents a significant challenge for reflection removal techniques due to its intensity. Our method successfully eliminates the reflective object, accurately revealing the underlying texture and color information. This performance is superior to other methods, highlighting our approach’s effectiveness in handling complex real-world reflections. In contrast, ERRNet, RAGNet, Dong et al., YTMT, DSRNet and Zhu et al. struggle to remove the object, leaving it almost entirely intact. Although IBCLN partially removes the reflection, it fails to recover the underlying color information, resulting in an incomplete outcome. This example clearly demonstrates our method’s advanced capability in accurately identifying and removing even strong and complex reflections, further proving its robustness in real-world scenarios.

The second example further showcases our method’s proficiency in handling reflections spread across an image. Here, the reflection is complex and covers a large area, which other methods fail to remove effectively. In contrast, our approach accurately targets and eliminates the majority of the reflection, preserving the integrity of the non-reflective elements.

Figure 4 illustrates the robustness of our method in real-world scenarios. These two cases were captured in real-life conditions by us. In the first example, a dense reflection covers the car window, a challenge that competing methods largely fail to address, with only Zhu et al. managing partial removal. However, our approach almost entirely separates the reflections, producing more visually appealing results. A similar outcome is observed in the second example, where our method successfully removes nearly all reflections. In contrast, all other methods struggle to handle this scenario effectively. These examples demonstrate the robustness of our decoupling paradigm, confirming its effectiveness in real-world scenarios.

These results demonstrate the effectiveness of our decoupling routine, offering several key advantages: 1) accurate identification and separation of reflection components from underlying content, 2) robust performance in removing dense reflections common in real-world scenarios, and 3) strong generalizability across diverse conditions. Collectively, these findings validate the theoretical soundness and practical efficacy of our proposed method.

### 4.3 Ablation Studies

To better verify the effect of our prompt generator and reversible network structure, we bring a series of ablation studies, including different settings of network structure and prompt generator. The results are gathered in Tab. 3. We present the results of our prompt generator on the left side and the results of our network structure on the right side.

**Table 3: Ablation studies on the prompt generator and different network configurations.**
| Setting | Prompt | Pre. | Average |  | Setting | Dual-stream | Ref. Loss | Invertibility | Average |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR | SSIM |  | PSNR | SSIM |  |  |  |  |  |  |  |
| A | $\times$ | $\times$ | 25.52 | 0.909 |  | D | $\checkmark$ | $\checkmark$ | $\checkmark$ | 26.37 | 0.917 |
| B | $\times$ | $\checkmark$ | 25.99 | 0.910 |  | E | $\times$ | $\times$ | $\checkmark$ | 25.99 | 0.914 |
| C | $\checkmark$ | $\checkmark$ | 26.03 | 0.913 |  | F | $\times$ | $\checkmark$ | $\times$ | 24.05 | 0.884 |
| Ours | $\checkmark$ | $\times$ | 26.65 | 0.917 |  | Ours | $\times$ | $\checkmark$ | $\checkmark$ | 26.65 | 0.917 |

Discussion on transmission-rate-aware prompt generator.
To inform the model with the transmission rate, a straightforward approach is to adjust the input image to enhance it globally using the estimated transmission rate. Specifically, for $I:=aT+bR+\phi(T,R)$, we adjust the input $I$ to $\frac{1}{a}I:=T+\frac{b}{a}R+\frac{1}{a}\phi(T,R)$. This operation is denoted as Pre. in Tab. 3. As shown in Tab. 3, if we remove all transmission-rate-aware techniques (setting A), the average performance drops by 1.13 dB. If we adopt the straightforward method described above (setting B), the performance recovers by 0.47 dB. This confirms the importance of informing the model with the transmission rate.

However, as we analyzed in Section 3.2, directly adjusting the input image is far from optimal. Due to potential inaccuracies in the estimation in some scenarios, directly adjusting the model can introduce an additional shift that is difficult to correct during second-stage training. A more subtle and flexible approach is to reweight the feature channels with our transmission-rate-aware prompt.

To verify this, we both adjust the input and add a transmission-rate-aware prompt to the feature (setting C). The performance remains nearly the same as in Setting B, indicating that adjusting the input makes it challenging for the model to recover from incorrect estimations. Finally, our model with the proposed transmission-rate-aware prompt outperforms all variants, demonstrating its efficacy.

Discussion on model design.
To verify the rationality of our design of the decoupling model, we created three new variants of our model. We modify our RDNet to a DSRNet-style one, where two streams estimate transmission and reflection separately in a single column, and interact with each other. This variant is denoted as Dual-stream (Setting D). As shown in Tab. 3, even with double computation, the performance still drops by 0.28 dB. This confirms the superiority of our decoupling design compared to the dual-stream design. Secondly, we removed the reflection part ($c_{1}\|\hat{R}-R\|^{2}_{2}$) in the content loss function (Eq. (5)), leaving only the transmission part ($c_{0}\|\hat{T}-T\|^{2}_{2}+c_{2}\|\nabla\hat{T}-\nabla T\|_{1}$) in the training process. This variant is denoted as Ref. Loss. (Setting E). A performance drop of 0.66dB can be observed. This confirms the necessity of the reflection loss function. Without regularization predicting the other component, the network weakens its ability to clearly identify both components in single-stream feature maps.

To verify the necessity of the invertibility of the network in the reflection removal task, we replace the reversible connection with the U-Net connection (Ronneberger et al., ) (Setting F). Although it requires slightly more parameters and much more memory, a massive performance drop of 2.6 dB can be discovered, indicating the importance of invertibility design.

## 5 Conclusion

In this paper, we proposed RDNet, a novel model for addressing key challenges in the task of single image reflection removal. Specifically, RDNet tackles the limitations of insufficient utilization of multi-scale, pretrained hierarchical information and information loss during feature decoupling. The multi-column reversible structure enables the preservation of rich semantic features, which are then effectively leveraged in the multi-scale processing of each column. Furthermore, the proposed Transmission-rate-Aware Prompt Generator alleviates the inherent conflict between complex reflection parameters and limited training data. Through these innovations, RDNet demonstrates an enhanced capability for robust reflection removal. Our method demonstrates superior performance compared to state-of-the-art techniques across a range of real-world benchmark datasets, highlighting its robustness and adaptability in diverse reflective scenarios. Ablation studies further validate the effectiveness of our key contributions, confirming the advantages of our design choices. It is positive that our work opens up new avenues for research in reflection removal, and has the potential to impact various applications in computer vision and image processing significantly.

#### Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant nos. 62372251 and 62072327. Mingjia Li is also supported by Cloud TPUs from TPU Research Cloud (TRC).

## Appendix

## Appendix A Further discussion on the TAPG

Figure: Figure 5: Visualization of the drawback of Mean Squared Error (MSE). Both color shifts and noise degradation exhibit the same MSE relative to the ground truth.
Refer to caption: extracted/5916962/9af594b6e4b380c0cd289176c8a4dae.png

As illustrated in Fig. 5 with a toy visualization of a pure white image, even though the color bias and noise exhibit the same Mean Squared Error (MSE) relative to the ground truth, a linear estimation can instantly correct the image’s color shift, whereas noise requires more complex operations to address. Metrics like MSE and Mean Absolute Error (MAE) struggle to compel the network to effectively recognize and rectify the linear degradation in the physical formulations.
In this context, by pre-calibrating the features with a transmission-rate-aware prompt, we can significantly mitigate the effects of linear degradation, such as color and intensity inconsistencies.

## Appendix B Qualitative Comparisons

More visual cases. We exhibit a total of nine additional cases: two cases from the Real20 dataset in Fig. 9, three cases from the Solid dataset in Fig.7, two cases from the Postcard dataset in Fig. 8 and two real-world cases captured by us in Fig. 6. As illustrated, our method excels at revealing the information obscured by reflections and is highly effective in removing the majority of the reflections.

**Table 4: The experiment of changing numbers of columns. The best results are indicated in bold.**
| Num Col | Average |  |
| --- | --- | --- |
| PSNR | SSIM |  |
| 2 | 26.25 | 0.914 |
| 4 | 26.65 | 0.917 |
| 6 | 26.19 | 0.910 |

## Appendix C Additional Experiments

The ablation study for the number of columns. In this study, we investigate the effect of varying the number of columns on the overall performance in Tab. 4. Specifically, we adjusted the number of columns after the first PHE column, experimenting with configurations of 2, 4, and 6 columns. Our findings indicate that a configuration with 4 columns yields the highest performance. In contrast, configurations with 2 and 6 columns resulted in performance drops of 0.4dB and 0.46dB in PSNR, respectively. This suggests that an optimal balance exists, where too few or too many columns can detract from the model’s performance.

Figure: (a) Input
Refer to caption: extracted/5916962/fig/ourreal1/stitch_images/0input-4loca.png

Figure: (a) Input
Refer to caption: extracted/5916962/fig/14_case_new/stitch_images/0.png

Figure: (a) Input
Refer to caption: extracted/5916962/fig/sup_case4/stitch_images/0.png

Figure: (a) Input
Refer to caption: extracted/5916962/fig/10case_new/stitch_images/0input-4Loca.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Does the paper analyze failure cases, particularly in challenging situations with complex or severe reflections? Would such an analysis provide a more comprehensive understanding of the model's limitations?
2. Does the study address whether the method can effectively handle high-resolution images, particularly in real-world applications where reflection removal often requires high-resolution processing (e.g., 4K or 8K images)?
3. For other questions,  please refer to the part of the weakness.
soundness: 3
strengths: 1. The manuscript is clearly written and easy to follow.
2. The ablation study convincingly demonstrates the effectiveness of the proposed methods.
3. The proposed method achieves improved performance over existing approaches.
summary: This work addresses the problem of single-image reflection removal, introducing a novel Reversible Decoupling Network (RDNet) as its core approach. The RDNet is built around two key components: a reversible encoder, which preserves crucial information, and a transmission-rate-aware prompt generator, which dynamically calibrates features to improve effectiveness. Experimental results demonstrate that the proposed network achieves superior performance compared to existing methods.
weaknesses: Novelty:

The design of this network is not particularly novel. It appears to be a combination of existing methods. For instance, components such as the multi-scale reversible column encoder, hierarchical decoder, and transmission-rate-aware prompt generator have been utilized in previous works such as HRNet, and ConvNext.


Computational Efficiency: 

Given that the network is relatively complex, there’s a lack of discussion around computational efficiency, which is important for real-time or resource-constrained applications. Including information on memory usage, inference speed, and model scalability would be useful for practical implementation.


Artifacts and Color Distortions Problem:

Figures 3 and 7 exhibit artifacts and color distortions in the restored images. For example, some content are changed compared to GT images.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. The PSNR and SSIM primarily focus on pixel-wise evaluation. The authors should also include LPIPS for a more comprehensive perceptual assessment.

2. It is important to provide comparisons regarding FLOPs and parameters. Additionally, running time is a critical factor for real-world applications, so an analysis of the proposed method’s running time should be included.

3. The experimental comparisons should be supplemented with the latest state-of-the-art methods. The authors should also clarify why the results for these comparison methods are comparatively low.
soundness: 2
strengths: 1. Overall, the writing of this manuscript is good and easy to follow. 
2.  The ablation study can prove the effectiveness of the proposed methods. 
3.  The proposed method outperforms current methods.
summary: This work focuses on the problem of single-image reflection removal. Its core insight is the proposed Reversible Decoupling Network. The proposed network's core components include a reversible encoder and a transmission-rate-aware prompt generator. The reversible encoder is designed to preserve important information, and the transmission-rate-aware prompt generator dynamically calibrates features to enhance overall effectiveness. The results show that the proposed network performs better than the existing methods.
weaknesses: 1. The designs of this network are not very new. The proposed network looks more like a complication of existing methods. For example, the multi-scale reversible column encoder, the hierarchy decoder, and the transmission-rate-ware prompt generator have been used in existing methods (e.g., HRNet, ConvNext model).
2. The proposed network looks complex.  There is a lack of comparison in FLOPs and Params in the main comparison (Table 1) and ablation study (Table 3).
3. The paper says " Our method demonstrates superior performance compared to state-of-the-art techniques across a range of real-world benchmark datasets" However, the experimental results of the proposed network were not compared with the latest SOTA methods, such as  "Revisiting Singlelmage Reflection Removal in the Wild (CVPR 2024). " 
4. In Figure 3 and Figure 7, there are some artifacts and color distortion in the restored images. For example, the shoes are very different compared to the ground truth.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1.The methods section is not clear enough, for example, line 243 “Bidirectional Interaction Level”, but there is no further explanation and no reference to related work. What is the motivation for using the Bidirectional Interaction Level?
2.It would also be helpful to see results on a unified dataset, including visualizations. Could the authors provide results for the specified methods under consistent training and testing conditions? Such as:
a. Johnson, Justin, Alexandre Alahi, and Li Fei-Fei. "Perceptual losses for real-time style transfer and super-resolution." Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. Springer International Publishing, 2016.
b.Wen, Qiang, et al. "Single image reflection removal beyond linearity." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
c.Kim, Soomin, Yuchi Huo, and Sung-Eui Yoon. "Single image reflection removal with physically-based training images." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
c.Dong, Zheng, et al. "Location-aware single image reflection removal." Proceedings of the IEEE/CVF international conference on computer vision. 2021.
d.Song, Zhenbo, et al. "Robust single image reflection removal against adversarial attacks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
e.Wang, Mengyi, et al. "Personalized single image reflection removal network through adaptive cascade refinement." Proceedings of the 31st ACM International Conference on Multimedia. 2023.
soundness: 3
strengths: 1.This work presents RDNet, which incorporates a multi-column reversible encoder to effectively preserve multi-scale semantic information. By decoupling transmission and reflection features, RDNet minimizes information loss during feature interactions, significantly enhancing reflection removal accuracy.
2.The proposed transmission-rate-aware prompt generator dynamically adjusts feature representations by learning channel scaling factors from the data. This design allows RDNet to achieve strong generalization and robustness across various real-world reflection scenarios.
3.The proposed method outperforms state-of-the-art methods in reflection removal both qualitatively and quantitatively.
summary: This paper introduces the Reversible Decoupling Network (RDNet), a novel approach to single-image reflection removal that overcomes key limitations in existing methods. RDNet features a multi-column reversible encoder that preserves hierarchical semantic information by decoupling transmission- and reflection-related features, thus preventing information loss during feature interactions across scales. Additionally, an adaptive transmission-rate-aware prompt generator dynamically adjusts features by learning channel scaling factors from data, which enhances RDNet's generalization and robustness in real-world reflection scenarios.
weaknesses: 1.The Bidirectional Interaction Level in Figure 2 could benefit from clearer explanation, as the current description in the text is brief and may lead to misunderstandings.
2.The paper compares the proposed method with several existing reflection removal techniques. However, distinct datasets are used in the comparative experiments, which is unnecessary. Additionally, expanding the range of comparison methods would ensure a more comprehensive evaluation, as the current selection may not sufficiently demonstrate the method's effectiveness.