Single Image Reflection Separation via Component Synergy
Qiming Hu Xiaojie Guo*
College of Intelligence and Computing, Tianjin University, Tianjin, China
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>, <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Abstract
The reflection superposition phenomenon is complex and
widely distributed in the real world, which derives various
simplified linear and nonlinear formulations of the problem.
In this paper, based on the investigation of the weaknesses
of existing models, we propose a more general form of
the superposition model by introducing a learnable residue
term, which can effectively capture residual information
during decomposition, guiding the separated layers to be
complete. In order to fully capitalize on its advantages, we
further design the network structure elaborately, including
a novel dual-stream interaction mechanism and a powerful
decomposition network with a semantic pyramid encoder.
Extensive experiments and ablation studies are conducted
to verify our superiority over state-of-the-art approaches on
multiple real-world benchmark datasets. Our code is pub-
licly available at https://github.com/mingcv/DSRNet.
1. Introduction
As a typical layer superimposition scenario, pictures
captured through glass-like surfaces may be blended with
undesired reflection, which not only impairs the aesthetic
value but also hinders the downstream tasks [31]. In the
meantime, scenes of interest are possibly concealed in re-
flection [35]. Therefore, both the parts transmitted through
a surface (transmission layer, T) and those reflected (reflec-
tion layer, R) are desired to be reconstructed from a super-
imposed image I to fulfill the practical demands.
Following a common assumption [17], I is linearly com-
posed by T and R expressed as I = T + R, which has been
popular due to its simplicity. However, in real-world sce-
narios, the reflection and transmission layers are likely to
be weakened due to diffusion and other problems during the
superimposition [35]. Therefore, several methods [34, 39]
introduce scalars α and β to the two components, respec-
tively, obtaining I = αT + βR. However, the linear su-
perimposition model is often violated due to over-exposure and other phenomena [37]. Thus, an alpha-matting map W
is introduced to the model as I = W ◦ T + ¯W ◦ R with
¯W = 1 − W, which, however, increases the degree of free-
dom and makes this ill-posed problem much harder. In view
of all these limitations, it is not a trivial task to represent dif-
ferent kinds of reflection scenarios with a single model. In
this work, we take a step forward in advancing the solution
and deliver a more general form of the superimposition pro-
cedure by introducing a residue term as follows:
I = ˜T + ˜R = T + R + Φ(T, R), (1)
where T and R denote the groundtruth scenes of the trans-mission and reflection layers, while their respective infor-
mation contained in I after superimposition and other degra-
dations and finally reach the camera sensors is represented
by ˜T and ˜R. Φ(T, R) = I − T − R is the residue of the
reconstruction, which can be caused by attenuation, over-
exposure, etc. Φ(·, ·) represents a group of functions that
can be used to model the residue in specific situations. For
example, Φ(T, R) = 0 in [23, 5, 17, 41, 36, 12], Φ(T, R) =
(α−1)T+(β−1)R in [34, 39], Φ(T, R) = −( ¯W◦T+W◦R)
in [37, 42]. In an effort to rule these different cases, a
learnable module is desired to be introduced to model the
residue Φ(T, R). To clarify it, we provide a visual exam-
ple in Fig. 1, where the first row shows a superimposed
image I, its groundtruth transmission T (GT) and reflec-
tion R (GT) layers collected by SIR2 dataset. According to
the linear model, the guidance for reflection restoration is
˜R = I − T with incomplete signals rather than R (GT) that
contains full information, which not only impairs the train-
ing of networks but also hinders the downstream usages.
Therefore, as we point out in Eq. (1), a promising way out
of this dilemma is to utilize ground-truth layers T and R as
guiding signals and employ an extra residual term to han-
dle the non-linearity (“Residue” in Fig. 1). As shown, the
components that violate the linear assumption are separated
from the layer predictions ˆT and ˆR, and thus more com-
plete and precious layer separations than previous methods
are obtained.
To further exploit the synergy between the components
T and R, we deliberate upon the facilitation of inter-
component feature interaction. The efficiency of a dual-
stream interactive network has been verified in SIRS prob-
lem by [12]. Building upon their analysis, we subse-
quently advance a more effective scheme. Inspired by the
gated mechanisms [2, 32], we present a simple yet effec-
tive mutually-gated interaction diagram (MuGI) serving as
a better feature interaction candidate, by means of which we
design the MuGI block to build our decomposition network.
Moreover, the hypercolumn [41], as a commonly used se-
mantic information encoder in the SIRS task, tends to ag-
gregate features in a lossy way. Therefore, we propose a
dual-stream pyramid fusion network (DSFNet) to replace
it, which hierarchically decomposes and fuses the multi-
scale semantic information leveraging our proposed MuGI
blocks and dual-stream fusion blocks (DSF Block). The
roughly decomposed features of layers are further delivered
into the dual-stream fine-grained decomposition phase (DS-
DNet). The two sub-networks constitute the main branch
of our Dual-stream Semantic-aware network with Residual
correction, namely DSRNet.
In summary, our main contributions are as follows:
• We build a general form for SIRS by introducing a
learnable residue term, which is more flexible to dif-
ferent scenarios and boosts the separation of layers; • We exploit the synergy of features via mutually-gated
interaction block within a dual-stream semantic-aware
network, which facilitates information usage;
• Extensive experiments are conducted to demonstrate
the effectiveness of our design. Overall, it achieves
state-of-the-art performance against the alternatives on
multiple real-world benchmarks for SIRS.
2. Related Work
Plenty of efforts have been devoted to handling the im-
age reflection separation problem in the past decades. They
either leverage multiple images to acquire more cues for the
layer decomposition or exploit extra priors to manage the
problem through a single image. In what follows, we orga-
nize the previous methods based on the images they require.
Multiple Image Reflection Separation. In general, the
transmission component tends to be unpolarized, yet the re-
flection component varies when rotating the polarization fil-
ter mounted in front of a camera sensor. Inspired by this
phenomenon, a variety of methods [28, 14, 13, 27, 21, 16]
resort to the physical solution that separates different com-
ponents through a sequence of images with varied polariza-
tion orientations. Besides the polarization cues, the layer
separations can also be indicated by different focuses [6],
stereo information [31], flash on/off [1, 15], relative mo-
tions [29, 8, 22, 38, 40, 25, 26] and scene consistency
[9, 30, 10]. Although involving multiple images mitigates
the ill-posedness of the separation problem, this pipeline re-
quires professional devices (such as polarizers) and manual
operations, which limits its application.
Single Image Reflection Separation. Compared
with multiple-image solutions, single-image-based methods
show more merits, being more flexible and requiring less
manual operations. However, as there is no such thing as
a free lunch, single-image schemes call for additional pri-
ors to cope with its ill-posedness nature, such as gradient-
based constraints [19, 18, 17, 23] and manual annotations
[17]. Specifically, Levin et al. [19, 18] impose sparse con-
straints on gradients to acquire the separations that have
fewer edges. Levin and Weiss [17] demand several man-
ual annotations of respective layers to obtain favorable de-
compositions, which involves additional human labor. Li
and Brown [23] suppose that one layer is smoother than the
other, thus applying unbalanced penalization on the gradi-
ents of the two layers to decompose them. Although these
methods work fine in restricted scenarios, the complicated
real-world conditions can considerably surpass their as-
sumptions, leading to unsatisfactory results.
The advent of deep learning technology mitigates the
incongruity between the volume of data and modeling ca-
pacity. Certain underlying assumptions can be subtly em-
bedded in the process of data synthesis, and subsequently learned by deep models. Concretely, CEILNet [5] imposes
the relative smoothness prior to the synthesis of reflection
layers and combines them with transmission layers by ad-
dition. An edge-aware network is developed to capture the
transmission components, which yet ignores high-level se-
mantics that are likely to aid the SIRS task. Zhang et al.
[41] hence introduce HyperColumn features [11] using a
pre-trained VGG-19 network to acquire semantic aware-
ness, besides the perceptual and adversarial losses. More-
over, the exclusivity loss is developed to penalize the in-
tersected gradients. ERRNet [36] goes a step further by
leveraging an additional set of misaligned pairs. How-
ever, it appears to overlook the estimation of the reflection
layer, which is essentially an image component and poten-
tially important to distinguish the transmission parts. Li et
al. [24] thereby presents a two-stage network (RAGNet) to
first estimate reflection components and predict transmis-
sion ones guided by them, whereas the reflection estima-
tion is isolated from the transmission in this way. Further,
Hu and Guo [12] come up with the YTMT strategy, which
pays equal attention to the two components, and a dual-
stream interactive network is developed to restore the two
layers simultaneously. However, their linear assumption
makes the predicted reflection components tend to be weak
in many cases. Other than them, BDN [39] and IBCLN [20]
make use of reflection models weighted by scalars, and it-
eratively estimate both the components. This scheme pre-
vents the reflection from being too weak, but its restorations
often struggle to be free from the transmission parts. Fur-
thermore, Wen et al. [37] simulate the nonlinear superim-
position phenomenon by predicting a three-channel alpha
blending weight map with the adversarial guidance of col lected unpaired images. Dong et al. [3] develops an iter-
ative network and estimates a probabilistic reflection con-
fidence map in each iteration. However, it is not an eas-
ier task to estimate an alpha blending map than to deter-
mine each of the blended components, while the residual
estimation only needs to retain the redundant information
during decomposition, which is more practical than the for-
mer. Considering the drawbacks of the previous reflection
models, we propose a general form of them with a learn-
able residue term, which is shown to be more effective and
flexible.
3. Methodology
As shown in Fig. 2, our proposed DSRNet comprises
two cascaded stages and a learnable residue module (LRM).
In what follows, we first clarify the motivation of LRM and
then introduce the MuGI block and the architecture of the
DSRNet followed by its training loss functions.
3.1. Residue of the Linear Combination
It often occurs that a superimposed image I cannot be
perfectly represented by the linear combination of T and R,
leading to a residue term, as discussed in Section 1. Un-
der its disturbance, the error of the additive reconstruction
criterion ε = ∥I − (ˆT + ˆR)∥ remains large even though the
predicted layers can well reconstruct the ground truths. Fur-
ther minimizing ε will put redundant information to either ˆT
or ˆR, which induces the deviation of predictions from their
ground truths instead. Moreover, there are many kinds of
simplified physical models proposed by previous work with
their own shortcomings. Naturally, a unified model is desired to be put forward. Therefore, we introduce an extra
residue term to offset the error in additive reconstruction,
which, at the same time, unifies the different physical mod-
els. As depicted in Fig. 2, we leverage a learnable residue
module (LRM) after Stage 2 to estimate the residual infor-
mation during the decomposition by analyzing the features
of ˆT and ˆR before the final layer. LRM consists of an in-
teractive and a fusion part to collect and merge residual in-
formation from two branches. The dual-stream signals are
concatenated before entering the fusion part. To constrain
the space of residue, the tanh(·) function is utilized as the
final activation. With the participation of LRM, we define
the following reconstruction loss with residual rectification
(R3 Loss):
Lrec := ∥I − (ˆT + ˆR) − Φ(ˆT, ˆR)∥1, (2)
where Φ denotes the LRM and ∥ · ∥1 means the ℓ1 norm.
Guided by this objective, the information beyond the addi-
tive model will flow to the residue term. Notably, this term
can be totally discarded during the testing phase, avoiding
extra computational costs.
3.2. Mutually-gated Interactive Block
Now that we employ the LRM to model the residual
components during the additive reconstruction, the rest part
can be seen as a linear model again as ˜I = I − Φ(ˆT, ˆR) =
ˆT + ˆR. As displayed in Fig. 3 (a), we exploit the syn-
ergy between ˆT and ˆR by developing the Mutually-Gated
Interactive mechanism, namely MuGI:
(ˆFT = G1(FT) ◦ G2(FR);
ˆFR = G1(FR) ◦ G2(FT), (3)
where FT, FR ∈ RH1×W1×C1 denote intermediate fea-
tures of streams for predicting ˆT and ˆR, respectively, and
ˆFT, ˆFR ∈ RH2×W2×C2 represent the outputs of the mu-
tual gate. G1 and G2 are functions selecting which part of
features to engage the interaction. ◦ indicates the element-
wise multiplication. Here we use a simple implementation
of MuGI to evaluate its capability of solving SIRS, which
divides a feature map in half at the channel dimension, and
G1(·) always selects the former half, while G2(·) chooses
the latter. We have H1 = H2, W1 = W2 and C2 = C1/2
in this way. Such a gated mechanism captures the mutual
dependency of the two components in the feature space, say
if either stream contains signals the other one desires, they
tend to be split into the latter half (selected by G2(·)) and
transferred into the sibling stream by the gate.
Our simple implementation of MuGI makes it plug-and-
play. Therefore, in general, most of the single-path blocks
can be converted into a dual-stream one by parallel setting
two of them and relating them with MuGI. In this paper,
we follow a state-of-the-art block design presented by [2] and convert its single-path NAF block into a dual-stream
mutually-gated interactive block (MuGI block). Its diagram
is depicted in Fig. 3 (b), in which the simple gates are re-
placed by MuGI gates. Specifically, the layer normalization
makes dual-stream features comparable, 1 × 1 convolution
doubles the channel dimensions in order to conduct MuGI
without information loss which reduces dimensions by half
during the interaction, channel attention and the following
1 × 1 convolution serve as fusion/reweighting roles.
3.3. Dual-stream Semantic-aware Network
The semantic information provided by pre-trained mod-
els is usually introduced to alleviate the impact of ill-
posedness in SIRS. The HyperColumn [41, 36, 12], as a
prevalent multi-scale semantic feature extractor in SIRS,
aggregates semantics through interpolating the features ex-
tracted by a pre-trained model into the same scale as the
input images. It then concatenates them together and em-
ploys a 1 × 1 convolution to rapidly reduce the channel di-
mensions (typically from 1475 to 64 or 256), before feed-
ing them into the decomposition networks. This strategy
omits the inner relationship of multi-scale features and may
discard informative signals. To overcome the drawbacks
mentioned above, we present a dual-stream pyramid fusion
network (DSFNet), which hierarchically aggregates the ex-
tracted features by jointly upsampling and interacting. As
shown in Fig. 2, given the hierarchical features extracted
by a pre-trained deep network (e.g., VGGNet-19), DSFNet
gathers them in a bottom-up manner with MuGI blocks and
dual-stream fusion blocks (DSF Block), the structure of
which is shown in Fig. 2. “Dual Conv” means two paral-
lel convolutional layers. The features extracted by the pre-
trained network at each level first interact through a MuGI
block, thereby features in the same scale are related and
transformed into dual-stream features (FT = FR for these
blocks). After the interaction, the deepest features extracted
by VGGNet are upscaled and then fused with shallower fea-
tures through the DSF block, in which the dual-stream fea-
tures come from the two adjacent scales are concatenated at
channel dimension, fused by a 1x1 convolutional layer and
then upscaled at each stream. The MuGI blocks are further
employed to promote cross-scale interactions. Note that,
shallow features extracted from RGB inputs are fused at the
top of DSFNet, which preserves fine-grained details to re-
store both layers. The aggregated features provide rough
separations of layers and are further refined by the second
stage. The following stage (DSDNet) is constructed by the
proposed MuGI blocks in a U-shaped manner. More details
can be found in the supplementary materials.
3.4. Loss Function
Besides the R3 loss demonstrated in Section 3.1, we
further introduce pixel, perceptual, and exclusion loss for
pixel-wise and semantic fidelity as well as the gradient in-
dependence of layers, which are described as follows:
Pixel Loss. The pixel loss is used to constrain the con-
sistency of the reconstructed layer and groundtruth in both
the natural image domain and gradient domain, minimizing
their errors as follows:
Lpix := ∥ˆT − T∥2
2 + ∥ ˆR − R∥2
2
+ α(∥∇ˆT − ∇T∥1 + ∥∇ ˆR − ∇R∥1), (4)
where ∥ · ∥2 denotes the ℓ2 norm. ∇ stands for the gradi-
ent operator, which extracts gradients in both vertical and
horizontal directions. We express one of them in the formu-
lation for clarity and average them in practice. α is set as 2
in our experiments.Perceptual Loss. The pixel-wise losses cannot assure
the multi-scale consistency between predictions and their
groundtruths, which may result in overwhelming punish-
ment for the whole image due to small differences in local
brightness. Therefore, the perceptual loss is further intro-
duced as follows:
Lper := X
i
ωi∥ϕi(ˆT) − ϕi(T)∥1, (5)
where ϕi(·) designates the features drawn by layer i ∈
{2, 7, 12, 21, 30} of a VGG-19 model. ωis are combining
weights of terms at different layers. We follow the setting
of hyper-parameters in [36].
Exclusion Loss. We introduce the exclusion loss to
strengthen the gradient independence prior and reduce the
structural coupling in estimated separations as below:
Lexc := 1
N
N −1X
n=0
∥Ψ(ˆT↓n, ˆR↓n)∥2
2,
Ψ(ˆT, ˆR) := tanh

η1|∇ˆT|

◦ tanh

η2|∇ ˆR|

,
(6)
where ˆT↓n and ˆR↓n represent taking down-sampling by 2n
times (2N at most) of ˆT and ˆR. η1 and η2 are normalization
factors, which are identical to [41].
Gathering all the loss terms yields the final objective as:
Lall := Lpix + β1Lper + β2Lexc + β3Lrec, (7)
where β1 = 0.01, β2 = 1, and β3 = 0.2 are set empirically.
epochs at most to reach favorable overall performance. The
learning rate is initialized as 10−4 and fixed during the train-
ing phase with a batch size of 1.
Dataset. Our training dataset embraces both real and syn-
thesized images. Following [5, 12], the training dataset are
composed of 90 real pairs from [41] and 7,643 synthesized
pairs from the PASCAL VOC dataset [4]. For synthesized
data, transmission and reflection layers are weakened by co-
efficients γ1 ∈ [0.8, 1.0] and γ2 ∈ [0.4, 1.0] during blending
them with the following model:
Isyn = γ1Tsyn + γ2Rsyn − γ1γ2Tsyn ◦ Rsyn, (8)
where Tsyn, Rsyn and Isyn represent the transmission, re-
flection, and superimposed layers during synthesis, respec-
tively. This formulation is inspired by the “screen” blending
mode in digital image processing, which always reserves
lighter colors for the blending layers.4.2. Performance Evaluation
Quantitative comparison. A comparison is made be-
tween state-of-the-art methods as shown in Table 1, involv-
ing Zhang et al. [41], BDN [39], ERRNet [36], IBCLN
[20], RAGNet [24], DMGN [7], Zheng et al. [43], YTMT
[12] and ours on four real-world dataset, including Real20
[41] and three subsets of the SIR2 Dataset [33]. Besides,
to meet the configuration of [3], we train our DSRNet additionally under their training settings, which includes 200
extra real pairs provided by the “Nature” dataset [20], and
13,700 synthesized image pairs provided by [41] instead.
It turns out that our method shows its superiority over
other competitors on all the testing datasets in both settings,
gaining 0.50dB and 1.54dB in terms of average PSNR in
the two settings, respectively. Given the four real-world
datasets contains a variety of scenes, illumination condi-
tions, and glass thickness, it is not a trivial task to achieve
the best performance of all metrics on these datasets si-
multaneously. The experimental results demonstrate that
our proposed SIRS scheme has significantly higher perfor-
mance and stronger generalization ability, which explains
our main contributions.
Qualitative comparisons. To further explain our perfor-
mance advantages, we provide visual comparisons of trans-
mission layers in Fig. 4 against state-of-the-art methods, in-
cluding Zhang et al. [41], BDN [39], ERRNet [36], IBCLN
[20], RAGNet [24], YTMT [12], and Dong et al. [3]. As
can be observed in Fig. 4, the method proposed by Zhang
et al. fails to handle the case containing scattered reflec-
tion components in the second row. BDN has trouble deal-
ing with the images with highlights like in the first row and
even aggravates the reflections. The problem is likely to be
caused by the linear reflection model, which lacks the abil-
ity to model specular highlights. For the results of ERRNet,
which contains only a single branch to estimate transmis-
sion layers, it lacks the direct modeling of the reflections,
therefore only removing parts of them. IBCLN also has
trouble removing highlights and scattered reflections. The
reflection layers in RAGNet are estimated without the par-
ticipation of the transmission, hence showing inferior per-
formance. YTMT can better cope with the highlights in the
last row owing to its dual-stream design but is still limited
by the linear assumption and cannot remove strong reflec-
tions. The method proposed by Dong et al. shows inferiorresults of the YTMT feature interaction mechanism (w/
YTMT Inter.) by replacing MuGI with the YTMT mech-
anism. It can be seen that the model with the YTMT mech-
anism is only slightly better than the scheme without inter-
action, which demonstrates the superiority of the proposed
MuGI mechanism.
Feature Encoder. In comparison to the hypercolumn
framework introduced by the previous method (referred to
as ”w/ HyperColumn”) and the approach that without us-
ing any feature encoder (”w/o Feature Enc.”), our method
demonstrates noteworthy performance advantages. These
advantages stem from two main factors. Firstly, our ap-
proach employs a hierarchical and gradual reduction of
channel dimensions, followed by fusion operations. Sec-
ondly, DSFNet leverages a feature interaction mechanism,
allowing high-level semantic features to interact before be-
ing fed into the subsequent stage.
5. Conclusion
In this paper, we modified the commonly-used linear
model in the single image reflection separation task and
proposed the reconstruction loss with residual correction.
The proposed model is more flexible and effective com-
pared with the previous methods. Meanwhile, we further
improved the feature interaction mechanism in dual-stream
networks and the usage of hierarchical semantic informa-
tion, proposed the MuGI as a novel interaction paradigm,
and a dual-stream semantic-aware network, namely DSR-
Net. Extensive experiments revealed that our proposed
method has achieved state-of-the-art performance on all
real-world benchmark datasets and verified our contribu-
tions. In the future, further constraints are desired to be
applied to the residue term to reduce its solution space, and
more reflection models are hopefully to be covered by it to
fit a wider range of reflection phenomena in the real world.

I need a report on Implementing this paper's model as a machine learning app called ReflectionEraser.
Section name Discusses about Marks
Abstract
A short intro, the main contribution of this work, what you have achieved, the GitHub repo link
1
Introduction
Introduction, motivation, background, and related works
2
Methodology
Provides the detailed steps and processes followed during the project, enabling others to replicate. Subsections can discuss datasets, data preprocessing, model architecture/algorithms, training procedures, etc.
3
Result analysis
Experimental setup, evaluation metrics, quantitative analysis (showing necessary tables and figures), error analysis, etc. 
5
Prototype
A brief discussion of the prototype that you have made for the project
2
Conclusion
Concluding remarks, limitations of this project, future work
1
References
Cite the relevant papers following IEEE style (minimum 10 references)
1