Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation
Authors: Qiming Hu
Xiaojie Guo
Venue: NeurIPS 2021 Poster (2021)
Keywords: reflection separation
dual-stream
interactive network
ReLU rectifier

Abstract: Single image reflection separation (SIRS), as a representative blind source separation task, aims to recover two layers, $\textit{i.e.}$, transmission and reflection, from one mixed observation, which is challenging due to the highly ill-posed nature. Existing deep learning based solutions typically restore the target layers individually, or with some concerns at the end of the output, barely taking into account the interaction across the two streams/branches. In order to utilize information more efficiently, this work presents a general yet simple interactive strategy, namely $\textit{your trash is my treasure}$ (YTMT), for constructing dual-stream decomposition networks. To be specific, we explicitly enforce the two streams to communicate with each other block-wisely. Inspired by the additive property between the two components, the interactive path can be easily built via transferring, instead of discarding, deactivated information by the ReLU rectifier from one stream to the other. Both ablation studies and experimental results on widely-used SIRS datasets are conducted to demonstrate the efficacy of YTMT, and reveal its superiority over other state-of-the-art alternatives. The implementation is quite simple and our code is publicly available at https://github.com/mingcv/YTMT-Strategy.

Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation
Qiming Hu, Xiaojie Guo"
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China huqimingotju.ce edu. edu.cn en xj m.m.a.guue max.guu@gmail... com
Abstract
Single image reflection separation (SIRS), as a representative blind source sepa- ration task, aims to recover two layers, i.e., transmission and reflection, from one mixed observation, which is challenging due the highly ill-posed nature. Exist- ing deep learning based solutions typically restore the target layers individually, or with some concerns the end of the output, barely taking into account the interaction across the two streams/branches. In order to utilize information more efficiently, this work presents a general yet simple interactive strategy, namely your trash is my Ireasure (YTMT), for constructing dual-stream decomposition networks. To be specific, we explicitly enforce the two streams to communicate with each other block-wisely. Inspired by the additive property between the two components, the interactive path can be casily built via transferring, instead discarding, deactivated information by the ReLU rectifier from one stream to the other. Both ablation studies and experimental results on widely-used SIRS datasets are conducted to demonstrate the efficacy of YTMT, and reveal its superiority over other ttate-of-the-ee art alternatives. The implementation is quite simple and our code is publicly available at http:///gttuall.oo//mmgg////////
Introduction
Blind source separation, long-standing problem in signal processing, aims recover multiple intrinsic components from their mixture, the difficulty of which comes from its ill-posedness, i.e., without extra information, there an infinite number of feasible decompositions. Particularly in computer vision, image reflection separation (IRS) is representative scenario that often occurs when taking pictures through a transparent medium such as glass. such cases, the captured images will contain both the scene transmitted through the medium (transmission) and reflection. On the one hand, reflections are annoying for high-quality imaging, and may interfere with the performance of most, if not all, of classic and contemporary vision oriented algorithms such as object detection and segmentation. On the other hand, one may also want to see what happens in the reflection. Hence, developing effective transmission-refletiioo decomposition techniques is desired.
Formally, the captured superimposed image can be typically modeled linear combination of a transmission layer T and a reflection R, i.e. 1 = T + R. Over last decades, a large number of schemes have been devised solve the decomposition problem. Various statistical priors and regularizers have been proposed to mitigate the ill-posed dilemma, while diverse deep networks have been recently built for the sake of performance improvement, please see Sec. for details. However,
* "Corresponding Author
Maan problems follow the same additive model, such as denoising (/ = B + N, where and denote clean image and noise, respectively), and intrinsic image decomposition (log / = log A + log S, where A and S stand for albedo and shading, respectively.) The proposed strategy be potentially applied to all these tasks, but due page limit, concentrate the task of SIRS verify primary claims in this paper.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

in the literature, the additive property, say I T + R, has been hardly investigated, except for acting as a reconstruction constraint.
Let us consider that, for any estimation pair 7 and A satisfying the additive property, there always exists an crror/residual Q subject to / - T' = +)+ Q). Once Q is somehow obtained, the only thing needed to do is subtracting it from T, and adding it into R instead of simply discarding. Under the circumstances, no information is trash, only misplaced. To be symmetric, we rewrite Q as Q := QT - QR. yielding T = T - QT + QR and R = R - QR + QT. In other words, the two targets including T and R can gain from each other by exchanging, rather than discarding, their respective "trash" factors QT and Qn. Driven by the above fact, question naturally arises: Can such an interaction/exchange be applied to intermediate deep features of dual-stream networks?
Contributions. This paper answers the above question by designing a general interactive dual- stream/branch strategy, namely your trash is my treasure (YTMT). An obstacle to realizing YTMT was how to determine exchanging information. Intuitively, activation functions are competent for the job, which are developed to select (activate) a part of features from inputs. In this work, we adopt the ReLU that is arguably the most representative and widely-used activation manner, while others could be also qualified like [4, 24, 12, 2]. Please notice that, instead of simply discarding the deactivated features (trash) of the one stream, we alternatively deliver them to the other stream as compensation (treasure). By doing so, there are two main merits: 1) the information losing and dead ReLU problems can be consequently significantly accelerated. The implementation is quite simple and flexible. We provide two optional YTMT blocks examples and apply them on both plain and UNet architectures to verify the primary claims. Both ablation studies and experimental results on widely-used SIRS datasets are conducted to demonstrate the efficacy of YTMT, and reveal its superiority over other of-t-offtheeaat alternatives.
T mitigated, and 2) the decreasing speed in training error be
2 Related Work
Over the past decades, much attention to resolving the image reflection separation problem has been drawn from the community. From the perspective of the required input amount, existing methods can be divided into two classes, i.e., multi-image based and single image based methods.
Multiple Image Reflection Separation (MIRS). Utilizing multiple images 28, 33, 27, 8, 31, 22, 9, 29, 40, 32, 41, 10] used to be a prevalent way to cope with the task, as more information complementary to single superimposed images, like varied conditions and relative motions, can
be explored from a sequence of images to accomplish the task. For instance, Agrawal el al. [1] use flash and no-flash image pairs remove reflections and highlights, while Szeliski el al. [33] employ focused and defocused pairs. A variety of approaches [7, 27, 8, 22, 33, 40, 41] have been proposed to seek relationships between different images, among which Farid and Adelson [7] employ independent component analysis to reduce reflections and lighting. Gai et al. [8] alternatively develop an algorithm to estimate layer motions and linear mixing coefficients for the recovery. Li and Brown [22] adopt the SIFT-flow to align the images, while Xue et al. [40] enforce a heavy tailed distribution on the obstruction and background components to penalize the overlapped gradients, making the two layers independent. Moreover, Yang et al. [41] introduce a generalized double-layer brightness consistency constraint to connect optical flow estimation and layer separation. Despite the satisfactory performance of MIRS methods, the need for specified shooting conditions and/or professional tools heavily limits their applicability.
Single Image Reflection Separation (SIRS). In practice, single image based schemes are more attractive. No doubt that, compared with those MIRS techniques, single image based ones are also more challenging due to less information available, demanding extra priors to regularize the solution. To mitigate the ill-posedness, Levin et al. [19, 18] favor decompositions that have fewer edges and corners by imposing sparse gradients on the layers, in the same spirit as [40]. Levin and Weiss [17] allow users to manually annotate some dominant edges for the layers as explicit constraints for the problem, which requires careful human efforts to obtain favorable results. Li and Brown [23] assume one layer is smoother than the other, and penalize differently on the two layers in the gradient domain to split the reflection and transmission. Despite progress made toward addressing the problem, the assumption and requirement could be frequently violated in real situations. Besides, these methods all rely on handcrafted features that may considerably restrict their performance.
2

---

Fusion & Alt.
Fusion & Att.
Att.
Att.
ReLU
Concat.
ConcaL.
Add.
Add.
Nop. ReLU
Nep. ReLU
Nep. ReLU
Noa. ReLU
ReLU
ReLU
ReLU
ReLU
1
Conv. 2d
Conv: 2d
1 Conv. 2d
Cony. 2d
Neg. ReLU
X1
8;
(2)
(b)
Figure I: The functional behaviors of ReLU and negative ReLU are plotted in (a). Two YTMT block options are given in (b) and (c). To fuse features from the normal and YTMT connections, the first option uses channel concatenation and 1 x 1 convolutions, while the second simply uses feature addition. Pixel and channel attention mechanism (denoted by Att.) is utilized in both (b) and (c).
With the emergence of deep neural networks, learning based methods [6, 44, 42, 38, 20, 39] have shown their power and become dominant for the task. Concretely, CEILNet [6] comprises of two stages, similarly [17, 37J. An edge map estimated in the first stage, which performs as the guidance for the second stage to produce the final transmission layer. However, the network is hard capture high-level information, thus having trouble guaranteeing the perceptual quality, Zhang al. [44] further take the perception cue into consideration by coordinating byper-column features [11], the perceptual and adversarial losses. The exclusivity loss (see also the gradient production penalization in [40]) is designed for assuring gradients to be exclusive between the two components. This manner is also adopted by [38]. The main drawback is the insufficient capacity in processing overlapped regions. Recently, ERRNEt [38] was proposed, which enlarges the receptive field by introducing channel attention and pyramid pooling modules. It does not explicitly predict the reflection layer, which directly discards possible benefit from the supervision of reflection. IBCLN [20] proposes an iterative boost convolutional LSTM network to progressively split the two layers. However, such an iterative scheme slows down the training and predicting procedure. The errors will also be accumulated due to the high dependency on the outputs of previous stages. Though the learning based strategies have further stepped forward in single image reflection separation compared with the traditional methods, they either restore the target layers individually, together with some concerns dealing with outputs (e.g., enforcing the linear combination constraint and/or concatenating them together as input). In other words, they barely take into account the interaction across the two streams/branches, which is key the target task and also other -traam decomposition tasks. This study is mainly to demonstrate the effectiveness such an interaction consideration.
cases with large wrasmmision-reflecte
Deep Interactive Dual-Stream Learning for SIRS
3.1 YTMT Strategy
In this part, the proposed deep interactive dual-stream strategy will be detailed by centering around the concept that your trash is my treasure, which would be beneficial to variety of two-component decomposition tasks using branch networks. Prior works [21, have shown evidence on the effectiveness of passing information between branches though, our YTMT strategy performs in novel and principled way. We first give the definition of the negative ReLU function as follows:
ReLU (x) := x ReLU(x) = min (xn(x)))) 0),
(1)
where ReLU((x) == max(x, 0). By the negative ReLU, the deactivated features can be easily retained. Figure (a) exhibits the behaviors of the ReLU and negative ReLU.
Here, let x be the input to the first layer of the network, and x (i (1,2) 2) for two branches) denotes
the feature obtained by the i-th branch after I stacked layers, i.e. x : Hxx))). The inputs to the
3

---

Addl
Camoat
Affeation
Collnn
VTME Blick
Max Pooling
VTNT Blckk
Max Ponling
STME Blick
Bilinerr Up
Bilinent Up
Add' Concet
YTM Bleck
Cancitoaation maati
VTMT Block
Signold
Bilincar
Hh
Add'
Concas
Addl Conee
(a) U-shaped YTMT network
ReLU
Hyyer Colama
TTVT Blnck
VEME Binck
x
x
Signooil
ReLU VeLU""RR1
YTMI
Tnput
(8))
Output
ReLil (R5)
-1
(b) Plain YTMT network
(c) Feature visualization
Figure 2: Illustration of the YTMT networks. (a) and (b) offer the YTMT versions modified on the U-shape and plain network architectures, respectively. The input first augmented by hyper column [44] and then fed into the YTMT blocks. (c) shows a visualized example of producing xf and x from x and x at the sixth YTMT block of the trained U-shaped YTMT network. For better view, all the features are normalized by softmax.
(I + 1)-th layer are as follows:
xf RcLU(X)) e ReLU (x%); x ReLU(X)) ReLU ReLU-(X))
where can be either the concatenation operation or the addition between features activated by the ReLU function (called normal connection) and those by the ReLU (YTMT connection), as shown in Fig. 1 (b) and (c). As be obtained from Eq. (2), the amount of information in x" and x is equivalent to that in x and žt) This property guarantees information flowing away from the interaction, which substantially avoids the problems of vanishing/exploding gradients and dead ReLU, Figure (c) depicts a visual example of producing x and x from x and x (in this case, L = 6). It shows that ReLU (xf) is complementary to the ReLU(X))) and ReLU (X²) ) is complementary to the Re By merging the complementary counterparts, there is no information wasted by rectifiers. addition, this strategy can significantly speed up similarly to the ResNet design [13], which will be empirically validated in Sec. 4. Moreover, generally speaking, our strategy can be compatible with most, if not all, of the activation pairs (e.g. Softmax and Softmin). But for the additive problems. the pair of ReLU and negative ReLU is more suitable due to its "either A or B" nature that satisfies the task of SIRS.
ReLU-(X;
decreasing of training error
We again emphasize that the proposed YTMT strategy is general and flexible, which can be imple- mented in various forms according to different demands. Figure/ provides two YTMT block options. The one in (b) fuses features from the normal and YTMT connections by channel concatenation and 1 x convolutions, while second (c) simply employs feature addition. In addition, pixel and channel attention mechanism is introduced in both (b) and (c) to select and re-weight the merged features (see [25] for details). Moreover, the YTMT blocks can be applied to most, not all, of dual- stream backbones by simple modification. In Fig. 2 (a) and (b), two commonly-use used architectures.

---

i.e. U-shaped [26] and plain [43] networks, are present. We will shortly show how to construct these YTMT based networks, and demonstrate their improvement over the backbones and the superior performance on specific applications over other competitors.
3.2 YTMT based Design for SIRS
As shown in Fig. 2 (a), we adopt the U-shaped network as the backhone for the task of SIRS, which can be readily implemented by replacing the convolutional blocks in the UNet architecture with proposed YTMT block options. Following the prior work, the input images are first augmented by the hypercolumn [44], gaining 1475 channels, then mapped to 64 via features and the original input. Each YTMT block is an interactive dual-stream module containing two convolutional layers, both followed by a dual-ReLU rectifier pair. The activations produced by the negative ReLU rectifiers are exchanged between the two streams, then merged by feature addition or concatenation operator before being fed into the attention block. We here use max-pooling and bilinear interpolation squeeze and expand the feature maps. Like in the single-stream UNet, there are skip connections between the encoder layers and the decoder layers (represented by dashed arrwss in orange in Fig. 2 (a)), but an extra skip connection is added between each encoder-decoder layer pair for the dual-stream design (represented by dashed arrows in blue). The features in skip connections are first fused with the up-sampled features and then fed into the YTMT blocks. After training the first stage to converge, we then freeze its parameters and initialize the second stage with them. The outputs of the first stage are then fed into the second one for training until it also converges. Under this design, the information of the transmission and reflection pairs will further interact during passing through the second stage. To be concluded, with consideration of YTMT block options ("C" for channel concatenation and "A" for feature addition) and using two-stage or not ("'S for single-stage and "T" for two-stage), YTMT based design for SIRS has several variants, including YTMTT-UCS, UAS, UCT, UAT}. based on the choice of model architecture ("U" stands for the U-shaped architecture. We omit the plain architecture for its inferior efficiency compared with the U-shaped on the task of SIRS). The performance difference between these YTMT variants will be studied Sec. 4.3.
convolution to fuse VGG
The objective function considered for SIRS consists of reconstruction loss, perceptual loss, exclusion loss, and adversarial loss. In what follows, each term is explained.
Reconstruction loss. Preceding methods have revealed that edges are essential for a valid separation [17, 40, 6, 35]. To make our model sensitive to the gradients, follow [6, 38] to penalize the gradient difference between predictions and targets besides the MSE term, via:
Lree == all T + bll RI2 + cclv VT + dll + R--1ll 11l,,
(3)
where || 1| stands for the f norm, and || 2|2 the f2 norm. In addition, we empirically set a = 0.3, 6 0.9, 0.6, d 0.2 in all of our experiments. Since reflections are usually weak, the penalization on VR- V R is omitted for stable training.
Perceptual Loss. The perceptual loss [14] assists models in achieving high perceptual quality. We minimize the & difference between the features of predicted components and those of ground-truths at layers 'conv2_2', 'conv3_2', 'conv4_2, and 'conv5_2' of a VGG-19 model [30] pretrained on the ImageNet dataset [3]. Denoting the features of the input at layer as ()..), we have the following function:
Eper : Eujllo,(T) (T) o5 (T))a + Euillo, (R) (R))
where wys balance the weights of different layers, (0.38, 0.21, 0.27, 0.18, 6.67} as default.
Exclusion Loss. Prior work [40, 44] shows that enforcing the exclusivity on the two components in the gradient domain is beneficial to separation tasks, which is defined as:
N-1 ITTL 11 with (Tf, R) tanh (Ar\V o tanh (AR)V (5)
Lose =
where Tin and Re are and ft down-sampled by 2n times. In addition, \r and Xn are normalization factors and represents element-wis multiplication.
5

---

Table 1:
Quantitative results on four real-workd benchmark datasets of different methods. The best
results are indicated in red and the second best results in blue.
Datasets
Metries
Input
CEIINNet
7hang al.
BDN
ERRNet
IBCLN
Lei et al.
YTMT-UCT
PSNR
19.16
18.45
22.55
18.41
22.89
21.86
22.35
23.26
Real20 (20)
SSIM
0.732
0.690
0.788
0.726
0.803
0.762
0.793
0.806
PSNR
23.74
23.62
22.68
22.72
24.87
24.87
23.81
24.87
Objects (200)
SSIM
0.878
0.867
0.879
0.856
0.896
0.893
0.882
0.896
PSNR
21.31
16.81
20.71
22.04
23.39
21.48
22.91
Postcard (199)
SSIM
0.877
0.834
0.797
0.859
0.876
0.875
0.873
0.884
PSNR
26.06
22.36
21.52
22.36
24.25
24.71
23.84
25.48
Wild (SS)
SSIM
0.890
0.821
0.832
0.830
0.853
0.886
0.866
0.890
0.035
Table 2: Quantitative comparison in reflection covery between the top-3 methods in transmission recovery. The best results are highlighted in red and the second best results in blue.
training eтrоr
w YIMT-UCS
0.030
0.025
0.020
0.015
Datasets
Metries
ERRNet
IBCLN
YTMT-UCT
PSNR
23.53
22,36
24,30
20
Real20
SSIM
0.446
0.469
0.542
epoch
Objects (200)
PSNR
26.02
19.97
SSIM
0.446
0,226
0.499
Figure 3: Training error on our training data of YTMT-UCS and dual-stream UNet without any feature interaction (w/o FI). The proposed interaction strategy leads to much faster de- creasing speed.
PSNR
22.47
13.16
23.41
Postcard (199)
SSIM
0.419
0.230
0.478
Wild (55)
PSNR
25.52
20.83
27.33
SSIM
0.460
0.298
0.590
Adversarial Loss, The adversarial loss regulates the SIRS solution on the real-world image manifold. With the adversarial training of the main network G and discriminator D, they seek the equilibrium by optimizing the adversarial loss Lade [15], as follows:
ce := - log(D(T,,)) f)) - log - (Ds(1))(TTT); L == -ooloog-1111 - D(T, f)) log())TT log(D(f, (6)
where DD stays invariant with [38].
Our overall objective function turns out to be:
Liot == Cree + Xp + de + AzLadv,
with A = 0.1, A3- = 1, and A3 = 0.01 empirically set.
4 Experimental Validation on SIRS
4.1
Implementation Details
Our training data consists of both real-world and synthesis images, as used in [38]. Among these data, 90 pairs of input and transmission groundtruth are collected by Zhang et al. [44], and 7,643 image pairs chosen from the PASCAL VOC dataset [5] to synthesize superimposed images following CEILNet [6]. The models are implemented Py PyTorch and optimized with Adam optimizer, keeping 31 = 0.9, and Bg 0.999. The learning rate is initialized as 10-4 and then reduced by half at epoch 60, 80, and 100, respectively. The training is stopped at epoch 120. All the models are trained on a single RTX 2080 Ti graphics card.
4.2 Comparison with State-of-the-art Methods
We select the proposed YTMT-UCT network to compare with the state-of-the-ard methods, including Zhang al. [44], BDN [42], ERRNet [38], IBCLN [20] and Lei al. [16], on four real-world dataset, involving Real20 [44] and three subsets of SIR² [34]. The PSNR and SSIM metrics are utilized to evaluate all the competing methods as shown Table Generally speaking, each component for blind source separation tasks should be treated equally. For the SIRS task, the reflection layer

---

Input
Zhang et al. [44] BDN [42]
ERRNet [38] IBCLN [20]
Ours
GT
Figure 4: Visual comparison of state-of-the-ane methods and ours on real-world testing dataset, including Real20 [44] and SIR² [6]. It shows that the results by our method are more visually striking
compared with those by the competitors, with less artifacts and color distortion (please pay attention the regions framed by red boxes).
Table 3: Quantitative results on the testing samples from Nature dataset of different methods. The best results highlighted in red and the second best results in blue.
Metrics
CEILNet-F
Zhang et al.
BDN-F
RmNet
ERRNet-F
IBCLN
YTMT-UCS
PSNR
19.33
19.56
18.92
19.36
22.18
23.57
23.85
SSIM
0.745
0.736
0.737
0.725
0.756
0.783
0.810
can be critical part of the information to reproduce the image shooting scene, as discussed in [36]. Therefore, we report the performance difference of the reflection recovery Table 2, among the methods with the top-3 transmission recovery. The visual comparison is also conducted in Fig. 6. Given ERRNet does not output reflection layers and the predicted reflections of IBCLN still mingle with large parts transmission layers, we compare their reflection layers by
T for faimesss
It turns out that the YTMT-UCT achieves the best results on almost all of the testing datasets, in terms of the recovery of both transmission and the reflection layers, which indicates that the YTMT strategy gives the most accurate separations with the help of frequent feature interaction between the transmission and reflection streams. The difficulty raising in Real20 comes from the largely overlapped regions and various reflection patterns, while the challenge of SIR? mainly low-quality transmission layers and reflections blended with the monochromatic background. Therefore, we further conduct the qualitative comparison among these methods in Fig. show our capacity for these challenging samples. As can be observed, the method proposed by Zhang er al. produces results with severe color distortion and cannot handle the aforementioned globally overlapping problem. BDN and IBCLN generalize unsatisfyingly on the Real20 dataset, failing to remove conspicuous reflections several cases, and some reflections are even enhanced by the BDN due to the inaccurate reflection estimation amplified by the cascade structure. Meanwhile, both the methods refine the

---

Input
CEILNet [6]
Zhang et [44]
BDN [42]
ERRNet [38]
IBCLN [20]
Ours
Figure 5: Visual comparison of YTMT-UAT and satte-of-the-ars methods real45 dataset[6]. It
shows that our method produces results with fewer residual reflections.
separations stage or perration-by-ierratinn where new outputs are concatenated to fed into the next stage/iteration, instead of inner interaction or feature exchanging. As consequence, the error would be rather accumulated. However, the YTMT strategy, there are more chances for the network (e.g., 14 times in the YTMT-UCT). Moreover, the separations are estimated in single network, rather than several sub-networks. We train the first stage of the network converge, and then use to initialize the second stage. The training of the second stage depends on not only the preliminary separations, but also the knowledge learned in the first stage. there is no better separation the network can exploit, the second stage will produce solution close to the preliminary separations. ERRNet introduces some useful network building blocks, but still leaves some obvious reflections. It merely estimates the transmission layer, discarding the interaction between two components. In conclusion, feature interaction is highly desired for the satisfactory separations. Moreover, the training eπrо shown in Fig. further accounts for the information efficacy of YTMT strategy, given the large margin of the convergence between the dual-stream UNet without any feature interaction and the YTMT-UCS solution. To evaluate our generalization ability to different data distributions, also follow the training setting of IBCLN [20] and report our results on the testing data of the Nature dataset proposed by We achieve 23.85 and 0.810 for PSNR and SSIM, respectively, compared with 23.57 and 0.783 of IBCLN, and surpass previous methods more than lot as shown in Table 3. Note that we reuse the numerical results proposed in the paper of IBCLN, since the same experimental setup is kept.
4.3 Ablation Study
To further assess the capacity of the YTMT strategy for the task, we have developed two baselines to illustrate the effectiveness of proposed YTMT strategy (see results in Table 4). The first model dual-stream UNet without any feature interaction. One might think is identical to a single UNet. However, constrained by the reconstruction loss term I|? + ft. - 7]I, the estimations of the two layers / and R are highly related to each other. Moreover, supervised by the ground-truth of both transmission and reflection layers, the optimization constrained by multiple regularizers are more likely to improve the generalization ability of the network. Thus we adopt it as one of the baseline networks, which means any interactive method should not have lower performance than The second model replaces all the negative Rel.U rectifiers by the normal ReLU in the YTMT-UCS to validate if the YTMT strategy really works to preserve the discarded information. The rest four models the YTMT variants introduced in the Sec. 3.2.

---

Table 4: Ablation study on different network architectures, including a dual-stream UNet without any feature interaction (FI), YTMT-UCS with all the negative ReLU rectifiers replaced by the normal
ReLU and four YTMT variants. The best results are highlighted in red and the second best results in blue.
Datasets
Metries
w/o
ReLU only
YTMT-UCS
YTMT-UCT
YTMT-UAS
YTMT-UAT
PSNR
20.37
22.79
23.05
23.09
23.26
23.39
Real20
SSIM
0.752
0.802
0,805
0.802
0.801
0,809
PSNR
23.85
24.06
24,46
24.58
24.71
25.40
Objects
SSIM
0.883
0.886
0.891
0.891
0.893
0.899
PSNR
20.82
22,02
22.66
22.75
22.45
23.01
Postcard
SSIM
0.863
0.869
0.885
0.884
0.871
0.874
PSNR
24.20
24.71
25.24
25,46
24.49
25.19
Wild
SSIM
0.859
0.887
0.892
0.870
0.880
Zhang et al. [44]
BDN [42]
IBCLN [20]
Ours
GT
Figure 6: Visual comparison on the layer decomposition between YTMT-UCT and state-of-the-ar
methods that output transmission-reflectios layer pairs.
With approximate number of parameters, YTMT solutions show obvious superiority over other alternatives, benefiting from the strategy introducing the complementary interaction between dual streams. Obviously, without any feature interaction, a dual-stream UNet is apparently inferior to those with feature interaction, which partly explains the performance gap between the IBCLN and ours. With only the positive ReLU rectifiers, the dual streams in the network will have the same activations, leading degraded performance. Moreover, leading results on the first two datasets, while YTMT-UCT exhibits better overall performance over other kate-of-the-ar methods, thus it is opted the default architecture for SIRS tasks. Also, shows that two-stage architecture can indeed further improve the results.
be seen that the YTMT-UAT shows
Conclusion
In this paper, we proposed general rule for deep interactive dual-streeam/branch learning, namely your trash is my treasure (denoted as YTMT), which says that two branches should communicate with each other frequently by exchanging, rather than discarding, the information useless to themselves. Activation functions are deemed to suitable for determining which part information exchange. As an example, the widely-used ReLU was chosen to validate the primary claims of this work. In addition, we offered several YTMT based designs based on both U-shaped and plain backbones to show the flexibility. Extensive experimental results on public SIRS datasets have been provided to verify the effectiveness of YTMT, and demonstrate clear advantages of our method comparison with other state-of-the--rt alternatives. Another merit of YTMT comes from its acceleration in decreasing error during training. positive that our strategy can derive diverse designs and be beneficial to many two-component decomposition tasks.

---

Acknowledgement
This work was supported by the National Natural Science Foundation of China under Grant nos. 62072327 and 61772512, and TSTC under Grant no, 20JCQNJC01500.
References
[1] Amit K. Agrawal, Ramesh Raskar, Shree K. Nayar, and Yuanzhen Li. Removing photography artifacts using gradient projection and flash-exposure sampling. ACM Trans. Graph., 24(3):828 835, 2005.
[2] Djórk-Améé Clevert, Thomas Unterthiner, and Sepp Hochreiter, Fast and accurate deep network learning by exponential linear units (elus). In ICLR, 2016.
[3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009,
[4] Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Gareia. Incorporat- ing second-orde functional knowledge for better option pricing. In NeurlPS, pages 472 478, 2000.
[5] Mark Everingham, Lue Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. IJC,, 88(2):303-333 -38,, 2010.
[6] Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, and David P. Wipf. A generic deep architecture for single image reflection removal and image smoothing. In ICCV, pages 3258-3267, 2017.
[7] Hany Farid and Edward H. Adelson. Separating reflections and lighting using independent components analysis. In CVPR, pages 1262-1267, 1999.
[8] Kun Gai, Zhenwei Shi, and Changshui Zhang. Blind separation of superimposed moving images using image statistics. TPAMI, 34(1):19-32, 2012.
[9] Xiaojie Guo, Xiaochun Cao, and Yi Ma. Robust separation of reflection from multiple images. In CVPR, pages 2195-2202, 2014.
[10) Byeong-Ju Han and Young Sim. Reflection removal using low-rank matrix completion. In CVPR, pages 3872-3880, 2017.
[11) Bharath Hariharan, Pablo Andrés Arbeláez, Ross B. Girshick, and Jitendra Malik. Hypercolumns for object segmentation and fine-gratd enaineai localization. In CVPR, pages 447-456, 2015.
[12] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, pages 1026-1034, 2015.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[14] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711, 2016.
[15] Alexia
The relativistic discriminator: a key element missing from standard
GAN. In ICIR, 2019.
[16] Chenyang Lei, Xuhua Huang, Mengdi Zhang, Qiong Yan, Wenxiu Sun, and Qifeng Chen. Polarized reflection removal with perfect alignment in the wild. In CVPR, pages 1747-1755, 2020.
[17] Anat Levin and Yair Weiss. User assisted separation of reflections from single image using a sparsity prior. TPAMI.
2007.
10

---

[18] Anat Levin, Assaf Zomet, and Yair Weiss. Learning to perceive transparency from the statistics of natural scenes. In NeurIPS, pages 1247-1254, 2002.
[19] Anat Levin, Assaf Zomet, and Yair Weiss. Separating reflections from a single image using local features. In CVPR, pages 906-313, 2004.
[20] Chao Li, Yixiao Yang. Kun He, Stephen Lin, and John Hoperoft, Single image reflection removal through cascaded refinement. In CVPR, pages 3562-3571, 2020.
[21] Siyuan Li, Wenqi Ren, Jiawan Zhang, Jinke Yu, and Xiaojie Guo. Single image rain removal via a deep coammosiiiooncoopssiiiiii network. Computer Vision and Image Understanding, 186:48 57, 2019.
[22] Yu Li and Michael S. Brown. Exploiting reflection change for automatic reflection removal. In ICCV, pages 2432-2439, 2013.
[23] Yu Li and Michael S. Brown. Single image layer separation using relative smoothness. In CVPR, pages 2752-2759, 2014.
[24] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlincarities improve neural network acoustic models. In Proc. icml, page 3, 2013.
[25] Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, and Huizhu Jia. Ffa-net: Feature fusion attention network for single image dehazing. In AAAI, pages 11908-11915, 2020.
[26] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241, 2015.
[27] Bernard Sarel and Michal Irani. Separating transparent layers through layer information exchange. In ECCV, pages 328-341, 2004.
[28] Yoav Y. Schechner, Nahum Kiryati, and Ronen Basri. Separation of transparent layers using focus. IJCV, 39(1):55-39, 2000.
[29] Christian Simon and Kyu Park. Reflection removal for in-vehicle black box videos. In CVPR, pages 4231-4239, 2015.
[30] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[31] Sudipta N. Sinha, Johannes Kopf. Michael Goesele, Daniel Scharstein, and Richard Szeliski. Image-based rendering for scenes with reflections. ACM Trans. Gruph., 1100::000:1100::10 10, 2012.
[32] Chao Sun, Shuaicheng Liu, Taotao Yang. Bing Zeng, Zhengning Wang, and Guanghui Liu. Automatic reflection removal using gradient intensity and motion cues. In ACM MM, pages 466-470, 2016.
[33] Richard Szeliski, Shai Avidan, and Anandan. Layer extraction from multiple images contain- ing reflections and transparency. In CVPR, page 1246, 2000.
[34] Renjie Wan, Boxin Shi, Ling- Yu Duan, Ah-Hwee Tan, and Alex C. Kot. Benchmarking single-image reflection removal algorithms. In ICCV, pages 3942-3950, 2017.
[35] Renjie Wan, Boxin Shi, Ling- Yu Duan, Ah-Hwee Tan, and Alex C. Kot. CRRN: multi-scale guided concurrent reflection removal network. CVPR, pages 4777-4785, 2018
[36] Renjie Wan, Boxin Shi, Haoliang Li, Ling-Yu Yu Duan, and Alex C. Kot. Reflection scene separation from single image. In 2020 IEEF/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 2395-2403. Computer Vision Foundation IEEE, 2020.
[37] Renjie Wan, Boxin Shi, Ah-Hwee Tan, and Alex C. Kot. Depth of field guided reflection removal. In ICIP, pages 21-25, 2016.

---

[38] Kaixuan Wei, Jiaolong Yang, Ying Fu, David P. Wipf, and Hua Huang. Single image reflection removal exploiting misaligned training data and network enhancements. In CVPR, pages 8178-811 1778-8187,22 2019.
[39] Qiang Wen, Yinjie Tan, Jing Qin, Wenxi Liu, Guoqiang Han, and Shengfeng He. Single image reflection removal beyond linearity. In CVPR, pages 3771-3778 2019.
[40] Tianfan Xue, Michael Rubinstein, Ce Liu, and William Freeman. A computational approach for obstruction-free photography. ACM Trans. Graph., 34(4):79:1-79:11 2015.
[41] Jiaolong Yang. Hongdong Li, Yuchao Dai, and Robby T. Tan. Robust optical flow estimation of double-layer images under transparency or reflection. In CVPR, pages 1410-1419, 2016.
[42] Jie Yang, Dong Gong, Lingqiao Liu, and Qinfeng Shi. Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal. ECCV, pages 675-691, 2018.
[43] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep CNN for image denoising. TIP, 26(7):3142-3155 2017.
[44] Xuaner Cecilia Zhang, Ren Ng, and Qifeng Chen. Single image reflection separation with perceptual losses. CVPR, pages 4786-4794, 2018.
12


Reviews:
[{"review_id": "Pl7_0-Kesc1", "reviewer": "Reviewer_zUwG", "summary": "In this work, the authors address the problem of how to get a good approximation of a scene affected by reflections produced by a pane of glass. To do this, the authors proposed a dual-stream decomposition strategy called YTMT, where the main idea is how to effectively exchange information to enforce a better approximation of the target scene (and also of the reflection layer). At the optimisation level, the authors use a composite loss function which involves the reconstruction term along with three additional terms to enforce better output quality based for instance in constraining the gradients of the image.  Experimental setting shows readily competing results with respect to the existing techniques for reflection removal. Moreover, authors show the potential of the technique for other tasks such as blind denoising.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths] \n--   The main strength of the paper is the idea of how to efficiently exchange/use information through a dual-stream decomposition strategy.  Moreover, the experimental results are competitive wrt the body of literature. \n\n[Weaknesses]\n------ A major point that turns down somehow the paper is the lack of formalism and intuition of the technique. More precisely:\n\n-- The authors consider the image reflection model as a linear superposition between two layers I=T+R. However, different works, even early ones in the area, for  example [*] [**] showed that the reflection is less in focus and often blurred then considering in the model a weighting factor w regarding the strength of the reflections and a blurring kernel k. Why do the authors not consider this more realistic scenario as a model?\n\n[*] Y.  Schechner,  N.  Kiryati,  and  R.  Basri,  “Separation  of  transparent layers  using  focus,”International  Journal  of  Computer  Vision  (IJCV),vol. 39, no. 1, pp. 25–39, 2000.\n[**]Li  and  M.  S.  Brown,  “Single  image  layer  separation  using  relative smoothness,”  inIEEE  Conference  on  Computer  Vision  and  PatternRecognition (CVPR), 2014, pp. 2752–2759.\n\n-- The main driving optimisation (7) is a combination of existing works and ideas that has been already applied to the task at hand including the gradients constraint and the exclusion term. Firstly, can the authors provide further thoughts on the level of novelty on the loss? Secondly, authors argue that they use an adversarial loss to ‘regulate the solution on the image manifold’  Can the authors argue where is coming that regularisation and where the space of images is modelling as a manifold?\n\n-- The authors in the introduction motivate the approach and discuss an inherent ‘error/residual Q’. How does it fit into the big picture of the main driving optimisation and how Q is obtained? \n\n-- From (3) authors set ‘a’ ‘b’ ‘c’ and ‘d’ empirically but what  was the range of values searched to set such values? Similarly for the set of  \\lambda in (7). What is the intuition behind those parameters wrt the performance (e.g. what is the effect of increasing or decreasing them)? \n\n\n------------ [Experimental Results] Although the experimental setting has merit, there are some points to clarify. \n-- The authors are using SSIM and PSNR metrics for the quantitative results. However, several works including [a] and [b] have discussed that need for more perceptual meaningful metrics such as the sLMSE [b]. Why not using more perceptural metrics such as the sLMSE? That avoid for example  that a single edge error to dominate the overall performance\n\n-- What is the motivation of Table 2? One is interested in recovering the target scene rather than the reflection itself. Whilst the reflection layer indeed helps in the model formulation the relevant output is the target scene (transmission layer).\n\n-- From Table I, It is unclear the level of performance gain from the proposed technique. The numerics seem not to have a statistical significant difference wrt ERRNet (and Lei et al). Authors should argue the main benefit of the proposed technique. \n\n-- The authors in the opening of the paper (footnote 1 in page 1) mentioned that the proposed technique can be applied to other tasks but they are not being explored in the paper as the main topic is reflection removal. However, authors actually include experiments regarding blind denoising -- Therefore, the title is not matching the content of the paper. Authors need to further motivate the inclusion of section 5\n\n\n[a]Wan, R., Shi, B., Duan, L. Y., Tan, A. H., & Kot, A. C. (2017). Benchmarking single-image reflection removal algorithms. In Proceedings of the IEEE International Conference on Computer Vision (pp. 3922-3930).\n[b] Grosse, M. K. Johnson, E. H. Adelson, and W. T. Free-man. Ground truth dataset and baseline evaluations for in-trinsic image algorithms. InProc. ICCV, 2009.\n\n\n\n------------------------------------------------------------\nPOST-REBUTTAL\n\nAfter reading the rebuttal carefully, the authors addressed my concerns and even provided further experiments to further support their statements. * And additional minor comment is regarding Q9, what I wanted to covey was that, from a reader point of view, it would be better to have only results on the task of reflection removal rather than adding another task (blind denoising) with a short description. Overall, I’m more positive about this paper than before and this is therefore reflected in the final suggested score -- expecting the authors to implement the points from the rebuttal in a the revised version for completeness. \n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is not a clear discussion on the limitations of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "BlaHKLtom1q", "reviewer": "Reviewer_myi5", "summary": "This paper presents a nonlinear module that leverages both positive and negative ReLUs for feature separation. This enables applications such as reflection removal, blind denoising and other layer separation tasks. The paper shows several variants of adding the proposed module into existing CNN frameworks. Results show superior performance on several benchmark datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of this paper is simple and intuitive. It's executed well with promising results and good ablation study. The novelty of this paper is the way of exploiting all features learned in a CNN and identifying its effectiveness in layer separation tasks where pixels are interfered but useful. The training scheme, data generation, objective functions are all from prior works, but the simple design of keeping all features in the network by adding dual-ReLU branch is interesting and novel, and can potentially inspire further research works.\n\nNevertheless, I have the following concerns:\n\n**Results**\n\nAlthough the model is tested on several datasets, not all results are shown in the supplement and the paper. Quality degrades (compared to results in the paper) for some images in fig 4 and 5 in the supplement. The limitation of the relatively worse performance isn't clear.\n\n**Ablation**\n\nWhile the paper presents several ablation studies on how features are fused and the backbone architectures, some supporting experiments are missing for the key claim of keeping the 'trash' features. In other words, is the benefit from having the negative branch?\n\nWhat happens if the model only keeps the negative ReLU or only the positive ReLU (which is similar to the vanilla backbone)? For example, to keep the number of parameters the same, how does the model perform if replacing all negative ReLU with positive ones and vice versa? What features are lost and how does it show up in the image space? (e.g. is it like residual artifacts? etc.)\n\nThe paper also mentions the design alleviates dead ReLU. Can the two branch use a Leaky ReLU? What would be the difference? There are many interesting questions raised from the proposed framework, but the paper seems to only touch on a few. I'd hope to get more insights from the authors and how they made their design choice of only experimenting with ReLU and how they are thinking about this problem.\n\n**Others**\n\nLine 102 mentions the iterative approach suffers from accumulated errors. However, how is the proposed two-stage method handling this problem, given the second stage also depends on the output from the first stage?\n\n\n## **Post-rebuttal**\n\nThe authors have addressed my concerns with more quantitative results and intuitive explanations. I'd encourage the authors to add some of these explanations into the main paper for clarity. I support acceptance of this paper. The simple design of positive-negative ReLU to preserve all features in the network is interesting and novel, and can potentially inspire further research works.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "**Missing references**\n\nThe paper is missing a set of works on internal learning for layer separation: \n- Ulyanov, Dmitry, Andrea Vedaldi, and Victor Lempitsky. \"Deep image prior.\" CVPR 2018.\n- Gandelsman, Yosef, Assaf Shocher, and Michal Irani. \"\" Double-DIP\": Unsupervised Image Decomposition via Coupled Deep-Image-Priors.\" CVPR 2019.\n\n**Writing**\n\nWriting is mostly clear.\n- line 164 & 238: the choose of -> the choice of", "review_text": "", "extra_scores": {"ethical_concerns": "Not that I'm aware of.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "9chIhTDhuD", "reviewer": "Reviewer_SZth", "summary": "\nThis paper presents a method that recovers reflection and transmission components from a given image degraded with reflections. The proposed network employs a dual-stream interactive strategy where the discarded information (by ReLU) from one branch helps improving the feature representation of the other branch and vice versa. Experiments are performed for several image processing tasks, including image reflection removal, denoising, demoireing, and intrinsic image decomposition.  \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is simple and provides favorable results for image decomposition tasks, especially for image reflection removal.  However, I have the following concerns.\n\n\n* Image reflection removal is a position-sensitive task where pixel-to-pixel correspondence from the input to output images is required. For such restoration tasks, those methods that operate at high-resolution (having no downsampling operations) usually perform better than the UNet-like architectures. Why did you use YTMT-UCT as a default choice over the YTMT-PCT version? \n    \n    \n\n* Among the UNet versions of YTMT, Why do you opt for YTMT-UCT as a default architecture to perform main experiments, when in the ablation experiments (Table 4) YTMT-UAT shows overall better performance than the YTMT-UCT version?\n    \n\n\n* The evaluation for the image denoising task is not convincing. The proposed method performs poorly when compared to DnCNN (which is five years old method) for Gaussian noise levels up to sigma=75. The difference in performance might increase even further when compared to recent image denoising approaches such as RIDNet, RDN, PAN, etc. It is only when the noise levels are sigma=80 and above, the proposed method shows performance improvement over DnCNN. Don't you think that the experimental setting with such high noise levels is unrealistic and impractical?  \n\n\n\n* Since the evaluation for the image reflection removal task is performed on real-world datasets, it would have been interesting to see the performance of YTMT on \\textit{blind} image denoising task on real datasets such as SIDD and DND.\n\n    \n\n* In Table 1 you are comparing the results produced by the proposed YTMT method with those of the IBCLN algorithm (CVPR20). The authors of IBCLN also provided a new real-world dataset called \\textit{Nature}. Did you compute the image quality scores on the Nature dataset?\n    \n\n\n* The main architecture diagram (Figure 2a) is complicated and the text explaining it isn't well written either (Sec 3.2). While some parts are fine, the overall paper is not easy to follow.\n\n\n\n * * * *  *  *  *  *\n\n### **Post Rebuttal Comments**\n* The authors have provided a convincing response to my concerns. Overall, the idea of this paper is simple and intuitive and provides good results. I recommend this paper for acceptance. However, I would suggest the authors to revise the paper to improve the text and simplify the figures.   ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "~ 24 hours"}, "extra_text": {}}, {"review_id": "3k2UFzbV9o4", "reviewer": "Reviewer_XY18", "summary": "This paper proposes a dual-stream network design equipped with an interactive strategy (coined YTMT) to tackle the blind source separation tasks, especially the single image reflection separation. The underlying design motivation comes from a very simple mathematical derivation, i.e., any information discarded from one layer might be beneficial to reconstruct another layer. They demonstrate performance improvement on the task of single image reflection removal. The possible extensions to other source separation tasks, e.g., the blind image denoising and intrinsic image decomposition, are also investigated. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n* The introduction part is well written; the design motivation is clear and reasonable. \n* Good performance on SIRS against state-of-the-art methods. \n* The potential applications to other blind source separation tasks. \n\n## Cons\nThough the underlying principle looks sound, it largely relies on the assumption that blind source separation could be better handled by  dual-stream networks. Such an assumption is, however, not established in majority of source separation tasks. To separate two layers, one may argue only a single network with two output branches at tail is sufficient to interact information within shared backbone in an implicit manner. The authors should consider adding such an ablation study (with comparable computation overhead) to demonstrate the benefit of dual-stream network design.\n\nBesides, in Table 3, the performance of two-UNet should be identical to a single UNet, as there is no interaction from another stream, thus the comparison there is not so meaningful. The descriptions of Split UNet and Concat. UNet are also unclear, the authors should give some explanations on why they design such architectures for ablation. The ablation study in Table 4 looks redundant, since neither the two stage design nor the attention module is the authors' contribution. The authors should reorganize the paper by moving other content, e.g., the intrinsic image decomposition, from the suppl. material into the main paper. \n\n## Summary\nOverall, the proposed idea is simple yet reasonable, but more experimental verifications are needed to justify its superiority. \nI tentatively give a borderline score and would change it according to the authors' response. \n\n## Post rebuttal\nAfter reading the rebuttal and other reviewers' comments, I think most issues raised by the reviewers are repairable, and many of them have been already addressed in the rebuttal. As a result, I lean to borderline accept and it's ok for me if the rejection is made on this paper. \nAlso, I highly recommend the authors refine the method and experiment parts, particularly, describing the abbreviations (e.g., Concat. UNet, YTMT-UCS) carefully to avoid potential confusion and distraction. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers in general liked the idea and results of the paper: simple designs of keeping all features in the network by adding dual-ReLU branch; good results on SIRS against SOTA. Many issues were raised, e.g. lack of comparison, formalism and intuition of the technique. The reviewers were satisfied with the rebuttal and bumped up the score. Please incorporate the post-rebuttal comments from the reviewers, e.g. having only results on the task of reflection removal rather than adding another task with a short description (can use the space to make the figures larger), and improving the text and simplifying the figures.
Decision: Accept (Poster)