Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation

1 Introduction . Blind source separation , a long-standing problem in signal processing , aims to recover multiple intrinsic components from their mixture , the difficulty of which comes from its ill-posedness , i.e. , without extra information , there is an infinite number of feasible decompositions . Particularly in computer vision , image reflection separation ( IRS ) is a representative scenario that often occurs when taking pictures through a transparent medium such as glass . In such cases , the captured images will contain both the scene transmitted through the medium ( transmission ) and reflection . On the one hand , reflections are annoying for high-quality imaging , and may interfere with the performance of most , if not all , of classic and contemporary vision oriented algorithms such as object detection and segmentation . On the other hand , one may also want to see what happens in the reflection . Hence , developing effective transmission-reflection decomposition techniques is desired . Formally , the captured superimposed image I can be typically modeled as a linear combination of a transmission layer T and a reflection R , i.e . I = T + R.1 Over last decades , a large number of schemes have been devised to solve the decomposition problem . Various statistical priors and regularizers have been proposed to mitigate the ill-posed dilemma , while diverse deep networks have been recently built for the sake of performance improvement , please see Sec . 2 for details . However , ∗Corresponding Author 1Many problems follow the same additive model , such as denoising ( I = B +N , where B and N denote clean image and noise , respectively ) , and intrinsic image decomposition ( log I = logA+ logS , where A and S stand for albedo and shading , respectively . ) The proposed strategy can be potentially applied to all these tasks , but due to page limit , we concentrate on the task of SIRS to verify primary claims in this paper . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in the literature , the additive property , say I = T +R , has been hardly investigated , except for acting as a reconstruction constraint . Let us consider that , for any estimation pair T̂ and R̂ satisfying the additive property , there always exists an error/residual Q subject to I = T̂ + R̂ = ( T +Q ) + ( R−Q ) . Once Q is somehow obtained , the only thing needed to do is subtracting it from T̂ , and adding it into R̂ instead of simply discarding . Under the circumstances , no information is trash , only misplaced . To be symmetric , we rewrite Q as Q : = QT − QR , yielding T = T̂ − QT + QR and R = R̂ − QR + QT . In other words , the two targets including T̂ and R̂ can gain from each other by exchanging , rather than discarding , their respective “ trash ” factors QT and QR . Driven by the above fact , a question naturally arises : Can such an interaction/exchange be applied to intermediate deep features of dual-stream networks ? Contributions . This paper answers the above question by designing a general interactive dualstream/branch strategy , namely your trash is my treasure ( YTMT ) . An obstacle to realizing YTMT was how to determine exchanging information . Intuitively , activation functions are competent for the job , which are developed to select ( activate ) a part of features from inputs . In this work , we adopt the ReLU that is arguably the most representative and widely-used activation manner , while others could be also qualified like [ 4 , 24 , 12 , 2 ] . Please notice that , instead of simply discarding the deactivated features ( trash ) of the one stream , we alternatively deliver them to the other stream as compensation ( treasure ) . By doing so , there are two main merits : 1 ) the information losing and dead ReLU problems can be consequently mitigated , and 2 ) the decreasing speed in training error can be significantly accelerated . The implementation is quite simple and flexible . We provide two optional YTMT blocks as examples and apply them on both plain and UNet architectures to verify the primary claims . Both ablation studies and experimental results on widely-used SIRS datasets are conducted to demonstrate the efficacy of YTMT , and reveal its superiority over other state-of-the-art alternatives . 2 Related Work . Over the past decades , much attention to resolving the image reflection separation problem has been drawn from the community . From the perspective of the required input amount , existing methods can be divided into two classes , i.e. , multi-image based and single image based methods . Multiple Image Reflection Separation ( MIRS ) . Utilizing multiple images [ 7 , 28 , 33 , 27 , 1 , 8 , 31 , 22 , 9 , 29 , 40 , 32 , 41 , 10 ] used to be a prevalent way to cope with the task , as more information complementary to single superimposed images , like varied conditions and relative motions , can be explored from a sequence of images to accomplish the task . For instance , Agrawal et al . [ 1 ] use flash and no-flash image pairs to remove reflections and highlights , while Szeliski et al . [ 33 ] employ focused and defocused pairs . A variety of approaches [ 7 , 27 , 8 , 22 , 33 , 40 , 41 ] have been proposed to seek relationships between different images , among which Farid and Adelson [ 7 ] employ independent component analysis to reduce reflections and lighting . Gai et al . [ 8 ] alternatively develop an algorithm to estimate layer motions and linear mixing coefficients for the recovery . Li and Brown [ 22 ] adopt the SIFT-flow to align the images , while Xue et al . [ 40 ] enforce a heavy tailed distribution on the obstruction and background components to penalize the overlapped gradients , making the two layers independent . Moreover , Yang et al . [ 41 ] introduce a generalized double-layer brightness consistency constraint to connect optical flow estimation and layer separation . Despite the satisfactory performance of MIRS methods , the need for specified shooting conditions and/or professional tools heavily limits their applicability . Single Image Reflection Separation ( SIRS ) . In practice , single image based schemes are more attractive . No doubt that , compared with those MIRS techniques , single image based ones are also more challenging due to less information available , demanding extra priors to regularize the solution . To mitigate the ill-posedness , Levin et al . [ 19 , 18 ] favor decompositions that have fewer edges and corners by imposing sparse gradients on the layers , in the same spirit as [ 40 ] . Levin and Weiss [ 17 ] allow users to manually annotate some dominant edges for the layers as explicit constraints for the problem , which requires careful human efforts to obtain favorable results . Li and Brown [ 23 ] assume one layer is smoother than the other , and penalize differently on the two layers in the gradient domain to split the reflection and transmission . Despite a progress made toward addressing the problem , the assumption and requirement could be frequently violated in real situations . Besides , these methods all rely on handcrafted features that may considerably restrict their performance . With the emergence of deep neural networks , learning based methods [ 6 , 44 , 42 , 38 , 20 , 39 ] have shown their power and become dominant for the task . Concretely , CEILNet [ 6 ] comprises of two stages , similarly to [ 17 , 37 ] . An edge map is estimated in the first stage , which performs as the guidance for the second stage to produce the final transmission layer . However , the network is hard to capture high-level information , thus having trouble guaranteeing the perceptual quality . Zhang et al . [ 44 ] further take the perception cue into consideration by coordinating hyper-column features [ 11 ] , the perceptual and adversarial losses . The exclusivity loss ( see also the gradient production penalization in [ 40 ] ) is designed for assuring gradients to be exclusive between the two components . This manner is also adopted by [ 38 ] . The main drawback is the insufficient capacity in processing cases with large transmission-reflection overlapped regions . Recently , ERRNet [ 38 ] was proposed , which enlarges the receptive field by introducing channel attention and pyramid pooling modules . It does not explicitly predict the reflection layer , which directly discards possible benefit from the supervision of reflection . IBCLN [ 20 ] proposes an iterative boost convolutional LSTM network to progressively split the two layers . However , such an iterative scheme slows down the training and predicting procedure . The errors will also be accumulated due to the high dependency on the outputs of previous stages . Though the learning based strategies have further stepped forward in single image reflection separation compared with the traditional methods , they either restore the target layers individually , or together with some concerns dealing with outputs ( e.g. , enforcing the linear combination constraint and/or concatenating them together as input ) . In other words , they barely take into account the interaction across the two streams/branches , which is key to the target task and also other dual-stream decomposition tasks . This study is mainly to demonstrate the effectiveness of such an interaction consideration . 3 Deep Interactive Dual-Stream Learning for SIRS . 3.1 YTMT Strategy . In this part , the proposed deep interactive dual-stream strategy will be detailed by centering around the concept that your trash is my treasure , which would be beneficial to a variety of two-component decomposition tasks using dual-branch networks . Prior works [ 21 , 35 ] have shown evidence on the effectiveness of passing information between two branches though , our YTMT strategy performs in a novel and principled way . We first give the definition of the negative ReLU function as follows : ReLU− ( x ) : = x− ReLU ( x ) = min ( x , 0 ) , ( 1 ) where ReLU ( x ) : = max ( x , 0 ) . By the negative ReLU , the deactivated features can be easily retained . Figure 1 ( a ) exhibits the behaviors of the ReLU and negative ReLU . Here , let x0 be the input to the first layer of the network , and x̃li ( i ∈ { 1 , 2 } for two branches ) denotes the feature obtained by the i-th branch after l stacked layers , i.e . x̃li : = Hli ( x0 ) . The inputs to the ( l + 1 ) -th layer are as follows : xl1 : = ReLU ( x̃ l 1 ) ⊕ ReLU − ( x̃l2 ) ; x l 2 : = ReLU ( x̃ l 2 ) ⊕ ReLU − ( x̃l1 ) , ( 2 ) where ⊕ can be either the concatenation operation or the addition between features activated by the ReLU function ( called normal connection ) and those by the ReLU− ( YTMT connection ) , as shown in Fig . 1 ( b ) and ( c ) . As can be obtained from Eq . ( 2 ) , the amount of information in xl1 and x l 2 is equivalent to that in x̃l1 and x̃ l 2 . This property guarantees no information flowing away from the interaction , which substantially avoids the problems of vanishing/exploding gradients and dead ReLU . Figure 2 ( c ) depicts a visual example of producing xL1 and x L 2 from x̃ L 1 and x̃ L 2 ( in this case , L = 6 ) . It shows that ReLU− ( x̃L1 ) is complementary to the ReLU ( x̃ L 2 ) and ReLU − ( x̃L2 ) is complementary to the ReLU ( x̃L1 ) . By merging the complementary counterparts , there is no information wasted by the rectifiers . In addition , this strategy can significantly speed up the decreasing of training error similarly to the ResNet design [ 13 ] , which will be empirically validated in Sec . 4 . Moreover , generally speaking , our strategy can be compatible with most , if not all , of the activation pairs ( e.g . Softmax and Softmin ) . But for the additive problems , the pair of ReLU and negative ReLU is more suitable due to its “ either A or B ” nature that satisfies the task of SIRS . We again emphasize that the proposed YTMT strategy is general and flexible , which can be implemented in various forms according to different demands . Figure 1 provides two YTMT block options . The one in ( b ) fuses features from the normal and YTMT connections by channel concatenation and 1× 1 convolutions , while the second in ( c ) simply employs feature addition . In addition , pixel and channel attention mechanism is introduced in both ( b ) and ( c ) to select and re-weight the merged features ( see [ 25 ] for details ) . Moreover , the YTMT blocks can be applied to most , if not all , of dualstream backbones by simple modification . In Fig . 2 ( a ) and ( b ) , two commonly-used architectures , i.e . U-shaped [ 26 ] and plain [ 43 ] networks , are present . We will shortly show how to construct these YTMT based networks , and demonstrate their improvement over the backbones and the superior performance on specific applications over other competitors .

Summary: In this work, the authors address the problem of how to get a good approximation of a scene affected by reflections produced by a pane of glass. To do this, the authors proposed a dual-stream decomposition strategy called YTMT, where the main idea is how to effectively exchange information to enforce a better approximation of the target scene (and also of the reflection layer). At the optimisation level, the authors use a composite loss function which involves the reconstruction term along with three additional terms to enforce better output quality based for instance in constraining the gradients of the image.  Experimental setting shows readily competing results with respect to the existing techniques for reflection removal. Moreover, authors show the potential of the technique for other tasks such as blind denoising.