Single Image Reflection Separation via Dual-Stream Interactive Transformers
Authors: Qiming Hu
Hainuo Wang
Xiaojie Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: Single Image Reflection Separation
Vision Transformer
Image Restoration
Reflection Removal
Primary area: deep_learning_architectures
TL;DR: This paper presents a dual-stream interactive transformer design to handle single image reflection separation problem,  considering both intra-layer and inter-layer feature correlations.

Abstract: Despite satisfactory results on ``easy'' cases of single image reflection separation, prior dual-stream methods still suffer from considerable performance degradation when facing complex ones, i.e, the transmission layer is densely entangled with the reflection having a wide distribution of spatial intensity. The main reasons come from the lack of concern on the feature correlation during interaction, and the limited receptive field. To remedy these deficiencies, this paper presents a Dual-Stream Interactive Transformer (DSIT) design. Specifically, we devise a dual-attention interactive structure that embraces a dual-stream self-attention and a layer-aware dual-stream cross-attention mechanism to simultaneously capture intra-layer and inter-layer feature correlations. Meanwhile, the introduction of attention mechanisms can also mitigate the receptive field limitation. We modulate single-stream pre-trained Transformer embeddings with dual-stream convolutional features through cross-architecture interactions to provide richer semantic priors, thereby further relieving the ill-posedness of the problem. Extensive experimental results reveal the merits of the proposed DSIT over other state-of-the-art alternatives. Our code is publicly available at https://github.com/mingcv/DSIT.

Single Image Reflection Separation via Dual-Stream Interactive Transformers
Qiming Hu, Hainuo Wang, Xiaojie Gue
College of Intelligence and Computing, Tianjin University, Tiajjin 300350, China huqimingot.ju edu. hainuoêtju. edu. en xj maax. maxx..uuogmail. com
Abstract
Despite satisfactory results "easy" cases of single image reflection separation, prior dual-stream methods still suffer from considerable performance degradation when facing complex ones, i.e., the transmission layer is densely entangled with the reflection having a wide distribution of spatial intensity. The main reasons come from the lack of concern the limited receptive field. To remedy these deficiencies. this paper presents a Dual-Stream Interactive Transformer (DSIT) design. Specifically, we devise a dual-attention interactive structure that embraces dual-stream self-attention and layer-aware dual-stream cross-attention mechanism simultancously capture intra-layer and inter-layer feature correlations. Meanwhile, the introduction of attention mechanisms can also mitigate the receptive field limitation. We modulate single-stream pre-trained Transformer embeddings with dual-stream convolutional features through crossaacchitecture interactions to provide richer semantic priors, thereby further relieving the ill-posedness of the problem. Extensive experimental results reveal the merits of the proposed DSIT over other state-of-the-ar alternatives. Our code is publicly available at http://hhhbb..ooo//mmn/////I11
the feature correlation during interaction, and
Introduction
When images are captured through glass-like mediums (semi-reflectors), the reflected scenes appear together with the transmitted ones to different degrees, influenced by many factors such as the material of medium and the illumination of both scenes, among others [42 62 This phenomenon poses significant challenges in various fields, like multi-view sterco imaging, mobile photography, security Therefore, successfully separating the superimposed layers can, on the one hand, enhance the capability of models to serve downstream applications. On the other hand, it paves the way for tackling broader spectrum of layer-decomposition tasks, such as image denoising and watermark/obstacle removal [51] (20].
surveillance, and autonomous driving 44 43
As long-standing blind source separation problem, single image reflection separation (SIRS) has always been challenging, due to the severe ill ill-posedness of disentangling two natural image signals. Generally, the superimposed images can be formulated follows:
T R + R).
It consists of an additive combination of target transmission and reflection layers (T and R. respec- tively) and a residual component, denoted by the mapping C of the two layers. Note that is used to describe the non- oon-linerr/liearr attenuation of the two layers, thus representing group of reflection models (23]. In the literature, two main routes of approaches have been delivered. One tendency (single-stream) is treat the reflection layer as noise/degradation, merely modeling the transmission layer. Alternatively, the other rising trend (dual-stream) pays attention to the reconstruction quality
CCoresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Fusion Ft
Mactix multiplication
3 Element-wise product
Relll
SA Map RPB
Flencntl-iicc
Charnel concar.
ReLU
Key
Inter- Window
Inier- Window
RelU
Combination
Softmax
Separation
Fusion
Fi
RelU"
Value
(a) YTMT behavior
Map
Quety
Fasion
Key
Windoww
Combization
Softmaa
FR
Fasion
FR
Value
Sepuration
Window
FFN
FR
(b) MuGI behavior
DAI behavior
Figure 1: Schematic illustration of dual-streamt interactive behaviors, including YTMT 22 MuGl
223, and our proposed Dual-Attention Interaction (DAI) mechanisms, where Fr and FR represent the feature flows of transmission layer and reflection layer respectively. The superscript / of the feature flows denotes the number of building blocks traversed to derive the flows.
of both the transmission and reflection layers. This work follows the latter principle because the reflection layer may also contain valuable information 43 and imposing constraints from both
perspectives can better regularize the decomposition [2 23
The dual-stream schemes, with IBCLN (29]. YTMT 122 and DSRNet 23 as representatives,
attempt to estimate both of the two layers with Siamese networks that employ two sub-networks with identical architecture and shared weights. Particularly, YTMT and DSRNet advocated dual-stream feature interactions to facilitate the information flow between the streams and finally reconstruct the decoupled layers. Though being effective, the interaction mechanisms used in these two methods do NOT explicitly assess the correlation between dual-stream features during interactions, as illustrated in Fig. (a) and (b). More concretely, assuming have dual-stream features
certain stage in the networks, previous interaction strategies directly pass the undesired information at the current stage (may be required again subsequent stages) from branch to the other without checking if the passed information is needed by the sibling. Actually, in intermediate blocks, some information is very likely delivered back and forth, making the separation process ineffective and inefficient. In contrast, the attention mechanisms in Transformers assign small weights to token pairs with low similarities, which are further suppressed by the Softmax function. In other words, employing cross- cross-attention mechanism seems to have the potential for improving dual-stream interaction. Furthermore, due to the presence of correlated scenes in reflection superposition phenomena across entire images, the task demands keen perception of global information in both streams. Motivated by the above analysis, we propose strategy, called dual-attention interaction, consisting of a dual-stream self- self-attention and a dual-stream cross-attention, as depicted in Fig. (c), to extract both the intra-layer and inter-layer feature correlations explicitly.
adopted networks (e.g., Hyper- Column (60 48 22 and Feature Pyramid (23)) pre-trained high-semantic tasks assist feature extraction. this work, we argue that Transformers, thanks to their generalization and selective at- tention characteristics, should be models typically have trouble in dense prediction due to the lack inductive biases [8] To make Transformers suitable, we develop Dual-Architecuue the interaction between semantically rich features extracted by a pre-trained Transformer and local dual-stream features extracted by a CNN. By this means, the Cross-Architecture Interactions (CAI) can balance the global and local perspectives and combine high-semantic priors with the low-semantic demands of the reflection separation.
In addition, most of recent state-of offte-of-th-ann methods 660 48
powerful the target task. However, pre-trained Transformer
Interactive Encoder (DAIE), which enables
In summary, our primary contributions are as follows:
We propose a novel Dual-A Dual-Attention Interaction (DAI) mechanism to energize Dual-Stream Inter- active Transformers. DAI introduces the explicit correlation assessment within dual streams to effectively address the challenge of reflection separation;

---

We customize bridge, namely the Aual-Architecture Interactive Encoder (DAIE), to connect the pre-trained Transformer model with the task of layer decomposition, which alleviates the inherent ill iol-posessess of the problem;
Through extensive experiments on multiple datasets, we demonstrate the efficacy of our design with superior performance over other SOTA competitors, both quantitatively and qualitatively. Moreover, the better generalizability compared to previous methods is also verified.
2 Related Work
Low-level Vision Transformers. Building upon the attention mechanism [3]. Transformers were initially developed by the community of natural language processing [41]. and soon became popular across various domains because of their remarkable modeling 17). scaling 24 [56] 36], and trans- ferring [T] 6 4 abilities. Introduced by [12]. Vision Transformers (ViT') have shown advantages on
large number of visual tasks [56 34 66 57 50 g.
For low-level purposes, IPT 6 was developed to handle multiple restoration tasks with a shared standard Transformer body, which required a large number of parameters for good performance without suitable task-related priors. Swin Transformer (35) introduced the shifted window attention mechanism, which reduced the computational cost of attention while incorporating inductive biases for images, inspiring a series of subsequent works. SwinIR [33] equipped cexidual-in-resddue structures [46, 61] with the Swin Transformer block, exploring its capabiliiy in low-level vision tasks. ELAN [39] performed multiple window attentions of varying sizes in parallel and fused them, using grouped four-directional offset convolution layers for local feature extraction and cross-window association. Chen et al. (7] introduced an overlapping attention mechanism in their HAT model to establish cinssw connections and employed same-task pre-training for better performance. Zhang et al. proposed ART [58]. which utilized a sparse window attention mechanism akin to dilated convolution, alternating it with window attention, thereby replacing the shifted window mechanism. UFormer [47 embedded window attention modules into a U-shaped network, which captured cross- window associations beyond the current scale. Moreover, Restormer (55) introduced a transposed attention mechanism. resembling channel attention. While faster, this approach somewhat neglected spatial correlations. Retinexformer (5) employed this design for low-light image enhancement. DAT [9] alternated between window self-attention and window transposed self-attention to address spatial correlation deficiencies. Overall, these methods mostly opt validate their designs on tasks like image super-resolution, which focus on reconstructing single component, overlooking the intrinsic advantages of attention mechanisms in component decomposition tasks.
Single Image Reflection Separation. Single image reflection separation, with looser data assump- tions, relies more heavily priors alleviate its inherent ill-posedness. Traditional methods developed priors like edge sparsity (8 27], manual annotation (26). or relative smoothness [II, some of which were further leveraged by deep learning methods. Although malltiple-image solu- tions [37] D 16 339 30, 18 38
53 [19] have shown satisfactory performance facing weaker ill-posedness, these methods typically rely on sequences of images captured with rotating polarizers or moving cameras, which limits their applicability.
51
In deep learning methods, CEILNet [14] applied the relative smoothness assumption to data synthesis and used an edge detection network to emphasize edge information. Zhang et al. [60 proposed gradient mutual exclusion loss promote edge sparsity and introduced the HyperColumn and perceptual loss to incorporate high-level semantics. ERRNeT [48] aimed to expand the receptive field and utilized non-aligned images for data augmentation. These methods employed end-to-end single- branch networks for estimation, but the lack of interaction between layers led to inefficiency and untidy separations. In another way, BDN (54) alternated between estimating transmission and reflection layers, considering their mutual dependency. RAGNet (32) estimated reflection first and then used its features modulate the estimation of the transmission layer and mask. Dong et al. [ emphasized the importance of the reflection layer, using multi-scale Laplacian features with LSTM for iterative estimation. RRW (25) utilized a cascaded reflection detector and remover and proposed a more reasonable data acquisition scheme. Song et al. [40] proposed robust SIRR model based on a multi- scale Transformer architecture, but it only learned to restore the transmission layer, and the usage of the Transformer was not well motivated. IBCLN [29 introduced convolutional LSTM network, using a dual-branch structure for reflection and transmission reconstruction but lacking interaction between branches for cross-verifying the accuracy of decoupling. Hu and Guo proposed a dual-stream
3

---

Upsampler
Connection
Architeere Interaction
Transmission Information Flow
Reflection Infommation Flow
Mixed Infontation Flow
Transformer Block x
DSLP Block x
RPB Table
DAIB
82:
DSLP Block
Fy
Ring dlar W-MSA
Trunsformer x ig
DSLP Block
DAIR
LayerNorm
x
Inter window combination
DS SA
Trunsformer Block By
DSLP Block
DAIB
LRPB Talle
DS-SA
Fy
Transformer Black x
DSLP Block x2
Layer awarr
DAIB
LayerNorm
FR
CrossArtu
Global Prior Extractor
Local Prior Extractor
RJN
F
Dual Architecture Interactive Encoder
Dual Stream Interactive Decoder
Dual- Strea-m Interactive Transformer (DSIT)
Dual-Atteation Interactive Block (DAIB)
Figure 2: (a) The overall architecture of our proposed Dual-Stream Interactive Transformer, which consists of a dual-are auulllarcitecctre interactive encoder and dual-stream interactive decoder, injecting the global prior into local features and aggregating them in dual-stream from bottom to up. (b) visual illustration of our proposed dual-attention interactive block, which provides both intra-layer self-attention and inter-layer cross-atee noss-attention, capturing holistic feature correlations.
interactive reflection separation strategy [22] called YTMT, assessing information utility through activation functions and exchanging low-value information, facilitating information retention and efficient interaction in the high-dimensional feature space. Their subsequent DSRNet (23) used MuGI for efficient interaction and introduced a non-linear residual component estimate the overexposure performance. However, neither the
during the layer superimposition, achieving state-of-the YTMT nor the MuGl mechanism explicitly assessed the correlation of exchanged information, which can introduce errors and lower the decoupling efficiency. To further alleviate the ill ill-posedness of the problem, a recent concurrent work (44) utilized wavelet priors and diffusion models to guide frequency-doman -awpenc---mmm----bbbi reflection removal. The other work [64] used natural language prompts for reflection separation, which, however, required accurate paired prompts, incurring additional costs.
3 Methodology
Interactive Encoder (DAIE) and Dual-Stream Interactive Decoder (DSID). The DAIE leverages both a pre- trained Transformer (Global Prior Extractor, GPE) and task-specific dual-stream CNN (Local Prior Extractor, LPE), capturing global and local features through specialized extractors. The mixed global information is then injected into the dual-stream local flows via Cross Architecture Interactions (CAI), ensuring comprehensive information utilization. Subsequently, the DSID separates and aggregates embeddings hierarchically through our newly developed Dual-Attention Interactive Block (DAIB, illustrated in Fig. 2 (b)). These components are detailed in the following subsections.
Our overall architecture is illustrated in Fig. (a). which comprises a
3.1 Dual-Attention Interactive Block
As depicted in Fig. (b). our proposed DAIB embraces a dual-stream design, taking both transmission and reflection feature flows (F! and FJN respectively) as inputs. After that, a layer normalization and two parallel attention mechanisms, namely dual-stream self-attention (DS-SA) and dual-stream cross- attention (DS-CA) are applied to the feature flows, capturing both inter- and intra-layer correlations. Subsequently, we derive the output feature flows F and FOT after passing the features through layer normalization and a forward network in the form of the dual-stream bocalty-preservinn

---

block (DSLP Block). The detailed computation procedure is displayed in Alg. D We provide detailed explanation of the dual attention mechanism the remainder of this subsection.
Efficient Dual-Stream Cross-Attention Mechanism. We present a simple yet effective cross- attention mechanism for dual-stream Transformer models via extension of the self-attention mechanism. Given the feature streams of transmission layer Fr E RN and reflection layer FR XNCC, we concatenate them along the token dimension form the input matrix XCA
Fr We then compute the query QCA. key KCA, and value VcA matrices for cross-
R2NxC
FR
attention by applying the linear transformations:
Qca = XcA KCA = XcA
VCA T W..W..
(2)
where Wg. We, W. RCDD denote the weight matrices that project the input features from C-- dimensional channels into D-dimensional hidden representations. The cross-attention score matrix
ACA E R2Nx2N computed
Fr FR
ACa E 2222 Softmax (Cr Kca) K(A) = Softmax
W W
FT
F
(3)
W.W,WIR F W W.W. W.
FR
Softmax(
F W.,,
where the intra-layer terms F. W,W,W.F. F7 and FR W, W. FR represent interactions within the trans- mission stream FT and the reflection stream FR. respectively. The inter-layer terms FTW, W. W FR and FR W, W F indicate interactions between Fr and FR. By denoting the Softmax function with a scaling factor
D as S.), the output matrix YeA is then calculated as:
S W. F,W, + S(F,W, W. FR W S(FRW, W. F, F.W. + S(FRW, W FRW.
Yca = ACA
(4)
further simplify the form of CA by introducing G(Z), Z2) S(Z, W, W z YZ W.. where
Z, E Rx and Z2 NxC can be chosen between Fr and FR. yielding the follows:
9(FT. FT) + G(T, FR)
FCA
YCA
(5)
((FR. + ((R,
FCA
We finally obtain the output of the dual-stream cross-attention FG* (FT, F(F,,FR) and FCA G(RR Fr) ((FR. FR), which are the combined effects of intra-layer and inter-layer interactions. Meanwhile, if we concatenate the dual-stream features along the batch dimension. obtaining the input matrix for the dual-stream self-attention mechanism XSA NxC we can further boost the parallelism of our model.
Dual-Attention Design. Based the above analysis, define the following dual-attention mechanism:
Ysa- DS-SA(QsAA KSA, VSA) SoftMax QSA K SAAVADDD
SA /DD B5A) VSA,
(6)
YC = DS-CA(QCs. Koa, VCA) SoftMax /T/VDD Bex)
where QSA KSa, and VSA are derived in the DS-CA. Note that, the number of tokens is doubled in DS-CA compared to DS-SA. To reduce the computation burden, we employ window-base attention mechanism for our dual-attention design. this way, QsA, KSA, VSA R22 x Nwx x Qcx, KCA, VCA E R x2N xD where NT denotes the total number of windows, Nw stands for window size. BSA Nww..N represents the relative position bias (35], which provides same bias values with respect to the same distance between two tokens in a window. It is obtained by indexing learnable lookup table BLUT E ()x/N/-11x(22//))))
through the predefined relative
indexes USA E XNw Each item of USA is mapped distance of two locations: üi b(p: pt)). p is a 2-D point in a window, the coordinates of which fall between 0 and /Nw 1, and each
5

---

F
FR
pCP
FT
FR
Inputs
Strl-Streamm Local Priors
Global Prines
Dual-Stream Features after CAI
Dual-Stream Features after DAIBs
Figure 3: Visualization of extracted local priors, global priors, their coos--achiteectur-iiteeeac dual-stream features and features after the DAIBS of two reflection-superingn -uperrioes inputs. All the above features are from the second level of our DSIT model and are channel-wise averaged to display.
distinct pj is mapped into single index by t(-). For DS-CA, we propose the Layered Relative Position Biases (LRPB). BCA & IR2 x2Nw which are indexed from the extended lookup table BCA E (12//W-1)x(2/NWr
by the layered relative indexes UcA E 2R2 x2Nw Each element is mapped by subtracting two 3-D points v/g = f(( - vz). e is 3-D point in a layered window, with an additional dimension representing to which layer the token belongs, and ('(.) maps 3-D locations into single indexes.
Dual-Stream Locality-Preserving Block. Since reflection separation is dense prediction task, primary consideration during architecture design is to maintain the local information. Therefore, we introduce the DSLP Block in our DSIT structure, which can be any convolutional dual-stream network modules. To focus on the enhanced interaction capabilities achieved by our dual-attention design, we avoid introducing additional novel local modules, opting instead employ the MuGI Block [23] as the implementation of the DSLP Block. This approach isolates the performance gains attributed solely to the dual-attention mechanism, DSRNet. One could, of course, substitute our design with alternative specialized dual-stream modules, potentially achieving even better model performance.
evidenced in comparisons with models like
3.2 Dual-Architecture Interactive Encoder
As depicted in Fig. 2 (a), our proposed DAIE integrates both global and local prior extractors. The single-stream global features modulate the dual-stream local features hierarchically through cross- architecture interactions (CAI), which are implemented with our proposed dual- attenttennion interactive blocks. Formally, we have DAIB (FI OF FTP) and DAIB(FOP FLP) ). where FOP and FLP F represent global and local features respectively. F denotes the transmission information flow and FR signifies the reflection stream. In an effort provide illustrate the feature visualization of DSIT in Fig. 3 As shown, the dual-stream local priors focus on different components of the inputs but lack precise layer-specific attention. After being modulated by the global priors FGP via our proposed CAI and aggregated with the lower stream, we obtain FCAI and FCAI, which are significantly separated. Furthermore, the modulated dual-stream features are fed into a group of DAIBS, resulting in feature separations FAAAIB FDAIB of higher quality.
intuitive understanding of our DAIE design, we
3.3
Loss Function
Pixel reconstruction loss. To compel the consistency of the restored layers T and R in the spatial domain with their ground-truth scenes T and R and the layer superimposition modeling, we introduce the following loss function:
Cpiz T - T + | R
+ a ( R) E(T,
where & denotes a learnable term to constitute the reflection superposition. || ||2 and || ||| represent the fg and ε norms, respectively. is hyperparameter to balance the intra-layer and inter-layer fidelity. By enforcing the reconstruction loss with learnable residual term, the restored layers appear to be cleaner and completed.

---

Table Quantitative results on four
testing datasets of SIRS models. The best results are
displayed in bold, while the second-best are underlined. i means data setting II is employed to train
the model. * represents additional prompts are introduced. A reflects extra data pairs are involved.
Real20 (20)
Objects (200)
Postcard (199)
Wild (55)
Average
Methods
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNN
SSIM
Zhang al.
60
22.55
0.788
22.68
0.879
16.81
0.797
21.52
0.832
20.08
0.835
BDN [54]
18.41
0.726
22.72
0.856
20.71
0.859
22.36
0.830
21.65
0.849
ERRNet 48
22.89
0.803
24.87
0.896
22.04
0.876
24.25
0.853
23.53
0.879
IBCLN
29
21.86
0.762
24.87
0.893
23.39
0.875
24.71
0.886
24.10
0.879
RAGNet 32
22.95
0.793
26.15
0,903
23.67
0.879
25.53
0.880
24.90
0.886
DMGN
15
20.71
0.770
24.98
0.899
22.92
0.877
23.81
0.835
23.80
0.877
Zheng et al.
63
20.17
0.755
25.20
0.880
23.26
0.905
25.39
0.878
24.19
0.885
YTMT 22
23.26
0.806
24.87
0.896
22,91
0.884
25.48
0.890
24.05
0.886
RohustSIRR 40
23,30
0.827
24.90
0.917
19.91
0.868
23.67
0.884
22.59
0.889
DSRNet 23
24.23
0.820
26.28
0.914
24.56
0.908
25.68
0.896
25.40
0.905
PrompIRR 44
24.11
0.813
24.17
0.859
23.03
0,895
26.43
0.930
23.95
0.880
Ours
25.06
0.836
26.81
0.919
25.63
0,924
27.06
0.910
26.27
0.917
Dong ef al
23.34
0.812
24.36
0.898
23.72
0.903
25.73
0.902
24.21
0.897
DSRNet
23
23,91
0.818
26.74
0.920
24.83
0.911
26.11
0.906
25.75
0.910
RRWIA
21.83
0.801
26.67
0.931
24.04
0.903
26.49
0.915
25.34
0.912
Zhong et al." 644
24.05
0.824
26.51
0.927
25.02
0.915
26.23
0.925
25.75
0.917
Ours!
25.22
0.836
27.27
0.932
25.58
0.922
27.40
0.918
26.49
0.922
Table 2: Quantitative results on the "Nature" testings set SIRS methods trained under data setting II.
The best results are shown in bold, and the second-best underlined.
Metrics PSNR
ERRNet-F
IBCLN
YTMT
Dong et al.
DSRNet
RRW
Zhong et al.
Ours
22.18
23.57
23.85
23.45
25.22
26.04
23.87
26.77
SSIM
0.756
0.783
0.810
0.808
0.832
0.846
0.812
0.847
Gradient reconstruction loss. Considering the gradient independence, as a traditional prior in blind-source decomposition, we simultancously encourage the models to restore the ground-truth gradient and penalize the intersected gradient as follows:
N N-1
Lyrad : VT
TTIl
+
VR
+
D n=0
R
(8)
D(T, R) == tanh (G tanh
Ez (RR)
where V denotes the difference operator of images. 1 RD are 2n down-sampled version of T and R. { and &2 are normalization factors. The exclusion term, introduced by [60], ensures the multi-scale exclusion of the two layers in the gradient domain.
Feature reconstruction Loss. To promote the perceived quality of decoupled layers, hamesss the following feature reconstruction loss:
Clea
Σ wi (o: (T)
(9)
where (-(-) represents the intermediate feature of the pre-trained VGG-19 model, where i {2, 7, 12, 12, 30) tells the layer id. w; balance the weights of hierarchies.
Total Loss. The full training objectives Lsotal defined as follows:
Liotal Cpia + X2 Xa
(10)
where A 1, A2 = 1, and 0 0.01 are coefficients for balancing different loss terms.
Experimental Validation
4.1 Implementation Details
Datasets. Our training datasets include both synthetic and real-world images. Following 2 we design two data settings for fair comparison: I. 7,643 synthesized pairs randomly sampled from the
7

---

Input
Zhang et al. 60
ERRNet 48
IBCLN 29
YTMT 22
Dong et al. |II
DSRNet 23
RRW 65
Ours
GT
Figure 4: Visual comparison of transmission layer predictions between previous state-of state-of-the-arts the--rtss and ours on samples from Real20 600 and SIR? datasets. Please note the areas the boxes.
PASCAL VOC dataset [13 in each epoch and 90 real pairs from [60]. II. 200 extra real pairs from the "Nature" dataset [29]. and 13, 700 synthesized pairs sampled from [60] instead. The training image size is fixed as 384 384. The window size of attention mechanisms, Nw fixed to 12 x 12, and the number of windows, Nr, varies depending the spatial scale of the features.
Training Strategy. Our models are all implemented via the PyTorch framework and optimized with Adam optimizer for 20 or 80 epochs based on different data settings. The learning rate is fixed as 10-4 with batch size of on a single RTX 3090 GPU. Given real-world data pairs are hard to acquire, we additionally propose data augmentation operation Reflection Mixup (RefMix) for the training real pairs, formulated as Taug = 7 T,, where y [0, 1] uniformly sampled.
4.2 Performance Evaluation
Quantitative comparison. As shown in Tabs. [ndd] 2 we make comparison between ours and state-of-the-ant methods on five real-world datasets, including Real20 [60]. Nature20 [29] and three subsets the SIR² Dataset 42). is noteworthy that our models trained on both data settings show superior performance over the alternatives on most testing sets, including those that involve extra real-world data [65] and language prompts (64). The superiority is attributed not only to the improved generalizability afforded by the hybrid Transformer architecture but also the dual- attention interactive design that directly assesses intra-layer and inter-layer correlations, which shows impressive efficiency on SIRS tasks and has high potential for other decomposition tasks.
Qualitative comparison. To evaluate our proposed model aesthetically, we first present a visual comparison of estimated transmission layers in Fig. The two superimposed input images

---

Input
IBCLN 29
YTMT 22
Dong et al. ||
De 23
Ours
Figure 5: Visual comparison of transmission predictions between previous state-of-the-an and ours in real-worke scenarios additionally captured in this paper. The broad advantages demonstrated by our method across these diverse conditions highlight its superior generalization capability.
sampled from Real20 and SIR² datasets, respectively. The two cases are representative since the first case is captured outdoors and contains both specular and weak reflections, while the second one is taken indoors with a relatively uniform reflection layer, which is highly entangled with the transmission structure. As can be seen, for the first case, ERRNet, YTMT, and Dong al. cannot recognize the reflection regions successfully. Zhang et al. removes the reflections at the cost of introducing color bias and artifacts. IBCLN, DSRNet, and RRW separate either the specular or the weak reflection parts, lacking the ability to correlate the reflection components of different intensities. As for the second one, most alternatives fail to separate the majority of the reflection layer. Although Dong et al. shows improvement over previous methods, it still leaves blurry reflection components in its result transmission layer. With a better layer modeling capability, our models conquer such a problem, providing strikingly clearer reconstructions.
Additionally, we specifically captured several in-the-wild test cases, as illustrated in Fig. 5 Unlike standard benchmarks that often incorporate artificial glass plates, these cases utilize real-world reflective surfaces, including reflections from accumulated water-condinn conditions entirely absent from the training set. The superior performance of our approach, in comparison to previous state-of- the-art methods, underscores its robust generalization capacity and practical applicability. We further compare the reflection predictions between dual-stream reflection separation models in Fig. 6 Notably, our DSIT model yields significantly more plausible results, exhibiting superior content fidelity and color accuracy, attributed to the enhanced information selection capabilities of our design.
4.3 Ablation Study
Model selection for GPE. Generally speaking, the design of our DAIE allows different choices of global prior extractors, which aim to provide semantic priors and/or non-local information. To elaborate on the efficacy of different GPE models, we compare the settings of using ResNet101 211, FocalNet-L [52), PVTv2-b4 [45), and Swin-L (35) models, which are all pre-trained on image classification tasks and finetuned on object detection tasks. As shown in Tab. 3 the 101 as a CNN backbonn provides limited global priors and poorly generalizes to the Real20 dataset. PVTv2 comprises few inductive biases, thus exhibiting inferior performance to the FocalNet and Swin Transformer, which preserve local dependency.
Design of CAI operation. Our cross- architeccee interaction mechanism is designed to exploit useful information from the priors extracted by GPE and LPE, and thus, various feature fusion operations can be taken. Here, we build baseline "Add" that simply sums the information flow provided by the different prior extractors. Further, we propose two variants based on Cross Attention design ("CrossAttt") and DAIB ("DAIB") mechanisms, respectively, As Tab. 3 shown, cross-attention is not

---

Input
IBCLN 29
YTMT P.P.
DSRNet 23
Ours
GT
Figure 6: Visual comparison of reflection layer predictions between previous state-of-theearts and ours on the SIR² dataset. Our method shows significant superiority over previous dual-stream arts.
capable of fusion cross- cessstrccitecture information, leading to worse performance than simple addition, while proposed DAIB bridges the local and global priors, obtaining superior results.
Design of DAIB module. To demon- strate the effectiveness of our DAIB de- sign, we constructed three baseline mod- els. The first baseline, "MLP FFN", re- places our DSLP Block with standard MLP module. The second, "w/o DS- CA", omits the DS-CA mechanism, and the third, "w/o DS-SA", removes the DS-SA mechanism. As shown in Tab. model performance degrades when sub- stituting DSLP Block with MLP, primar- ily due the reduced inductive biases and feature interactions. Moreover, the removal of either DS-CA DS-SA sults in inferior performance, particu- larly for DS-CA. This highlights the crit- ical role of our proposed dual-attention interactive mechanism.
Table 3: Ablation study on different factors of our design.
Real20 (20)
SIR² (454)
Factors
Instances
PSNR
SSIM
PSNR
SSIM
ResNet101
21.59
0.777
24.42
0.886
PVTv2-b4
23.16
0.793
24.44
0.890
GPE
FocalNei-L
24.13
0.824
25.52
0.910
Swin-L
25.06
0.836
26.32
0.920
Add
24.62
0.825
25.79
0.917
CAI
CrossAttn
24.50
0.819
24.76
0.896
DAIB
25.06
0.836
26.32
0.920
MLP FFN
23.65
0.817
25.38
0.909
w/o DS-CA
24.48
0.823
25.89
0.919
DAIB
wo DS-SA
24.47
0,827
26.
0.920
DAIB
25.06
0.836
26.32
0.920
w/ LRPB
24.93
0.821
25.99
0.917
LRPB
w/ LRPB
25.06
0.836
26.32
0.920
RelMix
24.72
0.823
26.27
0.915
RefMix
w/ RefMix
25.06
0.836
26.32
0.920
Design of LRPB mechanism. LRPB provides initial bias to each attention
point according to the spatial and layer locations, injecting a layer-aware prior into the attention mechanism. Through the ablation study of whether not equip with LRPB shown in Tab. "LRPB", we show its merit in handling the layer decomposition problem.
The usage of RefMix. To evaluate the effectiveness of our proposed RefMix, comparison of employing it or not is made in Tab. 3 showing that aids the reflection separation.
Due to the page limitation, more visual analyses are organized in the appendix.
5 Concluding Remarks
In this study, dual-stream interactive Transformer has been designed address the challenge of single image reflection separation. To harness high-quality priors from pre-trained Transformer models, we developed hullarchitecturr interactive encoder, which can effectively fuse multi-source information with adaptive emphases. Additionally, we introduced a novel dual-attention interactive block that utilizes both an effective dual-stream self-attention mechanism and layer-aware dual- stream cross-attenion module to separate the entangled features. Comprehensive experiments together with ablations have been conducted to verify the advances of our method. Looking forward, several interesting points deserve future exploration. For instance, larger vision foundation models can provide more substantial priors for low-level vision tasks. relative position bias design tailored for a specific vision task will be also beneficial. Furthermore, our RefMix technique is likely to extend its benefits to other data-hungry visual tasks. These considerations may inspire future research on low-level Transformer designs.
10

---

Acknowledgement
This work was supported by the National Natural Science Foundation of China under Grant nos. 62372251 and 62072327.
References
[1] Amit K. Agrawal, Ramesh Raskar, Shree K. Nayar, and Yuanzhen Li. Removing photography artifacts using gradient projection and flash-exposure sampling. TOG, 24(3):828-835 2005.
[2] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. VATT: transformers for multimodal self-supervised learning from raw video, audio and text. In NewrIPS, pages 24206-24221, 2021.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
[4] Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Pre-training of image transformers. In ICLR, 2022.
[5] Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang, Radu Timofte, and Yulun Zhang, Retinex- former: One-stage retinex-base based transformer for low-light image enhancement. In ICCV, pages 14470-12479) 2023.
[6] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, pages 12299-12310, 2021.
[7] Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong. Activating more pixels image super-resolution transformer. In CVPR, pages 22367-22377, 2023.
[8] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2023.
(9] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang. and Fisher Yu. Dual aggregation transformer for image super-resolution. In ICCV, pages 2278-112277 2023.
[10] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171-4186, 2019.
[11] Zheng Dong, Ke Xu, Yin Yang, Hujun Bao, Weiwei Xu, and Rynson W. H. Lau. Location-aware single image reflection removal. In ICCV, pages 4997-5006, 2021.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[13] Mark Everingham, Luc Van Gool, Christopher K. I. Williams. John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. IJCV, 88(2):303-33,, 2010.
[14] Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, and David P. Wipf. A generic deep architecture for single image reflection removal and image smoothing. In ICCV, pages 3258 1288-3267,222 2017.
[15] Xin Feng, Wenjie Pei, Zihui Jia, Fanglin Chen, David Zhang, and Guangming Lu. Deep-masking generative network: A unified framework for background restoration from superimposed images. TYP, 30:4867-4882 2021.
[16] Kun Gai, Zhenwei Shi, and Changshui Zhang. Blind separation of superimposed moving images using image statistics. TPAMI, 34:19-32, 2012.

---

[17] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In CVPR, pages 16081-160% 2022.
[18] Xiaojie Guo, Xiaochun Cao, and Yi Ma. Robust separation of reflection from multiple images. In CVPR. pages 2195-220000 2902,, 2014.
[19] Byeong-Ju Han and Yoe-Y Sim. Reflection removal using low-rank matrix completion. In CVPR, pages 3872 -8800, 2017.
[20] Junlin Han, Weihao Li, Pengfei Fang. Chunyi Sun, Jie Hong, Mohammad Ali Armin, Lars Petersson, and Hongdong Li. Blind image decomposition. In ECCV, pages 218 287, 2022.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[22] Qiming Hu and Xiaojie Guo. Trash or treasure? an interactive dual-stream strategy for single image reflection separation. In NeurIPS, pages 24683- 24694, 2021.
[23] Qiming Hu and Xiaojie Guo. Single image reflection separation via component synergy. In ICCV, pages 13138-133 13147, 2023.
[24] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint, 2020.
[25] Chenyang Lei, Xuhua Huang, Mengdi Zhang, Qiong Yan, Wenxiu Sun, and Qifeng Chen. Polarized reflection removal with perfect alignment in the wild. In CVPR, pages 1747-1755, 2020.
[26] Anat Levin and Yair Weiss. User assisted separation of reflections from single image using sparsity prior. TPAMI, 1944)11647-11554 2007.
[27] Anat Levin, Assaf Zomet, and Yair Weiss. Learning to perceive transparency from the statistics of natural scenes. In NeurIPS, pages 1247-1254, 2002.
[28] Anat Levin, Assaf Zomet, and Yair Weiss. Separating reflections from a single image using local features. In CVPR, pages 306-313, 2004.
[29] Chao Li, Yixiao Yang, Kun He, Stephen Lin, and John Hoperoft. Single image reflection removal through cascaded refinement. In CVPR, pages 3562-3571, 2020.
[30] Yu Li and Michael S. Brown. Exploiting reflection change for automatic reflection removal. In ICCV, pages 2432-2439, 2013.
[31] Yu Li and Michael S. Brown. Single image layer separation using relative smoothness. In CVPR, pages 2752-2759, 2014.
[32] Yu Li, Ming Liu, Yaling Yi, Qince Li, Dongwei Ren, and Wangmeng Zuo. Two-stage single image reflection removal with reflection-aware guidance. arXiv preprint, 2020.
[33] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCVW, pages 1833-1844, 2021.
[34] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao. Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In CVPR, pages 11999-12009, 2022.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 9992-10002, 2021.
[36] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4172-4182, 2023.
12

---

[37] Bernard Sarel and Michal Irani. Separating transparent layers through layer information exchange. In ECCV, pages 228-341.2 2004.
[38] Christian Simon and In Kyu Park. Reflection removal for in-vehicle black box videos. In CVPR, pages 4231-4239, 2015.
[39] Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. Image-based rendering for scenes with reflections. TOG, 31(4):00011100:10 2012.
[40] Zhenbo Song. Zhenyuan Zhang, Kaihao Zhang, Wenhan Luo, Zhaoxin Fan, Wenqi Ren, and Jianfeng Lu. Robust single image reflection removal against adversarial attacks. In CVPR, pages 14688-24696 2023.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neur/PS, pages 5998 -0008, 2017.
[42] Renjie Wan, Boxin Shi, Ling- Yu Duan, Ah-Hwee Tan, and Alex C. Kot. Benchmarking single-image reflection removal algorithms. In ICCV, pages 3942-3950, 2017.
[43] Renjie Wan, Boxin Shi, Haoliang Li, Ling-Yo Yu Duan, and Alex C. Kot. Reflection scene separation from single image. CVPR, pages 2395-2403, 2020.
[44] Tao Wang, Wanglong Lu, Kaihao Zhang, Wenhan Luo, Tae-Kyun Kim, Tong Lu, Hongdong Li, and Ming-Hsuan Yang. Promptrr: Diffusion models as prompt generators for single image reflection removal. arXiv preprint, 2024.
[45] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Compul. Vis. Media, 83)4415-424, 2022.
[46] Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In ECCVW, volume 11133, pages 63-79, 2018.
[47] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In CVPR, pages 17662- 17672, 2022.
[48] Kaixuan Wei, Jiaolong Yang, Ying Fu, David P. Wipf, and Hua Huang. Single image reflection removal exploiting misaligned training data and network enhancements. In CVPR, pages 8178-8187, 2019.
[49] Patrick Wieschollek, Orazio Gallo, Jinwei Gu, and Jan Kautz. Separating reflection and transmission images in the wild. In ECCV, pages 90-105, 2018.
[50] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, José M. Álvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurlPS, pages 12077-12090, 2021.
[51] Tianfan Xue, Michael Rubinstein, Ce Liu, and William T. Freeman. A computational approach for obstruction-fres photography. TOG, 14:79:1-79:11 2015.
[52] Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. In NeurIPS, 2022.
[53] Jiaolong Yang, Hongdong Li, Yuchao Dai, and Robby T. Tan. Robust optical flow estimation of double-layer images under transparency or reflection. CVPR, pages 1410-1419, 2016.
[54] Jie Yang, Dong Gong, Lingqiao Liu, and Qinfeng Shi. Seeing deeply and bidirectionally: deep learning approach for single image reflection removal. In ECCV, pages 675-691, 2018.
[55] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, pages 5718-5729, 2022.
13

---

[56] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. CVPR, pages 1204-1213, 2022.
[57] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In ICLR,, 2023.
[58] Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Accurate image restoration with attention retractable transformer. In ICLR, 2023.
[59] Xindong Zhang. Hui Zeng, Shi Guo, and Lei Zhang. Efficient long-range attention network for image super-resolution. In ECCV, pages 649-667, 2022.
[60] Xuaner Cecilia Zhang, Ren Ng. and Qifeng Chen. Single image reflection separation with perceptual losses. CVPR, pages 4786-4794 2018.
[61] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In ECCV, pages 294-310, 2018.
[62] Qian Zheng, Jinnan Chen, Zhan Lu, Boxin Shi, Xudong Jiang, Kim-Hui Yap, Ling- Yu Duan, and Alex C Kot. What does plate glass reveal about camera calibration? In CVPR, pages 3022-3032, 2020.
[63] Qian Zheng, Boxin Shi, Jinnan Chen, Xudong Jiang, Ling-Yt Yu Duan, and Alex C. Kot. Single image reflection removal with absorption effect. In CVPR, pages 13395-13400 2021.
[64] Haofeng Zhong, Yuchen Hong, Shuchen Weng, Jinxiu Liang, and Boxin Shi. Language-guie guided image reflection separation. In CVPR, pages 24913-24922, 2024.
[65] Yurui Zhu, Xueyang Fu, Peng-Taas Tao Jiang, Hao Zhang, Qibin Sun, Jinwei Chen, Zheng-Jun Zha, and Bo Li. Revisiting single image reflection removal in the wild. In CVPR, 2024.
[66] Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. In ICCV, pages 6725-6735, 2023.
14

---

A
Appendix
A.1 Algorithm for Dual-Attention Interactive Block
We offer the overall procedure of the DAIB in Alg. corresponding to the description in Sec. 3.1
A.2 Visual Analysis of Ablation Study
As shown in Fig. provide visual results of our ablation study, Among them, "GPE-ResNeel011" "GPE-PVTV2-br and "GPE--ocalNeetLL" show the effects of selecting different GPE models. ResNet101 provides limited global priors, thus exhibiting trivial performance. DSIT with PVTv2- b4 tends to concentrate on the wrong components during the layer decomposition due to a lack of inductive biases. producing unpleasant results. FocalNet-L replaces explicit query-key-value interactions with focal modulation module. Although it shows merits over the former two settings, the lack of explicit information assessment limits its usability in our framework. The settings of "CAI- Add" and "CAI-CrossAttrr fail to effectively integrate global and local priors, resulting in inferior performance and misguided attention. "DAIB-N -DAI-MMPP FFN" replaces the DSLP Block with MLP FFN, which also cannot separate the reflection successfully due to insufficient inductive biases. Moreover, we can observe that removing either attention mechanism our dual attention interaction design leads to appreciable performance degradation. Layered Relative Position Bias (LRPB) provides layer-aware attention initialization, while "w/p LRPB" loses the ability to separate reflections in some areas. As shown by "w/o RefMix", specific reflection patterns can be hard to separate after eliminating the reflection mixup augmentation.
A.3 Visual Illustration of the RefMix
We visualize the results of our Reflection Mixup (RefMix) augmentation in Fig. 8 The hyperparameter y controls the blending rate of and T. Since the image pairs are aligned, adjusting their blending rate is actually tuning the intensity of the reflection layer. Some regions covered by strong reflections can thereby be learned reasonably.
A.4 More Visual Comparisons
We provide more visual comparisons of estimated transmission and reflection layers in Figs. The overall superiority shown by our models demonstrates the effectiveness of DSIT design.
12
Note that the case is sampled from the Real20 dataset Fig. 12 and no ground- truth reflection layer provided with the dataset, transmission layer as the reference. It is clear that the hollows should not appear in the real reflection scene. Our models present potential ability of inpainting these areas with the aid of global priors.
put the result of the input image subtracted by the ground-truth
A.5 Limitations
Previous SIRS methods typically fail to deal with the regions dominated by reflections, which show negligible defocus effects, shown in Fig. [3 Although model takes significant step forward to solve the hard case beyond the former methods, still leaves some visible reflections due to the deviation of the case from the relative smoothness assumption. We can expect that it will be better handled when involving more real-world pairs.
15

---

Algorithm The computational process for Dual-Attention Interactive Block.
Require: Dual-stream features F/ and F
Ensure:
Interacted dual-stream features FT and FR.
1: LayerNorm:
2: FT =LN FR = (N(FFR)
3: Combine tokens both inter/intrawwindowwwwe
X Concat (FT FR dim==)
4:
5:
X = Concat ( (FT FR dim=1)
6:
Apply DS AS-SA and DS-CA, respectively:
SAA DS (X
7:
8:
X
= DS (
9:
Split tokens back dual streams:
10:
FSA
T
F Split(X) dim=0)
FCA
FF Split(X) dim=1)
12:
Combine the dual-attention results with skip connections:
13:
FDA
F
FSSA
FA
14:
FDA
R
F
F
F
15:
Apply the DSLP Block as FFN:
EEEN
DSLP (,
T
17:
Derive the outputs FT" FR..
FOT
FDA + FFFN
19:
FOT
R
RA + PEEN
Input
NellRe
GPE-PVIV2-64
CAI-Add
CAI-CrossAttn
DAIB-MLP FFN
DAIB-w/o DS-CA
DAIB-wio DS-SA
w/o LRPB
w/o RelMix
Figure 7: Visual results of DSIT variants shown in Tab.
The case is sampled from the SIR² dataset.
16

---

Tourin
OO
RICHARD C. BLUM
RICHARD BLUM
RCCAARD C. BLUM
RICHARD C. BLUM
RICHARD BLUM
RICHARD C. BLUM
==0
0=2
y
0.6
y 0.8
y=
Figure 8: The RefMix results according to different yys which enrich the real pairs during training.
Input
7hang
ERRNet
IBCLN
RAGNet
YTMT
Dong al.
DSRNet
RRW
Ours (data setting !)
Ours (data setting II)
GT
Figure 9: Visual comparison of transmission layer predictions between previous arts and ours. The case is sampled from the SIR² dataset.
17

---

Input
Zhang et al.
ERRNet
IBCLN
RAGNet
YTMT
Dong al.
DSRNet
RRW
Ours (data setting I)
Ours (data setting II)
GT
Figure 10: Visual comparison of transmission layer predictions between previous and ours. The case is sampled from the SIR² dataset.
Input
Zhang et
ERRNet
IBCLN
HI--
RAGNet
YTMT
Dong al.
DSRNet
RRW
Ours (data setting I)
Ours (data setting II)
GT
Figure 11: Visual comparison of transmission layer predictions between previous arts and ours. The case is sampled from the SIR² dataset.
18

---

Input
Zhang et al.
BDN
IBCLN
RAGNet
YTMT
Dong et al.
DSRNet
Ours (data setting I)
Ours (data setting II)
I-T
Figure 12: Visual comparison of reflection layer predictions between previous arts and ours.
15
Input
Zhang al.
ERRNeT
IBCLN
RAGNet
YTMT
Dong et al.
DSRNet
RRW
Ours (data setting I)
Ours (data sotting II)
GT
Figure 13: Our limitation illustration, when solving hard case, sampled from the SIR? dataset.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, we summarize our main claims and the scope of our research. In the introduction, we clearly state our contributions and validate them with detailed experiments Sec.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations of the work in Sec. A.5
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: the Experiments section, we provide detailed description of the configura- tion for the main experimental results of the paper. And we provide detailed description the module structure in the body, algorithmic procedures in the appendix, and code implementation in the supplementary material, ensuring the reproducibility of the main experimental results of the paper.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code has been provided in the supplementary material and will be made
open-source.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the Experiments section, provide detailed descriptions of all the training and testing details required.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The methods in this field typically do not introduce error bars, and we follow the setting for fair comparisons.
8. Experiments Compute Resources
20

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: In Sec. we provide detailed information on the computer resouress required for the experiments.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp///eurrpp..cc/oooo//c///hs
Answer: [Yes]
Justification: Our research adheres the NeurIPS Code of Ethics in every aspect.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the positive societal impact of our work in both the Introduction and Conclusion sections, and the problem typically does not involve negative societal impacts.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The assets (e.g., code, data, models) used in the paper respect the licenses and terms use of their creators or owners.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not include new assets.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
21


Reviews:
[{"review_id": "yJ4KVzE1i7", "reviewer": "Reviewer_bMoK", "summary": "This paper introduces a Dual-Stream interactive transformer to tackle the single image reflection removal task. The motivation of the proposed method is based on the drawbacks of existing methods that the dual-stream methods cannot assess the effectiveness of the information flowing between the two streams. Based on this analysis, this paper proposed . The proposed method constructs the encoder and decoder in dual-stream structure and crossover the outputs of the two sub-networks. Comprehensive experiments show that the proposed method outperforms the baseline methods.", "questions": "Generally, the paper is not well written. The equations, symbols and definitions are not clearly described in the text and therefore the idea of the methodology becomes confusing. The authors are high advised to polish the presentation of the manuscript and re-submit it. In addition, the paper does not provide enough evidence to its claim. It lacks novelty in that sense.", "limitations": "The limitation is addressed in the appendix: the proposed method fails to deal with the regions with dominant reflections.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. This paper demonstrates the proposed method consistently outperforms the baseline methods in benchmarking datasets on image reflection separation task.\n2. The motivation of the proposed method is based on observations, which is one of the existing problems of the reflection separation task.", "weaknesses": "1. The methodology section is not described clearly. First, in line 146, the superscript symbol k on feature parameter F_T^{k-1} is neither explained in the text nor in the figures. Does this number k indicate the network is running in recurrent manner?  And in line 149, the superscript is used to indicates layer normed features. Also in Figure 1, the superscript of feature F is using l instead.  Second, I would assume F_T represents transmission features and F_R represents reflection features.  Third, in line 151, the DS-SA  and DS-CA should be explained or at least come with citations. \n2. Line 155 and Figure 2, how does DSLP FNN extract local features? How are local features defined? What is the difference between the input of DSLP and the input of the Transformer block? \n3. In the introduction section, this paper criticizes the existing dual-stream methods about the insufficient assessment on the information flowing between the two streams. However, I could not see the assessment of the data or any experiments analyzing the information. To make the claim of the paper valid, the authors should show evidences that the proposed method is able to make the information positively effective for reflection separation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we sincerely appreciate your comments, which can help us further improve the quality of our paper. We also apologize for any confusion or inconvenience caused by writing issues. Below, we address the main concerns.\n\n**Q1**: What does the superscript symbol $k$ mean?  It is not consistent with the superscript of $\\mathbf{F}$ in Figure 1.\n\n**A1**: $\\mathbf{F}$ denotes the feature flow in our dual-stream architecture. Specifically, $\\mathbf{F_T}$ and $\\mathbf{F_R}$ represent the feature flows for the transmission and reflection layers, respectively. When the superscript of $\\mathbf{F}$ is a lowercase letter, such as in $\\mathbf{F}^k_\\mathbf{T}$, it indicates the feature flow of the transmission layer after passing through $k$ DAIBs (Dual-Attention Interactive Blocks). The superscripts $l$ in Figure 1 and $k$ in the methodology section have the same meaning.  Moreover, when the superscript is formed by uppercase letters, such as in $\\mathbf{F}^{\\text{LN}}_\\mathbf{T}$, it designates the feature flow after passing through a specific layer within a DAIB (in this case, LayerNorm). We appreciate your attention to this detail. We will definitely unify the notations in the next version for clarity.\n\n**Q2**: The DS-SA and DS-CA should be explained or at least come with citations.\n\n**A2**: The DS-SA (Dual-Stream Self-Attention) and DS-CA (Dual-Stream Cross-Attention) mechanisms are introduced in Section 3.1 of our paper, which are explained in the subsequent paragraphs following their first mention. In the revision, we will ensure that a brief explanation is provided at their first mention to clarify their roles as follows: \n\n\"Following the LayerNorm, we apply Dual-Stream Self-Attention (DS-SA) to $\\mathbf{X}^{\\text{LN}}\\_{0}$ and Dual-Stream Cross-Attention (DS-CA) to $\\mathbf{X}^{\\text{LN}}\\_{1}$, obtaining $\\mathbf{X}\\_{\\text{SA}}$ and $\\mathbf{X}\\_{\\text{CA}}$, respectively. The design of these two attention mechanisms will be detailed later in this section.\"\n\n**Q3**: (1) How does DSLP FFN extract local features? (2) How are local features defined?  (3) What is the difference between the input of DSLP and the input of the Transformer block?\n\n**A3**: (1) Extraction of local features by DSLP FFN: The DSLP FFN (Dual-Stream Locality-Preserving Feed-Forward Network) and DSLP Block can be any convolutional dual-stream network modules, such as the YTMT Block [1]  and MuGI Block [2]. In our experiments, we employ the MuGI Block. Since reflection decomposition is a dense prediction task, one of the primary functions of the DSLP is to maintain local information. DSLP achieves this goal by using convolutional structures to extract local features, like other hybrid Transformer architectures [4],[5],[6].\n\n(2) Definition of local features: Local features refer to the local correlation among adjacent pixels that often have similar values. CNNs capture the local structures through limited receptive fields, weight sharing, and/or spatial sub-sampling [3],[4]. \n\n(3) Differences between inputs to DSLP and Transformer blocks: Both the features for the Transformer block and the DSLP block originate from the input image $\\mathbf{I}$. However, the features fed into the Transformer block first undergo a PatchEmbed process. Before being input into the DSLP FFN, these features are reshaped after the LayerNorm to fit the convolution operation, like other hybrid Transformer architectures [5],[6].\n\nTo avoid any confusion, we will provide a detailed explanation in the next version to address these mentioned points thoroughly.\n\n**Q4**: How does the proposed method assess the information flow between the two streams? \n\n**A4**: We explain in detail how explicit correlation assessment works in the Common Issue in the Global Rebuttal. Additionally, the ablation study on DAIB in Table 3 of the main body of our paper shows that removing the cross-attention mechanism (\"w/o DS-CA\") leads to significant performance degradation on real-world datasets. This also demonstrates that our proposed method is able to make the information positively effective for reflection separation.\n\n**Q5**: Does the proposed method fail to deal with the regions with dominant reflections?\n\n**A5**: In the situation mentioned by the reviewer (Figure 13), some regions in the image are dominant with strong reflections (in other words, information of the transmission layer has been largely suppressed), which poses challenges not only to our method but also to previous state-of-the-arts. While other methods struggle to discriminate the reflection component from the entangled layers in such scenarios, our method removes significant portions of the reflections, way more effectively, which corroborates our generalization capability. \n\n[1] Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation.  NeurIPS 2021.\n\n[2] Single Image Reflection Separation via Component Synergy. ICCV 2023.\n\n[3] Object recognition with gradient-based learning. CGCV 1999.\n\n[4] CvT: Introducing Convolutions to Vision Transformers. ICCV 2021.\n\n[5] Incorporating Convolution Designs into Visual Transformers.  ICCV 2021.\n\n[6] Uformer: A General U-Shaped Transformer for Image Restoration. CVPR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qqU6uReWJ1", "reviewer": "Reviewer_k1uS", "summary": "The authors propose the Dual-Stream Interactive Transformer (DSIT) for single image reflection separation. DSIT is a dual-stream method designed for complex scenarios. Specifically, the authors design the Dual-Attention Interaction (DAI) to achieve feature aggregation of different dimensions through dual-stream self-attention and layer-aware cross-attention. They also propose the Dual-Architecture Interactive Encoder (DAIE) to leverage pre-trained Transformer models. Experiments demonstrate that the proposed method outperforms comparative methods.", "questions": "1. Clarify the implementation of CAI and other model settings, such as N_T, N_W, and the structure of DSLP.\n2. Provide a comparison of FLOPs in Tables 2 and 3.\n3. The proposed DSIT uses a Transformer as a pre-trained model and claims to introduce prior knowledge. How does the model perform without using a pre-trained model? Additionally, why does DSLP not use pre-trained models?", "limitations": "The authors discuss the method's limitations and societal impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors use DAI to enhance feature aggregation of transmission and reflection information, which is a reasonable approach. The ablation study also proves its effectiveness.\n2. The authors introduce a pre-trained Transformer to provide global information and improve feature modeling. As shown in Figure 3, the final F_T^{DAIB} and F_R^{DAIB} have clearer detailed features.\n3. Testing on multiple datasets shows that the proposed method outperforms state-of-the-art methods.\n4. The authors conduct thorough experiments on various modules in the ablation study, further proving the effectiveness of the proposed method.", "weaknesses": "1. The implementation of CAI is not clearly explained. In L194, CAI seems to be through DAIB(F^{GP}, F_T^{LP}) and DAIB(F^{GP}, F_R^{LP}), while the proposed DAIB in the method deals with F_T and F_R. It is recommended that this be clarified in Figure 2 and the text to improve clarity. Additionally, the implementation of CAI is overly complex. Considering the high computational complexity of DAIB, a simpler implementation should be considered, such as directly using F GP in DAIB(F_T^{LP}, F_R^{LP}).\n2. The authors do not provide detailed model settings, such as N_T, N_W, and the detailed structure of DSLP.\n3. There is a lack of comparison between FLOPs and Params. The proposed method uses multiple attentions to aggregate features, which introduces high computational complexity.\n4. In Figure 2, some parts of the Dual-Stream Information Flow overlap excessively, reducing distinguishability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our work. Below, we address your main concerns:\n\nQ1: (1) Clarify the implementation of CAI, (2) N_T, N_W, and (3) DSLP.\n\nA1: (1) CAI is a general design introduced to enable interaction and fusion of features extracted from different network architectures during the feature extraction phase. A specific implementation of this design is the dual-stream feature interaction module. We tested 3 different implementations of CAI as in the ablation study shown in Table 3 of the main paper. It turned out that our current choice (DAIB) is significantly superior to other alternatives, which exhibits the effectiveness of DAIB for both cross-architecture and cross-layer interactions. \n\n(2) In our experiments, the training image size is fixed at $384\\times384$. The window size of attention mechanisms, $N_W$, is fixed to $12\\times12$. The number of windows, $N_T$, varies depending on the spatial scale of the features. For instance, if the spatial scale is $384\\times384$, then $N_T = (384/12)\\times(384/12)=1024$.\n\n(3) The DSLP can be any convolutional dual-stream network module, such as the YTMT Block [1] and MuGI Block [2]. To highlight our main contribution and avoid other possible factors influencing performance, we simply adopt the MuGI Block in our experiments. Also, more powerful convolutional modules can be developed in future work.\n\nWe will clarify these model settings in our revised version and thoroughly proofread our paper to eliminate other writing problems. \n\nQ2: Simpler Implementation of DAIB.\n\nA2: In Table 3 of the main paper, we provided an ablation study on CAI. The \"Add\" variant represents the removal of the DAIB, directly adding the global priors on the dual-stream local priors. Although this setting reduces the complexity of the model (from 517 GFLOPs to 471 GFLOPs), it decreases the average PSNR from 26.27 to 25.74. To ensure superior performance, we retained the DAIB as the interaction module for CAI. However, considering broader applications, exploring more lightweight model designs can be more attractive, which may use a more lightweight backbone (MobileViT for example), reduce the number of channels and blocks, and employ knowledge distillation techniques. Thank you for your valuable suggestion, which inspires us to optimize the model for more practical use in the future.\n\nQ3: Provide a comparison of FLOPs and Params. \n\nA3: As shown in the Table below, we present a comparison in terms of GFLOPs (obtained via fvcore package at $384\\times384$ input resolution) and learnable parameters between previous state-of-the-arts and ours. It can be seen that the parameter amount of our model is comparable to that of RAGNet and DSRNet, but our model is of the lowest computational complexity among all the compared methods. This efficiency is achieved by our multi-scale and non-recurrent model design. Moreover, our method demonstrates the best average performance across all datasets, as evidenced by the average numerical metrics on all testing images from Real20 and SIR^2. \n\n|Models|ERRNet|IBCLN|RAGNet|YTMT|Dong _et al._|DSRNet|Ours|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Parameters (M)|18.58|24.37|146.72|76.90|**10.93**|124.60|131.76|\n|GFLOPs|820|708|610|838|666|743|**517**|\n|Performance|23.53/0.879|24.10/0.879|24.90/0.886|24.05/0.886|24.21/0.897|25.75/0.910|**26.49/0.922**|\n|\n\nQ4: Some parts in Figure 2 overlap excessively.\n\nA4: We will definitely polish the figures in our revised version, for better visual appeal.\n\nQ5: How does the model perform without using a pre-trained model?\n\nA5: As shown in the table below, DSIT (w/o Pretrain) refers to our model with the Transformer backbone randomly initialized, and trained from scratch alongside the main network. The results reflect a remarkable performance drop, which comes from the data-hungry nature of Transformer architectures; training from scratch on a relatively small dataset can negatively impact the performance of models [3]. This outcome also underscores the importance of high-semantic pretraining in addressing the ill-posed nature of the reflection separation task, which has been demonstrated in previous methods like the HyperColumn in YTMT [1] and DSFNet in DSRNet [2]. In future work, improvements of the pre-training technique, such as incorporating same-task pre-training strategies like HAT [4], could further enhance the model performance.\n\nQ6: Why does DSLP not use pre-trained models?\n\nA6: We have already employed the pretrained Global Prior Extractor to extract high-semantic prior, the information flow of which is task-agnostic and requires task-specific guidance to adapt to our current task. The proposed dual-stream Local Prior Extractor (LPE) serves this purpose, guiding the task-agnostic information to be more relevant to reflection separation. Therefore, there is no need to further introduce a pre-trained task-agnostic LPE.\n\nTo further validate this, we experimented as shown in the table below. In the DSIT with CNN Pretrain, we replaced the original LPE with a pre-trained RepVGG-B3 model. We used CAI to interact and fuse features extracted by the pre-trained models from both architectures. However, we observed that this approach not only increased the model's overall GFLOPs but also resulted in weaker performance, which confirms our opinion above that using a pre-trained CNN network in place of our LPE does not improve performance. Thank you for your insightful question.\n\n|Models|Average PSNR/SSIM|GFLOPs|\n|:-|:-:|:-:|\n|DSIT(w/o Pretrain)|25.07/0.897|517|\n|DSIT(CNN Pretrain)|25.83/0.911|525|\n|DSIT(Ours)|**26.27**/**0.917**|**517**|\n|\n\n[1] Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation.  NeurIPS 2021.\n\n[2] Single Image Reflection Separation via Component Synergy. ICCV 2023.\n\n[3] Training data-efficient image transformers & distillation through attention. ICML 2021.\n\n[4] Activating More Pixels in Image Super-Resolution Transformer. CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J3dEJfAh3m", "reviewer": "Reviewer_9gCf", "summary": "The paper proposes a transformer-based network for the single image reflection separation task. It focuses on the network architecture design, proposing several tailored designs, including dual-attention interactive block (DAIB), dual-stream self-attention (DS-SA), and dual-stream cross-attention (DS-CA).", "questions": "Please refer to the weaknesses for details.", "limitations": "Given the advancements in deep learning for computer vision, the proposed reflection separation task focuses on synthetic pairs. I suggest the authors shift their research focus towards reflections occurring in real-world scenes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe introduction is well-written.\n2.\tThe proposed method demonstrates state-of-the-art (SOTA) performance on the traditional single image reflection separation datasets.", "weaknesses": "1.\tThe method section is poorly written, making it difficult to follow due to the extensive use of abbreviations, some of which are not explained upon first mention, such as DS-SA and DS-CA.\n2.\tDespite a thorough reading of the methods and implementation details, it is unclear where the authors have applied the pretrained transformer. Specifically, I mean \"pretrained transformer.\" Additionally, the placement and role of the proposed CNN-based network within the DSIT framework are not clearly clarified.\n3.\tFor Eq. (3), the authors should clarify where the 'cross' is present. The equation seems to suggest a self-attention mechanism, as the query, key, and value all derive from the same features.\n4.\tIn Fig. 3, it is unclear if the increased intensity of red indicates regions of higher attention. If so, I cannot distinguish the differences between the local and global priors as revealed by the figure.\n5.\tI think the current setting of single image reflection separation does not align well with typical real-world scenarios. Reflections in real scenes typically occur in localized regions, such as the reflection of a building with a glass facade. Therefore, relying heavily on synthetic reflections with minimal real-scene data might not be practical. The authors should consider focusing on real-scene reflections.\nOverall, I think the novelty of the paper is limited, and the writing is poor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback. We appreciate your comments and try our best to address the issues as follows:\n\n**Q1**: The writing of the method section.\n\n**A1**: We apologize for any difficulties caused by the writing in the method section. In the next version, we will thoroughly proofread and clarify our paper. Specifically, we will ensure that the terms DS-SA and DS-CA are properly defined before their first use, as follows: \n\n\"Following the LayerNorm, we apply Dual-Stream Self-Attention (DS-SA) to $\\textbf{X}^{\\text{LN}}\\_0$ and Dual-Stream Cross-Attention (DS-CA) to $\\textbf{X}^{\\text{LN}}\\_1$, obtaining $\\textbf{X}\\_{\\text{SA}}$ and $\\textbf{X}\\_{\\text{CA}}$, respectively. The design of these two attention mechanisms will be detailed later in this section.\"\n\n**Q2**: (1) The appliance of the pre-trained transformer. (2) The CNN-based network placement.\n\n**A2**: (1) We utilize a pretrained Transformer architecture for the Global Prior Extractor, as shown in Figure 2 (a) in the main paper. The transformer blocks in GPE can load pretrained weights before training, which can leverage rich priors learned from large-scale datasets to aid in reflection separation. This practice is consistent with previous reflection separation methods, which have similarly used pretrained models to enhance feature extraction, such as the HyperColumn used in Zhang et al. [1] and YTMT [2], and the DSFNet in DSRNet [3].\n\n(2) As depicted in Figure 2 (a), the Local Prior Extractor, referred to as the CNN-based Network, is implemented using a convolutional dual-stream interactive network structure. In our implementation, each DSLP Block specifically utilizes the MuGI Block from the DSRNet [3] to highlight our main contribution and avoid other possible factors influencing performance.  Additionally, more advanced convolutional modules can be developed in future work.\n\nWe will highlight these points in our revision to avoid misunderstanding.\n\n**Q3**: Where is the \"cross\" presented? \n\n**A3**: We explain in detail how explicit correlation assessment works and where is the \"cross\" presented  in the A1 of common issues in the global rebuttal. We will clarify this point in our revised version.\n\n**Q4**: Elaboration on Figure 3.\n\n**A4**: In the figure, red regions in the feature visualization indicate high attention values, while blue regions represent low attention values. We will add a color bar in the revised version to clarify this. \n\nPlease note that the pretrained model used in transfer learning is task-agnostic. We do not expect the priors alone to distinguish between the reflection and transmission layers clearly. Instead, the priors provide varying degrees of attention to different components of the image. For instance, the global prior $\\mathbf{F}^{\\text{GP}}$, extracted using a pretrained Swin Transformer model, shows higher attention towards reflection components. The shifted window-based self-attention contributes to the coherent attention across the image. Additionally, the Local Prior Extractor captures local prior information for both the transmission stream $\\mathbf{F}^{\\text{LP}}\\_\\mathbf{T}$ and the reflection stream $\\mathbf{F}^{\\text{LP}}\\_\\mathbf{R}$. These local priors exhibit a \"granular\" texture and may not accurately attend to the correct components of transmission and reflection. After the Cross-Architecture Interaction with the global prior $\\mathbf{F}^{\\text{GP}}$, the attention of $\\mathbf{F}^{\\text{LP}}\\_\\mathbf{T}$ and $\\mathbf{F}^{\\text{LP}}\\_\\mathbf{R}$ towards the transmission and reflection layers becomes more appropriately focused. Through further interaction and optimization of Dual-Attention Interactive Blocks, the dual-stream features $\\mathbf{F}^{\\text{DAIB}}\\_\\mathbf{T}$ and $\\mathbf{F}^{\\text{DAIB}}\\_\\mathbf{R}$ turn out to be more fine-grained and accurate. This improvement has also been recognized by Reviewer k1uS (Strengths 3).\n\n**Q5**: Does the current setting align with real-world scenarios? \n\n**A5**: Please note that all the testing data used in our paper are actually captured in the real world, including Real20, Nature20, Wild, Objects, and Postcard.  These datasets encompass a variety of indoor and outdoor scenes, different glass thicknesses, and varying distances, thereby covering a wide range of real-world reflection scenarios. Again, we emphasize that our method does *NOT* focus on synthetic scenarios, as the reviewer criticized.\n\nMeanwhile, during the rebuttal period, we captured several reflection scenes in the wild according to what the reviewer described as typical real-world scenarios. We compared the inference results of state-of-the-art methods and our approach in the global rebuttal PDF. In Figure 1 of the rebuttal PDF, the first 3 examples primarily contain localized reflection regions. The 4-th example showcases the reflection phenomenon on a water surface, which is not covered in the \"minimal real-scene data\" during training mentioned by the reviewer. Our results are visually impressive, demonstrating our model's generalization capability. Besides, Figure 2 of the rebuttal PDF presents the predicted reflection layers from our method on these real-world images, along with the corresponding binarized maps. From these visualizations, it is evident that our method can effectively identify localized reflection regions.\n\nInspired by the suggestion of the reviewer, in future work, we may explore the introduction of a lightweight \"reflection localizer\". This would allow us to bypass the processing of non-reflective regions (like the manner of early exiting [4]), thereby accelerating the reflection separation process.\n\n[1] Single Image Reflection Separation with Perceptual Losses. CVPR 2018.\n\n[2] Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation.  NeurIPS 2021.\n\n[3] Single Image Reflection Separation via Component Synergy. ICCV 2023.\n\n[4] Adaptive Patch Exiting for Scalable Single Image Super-Resolution. ECCV 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "miMl228ucg", "reviewer": "Reviewer_gpBZ", "summary": "This work introduces ADI, a new interactive dual-stream transformer framework for single image reflection separation. It incorporates a dual-attention interaction to explicitly model the dual-stream correlation for improved reflection separation, achieving impressive performance compared to other SOTA methods.", "questions": "see the weaknesses.", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper addresses the limited interaction of previous dual-stream transformer frameworks for the SIRS task by presenting the ADI module, which simultaneously considers inter-window and intra-window fusion.\n2. The quantitative results are impressive compared to previous SOTA works.\n3. The discussion and motivation are well-explained and intuitively correct.", "weaknesses": "1. The design is quite plain, involving inter-patch/intra-patch attention.\n2. The choice of using gradient and feature loss should be analyzed in ablation studies to verify their contributions.\n3. Model complexity and inference time are missing in the main manuscript, making it difficult to determine whether the main contribution comes from the larger model capacity or the proposed algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the motivation, technical soundness, and state-of-the-art performance of our method. Below, we address the key concerns you raised:\n\n**Q1**: Why the design of inter-patch/intra-patch attention is quite plain?\n\n**A1**: Keeping the design simple is beneficial to validate its effectiveness. When the pipeline becomes complicated, it can be hard to determine which part of the design really works. Also, more powerful and complicated attention designs can be developed in future work, which can potentially enhance the performance further. \n\nThrough the derivation of LaCA (Layer-aware Cross-Attention) provided in the global Author Rebuttal, it is evident that even with a straightforward token concatenation operation and corresponding modifications to the relative position bias, we naturally introduce inter- and intra-layer attention for dual-stream information. This demonstrates the inherent relationship between dot-product self-attention mechanisms and dual-stream interactive modeling. \n\nIn the meantime, our strategy can also energize other multi-stream transformer designs. For instance, in high-resolution image generation scenarios where a single GPU cannot support the entire image generation process, multiple information streams of the model can be distributed across different GPUs. Each stream would handle a specific image patch, and inter-stream feature interactions would ensure the coherence of the generated patches, enabling them to be seamlessly stitched together into a complete image (like in [1]).\n\n**Q2**: What about the contributions of gradient and feature losses? \n\n**A2**: Both gradient and feature losses contribute positively to our method. The importance of gradient relationship has been emphasized by early works in SIRS [2],[3],[4]. Feature (or perceptual) loss, on the other hand, significantly aids in modeling natural images by ensuring that the reconstructed images are perceptually similar to the ground truth [5],[6].\nAs suggested by the reviewer, to confirm their contributions, we additionally conducted ablation experiments, as shown in the table below. We evaluated the impact of removing Gradient Loss (DSIT (w/o Gradient Loss)) and Feature Loss (DSIT (w/o Feature Loss)) on the performance of our model. The results indicate a noticeable decrease in performance when these losses are disabled, underscoring their necessity. Specifically, the gradient loss helps preserve edge details, while the feature loss enhances the perceptual quality of the outputs, both of which are beneficial for effective reflection separation.\n\n|          Models         |      Real20     |     Objects     |     Postcard    |       Wild      | Average PSNR/SSIM |\n|:-----------------------:|:---------------:|:---------------:|:---------------:|:---------------:|:-----------------:|\n| DSIT (w/o Gradient Loss) |   24.13/0.814   |   27.54/0.922   |   23.38/0.898   |   26.72/0.906   |    25.55/0.906    |\n|  DSIT (w/o Feature Loss) |   23.13/0.780   |   26.00/0.919   |   23.80/0.896   |   25.88/0.900   |    24.94/0.901    |\n|        DSIT(Ours)       | **25.06/0.836** | **26.81/0.919** | **25.63/0.924** | **27.06/0.910** |  **26.27/0.917**  |\n|\n\n**Q3**: Please provide the model complexity and inference time. \n\n**A3**: We have provided a comparison in model parameters, GFLOPs (for $384 \\times 384$ resolution inputs), inference time (for $384 \\times 384$ resolution inputs, tested on an RTX 3090 GPU), and the average PSNR/SSIM across 4 datasets (Real20, Objects, Postcards, and Wild) in the table below. The number of parameters of our model is comparable to RAGNet and DSRNet. Due to our multi-scale and non-recursive network structure, we achieve superior GFLOPs efficiency compared to all other methods. Despite being based on a Transformer architecture, our method demonstrates a slightly faster inference time than DSRNet, highlighting the efficiency of our design. The above results show that the improvement of our model is non-trivial. As further illustrated in the additional cases provided in the global rebuttal PDF, our method exhibits impressive generalization performance in real-world scenarios compared to previous state-of-the-art approaches. This reflects the strong capability in effectively modeling reflection scenes of our method.\n\n|        Models       |    ERRNet   |    IBCLN    |    RAGNet   |     YTMT    | Dong _et al._ |    DSRNet   |       Ours      |\n|:-------------------:|:-----------:|:-----------:|:-----------:|:-----------:|:-------------:|:-----------:|:---------------:|\n|    Parameters (M)    |    18.58    |    24.37    |    146.72   |    76.90    |   **10.93**   |    124.60   |      131.76     |\n|        GFLOPs       |     820     |     708     |     610     |     838     |      666      |     743     |     **517**     |\n| Inference Time (ms) |      72     |      75     |    **56**   |     101     |      123      |     256     |       215       |\n| Average PSNR/SSIM| 23.53/0.879 | 24.10/0.879 | 24.90/0.886 | 24.05/0.886 |  24.21/0.897  | 25.75/0.910 | **26.49/0.922** |\n|\n\n[1] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models. CVPR 2024.\n\n[2] User Assisted Separation of Reflections from a Single Image Using a Sparsity Prior. ECCV 2004. \n\n[3] Single Image Layer Separation Using Relative Smoothness. CVPR 2014.\n\n[4] A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing. ICCV 2017.\n\n[5] Image style transfer using convolutional neural networks. CVPR 2016.\n\n[6] Perceptual losses for real-time style transfer and super-resolution. ECCV 2016.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Common Issue**:

**Q**: Illustration of  Explicit Correlation Assessment and the Cross-Attention mechanisms in the proposed method.

**A**: We utilize the Layer-aware Cross-Attention (LaCA) mechanism for explicit correlation assessment between the two streams. A formal derivation of LaCA has already been provided in Appendix A.2 of our initial submission. We here again explain its principle as follows:

Given the feature streams of transmission layer $\textbf{T} \in \mathbb{R}^{N\times C}$ and reflection layer $\textbf{R} \in \mathbb{R}^{N\times C}$ , we concatenate them along the first dimension to form a matrix $\textbf{X} \in \mathbb{R}^{2N\times C} = \begin{bmatrix} \textbf{T} \\\ \textbf{R} \end{bmatrix}$, where $N$ represents the number of tokens and $C$ denotes the number of channels in each token, respectively.

The query $\mathbf{Q} \in \mathbb{R}^{N\times D}$, key $\textbf{K} \in \mathbb{R}^{N\times D}$, and value $\textbf{V} \in \mathbb{R}^{N\times D}$ matrices can be computed by applying the linear projections to the merged stream $\mathbf{X}$ via:

$$\textbf{Q} = \textbf{X}\textbf{W}_q , \quad \textbf{K} = \textbf{X}\textbf{W}_k, \quad \textbf{V} = \textbf{X}\textbf{W}_v ,$$
where $\textbf{W}_q, \textbf{W}_k, \textbf{W}_v \in \mathbb{R}^{C \times D} $ are the weight matrices of linear projections, changing the number of channels of each token from $C$ to a hidden dimension $D$.

The attention score matrix $\mathbf{A}\in\mathbb{R}^{2N\times 2N}$ is computed by:
$$
\textbf{A} = \text{Softmax}(\frac{\textbf{Q}\textbf{K}^T}{\sqrt{D}}) = \text{Softmax}(\frac{1}{\sqrt{D}}\begin{bmatrix}
 \textbf{T} \\\\
 \textbf{R}
\end{bmatrix}\textbf{W}_q\textbf{W}_k^T\begin{bmatrix}
 \textbf{T}^T & \textbf{R}^T
\end{bmatrix}) = \text{Softmax}( \frac{1}{\sqrt{D}}\begin{bmatrix}
 \textbf{T}\textbf{W}_q\textbf{W}_k^T\textbf{T}^T & \textbf{T}\textbf{W}_q\textbf{W}_k^T\textbf{R}^T \\\\
 \textbf{R}\textbf{W}_q\textbf{W}_k^T\textbf{T}^T & \textbf{R}\textbf{W}_q\textbf{W}_k^T\textbf{R}^T
\end{bmatrix} ),$$
where the intra-layer terms $\textbf{T}\textbf{W}_q\textbf{W}_k^T\textbf{T}^T$and $\textbf{R}\textbf{W}_q\textbf{W}_k^T\textbf{R}^T$ represent interactions within the transmission stream $\textbf{T}$ and the reflection stream $\textbf{R}$, respectively.  The inter-layer terms $\textbf{T}\textbf{W}_q\textbf{W}_k^T\textbf{R}^T$and $\textbf{R}\textbf{W}_q\textbf{W}_k^T\textbf{T}^T$ indicate interactions between the transmission stream $\textbf{T}$ and the reflection stream $\textbf{R}$.

By denoting the Softmax function with a scaling factor $\frac{1}{\sqrt{D}}$ as $\mathcal{S}(\cdot)$, the output matrix $\textbf{Y}$ is then calculated as:

$$\textbf{Y} = \textbf{A}\textbf{V} = \begin{bmatrix}
 \mathcal{S}(\textbf{T}\textbf{W}_q\textbf{W}_k^T\textbf{T}^T) \textbf{T} \textbf{W}_v + \mathcal{S}(\textbf{T}\textbf{W}_q\textbf{W}_k^T\textbf{R}^T) \textbf{R} \textbf{W}_v \\\\
 \mathcal{S}(\textbf{R}\textbf{W}_q\textbf{W}_k^T\textbf{T}^T) \textbf{T} \textbf{W}_v + \mathcal{S}(\textbf{R}\textbf{W}_q\textbf{W}_k^T\textbf{R}^T) \textbf{R} \textbf{W}_v 
\end{bmatrix}.$$

We further simplify the form of $\textbf{Y}$ by introducing a function $\mathcal{G}(\textbf{A},\textbf{B})=\mathcal{S}(\textbf{A}\textbf{W}_q\textbf{W}_k^T\textbf{B}^T)\textbf{B}\textbf{W}_v$, where $\mathbf{A}\in\mathbb{R}^{N\times C}$ and $\mathbf{B}\in\mathbb{R}^{N\times C}$ can be chosen between $\mathbf{T}$ and $\mathbf{R}$, yielding the follows:

$\textbf{Y} = \begin{bmatrix}
\mathcal{G}(\textbf{T},\textbf{T})+\mathcal{G}(\textbf{T},\textbf{R}) \\\\
\mathcal{G}(\textbf{R},\textbf{T})+\mathcal{G}(\textbf{R},\textbf{R})
\end{bmatrix}.$

We finally obtain the output of the dual streams as $\textbf{T}_o=\mathcal{G}(\textbf{T},\textbf{T})+\mathcal{G}(\textbf{T},\textbf{R})$ and $\textbf{R}_o=\mathcal{G}(\textbf{R},\textbf{R})+\mathcal{G}(\textbf{R},\textbf{T})$. 
Specifically, the output features of the transmission stream, $\mathbf{T}_o$, consist of two parts: intra-layer explicit correlation assessment $\mathcal{G}(\mathbf{T},\mathbf{T})$ and inter-layer explicit correlation assessment $\mathcal{G}(\mathbf{T},\mathbf{R})$. Similarly, the output features of the reflection stream, $\mathbf{R}_o$, include intra-layer explicit correlation assessment $\mathcal{G}(\mathbf{R},\mathbf{R})$ and inter-layer explicit correlation assessment $\mathcal{G}(\mathbf{R},\mathbf{T})$.

Additionally, to facilitate a more intuitive understanding of LaCA, we have included an illustrative example in Figure 3 of the **Global Rebuttal PDF** (attachment below), which sequentially displays the transmission stream $\mathbf{T}$, the reflection stream $\mathbf{R}$, the concatenated matrix along the token channel $\textbf{X}$, and 4 internal blocks of the attention matrix $\mathbf{A}^* = \begin{bmatrix}
\textbf{T}\textbf{W}_q\textbf{W}_k^T\textbf{T}^T & \textbf{T}\textbf{W}_q\textbf{W}_k^T\textbf{R}^T \\\\
 \textbf{R}\textbf{W}_q\textbf{W}_k^T\textbf{T}^T & \textbf{R}\textbf{W}_q\textbf{W}_k^T\textbf{R}^T
\end{bmatrix}$.

*The diagram clearly shows that the two submatrices along the main diagonal represent the self-attention of the layers, while the other two along the off-diagonal represent the cross-attention of the two layers*.

Meta-review: The reviewers acknowledge the technical contribution of DAI mechanism and the strong performance advantage of this paper. The rebuttal was successful, resulting in an increased average score, with previous negative scores now becoming supportive. After careful discussion and consideration, we are pleased to inform you that your paper has been accepted. There is one issue from the AC-reviewer discussion regarding the distinction in the technical aspects compared to YTMT [22]. The ACs have investigated this matter and found the technical components to be distinguishable. Moreover, the performance of this submission is significantly superior to YTMT, both quantitatively on benchmark and visually in real cases. The authors are encouraged to provide a clearer distinction in the final version. Additionally, a common suggestion from the reviewers is that the writing can be improved. While the ACs believe this will not influence the acceptance of the paper, given its merits and strong performance, the authors are advised to revise the writing substantially to enhance clarity.
Decision: Accept (poster)