Xformer: Hybrid X-Shaped Transformer for Image Denoising
Authors: Jiale Zhang
Yulun Zhang
Jinjin Gu
Jiahua Dong
Linghe Kong
Xiaokang Yang
Venue: ICLR 2024 poster (2024)
Keywords: Image denoising
hybrid Transformer blocks
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: In this paper, we present a hybrid X-shaped vision Transformer, named Xformer, which performs notably on image denoising tasks. We explore strengthening the global representation of tokens from different scopes. In detail, we adopt two types of Transformer blocks. The spatial-wise Transformer block performs fine-grained local patches interactions across tokens defined by spatial dimension. The channel-wise Transformer block performs direct global context interactions across tokens defined by channel dimension. Based on the concurrent network structure, we design two branches to conduct these two interaction fashions. Within each branch, we employ an encoder-decoder architecture to capture multi-scale features. Besides, we propose the Bidirectional Connection Unit (BCU) to couple the learned representations from these two branches while providing enhanced information fusion. The joint designs make our Xformer powerful to conduct global information modeling in both spatial and channel dimensions. Extensive experiments show that Xformer, under the comparable model complexity, achieves state-of-the-art performance on the synthetic and real-world image denoising tasks. We also provide code and models at https://github.com/gladzhang/Xformer.

Published as a conference paper at ICLR 2024
XFORMER: HYBRID X-SHAPED TRANSFORMER FOR IMAGE DENOISING
Jiale Zhang', Yulun Zhang': + Jinjin Gu Jiahua Dong', Linghe Kong',, Xiaokang Yang' Shanghai Jiao Tong University. *Shanghai AI Laboratory, Universiity of Sydney, "Shenyang Institute of Automation, Chinese Academy of Sciences
ABSTRACT
In this paper, we present hybrid X-shaped vision Transformer, named Xformer, which performs notably on image denoising tasks. We explore strengthening the global representation of tokens from different scopes. In detail, we adopt two types of Transformer blocks. The spatial-wise Transformer block performs fine-grained local patches interactions across tokens defined by spatial dimen- sion. The channel-wise Transformer block performs direct global context in- teractions across tokens defined by channel dimension. Based on the concur- rent network structure, we design two branches to conduct these two interac- tion fashions. Within each branch, we employ an encoder-decoder architecture to capture multi-scale features. Besides, we propose the Bidirectional Connection Unit (BCU) to couple the learned representations from these two branches while providing enhanced information fusion. The joint designs make our Xformer powerful to conduct global information modeling in both spatial and channel dimensions. Extensive experiments show that Xformer, under the comparable model complexity, achieves statc-of-the-ar art performance on the synthetic and real- world image denoising tasks. We also provide code and models at https: wtw.:ttubb..co//glccannngggg
INTRODUCTION
As fundamental vision task, image denoising aims to recover the high-quality image from its noisy counterpart. It has been a very challenging problem as the denoising process is hard to distinguish the tiny textures and details from the noise. Recently, deep convolutional neural networks (CNNs) have shown great power to solve this inverse problem (Zhang et al., 2017a 2020 Tian et al. 2020 Zhang et al. 202 la extracted to provide powerful image representations. However, the disadvantages of convolution are also obvious. Due to the poor receptive field scaling, CNN has limited ability to capture long-range dependencies among visual elements. Moreover, the convolution filters are parameter-dependent and content-indeependent and thus experience difficulty to show flexibility for the dynamic inputs. To address the above shortcomings, several recent works investigate the self-attention (SA) mechanism
Hu et al. 2021). With the help of convolution operations, deep features can be
replace the convolution and build the Transformer-base -aasd networks (Liang al. 2021 Chen al.,
2021 Zamir et al. 2022 Wang et al. 2022 Lee et al. 2022).
Transformer has shown state-of-the-art performance on high-level vision tasks (Xu al.
1021
Ali et al. 2021 Zhang & Yang 2021 Liu et al.] 2021 Chu et al. 2021: [aag et al. 1021
Xie et al. 2021 The SA mechanism has great power to capture content-dependent global rep- resentations while modeling long-distance relationships. Despite of the growing computational cost, researchers are investigating the employment of Transformer in solving low-level vision problems (Liang et al., 2021] Chen et al.] 2021). [iang et al.] (2021) proposed SwinIR based on Swin Transformer (Liu al.) (021) to utilize spatial-wise window-based SA blocks. The tokens extracted from square location. Zamir al. (2022) proposed Restormer to ap- ply SA across channel dimension rather than the spatial dimension. It is demonstrated that the channel-wise SA is able to model global connectivity. For further analysis, these two types of SA mechanisms are considered to focus on different respects of global information modeling. In detail, the spatial-wise SA is good at capturing local patch-level information and modeling fine-detailed spatial features. On the other hand, the channel-wise SA is capable of capturing
"Conresponding authors: Yulun Zhang, yulun 100@ gmail.com: Linghe Kong, linghe.kong @ sjtu.edu.cn. The work was mainly done when Yulun Zhang was at ETH Zürich.

---

Published as a conference paper at ICLR 2024
S:p Consectone
Skp Connector
A H
Dutpri
Candio
Upsample
Skip Conred
suaar
Aditions
Figure Architecture of our proposed Xformer. The modules include spatial-wise Transformer block (STB), channel-wise Transformer block (CTB), and bidirectional connection unit (BCU).
global channel-level information and modeling specific channel features. Especially, both types of information modeling are important for enhancing representation learning in Transformer.
Therefore, we explore adopting spatial-wise SA and channel-wise SA together in this paper. It remains challenging as there are gaps be- tween these two types of SA mechanisms. We consider designing a concurrent network with dual branches. Similar parallel ideas have also been investigated for other visual tasks in recent years (Chen et al. 2022 Pan et al., 2022. Peng
Neny Camb3
HUN
Lihanto
Reskerac
Xicener (HiEa))
Figure 2: Visual examples for Gaussian color im- age denoising with noise level o=50 on 100.
al. 2021). In special, the concurrent structure is beneficial for the network to build direct in- teractions between different branches. Furthermore, we apply spatial-wise Transformer block and channel-wise Transformer block in respective branch. Following previous works (Yue et al. 2020) (022), we employ an encoder-decoder structure within each branch obtain multi-scale features. In short, the spatial-wise branch can perform fine- ane-ggriied local patches interactions across spatial- ppaiial-dimnnsoon tokens. The channel-wise branch can perform direct global context interactions across channel-dimension tokens.
Cheng et al. |021 Zamir et al., (022).
For further investigation, the concurrent network enables dual branches to model patch-level and channel-level information respectively. However, there are still some limitations. The direct con- catenating operation in the end fails to effectively use these two types of features. In this way, each branch cannot capture information from different levels. Therefore, we propose the Bidirectional Connection Unit (BCU) as the bridge between two branches, which provides information fusion in an interactive manner. With BCU, the network can couple two styles of deep features. In detail, utilize convolution layers with 3x3 kernel to refine the learned deep features in corresponding branches. Then, we add the refined features to respective branches. Such a fusion operation can greatly enhance the global representation of tokens from different dimensions.
Based on the designs above, we present a hybrid X-shaped Transformer for image denoising, named Xformer, shown in Fig. [ We design concurrent network with two branches. Specifically, we separately utilize spatial-wiis SA blocks and channel-wise SA blocks in respective branch. Besides, employ the proposed BCU to bridge these two branches for information fusion. The joint designs enable our network to obtain stronger global representations in Transformer. More details can be found in Sec. 3 Our Xformer can achieve superior results against recent leading image denoising methods. As shown in Fig. 2 our proposed method obtains visually pleasing results while others suffer from the loss of details. Overall, we summarize our main contributions as follows:
We propose Xformer, an X-shaped Transformer with hybrid implementations spatial- wise and channel- Transformer blocks, thereby exploiting the stronger global repre- sentation of tokens in Transformer-base neural network.
We propose the Bidirectional Connection Unit (BCU) that is able effectively couple the learned representations from two branches of Xformer. This simple design significantly enhances the global information modeling of our method.
We employ Xformer to train efficient and effective Transformer-base based network for im- age denoising. We conduct extensive experiments on the synthetic and real-world noise removal tasks. Our method can achieve state-of-the-art performance.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Image Denoising. Due to the powerful generalizing ability from large- scale data, CNN-based meth- ods (Zhang et al. 2020 Tian et al. 2020 Zhang ct al., 202 Hu et al. 2021) have achieved superior performance over the traditional denoising algorithms (Perona & Malik 1990, Mairal et al., 2009) Elad & Aharon 2006). Zhang al. (2017a) proposed DrCNN as representative image denoising method, which trained mappings from noisy images to noises. For further improvements, many sub- sequent works used more elaborate network architecture designs, including encoder-decoder struc-
ture (Yue et al. 2020] Cheng et al. 2021] Zamir et al. (202T) non-local modules (Liu et al. 200 Zhang et al., 2019), attention mechanism (Zhang et al. 2021b), and dynamic convolution (Jiang et al., 2022). Unfortunately, most CNNs suffer from the limited ability to model long-range de- pendencies, which is crucial for recovering clean images. Very recently, researchers have started to utilize self-attention strategy to replace the single convolution operation. (Liang et al.] 2021 Chen
ct al. 2021 Zamir et al. 2022 Wang et al. 2022 Lee et al. 2022).
Vision Transtormer. In recent years, Transformer has achieved impressive success in machine translation tasks (Vaswani et al.] 2017). It also performs outstandingly to solve numerous high-
level vision problems (Zhang & Yang 202T, Chu et al., 2021, Wang et al., 202T, Xie et 2021)
due to the content- -eepended global receptive field of the network. Dosovitskiy et al. (2021 firstly proposed ViT introduce Transformer into image recognition. Inspired by these, more and more works started to apply Transformer to solve low-level vision tasks (Liang et al., 2021] Chen et al.,
202T Wang et al.] 2022; Zamir et al. 2022). Wann et al.] (2022) proposed general U-shaped Transformer named Uformer based U-Net (Ronneberger al. 20155) for image restoration. Za- mir et al. (2022) proposed a strong baseline model named Restormer and achieved state-of-the-art performance in several image restoration tasks. [hen et al. (2021) proposed IPT to apply standard Transformer blocks while using pre-training on additional datasets. To sum up, these works have not explored enhancing the global representation of tokens from different dimensions. In contrast, we design a general X-shaped Transformer to bridge this gap.
Concurrent Network. Compared to the commonly-used serial network, the concurrent network has parallel branches in the whole network architecture. Therefore, it has the natural advantage of simultancously conducting different types of representation learning and building direct interactions between dual branches. Recently, few efforts have been made to explore this field. Some works only focused on parallel blocks such as Chen et al. (002) and Pan et al. (222)). Peng et al. (202T) defined representative concurrent network named Conformer solve some high-level vision prob- lems. They designed two branches to respectively leverage convolution operators and self-attention mechanisms. Specifically, the CNN branch preserves fine-detailed local features. The Transformer branch captures long-range dependencies. In this paper, we also design concureent network. Dif- ferent from it, our proposed Xformer applies different Transformer blocks in two branches and we focus on low-level vision tasks. Besides, we propose the Bidirectional Connection Unit (BCU) to greatly enhance the information fusion within two branches.
3
METHOD
3.1 OVERALL PIPELINE
As shown in Fig. [ our Xformer is a hybrid X-shaped Transformer-basee network with two branches. Following the design of U-Net structures (Cheng et al. 2021 Zamir et al. 2022; Wang ct al. 2022), each branch is treated as a separate U-shaped network with skip connections between encoders and decoders. We utilize the spatial-wise window-based Transformer blocks (STBs) to construct the spatial-wise branch. The channel-wise cross-covariance Transformer blocks (CTBs) are used to construct the channel-wise branch. Then, we design the Bidirectional Connection Unit (BCU) to bridge the dual branches for feature complementarity. It can bring information fusion for different branches. Besides, we provide two additional designs. Firstly, we make the last encoder involving STBs of two branches share parameters for the purpose of computational efficiency. Sec- ondly, concatenate the output features from two branches and send them to new refinement module involving CTBs. The overall pipeline is as follows.
Given a degraded image I E IR Nxx CC.. where H, W, and Cin are the height, width, and input channels, our proposed Xformer first uses a 3x3 convolutional layer (Conv) to obtain the shallow feature Fo E H XC. where C is the size of new feature dimension. Next, the feature F, is
3

---

Published as a conference paper at ICLR 2024
Window reverse
Projection
Projection
reshape
+50+
softmax
softmax
LayerNorm
LayerNom
K"
-cfac
ccp
ext
C-MSA
Projection
reshape
LayerNoom
LayerNiom
window partition
Projection
(4) STB
(b) CTB
Figure Module architectures of spatial-wise and channel-wise Transformer blocks (STBI CTB).
concurrently sent to two 4-veell symmetric encoder-decoder branches. Through these two branches, it is transformed into two new deep features F,, F. E IR #x*xxxx xC, In detail, each encoder or decoder contains cascaded Transformer blocks. Encoders take Fo as input and reduce half of the spatial resolution while doubling the number of feature channels as the stage grows. Decoders take the low-resolution features input and reduce half of the feature channels while doubling the size of feature maps. By counting, input features experience three times of downsampling and upsampling. Besides, the up-sampled features concatenated with the corresponding features from encoders via skip connections for the recovery improvement. The concatenated features further pass through 1x1 Conv to reduce channels by half. For information fusion, the down-sampled features after the first and second encoders pass through the BCU and then integrate to the opposite branch through element-wise add. Similarly, the reduced features after the first and second decoders pass through the BCU and are added to the opposite branch. In the end, the new features F, and F- are concatenated and then flow to the refinement module. The output further transmitted into 3x3 Conv to obtain a residual image I, E TR. Hxxx..... Finally, the restored image generated by i I+I.
3.2 DUAL BRANCHES
-pae Branch. As shown in the upper part of Fig. I the spatial-wise branch adopts the encoder-decoder structure with skip connections. The components of encoders and decoders are cascaded spatial-wise window-based Transformer blocks (STBs). We provide the details of STB in
Fig. 3 a). We make X₁ as the output at the les block. We formulate the calculation process of STB as
X'r W-MSA(LN(xx-.)) X1-1,
(I)
FFN(LN(X<:)) + X'1,
where W-MSA means the window-based multi-heas self-attention. Here assume that the number of heads is 1 to transfer MSA to singe-head mode. Given the feature E # xC generated
by layer normalization (LN) Ba al., 2016), W-MSA first split it into non-overlapping GxG
xC for deh window. Next, performs linear projecting to generate query (Q'), key (K'), and value (V"), yielding Q'=X'WQ, K'=X and V'=X'VV, where W. .WV E RCCCC are learnable parameters. We formulate the calculation in inh window as
windows to get features X E IRGxCC
Q'K:" T
Softmax
VC + B)
(2)
where X" is the output feature map in deh window and B is the relative position encoding (RPE) (Shaw et al.) 2018: Raffel et al., (020). Furthermore, the features from all windows are projected together and reshaped to the feature map of size R #xXXC
the last output of W- SMA. For the Ford-forward network (FFN), we use the basic multi-layer perception (MLP) used in recent works (Liu al. 2021 Liang al. 2021) deal with the input features. In short, the STB utilizes non-overlapping windows to generate shorter token sequences for the self-attention computation, which can enable the network obtain fine-grained local patches interactions.
Channel-wise Branch. Similarly, this channel-wise branch contains a 4-level encoder-decoder structure. special, the encoders and decoders are constructed by cascaded channel-wiss Trans- former blocks (CTBs). The details of CTB are shown in Fig. b). Assuming that X is the output at the ken block, the calculation process can be formulated as
X' C-MSA -k-1)) + X
(3)
X (FN w) + +X'.
4

---

Published as a conference paper at ICLR 2024
where C-MSA means the channel-wise multi-head self-attention. We also assumee that the number of heads is and transfer MSA to a singe-head fashion. Given the normalized feature X E "xxWxx C-MSA first utilizes the projecting module to get prepared query, key and value. In order to intro- duce contextualized information into self-attention computation, we choose to use x depth-wise convolution (Conv) following 1 x 1 Conv generate query (Q), key (K), and value (V). We make p Q-W3 d W X, K=W W X, and V=Wa W X. where W means parameters of 1 point-
wise Conv and O() means parameters of 3x3 depth-wise Conv. Then, the obtained Q, K, V are reshaped into new feature maps of size RCXN where N xW. The query and key are further nor- malized to prepare for cross-covariance attention. The new transposed attention map is calculated by Q and K" with size of RCxx, The calculation process of the C-MSA is formulated as
Softmax (QK? /T)V,
(4)
where 7 is learnable temperature parameter and X E RCNN is the output. Then X is reshaped to the original feature size of IR # X xe. The output further passes through linear projecting layer. Added by the shortcut X;- new features x'x is transmitted to the following part. For the FFN, we introduce the gating mechanism and depth-wise convolutions proposed in the recent work (Zamir et al. 2022) to enrich the feature transferring, which is validated to be effective. Equipped with the used C-MSA and FFN, the CTB enjoys strong ability to capture direct global context interactions.
3.3 BIDIRECTIONAL CONNECTION UNIT
Motivation. As the dual branches enable the network to capture both patch-level and channel-level information, the information fusion is treated as an important step to enhance global information modeling. Simple concatenating operation is not able to effectively utilize information from differ- ent branches. The direct connection of dual branches is not the best choice. Therefore, we propose the Bidirectional Connection Unit (BCU) to couple the deep features from their respective feed- forward processes for feature complementarity. We demonstrate that the proposed BCU plays an important role to provide enhanced information fusion.
Specific Design. We design the BCU bridge the two branches in an interactive manner. We carry out the specific feature complementarity like the form of absolute position encoding (Vaswani al. 2017). On one hand, we add the global context information brought by channel-wise self- attention to the feature maps of the spatial-wist branch. On the other hand, we add the fine-grained patch-level information brought by local patches interactions to the feature maps of the channel- wise branch. In detail, the BCU contains two simple convolution layers. Specifically, we use 3x3 depth-wiss, wise convolution layer to refine the deep features from the spatial- wise branch for the purpose of saving computational consumption. We use common 3 3x3 convolution layer to refine features from the channel-wise branch to provide more channel-dimension interactions. With the 3 3x3 kernel size, the feature refinement process can provide extra contextualized information. The specific implementation is shown in Fig.
3.4 IMPLEMENTATION DETAILS
Specific Settings. Firstly. we set the layer numbers of both branches the same, which are [2. 4, 4, 6, 4, 4, 2]. The number of CTBs in the refinement stage is set to 4. Secondly, we set the number of heads in corresponding layers to [1, 2, 4, 8, 4, 2, 1]. The head number of CTBs in the refinement stage is set to 1. Meanwhile, the channels number of shallow features generated by the first convolution layer is set to 48. The expansion size of hidden layers in FFN set to 2.66. Thirdly, the window size in spatial- spat Transformer blocks is set to 16. Note that also utilize the shifted- window strategy (Liu et al. [202I). Besides, we use pixel-unshuffl and pixel- shuffle operations (Shil al. 2016) for downsampling and upsampling. Lastly, following the recent work (aairr et al. we use the progressive training strategy for fair comparisons.
2022
Loss Function. Following most recent works (Lai et al. 2017, Zhang et al. 2020 Zamir et al., 2022), use L1 loss function to optimize proposed Xformer. For image denoising, the goal of training Xformer is to minimize the L1 loss, which is formulated as
L IH THQll1:
(5)
where i TiQ is the output of our Xformer and TnQ is the corresponding ground-truth image.
5

---

Published as a conference paper at ICLR 2024
Method
All STB
All CTB
STB+CTB
Method
w/o BCU
BCU-1
BCU-2
Complete BCU
Method
w/o Shin
w/ Shin
Parauss
26,03
28.81
25,23
Pomms
24.70
24.71
25.22
25.23
Parame
25.25
25.
(LOPs(G)
422
40.9
42.2
FLOPs
42.2
PSNR (dB)
29.87
29.67
29.4
PSNR (dB)
29.82
29.84
29.92
29.91
PSNR (dB)
29.85
29.94
SSIM
0.8851
0,8330
0.8805
SSIM
0.3522
0.8848
0.8859
0.8865
SSIM
0.8852
0.8365
(a) Ablation study of block setting.
(b) Ablation study of BCU settings.
(c) Whether to shift.
CTB
BCU
Structure
Params (M)
FLOPs
PSNK (cBB)
Method
SwinIK
Restormer
Xfoemer (ours)
single-branch
26.48
40.6
29.54
0.8833
Params (M)
11.50
26
25,23
single branch
26.11
38.7
29.68
0.8829
PLOPs (G)
2012
38.7
42.2
two branches
24.70
40.9
29.82
0.5842
McMaster
30.22
30.30
30.38
two nrn-branches
25.25
42.2
20.94
0,8805
Usbaa100
29.82
30.02
30.36
(d) Ablation study of designed models with different branches.
(e) PEOaaS--LLOPPPSEEE comparisons.
Table 1: Ablation experiments (a-d) and model complexity comparisons (e). For ablation, we train models on Gaussian color image denoising task with o= for 100k iterations and test on Urban 100.
Features from the last CTB Features from the last STB
Features from the last CTB Features from the last STB
Urb w/o BCU
w/o w/ BCU
Urban100_HR
w/o
w/ BCU
Figure 4: Visualization of feature maps from the last STB and CTB in the encoder-decoder module of dual branches. We compare different situations about whether to use BCU.
4
EXPERIMENTAL RESULTS
4.1
EXPERIMENTAL SETTINGS
Data and Evaluation. We conduct experiments on two denoising tasks, including synthetic image
denoising using noisy images generated with additive white Gaussian noise and real image denoising using real-world noisy images. For Gaussian denoising, following the previous work (Liang et al.
2021
we use DIV2K (Timofte et al. 2017), Flickr2K (Lim et al. 2017). BSD500 (Arbelaez al.
2010
and WED Ma et al. (0166) as training data. Set12 (Zhang et al. 2017a), BSD68 (Martin Kodak24 (Franzen 1999). McMaster (Zhang et al.) 2011 and Urban100 (Huang 2015) are the testing data. For real image denoising, same with Restormer (Zamir et al.
2001).
et al.
2022.
we use SIDD (Abdelhamed al., 2018) to train our model. The evaluation is performed on patches of the SIDD validation set (Abdelhamed et al. 2018) and 50 pairs of images from the
1,280
DND (Plotz& & Roth (017). Note that we evaluate the performance with the commonly-usee PSNR and SSIM (Wang et al. (2004) metrics. Besides, we also provide comparisons of FLOPs and model size. We set the input image size to x 128x 128 when calculating FLOPs.
Training Settings. We perform data augmentation on the training data through random horizontal or vertical flips and rotation of 90º, 180", and 270". Using progressive training strategy proposed by Restormer (Zamir et al. 2022). we set the batch size and patch size pairs to (64,1282) (40.160²), (32,1922), (16,256*). (8.320").
(8.3843) at training iterations [Ok, 92k, 156k, 204k, 240k, 276k]. AdamW (Loshchilov & Hutter, 2019) is used to optimize our model with 81 0.9, /92 0.999, and weight decay 10-1 We train our model for total 300k iterations and the initial learning rate is
set to 3x-0-- and gradually reduced to 10-6 through the cosine annealing (Loshchilov & Hutter, 2017). Our Xformer is implemented on PyTorch (Paszke et al. 2017) using 4 Nvidia A100 GPUs. 4.2
ABLATION STUDY
For ablation experiments, we train all the models on Gaussian color image denoising task with noise level 5=50. We train these models for 100k iterations. The evaluations are performed on 1rban100 dataset. We also report the model size and FLOPs. The results are shown in Tab. [
Impact of STB and CTB. We design the ablation study to support using both STB and CTB.
Ablation design. As shown in Tab. la we present three different network designs. Specifically, using all STB or CTB means that we replace all the Transformer blocks in Xformer with STB or CTB. For fair comparisons, we design these models with comparable complexity.
Analyses and conclusion. As we can see, the model using all CTB gets poor performance since it pays less attention to patch-level information. Furthermore, the model using all STB obtains suboptimal performance. However, using STB and CTB together in dual branches achieve the best performance gains. The joint application of hybrid Transformer blocks can simultaneously obtain patch-level and channel-level information, which is very important.

---

Published as a conference paper at ICLR 2024
HQQ
Noisy (o==50)
BM3D
IRCNN
DoCNN
Urban100: img.012
RNAN
RDN
SwinlR
Restormer
Xformer (ours)
HQ
Noisy (a=50))
Uirhan 100: img 033
RDN
Restoreer
Xformer (ours)
Figure 5: Visual comparisons Gaussian color and gray image denoising (a=50).
Impact of BCU. We further discuss the impact of the BCU by the ablation study.
Ablation design. We set four different experimental conditions. As shown in Tab. Tb we compare the results of models without BCU, using single-direction BCU, and using complete BCU, Note that using single-direce directio BCU means that we only use the DWConv or the Conv provide the information transmission from single direction. Furthermore, BCU-1 denotes the model using DWConv and BCU-2 denotes the model using Conv.
Analyze the importance of using BCU. As we can see, the model with complete BCU achieves PSNR gain of 0.12 over that without BCU, which indicates that the BCU is an important component.
Analyze the necessity of the interactive manner. The models using single-direction BCU have sub- optimal performance. Moreover, we find that the model with BCU-2 yields 0.08 dB gain over that with BCU-1, It reveals that the information flow from the channel-wise branch has bigger impact.
Visual analyses and conclusion. We provide visual results Fig. We visualize the deep features from the last STB CTB in corresponding branches. We network with BCU can capture more extra information and thus show better visual results. We conclude that proposed BCU provide effective information fusion for the dual branches. With BCU, our proposed network can achieve promising performance.
that the STB and CTB in the
Impact of Different Branches. We also discuss the importance of different branches.
Ablation design. We design the networks with comparable model size and FLOPs. As shown in Tab. [d] present four different networks, including using STB-hased branch, using CTB-based branch, using dual branches without BCU, and using dual branches with BCU. The model using single branch is degraded complete U-shaped network like the architecture of Restormer.
Analyze the results of using single branch. The model using STB-based branch performs better than that using CTB-based one. demonstrates that the patch-level information deserves
attention.
Analyze the results of using dual branches without BCU. We find that the model using dual branches without BCU gets limited performance. It confirms the statement discuss in Sec. 3.3 The simple concatenating operation fails to effectively utilize the obtained information. Thus, the direct connection of dual branches even brings unsatistied results.
Analyze the results of using dual branches with BCU. As we can see, with BCU, our model using dual branches can achieve greatly enhanced performance. further validates that the BCU plays an important role in fusing patch-level and channel- information in our concurrent network.
Conclusion. Thanks these two joint designs, our proposed method can explore enhancing global information modeling in Transformer and thus outperform previous promising methods.
Impact of Shift. Table Ic shows the comparisons about whether to use shift. As we can see, the model using shift operation obtains better performance gains. It is because shift operation can bring more global receptive fields for window-based self-attention. Therefore, we choose to use shift.
4.3 GAUSSIAN IMAGE DENOISING RESULTS
We provide the comparisons of our Xformer with recent representative image denoising methods both Gaussian color and grayscale image denoising tasks. As shown in Tabs. and Tab.
7

---

Published as a conference paper at ICLR 2024
Dataset
BM3D
DECNN
IRCNN
FFDNet
NLRN
MWCNN RNAN RDN DRUNet SwinIR Restormer Xformer (ours)
15
32.37
32.86
32.76
32.75
33.16
33.15
33.25
33.36
33.42
33,46
Set12
25
29.97
30.44
30.37
30.43
30.80
30,79
30.94
31.01
31,08
31.16
50
26.72
27.18
27.12
27.32
27.64
27.74
27.70 27.60
27.90
27.91
28.00
28.10
31.08
31,73
31.63
31.63
31.88
31.91
31,96
31,98
BSD68
25
28.57
29.23
29.15
29.19
29.41
29.41
29.48
29.50
29.52
29,55
50
25.60
26.23
26.19
26.29
26.47
26.53
26.48 26.41
26.59
26,58
26.62
26.65
15
32.35
32.64
32,46
32.40
33.45
33.17
33.44
33.70
33.79
33.98
Urban100
25
29.70
29.95
29.80
29.90
30,94
30,66
31.11
31.30
31.46
31.78
50
25.95
26.26
26.22
26.50
27.49
27.42
27.65 27.40
27.96
27.98
28.29
28.71
Table PSNR (dB) comparisons for Gaussian grayscale image denoising on three benchmark datasets, The underlined and bold numbers indicate the second best and the best results.
Datasct
BM3D DrCNN IRCNN FFDNet RNAN RDN DRUNet P3AN IPT SwinlR Restormer Xformer (ours)
15
33.52
33,90
33.86
33.87
34,30
34.42
34.40
34.43
CBSD68
25
30.71
31.24
31.16
31.21
31.69
31.78
31.79
31.82
50
27.38
27.95
27.86
27.96
28.27 28.31
28.51
28.37 28.39
28.56
28.60
28.63
15
34,28
34.60
34.69
34,63
35.31
35,34
#35.35
35.39
Kodak24
25
32.15
32.14
32.18
32.13
32.89
32.89
#32.93
32,99
28.46
28.95
28.93
28.98
29.58
29.66
29.86
29.69
29.64
29.79
#29.87
29.94
34.06
33.45
34.58
34,66
35,40
35.61
35.61
35.68
McMaster
25
31.66
31.52
32.18
32.35
33.14
33.20
33.34
33.44
50
28.51
28.62
28:91
29.18
29.72
30.08
29.98
30.22
30.30
30.38
15
33.93
32.98
33.78
33.83
34.81
35.13
35.13
35,29
Urban100
25
31.36
30,81
31,20
31.40
32.60
32,90
32,96
33.21
27.93
27.59
27.70
28.05
29.08
29.38
29.61
29.51
29.71
29.82
30.02
30.36
Table 3: PSNR (dB) comparisons for Gaussian color image denoising four benchmark datasets. The underlined and bold numbers indicate the second best and the best results. ddenotes results which are obtained by testing with officially provided pre-trained models.
2007) is the classical denoising method. The CNN-based methods in- clude DrCNN (Zhang al., 2017a), IRCNN (Zhang et al. (017b), FFDNet (Zhang et al.) 2018).
BM3D (Dabov et al.
NLRN (Liu et al. 201 MWCNN (Liu et al. 2018b). RNAN (Zhang et al. 2019), RDN (Zhang al. 2020). DRUNet (Zhang al. 2021a), and P3AN (Hu et al. (221)). The Transformer- methods include SwinIR (Liang et al. 2021). IPT (Chen et al. 2021), l), and Restormer (Zamir et al.,
2022). All the results are obtained by open available data. We get some results by using officially
provided pre-trained models. Following most recent works (Liang et al., 2021 Zamir et al., 2022
we set the additional noise level 15, 25, and 50. We also provide the visual comparisons in Fig.
The results of model parameters and FLOPs comparisons are shown in Tab. Te
Quantitative Comparisons. We present the PSNR results of all compared approaches in Tab. 2 and Tab. The corresponding scores are obtained by testing on several benchmark datasets for Gaussian grayscale and color image denoising. As we can see, our Xformer achieves the best PSNR performance across all evaluation datasets. Specifically, for the evaluation on high-resolution Ur- ban 100 dataset (Huang et al.) 2015) under the challenging noise level 50, Xformer obtains 0.42 dB performance gain over the previous best Transformer- 2022), as shown in Tab. 2 Similarly, Table [3] also shows that the best performance is achieved by Xformer for Gaussian color image denoising. Compared to the results of SwinIR (Liang et al.) 2021), Xformer has better performance gains while maintaining 4.76x fewer FLOPs. It also worth mentioning that our proposed method has the comparable model size and FLOPs with Restormer. In short, the experimental results demonstrate that proposed Xformer becomes new promising Transformer- based network for the Gaussian image denoising.
network Restormer (Zamir et al.,
Visual Comparisons. The visual comparisons for Gaussian color and grayscale image denoising on some challenging examples are shown in Figs.5) The noise level is set to 50 and results are obtained by testing on Urban100. We can see that our Xformer is able to remove heavy noise corruption for color image denoising. Compared to some previous denoising methods, our method obtains visually pleasing results. Besides, for the gray image denoising, the detailed textures and high- sighfrreuency components of the original images reserved by using method. However, others suffer from the heavy blurring and missing details. It demonstrates that our Xformer performs excellently for both color and grayscale image denoising for visual results.
8

---

Published as a conference paper at ICLR 2024
Dataset Method
DaCNN
CBDNet
RIDNer
AINDNet
VDN
SADNei
DANe:
CyclelSP
MIRNet
Deamñet
DMGL
MAXIM
Utormer
Restormer Xfermer
SIDD
PSNR
25.65
23.66
30.78
38.71
39,05
39.28
29,45
39.47
29.52
39.72
39.47
38.94
39.95
39.89
40.02
29.98
SSIM
0.685
0.583
0.501
0.951
0.954
0.056
0.957
0.957
0.957
0,959
0.957
0.953
0.960
0.900
0.360
[L960
DND
PSNR
34.51
32.47
38.06
39.26
79.37
90. 88
19.59
19.58
39.56
39.88
39.63
39.77
39.84
4004
40.03
40.19
SSIM
0.51
0.790
0,912
0,953
0.951
0.952
0.952
0.955
0.956
0,956
0.953
0.956
0.954
0.956
0.956
0.957
Table 4: PSNR (dB) and SSIM comparisons for real image denoising two henchmark datasets. The underlined and bold numbers indicate the second best and the best results.
Model Size Comparisons. Table 1c provides comparisons of parameters number and FLOPs with existing state-of-the-art methods. We calculate the FLOPs assuming that the input size is 128. The PSNR scores reported on the popular benchmark datasets McMaster (Zhang 3 128 x
eet al. 2011) and Urban (Huang et al. 2013) under Gaussian color image denoising with noise level 0=50. We mainly compare our proposed method recent Transformer-basee networks, in- cluding SwinlR and Restormer. We find that our Xformer enjoys very low FLOPs when compared to SwinIR. Meanwhile, it is seen that our method has comparable model size and FLOPs with Restormer. However, our Xformer can achieve the best performance among them. In special, it obtains 0.34 dB higher PSNR score over Restormer on Urban100. indicates that our method has an acceptable computational and memory cost while performing promising image denoising.
4.4 REAL IMAGE DENOISING RESULTS
As shown in Tab. We show the quantitative comparisons of our Xformer with other state-of-the- art methods on real-world image denoising task. In detail, we report the evaluation results from the
classic denoising method BM3D (Dabov et al.] (007), CNN-based methods DrCNN (Zhang et al.
2017a), CBDNet (Guo al. 2019), RIDNet (Anwar & Barnes 2019), AINDNet (Kim al. 2020).
VDN Yue et al., 2019), SADNet (Chang et al. 2020). DANet Yue et al. (020), CycleISP (Zaiir
ct al. 2020a). MIRNet (Zamir et al. 2020b), DeamNet (Ren et al. 2021). DAGL (Mou et al. 202 MAXIM (Tu et al., (022). and Transformer-based methods Uformer (Wang et al.) 2022) and Restormer (Zamir et al., [022]. Following the recent work (Zamir et al.) 2022), we only use
SIDD (Abdelhamed et al. 2018) dataset to train our models. Then, the trained models are directly used to perform evaluations on the DND (Plotz& & Roth 2017) benchmark. As DND does not provide ground-truth labels, the corresponding results are obtaied by uploading images to the online server. Note that all the results are obtained from the open-souree data.
Quantitative Comparisons. Table 4 shows the PSNR and SSIM scores of recent approaches on real-world image denoising. As we can see, our proposed Xformer outperforms all the state-of-the- art methods on the DND dataset (Plotz & Roth (017) and achieves comparable performance on the SIDD dataset (Abdelhamed et al.) 2018). Compared to all the CNN-based methods, our Xformer has the obvious performance improvement. Furthermore, compared to Restormer (Zamir et al.) 2022), method performs better with comparable mode complexity. Besides, our Xformer achieves higher performance gains than Uformer (Wang et al. 2022) while maintaining 2.01 x fewer model parameters. In special, Restormer only utilized channel-wise self-attention and paid less attention to local patches interactions. On the contrary, Uformer focused on spatial-wise self-attention and neglected the channel-dimensie token interactions. In contrast, our proposed Xformer explore building interactions among tokens from both spatial and channel dimensions. Through fusing both patch-level and channel-level information, our method is able to obtaín better performance.
CONCLUSION
In this work, we propose a hybrid X-shaped vision Transformer, named Xformer, for image de- noising task. We design a concurrent network structure to utilize spatial-wise window-based Trans- former blocks and channel-wise Transformer blocks respectively in two branches. Our proposed Xformer can enable each branch to proceed representation learning from the corresponding dimen- sion, spatial or channel. Besides, we propose the Bidirectional Connection Unit (BCU) to bridge the separate branches. Specifically, the BCU provides information fusion in an interactive manner and greatly enhances the global information modeling ability of both branches. We conduct ex- tensive experiments on the synthetic and real-world image denoising tasks. Experimental results demonstrate that our Xformer can outperform recent ofate-of-the-ar methods both quantitatively and visually with comparable model size and computational cost.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported part by NSFC grant (62141220, U19B2035) and Shanghai Municipal Science and Technology Major Project (2021SHZDZZ00102.,
REFERENCES
AbdeIrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In CVPR, 2018.
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, al. Xeit: Cross-covariance image transformers. In NeurlPS, 2021.
Saeed Anwar and Nick Barnes. Real image denoising with feature attention. In ICCV, 2019.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hier- archical image segmentation. TPAMI, 2010.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 1rXiv::607.6444 2016.
arXiv preprint
Meng Chang, Qi Li, Huajun Feng, and Zhihai Xu. Spatial-adaptive network for single image de- noising. In ECCV, 2020.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma. Chun jing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, 2021.
Qiang Chen, Qiman Wu, Jian Wang, Qinghao Hu, Tao Hu, Errui Ding, Jian Cheng, and Jingdong Wang. Mixformer: Mixing features across windows and dimensions. In CVPR, 2022.
Shen Cheng, Yuzhi Wang, Haibin Huang, Donghao Liu, Haoqiang Fan, and Shuaicheng Liu. Nbnet: Noise basis learning for image denoising with subspace projection. In CVPR, 2021.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In
NeuriPS, 2021.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik. and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. TIP, 2007.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x 16 words: Transformers for image recognition at scale. ICLR, 2021.
Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. TIP, 2006.
Rich Franzen. Kodak lossless true color image suite. source: http://vk. us/graphics/kodak, 1999.
Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind de- noising of real photographs. In CVPR. 2019.
Xiaowan Hu, Ruijun Ma, Zhihong Liu, Yuanhao Cai, Xiaole Zhao, Yulun Zhang, and Haoqian Wang. Pseudo 3d cotreloorelatoon network for real image denoising. CVPR, 2021.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from trans- formed self-exemplars. In CVPR, 2015.
Yifan Jiang, Bart Wronski, Ben Mildenhall, Jon Barron, Zhangyang Wang, and Tianfan Xue. Fast and high-quality image denoising via malleable convolutions. ECCV, 2022.
Yoonsik Kim, Jae Woong Soh, Gu Yong Park, and Nam Ik Cho. Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. In CVPR, 2020.
10

---

Published as a conference paper at ICLR 2024
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR, 2017.
Hunsang Lee, Hyesong Choi, Kwanghoon Sohn, and Dongbo Min. Knn local attention for image restoration. In CVPR, 2022.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCVW, 2021.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- networks for single image super-resolution. In CVPRW, 2017.
Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas Huang. Non-local recurrent network for image restoration. In NeurIPS, 2018a.
Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo. Multi-level wavelet-cnn for image restoration. In CVPRW, 2018b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Nya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang. Waterloo exploration database: New challenges for image quality assessment models. TIP. 2016.
Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, and Andrew Zisserman. Non-local sparse models for image restoration. In ICCV, 2009.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001.
Chong Mou, Jian Zhang, and Zhuoyuan Wu. Dynamic attentive graph learning for image restoration. In ICCV, 2021.
Xuran Pan, Chunjiang Ge, Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, and Gao Huang. On the integration of self-attention and convolution. In CVPR, 2022.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Yc. Conformer: Local features coupling global representations for visual recognition. In ICCV, 2021.
Pietro Perona and Jitendra Malik. Scale-space and edge detection using anisotropic diffusion. TPAMI, 1990.
Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In CVPR, 2017.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 2020.
Chao Ren, Xiaohai He, Chuncheng Wang, and Zhibo Zhao. Adaptive consistency prior based deep network for image denoising. In CVPR, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.

---

Published as a conference paper at ICLR 2024
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. arXiv preprint arXiv: 2018.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
Chunwei Tian, Yong Xu, and Wangmeng Zuo. Image denoising using deep enn with batch renor- malization. Newral Networks, 2020.
Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee, et al. Ntire 2017 challenge on single image super-resolution: Methods and results. CVPRW, 2017.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mlp for image processing. In CVPR, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. ICCV, 2021.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: general u-shaped transformer for image restoration. CVPR, 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. TIP, 2004.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transform- ers. In ICCV, 2021.
Zongsheng Yue, Hongwei Yong, Qian Zhao, Deyu Meng, and Lei Zhang. Variational denoising network: Toward blind noise modeling and removal. In NeurlPS, 2019.
Zongsheng Yue, Qian Zhao, Lei Zhang, and Deyu Meng. Dual adversarial network: Toward real- world noise removal and noise generation. In ECCV, 2020.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Cycleisp: Real image restoration via improved data synthesis. In CVPR, 2020a.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Learning enriched features for real image restoration and enhance- ment. In ECCV, 2020b.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021.
Syed Waqas Zamir, Aditya Arora, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning deep enn for image denoising. TIP. 2017a.
Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep enn denoiser prior for image restoration. In CVPR, 2017b.
12

---

Published as a conference paper at ICLR 2024
Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward fast and flexible solution for based image denoising. TIP, 2018.
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. Plug-and-play image restoration with deep denoiser prior. TPAMI, 2021a.
Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. Color demosaicking by local directional inter- polation and nonlocal adaptive thresholding. JEI, 2011.
Qinglong Zhang and fu-Bin Yang. Rest: An efficient transformer for visual recognition. In NeurIPS, 2021.
Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu. Residual non-local attention net- works for image restoration. In ICLR, 2019.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image restoration. TPAMI, 2020.
Yulun Zhang, Kunpeng Li, Kai Li, Gan Sun, Yu Kong, and Yun Fu. Accurate and fast image denoising via attention guided scaling. TIP, 2021b.
13


Reviews:
[{"review_id": "JnabSIzSCZ", "reviewer": "Reviewer_V2SV", "summary": "This paper proposes XFormer for image denoising, which aims to combine SwinIR which utilizes spatial-wise self attention and Restormer designing channel-wise self attention for image denoise, thereby leveraging the advantages from both methods. The designs, including dual-branch architecture and the bidirectional connection unit for bilateral interactions between two branches, are straightforward.m The paper is motivated well, whilst the technical novelty is incremental, especially compared to SwinIR and Restormer.", "questions": "It is suggested to investigate both theoretically and experimentally what kind of denoising scenarios (noises) are STB and CTB suitable for, respectively.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is motivated well. It is reasonable to combine the advantages of both spatial-wise self attention and channel-wise self attention to capture both the local fine-grained features and global features across channels.\n\n2. The paper is organized well and easy to follow despite some typos.", "weaknesses": "1. The technical novelty is incremental. There are two core designs: the dual-branch architecture and the bilateral interactions between two branches, which are both typical designs and have been extensively explored in other work. Thus, the technical novelty is limited, especially compared to SwinIR and Restormer.\n\n2. Compared to Restormer, Xformer has limited performance improvement, especially on real image denoising scenarios which is more important for evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ou9GWsi75i", "reviewer": "Reviewer_GouV", "summary": "The authors proposed a hybrid X-shaped transformer for high-quality image denoising. The idea is good. Specifically, the technique consists of spatial-wise transformer blocks (STB) and channel-wise transformer blocks (CTB) to model global information. The authors provide extensive ablation studies to support the effectiveness of each proposed component, like STB, CTB, and BCU etc. The main comparisons with recent methods further show that the proposed method Xformer achieves better performance than others quantitatively and visually.", "questions": "Please refer to the Weaknesses for details.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The idea is good and novel. The proposed Xformer exploits stronger global representation of tokens with a hybrid implementation of spatial-wise and channel-wise Transformer.\n\nThe bidirectional connection unit (BCU) is proposed to couple the learned representations from two branches of Xformer. It is simple but effective according to the ablation.\n\nThe authors provide extensive ablations to show the effects of some key components, like STB, CTB, BCU, and shift operation.\n\nThe main comparisons are also extensive. The authors provide both Gaussian and real image denoising results, where the proposed Xformer achieves better average quantitative results and also shows better visual results.\n\nThe writing is good and the work is well-prepared. The overall paper framework is well-organized.\n\nThe authors provide more results and analyses in supplementary file, where a sample code is also available. Such a code makes the reproducibility more faithful.", "weaknesses": "Some details are not clear enough for better understanding. How did the authors determine the final model when training is finished? For example, did the authors choose the model based on the best validation performance or just use the model from the final iteration?\n\nIn the ablation study, Table 1 (b), it seems that w/o BCU and BCU-1 is comparable, BCU-2 and Complete BCU is comparable. Please give more analyses about their difference.\n\nIf the proposed method could be used for other image restoration tasks? If so, please give some comments and discussions. Or is it specifically designed for image denoising?\n\nThe Xformer shows very good performance. Are there any failure cases for image denoising? Namely, the proposed method can hardly recover good details either.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4i7Ud1Jozd", "reviewer": "Reviewer_wiTn", "summary": "In this paper, the authors propose a hybrid X-shaped vision transformer for image denoising, named Xformer. Xformer has two branches with one containing the spatial-wise transformer blocks and the other containing the channel-wise transformer blocks. Between these two branches, there are the bidirectional connection units which couple the learned representations from these two branches. The experimental results show that the proposed method performs well on the synthetic image denoising dataset, but the method does not achieve the SOTA on the real-world image denoising dataset.", "questions": "See the weakness section.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The X-shaped architecture is elegant and reasonable.\n2. The experimental results on the synthetic dataset are good.\n3. The overall paper writing is good.", "weaknesses": "There are several places that are not intuitive or clear:\n1. The authors claim that \"we make the last encoder involving STBs of two branches share parameters for the purpose of computational efficiency.\" However, it is unclear how much the performance will be influenced by the parameter-sharing strategy. It is also not clear why it is critical to share parameters for this place in the network. Why not share parameters in other places?\n2. The authors claim that \"In short, the STB utilizes non-overlapping windows to generate shorter token sequences for the self-attention computation, which can enable the network to obtain fine-grained local patches interactions.\" Shorter token sequences? Compared to what? Why does the shorter token sequences can enable the network to obtain fine-grained local patches interactions?\n3. The authors claim that \"In order to introduce contextualized information into self-attention computation, we choose to use 3×3 depth-wise convolution (Conv) following 1×1 Conv to generate query (Q), key (K), and value (V).\" Why not directly use a vanilla 3×3 convolution?\n4. The authors claim that \"Specifically, we use a 3×3 depth-wise convolution layer to refine the deep features from the spatial-wise branch for the purpose of saving computational consumption.\" Why not also using the 3×3 depth-wise convolution layer to refine the deep features from the channel-wise branch? Will it influence the performance compared to using the 3×3 vanilla convolution?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sr3XTfyuxg", "reviewer": "Reviewer_Md3A", "summary": "In the proposed architecture, the authors design two branches to conduct these interaction modes. Both branches use an encoder-decoder setup to capture multi-scale features. An essential addition to this structure is the \"Bidirectional Connection Unit (BCU)\", which couples the learned representations from the two branches and facilitates better information fusion.\n\nThe combined designs enable the Xformer to effectively model global information in both spatial and channel dimensions. Through extensive experiments, the authors demonstrate that the Xformer achieves state-of-the-art performance on both synthetic and real-world image denoising tasks, all while maintaining comparable model complexity.", "questions": "Please see the weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper stands out in its innovative approach to image denoising by proposing a hybrid X-shaped Transformer. The clear presentation, combined with extensive experiments and state-of-the-art results, underscores its significance in the domain. The novel components, especially the BCU, and the creative combination of spatial and channel-wise blocks, emphasize its originality. The potential impact of this work on the broader image processing community is considerable.", "weaknesses": "The hybrid nature of the model, with its dual branches and BCU, might be challenging for some readers to grasp fully. While the description seems structured, visual aids might be lacking.\n7.Questions：Could you provide more insights into the design rationale behind the Bidirectional Connection Unit (BCU)? How does it ensure efficient information fusion between the two branches?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)