CascadedGaze: Efficiency in Global Context Extraction for Image Restoration
Authors: Amirhosein Ghasemabadi
Muhammad Kamran Janjua
Mohammad Salameh
CHUNHUA ZHOU
Fengyu Sun
Di Niu
Venue: Accepted by TMLR ()

Abstract: Image restoration tasks traditionally rely on convolutional neural networks. However, given the local nature of the convolutional operator, they struggle to capture global information. The promise of attention mechanisms in Transformers is to circumvent this problem, but it comes at the cost of intensive computational overhead. Many recent studies in image restoration have focused on solving the challenge of balancing performance and computational cost via Transformer variants. In this paper, we present CascadedGaze Network (CGNet), an encoder-decoder architecture that employs Global Context Extractor (GCE), a novel and efficient way to capture global information for image restoration. The GCE module leverages small kernels across convolutional layers to learn global dependencies, without requiring self-attention. Extensive experimental results show that our approach outperforms a range of state-of-the-art methods on denoising benchmark datasets including both real image denoising and synthetic image denoising, as well as on image deblurring task, while being more computationally efficient.

Published in Transactions on Machine Learning Research (05/2024)
GascadedGaze: Efficiency in Global Context Extraction for Image Restoration
Amirhosein Ghasemabadi Muhammad Kamran Janjua'? Mohammad Salameh Chunhua Zhou', Fengyu Sun Di Niu'
Depp.. ECE, University of Alberta, Canada, 'Huawei Technologies, Canada, 'Haawei Kirin Solution, China (ghasemab, )mii)) Qualberta.ca, (kamranjjanjuc mohammad. salameh, zhouchunboua Ghuawei..com com, sunfenggu@hhssliioonn com "Equal Contribution
Reviewed on OpenReview: https: openreview. net/ forum? id= C3FXHaNVuQ
Abstract
Image restoration tasks traditionally rely on convolutional neural networks. However, given the local nature of the convolutional operator, they struggle capture global information. The promise of attention mechanisms in Transformers is to circumvent this problem, but it comes at the cost of intensive computational overhead. Many recent studies in image restoration have focused on solving the challenge of balancing performance and compu- tational cost via Transformer variants. In this paper, we present CascadedGaze Network (CGNet), an encoder-decoder architecture that employs Global Context Extractor (GCE), a novel and efficient way to capture global information for image restoration. The GCE module leverages small kernels across convolutional layers to learn global dependencies, without requiring self-attention. Extensive experimental results show that our computa- tionally efficient approach performs competitively to range of state-of-the-art methods on synthetic image denoising and single image deblurring tasks, and pushes the performance boundary further on the real image denoising lask. We release the code at the following link: tiggttubb.... com/ Ascend ReeeeaddReeecccc/ Caa.cchhcaacceeaeeeeeeeee
1 Introduction
Image restoration refers to recovering the original image quality by addressing dogradation introduced dur- ing the capture, transmission, and storage processes. This degradation includes unwanted elements like noise, blurring, and artifacts. Given that infinitely many feasible solutions may exist, image restoration is
considered an ill-posed problem. It challenging task as it involves processing high-frequency elements like noise while preserving crucial image characteristies such edges and textures (Su et al.) 2022b). To tackle this complexity, current image restoration ion techniques leverage deep neural networks. These networks have demonstrated remarkable progress across various restoration tasks, achieving state-of-the-art results on
several benchmark datasets (Li et al. 2023b Zamir et al. 2021 Wang et al. 2022b Cheng et al. 2021 Chu
et al.
2022
While convolutional neural networks (CNNs) have been widely used for image restoration (Chen et al. 2021
Fan et al. 2022; Chang al. 2020; Yue al., 2020),, their limited receptive field size restricts their ability to capture long-range dependencies and global context effectively. Conversely, Transformers excel at modeling global interactions and dependencies, making them well-suited for image restoration tasks that require a holistic understanding of the image content (Dosovitskiy et al.] 2020 Vaswani et al. 2017 Ramachandran
et al. 2019 Touvron et al. 2021]. However, Transformers come at the cost of intensive memory consumption and quadratic computational complexity self-attention as image spatial resolution increases.
Due the computational overhead of Transformers, especially self-attention, there has been growing interest in developing efficient types of Transformers. Various techniques have been proposed to address this

---

Published in Transactions on Machine Learning Research (05/2024)
Real Image Denoising
Gaussian Image Denoising
40.40
CGNet
CGNet
30.05
40.30
NAFNet
30.00
Restormer
PSNR (dB)
PSNR (dB)
29.95
40.20
29.90
40.10
ART
29.85
CAT
Restormer
CODE
40.00
HINet MAXIM
29.80
SwinIR
GRL-S
75
100
125
150
175
1000
2000
3000
4000
MACs (G)
MACS (G)
Figure 1: Computational Efficiency Performance. Left: PSNR vs. MACs (G) comparison SIDD real image denoising. Right: PSNR MACs (G) comparison Gaussian image denoising tested Kodak24 dataset with noise level = 50. Our model achieves state-of-the-art results and is computationally efficient.
challenge, including local attention [Wang et al. 2022b Liang et al. 2021) which applies self-attention to smaller input patches instead of the entire input. Channel attention introduced by Restormer (Zamir et al. 2022) is another method that applies the attention mechanism to the channel dimension rather than the spatial dimension. Even though these methods have demonstrated improved computational efficiency, they do not fully capture long-range spatial dependencies. Building upon efficient allention mechanisms, various architectures have emerged, combining existing mechanisms introducing novel attention methods to learn
global context. Nonetheless, these approaches, including (Li ct al. 2023b Zhang et al. 2022
Chen et al.
2022b Zhao et al. 2023), still require significant. computational resources.
this paper, we address the substantial computational overhead associated with learning global dependen- cies. We propose CascadedGaze Network (or CGNet), a fully convolutional encoder-decoder based restora- tion architecture, which uses Global Context Extractor (GCE) module to effectively capture the global context without relying self-attention mechanism, thus achieving both state-of-the-art performance and computational efficiency simultaneously in image restoration tasks. The name "CascadedGaze" reflects the cascading convolutional layers within the GCE. CGNett draws inspiration from recent work Metaformer (Yu et al. 2022) which challenges the prevailing belief that attention- eatention-based token mixer modules are essential for the competence of Transformers. Metaformer demonstrated that these attention- modules can be replaced with simpler components while maintaining impressive performance.
We empirically demonstrate the efficacy of CGNet and the GCE when applied to image restoration tasks. We observe competitive performance range of benchmark datascts while maintaining lower computational complexity and run-time compared to previous methods (Figure) D. In real image denoising (SIDD dataset), our method pushes the performance boundary further by surpassing the previous best-reported results. Further, on synthetic image denoising and single image motion deblurring (GoPro dataset), our method matches the state-of-the-art performance whilst being lower on MACs (G). These results emphasize the effectiveness of the proposed approach across various restoration tasks.
2 Related Work
The problem of image restoration well-studied computer vision literature (Fattal 2007 HcK SUNJ.
2011 Kopf et al.] 20088 Michaeli & Irani 2013). with competitions and challenges organized around designing methods across various restoration domains [Ignatov & Timofte 2019 Abdelhamed al.
2019
2020
2

---

Published in Transactions on Machine Learning Research (05/2024)
et al. 2023c). In recent times, learnable neural network based approaches outperform the more traditional restoration methods (Chen et al., 2022a; Zhang et al. 2023 Chen et. al. 2021 Zamir et al. 2021) even without any prior assumptions on the degradation process. Since these learnable approaches are data- driven, the availability of large-scale benchmark datasets allows these methods to estimate the distribution of degraded images empirically. This gain in performance afforded by several stacked convolutional layers that downsample and upsample the feature maps throughout the network. Furthermore, most of these networks are constructed in U-Net (Ronneberger al.) 2015) fashion, where stacked convolutional layers form U-shaped architecture with ssip-connections providing the necessary signal over longer range.
Transformers in Restoration Transformers have seen significant surge in their usage across the suite of computer vision lasks, including image recognition, segmentation, and object detection (Dosovitskiy et al.
2020 Ramachandran et al. 2019 Touvron et al., 2021 Yuan et al. 2021 Liu et al. 2021 Carion et al. [020]] albeit they were originally designed for natural language tasks (Vaswani et al. 2017). Vision Transformers
decompose images into sequences of patches and learn their relationships, demonstrating remarkable capa- bilities to handle long-range dependencies and adapt to diverse input content relying solely on self-attention to learn input and output representations. They have also been applied to low-level vision tasks like super- resolution, image colorization, denoising, and deraining (Zamir et al. 2022. Wang et al. 2022b Liang et al.
2021 Tu et al. 2022 Li et al.) 2023b Zhao et al. 2023). Unlike high-level lasks, pixel-level challenges necessitate manipulating individual pixels or small pixel groups in an image to enhance or restore specific details. Although these architectures can learn long-term dependencies between sequences, the computa- tional intractability hinders their realization and adoption in essourc-constrriing applications (Han et al, 2022
Lin et al. 2022). Specifically, the complexity increases quadratically with an increase in the input size.
Efficient Transformers Recent approaches seek alternative stralegies that reduce complexity while en- suring the generation of high-resolution outputs (Liu et al., 2022a Hatamizadeh et al. 2023 Liu et al,, 2022c Tang ct al. 2022). One such approach is locality-constrained self-attention in Swin Transformer de- sign LLuu et al. 2021). However, since self-attention is applied locally, the context aggregation is restricted to local neighborhoods. Some methods like CAT (Chen et al., 2022b) try to address the locality issue by using rectangle-windor self-attention which utilizes horizontal and vertical gectangle-windor attention to expand the attention area. A recent work, ART (Zhang et al. 2022), focused on combining sparse and dense attention, wherein the sparse attention module provides wider receptive field and dense attention functions in a more local neighborhood. Low-rank factorization and approximation methods are two other efficient techniques employed to reduce the computational complexity of self-attention in Transformers (Wang et al., 2020 Xiong et al. 2021 Lu et al. 2021 Ma et al. 2021). However, these methods can lead to loss of information, are sensitive to hyper-parae pareepaaaametess, and are potentially task-dependent.
Fully Convolutional Methods in Restoration Prior to the surge of Transformers, restoration hod utilized convolutional neural networks in their design [Tu et al. 2022 Zamir et al. 2021 Zhang et al. 2020b) Zamir et al. 2020). HINet (Chen et al.] 2021), multi-stage convolutional method, introduced the half- instance normalization block for image restoration. This was opposed to the batch normalization given the high variance between patches of images, and difference in training and testing settings, a normal practice in low-vision tasks such as restoration. SPAIR (Purohit et al.] [202]] designed distortion-guided networks consisting of two main components: network to identify the degraded pixels, and a restoration network to restore the degraded pixels. Building on Restormer's (Zamir et al., 2022) computational savings by introducing channel attention instead of spatial attention and prioritizing simplicity in design, NAFNet (Chhen] lt al. 2022a proposed a simplified version of channel attention, achieving state-of-the-art performance while being much more computationally efficient. For a more detailed survey on deep learning based restoration mothods, we refer the reader to the recent survey work (Su et al. 2022a
3 Methodology
We aim to develop a module capable of efficiently learning local and global information from the input data. To this end, we propose a cascaded fully convolutional module that progressively captures this information. It serves a low-cost. alternative for the attention mechanism. We further introduce the Range Fuser module
3

---

Published in Transactions on Machine Learning Research (05/2024)
[a] CascadedGaze Network (CGNet)
(b) CascadedGaze Block
[c) Global Context Extractor
(d) Range Fuser
Input
Output
LayerNorm
k₃ R
Upsample
3x3, comv
Head
1 1, point-cony
Fa
F
point comv
Concatenate
GELU
depth-conv
Encoder
Decoder
GELU
Residual Connection(s)
Nr &z point conv
Channel
Residual Connection(s)
Reduction
GCE Module
GELU
Encoder
Decoder
Legend
Range Fuser
Kg R3
k1xk1, depth-conv,
1 point-conv
stride=k1
Middle
k2xk2, depth-conv, stride=k2
LayerNorm
GELU
k3xk3, depth-conv, stride=k3
ChanneiMLP
Element-wise
Addition
Figure 2: Architecture Diagram. (a) Illustration of the overall architecture of CascadedGaze network (CGNet). Each encoder layer comprises N₂ CascadedGaze blocks. (b) The CascadedGaze blocks are composed of (c) GCE module and (d) Range Fuser. GCE Module has three depthwise convolutions, followed by pointwise convolutions and GELU.
in order to aggregate the learned local and global context. Both of these modules are coupled with the restoration architecture, which we refer to as CascadedGaze Network (CGNet). In this section, we discuss the overall architecture the proposed approach, followed by the two proposed modules, namely (a) Global Context Extractor (GCE) and (b) Range Fuser. Finally, we go through the details of construction steps keep the architectural construction computationally tractable and efficient.
3.1 Overall Pipeline
We adopt the widely acknowledged U-shaped Net architecture, composed of several encoder-decoder blocks, which has emerged as standard in image restoration tasks (Elad et al.) 2023). We follow the supervised setting wherein the dataset D realized by pairs of degraded and ground-truth (degradation- free) images D = (ff. (f,, ....... . (L, IN) where n is the total number of images, while I, and I denote the ith degraded and ground-truth images respectively.
W
W
ky,b
Tn Ryyk
Local
Consider degraded input image xxx33 where H, and W denote the height, and width re- spectively (or spatial dimension of the image) and denotes the number of channels. Input Image is first fed the convolutional layer that transforms the image into a feature map Fu #xCC This feature map then passes through four encoder-decoder stages. At each encoder stage, the input resolution is halved, and the number of channels is doubled. The
Figure 3: GCE module. We visualize the depthwise separable convolution layers to elucidate the capturing of context at different levels. The spatial range of each convolution is depicted in the input feature block with their corresponding colors.

---

Published in Transactions on Machine Learning Research (05/2024)
spatial dimension of the feature map is at the lowest in the middle block. Each encoder block is composed of Ngx CascadedGaze (CG) blocks. Since the U-Net structure is symmetric, each decoder block operates on input from the previous block and its corresponding encoder block through a skip connection. In each decoder stage, a pixel shuffling operation progressively restores the fcature map's original resolution. The output from the last decoder block, Fa: then fed through the head of the net before the restored image In E xx is output. We defer the reader to Figure for visualization of the entire architecture.
3.2 Global Context Extractor (GCE) Module
We draw insights from the work Metaformer (Yu et al. 2022 and find that it is possible to gain competitive performance by retaining the core structure of the Transformerr but replacing the self-attention mechanism with a more efficient alternative. We achieve the aforementioned goal by using convolutional layers with small kernel sizes the building block of our Global Context Extractor (GCE) module to extract and aggregate features from a large area of the input feature map.
Each GCE module is composed of up to three convolution layers denoted by 4, 12, and 13, respectively. For every convolution inside the GCE module, we set the stride to be equal to kernel size at each layer resulting in non-overlapping patches and subsequent reduction in the spatial dimension. The output spatial resolution of any convolution layer can be derived from Nout mis +2p k + 1], where p denotes padding, 8 denotes stride, and denotes kernel size. In GCE, we set k, and p O, and re-write the above formula as nout = [
Consider the input feature map Fj E H that is fed to the i-th encoder. Let GO denote the j-th GCE module in i-th encoder. At the first convolution layer, 4, has a kernel size of k1, and will aggregate spatial features from a k x k neighborhood of the input feature map. Let the output of I, be denoted by A loral then we can write it formally as
A local G21l1(F)) E E x
(1)
where ky convolution layer, 12, with a kernel size of k2, will aggregate information from K2 x k2 patches of summary tokens from the previous layer. Each of these summary tokens represents a k1 x ly area of the original imput feature map, so the output of the second convolution can be described as aggregated information from a ki x kikę neighborhood of the input feature map. If the kernel size of the last convolution layer, la is ka, can write similar formal construction for l2 and la-
H. and W. and C' denote the spatial dimension of the output feature map. Similarly, the second No
# X 4n x62 A middle = (A local E x22
W XC
(2)
A glohal G]]la(( middle) E R H x xxggg XC
w
(3)
where A local muiddle and Aliobal denote local, middle, and global context, respectively.
Comparison to Sele-Attention Self-attention in VIT (Doovvisskiy et al. 2020) functions patches of images generated by splitting the image into fixed-sized pieces. Each patch, or its linear projection, is coupled with 1D positional embedding indicating its position in the sequence. Self-atlention then computes the attention score by attending to each sub-sequence (or patch) within the sequence (or image). In contrast, each subsequent layer in GCE operates on the patches generated by the layer proceding it. In Eq. 2 A middle
operates on the patches generated by the preceding layer, A local, Similarly, Aglobal operates on the patches generated by A middle drawing parallels to self-attention.
3.3 Range Fuser
The extracted local and global features have different spatial sizes. To enable proper concatenation, we employ upsampling with nearost-ncighbor interpolation to match the spatial dimensions. This is non- learnable layer and hence does not affect the model size. We concatenate the upsampled feature maps along the channel dimension and obtain features with the original spatial dimensions but with inflated channels.
5

---

Published in Transactions on Machine Learning Research (05/2024)
We recognize the varying importance of channels and draw inspiration from (Chen et al. 2022a by employing Simple Channel Attention (SCA) ) to re- re-weight each channel. This approach allows for accentuating important channels while suppressing less informative ones, resulting in more refined and focused representation of the aggregated features.
Further, we employ a single pointwise convolution to streamline the representation further and reduce the channel dimension the input size. This yields a compact, refined input representation that seamlessly incorporates local and global information. Combining SCA and pointwise convolution ensures that model retains the essential details while suppressing noise and improving performance and robustness.
3.4 On Computationally Efficient Construction
To further reduce the computational overhead, we merge similar channels by element-wise summation before feeding them to GCE. We explore two channel-merging options in this regard.
DynamicMerge: We employ a dynamic channel merging technique by leveraging the token merging approach introduced by (Bolya et al. 2023) for merging similar tokens within Transformers. This adaptation relies on a selected similarity metric, such as Mean Absolute Error or correlation, to assess the channel similarity and facilitate the merging process. The similarity may be calculated among the channels themselves or the kernel weights of the depthwisc convolution layer corresponding each channel.
StaticMerge: As opposed to a dynamic merging strategy, we also explore statically merging based on fixed index. We achieve this by merging even channels with odd channels.
We ablate cach method and find that static merging of channels (StaticMerge) performs the best our case. This is preferable given that there constant computational cost to the operation; more discussion on the ablation experiments to follow.
CascadedGaze Block
3.5 Comparison to NAF Block
LayerNorm
NAF Block
Layertorm
We compare the proposed CacadedGaze (CG) block and draw par- allels with the previously introduced NAF block in (Chen et al. 2022a). The key difference lies in the CG block's Global Con- text Extractor (GCE), which utilizes cascading receptive fields for a compact understanding of input fcatures at both local and global scales. Unlike the NAF block that relies on Simple Channel Atten- tion (SCA) to re-weight channel-wise feature maps, CG block learns local and global compact aggregated representation without the computational overhead of Transformer-lik architectures.
sxd,
point-come 3x3,
depth-corv
GELU
depth comm
Channel Reduction
SimpleGate
GCT Module
SCA
Sxd, comM
Range
Layer/orm
4 Results
LayerNorm
ChansettLP
We evaluate CGNet on benchmark datasets for three image restora- tion tasks (a) real image denoising, (b) Gaussian image denoising, and (c) single image motion deblurring. We discuss these restora- tion tasks, and datasets, and then describe our experimental setup, hyperparameters, and training protocol, and results.
ChannetMLP
Figure Block Comparison. Ca- cadedGaze Block and NAF Block com- parison diagram.
4.1 Datasets
For image denoising, we train our models on both synthetic benchmark datasets (Gaussian image denois- ing) and the real-world noise dataset (real image denoising). The Smartphone Image Denoising Datasct (SIDD) (Abdelhamed al. 2018) is a real-world noise dataset composed of images captured from different smartphones under various lighting and ISO conditions, inducing variety of noise levels in the images. The

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: Scores on Gaussian Image Denoising task. We report PSNR scores along with MACs (G) and inference time (milliseconds) (denoted Inf.) calculated for an image size of 512 512 one a single NVIDIA Tesla v100 PCIe 32 GB GPU. Our method is lower in MACs and is faster than previous methods while remaining comparable, if not better. Notably, our model outperforms Restormer, which has the closest MACs and inference time to across all test datasels except for McMaster. The best results are highlighted in red, while the second bests are in blue. We do not highlight ART (Zhang et al. 2022 due to significant differences in MACs.
CHSS168
Kodak24
McMaster
Urban100
Method
MACs + (C)
Inf. 1
2001
1909
2011
20155
15
2021
2991
1850
34.42
31.78
28.56
35.34
32.89
29.79
35.61
33.20
30.22
35.13
32.90
29.82
Restormer
564
350
34.40
31.79
28.60
35.47
33.04
30.01
35.61
33.34
30.30
35.13
32.96
30.02
2022
GRLSS
975
34.36
35.32
32.88
20.77
35.24
333.07
2023b
Alt1
22022
34.46
35.30
32.90
20.87
30.31
35.20
33.
CODDE
221233
180
600
34.33
31.69
28.47
35.32
32.88
29.82
35.33
33.11
30.03
CCNNet
(Ours)
444
215
34..4
31.79
28.60
35.52
33.07
30.06
35.58
33.28
30.22
35.18
32.98
30.07
Noisy Input
Restormer / 28.02
CGNet/ / 30.98 / 28.12
Figure 5: Qualitative Comparison on Gaussian Denoising. Visual results on Gaussian image denoising
on Kodak24 (Franzen 1999) dataset. We compare with Restormer ZZamir et al. 2022), the best method in the literature on the dataset. Our method, CGNet, restores finer details and pleasing outputs. The corresponding PSNR scores for each image are mentioned the top of the figure.
synthetic benchmark datasets are generated with additive white Gaussian noise on BSD68 MMrtinn et al. 2001), Urban100 (Huang et al. 2015), Kodak24 (Franzen 1999) and McMaster (Zhang et al. 2011). For image motion deblurring, we employ the GoPro dataset (Nah al. 2017) the training data. The GoPro dataset contains dynamic motion blurred scenes captured from consumer-grade camera. In all cases, we
adopt the standard data preprocessing pipeline following (Chen et al. 2021 2022a Zamir et al. 2022

---

Published in Transactions on Machine Learning Research (05/2024)
Noisy Input
Restormer
NAFNet
Ours
35.1
35.08
35.12
ubraty for exar lirrrr for esan CS library for exan
dd ta the Co ROP e the CD-ROM e the CD-ROM d the CD-ROM
as
as
as
colos and lightlng
coler and lighting coler and lighting
color and lighting
screen to world
screen-to-world
screen-to-world
screen-to-world
world- to screen
world-to-screen
world-to-screen
worid-to-screen
35.03
35.91
36.24
Figure 6: Qualitative Comparison on Real Image Denoising. Denoising results validation images from SIDD dataset (Abdelhamed et al. 2018). CGNet (Ours) restores visually pleasing images in variely of scenes and objects; additionally, the PSNR scores quantitatively confirm CGNet's performance boost in these images.
Table 2: Scores on Image Denoising on SIDD datasct. CGNet achioves statc-of-tho-art results on the SIDD dataset while being faster with lower MACs. The inference lime is calculated on a single NVIDIA Tesla v100 PCTe 32 GB GPU, and the MACs calculated for an image size of 256 256 3. Note that we do not report inference time for methods scoring much lower PSNR on the task. The best results are highlighted in red, while the second best in blue.
Smartphone Image Denoising Dataset (SIDD)
MACs
Inference Time (ms)
Method
PSNR 1
SSIM
(G)
MPRNet
588
39.17
0.958
Zamir et. al.
2021
CycleISP
189.5
39.52
0.957
Zamir et al.
2020
H1Net
170.7
39.99
0.960
Chen al.
2021
MAX
169.5
39.96
0.958
Tu ct
2022
CA1
135.7
390
40.05
0.960
Chen et al.
2022b
Restormer
141.0
102
40.02
0.960
Zamir et al.
2022
NAFNet
63.6
53
40.30
0.962
Chen et al.
20/22a
CGNet
62.1
52
40.39
0.964
(Ours)
4.2
Experimental Setup
CGNet for the denoising and deblurring tasks comprises sequence of four encoder blocks, one middle block, followed by a sequence of four decoder blocks, with skip connections between corresponding encoder /ecoder blocks. To reduce computational expenses, we strategically use CGE blocks in the encoder and simple NAF

---

Published in Transactions on Machine Learning Research (05/2024)
Reference
Blurred
Restormer
NAFNet
CGNet
PSNR
29.9
32.39
32.63
PSNR t
28.12
29.82
29.99
PSNR t
33.96
35.25
35.52
Figure Qualitative Comparison on Image Deblurring. Visual results on Single Image Motion Deblurring sample images from validation set of GoPro dataset Nah et al. (2017). CGNet (ours) results are much more closely aligned with the ground truth in terms of reconstruction, and are sharper.
blocks (Chen et al. 2022a in other places which is discussed in detail in the ablation section. Below, we elaborate on the specifications of each model.
Real Image Denoising The encoder comprises 2, 2, 4, and 6 CascadedGaze blocks, respectively. The rest of the network is composed of NAF blocks with 10 at the middle layer and 2, 2, 2 and 2 for the four decoder blocks, respectively. We set the width of the network to 60. The restored image is taken from the head of the network, which is a convolutional layer applied to the output of the last decoder.
Gaussian Image Denoising. For a fair comparison with previous methods in the literature, we increase the size of our network specifically, increasing the number of blocks and the width. The encoder has 4, 4, 6, and blocks, the middle layer has 10 blocks at each stage, and the decoder has 2, 2, 2, and 4 blocks. We set the width of the network to 70.
Image Deblurring. The first three encoder blocks have Cascaded block each, while the fourth encoder comprises 2 CascadedGaze blocks followed by 25 NAF blocks. The remaining middle and decoder blocks also comprise 1 NAFNet block cach. We set the width of the network to 62 in this case. For the deblurring task, we follow the architectural modifications proposed in the work (Liu et al. 2022b). Unlike the single output in the denoising model, we modify the head of the network to accommodate for K multiple outputs allowing the network to output multiple feasible solutions. We set the value K = 4 for all of these experiments. Since the models are trained on 256 256 patch sizes, testing on larger sizes degrades performance; therefore, we finetune the model on 384 x 384 patches for more epochs following (Zamir et al., 2022 additionally, we use TLC proposed by (Cuu et al. 2022) for inference image deblurring task.
Shared Configuration. We train the models in all tasks for 400K iterations, with AdamW as the opti- mizer (31 0.9, 82 = 0.9), and minimize the negative PSNR loss function (i.e., maximize the PSNR). We use cosine anncaling scheduler that starts with the lcarning rate of 1 and decays to 1e-7 throughout learning. All of our models are implemented in the Torch library, trained on 8 NVIDIA Tesla v100 PCTe 32 GB GPUs For inference, we utilize a single GPU. During training for real denoising and motion deblurring experiments, we set the image patch size to 256 256. For Gaussian denoising, we follow (Zamir et al. (022) progressive training configuration and start with the patch size of 160 and increase it to 192, 256, 320, and 384 during training. The reported results averaged over three runs. We compute Peak Signal-to-Noise
9

---

Published in Transactions on Machine Learning Research (05/2024)
Table 3: Scores on Singe Image Motion Deblurring on GoPro dataset. Our method scores the highest PSNR on the task, achieving sate-of-theeart results. The best results are highlighted in red, while the second bests are in blue.
GoPro Motion Deblurring
SRN Tao
DBGAN Zhang al. 2020ml
SPAIR Purohis
MPRNet
HINet
HI-Diff
Method
2021
Zamir et
2021
et
2024
PSNR
31.10
32.00
32.80
33.33
SSIM
0.934
0.942
0.953
0.959
0.959
0.964
NAFNet
MAXIM
Restormer
NeFNet Chen 2022a
D
CGNet (Ours)
Method
MH-C
Tu
2022
Zamir 2022
Xia al.
2023
Liu et al.
2022b
PSNR
32.92
33.71
SSIM
0.961
0.967
0.967
Ratio (PSNR) metric and Structural Similarity Index (SSIM) in line with the standard evaluation protocol followed by literature on image restoration
(Chen et al. 2022a Purohit et al. 2021 Zamir et al. 2022)
4.3 Results Discussion
Real Image Denoising. We perform experiments on the Smartphone Image Denoising Dataset eepee
(SIDD) (Abdelhamed ct al. 2018) part of the real-world denoising experiments. Table 2 compares CGNet with previously published methods in the literature. Our proposed approach archives 0.09 dB gain over the previous best method NAFNet (Zamir et al., 2022). We provide visual results on sample images from the SIDD dataset in Figure G our method restores results more faithfully and closer to the ground truth.
Gaussian Image Denoising. We present results of CGNet on Gaussian image denoising four datasets with three different noise levels (o = 15, 25, 50) in Table | As the spatial size of images in the test dalasets is larger than 512, the reported MACs (G) values are calculated for an image size of 512 x 512. Our method is comparable to current stato-of-the-art mothods, pushing the boundary on a. few datasets while being significantly faster in inference time, and lower on MACs (G). We beat Restormer (Zamir al., 2022 in all datasets except McMaster. Even though we have reported ART (Zhang et al. 2022). we note that CGNet is not comparable as ART's MACs (G) is 10x larger. Also, is computationally intractable on a limited budget, given that observe an Out of Memory (OOM) error when running inference on ART. We present few visual results in Figure on the Kodak24 dataset.
Single Image Motion Deblurring. Table 3 lists the results of our approach on single image motion deblurring task the GoPro dataset (Nah ct al.) 2017). Our model gains +0.06 and +0.02 dB in PSNR compared to NAFNet and NAFNet multi-head methods, showing the effectiveness of our method in different restoration tasks. Furthermore, visual results on images from the GoPro dataset are also provided in Figure]
4.4 Visualizing GCE Module
We visualize the GCE module, mainly looking at how local and global layers learn input context. Figure plots the activations of A locaa and Aglobal recall Eq. [ and Eq. 3 from G i.e. the first GCE module of second encoder block. The layer operating the local context learns structure local to foreground objects occupying considerable pixel space in the image (for example, the cars), while the global context is much broader with considerably activated objects present even in the background (for example, trees and sidewalk).. For each image, the local context. acts like an edge detector learning low-level features local to the objects. Notice how objects much further away in the distance are void of sharp edges. On the other hand, the global context learns higher abstractions of the image than low-level fcatures. Such analysis of neuron activations understand context is well explored in interpretability literature, both language processing (Sajjad et al. 2022), and vision (Zeiler & Fergus 2014
g
10

---

Published in Transactions on Machine Learning Research (05/2024)
Input
Local Context
Global Context
Figure 8: GCE Module Visualized. Visualization of the local and global context taken from the outputs of the Global Context Extractor (GCE) module. Results visualized on images taken from validation set of GoPro dataset (Nah et al., 2017). The local context. is adept at learning local structure and features edges, whereas the global context extracting high-level features and shapes.
4.5 Ablation Study
We ablate the proposed CaseadedGazeeNet to understand what components necessitate efficiency and per- formance gains. All experiments are conducted on real-image denoising task using a smaller variant of our model with a singular block at each level of the architecture, and a width of 8, Our smaller models operate within computational budget of approximately 0.5 MACs (G), and are trained for total of 200K itcrations while the remaining settings are the same as those of the main model. In all the cases, the combinations we adopt for the CascadedGazeNet are in bold.
Channel Merging Method. We employ a merging algorithm before using the GCE module to reduce its computational overhead further. As shown in Table our ablation study showed that merging channels basod on fixed index, referred to as StaticMerge, during both training and inference outperforms dynamically merging similar channels (referred to DynamicMerge). The simplicity of the method makes it easier for the model to learn, as it does not have to adapt to a different combination of channels for each batch of data.
Kernel Sizes. The choice of kernel sizes significantly impacts the performance of a CNN. In general, it is better to use smaller kernel sizes at the beginning of the GCE and then use larger kernel sizes for later convolutional layers. Smaller kernel sizes allow the GCE to extract more detailed information from the imput image. Then, larger kernels at the end of the GCE aggregate these details to capture more global information. We follow this intuition and ablate two kernel choices for each layer in the GCE module. We aim to understand whether smaller kernel to larger kernel sizes better for design or vice-versa. Table 5 shows our experiments' results on kernel size choices. Our results show that the best choice is to utilize a smaller larger kernel size design, where the initial layer extracts local, while the last layer learns global context.
Global Context Extractor Module Placement. The GCE module is a resource-intensive component, and incorporating it extensively throughout the network is impractical. This stems from the trado-off between performance enhancements and computational costs, necessitating careful equilibrium. Intuitively, since GCE extracts both local and global information, is best suited for the encoder part. This helps the
11

---

Published in Transactions on Machine Learning Research (05/2024)
Table 5: Kernel Size Study. Comparison of different kernel sizes for the GCE module in the architecture. We ablate two combinations to determine the optimal employment of larger kernels at the initial and final stages.
Table Comparison of Differ- ent Convolutional Layers. We mainly ablate the order of point- wise (PW) and depthwise convolu- tions (DW) and compare these com-
Table GCE Block Place Study. We ablate the placement of GCE blocks throughout the net- work. Enc refers to Encoder blocks, while Mid refers to Middle blocks, and Dec refers to Decoder blocks.
binations with the standard convo-
lutional layer.
GCE Location PENR
MAC₂ (G)
Parama (M)
X
39.33
30.32
59.32
0.520
0.506
0:400
0.446
0.848
0,406
0.737
Kernel
3.3. 3] 39.25
3,3, 6
39.32
PSNR MACs (G)
0.442
0.446
0.406
0.408
Params
(M)
Type
PW
Standard
Convolution
DW+PW
39.32
39.32
39.33
PSNR
0.480
0.464
0.448
MACs (G)
0.498
0.406
(M)
Params
model utilize information to capture fine-grained non-corrupted information. However, we ablate the GCE placement, cumulatively increasing the GCE blocks throughout the network. The results summarized in Table 4 Our experiments back up the idea that placing the GCE module in the encoder blocks yields the best balance between performance and computational efficiency.
Channel Expansion before GCE. We investigate the effects of channel expansion at the beginning of the CascadedGaze block using a point-wise convolution operation. Specifically, we try expanding by x2, keeping it as is, and expanding by x2 while utilizing channel merging (StaticMerge) to reduce the number of channels by half. In agreement with intuition, we find that expanding the channels by x2, and performing roduction by channel merging (StaticMergo) works the best while maintaining a balance between the MACs (G) and PSNR score. The complete analysis is shown in Table S
Convolutional Layer Type. The type of convolutional layers used in the GCE module significantly impacts our model's size and computational efficiency. Therefore, we ablate the choice by considering three options: standard convolution, pointwise convolution depthwise convolution (PW+DW), and depthwise convolution + pointwisc convolution (DW+PW) As shown in Table using depthwise convolution to reduce the spatial dimension and then applying the pointwise convolution significantly makes our model smaller while maintaining competitive performance.
Table 8: Comparison of Channel Expansion. We Table 7: Comparison of Channel Merging. We compare channel expansion before passing to GCE report comparison two channel merging methods. module. When channels are reduced to half ç, wo DynamicMorge has different strategies, whereas Stat- use the StaticMerge technique to achieve the desired icMerge is a fixed merging method, as discussed before. reduction.
Method
Strategy
PSNR
Time (mx)
Inference
Expansion
Channels PSNR
MACs (G)
Params
Statichleree
Factor
(M)
Chaancl Cosine Similarity
x2
2 C
39.33
0.507
0.569
DynamicMerge
Kernal Cosene Simalarity
14.5
Kernal MAR
11.
x
39.28
0. 0.401
0.355
X2
(2 x C)/2
39.32
0.446
0.406
5
Conclusion
We introduced method to learn the local and global context for image restoration tasks in computa- tionally efficient manner. Inspired by the self-attention mechanism in Transformers, we proposed a module termed Global Context. Extractor (GCE), for fully convolutional architectures. We constructed a restoration architecture, termed CascadedGaze Network (CGNet), utilizing the introduced GCE module and empirically verified the effectiveness in terms of overall performance and computational tractability. We hope that our work will spur interest in efficient architecture construction to learn the global context for various low-level vision tasks.
12

---

Published in Transactions on Machine Learning Research (05/2024)
References
Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smart- phone cameras. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp.
1692-1700, 2018.
Abdehrahman Abdelhamed, Radu Timofte, and Michael S Brown. Ntire 2019 challenge on real image denoising: Methods and results. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
AbdeIrahman Abdelhamed, Mahmoud Afifi, Radu Timofte, and Michael S Brown. Ntire 2020 challenge on real image denoising: Dataset, methods and results. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops, pp. 496 497, 2020.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster, 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. Springer, 2020.
Meng Chang, Qi Li, Huajun Feng, and Zhihai Xu. Spatial-adaptive network for single image denoising. In Computer Vision Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 28, 2020, Proceed- ings, Part XXX 16, pp. 171-187. Springer, 2020.
Kang Chen and Yuanjie Liu. Efficient image deblurring networks based on diffusion models. arXiv preprint 00l::400005007 2024.
Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. Hinet: Half instance normalization network for image restoration. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 182-192, 2021.
Liangyu Chen, Xiaojic Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In European Conference on Compuler Vision, pp. 17 33. Springer, 2022a.
Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xin Yuan, et al. Cross aggregation transformer for image restoration. Advances in Neural Information Processing Systems, 35:25478- 254400, 2022b.
Zheng Chen, Yulun Zhang, Ding Liu, Jinjin Gu, Linghe Kong, Xin Yuan, et al. Hicrarchical intogration diffusion model for realistic image deblurring. Advances in Neural Information Processing Systems, 36, 2024.
Shen Cheng, Yuzhi Wang, Haibin Huang, Donghao Liu, Haoqiang Fan, and Shuaicheng Liu. Nbnet: Noise basis learning for image denoising with subspace projection. In Proceedings of the TEEE/CVE conference on computer vision and pottern recognition, pp. 4896 4906, 2021.
Xiaojie Chu, Liangyu Chen, Chengpeng Chen, and Xin Lu. Improving image restoration by revisiting global information aggregation. In European Conference on Computer Vision, pp. 53-71. Springer, 2022.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint ar arXiv:2209.1 14687, 2022.
Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusson for image restoration. arXio preprint 11445 2023.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth Transformers for image recognition scale. arXiv preprint arXiv:2010. 1...11199 11929, 2020.
16x16 words:
Michael Elad, Bahjat Kawar, and Gregory Vaksman. Image denoising: The deep learning revolution and beyond-a survey paper-. Xix preprint -xXx:2301.03300 2023.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Chi-Mao Fan, Tsung-Jung Liu, Kuan-Hsien Liu, and -sinng Chiu. Selective residual m-net for real image denoising. In 2022 30th Europpnn, Signal Processing Conference (EUSIPCO), pp. 469-473. TEEE, 2022.
Raanan Fattal. Image upsampling via imposed edge statistics. ACM SIGGRAPH 2007 papers, pp. 95-cs. 2007.
Rich Franzen. Kodak lossless true color image suite. source: http://rOk. (graphics/kodaa) 4(2):9, 1999.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1):87-110, 2022.
Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, Jose M. Alvarez, Jan Kautz, and Pavlo Molchanov. Fastervit: Fast vision transformers with hicrarchical attention, 2023.
M HeK and TANG X O SUNJ. Single image haze removal using dark channel prior. IEEE Transactionson Pattern Analysis and Machine Intelligence, (311))23341, 2011.
Jia-Bin Huang, Abhishek Singh, and Narendra Abuja. Single image super-resolution from transformed exle-exemplars In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 5197-5206, 2015.
Andrey Ignatov and Radu Timofte. Ntire 2019 challenge image enhancement: Methods and results. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593- 35:23593-23606, 2022.
Johannes Kopf, Boris Neubert, Billy Chen, Michael Cohen, Daniel Cohen- -rr, Oliver Deussen, Matt Uyt- tendaele, and Dani Lischinski. Deep photo: Model-based photograph enhancement and viewing, ACM transactions on graphics (TOG), (3):11-00 2008.
Xin Li, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng, Xinchao Wang, and Zhibo Chen. Diffusion models for image restoration and enhancement-a comprehensive survey. arXiv preprint ar wrXiv:2308.09384 2023a.
Yawei Li, Yuchen Fan, Xiaoyu Xiang, Denis Demandolx, Rakesh Ranjan, Radu Timofte, and Lue Van Gool. Efficient and explicit modelling of image hicrarchics for image restoration. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pullern Recognition, pp. 18278 18289, 2023b.
Yawei Li, Yulun Zhang, Radu Timofte, Lue Van Gool, Zhijun Tu, Kunpeng Du, Hailing Wang, Hanting Chen, Wei Li, Xiaofei Wang, et al. Ntire 2023 challenge on image denoising: Methods and results. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1904 1920, 2023c.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the TEFE/ /CV international conference compuler 2021.
vision, pp.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. AT Open, 2022.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pechenizkiy, Decebal Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv preprinl arXic:2207.03022 2022a.
Sidun Liu, Peng Qiao, and Yong Dou. Multi-outputs is all you need for deblur. orXiv preprint arXiv:220..1 13029, 2022b.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 10012 10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A couvnel for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976 11986, 2022c.
Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. Soft: Softmax-free transformer with linear complexity. Advances in Neural Information Processing Systems, 34:21297-21309 2021.
Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. Luna: Linear unified nested attention. Advances in Neural Information Processing Systems, 34:2441-2455 2021.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithuns and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pp. 416- 423. IEFE, 2001.
Tomer Michaeli and Michal Irani. Nouparametric blind super-resolution. In Proceedings of the IEEE Inter- national Conference on Computer Vision, pp. 945-952, 2013.
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, July 2017.
Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1586-1595, 2017.
Kuldeep Purohit, Maitreya Suin, AN Rajagopalan, and Vishmu Naresh Boddeti. Spatially-adaptive image restoration using distortion-guidee guided networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2309-220000 2019, 2021.
Prajit. Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand- alonc self-attention in vision models. Advances in neural information processing systems, 32, 2019.
Jacsung Rim, Hacyun Lee, Jucheol Won, and Sunghyun Cho. Real-world blur dataset for learning and bench- marking deblurring algorithms. In Computer Eision-ECCH 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXV 16, pp. 184 201. Springer, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer- Assisted Intervention MICCAI 2015: 18th International Conference, Munách, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241, 2015.
Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation deep nlp models: A survey. Transactions of the Association for Computational Linguistics, 10:1285-1303, 2022.
Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human- aware motion deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5572-5581, 2019.
Jingwen Su, Boyan Xu, and Hujun Yin. A survey of deep learning approaches to image restoration. Neuro- computing, 487:46 65, 2022a.
Jingwen Su, Boyan Xu, and Hujun Yin. A survey of deep learning approaches to image restoration, Neuro- computing, 487:46-60 65, 2022b.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Chao Xu, and Yunhe Wang. Ghostnetv2: enhance cheap operation with long-range attention. Advances in Neural Information Processing Systems, 2022.
Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. recureett network for deep image deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8174- 8182, 2018.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training dala-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10333 PMLR, 2021.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mlp for image processing. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5769 5780, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. aiviv preprinl or arXiv:2006.00 04 768, 2020.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint Xix:22221 00490, 2022a.
Zhendong Wang, Xiaodong Cun, Jiammin Bao, Wengang Zhou, Jianzhunng Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17683 17693, 2022b.
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool. Diffir: Efficient diffusion model for image restoration. arXiv preprinl ar urXiv:2303.004/ 09472, 2023.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström- algorithm for approximating self-attention. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 141481 2021.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 10819-10829, 2022.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenct. In Proceedings of the IEEE/CVE international conference on compuler vision, pp. 558 567, 2021.
Zongsheng Yue, Qian Zhao, Lei Zhang, and Deyu Meng. Dual adversarial network: Toward real-world noise 2020: 16th European Conference, Glasgow,
removal and noise generation. In Computer UK, August 28-28 28, 2020, Proceedings, Part X 16, pp. 41-58. Springer, 2020.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Cycleisp: Real image restoration via improved data synthesis. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 2696 2705, 2020.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF conference on computer vision and poltern recognition, pp. 14821-1483) 2021.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Huung Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5728 5739, 2022.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Eision-ECVV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt I 13, pp. 818 833. Springer, 2014.
Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Accurate image restoration with attention retractable transformer. In The Fleventh International Conference on Learning Representations, 2022.
Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2737 2746, 2020a.
Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic imaging, 20(2):0 020116,, 2011.
Yi Zhang, Dasong Li, Xiaoyu Shi, Dailan He, Kangning Song, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Kbnet: Kernel basis network for image restoration, 2023.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image restoration. IEEE transactions pottern analysis and machine intelligence, 23(7))2480-2443 2020b.
Haiyu Zhao, Yuanbiao Gou, Boyun Li, Dezhong Peng. Jiancheng Lv, and Xi Peng. Comprehensive and delicate: An efficient transformer for image restoration. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14122-14132, 2023.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Table 9: Generalization Experiments. On several denoising, and deblurring datasets (synthetic and real-world), we report performance on the generalization experiments. The MACs(G) are computed on an input size of (512 512 x 3).
Generalization Experiment(s)
Real- World Denosing
Synthetic
Real- World Blur
Method
MACs (G)
Params
DND
GoPro
RealBlurR
RealBlurJ
2017
20177
22000
2020
PSNR
SSTM
PSNR
SSIM
PSNR
SSIM
PSNR
SSTM
NAPNet
[202aa
254.37G
113.96M
38.41
0.943
33.71
0.967
36.07
0.954
28.18
0.848
CGNet
39.41
0.950
33.77
0.968
36.04
0.964
28.28
0.853
248.18G
119.22M
(Ours)
(+ 1.0) dB
(+000007)
(+0.06) dB
(+0.001)
(-0.03) dB
(--)
(+0.1) dB
(+0.0005)
Table 10: Synthetic Gaussian Denoising Dataset. Experiments comparing NAFNet, and CGNet (ours) on four synthotic Gaussian denoising datasets with noise level a = 25. * We trained NAFNet given the lack of open-source official results from the authors.
Synthetic Gaussian Denoising
CBSD68
Kodak
McMaster
Urban100
0 = 25
(2001)
(1999)
(2011)
(2015)
NAFNet* *
31.75
33.01
33.26
32.83
(2022a)
CGNet
31.79
33.07
33.28
32.98
(Ours)
(+0.04) dB
(+0.06) dB
(+0.02) dB
(+0.15) dB
Appendix
A Generalization Experiments
In this section, we discuss the generalization experiments denoising and deblurring tasks. We test proposed method four new datasets and compare it with the methods in the literature. More specifically, we test our image denoising method, trained on SIDD dataset (Abdelhamed et al.) 2018), on the Darmstadt Noise Dataset (DND) (Plotz & Roth, 2017). Further, we test the single image deblurring method, trained on synthetic blur datasets GoPro (Nah et al. 2017), on another synthetic blur dataset HIDE (Shen et al. 2019), and two real-world blur datasets, RealBlurR (Rim et al. 2020) and RealBlurJ (Rim el al. 2020). Our method non-tr transformerr architecture and is comparable in parameters and MACs (G) to the previously introduced NAFNet (Chen et al. 2022a);; therefore, emphasize the comparison with the aforementioned method. We use the official models trained SIDD and GoPro, which were published by the authors T and only run inference for comparison.
A.1 Synthetic Image Denoising
To compare NAFNet on synthotic Gaussian image denoising task with noise level a = 25, we train the method since there is no official trained model made available by the authors. We increase the model capacity and size to match MACs(G) of CGNet for fair comparison. The results, PSNR scores, are reported in Table 10: our method scores higher on the metric across all four datasets.
A.2 Real Image Denoising
Darmstadt Noise Dataset (DND) PPloz & Roth 2017) consists 50 pairs of real-noise and its ground-truth images captured with several different consumer-grade cameras. The reference image is taken with the base ISO level, while the noisy image is captured with higher ISO. The dataset. is prepared following a series of procossing steps to handle camera shift alignment, exposure time adjustment, and intensity scaling.
18

---

Published in Transactions on Machine Learning Research (05/2024)
CGNet
32.48
37.04
34.48
CoNet
NAFNet
NAFNet
CGNet
Discoleraton
Discoloration
NAFNet
30.13
35.78
32.87
Figure 9: Qualitative Comparison on Real Image Denoising. Visual Results of Real-World Denoising on the DND dataset. The zoomed-in regions are provided for visualization, while the PSNR scores are written on top of the restored frames.
It common practice in image restoration literature evaluate methods trained on the SIDD dataset on the DND benchmark test since both are real-world noised datasets, albeit with different capturing devices (smartphones and consumer-grade cameras, respectively). We compare proposed CGNet method with NeFNet given that. both are non-Transformer architectures and are comparable in MACs (G). We observe that our method scores +1.0 dB higher in PSNR on the task compared to NAFNet in Table g and the restored images are artifacts-free and faithful to the ground-truth, see Figure g
A.3 Real-World Image Deblurring
On real-world image deblurring task, compare both GoPro dataset trained models on two different real- world blurry datasots: RealBlurR, and RealBlur. (Rim et al., 2020). The datascts are constructed with an acquisition system designed to capture aligned pairs of blurred and sharp images, followed by post- processing techniques to construct high-quality ground-truth images. On RealBlurJ, our proposed method (CGNet) scores higher both on PSNR and SSIM metrics, while is comparable NAFNet on the Real- Blur dataset. In conclusion, we find that CGNet generalizes better than NAFNet on unseen degradation (denoising, deblurring bebhurring-both both synthetic and real-world).
19

---

Published in Transactions on Machine Learning Research (05/2024)
Table 11: Comparison on Single Image Motion Deblurring with Diffusion Methods. We report
results on the single image motion deblurring task on the GoPro dataset Nah al. (2017) with recent
diffusion-basee based restoration models.
DiffIR
InD
HI-Diff
DvSR
DvSR-SA
Swintormer
CGNet
Methods
(2023)
(2023)
(2024)
(2022)
(2022)
(2024)
(Ours)
PSNR
33.20
31.49
33.33
31.66
33.23
33.38
33.77
SSIM
0.963
0.946
0.964
0.948
0.963
0.965
0.968
B
Restoration Problems and Diffusion Models
Diffusion models have revolutionized the generative modeling landscape, and now have started to find their adoption, although gradual, in the restoration community. In supervised restoration lasks, agreement of the restored results with the ground truth is of high importance to avoid unwanted artifacts, and hallucinated details, problem inherent naive diffusion models (Chen et al. 2024). However, more recently, diffusion models have shown competitive performance on several restoration lasks, such image deblurring, image in- in-painting, image super- resoluton, and image dehazing (Kawar et al. 2022 Wang et al. 2022a Chung al.
2022 Delbracio & Milanfar 2023). Generally, diffusion models require several steps (due to their iterative nature) to produce high-fidelity output and hence, encounter computational intractability issues. Recent
works on diffusion models for restoration, such as (Xia et al. 2023 Chen et al., 2024 Whang et al. 2022)
now consider computational efficiency to an important design decision of the architecture formalism. We
refer the reader the recent survey work on diffusion methods for image restoration (Li al. 2023a for an in-depth literature summary.
compare proposed approach with recent diffusion models based image deblurring chitectures on the GoPro dataset (Nah et al. 2017). Notably, diffusson models are known to score the lower end on distortion metrics such as PSNR, and SSIM due to the prevalence of undesired artifacts in the restored results, or misaligned generations (Chen et al. 2024
In Table 11
Whang et al. 2022
20


Reviews:
[{"review_id": "AvnmwIgurN", "reviewer": "Reviewer_w7yt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe relevant work statement is more comprehensive, and the language expression is smooth.\n2.\tA new CascadedGaze Network is proposed for image restoration tasks, and good results are obtained on PSNR and MACs.\nWeaknesses:\n1.\tThe overall innovation of the paper is limited. It only integrates U-Net, cascaded pyramid convolution, and NAFNet's Simple Channel Attention (SCA), and almost no original content exists.\n2.\tThe fonts in Figures 2, 3, and Table 1 are too small and can hardly be read on printed paper.\n3.\tThe performance improvement of the experimental part compared to NAFNet is very small, which is not enough to support the advantages of the entire paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article proposes a CascadedGaze Network (CGNet), an encoder-decoder architecture that employs Global Context Extractor (GCE) to capture global information for image restoration. Compared with the traditional convolution network, it considers the global context of the image. Compared with the traditional transformer network, it reduces the computational overhead. Judging from the experimental results, there is a certain performance improvement in RSNR and MACs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tFigure texts need to be enlarged.\n2.\tIt is necessary to propose more novel network architecture or algorithm theory rather than simple model splicing.\n3.\tMore favorable experimental results compared with NAFNet need to be added to prove the value of this work.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vphc2BX3MV", "reviewer": "Reviewer_DDmx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is very well-written and, as far as I can tell, the experimental setup follows current best practices.\nThe design choices are clearly motivated and I appreciate in particular that Section 4.4 (Figure 7) also verifies that they agree with experiments.\n\nWhile the efficiency improvement in terms of MACs is evident, I would however argue that the PSNR improvements are insignificant. In general, I think the abstract and introduction make too strong claims about being state-of-the-art, while the experimental section gives a more balanced account, e.g. _\"Our method is comparable to current state-of-the-art methods, pushing the boundary on a few datasets while being significantly faster in inference time, and lower on MACs (G).\"_\n\nRelated to these SotA claims, why is there no reference to the NTIRE 2023 Image Denoising challenge? I'm also missing an account of recent advances in diffusion models for image restoration, see e.g. this survey: https://arxiv.org/abs/2308.09388", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose two architectural modifications to make neural networks for image restoration more computationally efficient without sacrificing performance. First, a Global Context Extractor, inspired by Metaformer, that captures global information without self-attention. Second, a Range Fuser that combines channels corresponding to different scales.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Tone down the claims on state-of-the-art performance.\n2. Relate or, better, compare with recent diffusion models for image restoration.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "r3OklXChNb", "reviewer": "Reviewer_dgkX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Proposed a novel convolutional architecture that works as well as transformers but is more efficient. \n\n2. The new component of the network Global Context Extractor (GCE) is well motivated. \n\n3. The empirical result demonstrates the advantage of the proposed CGNet network.\n\nWeaknesses:\n1. The performance is very close to NAFNet, especially on the SIDD) and GoPro datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new network architecture CascadedGaze Network (CGNet), which is a convolutional network that captures the global features using a Global Context Extractor (GCE). The idea is to achieve the success of transformers but be more efficient by proposing a component that does a similar job of self-attention but is convolutional. The effectiveness of the proposed CGNet architecture was verified on the SIDD dataset, the Gaussian image denoising dataset, and the motion deblurring dataset GoPro.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More comparison with NAFNet or exploring ways to combine GCE with NAFNet.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 3 reviewers are initially skeptical w.r.t. the significance compared to the baseline NAFNet. After the authors' revision and rebuttal, those concerns were largely lifted. All reviewers were convinced that CGNet improves over NAFNet and recommended acceptance. Editor hence follow the reviewers' unanimous recommendations to recommend acceptance of the paper.
Decision: Accept (poster)