Sharing Key Semantics in Transformer Makes Efficient Image Restoration
Authors: Bin Ren
Yawei Li
Jingyun Liang
Rakesh Ranjan
Mengyuan Liu
Rita Cucchiara
Luc Van Gool
Ming-Hsuan Yang
Nicu Sebe
Venue: NeurIPS 2024 poster (2024)
Keywords: Low-level Vision
Image Restoration
Vision Transformer
Primary area: machine_vision
TL;DR: We propose SemanIR, a Transformer-based approach for image restoration that optimizes attention computation by focusing on semantically relevant regions, achieving linear complexity and state-of-the-art results across multiple tasks.

Abstract: Image Restoration (IR), a classic low-level vision task, has witnessed significant advancements through deep models that effectively model global information. Notably, the emergence of Vision Transformers (ViTs) has further propelled these advancements. When computing, the self-attention mechanism, a cornerstone of ViTs, tends to encompass all global cues, even those from semantically unrelated objects or regions. This inclusivity introduces computational inefficiencies, particularly noticeable with high input resolution, as it requires processing irrelevant information, thereby impeding efficiency. Additionally, for IR, it is commonly noted that small segments of a degraded image, particularly those closely aligned semantically, provide particularly relevant information to aid in the restoration process, as they contribute essential contextual cues crucial for accurate reconstruction. To address these challenges, we propose boosting IR's performance by sharing the key semantics via Transformer for IR (i.e., SemanIR) in this paper. Specifically, SemanIR initially constructs a sparse yet comprehensive key-semantic dictionary within each transformer stage by establishing essential semantic connections for every degraded patch. Subsequently, this dictionary is shared across all subsequent transformer blocks within the same stage. This strategy optimizes attention calculation within each block by focusing exclusively on semantically related components stored in the key-semantic dictionary. As a result, attention calculation achieves linear computational complexity within each window. Extensive experiments across 6 IR tasks confirm the proposed SemanIR's state-of-the-art performance, quantitatively and qualitatively showcasing advancements. The visual results, code, and trained models are available at: https://github.com/Amazingren/SemanIR.

Sharing Key Semantics in Transformer Makes Efficient Image Restoration
Bin Ren 1,2,3 R.n..2.3 Yawei Lift
Jingyun Liang' Rakesh Ranjan Mengyuan Liu Nicu Sebe
Rita Cucchiara" Luc Van Gool Ming-Hsuan Yang*
University of Pisa Universitty of Trento "INSAIT, Sofia University 'ETH Zürich MMet Reality Labs
State Key Laboratory of General Artificial Intelligence, Peking University. Shenzhen Graduate School University of Modena and Reggio Emilia *University of California, Merced
Abstract
Image Restoration (IR), a classic low-level vision task, has witnessed significant advancements through deep models that effectively model global information. No- tably, the emergence of Vision Transformers (ViTs) has further propelled these advancements. When computing, the self-attention mechanism, cornerstone of ViTs, tends to encompass all global cues, even those from semantically unrelated objects regions. This inclusivity introduces computational inefficiencies, par- ticularly noticeable with high input resolution, as it requires processing irrelevant information, thereby impeding efficiency. Additionally, for IR, is commonly noted that small segments of degraded image, particularly those closely aligned semantically, provide particularly relevant information to aid in the restoration pro- cess, as they contribute essential contextual cues crucial for accurate reconstruction. To address these challenges, we propose boosting IR' 's performance by sharing the key semantics via Transformer for IR (i.e.. SemanIR) in this paper. Specifically, SemanIR initially constructs sparse yet comprehensive key-semantic dictionary within each transformer stage by establishing essential semantic connections for every degraded patch. Subsequently, this dictionary is shared across all subsequent transformer blocks within the same stage. This strategy optimizes attention calcula- tion within each block by focusing exclusively on semantically related components stored in the key-semantic dictionary. As result, attention calculation achieves lin- ear computational complexity within each window. Extensive experiments across 6 IR tasks confirm the proposed SemanIR's 's state-of-the teee-o--theettt art performance, quantita- tively and qualitatively showcasing advancements. The visual results, code, and trained models available at https: h/tp://ghtttt com/ Maazzinrren/Seaaae IR.
1 Introduction
Image restoration (IR) stands as fundamental task within low-level computer vision, aiming to enhance the quality of images affected by numerous factors, including noise, blur, low resolution, compression artifacts, mosaic patterns, adverse weather conditions, and other forms of distortion. This capability holds broad utility across various domains, facilitating information recovery in medical imaging, surveillance, and satellite imagery. Furthermore, bolsters downstream vision tasks like object detection, recognition, and tracking [74, 60). Despite notable progress in recent years, prevalent IR methods encounter challenges in effectively addressing complex distortions or preserving/recovering crucial image details [48]. Achieving high-quality image recovery necessitates meticulous exploration of the rich information present in degraded counterparts.
Work done during visiting at Zürich INSAIT Sofia University. Project Leader & Corresponding Author. Email: 1i. yawei. ai@gmail
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Filber
Flatten
Example Input
(b)
(c)
(e)
Figure I: (a) The CNN filter captures information only within a local region. (b) The standard MLP/Transformer architectures take full input in long-sequence manner. (c) The window-size multi- head self-attention (MSA) mechanism builds full connection within each window. (d) Position-fixe sparse connection. (c) The proposed Key-Semantie connection.
In modern IR systems, representative networks for learning rich image information are typically constructed using three fundamental architectural paradigms. i.e., convolutional neural networks (CNNs) [42, 94], Multilayer perceptrons (MLPs) [5, 82], and Vision Transformers (ViTs) [84, 20]. The imput image is treated regular grid of pixels in the Euclidean space for CNNs (Fig. I(a)) or a sequence of patches for MLPs and ViTs (Fig. I(b).. However, degraded inputs usually contain irregular and complex objects. While these choices perform admirably in scenarios with regular or well-organized object boundaries, they have limitations when applied to degraded images with more flexible and complex geometrical contexts.
Additionally, CNNs struggle to model long-range dependencies due to their limited receptive field (Fig. ((a)). In contrast, MLPs and ViTs can capture long-range relations effectively, although at the cost of losing inductive bias and incurring a heavy computational burden, i.e., quadratic complexity increases with higher input resolution| 82, 84, 20, 70J. To address these limitations, recent IR methods have explored strategies for complexity reduction. A common approach is to implement MSA within local image regions [53, 50]. For example, SwinlR [50] and GRL [48] employ full MSA region- fixed anchored stripe MSA, but they still struggle with irregular object connections. Furthermore, prior research [111] highlights that smooth image content is more prevalent than complex image details, underscoring the need for distinct treatment based on semantic content.
In this paper, we introduce a novel approach, SemanIR, to address the limitations above. Specifically, within each transformer stage, we first construct key-semantic dictionary, which stores only the top-k semantically related relations for each given degraded patch with the k-nearest neighbors (iee. KNN) algorithm. Then, the attention operation within each transformer layer occurs only among the top-k patches. This design brings two main advantages, i.e., 1) Each degraded patch benefits from its semantically similar patches, typically containing comparable contextual or textural information, while excluding the side effects from other patches that contain entirely unrelated information. 2) Compared to the conventional window-wise attention, which built a dense connection between all the sparse but position-fixed fixed
patches (Fig. 1()) that leads to highly computationally demanding, manner (Fig. (dd) which introduces irrelevant semantics. Our key-semantic connection (Fig. 1(e)) leads to a sparse yet more meaningful attention operation, which allows our method to achieve the same receptive field as previous ViTss methods while maintaining lower computational costs. This is not like previous token merging or pruning methods [6, 91, 69] that may merge unrelated information or prune some semantically related information. In addition, to make the proposed method more efficient, instead of creating a key-semantic dictionary for each transformer layer, we create it just once at the beginning of each transformer stage and then share it with all the following transformer layers within the same stage. This not only largely reduced the computation burden but also made our methods different from other token pruning and merging methods [106, 89, 107], which include dynamic patch skipping/selection within each attention layer or involve an additional offset generation network. Meanwhile, merging or pruning tokens will lead to loss of information corresponding patches, which is not preferred in image restoration [85]. addition, such a sharing strategy allows each degraded patch to be continuously optimized by its semantically related patches within each stage.
It is also worth noting that the implementation of the attention layer of our SemanIR is achieved in three interesting manners (i.e., Triton [19]', torch-mask, and torch-gather), which are discussed in our ablation stuiies Overall, our method's suitability for image restoration comes from the
souncs GPU programming tool https: hwt.:paaa..caaeeeeeerc////toooot

---

utilization of semantic information, the preservation of the details, and the effective KNN strategy. The contributions of this work are:
1. For each degraded input patch, we propose to construct a key-semantic dictionary that stores its most semantically relevant & patches in sparse yet representative manner. This strategy excludes the side effects of a given degraded patch from semantically unrelated parts.
2. Based on the constructed key-semantic dictionary, we propose share the key semantic information across all the attention layers within each transformer stage, which not only makes each degraded patch well-optimized but also largely reduces the computational complexity compared to conventional attention operations.
3. Extensive experimental results show that the proposed SemanIR achieves state-of-the-an performance on 6 IR tasks, i.e., deblurring. JPEG compression artifact removal (JPEG CAR), denoising, IR in adverse weather conditions (AWC), demosaicking, and classic image super-resolution (SR).
2 Related Work
Image Restoration (IR), as a long-standing ill-posed inverse problem, is designed to reconstruct the high-quality image from the corresponding degraded counterpart with numerous applications [72, 3, 49]. Initially, IR was addressed through model-based solutions, involving the search for solutions to specific formulations. However, learning- based approaches have gained much attention with the significant advancements in deep neural networks. Numerous approaches have been developed, including regression- based [51,41, 41, 50, 48, 104] and generative model-based pipelines [25, 87, 56, 92]. pintesssionbbasee method for image restoration.
In this paper, we propose
Non-Local Priors Modeling in IR. Tradition model- IR methods reconstruct the image by regularizing the results (e.g.. Tikhonov regularization [27]) with formulaic prior knowledge of natural image distribution. However, it challenging for these methods to recover realistic detailed results with hand-designed priors. Besides, some other classic method finds that self-similarity is an effective prior, which leads to an impressive performance [7. 17J. Apart from traditional methods, the non-local prior has also been utilized in modern deep learning networks [86, 48, 109], typically captured by the self-attention mechanism. More recently, the overwhelming success of transformers [84] in the natural language processing domain [35] and the classic vision community [20, 8, 81, 90, 10] has led to the development of numerous ViT-based IR methods. These methods aim to enhance the learning ability for modeling non-local priors [50, 93, 48 14, 15, 98] and consequently archives better performance. Meanwhile, this raises a question: are all non-local priors essential for IR?
Key-Semantic Non-local Prior Exploration for IR. To answer the question, we found many methods demonstrating the effectiveness of modeling key semantics within ViTs. For example, KiT [43] proposed increasing the non-local connectivity between patches at different positions through KNN matching. This approach aims to better capture the non-local relations between the base patch and other patches in each attention calculation. However, it results in significant additional computational costs due to the KNN matching. DRSformer [13] proposed a top-k selection strategy that chooses the most relevant tokens to model the non-local priors for draining after each self-attention calculation without reducing the computation complexity, since after each attention calculation, the DRSFormer utilized (mask, top-k, scatter) operations at each transformer layer. Consequently, this inevitably increases the computation cost. Similar conclusions can be also drawn from the graph perspective solutions [28, 73, 61, 32] for various IR tasks, like facial expression restoration [54], image denoising [77], and artifact reduction [61]. [32] construct the graph with transformer-be based architecture where each patch is connected to all other patches. All these methods suggest that if the semantically related information can be addressed, the degraded image can be restored with better performance. However, the efficiency issue, which is extremely unignorable, remains untouched within the aforementioned methods. is particularly crucial for ViTssbased image restoration methods, which often need address high-resolution degraded input images. VaV [107] reduces computational costs by storing attention scores from few initial layers and reusing them in subsequent layers. However, this approach does not change the computation cost of attention itself; it merely reuses previously computed scores. In this paper, we propose sharing key semantics within each transformer stage, demonstrating its efficiency and effectiveness through experimental and theoretical analysis. Our method, SemanIR, reduces computation in both training and inference
3

---

Semeiil-SSpp
Semaalik Staee
Sommniik
SeroaLX
(a) The architecture of the proposed SemanIR for image super-csooution
Triling
Five
Kry Semsrtic Artetiinn
FFN
QKV Proj
(b) The propoced Sey-Saemntic Transformer Layer
(c) Key-Semintic Dictionary
(4) Key-Semantic Atlentoon
Extractor
Imuge Recunstractor
Seman Dictionary Construction
Element-wise addition
Matrix Multipiicttinn
Figure 2: The proposed SemanIR mainly consists of convolutional feature extractor, the main body of SemanIR for representation learning, and an image reconstructor. The main body in columnar shape shown here is for image SR, while the U-shaped structure (shown in Appx. A.2) is used for other IR tasks. (b) The transformer layer of our SemanIR. The toy example of k=3 for (c) the Key-semantic dictionary construction and (d) the attention of each Layer.
by using a semantic dictionary to filter out irrelevant patches during training and optimizing attention operations with Triton kernels during inference.
3 Methodology
To comprehensively study the effectiveness the proposed method that is architecture-agnostic for various IR tasks, adopted two of the most commonly used architectures i.e. the columnar architecture (shown in Fig. 2 (a)) for image SR and the U-shaped architecture (shown the Appendix, i.e., Appx. A.2) for other IR tasks. In the following, dictionary in Sec. 3.1. Based on the key-semantic dictionary, then we explain why sharing it works for IR, and we introduce the basic unit, the key-semantic transformer layer in Sec. 3.2. Finally, two interesting discussions (Sec.3.3) are introduced regarding the implementation style of the Key-Graph attention and two top-k settings during the training. The efficiency analysis is provided in Appx. A.3.
first show how to construct the fey-semantic
3.1 Key-Semantic Dictionary Construction
Consider the input feature Fin E HxXCC where H. W, and C denote the height, the width, and the channel. VIT's are good at modeling global dependencies for Fin. This is achieved by the MSA, the core of ViTs, by connecting all other patches to a certain patch. Specifically, F,, first split into N non-overlapping patches, forming its patch representation P {pilp; E RRauxe where ht, wr and e are the height, the width, and the channel of each patch. To achieve such global connectivity D, P is linearly projected into Query (Q), Key (K), and Value (V) matrices, which are denoted as Q = PW gras K- PW key, and V = PW val- W Warw/Ken /ey/eeal represents the learnable projection weights. Then D is performed by softmax function follows:
1, 2, 3, ... N).
exp(QrK,)
Di Ex=I.., exp Expp(Q/K//VV
1,2,3, N,
(1)
where d is the dimension of Q and K. Then each patch is aggregated via E, D.y However, Diş functions as full semantic dictionary, where each patch is connected to all other patches regardless of their semantic relatedness. e.g., given a sub-gras sub-graph with green dog patch shown Fig. I(c), the tree-related patches are also considered. Since such an operation occurs at each attention calculation step in ViTs, inevitably increases the computational cost, especially for large-scale inputs. In addition, for IR, degraded patch usually benefits from its most semantically related patches, as they share similar texture and geometric information. This naturally raises the question: Can we build key-semantic dictionary, Dx, where each patch is connected only to its most related patches?

---

Algorithm 1 Key-Semantic Transformer Stage (i.e., SemanIR Stage)
Input: input feature Fin. numbers of SemanIR layer Nayer KNN value k, the patched feature P
Output: aggregated feature Fout
I: Dx= < KeySemanDiicionaryy (icciiooarry_oossnncor(.... Sec. 3.1
2: for i 1 to Mayyr do
Q. K, V t- Sinear_Proj(?
4: P
Semanll SernanIR_Aut(Q, K, V; V,D)) Sec. 3.2
5: p E-P+ FFN(P) // See. 3.2
6: end for
7: Fout f- Fin + Conv(UnPPacc((P))
8:
return Fout
To mitigate this problem, given P, we first construct fully connected dictionary D by calculating its self-similarity Sim() via a naive dot product operation as ji. = Sim(i,j) pi Pj which describes the correlation among all the patches, with higher values indicating stronger correlations. To reduce the side influence of patches with low correlation (e.g. the tree-related patches at the upper left part in Fig. (c)) for the green background dog destination patch, we keep only & highly related patches and exclude the remaining. This is achieved by a KNN algorithm from D as follows:
DK(i.j) - (D(i.)), D(i,)) 2 Sim(i,) ) and + 10. 10 otherwise,
(2)
where Sim(i, )a denotes the ken largest connectivity value of patch ps. As a result, Dx con- tains only the patches with high correlation (e.g.. dog-related patches Fig. ((e)) for the desti- nation patch (e.g., the green dog patch). We formalize the key-semantic dictionary construction as KeySemanDDccioonary CoriiinnryCCossuuuttorrrrr in Alg. 1. Although such a dictionary allows the subsequent attention operation to focus on the most semantically related patches, constructing it before each attention operation significantly increases computational costs. Mcanwhile, we observed that IR architectures typically transformers stage-by syrre--b-ttage (See Fig. 2(a)). This means that in each stage, several transformer layers are directly connected sequentially and operate at the same semantic level. Inspired by this, we propose share the same key-semantic dictionary for all the transformer layers.
3.2 Sharing Key Semanties Cross Transformer Layers
The structure of each transformer layer is shown in Fig. 2(b), which consists of a key-semantic attention block followed by a foed-forward network (FFN). Specifically, given an input F... we form each transformer layer as output, and / is the trainable parameters. Previous methods tried to reduce the computation cost mainly by applying some techniques (i.e., T) like token merging or pruning after each attention calculation or the entire transformer layer, and it can be formalized as 2 = T(FFN (fFFN(f((En)) (F.))))) or (TFN(T fo (F..)))) However, the main computation cost from MSA is still untouched.
(FF.))(F..)), where ff means the transformer layer, = denotes the
Owing the permutation-invaricc property (i.e., (T (x) here T E N means any token level permutation matrix) inherent in both the MSA and the FFN [84, 44], the transformer layer consistently produces identical representations for patches that share the same attributes. regardless of their positions or the surrounding structures [9]. In other words, patches at the same location are consistently connected to other patches possessing the same attributes as they traverse through the various layers within the same stage. It enables Dx to serve reference permutation for each attention in the subsequent transformer layers, facilitating efficient yet highly semantics-relatt attention operations. This distinguishes our method from previous token merging/pruning [106, 89] or sparse ditention-based methods (Fig. I (d) that only activate patches in a grid -ixed manner[98).
The workflow is intuitively illustrated in Fig. 2 (c) and (d). Initially, the patch P is linear projected via Linear [roj() (The 3rd step Alg. 1) into Q, K, and V. For each patch p; in Q, instead of calculating the self-attention with all fu patches in K&V, only i essential patches are selected via the semantic lookup via the indices from Dx in them, forming the K&V. Then the attention matrix Del is obtained by: Dott Softmaxx (QKT /Vd), which captures the pair-wise relation between each destination patch p; in Q with only patches in K&V that are semantically highly related to p.. For other unselected patches in K &V, we aim to maintain their position in their corresponding
5

---

Table I: GPU memory footprint of different implementations (i.e., Triton, Torch-Gather, and Torch-Mask) of our key-graph attention block. N the number of tokens and is the number of nearest neighbors. OOM denotes "ou "out of memory".
Table 2: Single-image motion deblurring results. Go-
Pro [62] dataset used for training.
Gol'r PSNRT SSIM
HIDE
Average
Method
PSNR?
SSIM
PSNR1 SSIMf
DeblurGAN [39]
28.70
0.858
24.51
0.871
26.61
0.865
Nah [62]
0.914
25.73
0.874
27.41
RablurGANv22 [40]
29.55
0.934
26.61
0.875
28.08
0.905
SRN
30.26
0.934
28.36
0.915
29.31
0.925
N
Triton
Torch Gather
Torch Mask
Gao et al. [24]
30,90
0:935
29.11
0.913
30.01
0.924
DBGAN [103]
31.10
0.942
28.94
0.915
30,02
0.929
512
0.27 GB
0.66 GB
0.36 GB
MI-RNN [64]
31.15
0.945
29.15
0.918
30.15
0,932
1024
0.33 GB
1.10 GB
0.67 GB
DMPHN 196]
31.20
0.940
29.09
0.924
30,15
0.932
2048
Suin et af. [79]
31.85
0.948
29.98
0.930
30.92
0.939
4096
2.61 GB
6.83
CODE [1]
31.94
29.67
30.81
8192
10.21 GB
10.57 GB
26.42 GB
SPAIR [65]
32.06
0.953
30.29
0.931
31.18
0.942
[16]
32.45
0.957
29.99
0.930
31,22
0.944
k
Triton
Torch-Gateer
Torch-Mask
IPT [30]
32.52
MPRNet (94)
32.66
0.959
30.96
0.939
31.81
0.949
GB
KiT [43]
32.70
0.959
30.98
0.942
31.84
0.951
64
5.82 GB
27.56 GB
13,93 GB
NAFNet [11]
32.85
0.960
128
6.45 GB
COM
14,43 GB
Restormer [93]
32.92
0.961
0.942
32.07
0.952
256
7,70 GB
0OM
15,43 GB
Ren et al. [71]
33.20
0.963
30,96
0.938
32,08
0.951
512
10.20 GB
00M
17.43 GB
SemanIR (ours)
33.44
0.964
31,05
0.941
32.25
0.953
places without any computation. Based on Ditt, the attention outputs the updated feature P via:
in the 4th step of Alg. 1. This differs from the conventional MSA, which calculates the relation of each patch in Q and all patches in K&V. Finally, with FFN, the output of each transformer layer is achieved via the 5th step of Alg. 1.
P = V. We formulate these two procedures as SemanIR_Att
Conversely, our design offers two advantages. Firstly, the computational cost can be significantly reduced within each attention window (detailed analysis can be found in the Appx. A.3), enhancing efficiency. Additionally, sharing the key semantics across transformer layers within each stage acts as a loop that continuously optimizes a degraded patch with its most semantically related patches, ensuring the performance of the proposed method (supported by our experimental results in Sec. 4).
3.3 Discussion
Fixed top-k vs. Random top-k Training Strategies. In the fixed top-k approach, k remains constant at 512 during training. In contrast, in the random top-k method, k is randomly selected from the set [64, 128, 192, 256, 384, 512). It is important to note that even in the random top-k setting, fixed k value is maintained for all patches/pixels in each iteration. During inference, the random top-k strategy offers more flexibility and requires training only single model, making more user-friendly and less resource-inge inttuuvveeenetvve....
Implementation of the Attention of SemanIR. To achieve the attention operation of the proposed SemanIR, we explored three different manners for the implementation, i.e., (i) Triton, (ii) Torch- Gather, and (iii) March-Mask. Specifically, (i) is based on FltshAttention [19], and a customized GPU kernel is written for the operators proposed in this paper. Parallel GPU kernels are called for the nodes during run time. (ii) means that we use the forch.gather() function in PyTorch to choose the corresponding Qyather and Kgather based on Dx, then the attention operation is conducted between Qaather and K Kaather. (iii) denotes that keep only the value of selected patches of DR and omitting other patches with low correlation via assigning those values to - -o guided by Dk- Discussions of the pros and cons regarding these manners provided in Sec. 4.1.
4 Experiments
In this section, we first analyze three important ablation studies of our SemanIR, followed by extensive experiments on 6 IR tasks, i.e., deblurring. JPEG CAR, image denoising, IR in AWC, image demosaicking, and image SR. More details about the architecture design, training protocols, the training/testing dataset, and full guantitative/addirional qualitative results are shown in Appx. A to F. The best and the 2nd-best results are reported in red and blue, respectively. Note that denotes a single model that is trained to handle multiple degradation levels i.e., noise levels, and quality factors.

---

Kodnk24-- - Sigms 15
Sei12 Seil2-Simm 25
Livel QQ
Mongo Senle
35.5
26.3
35.6
PSNR [a0)
PSNR (a0)
PSNK (48)
36.5
252
34.0
36,0
Random topk
Random tupk
Random topk Fixed topk
Random topk
33.5
Fixed topk
Fixed topk
Fixed topk
S12
354
394
k
A
k
(a) Color Image Denoising.
(b) Gray Image Denoising.
(c) Color Image JPEG CAR. (d) Gray Image JPEG CAR.
Figure The impact of with different inference k value. Circle size represents FLOPs.
query region
without top-k
(c) with top---8
with top-k
tip-k
(f) with
Figure 4: The impact of & with different inference & value.
Table 3: The efficiency comparisons results Urban 100 dataset.
Task
Method
Architecture
Params [M]
FLOPS [G].
Runtime [ms]]
PSNR1
SwinIR [50]
Columnar
11.90
215.32
152.24
27.45
CAT [15]
Columnar
16.60
387.86
357.97
27.89
SR
HAT [14]
Columnar
20.77
416.90
368.61
28.37
SemanIR-S (Ours)
Columnar
12.02
290.20
211.94
28.34
SwinIR [50]
Columnar
11.75
752.06
1772.84
27.98
Denoising (a = 50)
Restormer [93]
U-shupe
26.10
154,88
210.44
28.29
(The same architecture for other IR tusk)
GRL [48]
Columnar
19.81
1361.77
3944,17
28.59
SemanIR (Ours)
U-shape
25.85
135.26
240.05
28.63
4.1 Ablation Study
The impact of the implementation of SemanIR Attention is assessed in terms of (i) Triton, (ii) Torch-Gather Gather, and (ii) Torch-Mask under different numbers of (various from 512 to 8192) and K (various from 32 to 512). The results of the GPU memory footprint are shown in Tab. 1, which indicate that Torch-Gather brings no redundant computation while requiring a large memory footprint. Though Torch-Mass brings the GPU memory increase, the increment is affordable compared to Torch-Gather and also easy to implement. Triton largely saves the GPU memory while at the cost of slow inference and difficult implementation for the back-propaggation process. To optimize the efficiency of our SemanIR, we recommend employing Torch-Mask during training and Triton during inference, striking balance between the efficiency and the GPU memory requirement.
The Impact of the in Key-Semantie Dictionary Construction. Three interesting phenomena observed from the results shown in Fig. 3 regarding the two top-k training strategies (Sec. 3.3). (1) The PSNR can largely increase with the increase of l in a fixed manner. (2) When reaches a certain number (i.e., 384), the performance improvements become marginal, supporting that only the most semantically related patches contribute significantly to the restoration. (3) The randomly sampled strategy has very stable and better performance compared to the fixed top-k manner especially when the inference k is fixed to small number (i.e., 64, 128, 256). We conclude that a random sampled strategy is more general and stable. It can also make the inference process more flexible regarding different computation resources. Meanwhile, we set a query region in the input and provided detailed comparison from the attention-based activation map together with the input query region in Fig. 4. Fig. 4(a) shows the query region input. Fig. 4(b) displays the activation map generated using standard attention mechanisms. Fig. 4(c-f) illustrate activation maps using our key-semantic dictionary with different lt values (18, 16, 64, 256]) during inference. The comparisons indicate that increasing i allows for connections to more semantically related regions. However, when ie is set too high (e.g., ho 256 shown in Fig. 4(f)), the activation map may include some semantically unrelated regions. This aligns with the findings and the results depicted in Fig 3, where increasing the & beyond certain point (e.g., from 396 to 512) does not further improve PSNR. More ablation results can found our Appx. D about the effect of the noise level and quality factor for denoising and JPEG CAR.
statement
7

---

Table 4: Color image JPEG compression artifact removal results.
Sct QF
JPEG
QGAC [2]]
FBCNN
[33]
DRUNet
[99]
SemanII (Ours)
SwinIR [50]
GRL-S
[48]
SemanIR
(Our))
PSNRT
SSIM
PSNRT
SSIM
PSNRT
SSIMT
PSNRT
SSIMT
PSNRT
SSIM
PSNRT
SSIMT
PSNRT
SSIMT
PSNRT
SSIMT
10
25
69
0.7430
27.62
0.8040
27.77
0.8030
27.47
0.8045
28.19
0.8146
28,06
0.8129
28.13
0.8139
28.31
0.8176
BSDSS00 LIVE
20
28.06
0.8260
29.88
0.8680
30.11
0,8680
30.29
0.8743
30.53
0.8781
30.44
0.8768
30.49
0.8776
30.61
0.8792
30
29.37
0.8610
31.17
0.8960
31.43
0.8970
31.64
0.9020
31.89
0.9051
31.81
0,9040
31.85
0.9045
31.94
0.9058
40
30.28
0.8820
32,05
0.9120
32.34
0,9130
32.56
0.9174
32.81
0.9201
32.75
0,9193
32.79
0.9195
32.85
0.9204
10
25.84
0.7410
27.74
0.8020
27.85
0.7990
27.62
0.8001
28.25
0.8076
28.22
0,8075
28.26
0.8083
28.37
0.8102
20
28.21
0.8270
30.01
0.8690
30.14
0.8670
30.39
0.8711
30.55
0.8738
30.54
0.8739
30.57
0.8746
30.63
0.8750
30
29.57
0.8650
31.33
0.8980
31.45
0.8970
31.73
0.9003
31.90
0.9026
31.90
0,9025
31.92
0.9030
31.96
0.9035
40
30.52
0.8870
32.25
0.9150
32.36
0.9130
32,66
0.9168
32.84
0.9190
32.84
0.9189
32.86
0.9192
32.88
0.9193
36
Ditasets
Datnscts
Datssiet
Datssicts
McManter
Set12
10.0
42.5
Classic5
CDSDS8
BSD66
BSDS500
BSDS500
Kodak24
Urbus100
37.5
Lrbann
40.0
Urbanl
PSNR (dR)
L1bbn
PSNR (BB)
30
25.0
37.5
Livel
PSNR
32.5
35.0
26
30.0
32.5
30.0
Sigma
Sigma
(a) Color Image Denoising, (b) Gray Image Denoising,
(c) Color Image JPEG CAR, (d) Gray Image JPEG CAR,
Onc-for-multi Sigma
One-for-multi Sigma
One-for-mutti QF
One-for-multi QF
Figure 5: One model trained to handle multi-degradations for denoising (a-b) and JPEG CAR (c-d).
Table 5: Gray image denoising PSNR
Table 6: Color image denoising PSNR.
Window Size
Set12
BSD68
Urban100
Window Size
Memaster
CBSDD6
Kodak24
Urban100
31.01
29.19
31.33
8
33.20
31.72
32.84
32.89
16
31.06
29.51
31,45
16
33.24
31.74
32.86
32,95
32
31.17
29.50
31.88
32
33.38
31.75
32.97
33.27
Efficiency Analysis. We compare our SemanIR method with four recent promising approaches on both 4x super-resolution (SR) and grayscale denoising tasks using the Urban dataset. Key metrics, including trainable parameters, FLOPs, runtime, and PSNR, are reported in Tab. 3. The results show that HAT and SemanIR achieve top-tier PSNR performance, reaching 28.37 dB and 28.34 dB, respectively, while SemanIR-S is much faster and uses 41.7% fewer parameters than HAT, making it more computationally efficient. SwinIR runs slightly faster than SemanIR-S but at the cost of significant 0.89 dB loss in PSNR. In comparison, SemanIR-S offers better accuracy and speed than both CAT and HAT, solidifying its effectiveness. Furthermore, the comparison between SwinIR and SemanIR for denoising reveals that, although SwinlR has fewer trainable parameters, its FLOPss are substantially higher, indicating that SemanIR is more efficient terms of computation. These results highlight that SemanIR-S S strikes an optimal balance between performance and efficiency, making it highly competitive for both SR and denoising tasks.
The Impact of One Model is Trained to Handle Multiple Degradation Levels. The experiments were conducted for both denoising and JPEG CAR with both color and grayscale images. For denoising, is set to [15. 25, 50, 75). For JEPG CAR, QF is set to [10, 20, 30, 40, 50, 60, 70, 80, 90). The results in Fig. 5 indicate that the PSNR for both tasks across all the datasets, under both color and grayscale settings, decreases when the degraded level increases. It means that maintaining decent generalization ability with one model to handle multiple degradation levels is not easy. Meanwhile, the proposed SemanIR can still outperform other methods on various tasks (See Tab. 7 and Tab. 4), which mcans that capturing the key semantics is also essential for model's 's generalization ability.
The Impact of the window size. The windows indeed contain mixed information from different semantic parts. Yet, is precisely this semantic distinction that motivates us develop a selection mechanism for semantic information using KNN. We have conducted ablation studies of the window size (i.e,, on both gray and color image denoising with = 25). The results are summarized in Tab. 5 and Tab. 6. With the increase of the window size, the semantic relevant information for each token is increased, thus leading to a PSNR gain for different IR tasks. For gray image denoising, larger window sizes lead to improved PSNR performance, consistently yielding higher PSNR values across different datasets. Similarly, in color image denoising, larger window sizes also result in better outcomes. For example, in the McMaster dataset, the PSNR increases from 33.20 dB with
8

---

Table 7: Color and grayscale image denoising PSNR results.
Color
Grayscale
Method
a.
CBSD68
McMaster
Urban 100
Set12
BSD68
Urhan 190
1-15 o-
or -5000
25-25
--0
c715
0-25
DDCCN [86]
0.56
33,90
31.24
27.95
33.45
31.52
38.62
$2.98
30.81
27.59
32.67
30.35
27.18
51,62
29,
26.25
52.28
29
80
36.35
*FFDNe [101]
0.49
33.87
31.21
27.96
34,66
32.35
29.18
33.83
31,4
28.05
32.75
30.43
27.32
31,63
29.
19
26.29
32.40
29.90
26.50
DRUNet [99]
52.64
34,30
31.69
28.51
35.40
33.14
50.08
34.81
32.60
29.61
33.25
30,94
27
31.91
29. 48
26.59
33.44
31.11
27.96
Restormer [96]
26.13
34,39
31.78
28.59
35.55
33.31
30.29
35.06
32.91
30.02
33.35
28.01
31,95
29:51
26.62
33.67
31.39
28.33
"SemanRR (Oors)
25.82
34.42
31,78
28.57
35.65
33.40
30.34
35.57
35,26
30.41
33,47
31,16
28.
31,95
29.49
26 54
34.05
31
31.84
28.83
DaC.N [36]
0.56
33.90
31.24
27.95
33.45
31.52
28.62
32.98
30.81
27.59
32.86
30,44
27.18
51.73
29:23
26.23
32.61
29.95
26.26
EDTT- [47]
11,48 34.39
31.76
28.
15.61
33.34
50.25
35.22
35,07
30.16
DRUNe: [99]
32.64 34,30
31.69
28.51
35.40
33.14
30.08
34.81
32,60
29.61
33.25
30.94
27.90
31,91
29.48
26.59
33.44
27.96
SwintR [50]
11.75 34.42
31,78
28. 56
35.61
33.20
50.22
35.13
32,90
29.82
33.36
31.01
27:91
31,97
29.
58
33.70
31.30
30
27.98
Restommer [93]
26.13
31.79
36.61
32,96
30.02
33,42
31.08
28.00
31,96
29.52
33.79
31.46
28.29
Xformer 197)
25.23
34,43
31,82
28.63
35.68
33.44
30.38
35.29
3
30.36
33.46
31,16
28. 10
31.98
29.55
26.65
33.98
31.78
28.71
SemanIR (Ouns)
25.82
31,79
28.60
35.65
33.43
30.30 38
35.38
33.29
30.51
33.48
31,18
28.
31.97
29.52
26.53
34.09
31.87 28.86
Table 8: IR in adverse weather conditions.
Table 9: Image demosaicking PSNR results.
Type
Testl (rain=fog)
Snow 1000---
RainDrop
Datasess
Kodak
McMaster
Method
PSNRT
Method
PSNRT
Mothod
PSNRT
pix2pis [31]
19.09
DesnowNet [52)
27.17
AuGAN [66]
30.55
MMNet [38]
40.19
37.00
Task
Specitic
IRGAN [45]
21.56
JSTASR (12)
25.32
Quan [68]
31.44
DDR [88]
11.11
37.12
Deegloint [26)
42.00
39.14
SwinlR (50)
23.23
SwinlR
28.18
SwinIR
30.82
REDD (29]
42.49
39.25
MPRNet [9:1]
21.90
DDMSNET [102]
CCN [67]
31.34
DRUNS 199)
42.68
Mulis
All-in-One [46]
24.71
All-in-Chn
28.33
All-in-One
$1.12
RNAN [109]
43.16
39.70
TransWa. Wea. (83)
27.96
TransWea.
28.48
Wea.
28.84
GRL [48]
43.57
SemanIR (Ours)
29.57
SemanIR (Ours)
30.76
SemaalB (Ours)
30.82
SemaeIR (Ours)
43.62
40.68
window size of 8 to 33.38 dB with window size of 32. These results suggest that larger window sizes enhance performance by capturing more contextual information.
4.2 Evaluation of SemanIR on Various IR Tasks
Evaluation Image deblurring. Tab. 2 shows the quantitative results for single image motion deblurring on synthetic datasets (GoPro [62], HIDE 176)). Compared to the previous state-of-the-art Restormer [93], our SemanIR achieves significant PSNR improvement (i.e.. 0.52 dB) on the GoPro dataset and the second-best on the HIDE dataset. The visual results are shown in the Appx. E.
Evaluation on JPEG CAR. The experiments for color images are conducted with 4 image quality factors ranging from 10 to 40 under two settings (i.e., i a single model is trained to handle multiple quality factors, and each model for each quality). The quantitative results shown in Tab. 4 indicate that our SemanIR achieves the best results on all the test sets across various quality factors among all the comparison methods for the color images. The visual comparisons in the Appx. E further supports the effectiveness of our method.
Evaluation Image Denoising. We show color and grayscale image denoising results in Tab. 7 under two settings (i.e., 9 one model for all noise levels o = {15, 25, 50} and each model for each noise level). For fair comparison, both parameters and accuracy are reported for all the methods. For t, our SemanIR performs better on all test sets for color and grayscale image denoising than others. It is worth noting that we outperform DRUNet and Restormer with lower trainable parameters. For another setting, the proposed SemanIR also archives better results on CBSD68 and 1r0 for color image denoising, and on Set and Urban 100 for grayscale denoising. These interesting comparisons validate the effectiveness of the proposed SemanIR and also indicate that our method has higher generalization ability. The visual results in Appx. E also support that the proposed SemanIR can remove heavy noise corruption and preserve high-frequency image details, resulting in sharper edges and more natural textures without over-smoothness or over-sharpness problems.
Evaluation in AWC. We validate SemanIR in adverse weather conditions, including rain+fog (Testl), snow "SnowTest100e achieves the best performance on Testl (i.e., 5.76% improvement) and Snow 100-LL (i.e. 8.01 % improvement), while the second-best PSNR on RainDrop compared to all other methods. See Appx. E for Visual comparisons.
and raindrops (RainDrop). PSNR is reported in Tab. 8. Our method
Evaluation on Image Demosaicking. The quantitative results shown in 9 indicate that the proposed SemanIR performs best on both the Kodak and MaMaster test sets, especially 0.05dB and 0.45dB absolute improvement compared to the current thth-arf-

---

Table 10: Classical image SR results. Both lightweight and accurate models summarized.
Method
Scale
Params
BSD100
Urhan100
Manga109
PSNRT
SSIMT
PSNRT
SSIMT
PSNR
SSIMT
PSNRT
SSIMT
PSNRT
SSIMf
RCAN [108]
2x
15.44
38.27
0.9614
34.12
0.9216
32.41
0.9027
33.34
0.9384
39.44
0.9786
SAN
2x
15.71
38.31
0,9620
34,07
0,9213
32.42
0,9028
33.10
0.9370
39.32
0,9792
HAN [63]
2x
63.61
38.27
0.9614
34.16
0.9217
32.41
0.9027
33.35
0.9385
39.46
0.9785
[10]
2x
115.48
38.37
34.43
32.48
33.76
SwinlR [50]
2x
11:75
38.42
0.9623
34,46
0.9250
32.53
0.9041
33.81
0.9427
39.92
0.9997
CAT-A [15]
2x
16.46
38.51
0,9626
34.78
0,9265
32.59
0.9047
34,26
0.9440
40.10
0.9805
ART [98]
2x
16.40
38.56
0,9629
34.59
0,9267
32.58
0,9048
34.30
0,9452
40.24
0.9808
EDT [47]
2x
11.48
38.63
0.9633
34.80
0.9273
32.62
0,9052
34.27
0.9456
40.37
0.9811
SemanIR-S (Ours)
2x
11.87
38.57
0,9651
34.99
0,9300
32.65
0.9078
34,86
0.9472
40.45
0.9824
SemanIR-B (Ours)
2x
19.90
38.61
0,9654
35.08
0,9304
32.69
0,9084
34,99
0.9455
40.59
0,9830
RCAN [108]
4x
15.59
32.63
0.90022
28.87
0.7889
23:77
0.7436
26.82
0.8087
31.22
0.9173
[1N
4x
15.86
32.64
0.9003
28.92
0.7888
27.78
0.7436
26.79
0.8068
31,18
0.9199
HAN [63]
4x
64.20
32.64
0.9002
28.90
0,7890
27.80
0,7442
26.85
0.8094
31.42
0.9177
IPT [10]
4x
115.63
32.64
29:01
27.82
27.26
SwinlR [50]
4x
11.90
$2.92
0,9044
29.09
0.7950
27.92
0.7489
27.45
0.8254
52.05
0.9260
CAT-A [15]
4x
16.60
33.08
0.9052
29.18
0.7960
27.99
0.7510
27.89
0.8339
32.39
0.9285
ART [98]
4x
16.55
33.04
0,9051
29.16
0.7958
27.97
0.751
27.77
0.8321
32.31
0,9283
EDT [47]
4x
11.63
33.06
0.9055
29:23
0,7971
27.99
0,7510
27.75
0.8317
32,39
0.9283
SemanIR-S (Ours)
4x
12.02
33.02
0,9082
29:29
0.8026
27.96
0.7582
28.34
0.8467
32.48
0.9322
SemanIR-B (Ours)
4x
33.08
0.9090
29.34
0.8037
27.98
0.7599
28.51
0.8467
32.56
0.9335
Trpa: Siees 1i 6o
ART
CATA
1DT
Sumuall (Dun))
Groed Track
Figure Visual comparison of classical image SR (4x) on 100a Best viewed by zooming.
Evaluation on SR. For the classical image SR, compared our SemanIR with both recent lightweight and accurate SR models, and the quantitative results are shown in Tab. 10. Compared to EDT, SemanIR-bai achieves significant improvements on Urban 100 (i.e., 0.72 dB and 0. 0.76dB for 2x and 4x SR) and Mangal09 datasets (i.e., 0.22dB and 0.17 dB for 2x and 4x SR). Even the SemanIR-sman consistently ranks as the runner-up across the majority of test datasets, all while maintaining a reduced number of trainable parameters. Visual results in both Fig. and Appx. also validate the effectiveness of the proposed SemanIR. Specifically, it is clear from the zoomed part in Fig. 6 that SemanIR can restore more details and structural content compared to other methods.
5 Conclusion
In this paper, we propose novel approach, SemanIR, for ViTs-based image restoration, which experimentally validated that global cues are essential to restore degraded images well, but the most semantically related global cures play the major role. Specifically, we propose to construct a semantic dictionary (i.e., naively by self-similarity is enough) for storing only the most related &s semantic information and then use it as a reference for guiding the attention operation for making the attention operation pay more attention only to these key semantics. Fur- thermore, we share the key-semantic dictionary with all the upcoming transformer layers within the same stage since each stage of the transformer is typically at the same semantic level. This strategy significantly reduces the computational cost for IR and functions as loop optimization, continuously restoring degraded patches with their most semantically related patches, which share similar texture or structural information. Extensive experiments on 6 IR tasks validated the effectiveness of SemanIR, demonstrating that our method achieves new state-of-the-an ahr performance.
capture the key semantics,
10

---

Acknowledgments and Disclosure of Funding
We thank Danda Pani Paudel for the valuable discussions on this work. This work was partially supported by Shenzhen Innovation in Science and Technology Foundation for The Excellent Youth Scholars (No. RCYX2023121 1090248064), the National Natural Science Foundation of China (No. 62203476), the MUR PNRR project FAIR (PE00000013) funded by the NextGenerationEU, the PRIN project CREATIVE (Prot. 2020ZSL9 the EU Horizon project ELIAS (No. 101 (0227),,, and the Ministry of Education and Science of Bulgaria (support for INSAIT, part of the Bulgarian National Roadmap for Research Infrastructure).
References
[1] Eirikur Agustsson and Radu Timofte. NTIRE 2017 challenge on single image super super- resolution: Dataset and study. In CVPRW, pages 126-135, 2017. 17
[2] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE TPAMI, 33(5):898 916, 2010. 17
[3] Mark R Banham and Aggelos K Katsaggelos. Digital image restoration. IEEE Signal Processing Magazine, 14(2):24-41, 1997.
[4] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low- complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012. 17
[5] Christopher M Bishop and Nasser M Nasrabadi. Pattern Recognition and Machine Learning. volume 4. Springer, 2006. 2
[6] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your VIT but faster. In ICLR, 2023. 2
[7] Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. CVPR, volume 2, pages 60-65, 2005.
[8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020. 3
[9] Dexiong Chen, Leslie O' Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In ICML, pages 3469-3489, 2022. 5
[10] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, pages 12299-12310, 2021. 3, 6, 10
[11] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In ECCV, pages 17-33, 2022. 6, 18, 19
[12] Wei-Ting Chen, Hao-Ya Yu Fang, Jian-Jiun Ding, Cheng-Che Tsai, and Sy-Yen Kuo. JSTASR: Joint size and transparency-awaie snow removal algorithm based on modified partial convolu- tion and veiling effect removal. In ECCV, pages 754-770, 2020. 9, 17
[13] Xiang Chen, Hao Li, Mingqiang Li, and Jinshan Pan. Learning a sparse transformer network for effective image deraining. In CVPR, pages 5896-5905, 2023. 3
[14] Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong. Activating more pixels in image super-resolution transformer. In CVPR, pages 22367-223777 2023. 19
[15] Zheng Chen, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Cross aggregation transformer for image restoration. NewrIPS, 35:25478-25490, 2022. 7, 10
[16] Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Won Jung, and Sung-Jea Ko. Rethinking coarse-to-fine approach in single image deblurring. In ICCV, 2021. 6
11

---

[17] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-D transform-domain collaborative filtering, IEEE TIP. 16(8):2080- 2095, 2007.
[18] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In CVPR, pages 1105--11074, 2019. 10
[19] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory efffeitteffcceet exact attention with 1O-awareness. In NeurIPS, 2022. 2,
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In JCLR, 2020. 2,
[21] Max Ehrlich, Larry Davis, Ser-Nam Lim, and Abhinav Shrivastava. Quantization guided JPEG artifact correction. In ECCV, pages 293-309, 2020.
[22] Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Pointwise shape- mhape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images. IEEE TIP, 16(5):1395- 1411, 2007. 17
[23] Rich Franzen. Kodak lossless true color image suite. source: http://rOk. us/graphics/kodkk 4(2), 1999. 17
[24] Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In CVPR, 2019. 6
[25] Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super- resolution. In CVPR, pages 10021-10030, 2023.
[26] Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. Deep joint demosaicking and denoising. ACM TOG, 35(6):1-12, 2016.
[27] Gene H Golub, Per Christian Hansen, and Dianne P O'LLerr. Tikhonov regularization and total least squares. SIAM Journal on Matrix Analysis and Applications, 21(1):185 194, 1999. 3
[28] Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In IJCNN, volume 2. pages 729-734, 2005. 3
[29] Yu Guo, Qiyu Jin, Gabriele Facciolo, Tieyong Zeng, and Jean-Michel Morel. Residual learning for effective joint demosaicing-denoistie arXiv preprint arXiv:2009.06205, 2020.
[30] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. CVPR, pages 5197-5206, 2015. 17
[31] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, pages 1125-1134, 2017. 9, 17
[32] Bo Jiang, Yao Lu, Xiaosheng Chen. Xinhai Lu, and Guangming Lu. Graph attention in attention network for image denoising. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023. 3
[33] Jiaxi Jiang, Kai Zhang, and Radu Timofte. Towards flexible blind JPEG artifacts removal. In ICCV, pages 4997-5006, 2021. 8
[34] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711, 2016. 19
[35] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: survey. ACM computing surveys, 54(10s):1-41,
2022. 3
12

---

[36] Daisuke Kiku, Yusuke Monno, Masayuki Tanaka, and Masatoshi Okutomi. Beyond color difference: Residual interpolation for color image demosaicking. IEEE TIP. 2583):1288-1300 2016. 9
[37] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: (412::1412.6980) 2014. 19
[38] Filippos Kokkinos, Stamatios Lefkimmiatis, and A. Iterative joint image demosaicking and denoising using a residual denoising network. JEEE TIP, 28(8):4177-4188 2019.
[39] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas. DeblurGAN: Blind motion deblurring using conditional adversarial networks. In CVPR, 2018. 6
[40] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. DeblurGAN-v2: Deblur- ring crnders-of-magniuuee faster and better. In ICCV, 2019.
[41] Wei-Sheng Lai, Jia-Bin Huang. Narendra Ahuja, and Ming-Hsuan Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR, pages 224-632, 2017. 3
[42] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11):2278 23244, 1998. 2
[43] Hunsang Lee, Hyesong Choi, Kwanghoon Sohn, and Dongbo Min. Knn local attention for
image restoration. In CVPR, pages 2139-2149, 2022. 3, 6, 18
[44] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. cnntuuttaioonivvarian neural networks. In
Set transformer: A framework for iitention-based ICML, pages 3744-3753. 2019. 5
[45] Ruoteng Li, Loong-Fah Cheong, and Robby T Tan. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In CVPR, pages 1633-1642, 2019. 9, 17, 24
[46] Ruoteng Li, Robby T Tan, and Loong-Fal Cheong. All in one bad weather removal using architectural search. CVPR, pages 3175-3185, 2020. 9, 17,
[47] Wenbo Li, Xin Lu, Jiangbo Lu, Xiangyu Zhang, and Jiaya Jia. On efficient transformer and image pre-training for low-level vision. arXiv preprint urXiv:2112. 10175, 2021. 9, 10
[48] Yawei Li, Yuchen Fan, Xiaoyu Xiang, Denis Demandolx, Rakesh Ranjan, Radu Timofte, and Lue Van Gool. Efficient and explicit modelling of image hierarchies for image restoration. In CVPR, pages 18278-18289, 2023. 2, 7, 8,
[49] Yawei Li, Kai Zhang, Jingyun Liang. Jiezhang Cao, Ce Liu, Rui Gong, Yulun Zhang, Hao Tang, Yun Liu, Denis Demandolx, Rakesh Ranjan, Radu Timofte, and Luc Van Gool. LSDIR: A large scale dataset for image restoration. In CVPRW, pages 1775-1787, 2023. 3
[50] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. SwinlR: Image restoration using Swin transformer. In ICCVW, pages 1833-1844, 2021. 2, 3, 7, 9, 10, 17
[51] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW, pages 1132-1140, 2017. 17
[52] Yun-Fu Liu, Da-Wei Jaw, Shih-Chia Huang, and Jenq-Neng Hwang. DesnowNet: Context- aware deep network for snow removal. IEEE TIP, 2766):304,,3773, 2018. 9, 17
[53] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[54] Zhilei Liu, Le Li, Yunpeng Wu, and Cuicui Zhang. Facial expression restoration based on improved graph convolutional networks. In MMM, pages 527-539. Springer, 2020. 3
13

---

[55] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018. 19
[56] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Image restoration with mean-reverting stochastic differential equations. arXiv preprint arXiv:2301. 2023.
[57] Kede Ma, Zhengfang Duanmu, Qingho Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang. Waterloo exploration database: New challenges for image quality assessment models. IEEE TIP, 26(2):0004-1016 2016. 17
[58] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, pages 416-423, 2001. 17
[59] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using mangal09 dataset. Multimedia Tools and Applications, 76(20):2181-22 112388 2017. 17
[60] Rafael Molina, Jorge Núñez, Francisco Cortijo, and Javier Mateos. Image restoration in astronomy: a Bayesian perspective. IEEE Signal Processing Magazine, 18(2):11-29) 2001. 1
[61] Chong Mou, Jian Zhang, and Zhuoyuan Wu. Dynamic attentive graph learning for image restoration. In ICCV, pages 5328-4333 2021.
[62] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, pages 3883-3891, 2017. 6, 9, 17
[63] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen. Single image super-resolution via a holistic attention network. In ECCV, pages 191-207 2020. 10
[64] Dongwon Park, Dong Un Kang, Jisoo Kim, and Se Young Chun. Multi-temporal recurrent neu- ral networks for progressive non-uniform single image deblurring with incremental temporal training. In ECCV, 2020.
[65] Kuldeep Purohit, Maitreya Suin, AN Rajagopalan, and Vishnu Naresh Boddeti. Spatially- adaptive image restoration using distortion-guided networks. In ICCV, 2021. 6
[66] Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from a single image. In CVPR, pages 2492-2491, 2018. 9, 17
[67] Ruijie Quan, Xin Yu, Yuanzhi Liang, and Yi Yang. Removing raindrops and rain streaks in go. In CVPR, pages 9147-9156, 2021. 17
[68] Yuhui Quan, Shijie Deng, Yixin Chen, and Hui Ji. Deep learning for seeing through window with raindrops. In ICCV, pages 2463-2471, 2019. 9, 17
[69] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynam- VCV: Efficient vision transformers with dynamic token sparsification. NeurIPS, 34:13937- 13949, 2021. 2
[70] Bin Ren, Yahui Liu, Yue Song, Wei Bi, Rita Cucchiara, Nicu Sebe, and Wei Wang. Masked jigsaw puzzle: A versatile position embedding for vision transformers. In CVPR, pages 20392-20391, 2023.
[71] Mengwei Ren, Mauricio Delbracio, Hossein Talebi, Guido Gerig, and Peyman Milanfar. Multiscale structure guided diffusion for image deblurring. In ICCV, pages 10721-10733, 2023. 6
[72] William Hadley Richardson. Bayesian- iterative method image restoration. Journal of the Optical Society of America, 62(1):55-59, 1972. 3
14

---

[73] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfar- dini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2008. 3
[74] MI Sezan and Henry Stark. Image restoration by the method of convex projections: Part
2 aapplicatiooss and numerical results. IEEE TMI, 1(2):55-01,, 1982.
[75] HR Sheikh. Live image quality assessment database release 2. http://live. ece. utexas. edu/reseeacchqualltyy 2005. 17
[76] Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human-aware motion deblurring. In ICCV, pages 2572-5581, 2019. 9, 17
[77] Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In CVPR, pages 3693-3702, 2017. 3
[78] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale
image recognition. In ICLR, 2015. 19
[79] Maitreya Suin. Kuldeep Purohit, and A. N. Rajagopalan. Spatially-attentive patch-hierardee hierarchical network for adaptive motion deblurring. In CVPR, 2020. 6
[80] Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In CVPR, 2018.
[81] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In
ICML, 2021.
[82] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. MAXIM: Multi-axis mip for image processing. In CVPR, pages 5769-5780, 2022.
2
[83] Jeya Maria Jose Valanarasu, Rajeev Yasarla, and Vishal M Patel. Trans Weather: Transformer- based restoration of images degraded by adverse weather conditions. In CVPR, pages 2353- 2363, 2022. 9, 17, 20
[84] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017. 2, 3,5
[85] Shashanka Venkataramanan, Amir Ghodrati, Yuki M Asano, Fatih Porikli, and Amir Habibian. Skip-attention: Improving vision transformers by paying less attention. In ICLR, 2024. 2
[86] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018.
[87] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. ICLR, 2023. 3
[88] Jiqing Wu, Radu Timofte, and Luc Van Gool. Demosaicing based on directional difference regression and efficient regression priors. IEEE TIP, 2588)3382233874 2016.
[89] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In CVPR, pages 4794-4803, 2022. 2,
[90] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self att-nattention network for referring image segmentation. In CVPR, pages 10502-10511, 2019.
[91] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Paylo Molchanov. A-VIT: Adaptive tokens for efficient vision transformer. In CVPR, pages 10809-108188, 2022. 2
[92] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. ResShift: Efficient diffusion model for
image super-resolution by residual shifting, arXiv preprint arXiv:2307.123443 2023. 3
15

---

[93] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, pages 5728-5739, 2022. 3, 6, 7, 18
[94] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Yang. and Ling Shao. Multi-stage progressive image restoration. In CVPR, pages 14821-14831, 2021. 2, 6,
[95] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In Curves and Surfaces, pages 711-730. Springer, 2010. 17
[96] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hierarchical multi-ptchh network for image deblurring. In CVPR, 2019. 6
[97] Jiale Zhang, Yulun Zhang, Jinjin Gu, Jiahua Dong, Linghe Kong, and Xiaokang Yang. Xformer: Hybrid X-shaped transformer for image denoising, arXiv preprint arXiv:2303.06440, 2023. 9
[98] Jiale Zhang. Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Accurate image restoration with attention retractable transformer. In ICLR, 2023. 5, 10
[99] Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. Plug- and-play image restoration with deep denoiser prior. IEEE TPAMI, 2021. 8, 9
[100] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep cnn for image denoising, IEEE TIP, 26(7):3142-3155, 2017. 17
[101] Kai Zhang, Wangmeng Zuo, and Lei Zhang. FFDNet: Toward a fast and flexible solution for enn-based image denoising. IEEE TIP, 27(9):4608 4622, 2018. 9
[102] Kaihao Zhang, Rongqing Li, Yanjiang Yu, Wenhan Luo, and Changsheng Li. Deep dense multi- scale network for snow removal using semantic and depth priors. IEEE TIP, 30:7419-7431, 2021. 9, 17
[103] Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In CVPR, 2020. 6
[104] Leheng Zhang, Yawei Li, Xingyu Zhou, Xiaorui Zhao, and Shuhang Gu. Transcending the limit of local window: Advanced super-resolution transformer with adaptive token dictionary. arXiv preprint arXiv:2401.08200 2024. 3
[105] Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic Imaging, 20(2):22006, 2011. 17
[106] Qiming Zhang, Jing Zhang. Yufei Xu, and Dacheng Tao. Vision transformer with quadrangle attention. IEEE TPAMI, 2024. 2,
[107] Shuoxi Zhang, Hanpeng Liu, Stephen Lin, and Kun He. You only need less attention at each stage in vision transformers. In CVPR, pages 6057-6066, 2024. 2.
[108] Yulum Zhang, Kunpeng Li, Kai Li. Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In ECCV, pages 286-301, 2018.
10
[109] Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu. Residual non-local attention networks for image restoration. arXiv preprint 1rXiv:1903.10082, 2019. 3,9 9
[110] Haiyu Zhao, Yuanbiao Gou, Boyun Li, Dezhong Peng, Jiancheng Lv, and Xi Peng. Com- prehensive and delicate: An efficient transformer for image restoration. In CVPR, pages 14122-14132, 2023.
[111] Maria Zontak and Michal Irani. Internal statistics of single natural image. In CVPR, pages 977-984, 2011. 2
16

---

A Experimental Protocals
A.1 Training/Testing Datasets
JPEG compression artifact removal. For JPEG compression artifact removal, the JPEG image is compressed by the cv2 JPEG compression function. The compression function is characterized by the quality factor. We investigated four compression quality factors including 10, 20, 30, and 40, The smaller the quality factor, the more the image is compressed, meaning a lower quality.
The training datasets: DIV2K (1), Flickr2K [51], and WED [57]-
The test datasets: Classic5 [22]; LIVEI [75], 1rban100 [30], BSI5500 [2].
Image Denoising. For image denoising, we conduct experiments on both color and grayscale image denoising. During training and testing, noisy images are generated by adding independent additive white Gaussian noise (AWGN) to the original images. The noise levels are set train individual networks at different noise levels. The network takes the noisy images as input and tries to predict noise-free images.
15, 25, 50. We
The training datasets: DIV2K [1], Flickr2K [51], WED [57], and BSD400 [58].
The test datasets for color image: CBSD68 [58], Kodak24 [23]. McMaster [105], and Urban [30].
The test datasets for grayscale image: Set12 [100], BSD68 [58], and Urban100 [30].
Image Demosaicking, For image demosaicking, the mosaic image is generated by applying a Bayer filter on the ground-truth image. Then the network try to restore high-quality image. The mosaic image is first processed by the default Matlab demosaic function and then passed to the network as input.
The training datasets: DIV2K [1] and Flickr2K [51].
The test datasets: Kodak [23], McMaster [105].
IR in Adverse Weather Conditions. For IR in adverse weather conditions, the model is trained on combination images degraded by variety adverse weather conditions. The same training and test dataset is used as in Transweather [83]. The training data comprises 9,000 images sampled from Snow 100K [52], 1,069 images from Raindrop [66], and 9,000 images from Outdoor-Rain [45]. 100K100K includes synthetic images degraded by snow, Raindrop consists of real raindrop images, and Outdoor- Rain contains synthetic images degraded by both fog and rain streaks. The proposed method is tested on both synthetic and real-world datasets.
The comparison methods in Tab. 6 of main manuseript: pix2pix [31], HRGAN [45], SwinIR [50], All-in-One [46], Transweather [83], DesnowNer [52], JSTASR [12], DDM- SNET [102), Attn. GAN [66], [68], and CCGAN [67].
The test datasets: testl dataset [46, 45), the RainDrop test dataset [66], and the Snow 100k-I test.
Image SR. For image SR, the LR image is synthesized by Matlab bicubic downsampling function before the training. We investigated the upscalingg factors x2, x3, and x4.
The training datasets: DIV2K [1] and Flickr2K [51].
The test datasets: Set5 [4], Set14 [95]. BSD100 [58], Urban 100 [30], and Manga 109 [59].
Image Deblurring, For single-image motion deblurring.
The training datasets: GoPro [62].
The test datasets: GoPro [62] and HIDE [76].

---

Vlow (2 1/31
nigh
1/9)
Cony
Feature Extractor
BeemanRR-Stage
SemanIR SemanIR-Stage
1st Scale
Sernan SStage
ot
2nd Scale
SemanIR Stage
SemanIR- Stage
(M/2 x Wk/2 x 201
3rd Scale
SemanIR-SSun Stage
SemanIR-Stage
(M/4 W/4
4th Scale
SemanIR SemannR-Stage
SemanIR- Stage
(M/8 W//6 8C]
Figure 7: The U-shaped hierarchical architecture (Archi- V2) the proposed SemanIR for Image Restoration. Note that this U-shaped one is used for image JPEG CAR, image denoising, image demosaicking, IR AWC, and image deblurring. Symbols @ and C denote the element- addition and channel-wise concatenation. The downsample and upsample operations are denoted by red and green arrows.
Table 11: The details of the transformer stages and layers per stage of SemanIR for both architectures.
Archi-VI (Columnar-shanl shape)
Archi V2 (U-shape)
Seman small
Seman base
Down Stages
Up Stages
Final Stage
Num. of SemanIR Stages Num. of SemanIR layer stage
6
6
6
6
A.2 Model Architecture
In the proposed SemanIR, we adopt two kinds base architecture i.e., the widely used multi-stage one shown in Fig. of our main manuscript (Archi-VI) and a U-shaped hierarchical one shown in Fig. 7 (Archi-V2) for taking patterns of various scales into account (Note that 1/3 of Tow and Thigh in Fig. denotes the grayscale/color image cases). This is consistent with previous methods such as Restormer [93], KiT [43], and NAFNet [11].
Note that The feature extractor for both architectures is implemented as a simple convolution and converts the input image into feature maps. The image reconstructor for Archi-VI takes the rich features calculated by the previous operations and estimates a recovered image.
In addition to introducing the two base architectures of the proposed SemanIR, we have provided comprehensive details of its structure in Table 11. This table outlines the number of SemanIR stages and the distribution of layers within each stage, offering thorough understanding of our model's architecture.
A.3 Efficiency Analysis
We provide a complexity comparison among the standard multi-head self-attention (MSA), the Window-wise MSA (W-MSA), and the proposed KeySemanIR MSA (SemanIR-MSA) in the Tab. 12. (H, W, C`) indicate the feature size, M represents the window size, and h denotes the number of heads. It is commonly demonstrated and proven that the complexity of the W-MSA is much lower than that of the standard MSA, i.e.,
C + HWC) < C +2(HW)O)
(3)
18

---

Table 12: The Computation Complexity Comparison.
Time Complexity
Space Complexity
MSA
O(AHWC² 2(HW)
O + 2h( (HW)))
W-MSA
OAHHW + H(C)
O HWC)
SemanIR MSA (Ours)
O(4HWC2 + +2kHWO)
O(AHWC +22kHWC)
To better understand the efficiency of the proposed method, it should considered together with all transformer layers within certain stage. Specifically, take stage, which contains 6 transformer Layers, as an example (To simplify the illustration, we omit the convolution operation at the end of each stage).
First, the total complexity of W-MSA within each stage can calculated as:
O |4HWC 2
Second, similarly, the complexity of the proposed SemanIR-MSS be calculated as follows. (O(HWC) indicates the complexity of the key-semantic dictionary construction at the start of each transformer stage. All the other layers then share hence it calculated only once).
O x [4HW 2 + +2kHWC] + (HW)C)
(5)
Third, a simple subtraction can be done as follows to validate that the proposed method more efficient compared the W-MSA within each Transformer stage:
(6 (4HWC² + H(MM)HWC] - (6 |4HWC2 2kH (HW)
(6)
=O((12M2 12k HW)HWC)
In the last equation, to provide further clarity, let's consider common setting where the window size M = 7 and the patch size is 16. The height H/ and the width W of the feature map are 64. The number of pixels within the window is approximately (7 x (16 x 16). In the proposed SemanIR, the k value is set to 512 or randomly sampled from [64, 128, 256, 384, 512]. To this end, we have:
O((12M2 - 12k HW) = O(12 (7 x 7) X (16 16) 12 x 512 64 x 64) O(150528 6144 4096) >>
This shows that the complexity is significantly greater than zero.
Based on the above analysis, it can be concluded that, together with the proposed transformer layer, constructing the key-semantic dictionary at the start of each stage leads to greater efficiency.
A.4 Training Details
Our method explores IR tasks, and the training settings vary slightly for each task. These differences encompass the architecture of the proposed SemanIR, variations in the choice of the optimizer, and loss functions. Each experiments are conducted on 4 NVIDIA Tesla V100 32G GPUs.
Architecture. We use the columnar multi-stage architecture (without changing the feature map resolution and number of channels) for image SR and the U-shaped architecture for the other tasks including image denoising, image deblurring, and other tasks. The strategy of using multiple architectures is also explored by the previous method [14, 11].
Optimizer. We adopt the same optimizer as all other comparison methods, i.e., Adam [37], for IR in AWC, and AdamW [55] for the rest IR tasks.
Loss Function. We adopt the same loss function as all other comparison methods, i.e., smooth L1 loss and VGG loss [34, 78] for IR in AWC, the Charbonnier loss for Deblurring, and L1 loss for the rest IR tasks.
Batch Size and Patch Size, We keep the similar batch size other comparison methods, i.e., (Batch size = 16, Patch Size = 64) for JPEG CAR, denoising, demosaicking, and SR. (Batch Size = 32, Patch Size 16) for IR in AWC. (Batch Size = 8, Patch Size 192) for deblurring.
19

---

BSS08500 QF 10
ISS08900-0 -QF 20
BS028500 OF
HS808500-0
25.35
33.0
31.9
20.30
PSNR (48)
PSNR (4B)
38.28
PSNR
$0.5
31.9
23.20
32.6
28:15
Ranion teph Flxod tppk
30.4
tayk Fissl topk
11.7
Rasken togh Fined ropk
Randor lopk
32.5
Flood topk
125
:92
256
256
125
354
64
129
152
512
(b)
[ivel QF
iiel QF
Fivl OF 30
-QF 40
26.3
30.6
31.9
PSNR (60)
26.2
40.5
PSNR (MB)
PSNR
PSNR
$1.6
26.1
12.6
Randoon xops Faood xopk
Randan tonk
Randon topk
Rondon topk
Fivel topk
Fioed topk
Flood topk
12% 192
192
256
123
364
123 192
512
(g)
ixel OF
OF
livel OF 30
QQF 40
28.3
30.6
31.9
32.5
18.2
PSNR 50.5
PSNRR (88)
PSNR (4B)
02.
PSNR
31.6
25.1
Randon ligh Floed ropk
toyk Fleed ropk
Rasdon togk Fleed ropk
Rondon lopk Flood topk
192 250
192 256
125
354
129 182
512
&
(
Figure 8: Ablation study on the impact of & for Color JPEG CAR on CBSD68 (a-d), Live (e-h), and U00 (i-l) datasets with QF = [10, 20, 30, 40).
Learning Rate Schedule. For all the IR tasks, similar to other comparison methods, we set the initial learning rate to x 10 10-4, and then the half-decay is adopted during the training. Note that the training iteration for JPEG CAR, denoising, demosaicking, and SR is set to 1M. For IR in AWC and debluriing, it is set to 750K.
A.5 Evaluation Introduction
Note that the results of all the comparison methods are reported from their original papers. The details of the evaluation metric (ie. SSIM, PSNR) are described as follows:
JPEG compression artifact removal. For color image JPEG compression artifact removal, the PSNR is reported on the RGB channels while for grayscale image JPEG compression artifact removal, the PSNR is reported on the channel.
Image Denoising, For color image denoising, the PSNR is reported on the RGB channels while for grayscale image denoising, the PSNR is reported on the Y channel.
Image Demosaicking, For the comparison between different methods, PSNR is reported on the RGB channels.
IR in Adverse Weather Conditions. We adopted the same PSNR evaluation metrie used in Transweather [83].
Image SR. The PSNR is reported on the Y channel.
Image Deblurring. The PSNR and SSIM on the RGB channels are reported.
B Limitations
This study faces a task-specific limitation: each image restoration task requires training a separate network. While efforts have been made to train models for varying degradation levels within specific
20

---

250
210
Ronken
Rusbe
Radie
Rancer
Bradon Footwoo
Jusdon Proedoggh
Fron upo
Frod
Froe
114
(4)
(6)
(6)
(
Sege
Hi-
Ufeen 100
Utberte Nyge
Uiber Sigee
11 100
PINA
up-
Rec
Janoe spk
Rcaka
Juslon ops
101
13c
Ze
24
15
12
i
(
Figure 9: Ablation study the impact of & for Color Image Denoising CBSD68 (a-c), Kodak24 (d-f), McMaster (g-i), and Urban 100 (j-l) datasets with {15, 25, 50).
val_psnr_y
H
38.2
38
37.8
37.6
200k
400k
600k
800k
995002
Run +
Smoothed
Value
Step
Relative
SemanIR-B 2x
38.3431
38.3355
995,002
6.942 day
SemanIR-S 2x
38.3016
38.2943
995,002
4.211 day
Figure 10: Training log shows the convergence of the proposed SemanIR during training. The
upscaling factor is 2x.
types, such image denoising and removal of JPEG compression artifacts, this approach still leads to inefficiencies in model training. It constrains the utility of the trained networks. A potential future enhancement involves developing mechanism enabling network handle diverse image degradation types and levels. Another challenge is the substantial parameter requirement of the proposed SemanIR, which operates within tens-of-millions parameter budget. Deploying such large image IR network on handheld devices with limited resources is challenging, if not unfeasible. Therefore, a promising research direction is the creation of more efficient versions of SemanIk, integrating non-local context more effectively, to overcome these limitations.
C Impact Statement
This paper introduces transformer-based approach that significantly enhances the efficiency and performance image restoration tasks, including image deblurring, JPEG CAR, image denoising, IR in adverse weather conditions, demosaicking, and image super-resolution. The proposed SemanIR's ' notable efficiency improvement holds promise for resource-effective implementations in real-world applications. This elevated performance creates opportunities for enhanced image quality across diverse domains. While our primary contribution lies in the technical aspects of Machine Learning, are cognizant of potential societal impacts, particularly in healthcare, surveillance, and digital
21

---

11.0
31.
302
(8)
(c)
Sigre
Ut figne
PAN LP4
33.00
124
21
112
K
1M
(h)
(
Figure 11: Ablation study on the impact of ls for Grayscale Image Denoising on BSD68 (a-c), Set12 (d-f), and Urban (g-i) datasets with o = 115, 25, 50}.
imaging. As with any technology, ongoing vigilance in ethical considerations during deployment is essential, ensuring responsible use and proactively addressing any unintended consequences.
D More Ablation Analyses
Besides the ablation studies presented in our main manuscript, we further provide the following two analyses:
Convergence Visualization. The training log of the proposed SemanIR for image SR is shown in Fig. 10. The log is reported for the PSNR on the Set5 dataset during training. Two versions of the proposed method including SemanIR-S and SemanlR-B are shown in this figure. As shown in this figure, the proposed network converges gradually during the training.
The Impact of the k in Key- Key-Semantic Dictionary Construction under Various IR Tasks. To explore how the k value of top-k will affect the IR performance of the proposed SemanIR. We conduct exhaustive experiments on JPEG compression artifact reduction for color images under different QF values (i.e., QF = [10, 20, 30, 40]), image denoising for both color and grayscale images under different noise levels (i.e., 3x, 4x) with the proposed SemanIR. Note that all the experiments for each IR task are conducted under two kinds of top-k settings, i.e., (i) l was randomly sampled from the range [64, 512] during the overall training phase, and (ii) k: was held constant at 512 throughout the training phase. For inference, & was configured to the specified value for both settings.
[15, 25, 50]), well image SR under different scales (i.e., 2x,
The results of the JPEG CAR in terms of the hyper- arper-parametes & under different training settings during inference for color image are shown in Fig. 8. It is clear that for the color JPEG CAR task when l is set to 64 during inference, there is a huge performance cat between the random top-k setting and the fixed top-k setting. In addition, the fixed top-k setting performs well or sometimes even a bit better than the random top-k setting only when & is also set to the same number (i.e., 512). With the decrease of k during inference for the fixed top-k setting, the PSNR drops largely marginally for all the datasets under every kind of degraded QF factor.
The results of the image denoising in terms of hyper- hyper- parameters A under different training settings during inference for both color image and grayscale image are shown in Fig. and Fig. 11. All the experimental results various datasets (i.e., BSD68/CBBDDDR share a similar trend for image denoising compared to the JPEG CAR task. The random top-k setting can maintain relatively stable PSNR score under different & during inference compared to its fixed counterpart. In addition, a decent result can be obtained for the fixed top-k setting only when the is set to the same (i.e., 512) during the inference.
Kodak24, McMaster, and Urban 100)
The results of the image SR terms of pypee--arameters under different training settings during inference for color images with different scale factors (i.e., 2x, 3x, and 4x) are also provided in Fig. 12. All experiments are conducted various datasets (i.e., BSD100, Manga109, Set5, Set14, 1,, and Urban 100). It shows that for datasets like BSD100, Set14, and Urban 100, similar trend can be also
22

---

323
1730
Keilen
32.
(s)
(2)
[6)
Suitt
312
14.9
PONA L11
24,9
344
U 000 Suau
11 100 Sudue
17:21
(6o)
60
Figure 12: Ablation study on the impact of & for Image SR with SemanIR-B on BSD100 (a-c), Manga109 (d-f), Set5 (g-i), Set14 (i-I) and Urban100 (m-o) datasets with scale (2x, 3x, and 4x).
observed in Fig. 12 compared to JPEG CAR and image denoising tasks, Le., the random top-k setting performs more stable regardless the change of the K during the inference. However, for Mangal09 and Set5 dataset. The best PSNR is obtained by the fixed top-k setting (in (d) - (i) in Fig. 12).
In general, based on all the experimental results mentioned above, we conclude that (1) the random top-k setting performs better than the fixed l setting, and usually outperforms the latter by a large margin when k is fixed to small values (i.e., 64, 128, 192, or 256.). (2) For the fixed top-k setting, if k is set to big enough (i.e., 512) during inference, the fixed top-k setting can also achieve comparative performance even better performance compared to the random top-k setting for several experiments (e.g.color JPEG CAR in Fig. (g) and Fig. 8 (k)). However, it is not always possible that the large fixed ls setting be generalized to limited computation resources, and the model trained with large fixed A usually needs the same k for inference to maintain the performance, which leads to heavy computation resources needed even for inference.
To this end, we propose to decouple the way to use between training and inference. i.e., we can use the random sample i during training while an optional fixed & during inference without degenerating the overall performance. It makes it possible to deploy models that heavily rely on large GPU memory during training but to limited GPU resources while maintaining reliable performance during inference. This is also consistent with the way we implement the proposed attention block (ie. we adopt Torch-Mask version that requires affordable large GPU memories during training compared to Torch Gathhrrthhrrree while adopting the Triton version during inference) of SemanIR.
In addition, setting predetermined k value for each patch/pixel enhances computational efficiency. A fixed & value facilitates parallel computation, particularly in attention operations. Conversely, making A values learnable for each patch or pixel would significantly increase the complexity of the attention operation. Nonetheless, exploring the potential of learnable values for each patch or pixel represents an intriguing avenue for further investigation.
E
More visual Results
To further support the effectiveness of the proposed SemanIR intuitively. We provide more visual comparison in terms of image deblurring, JPEG CAR, image denoising, and image SR below.
Image Deblurring The visual results for single image motion deblurring are shown in Fig. 14. As shown in this figure, the proposed method can effectively remove the motion blur in the input images
23

---

Raining Input
RESCAN
All-in-One
Trans Weather
SemanIR (Ours)
Figure 13: Visual comparison for restoring images in AWC. Best viewed by zooming.
and restore more details such the facial contour, and the characters compared MPRNet Restormer.
JPEG Compression Artifact Removal For JPEG compression artifact removal, the visual results for color images on the Urban 100 dataset are shown in Fig. 15. The proposed method achieves state-of-the-art performance in removing the blocking artifacts in the input images.
Image Denoising The qualitative results for image denoising on the BSD68 and the Urban 100 dataset are shown in Fig. 16 (grayscale image) and Fig. 17 (color image). It is clear that for both the grayscale and color inputs, the proposed SemanIR can remove the noise in the noisy input images and recover more realistic textural details in the restored images.
IR Adverse Weather Conditions. The qualitative results for in AWC on the Testl [46, 45] dataset are shown in Fig. 13. It shows a challenging case but our method can restore better structural content and clearer details.
Image SR. The comparison of visual results of different image SR methods is shown Fig. 18 and Fig. 19. Fig. 18 shows the results the Urban 100 dataset, and Fig. 19 shows the results the Mangal 09 dataset. The proposed SemanIR can restore more missing details in the LR images compared to other state-of-the-art methods like Swin ART, CAT, and EDT.
24

---

Blurring Input
MPRNet
Restormer
SemaniR (Ours)
GOPRO0384 11 00 000003
GOPRO0384 11 000058
Pott Hut
GOPRO0384 11 004005
Pizza Hw
GOPR00384 11 05 004043
7100
$720
20
5512 70077556
2012
5513,551 5515
1504 7508
7 750 7508
GOPRO0410 11_0000111100_____11111111 111111100000220 00003023
Figure 14: Visual comparison with single image motion deblurring on GoPro dataset. Best viewed by zooming.
25

---

Input
DRUNEt
SwiaIR
GRL
SemantR (Ours)
Uifben100: Img. 053 10
Libenin0: img 034 (q 10;
ing (q--10)
Uiban102: img. 030 10)
Urbanttt: 10)
Llban100: ling 088 (q 10);
Figure 15: Visual comparison of color JPEG CAR on Urban 100 dataset. Best viewed by zooming.
26

---

DSD68: test001 (GTT)
Input (Sigma-50)
SwinIR
Restormer
SemanIR (Ours)
BSD68: tes1024 (GT)
Inpul (Sigma-50)
SwinIR
Ressomer
SemanIR (Ours)
BSD68: test038 (GT)
Input (Sigma-50) (S0)
Swin
Restormer
SemanIR (Ours)
BSD68: test040 (GT)
Input (Sigma-50)
SwinIR
Restormer
SemanIR (Ours)
Figure 16: Visual comparison with image denoising on BSD68 dataset. Best viewed by zooming.
27

---

Impui.
DRUNd
SwinlR
SemsmIR (Ouzs)
Utban (00: img. 002 (Sigma-550)
Urbaal L00: ing (Sigme-55)
O
O
img._ (Sigme-50) 50)
Urban img. (5igma-50)
ing, (Signn-30 50)
Ubb ing. US4 (Sigme--50) 30)
Figure 17: Visual comparison with image denoising on Urban 100 dataset. Best viewed by zooming.
28

---

Odhan00) ing. 034, (LXx4)
Swinilt
AXI
CARA
FIST
Seaveil (1rg)
ETT
11
Urhaa100 www.toeeengg_ee0 Jng, D08,
Swinit
ART
CATA
01
C1rbnn000_ 237,
Swinitt
AST
C4T4
SceuudR (Oui))
EDT
Semailk (Oun)
Figure 18: Visual comparison (4x) with image SR on Urban100 dataset. Best viewed by zooming.
29

---

Seput CLIX 4e))
Swizll
CAL-A
E01
SemaziR (Cun))
GT
K
2
2
ir PRsDcIo
ire PRsDeCE
2
ir5c
RESOcID
PRSSeN
PRESOCD
PRESENTED
Kasgi Abzure
JUMP
JUMP CI
JUMP
JUMP G
JUMP
JUMP
D SKISS
D D
D FInee
D
D
D DISI
KSSSS
V
TEAIN DORNCH
FTAIN CROMIES
PRAIN DOMMOO BP
FRAIN CONACR
SRAIN COONNE BP
SRAIN COMICR SET
Mit
/ I BIHHONSS DNNOSN
/ HONONN BNYSVN
WHHONNI INVWYY
/IIT w
VETHONIE INISST
/ 7 VINHONINS MANSYY
/
Figure 19: Visual comparison (4x) with image SR on Mangal09 dataset. Best viewed by zooming.
30

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
We claims the main contribution of our work in both the abstract and the introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations in Appx. B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification:
We provided the efficiency analyses in Appx. A.3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the experimental settings including training/test datasets, training details, and evaluation provided in our Appx. A.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

reproducible, they Answer: [Yes]
Justification:
We provided our code as the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.co wttpccccppccccccee.cc.ceccccclct ppppppcgguiuls////pppppppppaaael Sadeee//ooeSumiiieecc wwwssssssssoiiiiiee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the experimental settings including training/test datasets, training details, and evaluation are provided in our Appx. A.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The error bars are not reported because we explored various IR tasks, would be too computationally expensive to report the error bars of each of the comparison results, instead, we directly report the results from their papers.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
33

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided the corresponding computation details in Appx. A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https / ///urnruri...// cc httppppllc/chccccudeeeeeeeee
Answer: [Yes]
Justification: The research conducted in this paper conforms in every respect with the NeurIPS Code of Ethics, ensuring adherence to all relevant ethical guidelines, including respect for persons, beneficence, justice, transparency, and compliance with legal and professional standards.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the impact statement in Appx. C.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
34

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
This paper focuses on the classic low-level vision part, i.e.. Image Restora- tion (IR) with the current open-source dataset, and tries emphasize the key-semantic information for more efficient solution for IR. There is no such risks.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have correctly cited the corresponding papers for all publicly released datasets used in this work.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasttss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
36

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "yxuKWVjxag", "reviewer": "Reviewer_xhpu", "summary": "Unlike traditional transformers where the multi-head self-attention layer calculates the correlation between one patch and all patches, the method proposed by the authors computes the correlation among the top k semantically similar patches, allowing image restoration with lower computational cost. \n\nAdditionally, by generating the key semantic dictionary only once at the beginning and sharing it across all transformer layers, the computational burden is significantly reduced.", "questions": "What is the specific difference between the KNN matching method used in KiT, DRSformer, and the method proposed by the authors?\n\nHow does the performance of the proposed method compare to previous token merging or pruning methods? If previous methods yield better results, why did the authors choose to use the KNN method?", "limitations": "The authors clearly outline the limitations of their research, informing the readers.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors logically explain the proposed method. They provide detailed information on the KNN algorithm and Key-Semantic Dictionary Construction and clearly illustrate how it is utilized through text and figures.\n\nThey also demonstrate the performance of the proposed method through numerous experiments. They conducted various experiments on hyperparameters and quantitatively measured the efficiency of the proposed method by assessing FLOPS, the number of parameters, and runtime.", "weaknesses": "The explanation of the difference between the KNN matching method used in KiT and DRSformer and the KNN method proposed by the authors is lacking.\n\nThe authors mention that their method differs from previous token merging or pruning methods, but there are no comparative results to show which method reduces computational cost more effectively.\n\n In Figure 2 (d) Key-Semantic Attention should be corrected from \"topk\" to \"top-k\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer xhpu:\n### Q1:  What's the difference between SemanIR and other methods like KiT or DRSformer?\n**A**: Please refer to our 1st answer in the shared \"Author Rebuttal\". \n\n### Q2: What is the difference between SemanIR and token merging and pruning methods?\n**A**: The key focus of our method significantly diverges from previous token merging or pruning techniques, aiming not only for efficiency but, more crucially, for effective performance in regression-based dense prediction image restoration tasks. While improving efficiency aligns with the goals of token merging methods, the proposed SemanIR is specifically tailored for image restoration within a regression-based paradigm, as mentioned in Line 92 of our original manuscript. For image restoration, every token encodes information about a local patch. Merging or pruning tokens will lead to a loss of information in corresponding patches, which is not preferred in image restoration [4].\nOur method’s suitability for image restoration can be attributed to the following points:\n\n- **Utilization of Semantic Information**: Unlike token merging or pruning methods [1, 2, 3], which reduce the number of tokens by combining or removing redundant ones with higher semantic similarity, SemanIR leverages these semantic-close tokens to enhance restoration. By ensuring that each token can benefit from others, our approach improves the overall quality of restoration.\n\n- **Preservation of Detail**: Token merging and pruning methods may reduce computational costs but can lead to the loss of crucial detailed information, such as texture, color, and local structures. In contrast, SemanIR maintains detailed information by focusing on semantic relevance rather than merely reducing token counts.\n\n- **Effective KNN Strategy**: In SemanIR, for each degraded pixel, we select its top-k most semantically close neighbors to contribute to the restoration of that pixel. This KNN strategy not only excludes the negative contributions from semantically unrelated neighbors but also enhances efficiency, distinguishing it from traditional approaches.\n\nTo conclude, although token merging and pruning techniques, such as those in [1, 2, 3], are effective for reducing computational complexity and are often suited for classification tasks [3], they are less appropriate for dense prediction tasks like IR. SemanIR’s approach addresses the specific needs of image restoration tasks more effectively by balancing efficiency with the preservation of detailed image information.\n\n[1] Token merging: Your ViT but faster. *ICLR'23*\n\n[2] DynamicViT: Efficient vision transformers with dynamic token sparsification. *NeurIPS'21*\n\n[3] A-ViT: Adaptive tokens for efficient vision transformer. *CVPR'22*\n\n[4] Skip-Attention: Improving Vision Transformers by Paying Less Attention. *ICLR'24*\n\n### Q3:  Notation correction:\n**A**: Thank you for pointing this out. We have revised Fig. (d) in the PDF file (Fig. 5) to include additional details, making the figure easier to follow and understand.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MAYP8DrzIu", "reviewer": "Reviewer_hRwb", "summary": "The paper proposes an efficiency-first modification to the self-attention mechanism in ViTs. The main premise of the work is to construct a key-semantic dictionary which relates each key to its semantically-relevant patches, and then share the dictionary across Transformer blocks in the same stage for computational efficiency.", "questions": "**Q1.** For the columnar architecture style, have the authors considered sharing the dictionary across different stages? Specifically in later stages, degraded information has mostly been recovered, so sharing across Transformer stages might be reasonable.", "limitations": "The paper address the limitations, and the societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1.** The proposed method is fairly straightforward, and offers a decent alternative to dense self-attention for image restoration tasks.\n\n**S2.** The proposition to share a dictionary, once constructed, across Transformer blocks allows for benefits in terms of FLOPs (G), and parameters.\n\n**S3.** Extensive experiments are conducted on several image restoration tasks.", "weaknesses": "**W1.** The proposed Key-Semantic dictionary computes the dot product to measure semantic similarity on windowed patches. While diagrammatically it is easier to illustrate, at feature level, due to mixing, etc., these windows might not necessarily contain information specific to one region. Have the authors considered window-size ablations?\n\n**W2.** There are several methods that focus on improving on efficiency of self-attention in context of image restoration. However, there is no comparison with different attention methods discussing how the propose method compares either in distortion metrics or computational performance [1], [2], [3].\n\n**W3.** The main computational efficiency is observed due to sharing the key information across Transformer layers. While maintaining the key-semantic dictionary is interesting, the idea of sharing information across Transformer layers has been explored previously [4].\n\n[1] CAMixerSR: Only Details Need More “Attention”\n\n[2] Skip-Attention: Improving Vision Transformers by Paying Less Attention\n\n[3] Learning A Sparse Transformer Network for Effective Image Deraining\n\n[4] You Only Need Less Attention at Each Stage in Vision Transformers", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer hRwb:\n### Q1:  Have the authors considered window-size ablations?\n**A**:  The windows indeed contain mixed information from different semantic parts. Yet, it is precisely this semantic distinction that motivates us to develop a selection mechanism for semantic information using KNN. We have conducted ablation studies of the window size (i.e., on both gray and color image denoising with $\\sigma=25$). The results are summarized in Tab.4 and Tab.5. With the increase of the window size, the semantic relevant information for each token is increased, thus leading to a PSNR gain for different IR tasks.\n\nTable 4: **Gray** DN on Set12, BSD68, and Urban100.\n| window-size | Set12 | BSD68 | Urban100 |\n| --| -- | -- | --|\n|8|31.01    |29.49  |31.33 |\n|16|31.06\t|29.51\t|31.45 |\n|32|31.17\t|29.50\t|31.88 | \n\n- For gray DN (Tab. 4), performance improves with larger window sizes, showing increased PSNR values across datasets.\n\nTable 5: **Color** DN on Mcmaster, Cbsd68, Kodak24, and Urban100.\n| window-size | Mcmaster\t|Cbsd68\t|Kodak24\t|Urban100\n| --| -- | -- | --| --|\n|8 |33.20\t|31.72\t|32.84\t|32.89 |\n|16|33.24\t|31.74\t|32.86\t|32.95 |\n|32|33.38\t|31.75\t|32.97\t|33.27 |\n\n- Similarly, for color DN (Tab. 5), larger window sizes yield better results. For instance, in the McMaster dataset, the PSNR increases from 33.20 dB with a window size of 8 to 33.38 dB with a window size of 32.\n\nThese findings indicate that larger window sizes enhance performance by capturing more contextual information, which will be discussed in the revised manuscript.\n \n\n### Q2:  Discussing how the proposed SemanIR compares to [1][2][3]\n**A**:\n- SemanIR vs. [1]: According to Tab. 9 in [1], [1] achieves 32.51 dB, 28.82 dB, and 27.72 dB PSNR on Set5, Set14, and BSD100 datasets, respectively. In comparison, SemanIR reaches 33.08 dB, 29.34 dB, and 27.98 dB PSNR on the same datasets. Both methods are designed for low-parameter settings. A detailed parameter comparison will be included in the revised manuscript. Note that [1] is not specifically optimized for other IR tasks.\n\n- SemanIR vs. [2]: Unlike SemanIR, [2] reuses earlier attention in subsequent layers, suggesting potential complementary benefits. [2] achieves an 11.3% FLOP reduction with performance comparable to the baseline, while SemanIR reduces FLOPs by 12.3% and improves PSNR by 0.39dB. We acknowledge the fairness of these comparisons and will conduct a similar evaluation in our IR settings, expecting similar results.\n\n- SemanIR vs. [3]: Please see the answer to the 1st question in the shared \"Author Rebuttal\", where we show that our SemanIR is consistently better than DRSformer on both deblurring and deraining.\n\n[1] CAMixerSR. *CVPR'24*\n\n[2] Skip-Attention. *ICLR'24*\n\n[3] DRSFormer. *CVPR'23*\n\n\n### Q3: While maintaining the key-semantic dictionary is interesting, the idea of sharing information across Transformer layers has been explored previously [4].\n**Answer**: Thank you for your feedback. We would like to address the points raised as follows:\n\n- **Timing of Related Work**: We note that [4] was made available on arXiv on June 1st, 2024, which is after our manuscript submission to NeurIPS. Therefore, our work was not influenced by this recent publication.  Nevertheless, we will include the following discussions in the revised manuscript.\n\n- **Focus of Our Work**: While [4] focuses on reducing computational costs in Vision Transformers (ViTs), our proposed SemanIR method is distinct in its primary application. SemanIR is designed specifically for image restoration, rather than the broader focus of LaViT on computational efficiency within ViTs. Although both works address computational efficiency, the objectives and methodologies are different.\n\n- **Difference in Computational Efficiency Approaches**: LaViT [4] reduces computational costs by storing attention scores from a few initial layers and reusing them in subsequent layers. However, this approach does not change the computation cost of attention itself; it merely reuses previously computed scores. In contrast, SemanIR introduces a novel approach for reducing computation during both training and inference. During training, SemanIR leverages a pre-constructed semantic dictionary to exclude irrelevant information from other semantically unrelated patches, thus enhancing restoration quality. During inference, our implementation with Triton kernels optimizes attention operations, directly reducing computational costs.\n\n[4] You Only Need Less Attention at Each Stage in Vision Transformers. *CVPR'24*\n\n\n### Q4: Sharing the dictionary across different stages?\n**Answer**: Thank you for your valuable suggestion regarding dictionary sharing across different stages of our columnar architecture. We appreciate the insight and acknowledge that this is a compelling idea worth exploring further. To address your suggestion, we conducted an analysis to visualize the self-similarity of features at the beginning of each stage in our model, which consists of a total of six stages. The visualization results are shown in Fig.1 of the rebuttal pdf file. Based on our observations:\n\n- **Stage-wise Semantic Similarity**: We noted that there are still significant differences in the semantic similarity maps across various stages. This suggests that sharing the dictionary from the early stages to the later stages could potentially lead to a performance drop due to the divergence in feature representations.\n\n- **Adjacency-based Sharing**: Despite the variability across stages, we observed that adjacent stages exhibit similar semantic similarities. This indicates that it might be feasible to share the dictionary every two or three stages. Such an approach could reduce computational costs while maintaining performance.\n\nGiven these findings, we recognize the potential benefits of exploring dictionary sharing. We plan to investigate this perspective in more detail in our future work to assess its impact on performance and efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kmahvDAoUD", "reviewer": "Reviewer_aTgi", "summary": "This paper propose SemanIR, a novel Transformer architecture for image restoration. The paper propose a new attention mechanism for better efficiency and efficacy, based on that within a degraded image, patches semantically close to the target patch to restore provide major information in the restoration process. They build a key-semantic dictionary, which stores the top-k closest patches, for each patch. This dictionary is shared across different transformer layers. This way, they successfully constrain the cost of attention operation with preservation of its meaningfulness and with global receptive fields. Their approach reach state-of-the-art performance with great efficiency on various image restoration tasks and benchmark datasets.", "questions": "1. Experiments on top-k selection\n\nThe experiment of fixed top-k  in Figure 3 seems to be conducted with k=512 during training, and k is changed only during training. What happens when using a different value for k during training, matching the k value with the k to use in inference? E.g., k=64 in both training & inference, k=256 in both training / inference. I wonder this since if k is fixed to k=64 both in training / inference without notable performance drop, why would one need to use a random top-k approach, when one could simply use a small k value in training, and opt to use a larger value? In addition, I wonder how the performance would differ when the k value in inference time is set to a value unused in the training process, such as 16 or 32.\n\n2. Visualization of attention maps / key-semantic dictionary?\n\nAs semantically close patches are used only in the attention process, I wonder how the visualized attention maps would look like.\n- Do the attention maps have a smooth attention across the top-k patches, or does it still rely heavily on a few nearest neighbors?\n- Could the authors provide an example visualization of top-k patch matches in the construction process? Does the visualized attention map and top-k match look as intended, similar to the concept visualization at Fig.1(e)?\n\n3. Non-sharing of key-semantic dictionary?\n\nKey-semantic dictionary is shared across multiple layers for efficiency. How does it change when it the dictionary is constructed after every layer? Ignoring the computational cost, does it lead to consistent improvement in performance?\n\n4. Complexity of key-semantic dictionary\n\nTo construct a key-semantic dictionary, the N = H x W tokens initially has to compute similarities against N tokens, Thus, I guess the complexity of the dictionary construction process to be of O(N^2) cost, which would still have quite a burdensome complexity. However, according to the Appendix, the complexity is O(HWC). Can the authors give further explanations on this?\n\n- typo: the K in the second row of Eq.6 seems to be small k, not a capital K.\n\n5. Why store the similarity, instead of index?\n\nIn construction of the key-semantic dictionary, it seems like the dictionary stores the similarity values of the top-k patches (Eq. 2). Is there a reason for storing the similarities values instead of storing the indices of top-k patches only?\n\n6.  NAFNet [11] baseline?\n\nNAFNet is a very strong baseline well-known in image restoration literature. Is there any reason there is no performance comparison to it?\n\n7. How are positional information of tokens (patches) handled?\n\nDoes the proposed attention mechanism consider positional information (either absolute or relative), or is it just the local features only that is used? I wonder how the positional information is handled both in key-semantic dictionary construction and attention layers. Are they just simply neglected?\n\n8. Windowed-attention?\n\nThe proposed method use an efficient attention mechanism while maintaining the global receptive field.\nBut according to the code from the supplementary material, it seems like SemanIR use a window-based approach from Swin-Transformers. Doesn't this limit the global receptive field and make it local, contradictory to the mentioned benefits of the proposed attention mechanism? I believe the proposed method has the potential to generalized application, even on vanilla Transformer architecture. Have the authors tried removal of windowed-attention, and applied it on vanilla Transformer architecture?", "limitations": "The limitations and potential impacts have been discussed appropriately.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using top-k semantically close patches (tokens) only sounds novel and smart, adaptively saving computational costs depending on the user's choice, with minimal negative effect on performance.\n\n2. The proposed method seems to be generally applicable, with potential to be adopted in tasks other than image restoration.\n\n3. The strong experimental results also confirm its effectiveness.", "weaknesses": "1. Overall, the experiments have been conducted extensively, on a variety of restoration tasks / benchmarks.\nYet, I think there could have been more experiments on the proposed method itself, than comparison to existing works.\nSee the Questions section (#1, #2, #3) below..\n\nAdditional comments:\n\n- I think the notation D is overused. In Eq. 1, it denotes the softmax output, which denotes the dictionary after a few lines (i.e., Eq.2). Although the subscript is slightly different, I found it confusing to read, as D_{i,j} and D(i,j) denotes completely different two things.\n\n- For future studies, open-sourcing the implementation is recommended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer aTgi:\n### Q1: Top-k selection discussion?\n**A**: See the answer to the 2nd question in the shared \"Author Rebuttal\".\n\n### Q2:  Visualization:\n**A**: We set a query region in the input and provided a detailed comparison from the attention-based activation map together with the input query region in Fig. 3 of the rebuttal pdf file. Fig. 3(a)  shows the query region input. Fig. 3(b) displays the activation map generated using standard attention mechanisms. Fig. 3(c-f) illustrate activation maps using our key-semantic dictionary with different top-k values ([8, 16, 64, 256]) during inference. To conclude:\n- **Normal Attention**: The activation map from normal attention (Fig. 3(b)) shows connections to regions that may not be semantically related to the query region. This indicates that normal attention can consider irrelevant regions.\n- **SemanIR with Key-Semantic Dictionary**: With the proposed SemanIR, using a smaller top-k value (e.g., top-k = 8 or 16) results in activation maps that connect only a limited number of neighboring patches with the closest semantic similarity. This is consistent with our intended concept, as demonstrated in Fig.1(e) of the original manuscript.\n- **Effect of Top-K Value**: Increasing the top-k value allows for connections to more semantically related regions. However, when the top-k value is set too high (e.g., top-k = 256 as shown in Fig. 3(f)), the activation map may include some semantically unrelated regions. This aligns with the findings and the results depicted in Fig. 3 of our manuscript, where increasing the top-k value beyond a certain point (e.g., from 396 to 512) does not further improve PSNR.\n\n### Q3: Non-sharing of key-semantic dictionary?\n**A**: We discuss this issue from the following perspectives:\n- **Potential Benefits**: Constructing a key-semantic dictionary for each layer could indeed enhance performance, as each layer would benefit from a dictionary specifically tailored to its unique context. This approach might allow each layer to utilize a more precisely matched dictionary, potentially improving the semantic relevance and accuracy of the attention process.\n- **Experimental Constraints**: For the sake of training efficiency, the *Torch-Mask* strategy is used. For a window-size of 32 with key-semantic dictionary sharing, this already leads to an explosion of memory. Creating layer-wise key semantics would further increase the memory footprint significantly. \n- **Empirical Evidence**: As an alternative, we have visualized the attention maps for each layer within the same stage in Fig. 4 of the rebuttal pdf file. It shows: (i) The attention maps exhibit only slight variations across layers, indicating that the semantic focus remains largely consistent. (ii) The activation regions in these maps are very similar, which supports the effectiveness of our approach of sharing the key-semantic dictionary across layers.\n\n### Q4: Complexity of key-semantic dictionary:\n**A**:  We acknowledge the mistake and appreciate the opportunity to clarify.\n- **Correction of Complexity**: The correct complexity for the similarity calculation process is indeed $\\mathcal{O}((HW)^{2}C)$, rather than $\\mathcal{O}(HWC)$. We apologize for this error.\n- **Revised Eq. 5**:\n\\begin{equation}\n\\begin{aligned}\n    \\mathcal{O}(6 \\times [4HWC^{2} + 2kHWC] + (HW)^{2}C)\n\\end{aligned}\n\\end{equation}\n- **Revised Eq. 6**:\n\\begin{equation}\n\\begin{aligned}\n    \\mathcal{O}(6 \\times [4HWC^{2} + 2(M)^{2}HWC] - (6 \\times [4HWC^{2} + 2kHWC] + (HW)^{2}C)) \\\\ = \\mathcal{O}((12M^{2} - 12k - HW)HWC)\n\\end{aligned}\n\\end{equation}\n- **Example Calculation**: Let’s consider a common setting as we indicate in our Appendix($M = 7$, patch size = 16, $H$ = $W$ =  64, k=512). We have: \n\\begin{equation}\n\\begin{aligned}\n  \\mathcal{O}((12M^{2} - 12k - HW)HWC) &= \\mathcal{O}(12 \\times (7 \\times 7) \\times(16 \\times 16) - 12 \\times 512 - 64 \\times 64) \\\\&= \\mathcal{O}(150528 - 6144 - 4096) >> 0\n\\end{aligned}\n\\end{equation}\nDespite the errors, the conclusions about the complexity remain valid. We appreciate your understanding and the opportunity to correct these details.\n\n### Q5: Why store the similarity, not index?\n**A**: In Eq 2 of our manuscript, we calculate the similarity values to construct the key-semantic dictionary but store only the indices. We have clarified this in the revised manuscript to prevent any misunderstandings.\n\n### Q6: Comparison to NAFNet[1]\nWe have included a performance comparison with [1] in Tab. 6:\n\nTable 6: The deblurring comparison on GoPro.\n| Method | PSNR | SSIM |\n| ---| ---| ---|\n|NAFNet [1] |32.85| 0.960|\n|SemanIR (Ours)|**33.44**|**0.964**|\n\nIt shows that SemanIR outperforms the strong baseline [1] in both PSNR and SSIM on the single-image motion deblurring task. We have included this important comparison in our revised manuscript.\n\n[1] Simple Baselines for Image Restoration. *ECCV'22*\n### Q7:  Positional Embedding (PE)?\n**A**:\n- For  $\\mathcal{D}_{K}$, the PE is not used but it is interesting to explore. \n- For attention, we adopted the relative PE. The full PE for all locations is first indexed from the relative positional encoding. During training, the positional encoding and corresponding value in the attention map of dissimilar tokens is masked by being set to infinity.\n\n### Q8: Windowed-attention?\n**A**:\n- For all experiments, we set window size = 32, which contains 1024 tokens. This are lots of tokens representing non-local connectivity and this size is already larger than the global range for tasks like classification.\n- The window-based calculation is chosen for its efficiency and reduced memory usage, as the masking strategy results in high memory consumption during training, making it costly for IR with vanilla ViT.\n- Applying the proposed SemanIR on vanilla ViT architecture would be a very interesting direction with proper design, which we would like to try in our future work for a more generalized exploration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EofaGspJfz", "reviewer": "Reviewer_YUtD", "summary": "This paper proposes a dictionary-based image restoration method that leverages the most relevant information to recover images with low computational costs. Specifically, the method constructs a key-semantic dictionary that stores the top-k semantically related regions for each patch and performs attention only among these related regions. Additionally, the dictionary is created only once at the beginning of each transformer stage to reduce the computational burden.", "questions": "See Weaknesses.", "limitations": "The limitations and broader impact have been discussed in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper presents an efficient and effective strategy for utilizing the most relevant patches for image restoration.\n2. Extensive experiments demonstrate the effectiveness of the proposed SemanIR method across several image restoration tasks.", "weaknesses": "1. The novelty of this paper is somewhat limited. Similar to KiT [1], the proposed method SemanIR utilizes the most relevant information for restoration by performing attention only among the related regions. The key difference between KiT and SemanIR is that SemanIR calculates the semantic relation only once and reuses the information in subsequent layers.\n2. Fig. 2 needs further improvement to increase its readability. The selected $\\hat{K},\\hat{V}$ are not shown in the calculation process of (d). Additionally, the dimensions of $\\hat{K},\\hat{V}$ are unclear in the statement. Suppose the dimension of $\\hat{K}$ is $k\\times c$, how could the dimension of $D^{att}_{K}=\\text{Softmax}_K(Q\\hat{K}^T/\\sqrt{d})$ be $hw\\times hw$ ?\n3. The intention behand constructing the random top-k strategy is unclear. Compared to the fixed top-k strategy, what are the benefits of a random $k$? It is apparent that the fixed top-k strategy would have inferior performance since the $k=[64,128,192,256,384]$ is different from the training setting $k=512$. As shown in Fig. 3, $k=256$ seems to be a suitable value for the proposed method. What is the need to increase $k$ to 512? \n4. In the IR in AWC task, some related works [2,3] are not compared.\n\n[1] KNN Local Attention for Image Restoration. CVPR'22\n\n[2] All-in-One Image Restoration for Unknown Corruption. CVPR'22\n\n[3] PromptIR: Prompting for All-in-One Blind Image Restoration. NIPS'23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer YUtD:\n### Q1: Comparison to KiT[1]\n**A**: Please refer to our answer to the 1st question in the shared \"Author Rebuttal\".\n\n[1] KNN Local Attention for Image Restoration. *CVPR'22*\n\n### Q2: Fig.2 needs to improve regarding the dimension \n**A**: Thanks for this very important suggestion, we have improved Fig.2 (d) and made it much clearer to follow. The revised figure is shown in Fig 5 of the rebuttal pdf file. We also demonstrated the difference in the calculation of the attention between training (*Torch-Mask* is used to exclude the possible negative connection below the threshold ) and inference (*Triton* kernel is design and used to greatly reduce the computation cost).\n\n### Q3:  Top-k selection discussion?\n**A**:  Please see the answer to the 2nd question in the shared \"Author Rebuttal\".\n\n### Q4:  Comparison to [2] and [3]?\n**A**: We compare the deraining results among AirNet[2] and PromptIR[3] under the single deraining degradation setting. The results are reported in Tab. 7. It shows that:\n- Our SemanIR outperforms AirNet by a large margin (i.e., 2.93 dB) on PSNR. The reason why the parameters of AirNet are only 8.93M is that it is built on CNNs.\n- Our SemanIR outperforms PromptIR by 0.79dB on PSNR with 27% fewer parameters even though both SemanIR and PromptIR are built with transformers.\n\nTable 7 The single-degradation deraining comparison on the Rain100L dataset.\n| Method|AirNet[2] | PromptIR[3] | SemanIR (Ours)|\n|---|---|---|---|\n|PSNR|34.90|$\\underline{37.04}$|**37.83**|\n|Params.|**8.93**M|35.59M|$\\underline{25.85}$M|\n\n[2] All-in-One Image Restoration for Unknown Corruption. *CVPR'22*\n\n[3] PromptIR: Prompting for All-in-One Blind Image Restoration. *NIPS'23*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear All,

We appreciate the dedicated efforts each of you has invested in evaluating our work and providing invaluable suggestions the positive feedback (i.e., logically explain, decent alternative to dense self-attention, numerous and various experiments, and strong experimental results). We have taken care to address each question (**Q**) with detailed answers (**A**), ensuring comprehensive coverage of concerns. All the Figs mentioned are provided in the attached pdf file. Below are the shared responses to all the reviewers:

### Q1: The difference between SemanIR and KiT[1] or DRSFormer[2]:
**A**: Besides the brief explanation in our manuscript ( Line 111), we offer a more detailed introduction below to emphasize the key differences:

**SemanIR vs. [1]**: 
- For k-NN matching, KiT performs KNN matching at each transformer layer, while SemanIR calculates a self-similarity once only at the beginning of each transformer stage and constructs the key-semantic dictionary $\mathcal{D}_{K}$ for sharing.  
- In terms of attention calculation within each transformer layer, SemanIR leverages the key-semantic dictionary so that only k of the $HW$ elements in $K$ and $V$ contribute to self-attention, with the rest excluded from the attention calculation. Most importantly for SemanIR in IR, the k selected elements are kept the same at each transformer layer within the same stage with the help of the key-semantic dictionary, which avoids the heavy KNN search within each layer like KiT, thereby enhancing efficiency. 
- Regarding experimental results, SemanIR also includes deblurring and deraining results. For draining, our method was trained and tested on the same datasets as KiT. Results shown in Tab. 1 indicate that SemanIR outperforms KiT in both deblurring and deraining tasks. 

Table 1: The comparison between KiT and the proposed SemanIR.
|Method|PSNR (Deblur:GoPro) | PSNR (Deblur: HIDE) | PSNR (Derain: 5 Test sets) | 
| - | - | - | - | 
| KiT[1] |32.70|30.98|32.81| 
| SemanIR (Ours) |**33.44**| **31.05**| **32.98** |

**SemanIR vs. [2]**:
-  As illustrated in Fig. 2 of the DRSFormer paper, its top-k sparse attention first computes the self-attention between all the tokens (i.e., the computation cost for $QK$ is not reduced) before performing the top-k and scatter operations, which means each token is still affected by all other tokens even some of the tokens are semantically unrelated. In contrast, SemanIR first selects the top-k elements in $K$ and $V$ to obtain $\hat{K}$ and $\hat{V}$, and then computes $Q\hat{K}^{\top}$ instead of $QK^{\top}$. This directly eliminates unnecessary contributions from semantically unrelated patches. 
-  After the attention calculation, the DRSFormer utilized (mask, top-k, scatter) operations at each transformer layer. This increases the computation cost while the proposed SemanIR doesn't need to do a top-k matching at each layer. This leads to significant efficiency improvement for SemanIR compared to DRSFormer (This is consistent with the results shown below in Tab.2). 
-  Regarding results, we used the same training datasets as DRSFormer and evaluated SemanIR on the Rain200H test set. The results shown in Tab. 2 indicate that while DRSFormer shows slightly higher performance on deraining, SemanIR is more efficient, with  **23%**, **44%**, and **89%** reduction in parameters, FLOPs, and runtime.

Table 2: The comparison between DRSFormer and  SemanIR (The efficiency is evaluated on one image with $H=W=256$).
|Method | PSNR (Derain: Rain200H) | Params.| FLOPs  |Runtime  | 
| -  | - | - | - | -|
| DRSFormer[2]|**32.17**|33.7 M|242.9 G|2200 ms|
| SemanIR (Ours) | 32.01| **25.85** M|**135.26** G|**240** ms|

As the results in the tables above and manuscript, the proposed SemanIR significantly differs from both KiT and DRSFormer, resulting in substantial enhancements in efficiency and performance for image restoration tasks, with notable improvements in runtime, parameters, and computational complexity, while maintaining competitive results in deblurring and draining.

[1] KNN Local Attention for Image Restoration. *CVPR'22*

[2] Learning A Sparse Transformer Network for Effective Image Deraining. *CVPR'23*

### Q2: Top-k selection
**A**: We examined the top-k value during training matches from the top-k value used during inference (i.e., fixed matching top-k) on JPEG CAR on the BSD500 dataset. The results in Fig. 2 indicate:
- Training with fixed matching top-k yields performance comparable to or slightly better than the random top-k approach when the top-k value is relatively small. When k=512 for both training and inference, performance is comparable to random top-k.
- However, using a fixed top-k value for training requires training multiple models for different k values (e.g., 6 models for k values ranging from 64 to 512). In contrast, the random top-k strategy offers more flexibility and requires training only a single model, making it more user-friendly and less resource-intensive.

When the top-k value during inference is set to a value not used during training. Tab. 3 shows:
- **Performance Degradation**: When using a top-k value that was not used during training, there is a notable drop in PSNR compared to other settings. This suggests that the model performance is sensitive to the specific top-k values used during training.
- **Comparison of Unseen top-k Values**: Among unseen top-k values, larger k values during inference tend to achieve better PSNR. This observation aligns with the findings from our ablation studies on window sizes.

Table 3: Inference with top-k=16, 32.
|| top-k=16 | top-k=32|Train: Random top-K(Average)|Train: Fix top-K=512(Average)|
|-|-|-|-|-|
|PSNR |30.16|30.23|30.62|30.54|

### Q3: Notation, Typo, and Open Source:
**A**: We corrected the usage of notation and addressed the typos as suggested throughout the manuscript. We have included the code in the supplementary materials, and the full training pipeline will also be released.

Meta-review: This paper propose a Transformer architecture for image restoration. It proposes an attention mechanism for better efficiency and efficacy. Their proposed method achieves state-of-the-art performance with great efficiency on various image restoration benchmarks. After the rebuttal, all the reviewers agreed to accept this paper.
Decision: Accept (poster)