Adaptive Window Pruning for Efficient Local Motion Deblurring
Authors: Haoying Li
Jixin Zhao
Shangchen Zhou
Huajun Feng
Chongyi Li
Chen Change Loy
Venue: ICLR 2024 poster (2024)
Keywords: Local motion deblurring
transformer
deep learning
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Local motion blur commonly occurs in real-world photography due to the mixing between moving objects and stationary backgrounds during exposure. Existing image deblurring methods predominantly focus on global deblurring, inadvertently affecting the sharpness of backgrounds in locally blurred images and wasting unnecessary computation on sharp pixels, especially for high-resolution images.
This paper aims to adaptively and efficiently restore high-resolution locally blurred images. We propose a local motion deblurring vision Transformer (LMD-ViT) built on adaptive window pruning Transformer blocks (AdaWPT). To focus deblurring on local regions and reduce computation, AdaWPT prunes unnecessary windows, only allowing the active windows to be involved in the deblurring processes. The pruning operation relies on the blurriness confidence predicted by a confidence predictor that is trained end-to-end using a reconstruction loss with Gumbel-Softmax re-parameterization and a pruning loss guided by annotated blur masks. Our method removes local motion blur effectively without distorting sharp regions, demonstrated by its exceptional perceptual and quantitative improvements (+0.28dB) compared to state-of-the-art methods. In addition, our approach substantially reduces FLOPs by 66% and achieves more than a twofold increase in inference speed compared to Transformer-based deblurring methods. We will make our code and annotated blur masks publicly available.

Published as a conference paper at ICLR 2024
ADAPTIVE WINDOW PRUNING FOR EFFICIENT LOCAL MOTION DEBLURRING
Haoying L1, Jixin Zhao', Shangehen Zhou', Huajun Feng". Chongyi Li & Chen Change Loy! Ss-Lab, Nanyang Technological University, Singapore ZZhejiang University, China Nankai University, China
(lhaoying, (eahr))@zuuuu......
(ZHA00388, sczhou, celoy)entu. edu..
[lichongy]
bechongyl/ennnaa..com @enankai.edu.co
ABSTRACT
Local motion blur commonly occurs in real-world photography due to the mix- ing between moving objects and stationary backgrounds during exposure. Exist- ing image deblurring methods predominantly focus on global deblurring, inadver- tently affecting the sharpness of backgrounds locally blurred images and wast- ing unnecessary computation
sharp pixels, especially for high-resolution im- ages. This paper aims to adaptively and efficiently restore high-resolution locally blurred images. We propose local motion deblurring vision Transformer (LMD- ViT) built on adaptive window pruning Transformer blocks (AdaWPT). To focus deblurring on local regions and reduce computation, AdaWPT prunes unneces- sary windows, only allowing the active windows to be involved in the deblurring processes. The pruning operation relies on the blurriness confidence predicted by a confidence predictor that is trained end-to-end using reconstruction loss with Gunbel-Softmax re-parameterization and pruning loss guided by anno- tated blur masks. Our method removes local motion blur effectively without dis- torting sharp regions, demonstrated by its exceptional perceptual and quantitative improvements compared to state-of-the-art methods. In addition, our approach substantially reduces FLOPs by 66% and achieves more than a twofold increase inference speed compared to Transformer-base deblurring methods. Codes and data are available at https:://////. /tlps////eaal...../ .wt.tgitthh........... 1o/ LMD-VE webpage
INTRODUCTION
Contrary to global motion blur, which typically affects entire image (Zhang et al. 2018). local motion blur is confined to speeific regions within an image. Such local blur generally the re- sult of object movements captured by stationary cameras (Li et al.) 2023 Schelten & Roth 2014). Applying global deblurring methods to images featuring local motion blur inevitably introduces un- wanted distortions in regions that were originally sharp. illustrated in FFiuree T Moreover, the processing of sharp regions, which is not required in this context, leads to unnecessary computa- tional expenditure. This wastage becomes particularly noticeable when dealing with high-resolution inputs. Existing local motion deblurring methods, such as LBAG (Li et al. 2023). address the is- sue by detecting local blur regions for targeted processing. While LBAG
gate structure to mitigate the deblurring impact on non-blurred regions, it still involves unnecessary computations as the entire image is processed by the network. In addition, the method's is reliance Convolutional Neural Network (CNN) architecture leads to limitations, as the interactions between the image and convolution kernels are content-independent and ill-suited for modeling long-rat mang-range dependencies. The Transformer architecture (Vaswani et al.) 2017). which excels at rong-range pixel interactions, has been successfully applied several image restoration problems (Liang et al., 2021a Guo al.,
This paper was completed in S-Lab, Nanyang Technological University.
* Conresponniig author.

---

Published as a conference paper at ICLR 2024
Locally blurred image
Blur annotation
AdaWPT 5
7daWPT
AdaWPT
Sharp region Uformer (global) Ours (adapted)
GT No.
Blur region Uformer (global) Ours (local)
GT No.
We introduce LMD-ViT, Transformer-based local motion deblurring method with an adaptive window pruning mechanism. We prune unnecessary windows based on the predicted blur- riness confidence supervised by our blur region annotation. In this process, the feature maps pruned at varying levels of granularity within blocks of different resolutions. The white masks in AdaWPT 5 to denote tokens be preserved, and regions without white masks are pruned. The grids denote window borders. Unlike global deblurring methods that modify global regions (Wang al., 2022 Zamir et al. 2022), LMD-VIT performs dense computing only on the active windows of blurry regions. Consequently. local blurs are efficiently removed without distorting sharp regions.
Figure 1:
2023 Wang al. 2022 Zamir et al. 2022). However, Transformers for deblurring tasks (Wang ct al. 2022; Zamir et al. 2022) usually conduct attention operations on every token, which not only distorts sharp tokens in locally blurry images (as shown in Figure 1) but also require substan-
tial memory and extend inference time. To mitigate these computational demands, strategies such
channel inter-dependence operations (Liang et al. 2021b), window-division (Wang et al. 2022
Yang et al., 2021) and token-reducing (Bolya et al.) 2022 Liang et al. 2022 Meng et al. 2022 Rao
al. 202 Yin et al. 2022) have been proposed.
Drawing inspiration from preceding work, in this paper, we propose a U-shaped local motion deblur- ring vision Transformer (LMD- ViT) with adaptive window pruning Transformer blocks (AdaWPT) as its core component. Our proposed LMD- VIT is the first to apply sparse vision Transformer to the local motion deblurring task. The core block, AdaWPT, focuses on locally blurred regions rather than global regions, which is made possible by removing windows unrelated to blurred areas, surpassing prior state-of-the-an methods in both deblurring performance and efficiency. Specifi- cally, we first train confidence predictor which is able to automatically predict theblurriness con- fidenceof feature maps. It is trained end-to-end by a reconstruction loss with Humbel-Softman re- parameterization, and pruning loss guided by our elaborately annotated local blur masks. We then design decision layer that provides binary decision maps which "1" represents the kept tokens in blur-related regions that require processing during inference while "0" represents the abandoned tokens in the other regions that can be removed. We also propose a window pruning strategy with Transformer layers. In detail, we apply window-based multi-head self-attention (W-MSA) (Wang) et al. 2022) and window-basee -baed feed-forward layers rather than enforcing these Transformer lay- ers globally. Only the selected windows are forwarded to these window-based Transformer layers, preventing unnecessary distortion of sharp regions while also reducing computations. To further enhance content interactions, AdP employs shifted window mechanism (Liang et al.] 2021a] Liu et al. 2021) and position embeddings (Wang et al.] (022) among the Transformer layers. Fur- thermore, we insert AdaWTP in LMD-Vis under different scales and receptive fields. Therefore, AdaWTP conducts coarse pruning of windows in low-resolution layers and more refined pruning in high-resolution layers, as shown in Figure I] which achieves a balance between computational com- plexity and deblurring performance. Compared to the CNN- local deblurring methods, our pro- posed method achieves improved visual and quantitative performances. Compared to Transformer- based local deblurring methods, speed up inference.
2

---

Published as a conference paper at ICLR 2024
5H E JAN
< sonu Lohtre
S00 LANVEY
10NA LAWKPV
2 LANAP
Whr
2014
S
3 Downy sample
8 Concatenate
Besides
4877
AdaWPT Layer
Figure 2: The architecture of LMD- VTT. LMD-ViT is built on a U-shape encoder-decoder structure with AdaWPT blocks, which prune tokens different resolutions. Each AdaWPT block contains several AdaWPT layers, predicting the blurriness confidence (C₁). the pruning decision (D,) and calculating the pruning loss (Cp). X₁ and X1+1 denote the ith and i + th feature map of each layer, respectively. "AVG", "g", "s" and "Lx" denote the average pooling operation, the pruning threshold in inference, the threshold of AVG, and the reconstruction loss respectively.
To summarize, our main contributions are: 1) the first sparse vision Transformer framework for local motion deblurring. LMD-ViT, which utilizes an adaptive window pruning strategy to focus computation on localized regions affected by blur and achieve efficient blur reduction without caus- ing unnecessary distortion sharp regions; 2) sophisticated blurriness prediction mechanism, integrating a confidence predictor and decision layer to effectively distinguish between sharp and blurry regions; 3) carefully annotated local blur masks for ReLoBlur dataset (Li et al. (023), which improve the performance of local deblurring methods. We explain more background information of our method in Appendix
METHODOLOGY
2.1 MODEL ARCHITECTURE
The architecture of our local motion deblurring vision Transformer (LMDDVVT) is shown the top Figure 2 which a U-shaped network with an encoder stage, a bottleneck stage, and a decoder stage with skip connections. An in-ppojectioonouttrrooii projection layer placed at the beginning/end of the network to extract RGB images to feature maps or convert feature maps to RGB images. The encoder, bottleneck, and decoder include a series of adaptive window-token pruning Transformer blocks (AdaWPT) and down- watsssaallliiuusaaaaaasnn
layers. As a key component, AdaWPT removes local blurs by a window pruning strategy with a confidence predictor, decision layer, and several Transformer layers. It is trained with reconstruction loss (Lx)) and pruning loss (Lp) constrained by our carefully annotated blur masks. AdaWPT can applied in any encoofr/deecode//botton block and is flexible prune at different resolutions. As shown in Figure I in proposed LMD- VIT, windows are pruned coarsely in low-rcsolution blocks and finely in high-resolution blocks. This strikes balance between computational complexity and accuracy. We detail the architectures and model paper-parameters of LMD- VIT in Appendix
2.2 ADAPTIVE WINDOW PRUNING TRANSFORMER (ADAWPT)
As shown at the bottom Figure an adaptive window pruning Transformer block (AdaWPT) includes several AdaWPT layers. During training, each AdaWPT layer comprises a confidence pre- dictor, a decision layer, a feature shift/reverse block, and several Transformer layers such as window- based multi-head self-attention (W-MSA) (Wang al. 2022), window-based locally-enhanced feed- forward layer (W-LeFF), and layer normalization (LN). AdaWPT is responsible for predicting the blurriness confidence by confidence predictor, generating pruning decisions from decision layer and pruning windows. It only allows the unpruned windows to be fed into the window-baa based Trans- former layers, significantly reducing computational costs and keeping sharp regions undistorted.
3

---

Published as a conference paper at ICLR 2024
Besides, feature shift/reverse block is inserted before/after pruning to promote feature interac- tions. In the inference phase, a window compound layer is incorporated to integrate the abandoned windows and selected windows into unified feature map. We detail the modules in AdaWPT in the following paragraphs.
2.3 CONFIDENCE PREDICTOR
The confidence predictor predicts the blurriness confidence for the input features X E R" Tokens with higher confidence more likely to be kept and others are removed. Following Rao et al. 2202), the confidence predictor employs MLP layers to produce feature embeddings e and predict the confidence map C using Softmax:
EW C; = Sooftax(ess,, 6 = Concat (MLP(MLP(X;). yyo D, MLP(X.)) ) N+, Σ Dj
(1)
where D, initialized using is the one-hot decision map to prune windows, which will be introduced in Section 2.4. i denotes the 7th AdaWPT block. The confidence predictor is trained using an end- to-end reconstruction loss (introduced in Section 2.7 with Gumbel-Softmax parameterization (see Section 2.4.. together with pruning loss (introduced in Section 2.7 guided by our blurry mask annotations (introduced in Section 2.6
2.4 DECISION LAYER
The decision layer samples from the blurriness confidence map to generate binary pruning decisions D. in which "I" represents tokens to be kept while "0" represents tokens to be abandoned. Although our goal is to perform window pruning, it is not practical to remove the zero tokens directly, for the absence of tokens halts backward propagation, and the different removal instances make parallel computing impossible in end-to-end training. To overcome this issue, we design the decision layer for training and testing, respectively.
In training, we apply the Gumbel-Softmax re- ce-parameterization (Jang et al. 2017) as the decision layer, since it assures the gradients to flow through the network when sampling the training decision map Der from the training confidence map C
D tr Humbel-Softmax (C
(2)
In testing, we apply Softmax with constant threshold the decision layer:
0. Softmax (C (x., v))) < Ø
(.,
(3)
Softmax (Cte (z, y))) 2 8,
where (x,y) denotes coordinate point. The abandoned windows, that is, windows irrelevant to local blurs, are further set to zero by
X = D Xt.
(4)
2.5 EFFICIENT TRANSFORMER LAYERS WITH WINDOW PRUNING STRATEGY
Considering the quadratic computation costs with respect a large number of tokens of high- resolution images, we employ Transformer layers in non-overlapping windows rather than in global manner accelerate training and inference. Specificallyy, we choose W-MSA layer (Wang et al. 2022) for the self-attention (SA) operation. For the feed-forward structure, we develop window-based locally-enhanced ferw-forward layer (W-LeFFF) as an alternative to LeFF (Wang et al.) 2022) which modifies global tokens. W-LeFF uses 3 x convolution with stride I and reflected padding 1 in independent windows. The reflected padding ensures the continuity at the window borders and obtains almost the same performance LeFF (Wang et al. 2022) (we compare the performance of LeFF (Wang et al. 2022) and W-LeFF in Appendix E.I
To enable parallel computing, regard each window as token group and prune based on windows.
Each window owns a 0/1 decision, which is calculated by average pooling the decision map with a threshold Windows with average
regarded as the kept windows and the others as the
4

---

Published as a conference paper at ICLR 2024
abandoned windows. To promote content interactions among non-overlapping windows, also apply relative position encoding (Wang et al. 2022) in the attention module and shift/reverse the windows by half the window size the beginning/end of the AdaWTP.
To accomplish both diffcrentiable training and fast inference, we propose different pruning strate- gies with W-MSA and W-LeFF in training and testing, respectively. In training, ensure back- propagation, all the windows including the abandoned windows go through W-MSA, W-LeFF, and LN sequentially to generate training features X X
X W-MSA(LN(X)")) + x!",, X W.LeFF(LN((X), ')) +
(S)
In testing. only the kept windows processed by Transformer layers, which release a great number of unnecessary tokens to deblur. To enable future stages to perform global operations, we mend the abandoned windows to their original locations to compound complete testing feature map X
X W-MSA(LN((X
2 8)) + X X W-LCFF(LN(X1:: 22s) X
(6)
where denotes the threshold in the average pooling operation (AVG).
2.6 BLUR REGION ANNOTATION
To obtain the ground-truth local blur mask for supervised training of the confidence predictor, we carefully annotate local blur masks for the ReLoBlur dataset (Li et al. 2023). Firstly, roughly identify the blurry regions by subtracting the blurry image and its corresponding sharp image to ob- serve the difference. This difference map helps us estimate the location of blurriness. Subsequently, we utilize ElSeg software to isolate moving objects and automatically segment their components. We then undertake manual refinement, guided by a principle: any diffuse spot encompassing more than 5 pixels is categorized as blurry, while those encompassing fewer are classified as sharp. To mitigate any potential leakage, we extend the blurry region by pixels ensure comprehensive cov- erage within the annotated mask. In our annotated blur masks, a pixel value of denotes blurriness, while a value of signifies sharpness, as shown in Figure I and |Fiuure 6
2.7 LOSS FUNCTIONS
To guide adaptive window pruning and locally deblurring, propose pruning loss and combine it with a weighted reconstruction loss to form the total loss:
f Lp + LR-
Pruning loss. We propose a pruning loss constrain the blurriness confidence prediction:
Cr do Σ Cross-Entropy(C (C,, Down-Sample" (M)),
(8)
where 00=0.01, and the blur mask M is down-sampled to match the resolution of the confidence maps, and i indicates a confidence predictor's location.
Reconstruction loss. To focus training local blurred regions, following Li et al. 2023), apply weights w on the reconstruction loss, which is combination of L1 loss, SSIM loss, and FFT loss:
LR WL (M- S', M S) + (1 - w)
S),
C = Ls S) + A >FFFT((
(9)
where S, and S" denote the sharp ground truth, and the deblurred output. w = 0.8, >== 100, and >3=0.1 denote the weights of the loss functions.
EXPERIMENTS AND ANALYSES
3.1 EXPERIMENTAL SETTINGS
We train LMD-ViT using AdamW optimizer (Kingma & Ba 2015 with the momentum terms of
(0.9, 0.999), batch size of 12, and an initial learning rate 2 x 10 that updated every 2k steps
EXSeg is provided http://hhhhcocooaaaaaaaaalaalale
5

---

Published as a conference paper at ICLR 2024
blur
(a) / 22.95dB (b) / 23.21dB
(c) 27.77dB
(d) / 26.78dB
Locally Blurod Image
(e) 24.31dB
(0) 25.26dB
(g) 25.01dB
(h) 28.BBBBBBB
GT
blur
(a) / 24.58BB
(b) 24.88dB
(c) 23. 23.40dB
(d) 25.28dB
'SRON
Locally Blured Image
(e) // 88dB (0) 28.84dB (g) / 29.72dB (h) 30 16dB
GT
Figure 3: Visual comparisons of state-of-the-aa methods for local motion deblurring. (a) DeepDe-
blur Nah et 2017); (b) DeblurGAN. v2 (Kupyn et al., 2019); (c) HINet (Chen al. 202 x (d)
MIMO-UNet (Cho et al. 202T); (e) LBAG (Li et al. 2023): (f) Restormer (Zamir et al.
2022 (g)
Uformer (Wang et al. 2022); (h) LMD-ViT (ours).
Ours
LBAG
Restormer
Uformer
Restormer
LBAG
0%
10%
30%
40%
50%
Locally Blurred Image
Uformer
Ours
Percentage
Figure 4: Results of user study for real-world local motion deblurring. The horizontal axis indicates the percentage of favoring each method.
by cosine annealing schedule [Loshchilov Hutter 2017). We set the window size of AdaWP
to 8 x 8, and the initial embedded dim to 32 which doubled after passing each down-sampling layer. LMD-ViT is trained on the GoPro dataset (Nah et al. 2017) and ReLoBlur dataset (Li et al., 2023) together, in order to enable both local and global motion deblurring. The sampling ratio of
GoPro training data (Nah et al. 2017) and the ReLoBlur training data (Li et al. [0023] close to 1:1. For fair comparison, train the baseline methods using the same datasets and cropping strategy. The model configurations of the compared deblurring methods follow their origin settings.
We evaluate our proposed LMD-ViT and baseline methods the ReLoBlur testing dataset (Li et al., 2023) with the full image size of 2152x 1436 on Nvidia A100 GPU. In testing, the inputs solely of locally blurred images without accompanying blur masks. In addition to the commonly- used PSNR and SSIM (Wang et al.) 2004) metrics, we calculate weighted PSNR and weighted SSIM (Li et al.
2023) better assess the local deblurring performance. We provide the evaluation results in the following sections and Appendix
3.2 EXPERIMENTAL RESULTS
Evaluations on public datasets. We first compare the proposed LMD- VIT with both baN-based
methods (Nah et al. 2017 Kupyn et al. 2019 Chen et al. 2021 Cho al. 2021
Li et al. 2023)

---

Published as a conference paper at ICLR 2024
Table I: Quantitative comparisons on the local deblurring dataset. "PSNR " "SSIM,", "Time" and "Params" denote weighted PSNR, weighted SSIM, inference time, and model parameters re- spectively. We bold the best result under each evaluation metric.
Categories
Methods
PPSNR 1SSIM *PSNR...
1SSIM
Time Params FLOPs
DeepDeblur Nah et al. 2017)
33.02
0.8904
28.29
0.8398
0.50s 11.72M 17.133T
DeblurGAN-v2 Kupyn
2019
33.26
0.8975
28.29
0.8489
0.07s 5. 5.076M 0.989T
CNNs
HINet Chen al. 2021
34.40
0,9160
28.82
0.8672
0.31s 88.67M 8.696T
MIMO-UNet (Cho el al. 2021
34.64
0.9247
29.17
0.8766
0.51s 16.11M 7,850T
LBAG (Li ct al. 20231
34.91
0.9265
29.26
0.8788
0.51ls 16.11M 7,852T
Restormer
Zamir el al.
2022
34.85
0.9271
29.28
0.8785
3.72s 16.
6,741T
Transformers
Uformer-B B (Wagg al.
2022)
35.14
0.9277
29.92
0.8865
1.31s 50.88M 4.375T
LMD-
35.42
0.9289
30.25
0.8938
0.56s 54.50M 1.485T
Input
w/o pruning
w/ pruning
GT
Figure 5: Evaluations for sharp region quality preservation with or without the pruning strategy. Without pruning, the network modifies global pixels and blurs the sharp backgrounds. In contrast, with pruning, sharp backgrounds are preserved well.
and transformer-basee methods (Zamir et al., 2022, Wang et al., 2022) on the ReLoBlur dataset LLi al.) (023) for local motion deblurring. As depicted in Figure and Figure 3] LMD-VVT exhibits superior performance compared to other state-of-the hanced details. Notably, the white stripes on the student's suit and the mobile phone show significant blur reduction without artifacts and closely resemble the ground truth. We notice that the number of preserved windows slightly exceeds the number of windows in the annotated blurry areas. This is to ensure that the preserved windows effectively cover much of the blurry area as possible. Quanti- tative evaluation results are presented in Table I which bolds the best local deblurring performance. Compared with CNN-based methods, our proposed LMD-ViT achieves improvement of 0.50 dB in PSNR and 0.95 dB in weighted PSNR, while maintaining a comparable or even faster inference speed. Compared with Transformer-based methods, LMD-VVT demonstrates significant reductions (-66% in FLOPs and inference time without sacrificing performance (PSNR +0,28dB), thanks to our adaptive window pruning modules. The PSNR and SSIM scores of LMD- ViT also exceed other Transformer- based deblurring methods, because our proposed pruning strategy prohibits the net- work from destroying sharp regions. However, global deblurring methods (i.e., Uformer (Wang et al. (202)) and Restormer (Zamir et 2022)) treat every region equally and may distort or blur the sharp regions inevitably.
methods, producing clearer outputs with en-
Additionally, we assess the global deblurring capabilities of LMD-VIT on the GoPro testing dataset (Nah et al. 2017) in Appendix C and present more comparisons between our proposed method and baselines with local blur masks in Appendix D
User study on real-world photos. To validate the effectiveness of our proposed model in real- world locally blurred images, we employ static Sony industrial camera and static Fuji XT20 SLR camera to capture 18 locally motion-blurred images, each with a resolution of x000 pixels. Subsequently, we conduct comparative evaluation against the top 3 methods as listed in Table As ground truths for these blurred images are unavailable, we organized a user study involving 30 par- ticipants with keen interest in photography. Each participant is presented with randomly selected
7

---

Published as a conference paper at ICLR 2024
Table The effectiveness of window pruning. "PSNR.. "SSIM," "Time", and "Params" denote weighted PSNR, weighted SSIM, inference time, and model parameters, respectively. When the pruning block is none, LMD-Vit does not prune and preserves all tokens.
No.
Pruning block
8
PPSNR
1SSIM
PPSNR...
1SSIM..
Time
Params
FLOPs
AdaWPT 1~9
0.5
35,42
0.9289
30.25
0.8938
0.56s
54.50M
1.485T
2
AdaWPT 2--8
0.5
35.41
0,9289
30.34
0.8928
0.70s
53.94M
1.592T
3
AdaWPT 3~7
0.5
35.44
0,9290
30.38
0.8936
0.77s
53,49M
2.021T
4
AdaWPT 4-66
0.5
35.37
0.9293
30.39
0.8931
1.07s
53.06M
3.105T
5
None
0.5
35.36
0.9280
30.22
0.8930
1.30s
50.39M
4.376T
6
AdaWPT I~
0.2
35.36
0,9289
30.21
0.8931
0,958
54.50M
1911T
7
AdaWPT 1~/9
0.3
35.41
0,9291
30.28
0.8934
0.80s
54,50M
1.61TT
8
AdaWPT 1~-9
0.4
35.43
0,9290
30.28
0.8935
0.69s
54,50M
1,556T
AdaWPT 1~
0.6
35.35
0,9284
30.18
0.8922
0.49s
54,50M
L.05TT
10
AdaWPT 1~99
0.7
35.32
0,9281
30.14
0.8918
0.43s
54.50M
1.327T
Table 3: The effectiveness of blur mask annotation. "PSNR.. and "SSIM,," " denote weighted PSNR, and weighted SSIM, respectively. Methods with "*" are trained with LBFMG's blur mask annotations 2. Methods without "*" are trained with blur mask annotations.
No.
Methods
1PSNR
1SSIM
1PSNR...
1SSIM...
LBAG* (L et 2023
34.83
0.9264
28.31
0,8711
LBAG
LI
el al. 2023
34.91
0.9265
29.26
0.8788
VID
35.31
0.9270
30.14
0.8911
LMD-VIT
35.42
0.9289
30.25
0.8938
deblurred images and tasked with selecting the most visually appealing one. The results, depieted in Figure 4] demonstrate that our proposed method exhibits robust performance on real-world locally motion-blurred images, showcasing sharper reconstructed edges and consistent content. Notably, emerges as the preferred choice among participants when compared to alternative approaches. Further details of the user study are provided in Appendix F2
3.3 ANALYSES
To examine the effectiveness of our proposed method, we conduct separate analyses focusing on the window pruning strategy and blur mask annotations in this section. Due to space constraints, details of pruning accuracy, and analyses of W-LeFF, the number of feature channels, joint training strategy, input resolutions, and special cases are presented in Appendix B
3.3.1 WINDOW PRUNING STRATEGY
We analyze the effects of window pruning strategy in three aspects: the number of pruning blocks, the pruning threshold 3 during inference, and pruning accuracy.
The number of pruning blocks. We fix the pruning threshold 9 0.5 and change the number of pruning blocks. For blocks without pruning, we force the decision maps D in Equation 4 to be the all-one matrix. From line line 5 of Table 2] we find that pruning more blocks results in fewer model parameters, higher inference speed, and more dropping scores. Notably, although we apply window pruning mechanism in all 9 blocks, the scores outperform other baseline models listed in Table Additionally, we conduct visual comparisons of all all-pruned architecture (line 1) with non-pruning architecture (line 5), as shown Figure 5] the all all-pruned network, most of the sharp regions pruned and are not destroyed. In contrast, the sharp regions turn blurry distorted when processed by the non non-prunce non-pruned network. This indicates that our adaptive window pruning mechanism prevent the sharp regions from being destroyed. The non-pruning network treats every region equally like global deblurring networks (e.g.. MIMO-UNet, (Cho et al. (202)). Uformer (Wang et al. 2022) and Restormer (Zamir al. 2022 that may harm the sharp regions inadvertently.
Pruning threshold 3. We fix the pruning blocks and adjust the pruning threshold 3 from 0.2 to 0.7 with 0.1 as the interval. Comparing line lines 6 to line 10 in Table find that the testing
8

---

Published as a conference paper at ICLR 2024
Locally bluned
LBFMG's annotations
Our annotations
Productions W LBFMG
Productions W OUTS
Ddblur LBFMG
D OUnS
GT
Figure 6: Visualizations of LMD-ViT's blurriness confidence map trained with different kinds of annotations. LMD-ViT predicts better blurriness confidence maps with our annotation than those produced with the annotations provided by LBFMG (Li et al. 2023
performance slightly varies with different pruning thresholds 3, resulting in different confidence levels and decision boundaries. Inferring with a lower nor a higher pruning threshold is neither reasonable, because the former makes the confidence predictor more inclusive, which potentially leads to fewer sharp windows to be pruned and a slower inference speed, while the latter makes the confidence predictor more conservative, which leads faster inference speed but filters the necessary blurry windows. To achieve balance, we choose 3 = 0.5 as it obtains relatively high evaluation scores and fast inference speed.
Pruning accuracy Our proposed AdaWPT enables blur region selection as well as removing sharp windows. From the first pruning block (AdaWPT 1) the pruming block (AdaWPT 9), the pre- served patches gradually correspond to the ground-truti blur annotations, shown in Figure To assess the local motion deblurring capability of our proposed LMD-ViT, we measure the mask pre- diction accuracy on the ReLoBlur test dataset using the formula: accuracy = TP+TN where TP the count of correctly identified blurry pixels, TNN represents accurately identified sharp pixels, and N stands for the total pixel count. Our model achieves peak accuracy of 94.51% among var- ious Ada WPT blocks, which predominantly reflects an effective recognition of local motion blurs. Notably, the accuracy rises from AdaWPT I to 8, while AdaWPT 9's accuracy dips possibly due to its skip connection with Ad 1. We detail the accuracies all blocks in Appendix E.5
3.3.2 BLUR MASK ANNOTATION
To verify the effectiveness our blur mask annotations, we conduct experiments on local deblur- ring method LBAG (Li et al. 2023) and our proposed LMD-VIT, as illustrated in Table Comparing line and line 2. we observe that our manually annotated blur masks enhance the evaluation scores. This observation is further supported by the comparison between line 3 and line 4. We provide visualizations mask predictions and deblurred results of LMD-ViT in Figure 6, is evident that LBFMG's² 2 masks lead to more recognition errors of local blurs. This is due the presence of holes and the noise in LBFMG's2 masks (Li et al. 2023). which may confuse AdaWTP in predicting masks and selecting blurry tokens. However, our smooth and continuous annotations greatly im- prove the performance of blur mask predictions, thus aiding LMD- ViT in presenting more promising deblurring results.
CONCLUSION
We have presented adaptive and efficient approach, LMD-VIT, a sparse vision Transformer for restoring images affected by local motion blurs. LMD-ViT is built upon our novel adaptive window pruning Transformer blocks (AdaWPT), which employ blur-aware confidence predictors estimate the level of blur confidence in the feature domain. This information is then used to adaptively prune unnecessary windows low-confidence regions. To train the confidence predictor, we designed an end-to-end reconstruction loss with Gumbel-Softmax prppaamtterization along with pruning loss guided by our carefully annotated blur masks. Extensive experiments demonstrate that our method effectively eliminates local motion blur while ensuring minimal deformation of sharp regions, sulting a significant improvement in image quality and inference speed. Due to limited page space, discuss the limitations and broader impacts in Appendix G
1.BFMG is automatic blur mask annotation method proposed by Li 2023

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This study supported under the RIE2020 Industry Aligmment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contributions from the industry partner(s). This study is also funded by the China Scholarship Council.
REFERENCES
Daniel Bolya, Cheng Yang Fu, Xiaoliang Dai, Peizhao Zhang. Christoph Feichtenhofer, and Judy Hoffman. Token Merging: Your vit but faster. In ICLR, 2022.
Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. HINet: Half instance nor- malization network for image restoration. CVPR, 2021.
Sung-Jin Cho, Seo-Wae Won Ji, Jun-Pyo Hong, Seung- Won Jung, and Sung-Jea Ko. Rethinking coarse- to-fine approach in single image deblurring. In ICCV, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Lanqing Guo, Siyu Huang, Ding Liu, Hao Cheng, and Bihan Wen. Shadowformer: Global context helps image shadow removal. In AAAI, 2023.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015.
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jifí Matas. Deblur- gan: Blind motion deblurring using conditional adversarial networks. In CVPR, 2018.
Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. DeblurGAN-v2: Deblurring (orders-of-magnitude) faster and better. In ICCV, 2019.
Haoying Li, Ziran Zhang, Tingting Jiang, Peng Luo, and Huajun Feng. Real-world deep local
motion deblurring. AAAI, 2023.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. SwinIR: Image restoration using swin transformer. In ICCVW, 2021a.
Jingyun Liang, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Mutual affine network for spatially variant kernel estimation in blind image ruper-resolution. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4096-4100 2021b.
Youwei Liang, Chongjian Ge, Zhan Tong. Yibing Song, Jue Wang, and Pengtao Xie. Not all patches what you need: Expediting vision transformers via token reorganizations. In ICLR, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hicrarchical vision transformer using shifted windows. In ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. AdaViT:: Adaptive vision transformers for efficient image recognition. In CVPR, 2022.
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017.
Yongming Rao, Wenliang Zhao, Benlin Liu. Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In NewrIPS, 2021.
10

---

Published as a conference paper at ICLR 2024
Wenqi Ren, Jiawei Zhang, Jinshan Pan, Sifei Liu, Jimmy Ren, Junping Du, Xiaochun Cao, and Ming--ssun Yang. Deblurring dynamic scenes via spatially varying recurrent neural networks. TPAMI, 2021.
Kevin Schelten and Stefan Roth. Localized image blur removal through non-parametric kernel estimation. In ICCV, 2014.
Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In CVPR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In CVPR, 2021.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. ICCV, 2021.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In CVPR, 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-
612. 2004.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. In NeurlPS, 2021.
Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-VIT: Adaptive tokens for efficient vision transformer. In CVPR, 2022.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
Jiawei Zhang, Jinshan Pan, Daoye Wang. Shangchen Zhou, Xing Wei, Furong Zhao, Jianbo Liu, and Jimmy Ren. Deep dynamic scene deblurring from optical flow. IEEE Transactions Circuits and Systems for Video Technology, 32(12)8220-82200 2021.
Shanghang Zhang, Xiaohui Shen, Zhe Lin, Radomír Měch, Joao P Costeira, and José MF Moura. Learning to understand image blur. In CVPR, 2018.
Shangchen Zhou, Chongyi Li, and Chen Change Loy. LEDNet: Joint low-light enhancement and deblurring in the dark. In ECCV, 2022.

---

Published as a conference paper at ICLR 2024
A RELATED WORK
Single image deep motion deblurring. The task of deep motion deblurring for single images orig-
inated from global deblurring (Zhang et al., 2018 Ren et al., 2021 Tao et al., 2018 Zamir et al.)
2021 Zhang al., 2021 Zhou et al., 2022). Pioneering deep global motion deblurring works utilize CNNS as basic layers and achieve promising improvements in image quality. Among them, Deep- Deblur (Nah et al. 2017). a multi-scale convolutional neural network. performs residual blocks to increase convergence speed. DeblurGAN (Kupyn et al., 2018) and DeblurGAN-v2 (Kupyn et al. 2019) introduce GANs and perceptual loss to improve subjective quality. HINet (Chen et al, 2021 applies Instance Normalization to boost performance. Recently, a CNN-based local motion deblurring method, LBAG (Li et al.] (023), bridges the gap between global and local motion de- blurring by inserting gate modules at the end of MIMO-UNet architecture (Cho et al.) (021). It predicts differentiable blur masks to reduce sharp backgrounds from modifications and guide the network to deblur locally. There are also CNN- based methods that involve spatially variant predic- tions for blind image reconstruction (Liang al., 2021b). Although the performance significantly improved, CNN-based methods suffer from the content-indepenel and convolution kernels, as well as the limitations of long-range dependency modeling.
interactions between images
Given the Vision Transformer's ' (ViT) (Dosovitskiy et al. 2021) ability to capture dencies, its application to global deblurring tasks has seen a surge of interest. For example, Uformer (Wang et al. 2022) employs window-based self-attention with learnable multi-scale restoration modulator to capture both local and global dependencies. Restormer (Zamir et al.) 2022) utilizes multi-head attention and feed-forward network to achieve long-range pixel interactions. In this local motion deblurring framework, LMD- ViT, that adaptively selects windows relevant blurry regions for window-based self-attention and feed-forward opera- tions, simultaneously benefiting from long-range modeling.
depen-
paper, we build a Transformer-base
Vision Transformer acceleration. Transformers have proven valuable in deblurring tasks, yet their direct application in local motion deblurring for high-resolution images presents challenges con- cerning computational efficiency. To solve the heavy computation problem of global self-attention in Transformers, researchers have presented several techniques. For example, Wang et al. adopted pyramid structures and rpatial-reduction attention (Wang al. [021] in image classification, object detection, and segmentation tasks. Some methods partition image features into different windows and perform self-attention on local windows (Vaswani et al,, 2021 Yang al., [202 Waan et al.) 2022) for image restoration tasks. Some image classification methods gradually reduce tokens in processing by hakennhalting (Meng et al. 2022 Rao et al. 2021 Yin et al. 2022) or token-merging
(Liang al., 2022 Bolya et al. 2022). Inspired by these advancements, we develop adaptive window pruning blocks (AdaWPI) to eliminate unnecessary tokens and focus deblurring only on blurred regions, which improves image quality and enables inference speed-up without compromis- ing sharp regions.
B LMD-VIT ARCHITECTURE AND MODEL
As shown in Figure 2 of our main paper, our proposed LiD-ViT is U-shaped network with one in-projection layer, four encoder stages, one bottleneck stage, four decoder stages, and one out- projection layer. Skip connections are set up between the encoder stage and the decoder stage. A lo- cally blurred input image B E I with a shape // x W x3 firstly goes through an in-projection block, which consists of 3 3 convolutional layer, LeakyReLU layer, and a layer normalization block, to extract low-level features as feature map X E R. The feature map then passes four encoder stages, each of which includes a series of AdaWPT Transformer blocks and one down- hown-sampling layer. AdaWPT uses a blur-aware confidence predictor and Gumble-Softmax parammtterizztionn to select blur-related tokens. Only the selected tokens are forwarded to Transformer layers includ- ing window-based self-attention (W-MSA), window-ban -aase LeFF), and layer normalization (LN). The wwwn-sampling layer down-sample the feature map size by 2 times and doubles the channels using 4 x 4 convolution with stride 2. The feature map's shape turns to H W bottleneck, where it can sense the longest dependencies in two AdaWPTs. After the bottleneck stage, the feature map goes through four decoder stages, each of which owns an up-sampling layer and a series of AdaWTP blocks. The up-sampling layer uses 2 2 transposed convolution with
-eed-forward layer (W-
2: x 2 x 3, E 2,, 4} after encoder stages, and has the smallest resolution in the
12

---

Published as a conference paper at ICLR 2024
sphphhall
Fizzetutt Pizza Hut
BUAH
TELL
TUAN
ABA
ABA
ABA
Globally blurred image
Blurred
Deblurred by LMD-VIT
Decision map
Figure 7: Global deblurring performance of our proposed LMD- ViT. The decision map denotes the pruning decisions. With a globally blurry input, the predictedblurrness confidenceapproaches 100%. Therefore, our network preserves all windows and nearly all the values in the decision map are equal to 1, which is represented as an all-white color.
Table 4: Quantitative comparisons the global deblurring dataset. All the methods trained with the Rel.ollur dataset (Li et al. 2023) and the GoPro dataset (Nah er al. 2017 together.
Evaluation metrics
Restormer Zamir
2022
Uformer-B Wang et
2022
LMD- ViT (ours)
PNSR
32.15
32.51
32.16
SSIM
0.9305
0.9376
0.9318
stride to reduce half of the feature channels and double the size of the feature map. The features put into the AdaWPT blocks are concatenations of the up-sampled features and the corresponding features from the symmetrical encoder stages through skip connections. Finally, the feature map passes the out-projection block which reshapes the flattened features to 2D feature maps and applies a 3 3 convolution layer to obtain a residual image R. The restored sharp image S' obtained by S' B ++R. R.
C GLOBAL DEBLURRING PERFORMANCES
Our proposed LMD-ViT integrates both local and global motion deblurring. To demonstrate its capability in global deblurring, we evaluate our model on the GoPro testing dataset (Nah et al. 2017). With a globally blurry input, the predicted blurriness confidence approaches 100%. Conse- quently, the decision layer of LMD-Vi VIT retains all windows, depicted in white in the 5th column of Figure We also compare our proposed models with other state-of-the-art Transformer-based deblurring methods in Table 4] The scores of our method slightly lag behind Uformer (Wang et al.) 2022) due to the utilization of window pruning strategy in LMD- VIT, which does not achieve 100% accuracy. As result, some tokens may miss deblurring manipulations in certain blocks, leading to less clear outputs. The visual comparison on the GoPro testing dataset (Nah et al. 2017) shows that LMD-ViT obtains comparable performance the state-of-the- global deblurring Transformers, as shown in |Figure 8. Our proposed method can effectively climinate global blurriness and restore sharp textures, showcasing its competitiveness against other baseline methods.
13

---

Published as a conference paper at ICLR 2024
Blurred
Restormer
Uformer
Ours
GT
Blurred
Restormer
Uformer
Ours
GT
Figure 8: Visual results of global motion deblurring.
Table Quantitative comparisons between LMD- VIT and baselines with local blur masks. "PSNR,", and "SSIM," denote weighted PSNR, and weighted SSIM, respectively. We bold the best result under each evaluation metric.
Methods
PPSNR
1SSIM
1PSNR
SSSIM
Inference Time
Restormer w/ mask
34.88
0,9259
29.44
0.88055
3.75s
Uformer-B w/ mask
35.23
0.9277
30.22
0.8935
1.33s
LMD-VIT (ours)
35.42
0.9285
30.25
0.8938
0.56s
D COMPARISONS BETWEEN LMD- VIT AND BASELINES WITH LOCAL BLUR MASKS
In Table I, both LBAG (Li et al. 2023) and our proposed LMD-ViT are trained utilizing local blur masks, whereas the baseline models, including Restormer (Zamir et al. 2022) and Uformer Wang et al.) 2022), are not originally designed with this feature. To ensure fair comparison, the top three baselines mentioned in Table I with local blur masks. Given that these baselines' architectures lack dedicated mask input, integrate the gated block from LBAG (Li al. 2023),
retrain
which is local blur-aware technique. Moreover, we apply our method's reconstruction loss (as detailed in Equation 9 to Uformer (Wang al., 2022) and Restormer (Zamir results in Table reveal that upon training with local blur masks, Uformer and Restormer indeed witness improvements in their PSNR and SSIM scores. Despite this enhancement, their performance still trails behind ours, and they require longer inference time. This substantiates the efficacy of our backbone and the adopted window pruning strategy.
2022). The
E FURTHER DISCUSSION OF ABLATION STUDY
E. THE EFFECTIVENESS OF W-LEFF
To save computational costs as well as inference time, we apply a window-based local-enhanced feed-forward layer (W-LeFF) instead of a locally-enhanced feed-forward layer (LcFF) (Wang al. 2022]. W-LeFF, only the non-pruned windows are processed by a feed-forward mechanism. To fair comparison between W-LeFF and LeFF, evaluate them the all-pruning LMD- VIT network architecture separately. As shown in Table 6 we observe that W-LeFF achieves nearly identical performance LeFF. Hence, substituting LeFT with W-LeFF does compromise the local motion deblurring capabilities while simultaneously accelerating the inference speed.
E.2
THE EFFECT OF FEATURE CHANNELS
The number of feature channels also affects the ability of neural networks. With a larger number of feature channels, neural network can capture more intricate and complex relationships in the input data, resulting in better performance. To verify the capability of our proposed network architecture, we train our network with dimensions
and 32 respectively, and compare it with CNN-based LBAG (Li et al.) 2023) with aligned model parameters, as shown in Table 7] The comparison between line and line shows an improvement with increased feature channels LMD-ViT because larger number of feature channels provides larger number of meaningful features or attributes, which can be beneficial for window selection and feature manipulation. The comparison between line 2 and line 4 implies that, under approximate model parameters, with the adaptive window pruning mechanism is more suitable for the local motion deblurring task with better evaluation scores, fewer model parameters, and faster inference speed.
proposed model
14

---

Published as a conference paper at ICLR 2024
Table 6: The effectiveness of W-LeFF in the proposed LiD-ViT, "PSNR,.. "SSIM, ", "Time", "Params", and "FLOPs" denote weighted PSNR, weighted SSIM, inference time, model parameters, and model complexity, respectively.
No.
Methods
Pruning block
PPSNR
1SSIM
1PSNN...
1SSIM..
Time
Params
FLOPs
LMD-ViT w LeFF
AdaWPT 179
35.45
0.9288
30.24
0.8935
0.86s
54,50M
1,485T 54.50M
LMD-VIT W-LeFF
AdaWPT 1"9
35.42
0.9285
30.25
0.8938
0.56s
1.485T
Table 7: The effect of feature dimension in the proposed LMD-Vi VIT. "Feature channels", "PSNR.." "SSIM,." ", "Time", "Params", and "FLOPs" denote the feature channels of each block, weighted PSNR, weighted SSIM, inference time, model parameters, and model complexity respectively.
No.
Methods
Feature channels
tPSNR 1SSIM *PSNR g
1SSIM, Time Params FLOPs
LBAG (Li 2023
32-64-128 60-120-240
34.91 0.9265
29.26
0.8788
0.51s 16.11M 7.852T
2
LBAG-Large [L. el al. 2023
34.93 0.9266
29.36
0.8871
1.13s 56,58M 17.659T
3
LMD-VVT-Smal
16-22-64-128-256
34.98 0.9259
29.89
0.8907
0.23s
21,59M 0.311T
4
LMD-ViT
32-64-128-256-512
35.42 0.9285
30.25
0.8938
0.56s 54.50M
1,485T
E.3 THE EFFECTIVENESS OF THE JOINT TRAINING STRATEGY
We train proposed LMD-ViT and baseline methods with the ReLoBlur dataset (Li et al. 2023)
and the GoPro dataset together mainly for two reasons. Firstly, joint training could prevent our model from over-fitting and improve the model's robustness. Secondly,
expect LMD- VIT both deblur globally and locally. Training with the ReLoBlur dataset (Li et al. 2023)
proposed
and the GoPro dataset (Nah et al. 2017) together improves both the local and global deblurring performances, as shown in Table 8
E.4 LOCAL DEBLURRING RESULTS UNDER DIFFERENT RESOLUTIONS
We enrich our findings by including results for various resolutions. Employing the nearest interpo- lation, Nown-sample the ReLoBlur testing data and conduct comparisons between proposed model and the top 2 baseline methods Table 9) We find that the local deblurring results of our method and the baselines drop. However, our proposed method outweighs baseline methods in terms of PSNR and SSIM scores. Moreover, it achieves fast inference in low-resolution images.
E.5 MASK PREDICTION ACCURACY, PRECISION AND RECALL
To further evaluate the local motion deblurring ability of our proposed LMD-ViT, we gauge the mask prediction accuracy, precision and recall of our model on the ReLoBlur testing dataset using the formulas:
TP + TN N
accuracy
TP
precision TP + FP TP
(10)
recall TP + TFN
where TP denotes the number of pixels correctly predicted as blurry, TN signifies the count of pixels accurately predicted as sharp, FN denotes the count of blurry pixels predicted as sharp, and N represents the total pixel count. The mask prediction accuracies, precisions and recalls exhibit variability across different AdaWPT blocks, as shown in Figure 9 Most AdaWPT blocks exhibit high prediction accuracies, indicating the effective recognition of local blurs by our proposed model. The first two learnable blocks show comparatively lower accuracies and precisions. The accuracy and precision of AdaWPT 9 drops, potentially influenced by its skip connection with Ada ddT 1.
15

---

Published as a conference paper at ICLR 2024
Table The effectiveness joint training with the ReLoBlur dataset (Li al. 2023 and the GoPro dataset (Nah et al. 2017). We evaluate in terms of the PSNR and SSIM metrics.
Training data
Training mask
Testing data
No.
ReLoBlur ReLoBlur + GoPro LBFMG's mask" Our mask
ReLoBlur
GoPro
35.24 dB 0.9283
31.60 dB 0,9252
2
35.29 dB 0.9280
31.74 dB 0,9271
35.31 0.9270
32.14 dB / 0.9217
4
35.42 dB 0.9285
32.16 dB 0.9318
Table 9: Comparisons for local motion deblurring under different resolutions. We test our proposed LMD-ViT with baseline methods with down-sampled ReLoBlur testing data. LMD-ViT signifi- cantly outweighs other methods and achieves efficient inference. "PSNR,, "SSIM and "Time" denote weighted PSNR, weighted SSIM, and inference time respectively. "g" denotes the down- sampled ReLoBlur testing dataset.
Image Resolution ReLoBlur*
Methods
TPSNR
1SSIM
*PSNR..
1SSIM
Time
FLOPs
Restormer Zamir et al.
2022
34.28
0.9130
29.01
0.8510
0.99s
1.723T
eL al. 2023
Uformer Wang al. 2022)
33.89
0.9053
28.65
0.8424
0.36s
1.158T
(1076 x 718)
LMD-ViT (ours)
34.59
0.9176
29.39
0.8586
0.23s
266.074G
Rel.oBlur
Restormer (Zamir et 2022)
33.10
0.8903
28.53
0.8376
0.26s
450.210G
[Li et al. 2023
Uformer Wang et al.
2022
32.88
0.8747
28.38
0.8293
0.25s
321.657G
(538 359)
LMD-VII (ours)
33.52
0.8978
28.85
0.8451
10s 162.627G
E.6 RESULTS OF SPECIAL CASES
We subjected our proposed model to testing using two distinctive input scenarios. the case of globally blurred images sourced from the GoPro testing dataset, the corresponding masks exhibit uniformly high values, and decision maps manifest predominantly an all-white color, as de- picted in Appendix Cland visually represented in Figure 7] Conversely, when inputting sharp images from the GoPro testing dataset, the mask values are uniformly low, resulting in decision maps that predominantly display all-black color, illustrated in FFigure 10 The results of these extreme experiments demonstrate the model's ability to effectively distinguish hetween blurred and sharp images, showcasing its robustness.
F
MORE VISUAL RESULTS OF LOCAL MOTION DEBLURRING
F.1 MORE VISUAL COMPARISONS BETWEEN LMD- VIT AND THE COMPARED METHODS
We provide more visual results of our proposed LMD-ViT ViT and other baseline methods for local motion deblurring, as shown in Figure 11 and Figure 12 The performance of LMD-ViT surpasses that of other otate-of-the-art methods, yielding output images with improved clarity and enhanced details.
F.2 MORE VISUAL RESULTS OF USER STUDY
We provide more visual results for real-world local motion deblurring, which used for conducting user study, as shown in Figure 13] We conduct a comparison between our method and the top methods listed in Table of the main paper. The visual results show that proposed method exhibits robust deblurring capability and outperforms the sfate-of-the-art methods on real-world locally blurred images.
G LIMITATIONS
Like other local motion deblurring methods (Nah et al. 2017 Kupyn et al. 2018; 2019) Chen et al.]
2021
Cho et al. 2021
Li et al. 2023 Wang et al. 2022 Zamir et al. (022), proposed L.D..
16

---

Published as a conference paper at ICLR 2024
1.0
Value
0.6
0.4
Accuracy
Precision
0.2
Recall
2
5
8
AdaWPT block number
Figure 9: The mask prediction accuracy, precision, and recall on the ReLoBlur testing dataset.
Poit
UCT
Sharp input
Decision map
Output
GT
Figure 10: Results of special cases, When inputting static sharp images, the decision maps tend to be all-zero metrics and colored in black. The output very consistent with the ground truth because LMT prohibits nearly all tokens from going through Transformer layers.
VIT is not capable of real-time deblurring. However, we plan to optimize our structure so that it can be applied in real time.
17

---

Published as a conference paper at ICLR 2024
Blurred
(c)
Locally blurred image
(c)
(f)
(g)
(h)
GT
Blurred
(a)
(b)
(c)
(d)
Locally blurred image
(c)
(f)
(g)
(h)
GT
B lurred
(a)
(b)
(c)
(d)
VILK
Locally blurred image
(f)
(g)
Blurred
(a)
(b)
(c)
(d)
Locally blurred image
(e)
(f)
(h)
GT
Blurred
(a)
(c)
(d)
Locally blurred image
(e)
f
(g)
(h)
GT
Figure 11: Visual comparing results of state-of-the-ee of--of--tth-tttt art methods for local motion deblurring. (a)
DeepDeblur (Nah et al. 2017); (b) DeblurGAN v2 (Kupyn et al. 2019) (c) HINet (Chen et al.
2021) I (d) MIMO-UNet (Cho et al. 2021); (e) LBAG (Li et al.) 20231 Restormer (Zamir al.
2022 (g) Uformer (Wang et al. 2022): ; (h) LMD- AMD-ViT (ours).
18

---

Published as a conference paper at ICLR 2024
Blurred
(a)
(b)
(c)
(d)
Locally blurred image
(f)
(g)
(h)
GT
blur
(a)
(b)
(c)
(d)
Locally blurred image
(e)
(f)
(g)
(h)
GT
blur
(a)
(bb
(c)
(d)
Locally blurred image
(e)
(f)
(g)
(h)
Blured
(a)
(b)
(c)
(d)
Locally blurred image
(e)
(f)
(h)
GT
Blured
(c)
(d)
Locally blurred image
(e)
(0
(h)
GT
Figure 12: More visual comparing results of state-of-the-art methods for local motion deblurring. The annotations the same as that in Figure 11
19

---

Published as a conference paper at ICLR 2024
Real-world locally blurred image
LBAG
Restormer
Ufommr
Ours
Figure 13: Visual results of user study for real-world local motion deblurring. We compare our proposed LMD-Vii with the top methods, i.e., LBAG (Li al., 2023), Restormer (Zamir et al., 2022), and Uformer (Wang et al. 2022), listed in Table of the main paper.
20


Reviews:
[{"review_id": "aAoyznWwwz", "reviewer": "Reviewer_ePoq", "summary": "The paper attempts to tackle local motion deblurring. Existing deblurring literature mostly focuses on general deblurring (without specifying global or local regions). This could be a disadvantage in computation and generalization in scenarios where only a small part of a high-resolution image is blurred. Therefore, this work proposes a transformer-based network called LMD-ViT. The authors make use of adaptive window pruning and blurriness confidence predictor in the transformer blocks to ensure the network focuses on local regions with blurs. Quantitative and qualitative results are presented on the ReLoBlur and GoPro datasets. The effectiveness of different design choices is analyzed.", "questions": "See Weaknesses.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "An adaptive window pruning strategy is adopted to focus the network computation on localized regions affected by blur and speed up the Transformer layers.\n\nA carefully annotated local blur mask is proposed for the ReLoBlur dataset to improve the performance of local deblurring methods.", "weaknesses": "The organization of the paper can be improved.\n\n1) The methodology (Sec. 2) consists of too many (unnecessary) acronyms. Moreover, there are some inconsistencies when citing previous works (for example, LBAG (Li et al., 2023), LBFMG (Li et al., 2023), etc.). It would be better for the submission would strongly benefit from polishing the writing.\n\nThe settings of the experiments need more explanation.\n\n2) It is not clear why the GoPro dataset is used for training along with the ReLoBlur training set. In previous works, such as LBAG, only the ReLoBlur dataset is used (see Table 4).\n\nThe novelty of the submission needs to be clarified.\n\n3) It would be better to discuss the differences between LBAG and Uformer. Compared to LBAG, it simply substitutes the CNN architecture with a Transformer. All the other modules, including sparse ViT, W-MSA (Window-based multi-head self-attention), and LeFF (locally- enhanced feed-forward layer), have been introduced in previous deblurring works.\n\nThe fairness of the experiments.\n\n4) The transformer baselines, such as Restormer and Uformer-B, are not trained with the local blur masks, which are deployed during their training by the proposed methods. This makes the comparison in Table 1 unfair.\n\nUnclear parts.\n\n5) Please explain in detail how the authors manually annotate the blur masks for the ReLoBlur dataset.\n\n6) The baseline results reported in Table 1 are higher than those in their original papers, e.g., LBAG. It would be better to give the reasons and more details when introducing Table 1. The baseline results reported in Table 1 are higher than those in their original papers, e.g., LBAG for 34.85 dB.\n\nTypo: \nTable table 4 in Appendix C.\nDecision map in Figure 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r3rtBP02tp", "reviewer": "Reviewer_SKAv", "summary": "The presented work delves into an interesting research problem: local motion deblurring. It introduces a novel approach known as LMD-ViT, constructed using \"Adaptive Window Pruning Transformer blocks (AdaWPT).\" AdaWPT selectively prunes unnecessary windows, focusing powerful yet computationally intensive Transformer operations solely on essential windows. This strategy not only achieves effective deblurring but also preserves the sharp regions, preventing distortion. Moreover, their method significantly accelerates inference speed. Additionally, they have provided annotated blur masks for the ReLoBlur dataset. The utility of this approach is showcased on both local and global datasets, where it demonstrates substantial performance improvements on local motion deblurring (the ReLoBlur dataset) and competes favorably with baseline methods in the realm of global deblurring.", "questions": "please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.\tThis paper addressed problems of single image local motion deblurring, which is very essential in today’s photography industry. The presented method is the first to apply sparse ViT in single image deblurring and may inspire the community to enhance image quality locally.\n2.\tThe proposed pruning strategy including the supervised confidence predictor, the differential decision layer and pruning losses are reasonable and practical. It combines window pruning strategy with Transformer layers, only allowing blurred regions to go through deblurring operations,  resulting in not only proficient deblurring but also the preservation of sharp image regions.\n3.\tThe quantitative and perceptual deblurring performances are obvious compared to baseline methods.\n4.\tThe presented method derives a balance between local motion deblurring performance and inference speed, as shown in the ablation study and experiments. The proposed method reduced FLOPs and the inference time largely without deblurring performances dropping on local deblurring data.\n5.\tThe authors provided annotated blur masks for the ReLoBlur dataset, enhancing the resources available to the research community.", "weaknesses": "1.\tThe authors did not mention whether the presented method LMD-ViT requires blur mask annotation during inference. This is crucial because if the method does require blur masks before inference, it would be helpful to provide instructions on how to generate them beforehand and assess their practicality.\n2.\tThe proposed method uses Gumble-Softmax as the decision layer in training and Softmax in inference. The equivalence of the two techniques in training and inference is not discussed.\n3.\tIn the user study experiment, the absence of an explanation regarding the camera equipment used is notable. This is important because when images from the same camera share a common source, the blurriness often exhibits a consistent pattern. Therefore, including images from the same camera would allow us to assess the proposed method's robustness.\n4.\tSome references are missing, like “Window-based multi-head self-attention” in page 2, and “LeFF” in Section 2.4.1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "53RSL4PPu1", "reviewer": "Reviewer_S2zK", "summary": "This paper proposes a local motion deblurring Transformer with adaptive window pruning, which only deblur the active (blurry) windows. The windows are classified as active/inactive according to the predicted bluriness confidence score.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, I think this paper has a novel idea and achieves good results in terms of performance and efficiency. I tend to accept this paper due to following reasons.\n\n1. adaptive window pruning that saves computation on unnecessary attention windows.\n\n2. bluriness confidence prediction that works well for both local motion prediction and global motion prediction\n\n3. annotated local blur masks on ReLoBlur\n\n4. well-designed experiments, well-presented figures and well-written paper.", "weaknesses": "Below are some concerns and suggestions.\n\n1. Since the confidence predictor only uses MLP layers. How many pixels did you shift? Is the feature shift necessary to enlarge the receptive field of the neighbourhood? \n\n2. What is the mask prediction accuracy on validation set?\n\n3. How did you decide the border when annotating the masks for blurry moving objects?\n\n4. If a patch is always abandoned, how is it processed? What layers it will be passed into during inference?\n\n5. It could be better to provide the results of two special cases (masks are all-ones/all-zeros) in the tables as a reference.\n\n6. Why did you only report real-world results on large images? Do you have a chart for comparison on PSNR/FLOPs/runtime under different image resolutions.\n\n7. The key of this method is the adaptive window pruning. It is better to provide an ablation study with the rest tricks as a baseline (i.e., no window pruning in training and testing).\n\n\nMinor:\n\n1. Unfinished paragraph in page 2.\n\n2. For figure 1, I think it might be better to visualise the attention window borders, for example, by adding solid lines to show the windows.\n\n3. The summarised contributions are a bit overlapped (point 1 and point 2). I think it's better to claim adaptive window pruning and bluriness confidence prediction as two contributions.\n\n4. I think there is no need to distinguish between AdaWPT-F and AdaWPT-P. Just be simple and united. The name of AdaWPT is enough.\n\n5. Comparison in Figure 6 is not visible.\n\n6. Is there an example on a globally clear image (e.g., a still scene). Contrary to Figure 7, will the decision map be all zeros?\n\n7. Are there other similar works in image restoration/deblurring? Are there some connections between this method and some \"blur kernel prediction + restoration\" methods (e.g. Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)