Stochastic Window Transformer for Image Restoration
Authors: Jie Xiao
Xueyang Fu
Feng Wu
Zheng-Jun Zha
Venue: NeurIPS 2022 Accept (2022)
Keywords: image restoration
transformer
stochastic window strategy
translation invariance
locality

Abstract: Thanks to the powerful representation capabilities, transformers have made impressive progress in image restoration. However, existing transformers-based methods do not carefully consider the particularities of image restoration. In general, image restoration requires that an ideal approach should be translation-invariant to the degradation, i.e., the undesirable degradation should be removed irrespective of its position within the image. Furthermore, the local relationships also play a vital role, which should be faithfully exploited for recovering clean images. Nevertheless, most transformers either adopt local attention with the fixed local window strategy or global attention, which unfortunately breaks the translation invariance and causes huge loss of local relationships. To address these issues, we propose an elegant stochastic window strategy for transformers. Specifically, we first introduce the window partition with stochastic shift to replace the original fixed window partition for training. Then, we design a new layer expectation propagation algorithm to efficiently approximate the expectation of the induced stochastic transformer for testing. Our stochastic window transformer not only enjoys powerful representation but also maintains the desired property of translation invariance and locality. Experiments validate the stochastic window strategy consistently improves performance on various image restoration tasks (deraining, denoising and deblurring) by significant margins. The code is available at https://github.com/jiexiaou/Stoformer.

Stochastic Window Transformer for Image Restoration
Jie Xiao, Xueyang Fu, Feng Wu, Zheng-Jun Zha
University of Science and Technology of China, Hefei, China
ustchbxj@mail. ustc. dustc.edu.com cn, fengwu, z Susttec..uuu.....ooo edu. en
Abstract
Thanks to the powerful representation capabilities, transformers have made impres- sive progress in image restoration. However, existing transformers-basee methods do not carefully consider the particularities of image restoration. In general, im- age restoration requires that an ideal approach should be translation-invarian to the degradation, i.e., the undesirable degradation should be removed irrespective of its position within the image. Furthermore, the local relationships also play vital role, which should be faithfully exploited for recovering clean images. Nevertheless, most transformers either adopt local attention with the fixed local window strategy or global attention, which unfortunately breaks the translation invariance and causes huge loss of local relationships. To address these issues, we propose an elegant stochastic window strategy for transformers. Specifically, we first introduce the window partition with stochastic shift to replace the original fixed window partition for training. Then, we design new layer expectation propagation algorithm to efficiently approximate the expectation of the induced stochastic transformer for testing. Our stochastic window transformer not only enjoys powerful representation but also maintains the desired property of transla- tion invariance and locality. Experiments validate the stochastic window strategy consistently improves performance various image restoration tasks (derain- ing. denoising and deblurring) by significant margins. The code available at https:// //gi github. com/ Swwriiiu/Stooooooo
Introduction
Image restoration aims to recover latent clean images from their noise-polluted counterparts, which lays the foundation for various vision tasks. Generally, image restoration methods should satisfy constraint: an ideal approach should remove undesirable degradation irrespective of its position within the image. In other words, maintaining the translation invariance is the fundamental require- ment for image restoration, which makes convolutional neural networks (CNNs) well-suited for this specific vision task [56, 23, 9, 73, 70, 61, 66, 58, 72, 6, 15]. comparison with multi-layer perceptrons (MLPs), the distinct characteristics of CNNs [26, 22, 17] are their built-in locality and weight sharing, resulting in the desirable property of translation invariance. These two priors play vital role in image restoration, i.e., pixels within local regions tend to exhibit strong correlations while translation invariance is one the desiderata for the potential image restoration method.
On the other hand, transformer- baase image restoration methods [2, 54, 33, 63] are not compatible with the translation invariance and locality. The patterns that transformers adopted for recovering high-resolution images can be summarized as small patch with global attention (e.g., 48 x 48 for IPT [2]) and large patch with local attention (e.g., Uformer [54] and SwinIR [33]). Nevertheless, neither of these two patterns can meet the requirements of translation invariance or locality. Here we detail
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Seniroa Camurod
0.9
Window
0.4
Reed
Stochertic
Radius
(a) The fixed window partition breaks the translation invari- ance.
(b) The fixed local window can- not faithfully capture local rela- tionships.
(c) The comparison of percentage of captured locality between the fixed and stochastic window strategy.
Figure I: Illustration of broken translation invariance and lost locality of the fixed window strategy.
the latter pattern while the former one is included in the supplemental material. The latter pattern partitions the feature map into fixed non-overlapped windows and self-attention is restricted inside these local windows. However, we argue that the fixed partition strategy will break the translation invariance and further lead to huge loss of local relationships. As shown in Figure (()), for the fixed window partition strategy, translation invariance can only be ensured among specific partitioned windows (blue windows) rather than general windows with arbitrary shifts (red windows), which disagrees with the translation invariance. The reason behind this is that the fixed window partition imposes the artificial prior to windows with specific shifts, which turns out to break the translation invariance. Figure 1(b) intuitively illustrates that the fixed window partition cannot capture intact local relationships. Without loss of generality, we consider the Euclidean distance, based on which the neighbor space will be the region encompassed by circle, in 2D plane to measure locality. For certain token (central red dot), many tokens (green dots) within its neighbor space (encompassed by the red circle with radius r) cannot participate in the calculation of self-attention, which leads to tremendous loss of local relationships. As the neighbor space expands, i.e., increasing radius the loss will more severe. Figure I(c) quantitatively depicts the proportion of local relationships that captured by the fixed window partition over the total relationships existing in the neighbor space. Ideally, when the feature map is divided into non-overlapping windows of size X 8, is expected that the relationships of token pairs with distance less than can be faithfully captured. However, can observe that there exists significant loss of local relationships even within the neighbor space whose radius is much smaller than A seemingly straightforward solution to this problem is to adopt the sliding window strategy to sean the whole feature map using the local window like CNNs. However, this strategy incurs huge overhead in terms of both memory and computing speed.
In this work, we propose a novel stochastic window strategy to impart translation invariance to the transformer and make it faithfully model local relationships. Specifically, instead of partitioning the feature map into fixed non-overlapped windows, we choose to cover the whole feature map with a stochastically shifted window partition (see Figure 2). By introducing stochastic shifts, all windows are treated equally so that the translation invariance and locality can be ensured. As shown in Figure (c)), our stochastic window strategy is able to capture complete local relationships until the radius of neighbor space reaches the window size. It is noteworthy that unlike the sliding window strategy, the time and memory overhead consumed for training stochastic window transformer is comparable to the fixed window strategy. For testing, we further design the layer expectation propagation algorithm to approximately marginalize the stochastic shifts, during which the translation invariance and locality can also be ensured.
In conclusion, the contributions are threefold:
We analyze the phenomena of translation invariance breaking and local relationships loss in existing transformer-baat based image restoration approaches. To the best of our knowledge, this is the first work to point out the importance of translation invariance for image restoration.
These windows processed using shared weights, e.g., Wo, WK, Wy for Q, K, V.

---

We propose new stochastic window strategy, which comprises stochastic windows for train- ing and layer expectation propagation for testing, compensate for the broken translation invariance of transformers and enable them to faithfully model local relationships.
Extensive experiments on various tasks, e.g., deraining, denoising and deblurrring, validate that the restoration performance can be consistently improved by equipping our stochastic window strategy.
2
Related works
Image Restoration. Image Restoration [67, 11, 5, 36, 34, 38, 62, 40, 29, 28] aims to restore the clean image from its degraded version. In recent years, remarkable progress against traditional model- driven methods [4, 53, 16] has been achieved due to the development of deep learning technologies [27], especially CNNs [26, 22, 17]. Instead of relying on preset image priors, learning based methods directly learn to project from noisy to clean ones from a large collection of noisy-clean image pairs. Numerous representative CNNs have sprung up across various classical image restoration tasks, including image denoising [25, 60, 68, 66, 7, 74, 32], image super-resolution [72, 6, 15, 20, 75], image deblurring [9. 23, 39, 23, 47, 65, 43, 42], image deraining [58, 50, 13, 64, 41, 30, 14), etc. In general, the translation invariance (derived from weight sharing) and locality have been hard-coded into the inherent structure of CNNs so that it seems CNNs are well-suited for image restoration tasks. However, compared with Transformer, CNNs are restricted by their limited flexibility.
Vision Transformer. Recently, with great success of transformer [49, 8] in the NLP field, Vision Transformers [10, 48, 52, 57,59, 3] have also been prevalent in vision community. ViT [iT treated image patches as token sequence and applied the vanilla transformer on it for image classification. With the goal of bringing in reasonable priors into transformer to improve efficiency, Swin [35] introduced local window based attention and established hierarchical architecture. Inspired by the key insights from high-level vision, a few transformers [2, 33, 54, 63, 55] for low-level vision have arose. But most of them directly transfer high-level designs (e.g.. local attention) into low-level vision without careful consideration of its particularities, As we discuss above, unlike CNNs, transformers do not possess the translation invariance and intact locality. In general, low-level tasks usually require to accomplish pixel-level regression, where more strict locality and translation invariance are expected. Besides, the CNNs-style sliding window strategy will incur huge burden terms of memory and computing speed. Therefore, an efficient mechanism to remedy the broken translation invariance and locality of transformer is essential for low-level vision.
3 Stochastic Window Transformer for Image Restoration
Transformers have attained impressive performance for image restoration due to their strong represen- tation ability. Given the quadratic complexity in both computation and memory usage, transformers for image restoration tend to employ the local attention. Except for high-efficiency, local attention should have been designed to model local relationships. However, as analyzed before, local attention breaks the ideal translation invariance and further leads to huge loss of local relationships. As shown in Figure 2, for feature map, the window partition with arbitrary shift Ew. contains a comparable quantity of local relationships and implies the translation invariance among divided windows. In other words, all the window partitions are equally informative in terms of translation invariance and locality and should be treated equally. Therefore, it is unreasonable for the fixed window strategy express infinite favoritism towards the certain partition (c.g., (0, 0) or (, {)) while simply discarding others. In this work, we propose the stochastic window strategy, by which all the window partitions are treated fairly. The network can be trained based on local attention but with stochastic rather than fixed shift. Therefore, transformer with the stochastically shifted window can be trained as efficiently as the fixed window. At test time, we propose the layer expectation propagation algorithm to approximate the expectation of the introduced stochastic shift, which also helps maintain the desired property of translation invariance and locality.
&
Figure 2: Window partition with shift (En, Ew). is the spatial size of local window.
3

---

3.1 Stochastic Window Strategy
Stochastic window strategy aims to impart the translation invariance to transformer and make faithfully exploit local relationships. Canonical transformer consists of alternate layers of self- attention (SA) and MLP. In order to promote efficiency, transformer tends to employ local window based attention and shifted window strategy is utilized to allow inter-window connections. Specifically, the feature map partitioned into non-overlapped windows and then SA is computed within local windows. Suppose the window partition is denoted by Par(:; 8, Eh, Sw), where size of the local window 8 and shift of the whole window partition is (Sh, Ew) (see Figure 2). With these notations, the shifted window transformer can be reformulated as:
21 SA(Par( s, 0, 0)) + 21-1,
21 MLP(= + =b1
$ 8 + It
21+1 = SA (1) 8,
2 2
31+1 = MLP((=+1) + 21+1,
where 2 and zς denote the output feature of SA and MLP for layer 1, respectively. To keep notation uncluttered, we overlook layer normalization in Equation (1). We figure out that all the window partitions are equally informative and preserving only one configuration each layer will lead to the broken translation invariance and tremendous loss of locality. The thought of informative equality among all the window partitions leads us to the stochastic window transformer, which is computed as:
2 (A(aaa(( CwD) + 21-1,
(L)) U(R.),
It = MLP(2)) (2)) + 21
where the set R. contains all possible shifts and U(.) denotes the uniform distribution. Given the periodicity of the window partition, R. can be simplified as
R2 = (0, ,,...., 8- 1] [0....,8-1]
where x means the Cartesian product. During training, (4) (l) are treated as i.i.d. random variables and sampled from the uniform distribution U.R,). Suppose the total number of SA layer is N, stochastic shifts from different layers ((C).<1))) ) are deliberately designed to be independent so that faithful locality and translation invariance can be ensured on layer level. By this treatment, despite all possible shifts are taken into account for each individual layer, single forward propagation only requires a set of sampled shifts {(C), <t. = so that the stochastic window transformer can be trained efficiently. For testing, from Bayesian perspective, stochastic shifts should be averaged according to their posterior distribution. Hence, the exact inference procedure is:
F(z)
Σ Sh
Fl;; Eh. co su..... EN ew -NJJUCE co ........ EN-1 -w EN-11
E N-1
Chi co.
e N- NN -Y-1) 2 ... -UGNN :.w ,
(4)
where F(-) denotes the function of the overall transformer and U is the uniform distribution. The derivation of Equation (4) follows {(C), el - independent. Apparently, according to Equation (4), the single exact inference requires forward propagation exponential times so that the compu- tational overhead of exact inference grows exponentially with the depth N, which is prohibitively expensive. Consequently, we turn to seek an approximate inference, which should also guarantee the translation invariance and intact locality, approximation in Dropout [46], we propose the general layer expectation propagation algorithm to approach the original exact inference process. The layer expectation propagation is formulated as:
replace the costly exact inference. Inspired by the
Σ
SA(Par(at 8, ch. U(() dl) + x
21
(8) +L)
E (L
[SA(Par(atst 8, che cc]I]] + #
test
(5)
test
(2((( + 2000

---

Encoder
Decoder
Leval-1
Lovol
MLP
Impur E doing
StoBlock
Down Sample
StoBlock
Down Sample
StoBlock
Leveli
Down
Samale
StoBlock
Up
Levelis
Sample
StoBlock
Up Sample
StoBlock
Up Sample
SteBleck
Reconstructio
Layer Norm
ge3
Layer Norm
(a) Architetture
(b) StoBlock
Figure 3: (a) The overall architecture of Stoformerit: (b) the structure of Stoformert block. StoWin- MSA is multi-head self-attention with stochastic window partition.
According to Equation (5), the single approximate inference merely requires one forward propaga- tion of the expected signal, which accelerates inference considerably. We then examine whether the stochastic window strategy is able to keep the translation invariance and model intact local relationships.
Translation Invariance. During training, the stochastic window strategy treats arbitrarily shifted window equally and eliminates the particularity of certain windows introduced by the fixed window partition (blue windows in Figure ((b).. Therefore, the stochastic window strategy can maintain the translation invariance during training. For testing, by the layer expectation propagation, each individual token is able to aggregate information from its neighbor space with shared weights like CNNs. Consequently, the translation invariance can also be well maintained. We also provide experimental evidence support the translation invariance in Section 4.2.
Locality, We have argued that the fixed window partition will lead to huge loss of local relationships since the majority of equally informative window partitions are simply discarded. As shown in Figure 1(b), there exists some token pairs whose distance small enough but they are not present in the same local window, which results in tremendous loss of locality. In contrast, with the stochastic window strategy, the window partition is randomly shifted, which will guarantee that any token pair will attend in the same window as long as their distance is smaller than the window size As shown in Figure (cc), the stochastic window strategy can capture intact local relationships when the radius of neighbor space is not larger than window size while the fixed window partition suffers from severe loss of local relationships. For testing, the proposed layer expectation propagation also considers all the shifted window partitions so that intact locality can be ensured as well.
Implicit Model Ensemble. Compared with the typical fixed window strategy (in particular, the observe significant performance improvements from extensive
shifted window strategy), experiments. In addition to the explanation from the compensation of the lost local information and the broken translation invariance, another explanation for performance improvements is that training with the stochastic window can be seen as training an ensemble of the fixed window transformer implicitly [18]. Each self-attention layer contains w² window partition, which results in 2N possible network combinations. For each training bini-batch, one of the 2NN networks is sampled and then updated. For testing phase, all the networks implicitly integrated using the proposed layer expectation propagation.
3.2 Network Architecture
The proposed stochastic window transformer provides an effective and efficient way to integrate the well-acknowwedged priors, i.e., locality and translation invariance, into transformer. For image restoration tasks which require pixel-level regression, faithfully exploiting these priors is especially important. To validate the effectiveness of the proposed stochastic window strategy, we integrate with the widely-used U-shaped architecture [44, 21J. The resulting transformer, named Stoformerti. not only enjoys the intrinsically strong representation ability of Transformer but also maintains the ideal translation invariance and intact local relationships. Figure 3 illustrates the overall architecture.
5

---

01
150
250
20
150
(a) Input
(b) Fixed window strategy
(c)
VDN
(d) Stochastic window strategy
Figure 4: Toy Experiment. The stochastic window strategy (d) helps to remedy the broken translation invariance caused by the fixed window partition (b).
3.3 Implementation Details
Loss Function. The loss function adopted for training is the Charbonnier loss [1], whose mathematical expression
L(I" = - 1| +
(6)
where I' and are the restored and ground-truth image respectively. The constant is empirically set 10-3
Training Detail. Stoformerit employs a four-level encoder-decoder structure. The numbers of StoBlock are {1, 2, 8} for level-1 to level-4 of Encoder and the blocks for Decoder are mirrored. The number of channel set to 32 and the window size is x We train the network with Adam optimizer (31 0.9, B2 0.999) with the initial learning rate 3 10 gradually reduced to 1 x 10-6 with the cosine annealing. The training samples are augmented by the horizontal flipping and rotation of 90" 180", 270". Please refer to the supplemental material for task-specific settings.
4 Experiments
4.1 Experimental Setup
In this section, we validate the effectiveness the stochastic window strategy. Except the previous slate-of-the-art methods are included for comparisons, we also elaborate four Stoformer variants: Stoformer&O. Stoformeri Stoformer&t, Stoformertt. The first symbol aims to indicate whether the stochastic window is adopted for training(f: : second symbol represents whether the layer expectation propagation is adopted for testing(t: layer expectation propagation; @: the fixed window). In all experiments, specify the fixed window (Equation (1)). These variants are identical except for the aforementioned window strategy. In particular, Stoformer is reduced to the traditional fixed window transformer and Stoformertt is the proposed stochastic window transformer.
the stochastic window; Q: the fixed window) and the
the shifted window to
4.2 Toy Experiment
To illustrate the broken translation invariance of the fixed window strategy and the remedy of proposed stochastic window strategy to this effect, a toy experiment is performed: we add a Gaussian noise patch with
50 whose spatial size is 64 x 64 size 256 x 256, the pixel value of which constantly set to the medium value 127. We move the Gaussian noise patch with stride and obtain total 192 x 192 36864 noisy images. Then denoise these noisy images using the fixed and stochastic window transformer, respectively and the PSNR distributions with respect (.r) are normalized the fixed range [0, and plotted in Figure 4. To further highlight the translation invariance of stochastic window transformer, we also include typical CNNs- denoiser VDN [60], which can ensure the translation invariance well due to inherent structure of CNNs, as the inference. Figure 4 indicates clearly that there are great variations, especially for the interior, the PSNR distribution of fixed window transformer while stochastic window transformer flattens the distribution significantly. Both stochastic window transformer and VDN contain the relatively uniform distribution for the interior, which indicates that the stochastic window strategy helps to maintain the desired translation invariance quite well.
8-bit pure color image with spatial

---

Blurry 30.10 dB
DBGAN 30.58 dB
MPRNet 31.60 dB
Stoformer00 31.82 dB
Input
Stoformert+
Stoformero 32.30 dB
Stofomeett 33.11 dB
Target
30.28 dB
c
Figure 5: Visual comparison image deblurring on the GoPro.
is worth noting that all these distributions have variations on the image boundary, which can be attributed to the difference of the contexts of the image boundary.
4.3 Experiments on Image Deraining
Table 1:
Quantitative results of image
Table 2: Quantitative results of image deblurring. Sto-
deraining on the SPA-Data.
formers are trained only on GoPro dataset.
SPA-Data
Method
PSNR
SSIM
Method
GoPro
HIDE
PSNR
SSIM
PSNR
SSIM
GMM [31]
34.30
0.9428
Nah et al. [39]
29.08
0.914
25.73
0.874
DDN [13]
36.97
0.9604
DeblurGAN [23]
28.70
0.858
24.51
0.871
SPANet [51]
40.24
0.9811
DeblurGAN [24]
29.55
0.934
26.61
0.875
JORDER-E [58]
40.78
0.9811
DBGAN [69]
31.10
0.942
28.94
0.915
RCDNet [50]
41.47
0.9834
IPT [2]
32.52
SPAR [40]
44.10
0.9872
MPRNet [62]
32.66
0.959
30.96
0.939
Uformer [54]
47.84
0.9925
SPAIR [40]
32.06
0.953
30.29
0.931
Stoformer O
47.80
0.9925
Stoformer
32.80
0.959
30.73
0.937
Stoformer f
46.95
0.9917
StoformerOt
31.62
0.950
29.94
0.928
Stoformer
33.17
0.963
30.91
0.940
Stoformert
48.85
0.9935
Stoformerti
33.24
0.964
30.99
0.941
Stoformertf 48.97
0.9938
We validate the stochastic window strategy image deraining task. Specifically, except the elaborate four Stoformer variants, existing seven deraining methods are included: GMM [31], DDN [13], SPANet [51], JORDER-E E [58]. RCDNet [50], SPAIR [40]. Uformer [54]. All these methods are evaluated on SPA-Data [51] and the performance comparison is shown in Table Due to the limited space, visualization of image deraining is included in the supplement material.
4.4 Experiments on Image Denoising
We conduct denoising experiments on the additive white Gaussian Noise benchmark datasets, which include Set12 [66], BSD68 [37], Urban100 [19], Kodak24 [12] and McMaster [71]. Following previous works [66, 60, 67]. a single model is trained tackle with various noise levels. Tables and 4 report PSNR scores of existing methods as well as Stoformer variants for color and grayscale image denoising, respectively. The supplemental material contains visualization of image denoising.
4.5 Experiments Image Deblurring
We also perform deblurring experiments on the benchmark datasets (GoPro [39] and HIDE [45]). The model is trained only the GoPro dataset and directly evaluated the HIDE. Table 2 presents PSNR and SSIM scores of different deblurring methods and Stoformers. Figure 5 presents an image deblurring example from GoPro [39]. More visual results are provided in the supplemental material.
Remark. From above experiments on various image restoration tasks, we can make the following observations and analyses:
7

---

Table 3: Gaussian color image denoising. single model is learned for various noise levels.
Mothod
CBSD68
Kodak22
McMaster
UrbanI00
o= 15
25
15
25
15
25
30
25
50
IRCNN [67]
33.85
31.16
27.86
34.69
32.18
28.
34.58
32.18
25.91
33.18
31.20
27.70
HFDNet [68]
33.87
31.21
27.96
34.63
52.
228
31.66
32.45
29.
33.83
31.40
28.05
DeCNN [66]
33.90
31.21
27.95
31.60
32.11
28.
33.15
31.52
28. 62
32.98
30.81
27.59
VDN [60]
33.90
31.35
28.19 19
FuncNet [36]
34.28
DRUNet 170]
34.30
$1.66
28.51
(55.11 .1
$2, 89
29. 86
35.
33.14
30.008
34.81
32.60
29.61
Restormer [63]
34.39
31.78
28.39
33.02
30.00
35.55
35.06
32.91
Stotomeer00
34.34
31.73
28.02
36.32 32
32.91
25,
36.53
33.35
30.34
36.04
32.83
29.86
Stofommerff
34.30
31.73
Stoformer 0
35.10
392 40
29. 13
355.50 50
$3..08
300000
36.
351.83
30. 80
35.37
33.14
30,00
Stoforertt
33.13
29.16
30.8
Table 4: Gaussian grayscale image denoising. A single model is learned for various noise levels.
BSD68
Urban 100
Set12
Method
σ 15
25
0 50
15
0
25
50
0 15
o
25
a = 50
DrCNN [66]
31.62
29. 16
26.23
32.28
29.80
26.35
32.67
30.35
27.18
FFDNet [68]
31.63
29.19
26.29
32.40
29.90
26.50
32.75
30.43
27.32
IRCNN [67]
31.63
29.15
26.19
32.46
29.80
26.22
32.76
30.37
27.12
DRUNet [70]
31.91
29.48
26.59
33.44
31.11
27.96
33.25
30.94
27.90
Restormer [63]
31.95
29.51
26.62
33.67
31.39
28.33
33.35
31.04
28.01
Stoformer
31.94
29.51
26.62
33.60
31.38
27.90
33.26
31.02
27.96
Stoformer
31.89
29.46
26.59
33.48
31.09
27.85
33.18
30.96
27.92
Stoformer
32.57
30.06
27.03
34.19
31.84
28.58
33.83
31.51
28.40
Stoformerit
32.57
30.06
27.07
34.24
31.92
28.72
33.85
31.53
28.46
S vs. StoformerOf: Without the stochastic window strategy for training, directly applying the layer expectation propagation will degrade performance dramatically, which reveals that the performance gain cannot simply attribute to feature ensemble at test time and the stochastic window for training matters.
Stoformer vs. Stoformert O: Even without the layer expectation propagation algorithm for testing, the stochastic window partition for training alone is also conductive to boost per- formance. This is reasonable since large amount of local information lost by Stoformer is re-exploited by StoformerjQ.
Stoformrrtt Others: Equipped with the stochastic window for training and layer expec- tation propagation for testing (Stoformerit), the model achieves the highest performance.
4.6 Analytic Experiments
Complexity Analysis. We provide a detailed analysis about the time and space complexity of the stochastic window strategy based single attention layer. Specifically, window strategy with fixed and sliding window strategy for training and testing, respectively. For training, our strategy as efficient the fixed window for both speed and memory cost, as shown in Table 5. But the fixed window loses translation invariance and locality. The sliding window can fulfill the translation invariance and locality while suffer from huge memory burden, which often incurs OOM. In contrast, the stochastic window enjoys efficient training while maintains translation invariance and locality. For testing, as shown in Table 6. the fixed window can inference efficiently but suffer from the broken translation invariance and locality loss. Compared with the sliding window, our stochastic window strategy requires more computations but less memory cost.
compare our stochastic
Table 5: Time and space complexity for training. B: batch size, (H, C): feature size, local window size, h: number of heads.
Strategy
Time Complexity
Space Complexity
Trans. Inva. Locality
Fixed window Sliding window
OSBHWC' BBHWC C?)
e(BHWC + h
0(BHWC3 +BHWC*
O(BHWCs2 + BBHhhs?
Stochastic window
@BBHWC2 -HWC
@(BHWC + BHWhsa)

---

Table 6: Time and space complexity for testing. window size, h: number of heads.
batch size, (H, W. C): feature size, local
Strategy
Time Complexity
Space Complexity
Trans. Inva. Locality
Fixed window Sliding window
GBHWWS -HWC
6(BHWC BHWhs²)
G(BHWCs BHWC*)
O(BHWCS BHWhs?)
Stochastic window
e BBHWCs BHWCR)
@(BHWC- + BlWhs?)
Eliminating Blocking Artifacts. Since the fixed window strategy implies infinite preference towards specific window partition, the learned feature map will consequently contain annoying blocking artifacts. In contrast, the proposed stochastic window strategy treats all window partitions equally, facilitating to keep translation invariance and model locality. Therefore, the blocking artifacts should be significantly eliminated by the stochastic window strategy. Figure 6 provides experimental evidence support our inference. Specifically, the learned feature map by the fixed window strategy (Figure 6(b)) contains obvious blocking artifacts while the stochastic window strategy (Figure 6(c)) can remove these side effects as expected. is surprising that with the stochastic window strategy, the feature is divided into non-overlapping local windows for further processing but in turn nearly no blocking artifacts are preserved. Please refer to the supplemental material for more visualization.
(a) Input
(b) Fixed window strategy
(c) Stochastic window strategy
Figure 6: The stochastic window strategy can eliminate the blocking artifacts.
Trade-off in Layer Expectation Propagation. Although the layer expectation propagation accel- erates inference significantly, the strict conduction still requires to take average of s features yielded by self-attention. Here, we investigate the influence of the number of averaged features to model performance. Figure 7 shows the trend of model performance and FLOPs of testing" 256 x 256 image with respect to the number of averaged features each layer. Figure reveals that PSNR first rises rapidly and then tends saturate with the increasing of the averaged feature number while FLOPs increases linearly with that number, which provides a promising trade-ofi off between performance and FLOPs. For instance, keep relatively high performance and efficient inference as well.
decrease the averaged feature number from 64 to 8 to
Boosting Model Generalization. Figure 8 presents the loss curve on SPA-Data during training and the PSNR curve on the test set plotted in Figure Compared with the fixed window strategy, our proposed stochastic window strategy attains higher PSNR on the test set with similar training error, which demonstrates that the stochastic window strategy boosts model generalization. This can be anticipated since the stochastic window strategy can exploit complete local relationships, which are discarded by the fixed window strategy.
Limitation and Discussion
In this work, we propose the stochastic window strategy and extensive experiments validate the effectiveness on several image restoration tasks, including image deraining, denoising and deblurring. Our experiments are mainly based on the widely-used U-shaped architecture. We will further validate the stochastic strategy more architectures (e.g., isotropic and multi-stage architecture). Indeed, the performing improvements can positively anticipated due the desired translation invariance and
NNte that the transformer with the stochastic window strategy can be trained as efficiently as the fixed window strategy.

---

SECCIDE A3 V'CC
7351
100%
03
Nurber sainngd Saature
Figure 7: Trade-off between per-
Figure 8: Training loss on the training set SPA-Data.
Figure 9: PSNR score the test set of SPA-Data.
formance and FLOPs of testing.
locality. Since the importance of translation invariance and locality is quite general, we also plan extend the stochastic window strategy based transformer more low-level vision tasks, such as image super-resolution, JPEG deblocking, and video restoration. Local attention based transformers, e.g., Swin Transformer [33], have achieved remarkable success on several CV tasks, including image classification, dense prediction, and semantic segmentation. However, they also compute attention based on specific local windows and cannot treat all the local windows fairly. The consequences that unexpectable information loss occurs during feature processing and the translation invariance is also absent. Our proposed stochastic window strategy can provide an effective mechanism towards this problem. Therefore. it is also promising to extend the strategy to other tasks where local attention based transformers have exhibited competitive performance.
6 Conclusion
In this paper, we analyze the deficiencies of existing transformers for image restoration. We figure that the fixed window strategy will inevitably lead the broken translation invariance and loss of locality. To tackle with these issues, the novel stochastic window strategy is proposed. Specifically, we utilize the window partition with stochastic shift replace the fixed window partition for training, which can ensure the translation invariance and intact locality. For testing, we propose the efficient layer expectation propagation to approximately take expectation of the introduced stochastic shift. Based on this strategy. we conduct extensive comparison experiments on various image restoration tasks to validate the effectiveness the proposed stochastic window strategy.
Broader Impacts
Nowadays, image acquisition system inevitably suffers from various degradation, ranging from inherent noise of capturing instruments, the shaking during shooting, unpredictable weather condition. Hence, image restoration itself has important value of research and application. Our proposed stochastic window strategy can make transformer more sophisticated restoring degraded images. However, from a societal point of view, negative consequences may also come along. For example, the deviation from the actual image textures caused by image restoration technology may affect fair judgment in medical and criminal situations. In these cases, it is necessary to combine expert knowledge to make rational decisions.
Acknowledgement
This work was supported by the National Key R&D Program of China under Grant 2020AAA0105702, the National Natural Science Foundation of China (NSFC) under Grants 62225207 and U19B2038, the University Synergy Innovation Program of Anhui Province under Grant GXXT-2019-025, and the USTC Research Funds of the Double First-Class Initiative under Grant YD2100002003.
References
[1] Charbonnier, I.. Blanc-Feraud, G. Aubert, and M. Barlaud. Two deterministic half-quadratic regularization algorithms for computed imaging, In ICIP, 1994.
10

---

[2] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao. Pre-trained image processing transformer. In CVPR. 2021.
[3] X. Chu, Z. Tian, Y. Wang. B. Zhang. H. Ren, X. Wei, H. Xia, and Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurlPS, 2021.
[4] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image denoising by sparse 3-d transform- domain collaborative filtering. TIP, 2007.
[5] K. Dabov, A. Foi, Katkovnik, and K. Egiazarian. Image denoising by sparse 3-d transform- domain collaborative filtering. TIP, 2007.
[6] T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, Second-order attention network for single image super-resolution. In CVPR, 2019.
[7] Z. Dai, H. Liu, Q. V. Le, and M. Tan. Coatnet: Marrying convolution and attention for all data sizes. In NeurIPS, 2021.
[8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480 04805, 2018.
[9] Dong, S. Roth, and B. Schiele. Deep wiener deconvolution: Wiener meets deep learning for image deblurring. In NeurIPS, 2020.
[10] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, Uszkoreit, and N. Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[11] Y. Fan, J. Yu, Y. Mei, Y. Zhang, Y. Fu, D. Liu, and T. S. Huang. Neural sparse representation for image restoration. In NeurIPS, 2020.
[12] R. Franzen. Kodak lossless color image suite, 1999, URL kodak/.
[13] X. Fu, J. Huang, X. Ding, Y. Liao, and Paisley. Clearing the skies: A deep network architecture for single-image rain removal. TIP, 2017.
[14] X. Fu, Q. Z.-J. 7ha, X. Ding, F. Wu, and J. Paisley, Successive graph convolutional network for image de-raining. IJC,, 2021.
[15] M. Haris, G. Shakhnarovich, and N. Ukita. Deep nack-projection networks for super-resolution. In CVPR, 2018
[16] K. He, J. Sun, and X. Tang. Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[18] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
[19] J.-B. Huang, A. Singh, and N. Ahuja. Single image super-resolution from transformed self- exemplars. In CVPR, 2015.
[20] Y. Huang, S. Li. L. Wang, T. Tan, et al. Unfolding the alternating optimization for blind super resolution. In NeurlPS, 2020.
[21] Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
[22] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, NeurlPS, 2012.
[23] O. Kupyn, Budzan, M. Mykhailych, D. Mishkin, and J. Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In CVPR, 2018.
11

---

[24] O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang. Deblurgan-v2. Deblurring (orders-of of-nagiit faster and better. In ICCV, 2019.
[25] Laine, T. Karras, Lehtinen, and T. Aila. High-quality self-supervised deep image denoising. In NewrIPS, 2019.
[26] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Newral Computation, 1989.
[27] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 2015.
[28] C. Li, C. Guo, L.-H. Han, J. Jiang, M.-M. Cheng, J. Gu, and C. C. Loy. Low-light image and video enhancement using deep learning: a survey. TPAMI, 2021.
[29] C. Li, C. Guo, and C. C. Loy. Learning to enhance low-light image via zero-reference deep curve estimation. TPAMI, 2021.
[30] X. Li, J. Wu, Z. Lin, H. Liu, and H. Zha. Recurrent squeeze-and-excitaton context aggregation net for single image deraining. In ECCV, 2018.
[31] Y. Li, R. T. Tan, X. Guo, J. Lu, and M. S. Brown. Rain streak removal using layer priors. In CVPR, 2016.
[32] Y. Li, X. Fu, and Z.-J. Zha. Cross-patche graph convolutional network for image denoising. In ICCV, 2021.
[33] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte. Swinir: Image restoration using swin transformer. In ICCV Workshops, 2021.
[34] D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang. Non-local recurrent network for image restoration. NeurlPS, 2018.
[35] Z.. Liu, Y. Lin, Y. Cao, H. Hu, Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[36] F. Luo, X. Wu, and Y. Guo. Functional neural networks for parametric image restoration problems. In NeurlPS, 2021.
[37] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001.
[38] C. Mou, J. Zhang, and Z. Wu. Dynamic attentive graph learning for image restoration. In ICCV, 2021.
[39] S. Nah, T. Hyun Kim, and K. Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017.
[40] K. Purohit, M. Suin, A. Rajagopalan, and V. N. Boddeti. Spatially-adaptiie image restoration using distortion-guided networks. In ICCV, 2021.
[41] D. Ren, W. Zuo, Q. Hu, P. Zhu, and D. Meng. Progressive image deraining networks: A better and simpler baseline. In CVPR, 2019.
[42] W. Ren, J. Zhang, L. Ma, Pan, X. Cao, W. Zuo, W. Liu, and M.-H. Yang. Deep non-blind deconvolution via generalized low-rank approximation. In NeurIPS, 2018.
[43] W. Ren, J. Zhang, J. Pan, S. Liu, J. Ren, J. Du, X. Cao, and M.-H. Yang. Deblurring dynamic scenes via spatially varying recurrent neural networks. TPAMI, 2021.
[44] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[45] 7. Shen, W. Wang, X. Lu, J. Shen, H. Ling, T. Xu, and I.. Shao. Human-aware motion deblurring. In ICCV, 2019.
12

---

[46] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting Journal of Machine Learning Research, 15 96):1999-19558 2014.
[47] X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia. Scale-recurrent network for deep image deblurring. In CVPR, 2018.
[48] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[49] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention all you need. In NeurlPS, 2017.
[50] H. Wang, Q. Xie, Q. Zhao, and D. Meng. model-driven deep neural network for single image rain removal. In CVPR, 2020.
[51] T. Wang, X. Yang. K. Xu, S. Chen, Q. Zhang, and R. W. Lau. Spatial attentive single-image deraining with high quality real rain dataset. In CVPR, 2019.
[52] W. Wang. E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[53] Y. Wang. J. Yang, W. Yin, and Y. Zhang. A new alternating minimization algorithm for total variation image reconstruction. SIAM Journal on Imaging Sciences, 2008.
[54] Z. Wang, X. Cun, J. Bao, and J. Liu. Uformer: A general u-shaped transformer for image restoration. In CVPR, 2022.
[55] J. Xiao, X. Fu, A. Liu, F. Wu, and Z.-J. Zha. Image de-raining transformer. TPAMI, 2022.
[56] L. Xu, S. Ren, C. Liu, and J. Jia. Deep convolutional neural network for image deconvolution. In NeurIPS, 2014.
[57] J. Yang, C. Li, P. Zhang, X. Dai, B. Xiao, L. Yuan, and J. Gao. Focal attention for long-range interactions in vision transformers. In NeurlPS, 2021.
[58] W. Yang, R. T. Tan, J. Feng, Z. Guo, S. Yan, and J. Liu. Joint rain detection and removal from single image with contextualized deep networks. TPAMI, 2020.
[59] Q. Yu, Y. Xia, Y. Bai, Y. Lu, A. L. Yuille, and W. Shen. Glance-and -aze vision transformer. In NeurIPS, 2021.
[60] Z. Yue, H. Yong, Q. Zhao, D. Meng, and L. Zhang. Variational denoising network: Toward blind noise modeling and removal. In NeurIPS, 2019.
[61] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao. Learning enriched features for real image restoration and enhancement. In ECCV, 2020.
[62] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao. Multi-stage progressive image restoration. In CVPR, 2021.
[63] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
[64] H. Zhang. V. Sindagi. and V. M. Patel. Image de-raining using conditional generative adversarial network. CSVT, 2019.
[65] Zhang, Pan, Ren, Y. Song, L. Bao, R. W. Lau, and M.-H. Yang. Dynamic scene deblurring using spatially variant recurrent neural networks. In CVPR, 2018.
[66] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. TIP, 2017.
[67] K. Zhang, W. Zuo, S. Gu, and I.. Zhang. Learning deep enn denoiser prior for image restoration. In CVPR, 2017.
13

---

[68] K. Zhang, W. Zuo, and L. Zhang. Ffdnet: Toward fast and flexible solution for enn enn-based image denoising. TIP, 2018.
[69] K. Zhang, W. Luo, Y. Zhong, Ma, Stenger, W. Liu, and H. Li. Deblurring by realistic blurring. In CVPR, 2020.
[70] K. Zhang, Y. Li, W. Zuo, L.. Zhang, L. Van Gool, and R. Timofte. Plug-and-play image restoration with deep denoiser prior. TPAMI, 2021.
[71] L.. Zhang, X. Wu, A. Buades, and Li. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal Electronic Imaging, 2011.
[72] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, 2018.
[73] Y. Zhang. Y. Tian, Y. Kong, B. Zhong, and Y. Fu. Residual dense network for image restoration. TPAMI, 2020.
[74] Y. Zhang, K. Li, K. Li, G. Sun, Y. Kong, and Y. Fu. Accurate and fast image denoising via attention guided scaling. 77P, 2021.
[75] Y. Zhang, H. Wang, C. Qin, and Y. Fu. Aligned structured sparsity learning for efficient image super-tesolution. In NeurlPS, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec.
6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See supplemental material.
(c) Did you report error bars (e.g., with respect the random seed after running experi-
ments multiple times)? [No] The random seed is fixed for reproducibility.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplemental material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] The data used for experiments are all publicly available. (e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] The data we used contain no personally identifiable information offensive content.
5. If you used crowdsourcing conducted research with human subjects...
14

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "lie0Cgor6n6", "reviewer": "Reviewer_86x1", "summary": "This paper is interesting. Spatial attention is essential in windowed transformer. By stochastic varying the window sizes and shapes, image restoration benefits from signficantly in various testsets compared with SOTAs. ", "questions": "1. Can this technique be extended to other CV tasks ?", "limitations": "1. Failure cases can be presented. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. First work of Stochastic Window transformer for image restoration \n2. New SOTAs for image restoration\nWeaknesses:\n1. Complexity of new stochastic window is not practically analyzed. \n2. Improvement on visual quality is not very obvious. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_R9GtChfZpl", "reviewer": "Reviewer_KzwD", "summary": "This paper raises an issue regarding the loss of translation equivariance in image restoration problems due to the transformer-based architectures. Often, attention or MLP layers operate on a predetermined window locations and the authors argue this could cause undesired artifacts or performance drops.\nThe authors proposes Stochastic Window Transformer by 1) allocating random window positions at training time and 2) averaging the results from multiple window positions.\nThe experiments were done in image deraining, denoising, and deblurring.", "questions": "I would suggest the authors check the definition of translation-equivariance and translation-invariance.\nAlso, the proposed method does not guarantee translation-equivariance.\n\nPlease refer to the weaknesses.", "limitations": "Limitations were addressed by the authors.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- translation equivariance\n\ntranslation invariance -> translation equivariance\nAs translation equivariance is the key property this paper is addressing, the authors should use a proper terminology. \nAn example task where translation invariance is needed is image classification, recognizing a number correctly whether it is shifted or not.\nGiven a translation function T, \nTranslation invariance: f(T(x)) = f(x)\nTranslation equivariance: f(T(x)) = T(f(x))\n\nL29 is the desiderata -> is one of the desiderata\nI would omit ‘any’ to be safe. There could be cases the position is important, ex) recovering from image distortion, vignetting, etc.\n\nL39-40 As attention layers are involved, I don’t expect the method to be translation equivariant in any ways.\n\n- locality\n\nL36, Figure 1b. If non-overlapping patches cause problems in locality, we could always use overlapping patches and crop out the boundaries at the cost of additional computation. It could be less efficient but solves the problem. Such strategy is commonly used to process high-resolution images even for translation-equivariant methods (CNNs).\n\n- layer expectation propagation algorithm\n\nHow does equation (5) work? Does it require inference over xi to compute expectation per attention block?\nThis introduces s^2 times more inference computations. Is the result worth the computation?\nWhile Figure 7 claims reducing the number of inference brings efficiency, however, x8 inference is still a very large burden.\nAlso, in the supplementary material, the PSNR differences between the inference results with and without the layer expectation propagation are minor.\n\n\n- Figure 4\n\nWhat is shown in Figure 4?\nWhat does the z-axis mean?\nL211 Is the PSNR normalization factor the same across 4b, 4c, and 4d?\nTranslation equivariance is one of the desired property but achieving high PSNR is also important.\nWhich approach achieved the best average PSNR?\n\nWhy want translation equivariance in transformers? People use transformers by sacrificing equivariance to achieve better results. Does bringing approximate translation equivariance lead to higher accuracy?\n\n- Figure 5\n\nWhile the PSNR gains in the Stoformer++ is big, the visual result doesn’t look that much different from other results.\nI don’t find this example to be appealing.\n\n- References\n\n[30] It is ICCV Workshop paper, not ICCV\n\n- Typos and Grammar\n\nL17 constantly -> consistently\nL32 What does ‘concluded’ mean? Is it ‘considered’?\nL167 By contrast -> In contrast\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "US2ILHA-SIU", "reviewer": "Reviewer_S7Qf", "summary": "This paper proposes a stochastic window strategy for training transformers and elaborates layer expectation propagation algorithm during the testing. This strategy consistently improves the performance of several image restoration tasks.", "questions": "See the Strengths And Weaknesses.", "limitations": "See the Strengths And Weaknesses.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea is interesting. With the stochastic window strategy, attention modules can avoid the broken translation invariance.\n2. The experiments show the improvements of each step clearly.\n\nWeaknesses:\n1. The training step is easy to understand. But, I do not fully understand the testing step. How to implement it to avoid the original expensive operations? Does the testing flops increase compared with the training? How about the detailed comparisons (flops and memory cost) of your method with the fixed(shift) window and sliding window during the training and testing?\n\n2. I encourage the author to adopt this strategy to existing transformer-based methods (e.g., Uformer[1], swinIR[2]) to evaluate the generalization on different backbones.\n\n3. L245. Without the layer expectation propagation algorithm, why does the model performs well?  The window during the testing is the same as the shift windonw and the local information has been broken as in previous works.\n\n4. L21-L47 in the Supplemental Material. Two algorithms mixed the dot production and matrix production.\n\n[1] Uformer: A general u-shaped transformer for image restoration\n\n[2] Swinir: Image restoration using swin transformer.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HuClDt4VCTO", "reviewer": "Reviewer_jX7b", "summary": "The paper points out the deficiencies, i.e., loss of locality and broken translation invariance, of existing transformers for image restoration tasks. To address these problems, the authors introduce the stochastically shifted window to replace the original fixed window with transformers and elaborate the layer expectation algorithm to take the average of corresponding stochastic factors. The paper also provides sufficient experimental evidence to support the effectiveness of the proposed method.", "questions": "1. See weaknesses.\n2. It is suggested to release the code to encourage further research about transformers on low-level vision.\n", "limitations": "In Limitation and Future Work (Sec. 4 of S.M.), the authors point out that they will validate the proposed stochastic window strategy on more architectures and image restoration tasks. In this paper, extensive experiments across multiple degradations (image deraining, denoising, deblurring) to demonstrate that the stochastic window strategy can bring consistent improvements. Moreover, the paper also provides clear and reasonable interpretations of the improvements. Hence, it can be anticipated that the stochastic window strategy can bring improvements to other image restoration tasks and architectures. As for the potential negative societal impacts, the authors also provide several solutions to mitigate them.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper provides significant insights about applying transformers to image restoration tasks. For low-level vision, researchers often directly apply high-level design keys of transformers into low-level vision without careful considerations. This paper exhibits consequent loss of locality and broken translation invariance of transformers. Given the potentially general impact of transformers on computer vision, analyzing the impropriety of transformers on specific tasks is of great meaning. \n2. The proposed method is quite novel and effective. To address the problem of the broken translation invariance and locality loss, the authors adopt the idea of sampling to uniformly sample one shift configuration (ξ_h, ξ_w) and design the efficient yet effective layer expectation propagation algorithm to take the expectation of the introduced stochastic factors. The authors also conduct extensive experiments to support the proposed method.\n3. The paper is well organized and easy to be understood.\n\nWeakness:\n1. Line 144-145 says that “stochastic shifts should be averaged according to their posterior distribution” and Eq. 4 implies that these random variables are averaged according to a uniform distribution. So, there should be some explanations about why the implied posterior distribution is uniform.\n2. The stochastic window transformer contains the layer expectation propagation algorithm to approximate the exact average (expectation) of stochastic shifts. It is suggested to add a concrete explanation about the reason of taking expectations. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept