Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a stochastic window strategy for training transformers and elaborates layer expectation propagation algorithm during the testing. This strategy consistently improves the performance of several image restoration tasks.
strengths_and_weaknesses: Strengths:
1. The idea is interesting. With the stochastic window strategy, attention modules can avoid the broken translation invariance.
2. The experiments show the improvements of each step clearly.

Weaknesses:
1. The training step is easy to understand. But, I do not fully understand the testing step. How to implement it to avoid the original expensive operations? Does the testing flops increase compared with the training? How about the detailed comparisons (flops and memory cost) of your method with the fixed(shift) window and sliding window during the training and testing?

2. I encourage the author to adopt this strategy to existing transformer-based methods (e.g., Uformer[1], swinIR[2]) to evaluate the generalization on different backbones.

3. L245. Without the layer expectation propagation algorithm, why does the model performs well?  The window during the testing is the same as the shift windonw and the local information has been broken as in previous works.

4. L21-L47 in the Supplemental Material. Two algorithms mixed the dot production and matrix production.

[1] Uformer: A general u-shaped transformer for image restoration

[2] Swinir: Image restoration using swin transformer.

questions: See the Strengths And Weaknesses.
limitations: See the Strengths And Weaknesses.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper raises an issue regarding the loss of translation equivariance in image restoration problems due to the transformer-based architectures. Often, attention or MLP layers operate on a predetermined window locations and the authors argue this could cause undesired artifacts or performance drops.
The authors proposes Stochastic Window Transformer by 1) allocating random window positions at training time and 2) averaging the results from multiple window positions.
The experiments were done in image deraining, denoising, and deblurring.
strengths_and_weaknesses: - translation equivariance

translation invariance -> translation equivariance
As translation equivariance is the key property this paper is addressing, the authors should use a proper terminology. 
An example task where translation invariance is needed is image classification, recognizing a number correctly whether it is shifted or not.
Given a translation function T, 
Translation invariance: f(T(x)) = f(x)
Translation equivariance: f(T(x)) = T(f(x))

L29 is the desiderata -> is one of the desiderata
I would omit ‘any’ to be safe. There could be cases the position is important, ex) recovering from image distortion, vignetting, etc.

L39-40 As attention layers are involved, I don’t expect the method to be translation equivariant in any ways.

- locality

L36, Figure 1b. If non-overlapping patches cause problems in locality, we could always use overlapping patches and crop out the boundaries at the cost of additional computation. It could be less efficient but solves the problem. Such strategy is commonly used to process high-resolution images even for translation-equivariant methods (CNNs).

- layer expectation propagation algorithm

How does equation (5) work? Does it require inference over xi to compute expectation per attention block?
This introduces s^2 times more inference computations. Is the result worth the computation?
While Figure 7 claims reducing the number of inference brings efficiency, however, x8 inference is still a very large burden.
Also, in the supplementary material, the PSNR differences between the inference results with and without the layer expectation propagation are minor.


- Figure 4

What is shown in Figure 4?
What does the z-axis mean?
L211 Is the PSNR normalization factor the same across 4b, 4c, and 4d?
Translation equivariance is one of the desired property but achieving high PSNR is also important.
Which approach achieved the best average PSNR?

Why want translation equivariance in transformers? People use transformers by sacrificing equivariance to achieve better results. Does bringing approximate translation equivariance lead to higher accuracy?

- Figure 5

While the PSNR gains in the Stoformer++ is big, the visual result doesn’t look that much different from other results.
I don’t find this example to be appealing.

- References

[30] It is ICCV Workshop paper, not ICCV

- Typos and Grammar

L17 constantly -> consistently
L32 What does ‘concluded’ mean? Is it ‘considered’?
L167 By contrast -> In contrast

questions: I would suggest the authors check the definition of translation-equivariance and translation-invariance.
Also, the proposed method does not guarantee translation-equivariance.

Please refer to the weaknesses.
limitations: Limitations were addressed by the authors.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is interesting. Spatial attention is essential in windowed transformer. By stochastic varying the window sizes and shapes, image restoration benefits from signficantly in various testsets compared with SOTAs. 
strengths_and_weaknesses: Strengths:
1. First work of Stochastic Window transformer for image restoration 
2. New SOTAs for image restoration
Weaknesses:
1. Complexity of new stochastic window is not practically analyzed. 
2. Improvement on visual quality is not very obvious. 
questions: 1. Can this technique be extended to other CV tasks ?
limitations: 1. Failure cases can be presented.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper points out the deficiencies, i.e., loss of locality and broken translation invariance, of existing transformers for image restoration tasks. To address these problems, the authors introduce the stochastically shifted window to replace the original fixed window with transformers and elaborate the layer expectation algorithm to take the average of corresponding stochastic factors. The paper also provides sufficient experimental evidence to support the effectiveness of the proposed method.
strengths_and_weaknesses: Strength:
1. The paper provides significant insights about applying transformers to image restoration tasks. For low-level vision, researchers often directly apply high-level design keys of transformers into low-level vision without careful considerations. This paper exhibits consequent loss of locality and broken translation invariance of transformers. Given the potentially general impact of transformers on computer vision, analyzing the impropriety of transformers on specific tasks is of great meaning. 
2. The proposed method is quite novel and effective. To address the problem of the broken translation invariance and locality loss, the authors adopt the idea of sampling to uniformly sample one shift configuration (ξ_h, ξ_w) and design the efficient yet effective layer expectation propagation algorithm to take the expectation of the introduced stochastic factors. The authors also conduct extensive experiments to support the proposed method.
3. The paper is well organized and easy to be understood.

Weakness:
1. Line 144-145 says that “stochastic shifts should be averaged according to their posterior distribution” and Eq. 4 implies that these random variables are averaged according to a uniform distribution. So, there should be some explanations about why the implied posterior distribution is uniform.
2. The stochastic window transformer contains the layer expectation propagation algorithm to approximate the exact average (expectation) of stochastic shifts. It is suggested to add a concrete explanation about the reason of taking expectations. 

questions: 1. See weaknesses.
2. It is suggested to release the code to encourage further research about transformers on low-level vision.

limitations: In Limitation and Future Work (Sec. 4 of S.M.), the authors point out that they will validate the proposed stochastic window strategy on more architectures and image restoration tasks. In this paper, extensive experiments across multiple degradations (image deraining, denoising, deblurring) to demonstrate that the stochastic window strategy can bring consistent improvements. Moreover, the paper also provides clear and reasonable interpretations of the improvements. Hence, it can be anticipated that the stochastic window strategy can bring improvements to other image restoration tasks and architectures. As for the potential negative societal impacts, the authors also provide several solutions to mitigate them.
review_title: null

Meta-review: metareview: The paper proposes a new stochastic window strategy for image restoration. The stochastic window transformer layer is invariant to translations and is applied to the image degradation and mitigates loss of locality, hence making the approach potentially more robust.
The reviews of the papers were mixed, with strong pro- and contra. One point of contention was whether the layer made sense at all, that is whether the invariance assumption is correct for image restoration. Another point of criticism was the additional computational burden imposed by the stochastic window layer. On the other hand reviewers liked SOTA performance, the novelty of the presented ideas and the paper writing. The rebuttal phase addressed many issues raised by reviewers, including the equivariance vs. invariance issue.
In my opinion the paper is an interesting and elegant theoretical contribution. I agree that *invariance w.r.t. image degradation* is indeed the right concept. While the additional runtime overhead may prevent the paper's method from application in time intensive scenarios, I still think that the approach has enough practical and theoretical strong points to merit publication.
confidence: Less certain
Decision: Accept