Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new Adaptive Fourier Neural Operator (AFNO) for mixing tokens in visual transformers. The idea is based on Fourier neural operators (FNO) that transform feature flow in the Fourier space. The difference w.r.t. existing FNO is in two modifications. First, the weight matrix is block-diagonal (analog of multi-head) and, second, use of an MLP instead of just linear weighting. The experiments show that the proposed method is competitive and often achieves results as good as with original self-attention (with fewer flops).
main_review: The paper proposes an alternative mechanism for self-attention in the context of vision transformers. The proposed method reduces the mixing complexity from quadratic to N log N, which could be important for long token sequences, e.g. high-resolution images.

The paper uses a well-established theory of Fourier transform motivating the proposal, which is neat.

It is hard from the paper presentation to compare with other efficient attention methods - the comparison is done using two metrics, accuracy and flops. Therefore it is not possible to judge if the paper achieved the state-of-the-art. 

The novelty is somewhat limited. The differences with GFN, i.e. adding MLP and channel mixing, do not seem particularly insightful or significant.

One weakness is that, at least on the chosen applications, the quadratic complexity of attention is not a bottleneck of the networks. At least it is not clear from the paper. For example, table 4 suggests that the mixer takes at most 35% of total compute, with the rest consumed by the point-wise MLP. From this angle, the significance of the work is not convincingly demonstrated. 

I find it weird that the paper on vision transformers does not cite the original Dosovitsky et al. 
summary_of_the_review: Despite limited novelty and significance, I found the work interesting and recommend acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: As is known, vision transformer has been becoming a more and more popular topic in the area of computer vision which is inspired by the success of this deep neural network fashion in other fields. However, the computing burden is a common disadvantage across the majority of transformer models compared with other deep neural networks. From a view of adaptive network operation in transformer, authors proposed to improve its expressiveness and generalization by imposing block-diagonal structure, adaptive weight-sharing, and sparsity.
main_review: It's an interesting idea and a good attempt that aims to propose a solution to reduce the computing burden while maintaining the performance of visual transformer models. The motivation of introducing a Fourier Neural operator is reasonable to improve its expressiveness and generalization by imposing block-diagonal structure, adaptive weight-sharing, and sparsity. However, from my view, the main weakness of this study is lacking comparison experiments on general computer vision tasks with state-of-the-art visual transformer models such as image classification, object detection (these tasks have been included in another visual transformer model "Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." arXiv preprint arXiv:2103.14030 (2021)."). For example, performance of swin transformer on imagenet 1k and 22k are both superior to this paper. According to the advantages shown in this study, it's better to be titled "an improved transformer model for few-shot segmentation" rather than the existing one which seems much more ambitious. When it comes to more details of this paper, the proposed methods perform very similar with or even worse than some existing ones while needing more network parameters (Table 4).

summary_of_the_review: It's an interesting study that aims to propose a solution to a common challenge in the visual transformer model. However, I can't support its acceptance considering the presented results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Vision transformers scale quadratically with the number of pixels. To cope with this challenge, this paper proposes Adaptive Fourier Neural Operator (AFNO) as an efficient token mixer that learns to mix in the Fourier domain. This is achieved by modifying FNO, including imposing a block-diagonal structure on the channel mixing weights, adaptively sharing weights across tokens, and sparsifying the frequency modes via soft-thresholding and shrinkage. The resulting model has a quasi-linear complexity and linear memory in the sequence size.
main_review: The idea of this paper is novel and interesting. My main concern is its experiments: I think that the experiments are insufficient to support the conclusion and demonstrate the effectiveness of the proposed method. This paper should provide extensive experiments and comparisons for image classification, object detection, instance segmentation, and semantic segmentation. The ablation studies should also be conducted on the ImageNet dataset for image classification. These tasks are the most fundamental ones, which can thus reflect the ability of the represent learning of networks. 

However, this paper provides results for some uncommon tasks, like image inpainting and few-shot segmentation. Hence, I have reasons to guess that the proposed method is only effective for these carefully chosen tasks. Although this paper provides some results for semantic segmentation and image classification, the experiments for semantic segmentation are only conducted using Segformer-B3 and the experiments for image classification are only conducted using ViT-S. This seems that this paper carefully chooses one model for each task, and the comparison to state-of-the-art transformer models is also missing. Therefore, I think that current experiments are insufficient to demonstrate the effectiveness of the proposed method.

summary_of_the_review: My main concern is about the insufficient experiments. Please see the above comments!
review_title: null

Meta-review: Overall, this paper receives positive reviews. The reviewers find the technical novelty and contributions are significant enough for acceptance at this conference. The authors' rebuttal helps address some issues. The area chair agrees with the reviewers and recommend it be accepted at this conference.
Decision: Accept (Poster)