Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors first explain the analogy between Transformer and the classic Evolutionary Algorithm (EA). Then, inspired by the dynamic local population in EA, the authors propose the EAT model, which integrates local operations (e.g., convolution, local attention, etc.) in the Transformer model. Besides, the authors design spatial-filling curve and task-related heads to improve the performance and generalization of the EAT model. Experiments on three tasks, including image classification, image retrieval and vision language navigation, demonstrate the effectiveness of the proposed EAT model.
main_review: This paper explains the rationality of Vision Transformer by the analogy between Transformer and the classic Evolutionary Algorithm (EA). Based on this analogy, the authors improve the Transformer model by three modules, including the integration of local operations in the attention layer, spatial-filling curve, and the task-related heads.

First, the analogy between Transformer (TR) and Evolutionary Algorithm (EA) is not much convincing. Convolution, attention, and fully-connected layers are just common operations to integrate low-level representations to form high-level semantics. These three operations share the same mathematical representation of X’ = WX. I didn’t see that the analogy between TR and EA could extend the explainability of TR.

Second, as for the local path and global path in EAT, this design is essentially a way to fuse global and local representations. For a representation, some channels are used for global operations (i.e., MSA), other channels are used for local operations (i.e., CNN or local attention). However, the fusion of global and local information has been discussed in many papers like [1,2,3,4]. And the novelty in this paper is limited.

Third, the ablation studies in Table 4 shows the effectiveness of spatial-filling curve and task-related heads. However, the improvements of EAT over the state-of-the-art methods on ImageNet in Table 1 are not significant (e.g., EAT-B obtains 82.0 while DeiT-B obtains 81.8 under the same number of parameters).
The quality of paper writing is good. It is easy for readers to understand the ideas of this paper. All the figures and tables are well illustrated. And the whole paper is well organized.

[1] Wang, Xiaolong, et al. "Non-local neural networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[2] Cao, Yue, et al. "Gcnet: Non-local networks meet squeeze-excitation networks and beyond." Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019.
[3] Qiu, Zhaofan, et al. "Learning spatio-temporal representation with local and global diffusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
[4] Srinivas, Aravind, et al. "Bottleneck transformers for visual recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

limitations_and_societal_impact: Overall, the contribution of this paper to the research community is not significant enough. The analogy between Transformer (TR) and Evolutionary Algorithm (EA) could not promote the explainability of TR. The integration of local and global representations is not novel in the literature of deep learning. The spatial-filling curve and the task-related heads do not show significant improvements in the experiments on ImageNet.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this article, the author
1. explain the transformer from the EA algorithm perspective
2. uses the dynamic local population strategy used in EA algorithm and transfer the strategy into the transformer
3. introduces the spatial filling curve to transform the image data
4. achieves SOTA results on a number of tasks.
main_review: pros:
1. This article has a certain degree of innovation
2. The result of this article is good

cons:

Q1. The writing of this article is not clear

Q1.1 This article contains too much information, the author has too many things to say. But the main contribution is not described clearly. I suggest the author use a few sentences to describe the core contribution of this article and write it in the first part. List by points.

Q1.2 The framework of this method seems complex. It is suggested to insert a diagram on the second page and highlight the core contribution.

Q2. The necessity of introducing EA

Q2.1 What is the meaning of introducing EA algorithm? What are the unique advantages of local population strategy in EA algorithm? Why is it useful to introduce local information because EA is similar to transformer? In my opinion, the introduction of EA algorithm has some over packaging.

Q2.2 Again local? The purpose of introducing transformer into CV task is to deal with the image information directly in a uniform way, without considering the sequence relationship between the image patches, and deal with the problem of CV from a global perspective. This article seems to reuse some features of images, such as local information and sequence to transformer? This sounds like considering some special features of the image and used in the transformer. I don't think it's bad, but I think it might be contrary to the original intention of the transformer.

Q3. Method & Experiments

Q3.1 Please report the FLOPs.

Q3.2 Please add all the Efficientnet models in Fig. 6. It seems that the improvement is incremental.

Q3.3 About local. It actually introduces some priors of CNN's local information into transformer to a certain extent. The idea of local is not new and needs to quote more relevant papers.

Q3.4 About SFC. If the input image is not a regular rectangle, what is the impact? Is the strategy adopted by the author the optimal one for 2D image arrangement?

========== post rebuttal ============

I have read the rebuttal and the rebuttal addresses most of my concerns. I will change my score from 4 to 6.

The rebuttal should be merged into main paper in the next version.
limitations_and_societal_impact: Please see the main review part.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper attempts to explain the rationality of Vision Transformer by the proven practical Evolutionary Algorithm and finds that both of them have consistent mathematical representation. Besides, for separating the architecture of transformer-based network from the data format adaptation, the authors introduce the spatial-filling curve into the vision transformer to sequence multiple dimensional data (e.g., image or video) into a uniform sequential format. Thereby the proposed transformer-based network can deal with multi-modal tasks.
main_review: It is interesting that the theory of EA is introduced to explain the rationality of Vision Transformer. The authors give the comparison of the basic operator in EA with the module in transformer. It is concluded that the mathematical representations of Mutation operator with FFN Module, Crossover Operator with SA Module are consistent. Based on those observations, it is better to give readers more insightful explanations on the success of transformer instead of only stating those phenomenon.

Serializing the multiple dimensional data into sequence with a consistent algorithm is indeed an elegant solution to adapt multi-modal data for transformer. Many algorithms like those in Figure 2 can competent this task, and Hilbert algorithm is selected for serializing the input data. So, what is the reason behind for the authors to make this choice. It would be more convincing if those space filling curve algorithms are compared in the experiment section.

limitations_and_societal_impact: The paper is easy to understand for me, but the writing should be checked carefully, such as “Imeges” in Table 1.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a novel perspective to explains the rationality of vision transformer by analogy with a heuristic evolutionary algorithm and proposes an improved EAT model borrowing the dynamic local population concept from EA. Authors introduce the spatial-filling curve to serialize the input image, so only one unified model is required to handle multi-modal tasks. The extensive classification and multi-modal experiments show that the proposed method is effective and efficient.
main_review: Strong Points 
- The perspective between Transformer and EA is interesting, and the improved EAT is reasonable and effective. The introduced spatial-filling curve makes it possible to deal with multi-modal tasks by one unified model.
- The paper is easy to follow, and its organization is sound. The authors provide the training code, and the reported results are convincing.
- The experiments for classification and multi-modal tasks are extensive, and detailed ablation experiments are carried out.
limitations_and_societal_impact: - Task-related head looks the same as the cross-attention structure in Transformer Decoder[61], and a similar structure is mentioned in Cait[57], so this contribution is optional.
- Spatial-filling curve does not seem to improve the performance, and the SIS mode stated in the paper is equivalent to the current token embedding in ViT.
- EAT increases by 0.5 points on average compared to the baseline in Table 1, but the reported highest Top-1 is 82.4 that is lower than current methods, e.g., SwinTransformer[36] and Cait[57]. Authors should compare with these methods or explain why it is not being compared.
- Why do authors claim that "... so that only 1D operators are required" in Figure 3? Adding 2D operations, such as 2D-convolution in CPVT[16], CeiT[70], and PVTv2[a], has been shown to improve the performance of Transformer in image classification tasks.
- Grammatical errors and typo issues lower the paper quality that can be seen in the following section.  

[a] Wang W, Xie E, Li X, et al. PVTv2:

Moreover, Here are more suggestions. 

- Some quantitative results in Appendix E should be presented in Table 1
- Why does DCN dramatically damage performance in Table 4?
- Figure 6 should be rearranged
- How to integrate CV and NLP features is unclear in multi-modal experiments, and more details should be displayed.
- Some grammar/typo issues.
    - L12, L51: multi-modal 
    - L102: their ways
    - L116: the definition of "h"
    - L161: the weight
    - L174: concepts
    - L314: remove the comma after "format"
review_title: null

Meta-review: This paper provides a view on the Transformer architecture as being related to that of an Evolutionary Algorithm. Based on this, the authors propose some improvements to the architecture and show convincing empirical results to demonstrate their efficacy and efficiency. The reviewers and discussion subsequent to the rebuttal support this view and I trust the authors will incorporate the reviewers' comments into the final manuscript.
Decision: Accept (Poster)