Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes a novel approach to deal with the computational problems of self-attention without introducing independence assumptions. The proposed approach is simple, easy to understand, and easy to implement.

However, evaluation for this paper is severely lacking. As it is, there is not enough information provided to adequately assess the proposed method's strengths in practice. The following should be added:

Evaluation on a variety of different tasks, such as image segmentation, temporally consistent object detection, object tracking, etc. Why are the evaluations limited to generative modeling? To prove the generality of the method (as claimed), it needs to be applied to various tasks.
Runtime (in inference) comparisons for each of the datasets and for each of the baselines. Additionally, a theoretical analysis for runtime in terms of the size of the input should be given (the column in Table 1 should have runtimes for each method clearly specified, and this should be done for each dataset and baseline)
Ablation study. What is the baseline architecture used without axial attention? There is only comparison to previous work which may have used a different architecture.

If these concerns are thoroughly addressed, I would be happy to increase my score.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper claims to propose a new approach to solve the computational problems of self-attention. However, the paper mainly focuses on adapting Transformer for image generation, which has far less applications. The whole paper needs to be rewritten to make their target and contribution clearer.

1. The authors overclaim that they provide a new approach for accelerating self-attention. However, they only adapted Transformer for image generation. In fact, Transformer does not equal to self-attention. Currently, two directional self-attention like Bert has much wider applications compared with Transformer like sequential self-attention. 

2. For a paper claim to improve self-attention, they should show its effectiveness on a broad range of tasks, with comprehensive experimental evaluation. However, authors mainly reported the image generation on several datasets. 

Overall, the authors need to rewrite the paper. They should either show more applications with the proposed self-attention approach or treat it as a new approach for image generation.
review_title: Official Blind Review #5

final_score: rating: 1: Reject
review_content: It is known that the standard self-attention method is computationally expensive and cost a significantly large amount of storage when the number of points to be attended is large.

This paper attempts to solve this problem and proposed the Axial Attention method. It is claimed to be able to save an O(N^(d-1)/d) factor of resources over standard self-attention.

The proposed method looks novel to me, but some of the related works are missing and the experiment session is insufficient. 

1)  The author should at least include the following works which also aim to reduce the cost of self-attention. Since the author did not mention these works which also focus on solving the same problem, It is hard for me to judge if the proposed method is better than existing works.
[a] CCNet: Criss-Cross Attention for Semantic Segmentation
[b] A^2-Nets: Double Attention Networks

2) self-attention has shown its effectiveness on a broad range of computer vision tasks, including image generation, detection, segmentation, and classification. I do not get why the proposed method is only benchmarked for generative models. Is it because the proposed method cannot be adopted on other popular CV tasks, such as detection, segmentation, and classification? Extra experiments should be included if the proposed method is not only designed for generative models.

3) The ablation study is missing. The author directly compared its own method with other existing methods that are implemented and trained with different hyperparameters. It is hard to know which indeed benefits the accuracy gain and how significant is the proposed method. 

4) In table 2 and 3, I do not see a clear advantage of the proposed method over the SOTA methods.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes axial attention as an alternative of self-attention for data arranged as large multidimensional tensors, which costs too much computational resource since the complexity of traditional self-attention is quadratic in order to capture long-range dependencies for full receptive fields. The axial attention is applied within each axis of the data separately while keeping information along other axes independent. Therefore, for a d-dimensional tensor with N = S^d, axial attention saves O(N^{(d−1)/d}) computation over standard attention. The proposed axial attention can be used within standard Transformer layers in a straightforward manner to produce Axial Transformer layers, without changing the basic building blocks of traditional Transformer architecture.  The authors did experiments on two standard datasets for generative image and video models: down-sampled ImageNet and BAIR Robot Pushing, and they claim that their proposed method matches or outperforms the state-of-the-art on ImageNet-32 and ImageNet-64 image benchmarks and sets a significant new state-of-the-art on the BAIR Robot Pushing video benchmark. 

Reasons to accept:

1.	Simple, easy-to-implement yet effective approach to adapt self-attention to large multidimensional data, which can save considerable computation for efficiency, while still have competitive performance.
2.	Clear writing, with sufficient but not redundant introduction of background knowledge and explanation of both the advantages and drawbacks of existing models (too large computational complexity on high-dimensional data).

Suggestions for improvement:

1.	It would be better if the authors can provide more analysis or case study to show the reason why Axial attention (Axial Transformer) can reach good performance even if it omits considerable operations compared to traditional Transformers, or to show why the attention operations within axis are important instead of attention operations between axis. 
2.	Definition of “axis” should be more clear in section 3 (there could be some ambiguities of “axis”).

review_title: Official Blind Review #2

Meta-review: This paper proposes a self-attention-based autoregressive model called Axial Transformers for images and other data organized as high dimensional tensors. The Axial Attention is applied within each axis of the data to accelerate the processing.

Most of the authors claim that main idea behind Axial Attention is widely applicable, which can be used in many core vision tasks, such as detection and classification. However, the revision fails to provide more application for Axial attention.

Overall, the idea behind this paper is interesting but more convincing experimental results are needed.

Decision: Reject