Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a multi-resolution based attention mechanism named MrsFormer. Various experiments have been conducted to validate the effectiveness of the proposed method over several baselines
strength_and_weaknesses: Strengths:
* The authors focus on an important problem in sequence modeling
* The technical details are presented well

Weakness
* The novelty needs to be further justified
* The draft needs to be better organized
* The experiments need to be strengthen
clarity,_quality,_novelty_and_reproducibility: The technical details are sufficient; however the experiments need include more state of the art methods as baselines to be more solid
summary_of_the_review: OnThe authors needs to better justify the novelty and the motivation of the proposed method. From the introduction, the authors only briefly introduce some technical description without any motivation or rationale of proposing MrsHA. I suggest the authors move the background knowledge parts (e.g., self-attention description) into Sec. 2 then highlight the motivation and justify the novelty.

The paper also misses many state-of-the-art efficient transformer variants such as Luna, Linformer, Performance, etc.. I suggest the authors include these relevant methods as baselines in the evaluation or better justify the criteria of why selecting the existing baseline methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores the idea of using wavelet transform to compress the input data length to achieve the approximation of attention map and value matric in a Transformer. The experimental results show that it outperforms some state-of-the-art methods. And the proposed method performs better computational and memory cost than the vanilla Transformer.
strength_and_weaknesses: The proposed approach has higher computational efficiency and lower memory overhead than the original Transformer at the cost of a small amount of performance. Although the proposed method is more efficient than the current version of the vanilla Transformer, a comparison with the computational efficiency and memory cost of existing methods is missing.
clarity,_quality,_novelty_and_reproducibility: The topic is a very interesting and a challenging problem. And the proposed Multiresolution-head Attention is somehow novel. The performance shows the advantage of the proposed method. Overall, the paper is well-organized.
summary_of_the_review: The paper proposes an improved model of Transformer, Multiresolution-head Attention, to reduce computation and memory overhead. The proposed method is more efficient through additional experiments and complexity analysis than the original Transformer. However, the experimental part lacks a comparison with the algorithmic complexity of existing methods. The overall organization of the paper is clear.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a technique to improve the efficiency of multi-head attention. 
The idea is to apply the multi-resolution decomposition with orthogonal bases inspired by wavelets to attention matrics or the V matrices. 
The Haar Wavelet (box-car like) is considered in this paper, resulting in up/down sampling combined with multi-head attention. 
Experimental results show that the proposed method improves FLOPS and memory uses compared to a naive softmax and the block-diagonal attention matrix approximation. 
strength_and_weaknesses: (+) Multi-resolution decomposition with orthogonal bases is a well-studied and proven-to-be-effective strategy for approximate signals. A reasonable choice. 

(+) The paper reads easily, and straightforwardly. No logical jumps. 

(--) Less technical novelty

(--) Less impact in experimental results

clarity,_quality,_novelty_and_reproducibility: The technical discussions are clearly written. Many readers may find the paper easy-to-read 
Thanks to the clear writings, reproducing may not be so hard. 

I find two major concerns during the review. 
(1) Novelty 
Signal decomposition with orthogonal, multi-resolution bases passes the "test-of-time": no doubt in its efficacy. 
If I correctly understand, the main idea is to simply plug-in the multi-reso decomposition with Haar-like Wavelet on the target matrices (Attention or V). I cannot find any other technical challenges or new ideas to solve the challenges. 
Given these, it is difficult for me to evaluate the paper novel enough to accept at ICLR. 

(2) impact of experiments
The experimental results are not strong. 
The results show that the proposed method is marginally better than the existing Multi-Reso. Approximations. However, the main goal of the paper is to improve the efficiency of Transformer. In that sense, other approaches (not Multi-Reso Approx.) presented in the related work did better jobs. 

For example, the FMMformer [Nguyen+, 2021] claims that the combination of low-rankness and sparsity yields linear-time complexity for self-attention. 
NAFNet [Chen+, 2022] reduces the computational cost (in terms of Multiply-ACcumulation Operations MACs) up to 8.4% of the naive Transformer while achieving a new SotA in denoising. 
Perceiver [Jaegle+, 2021] proposes to replace the pair-wise self-attention to the cross-attention between the low-dimensional Key latent vectors and the high-dimensional input vectors, yielding linear-time complexity with great recognition accuracy in several domains. 

Compared to these results, the gains achieved in this paper are not significant. Then it is difficult for me to claim the necessity of the Multi-Res. Approx. approach to improve the efficiency of the Transformer at this moment. 

I expect clearer advantages of the multi-res. approx. approch are provided through author-feedback time!


[Chen+, 2022] Chen, Chu, Zhang, and Sun, "Simple Baselines for Image Restoration", ECCV 2022. 
[Jaegle+, 2021] Jaegle, Gimeno, Brock, Zisserman, Vinyals, and Carreira, "Perceiver: General Perception with Iterative Attention", ICML 2021. 

Minor comments:
The discussion about the redundancy of the Multi-Reso Approximation in Sec 1.2: in general I agree to these claims, but it is better to have some citations to support the claims. 
summary_of_the_review: The technical idea is clear. But the novelty and the impact of experimental results are not enough for recommend to accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces MrsFormer, a transformer network with multiresolution attention. The authors apply a multi-resolution decomposition to attention matrices and to the V matrices. Experiments are conducted for time series classification and for image classification on ImageNet. The authors show that MrsFormer can improve efficiency while limiting the drop in prediction performances.   
The paper initially received two reject (3) recommendations and one borderline accept (6) recommendation. The main concerns pointed out by the reviewers related to the novelty of the approach, the significance in experiments and the lack of comparison to other efficient attention baselines. During the discussion period, the authors provided a combination of their approach with efficient attention methods, and new segmentation results on ADE-20K. Although these additional experimental results were appreciated by reviewers, they did not successfully answered to the concerns on novelty, and there remained a majority of reviewers voting for rejection after rebuttal.  

The AC carefully reads the submission and discussions. Several works have recently explored the idea of a multi-resolution approximation of attention matrices (Zeng et al., 2022; Fan et al., 2021; Tao et al., 2020; Li et al., 2022), as discussed in the paper. The difference in the submission is to approximate the output of an attention head rather than the attention matrices, but the experimental results only show marginal improvements of the proposed approach. The AC also considers that the experiments can be consolidated. On ImageNet, the generalization performances should be assessed on ImageNet-v2 since the ImageNet dataset does not have a separate validation and test set. In addition, the proposed approach is inferior to the softmax baseline on ImageNet, but seems to be superior for time series classification or on the new segmentation results: the authors did not comment if their approach might have an expected regularization effect, or if the improvements in performances are due to other factors.   
Therefore, the AC recommends rejection, but encourages the authors to re-submit their work based on the reviews' feedback.
justification_for_why_not_higher_score: - Limited novelty and experimental improvement compared to previous works in multi-resolution approximation of attention matrices
- Experiments needs consolidations and more analyses
Decision: Reject