Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel benchmark, called CAB,  to compare the efficiency of different attention mechanisms in transformers. The current widely used benchmark LRA focusses only on noncausal self-attention. CAB seeks to evaluate four types of attention mechanisms: the combinations of causal/ non-causal and self/ cross attention. The benchmark consists of nine datasets for seven real-world tasks across vision, NLP, speech and time-series forecasting. CAB  is also used to evaluate 8 baseline models covering the four attention mechanisms using standard statistical performance metrics and a proposed composite index as well their efficiency – relative running times and memory usage as a function of the sequence length, highlighting the strengths and weakness of efficient approach across the four attention types suggesting where more research is needed.

strength_and_weaknesses: **Strengths**

- CAB is a novel benchmark with a finer granularity of attention taxonomy than any existing benchmark. LRA addresses only one of the 4 types of attention mechanisms.
-  CAB comprises 7 tasks of interest to the research community on 9 datasets being used by the research community across 4 domains.
- The paper also evaluates the performance of 8 popular models as baselines as a part of CAB for new approaches to be benchmarked against.
- The paper further identifies the scenarios where efficient approaches are significantly lagging behind vanilla attention in performance as the unmet need for the research community to focus on.

**Weaknesses**

***Analysis of Results*** 
- The paper should provide more insights on why results from certain methods vary across various studies: local attention in (Xiong et al, 2022) vs LRA; S4D on LRA vs Sum, MLM tasks, etc.
- (Figure 3) In the ‘self’ regime, why is it that performance holds across LRA and NS but drops (in correlation) for causal-self? 
- (Figure 3) There is a clear blockiness along the diagonals indicating high positive correlation in performance of approaches within the ‘self’ scenarios (LRA, non-causal and causal) and within the ‘cross’ scenarios (non-causal as well as causal). What may be the reasons for high negative correlation in the off-diagonal blocks? In other words, can you kindly elaborate on footnote 4? Also, what explorations need to be done beyond this benchmark?
- It is not clear what ‘non-homologous information’ is. Kindly provide explanation and a reference for the term.

***Clarity***
- Figures showing the tradeoff between performance (accuracy) and cost (time, memory) are recommended to communicate the tradeoffs better.
- Writing should be improved in Sections 4 and 5. The authors should avoid using words/ phrases like ‘utterly defeated’, ‘embarrasingly’,  etc. 
- Appropriate capitalization should be used in the ‘References’ section.
- Some easily fixable typos.

clarity,_quality,_novelty_and_reproducibility: ***Clarity***: The paper is mostly clear and easy to read.

***Quality***: The overall quality of the submission is acceptable.

***Novelty***: The benchmark is novel, more comprehensive that the previous benchmarks and would be useful to compare future approaches and help spur research into efficient approaches.

***Reproducibility***:  Should not be a concern as the authors have promised that all the code will be released on github. 

summary_of_the_review: The paper introduces a new benchmark for comparing and evaluating efficient attention mechanisms. I expect the benchmark to be useful to the research community and is likely to help in guiding research into the design of novel efficient attention mechanisms, especially in the causal and cross-modal domains. There is an extensive benchmarking of exemplary baseline algorithms. The paper is mostly written well (though the writing, clarity and the discussions can be somewhat improved).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper:  The authors propose Comprehensive Attention Benchmark (CAB) under a fine-grained attention taxonomy with four distinguishable attention patterns, namely, noncausal self, causal self, noncausal cross, and causal cross attentions. The benchmark collects seven real-world tasks from diverse fields of computer vision, natural language processing, speech processing, and time series forecasting. And an interesting finding is that some efficient Transformers often achieve less competitive results in the causal cross scenario.
strength_and_weaknesses: Strength:
1. A benchmark to evaluate different types of attention patterns for the task with long sequences
2. The benchmark covers a broad number of tasks.

Weakness:
1. The number of baselines in figures b,c,d are limited. It would be better to list more.
2. LongT5, "Efficient Text-To-Text Transformer for Long Sequences", also works on the causal attention mechanism. All the datasets in LongT5 could also be a good benchmark. Language model is also a widely used task to evaluate causal attention. Thus, the benchmark may not be novel enough if only considering the NLP area.
clarity,_quality,_novelty_and_reproducibility: Can you list the SOTA results with model pertaining for NLP tasks?
summary_of_the_review: The benchmark is interesting and covers many tasks from different domains. The exploration of different attention mechanism is interesting. My major concern is that there has been benchmarks for NLP tasks. After all, I would like to see people working on this new benchmark.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose Comprehensive Attention Benchmark (CAB) under a fine-grained attention taxonomy with four distinguishable attention patterns, namely, noncausal self, causal self, noncausal cross, and causal cross attentions.

strength_and_weaknesses: Strength
1. the paper is well written and feeling pleasure to read. It summarize many eixting of Transformer.

2. The authors propose Comprehensive Attention Benchmark for ling sequence modeling.

Weaknessness
1. the novelty of this work is not enough for ICLR. It seems that this paper is more related to benchmark dataset.

2. They claim that  related codes will be released at https://github.com/Anonymous. However, I cannot find it.

clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well written and it is clear.

Quality: The quality of this work is good.

Novelty: The novelty is limited.

Reproducibility: unclear at this time.
summary_of_the_review: This paper proposes to evaluate efficient attentions under a more fine-grained attention taxonomy
with four distinguishable attention patterns, each of which reflects different attentive functionality.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents CAB, a new benchmark to test Transformers and other sequence models on long sequence data. The paper benchmarks a number of methods as a baseline.
strength_and_weaknesses: + Fills in a missing gap in the literature beyond LRA
+ Will be useful for researchers evaluating long-sequence models

- There appear to be a few basic mistakes in the methods characterization
- It would be nice to have a few more baselines
- Paper could use some copy edits
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clear to follow, but could use some copy editing. I think the focus on distinguishing self-attention from causal/cross/etc attention is a little bit misplaced.

Quality: I think the most useful contribution of this paper is simply aggregating more long-sequence benchmarks as "standard." The categorization of methods into self-attention, causal, cross, etc seems to have some basic mistakes.

For example, S4D is inherently a causal model (this is the definition of a state space model - it models how a state changes through time). The "bidirectional" or noncausal version of S4D is a modification on top. I am not as familiar with the details of the other methods, but this category error makes me concerned.

For a benchmark paper, it would be nice to see more baselines for exact attention, such as FlashAttention.

One question: which tokenizer do you use for PG-19? Making that clear up front would be helpful (it is one of the reasons that existing work is a little hard to compare).

Novelty: I think this type of benchmark has been missing from the literature, so it is welcome. However, the analysis and evaluation on top are a little bit lacking.

Reproducibility: Seems clear.
summary_of_the_review: I think this paper could make a good contribution to the literature, but the paper has a few basic mistakes that should be corrected. If my concerns are addressed in the rebuttal, I'm happy to raise my score above the acceptance threshold.

Update after rebuttal: my concerns have been addressed, so I am increasing my score to a 6.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper collects a benchmark with seven real-world tasks across different domains to study attention patterns (claims to be dedicated to sequence modeling tasks).

Strength: 
- The ablation study of various modes of attention, such as noncausal, cross-attention, and unidirectional (or causal) is helpful.

Weaknesses
- The performance gap for various models in many of the CAB benchmarks is not statistically significant. This means that the tasks chosen may not be the most appropriate settings to do this ablation study.
- CAB is meant to be for modeling long sequences, though there is for example super-resolution benchmark in there which I think is not the best benchmark to be used in long sequence modeling. In particular, as properly mentioned by Saharia et al. 2022: "Super-resolution works by learning to transform a standard normal distribution into an empirical data distribution through a sequence of refinement steps, resembling Langevin dynamics." This can be done by shorter tern autoregressive models as well. 
- Extrapolation is abstracted away to one single scenario which basically confirms the no-free lunch principle. The study of extrapolation in sequence modeling requires benchmarks with multiple characteristics such as changes in sampling frequency input signals, irregularly sampled data, learning dynamics beyond the training distribution, distribution shifts, and so many more. A great example of testing extrapolation capability was shown for the state-space models, S4, by Gu et al. 2022, on the Speech Command recognition benchmark, where they trained on sequences of length 16k, and then tested it with sequences sampled at 8kHz frequency. Perhaps this benchmark or variations of it could be tested in CAB. 
- The proposed benchmark is heavily built around attention-based models for long sequence modeling. This is a huge inductive bias that discards the opportunity of testing natural sequence modeling baselines such as state-space models [Gu et al. 2021], Fourier neural operators [Li et al. 2020], Physics informed neural networks [Lu et al. 2019], advanced RNNs such as Long Expressive Memory [Rusch et al. 2022], Closed-form Continuous-time networks [Hasani et al. 2022], and Gated RNNs with Weighted Time-Delay Feedback [Erichson et al. 2022], let alone SoTA transformer architectures such as Mega [Ma et al. 2022].

I believe a major revision of this paper is required to elaborate on the concerns raised. Thus, I do not recommend acceptance.
Decision: Reject