Title: CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling	
Benchmark: Comprehensive Attention Benchmark (CAB)
Summary: Current benchmarks testing different attention architectures for long-term modelling only focuses on the standard bidirectional (or noncausal) self attention, and completely ignores cross attentions and unidirectional (or causal) attentions. 

In this paper, we propose Comprehensive Attention Benchmark (CAB) with four distinguishable attention patterns, namely, noncausal self, causal self, noncausal cross, and causal cross attentions. In seven tasks, CAB validates efficient attentions in eight backbone networks to show their generalization across neural architectures.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Long sequence modelling capability of transformers with different attention mechanisms
Phenomenon defined by authors: No
Phenomenon definition: It is defined as the performance of the transformer models in handling long-sequence tasks. There are seven tasks, such as text-to-speech synthesis, summarization, Long Sequence Time-series Forecasting, etc. The sequence length considered 'long' would be different for each task. 
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Task definition: There are seven long-sequence tasks ranging from Computer Vision, NLP and time-series forecasting handled by transformers. Examples include Super-Resolution, Masked Language Modelling, Long Sequence Time-series Forecasting. 
Task item definition: A long-sequence task, the dataset for the task, sequence length, evaluation metric for performance, transformer model type, attention mechanism
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size detail: So one dataset for each task, seven tasks in total
Dataset metadata available: Yes
Dataset metadata detail: The backbone transformer architecture and attention architecture
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Response format detail: Different outputs for each task. For example, for summarisation, it produces a summary paragraph; for Super-Resolution, it convert low-resolution (16 × 16) face images into high-resolution (128 × 128) images.
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Distribution (perplexity, calibration, correlation), MCD, MSD, PSNR, SSIM
Metric definition detail: Different metrics used to evaluate each task.
Metric subscores: Yes
Metric subscores detail: Results are provided separately for each task with different metrics. 
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context