Title: CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling
Benchmark: CAB (Comprehensive Attention Benchmark)
Summary: CAB is a multimodal benchmark assessing long-range modeling in transformers across computer vision, natural language processing, speech processing, and time-series forecasting. It is publicly available and composed of 7 tasks, spanning 9 datasets, to measure noncausal self, causal self, noncausal cross, and causal cross attention with a custom metric. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Long-range modeling
Phenomenon defined by authors: Yes
Phenomenon definition: Long range modeling is implicitly defined as "longer sequence modeling in different domains" (1). 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: The four types of attention to be measured are noncausal self attention, causal self attention, noncausal cross attention, causal cross attention. 
Task definition: There are "seven tasks covering four research fields ... computer vision, natural language processing, speech processing, and time series forecasting" (3). The tasks are Text-to-Speech Synthesis (TTS), Summarization (Sum), Long Sequence Time-series Forecasting (LSTF), Point Cloud Completion (PCC), Langauge Modeling (LM), Masked Language Modeling (MLM), Super-Resolution (SR). 
Task item definition: The benchmark is composed of 9 distinct datasets with different features. 
Task definition detail: Each task has its own dataset and metric. 
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size detail: Each task has a different target sequence length.
Dataset metadata available: No
Sampling method: Unknown
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM post-processing (extracting answers, reformatting for automated scoring), Distribution (perplexity, calibration, correlation), Mel-Cepstral Distortion is a measure of audio quality for TTS. A custom index is defined to balance all the evaluation metrics. 
Metric definition detail: TTS uses MCD, MSD. Sum uses ROUGE. LSTF uses MSE, MAE. PCC uses CD and F-Score. LM and MLM uses PPL. SR uses PSNR and SSMI. The paper defines a custom metric, compositional index (CI), which is "a normalized score to balance the influence among evaluation metrics, and high CI represents excellence. It is computed as follows: a) we transform all evaluation metrics beforehand, so that a higher score indicates better performance; b) we then normalize each transformed metric with Z-score normalization; c) after normalization, the score of each evaluation metric is averaged within each task, and is further averaged across tasks" (5). 
Metric aggregation: Custom compositional index (CI)
Metric statistics: Simple mean/sum, custom normalized aggregate metric
Metric subscores: Yes
Metric subscores detail: Scores are reported for each attention type, per task, per sub-metric in each task, and with the total CI. 
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors highlight that evaluating long-range modeling requires assessing "standard bidirectional (or noncausal) self attention" and "cross attentions and unidirectional (or causal) attentions, which are equally important to downstream applications" (1). CAB is proposed to measure causal and cross self attention, in addition to noncausal self attention. 
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context