Title: PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding
Benchmark: PEER
Summary: A benchmark called PEER (a
comprehensive and multi-task benchmark for Protein sEquence undERstanding).
PEER provides a set of diverse protein understanding tasks including protein
function prediction, protein localization prediction, protein structure prediction,
protein-protein interaction prediction, and protein-ligand interaction prediction. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: The capability being measured in the PEER benchmark is protein sequence understanding. The benchmark evaluates model performance across a range of biologically relevant tasks, which include: Protein function prediction, Protein localization prediction, Protein structure prediction, Protein-protein interaction prediction, Protein-ligand interaction prediction
Phenomenon defined by authors: Yes
Phenomenon definition: The PEER benchmark includes seventeen biologically relevant tasks that cover diverse aspects of protein understanding, including protein function prediction, protein structure prediction, protein localization prediction, protein-protein interaction prediction and protein-ligand interaction prediction.

We represent a protein x as a sequence of amino acids (a.k.a., residues) x = (x₁, x₂, · · · , x_L) of length L. For each task, we list the task name and its acronym, task category, data source, protein sequence statistics, dataset statistics and evaluation metric.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task is defined as evaluating language models on a set of 17 biologically relevant benchmarks that test their ability to understand protein sequences. This includes predicting various properties and interactions of proteins, such as their function, structure, localization, and interactions with other proteins or ligands​

Task item definition: A single item in the task dataset typically consists of a protein sequence (a string of amino acids) and a corresponding label or target value, which varies by task—e.g., a fitness score (regression), a structural class (classification), or a binary interaction label.
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: across 17 tasks: 115,271
Dataset size notes: 274,179 and 28,743
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), Spearman’s ρ, L/5 precision, RMSE
Metric aggregation: Simple Mean
Metric statistics: simple mean, std
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Biology / 
New taxonomy: Specific Application / Physical Sciences