Title: ON PRE-TRAINED LANGUAGE MODELS FOR ANTIBODY
Benchmark: ATUE
Summary: This paper introduces the AnTibody Understanding Evaluation (ATUE) benchmark to systematically assess the representation capabilities of general and antibody-specific pre-trained language models across a range of antibody-related tasks. It also explores how incorporating biological mechanisms into pre-training can enhance model performance and evaluates the transferability of learned representations to real-world applications such as drug discovery and immune system analysis.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: LLMs capability to do  antibody representation learning and biological reasoning with sequence specificity
Phenomenon defined by authors: Yes
Phenomenon definition: how LLMs perform in antibody tasks with different specificity and how introducing specific biological mechanisms to the pre-training process can benefit the model. Additionally, authors evaluate if the learned antibody pre-trained representations can be applied to real-world antibody problems, like drug discovery and immune process understanding.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Evaluate the ability of pre-trained language models to perform on four supervised antibody-related prediction tasks—antigen binding, paratope prediction, B cell maturation classification, and SARS-CoV-2 antibody discovery—each varying in antibody specificity. These tasks assess whether the models can capture biologically meaningful information from antibody sequences.
Task item definition: N/A there are four tasks
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 3242, 1662, 88094, 22000
Dataset size notes: 15,128/3,242 , N/A
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), Matthews Correlation Coefficient (MCC), and AUC (Area Under the ROC Curve)
Metric aggregation: Simple Mean
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Biology / 
New taxonomy: Specific Application / Physical Sciences