Title: A SARS-CoV-2 Interaction Dataset and VHH Sequence Corpus for Antibody Language Models
Benchmark: AVIDa-SARS-CoV-2
Summary: AVIDa-SARS-CoV-2 is a dataset featuring the antigen-variable domain of heavy chain of heavy chain antibody (VHH) interactions obtained from two alpacas immunized with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) spike proteins. AVIDa-SARS-CoV-2 includes binary labels indicating the binding or non-binding of diverse VHH sequences to 12 SARS-CoV-2 mutants, such as the Delta and Omicron variants.  The authors report benchmark results for predicting SARS-CoV-2-VHH binding using VHHBERT pre-trained on VHHCorpus-2M and existing general protein and antibody-specific pre-trained language models.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: binding prediction, i.e. predict the binding site of an antibody.
Phenomenon defined by authors: Yes
Phenomenon definition: The antibody discovery task is a binary sequence classification that distinguishes antibodies that bind to SARS-CoV-2.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Binary classification: whether the antibody binds to a specific antigen at the antibody sequence level.
Task item definition: - VHH_sequence: Amino acid sequence of VHH
- Ag_label: Antigen Type
- label: Binary label represented by 1 for the binding pair and 0 for the non-binding pair
- subject_species: Species of the subject from which VHH was collected
- subject_name: Name of the subject from which VHH was collected
- subject_sex: Sex of the subject from which VHH was collected
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 77,003
Dataset size notes: 2M
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Biology / 
New taxonomy: Specific Application / Physical Sciences