SlowGuess/NeurIPS_2023_OCR huggingface dataset:
AVIDA-hIL6: A Large-Scale VHH Dataset Produced from an Immunized Alpaca for Predicting Antigen-Antibody Interactions
Hirofumi Tsuruta $^{1*}$ , Hiroyuki Yamazaki $^{1*}$ , Ryota Maeda $^{1*}$ , Ryotaro Tamura $^{1}$ , Jennifer N. Wei $^{2}$ , Zelda Mariet $^{2}$ , Poomarin Phlophyisut $^{2}$ , Hidetoshi Shimokawa $^{2}$ , Joseph R. Ledsam $^{2}$ , Lucy Colwell $^{2}$ , Akihiro Imura $^{1*}$
$^{1}$ COGNANO Inc.,  $^{2}$ Google LLC
{tsuruta, yamazaki, maeda, ryotarotamura, akihiroimura}@cognano.co.jp {weijennifer, zmariet, poomarin, simokawa, jledsam, lcolwell}@google.com
Antibodies have become an important class of therapeutic agents to treat human diseases. To accelerate therapeutic antibody discovery, computational methods, especially machine learning, have attracted considerable interest for predicting specific interactions between antibody candidates and target antigens such as viruses and bacteria. However, the publicly available datasets in existing works have notable limitations, such as small sizes and the lack of non-binding samples and exact amino acid sequences. To overcome these limitations, we have developed AVIDa-hIL6, a large-scale dataset for predicting antigen-antibody interactions in the variable domain of heavy chain of heavy chain antibodies (VHHs), produced from an alpaca immunized with the human interleukin-6 (IL-6) protein, as antigens. By leveraging the simple structure of VHHs, which facilitates identification of full-length amino acid sequences by DNA sequencing technology, AVIDa-hIL6 contains 573,891 antigen-VHH pairs with amino acid sequences. All the antigen-VHH pairs have reliable labels for binding or non-binding, as generated by a novel labeling method. Furthermore, via introduction of artificial mutations, AVIDa-hIL6 contains 30 different mutants in addition to wild-type IL-6 protein. This characteristic provides opportunities to develop machine learning models for predicting changes in antibody binding by antigen mutations. We report experimental benchmark results on AVIDa-hIL6 by using machine learning models. The results indicate that the existing models have potential, but further research is needed to generalize them to predict effective antibodies against unknown mutants. The dataset is available at https://avida-hil6.cognanous.com.
Antibodies are proteins that play an essential role in the immune system. When antigens such as viruses and bacteria invade the body, the immune system protects the body by producing large numbers of antibodies that bind to the antigens to inhibit their function or mark them for removal. Antibodies have become an important class of therapeutic agents to treat human diseases because of their high target specificity and binding affinity [2]. An essential step in therapeutic antibody discovery is the identification of specific interactions between antibody candidates and target antigens, which has traditionally relied heavily on expensive, time-consuming experiments [24]. Therefore,
computational approaches are increasingly used to complement and accelerate traditional processes for therapeutic antibody discovery [46, 25].
In particular, there is growing interest in using machine learning to predict antigen-antibody interactions [39, 28, 20], which can be used to virtually screen binding antibodies against specific target antigens. Schneider et al. [39] developed the structure-based deep learning for antibodies virtual screening (DLAB-VS) by using the structural antibody database (SAbDab) [17], which contains collections of antigen-antibody complex structures. Lim et al. [28] generated datasets of antibody sequences from mice immunized with cytotoxic T lymphocyte-associated antigen 4 (CTLA-4) and programmed cell death protein 1 (PD-1); then, they built deep learning models to predict binder and non-binder antibodies to CTLA-4 and PD-1. Huang et al. [20] proposed AbAgIntPre, a deep learning-assisted prediction method that was trained using only the amino acid sequences in two public databases: SAbDab and the coronavirus antibody database (CoV-AbDab) [35].
Despite these promising developments, progress in therapeutic antibody discovery has lagged behind progress in other areas of drug discovery. A major reason for this is the lack of availability of high-quality, large-scale datasets of antigen-antibody interactions. First, most existing datasets have small sample sizes. For example, as of May 2023, SAbDab and Lim et al.'s datasets [28] contain 5,737 and 3,064 binder samples, respectively. In addition, SAbDab only has samples for binding antigen-antibody pairs. In previous studies [39, 20] using SAbDab, antigens and antibodies were randomly paired to form non-binding pairs. CoV-AbDab contains 12,021 entries, from which more than 30,000 antigen-antibody pairs, including non-binding pairs, are available. However, CoV-AbDab provides only the variant name and not the amino acid sequence. As the variant name is defined by representative mutations, the exact amino acid sequence may vary between publications, thus making it difficult to use CoV-AbDab for antibody discovery because a single amino acid change can be critical for an antigen-antibody interaction.
To overcome these limitations, we have developed AVIDa-hIL6, an antigen-variable domain of heavy chain of heavy chain antibody (VHH) interaction dataset produced by an alpaca immunized with the human interleukin-6 (IL-6) protein. IL-6 is a relatively small protein, a simply structured, well-characterized cytokine that exists as a monomer in the body and is associated with many inflammatory diseases and cancers. To ensure a wide variety of antibody sequences, we used VHHs, whose simple structures enable much easier identification of full-length amino acid sequences by DNA sequencing technologies such as next-generation sequencing (NGS) than for conventional antibodies. By leveraging these advantages, AVIDa-hIL6 contains 573,891 antigen-VHH pairs, including 20,980 binding pairs, with their amino acid sequences. In addition, we have developed a novel labeling method to obtain reliable labels for binding and non-binding.
Furthermore, AVIDa-hIL6 contains information on the interaction of diverse VHHs with 30 different mutants produced by artificial point mutations, in addition to the wild-type IL-6 protein. As the COVID-19 pandemic has shown, viruses continuously evolve through mutation to evade the immune system. Because emerging mutations involving amino acid substitutions can lead to profound changes in antibody binding, prediction of their effects is critical in the development of therapeutic antibodies. Notably, AVIDa-hIL6 contains antibody sequences that are positive for most IL-6 mutants but negative for specific IL-6 mutants, or vice versa, thus providing important insights for understanding how antigen mutations affect antibody binding.
The main contributions of this paper are summarized as follows.
- We release AVIDa-hIL6, which is the largest existing dataset for predicting antigen-antibody interactions (10 times larger than any other public dataset) and contains amino acid sequences of antigens and antibodies and binary labels for binding and non-binding pairs.
- AVIDa-hIL6 has the wild type and 30 mutants of the IL-6 protein as antigens, and it includes many sensitive cases in which point mutations in IL-6 enhance or inhibit antibody binding.
- We have designed a novel data generation method, including data labeling, by using the immune system of a live alpaca. This method can be applied to any target antigen, in addition to IL-6.
- We report benchmark results for the prediction of antigen-antibody interactions by using machine-learning-based baseline models. These results confirm that AVIDa-hIL6 provides valuable benchmarks for assessing a model's performance in capturing the impact of antigen mutations on antibody binding.
Table 1: Characteristics of public datasets for predicting antigen-antibody interactions. The numbers of samples were obtained in May 2023.  $\checkmark^{*}$  denotes that relevant information was missing or not available for part of the dataset.
Dataset	#Samples	Sequence	Structure	Antibody type	Data collection
Binder	Non-binder	Antibody	Antigen
SAbDab-nano [40]	1,114	-	✓	✓	✓	VHH	Curation
sdAb-DB [47]	1,452	-	✓	✓*	✓*	VHH	Curation
SAbDab [17]	5,737	-	✓	✓	✓	Conventional,VHH	Curation
Lim et al. [28]	3,064	12,055	✓	✓	Conventional	Experiment
CoV-AbDab [35]	32,064	5,303	✓	✓*	✓*	Conventional,VHH	Curation
AVIDA-hIL6	20,980	552,911	✓	✓	VHH	Experiment
In this section, we put our work into context with public datasets for predicting antigen-antibody interactions. Recent advances in NGS technology now enable the construction of large-scale databases of antibody sequences, such as the observed antibody space (OAS) [31] and iReceptor [14]. However, those databases cannot be directly used as training data for predicting antigen-antibody interactions because of the lack of information on the antigen corresponding to each antibody. Thus, those databases are primarily used to build antibody-specific language models [37, 32, 26] and to generate new antibody sequences via deep generative models [3]. Here, we focus on datasets with information on antigen-antibody interactions, as summarized in Table 1.
Antibody Type. The datasets listed in Table 1 contain two types of antibodies: conventional and VHH. A conventional antibody comprises two pairs of heavy and light chains. A conventional antibody acts as a single functional unit by combining the heavy and light chains encoded on separate chromosomes. For example, in humans, the heavy chain is encoded on chromosome 14, and the light chain is encoded on chromosome 2 or 22. Therefore, purification of a single-cell lymphocyte is essential for DNA sequencing of the antigen-determining regions on an antibody. Such cell cloning comprises several steps and is time-consuming, making analysis on the order of 10 to the third power or more virtually impossible. In contrast, a VHH, found in camelids such as alpacas and llamas, comprises only heavy chains. The heavy-chain antibodies are derived from a single gene, e.g., they are encoded on chromosome 4 in alpacas; moreover, a VHH is the smallest functional unit of heavy-chain antibodies, and the sequencing for the antigen-determining regions does not require cell cloning. Thus, we can perform exhaustive analysis on the order of six powers of 10 or more by simply extracting DNA from a bulk sample of lymphocytes. Additionally, VHHs have recently gained interest as therapeutic agents because of their small size, high stability, good human tolerability, and relative ease of production [22, 21]. SAbDab-nano [40] and sdAb-DB [47] are public databases that collect only VHHs, but they both have too few samples for machine learning drug discovery or design applications. Hence, we use immunized alpacas as a data source to generate a large amount of VHH sequence data.
Sequence and Structure Information. SAbDab [17] and its sub-database, SAbDab-nano [40], collect all the available antigen-antibody complex structures in the Protein Data Bank (PDB) [8]. Also, some data in sdAb-DB [47] and CoV-AbDab [35] include structural information from the PDB. Because accurate knowledge of antibody structures is important for understanding the antigen-binding function of antibodies, SAbDab is increasingly used for antibody structure prediction via deep learning [38, 1]. However, experimental methods for antibody structure determination, such as X-ray crystallography and cryo-electron microscopy, are relatively expensive and time-consuming, making it difficult to increase the amount of data. More recently, machine learning methods such as AlphaFold [23] and RoseTTAFold [6], which accurately predict a protein's structure from the amino acid sequence, have greatly accelerated progress in the biological sciences. AVIDa-hIL6 focuses on amino acid sequences of antigens and antibodies to generate sufficient training data for machine learning.
Number of Labeled Samples. Some existing datasets only have samples for binding antigen-antibody pairs. One reason is that the identification of non-binding antigen-antibody pairs has little clinical significance. In previous studies [39, 20] using SAbDab, antigens and antibodies were randomly paired to form non-binding pairs. This process was based on the assumption that
Figure 1: Overview of the data generation process for AVIDa-hIL6.
randomly sampled pairs are unlikely to bind because of antibodies' high target specificity. Lim et al. [28] generated a dataset of antibody sequences labeled as "binder" and "non-binder" through an experimental approach using mice immunized with CTLA-4 and PD-1. The number of samples in Table 1 is the total for CTLA-4 and PD-1. As with AVIDa-hIL6, Lim et al. created their dataset from their original experiments, thus revealing the potential to increase the amount and diversity of data by the same approach using arbitrary antigens. CoV-AbDab collects antibodies that bind to at least one beta coronavirus and currently contains 12,021 entries. Because each entry has "Binds to" and "Doesn't Bind to" columns and contains zero to multiple antigens for a specific antibody, the number of samples in Table 1 was counted for each possible antigen/antibody pair. However, CoV-AbDab only provides variant names, e.g., SARS-CoV1_Omicron-BA2 or SARS-CoV2_Alpha, and each antigen's exact amino acid sequence is only available if it was provided in the original publication.
3 AVIDa-hIL6: Antigen-VHH Interaction Dataset Produced from Alpaca Immunized with Human IL-6 Protein
AVIDA-hIL6 is a dataset of antigen-VHH interactions with amino acid sequences and binary labels for binding and non-binding. In this section, we introduce the dataset generation process, dataset statistics, and verification of label reliability. A labeled dataset and the raw data are available at https://avida-hil6.cognanous.com. The dataset is released under a CC BY-NC 4.0 license.
3.1 Dataset Generation
Figure 1 shows an overview of the data generation process. The KYODOKEN Institutional Animal Care and Use Committee approved the protocols for the experiments (see Appendix A.1). Appendix A.2 gives the detailed experimental procedures and the amino acid sequences of the IL-6 proteins.
Step 1. Immunization To ensure the diversity of antibodies binding to the IL-6 protein that we used as an antigen, we used the immune system of a live alpaca. We introduced a site-directed mutation with alanine at intervals of three to six amino acids, like the alanine scanning technique [15], which is used in molecular biology to determine the contribution of a specific amino acid; as a result, we obtained 30 types of mutants in addition to the wild-type IL-6 protein. A mutant is denoted, for example, as IL6_P42A, which means that an amino acid in the wild type is substituted from proline to alanine at position 42. We immunized a single alpaca with a cocktail of 31 different IL-6 proteins four times at about two-week intervals. After each immunization, one blood sample and one or more lymph nodes from different body sites were collected, yielding a total of 12 libraries. We provide additional information about the collection process at https://avida-hil6.cognanous.com.
Step 2. Phage Library Construction We used phage display [43] to identify VHHs that bind to the IL-6 protein. Phage display is a technique for displaying the target proteins on a phage surface in a form that allows them to bind to other molecules. This versatile technique enables generation of protein libraries containing up to  $10^{10}$  different variants and is often used for affinity screening of antibodies for their binding partners [7]. We cloned the VHH genes obtained from each library into the pMES4 phagemid vector to display the VHHs on the phagemid surface. As a result, 12 phage libraries corresponding to each of the above libraries were generated and designated as the mother libraries.
Step 3. Affinity Selection Affinity selection by biopanning using the mother libraries can enrich a phage with displayed VHHs that bind to the target molecule. For the target molecules, we used the wild type and 30 mutants of IL-6 and a negative control sample that did not contain any IL-6 protein. Only experiments targeting the wild-type IL-6 protein were performed in triplicate to ensure reproducibility. The mother library was added to the container and incubated with target-coated magnet beads. Then, non-binding phages were washed away, and the remaining phages that bound to the beads were eluted. Consequently, by performing one round of biopanning on each of the 12 mother libraries, we generated a total of 408 sublibraries.
Step 4. Sequence Analysis The amino acid sequences of VHHs displayed on a phage surface can be identified by analyzing the phage genome's DNA with NGS technology. Approximately 100,000 paired reads were generated for each library by NGS, and singletons were removed to avoid sequencing errors. The DNA sequences were translated into amino acid sequences. We counted the number of occurrences of each unique VHH amino acid sequence from the paired reads, which reflected the concentration of each VHH in the library. For each of the 12 mother libraries before panning and 408 sublibraries after panning, we created a table with the VHH amino acid sequences and their read counts.
Step 5. Data Labeling We designed a labeling method to distinguish whether a VHH binds to each IL-6 protein type by applying a statistical test for differences in the proportions of each VHH in a library before and after panning. Here, we focused on examining the binding between a specific VHH and a specific target molecule. Let  $p_1$  and  $p_2$  denote the population proportions of a specific VHH in the libraries before and after panning. We identified some of the VHH sequences in the libraries by NGS analysis. Let  $n_1$  and  $n_2$  denote the libraries' total read counts before and after panning, respectively, and let  $x_1$  and  $x_2$  denote the read counts of a specific VHH in the libraries. Then, the respective sample proportions of a specific VHH in each library are  $\hat{p}_1 = \frac{x_1}{n_1}$  and  $\hat{p}_2 = \frac{x_2}{n_2}$ . Given that the minimum value of all possible  $n_1$  and  $n_2$  was over 10,000, we assumed that  $\hat{p}_1$  and  $\hat{p}_2$  follow normal distributions with mean  $p_1$  and  $p_2$  and variance  $\frac{p_1(1 - p_1)}{n_1}$  and  $\frac{p_2(1 - p_2)}{n_2}$ , respectively, according to the central limit theorem. Furthermore, the difference in the proportions  $\hat{p}_1 - \hat{p}_2$  can also be approximated by a normal distribution due to the reproductive property of the normal distribution. Thus, the test statistic  $Z$  under null hypothesis  $H_0: p_1 = p_2$  was calculated as follows.
$$
Z = \frac {\hat {p} _ {1} - \hat {p} _ {2}}{\sqrt {p (1 - p) \left(\frac {1}{n _ {1}} + \frac {1}{n _ {2}}\right)}} \tag {1}
$$
where  $p$  is the pooled proportion calculated as  $p = \frac{x_1 + x_2}{n_1 + n_2}$ . The p-value of  $Z$  was calculated using the standard normal distribution. In the same way, p-values were calculated for all VHH-target pairs in the sublibraries with respect to the 12 corresponding mother libraries. Because we had 12 sublibraries associated with the same target molecule, we adopted the smallest p-value, indicating the most significant difference in proportion, among identical VHH-target pairs. If a specific VHH's proportion in a sublibrary increased from the proportion in the corresponding mother library and the p-value was 0.05 or less (our chosen significance level), the VHH-target pair was labeled with "binder." Similarly, if the proportion decreased and the p-value was 0.05 or less, the pair was labeled with "non-binder." Finally, if the p-value exceeded 0.05, the pair was labeled with "non-significant." Our dataset contains 1,998,127 samples labeled non-significant, including 325,865 unique VHH sequences that are present in the alpaca body. These labels were not used for supervised learning to predict antigen-antibody interactions. However, these samples may be helpful for pre-training via self-supervised learning as used by existing antibody-specific language models [37, 32, 26].
Figure 2: (a) Number of samples for each antigen type. (b) Visualization of the labels for pairs of 100 VHH sequences and each antigen type. Each cell represents a unique VHH-antigen pair. White cells denote non-significant or noise labels. (c) Distribution of the pairwise identities of the VHH sequences.
The results of biological experiments always contain background noise, such as binding to contaminating proteins. Therefore, we developed a novel noise reduction algorithm to avoid false positives and improve label reliability. We reconfirmed VHHs labeled as "binder" to any of the IL-6 proteins by comparing the labels to negative control samples under the following conditions.
1. If the VHH was a non-binder to the negative control sample, the label remained "binder."
2. If the VHH was a binder to the negative control sample, the label was reassigned from "binder" to "noise" because of possible false positives.
3. If the VHH was "non-significant" with respect to the negative control sample, the ratio of the p-value of the negative control sample to that of the IL-6 protein was compared to  $10^{2.5}$ . This value was empirically determined by an author (a biologist) according to feedback from biological experiments in our previous studies [29].
(a) If the ratio of p-values was below  $10^{2.5}$ , the label was reassigned from "binder" to "non-significant" because of possible false positives.
(b) If the ratio of p-values was  $10^{2.5}$  or more, the label remained "binder."
We carefully verified the reliability of our labels, as discussed in section 3.3. The code for data labeling is available at https://github.com/cognano/AVIDA-hIL6.
3.2 Dataset Statistics
AVIDa-hIL6 contains 573,891 data samples, comprising 20,980 binding pairs and 552,911 non-binding pairs. The proportion of binding pairs is about  $3.7\%$ . Figure 2(a) shows the number of samples for each antigen type. Although the number of samples varied for each IL-6 protein type, we successfully generated at least 10,000 samples for the wild type and 30 different mutants. Furthermore, at least 250 binder VHH sequences existed for each IL-6 protein type. Because we labeled the VHH sequences in the mother library for each IL-6 protein type, AVIDa-hIL6 has information on whether the same VHH sequence binds to each of multiple targets. The number of unique VHH sequences in AVIDa-hIL6 is 38,599, including 4,425 sequences that bind to at least one IL-6 protein type. Importantly, 650 VHH sequences, about  $14.7\%$  of the VHH binders, show binding to specific IL-6 protein types but non-binding to others. We visualized whether 100 sequences extracted randomly from these 650 VHH sequences bound to each antigen type, as shown in Figure 2(b). These samples have valuable information on which mutations enhance or inhibit antibody binding, which should be strongly associated with the IL-6 protein's binding site. Furthermore, when focusing on the same VHH sequence, i.e., one row, we observe that mutants with mutations at closer positions tend to have the same label.
To gain a better understanding of the distribution of VHH sequences, we compared it to the distributions in the existing VHH datasets SAbDab-nano and sdAb-DB. We used only the binders from AVIDa-hIL6 because the existing datasets only contain binders. The numbers of unique VHH binders in SAbDab-nano, sdAb-DB, and AVIDa-hIL6 are 828, 1,414, and 4,425, respectively. To mitigate the computational complexity, we randomly sampled 700 unique VHH sequences from each dataset and calculated all pairwise sequence identities with Biopython v1.81 [13]. Figure 2(c) shows the
(a) Binder (ONU14)
(b) Non-binder (ONU82)
Figure 3: Immunofluorescence staining analysis using three VHHs labeled with (a) binder, (b) non-binder, and (c) noise. HA-tagged IL-6 proteins were introduced into cells and stained with His-tagged VHHs. Subsequently, HA-tags were visualized in green and His-tags in red. The ONU is our identifier to distinguish the VHH sequence.
(c) Noise (ONU250)