SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Numbers of samples in the training and test sets.
Dataset	#Samples	Antigen types
Binder	Non-binder	Total
Training	10,564	282,279	292,843	Wild-type, Q45A, D54A, G63A, C72A, L90A, P93A, D99A, F102A, G105A, E108A, T117A, S135A, E138A, F153A, D168A
Test	10,416	270,632	281,048	P42A, T48A, E51A, I57A, I60A, K69A, C78A, S81A, E87A, L120A, L126A, L129A, Q144A, D162A, T165A
distributions of sequence identities for these datasets. The results indicate that AVIDa-hIL6 has peaks at regions of higher sequence identity than the other datasets. Interestingly, AVIDa-hIL6 has a peak at  $97\%$  sequence identity, which is absent for the others. As a living organism's immune response progresses, effective antibodies with high binding affinity to the antigen are selected through a process called affinity maturation and are further mutated by repeated exposure to the antigen. Thus, these results may reflect that a live alpaca's immune system selects VHH sequences with high sequence identity that specifically bind to target IL-6 proteins through affinity maturation.
3.3 Label Reliability
To verify our label reliability, we tested the antibody binding ability by immunofluorescence staining. Because the number of VHHs that could be verified was limited by the time and cost of biological experiments, VHHs were selected under the following conditions to verify label reliability efficiently. First, we examined only the wild-type IL-6 protein as a target antigen. Next, the amino acid sequences of all labeled VHHs with higher than  $93\%$  identity were clustered by the UCLUST algorithm [18] to validate diverse sequences. When two or more VHHs with the same label were in the same cluster, the one with the highest read count was selected. We know empirically that if all the VHHs in a cluster have the same label, these labels are likely to be true. Therefore, such VHHs were excluded from the candidates. Then, VHHs with suspect labels were selected in order of their read counts. Finally, we tested 10 binder-labeled, six non-binder-labeled, and four noise-labeled VHHs for validation.
Immunofluorescence analysis showed that all 10 binder-labeled VHHs actually bound to the wild-type IL-6 protein, whereas the six non-binder-labeled and four noise-labeled VHHs did not. Figure 3 shows the results for a representative clone of the three types of labeled VHHs. Appendix A.3.2 gives the results for all the tested clones and their amino acid sequences. We could observe the overlapping of IL-6 signals and VHH signals in the binder group, whereas the VHH signals were lost in the non-binder group. The VHH signals did not coincide with the IL-6 signals in the noise group, which can be interpreted as noise-labeled VHHs binding nonspecifically to cells. These results indicate that our noise reduction algorithm contributed to reducing false positives. In addition, these results were also confirmed by kinetic assay via biolayer interferometry (BLI), as described in Appendix A.3.2. As a result, we could ensure that AVIDa-hIL6 has highly reliable labels.
4 Benchmarks
4.1 Benchmark Task
To demonstrate the use of AVIDa-hIL6 for antibody discovery, we performed an experiment on binary classification of whether a given antigen-antibody pair binds. By leveraging information on the binding of diverse antibodies to antigen mutants, we defined a benchmark task to assess the model performance in capturing the impact of antigen mutations on antibody binding. First, we randomly selected 15 mutants and reserved the data samples for those mutants as a test set. The remaining 15
mutants and the wild type were reserved for model training. Table 2 lists the numbers of samples in the training and test sets. As we used artificial point mutations, each mutant's sequence identity with respect to the wild type was the same, differing only in the position where the alanine was introduced. Next, we trained models by using only the wild-type IL-6 protein and evaluated their performance in predicting antibody bindings in the test set. Then, we randomly selected one mutant from the remaining 15 mutants outside the test set and added it to the training set to evaluate each model's predictive performance. By repeating this process, we tracked the model's predictive performance for unknown mutants contained only in the test set. Because the order of adding mutants to the training set affected the model performance, we ran the same experiment five times in shuffled order, and we report the averaged results here. For all model training, we randomly selected  $10\%$  of the training set for model validation. This experimental scenario assumes that antigen mutants emerge one after another to evade the immune system, as in the COVID-19 pandemic. In such a scenario, we evaluated the model's performance in predicting antibody candidates that will bind to future emerging mutants according to the binding information of antigens that have already been observed.
4.2 Baseline Models
We adopted three neural network-based models and one classical machine learning model as baselines. The model inputs were the amino acid sequence of an IL-6 protein with a length of 218 and a VHH with a maximum length of 152.
- AbAgIntPre [20] is a state-of-the-art model designed for antigen-antibody interactions based on amino acid sequences. It combines the composition of k-spaced amino acid pairs (CKSAAP) [11] encoding and a convolutional neural network (CNN) model with a Siamese-like architecture. We used the model parameters reported in the original paper [20].
PIPR [10] is a residual recurrent convolutional neural network (RCNN) for protein-protein interaction (PPI) prediction. Following the PIPR strategy, we used an amino acid encoding that combined a five-dimensional vector obtained from a pretrained skip-gram model using the STRING database [44] and a seven-dimensional vector describing the categorization of electrostaticity and hydrophobicity. We changed the number of RCNN units from five to three because our sequence length was more than nine times shorter than the protein input in the original PIPR. The other model parameters were the same as in the paper [10]. Although PIPR was not specifically designed for antigen-antibody interactions, such interactions that ignore non-protein antigens can be considered a subset of PPI, meaning that models designed for PPI can also apply to antigen-antibody interactions.
- A Multi-Layer Perceptron (MLP) with one hidden layer of 512 neurons was used as a simpler neural network-based model than the above two models. We used one-hot encoding to represent amino acid sequences. One-hot vectors of the VHHs and IL-6 proteins were flattened and concatenated for input to the MLP.
- Logistic Regression (LR) was used as a classical machine learning model that is commonly used for binary classification tasks. As with the MLP, we used flattened and concatenated one-hot vectors of the VHH and IL-6 proteins as input.
In this experiment, the three neural-network-based models were trained for 100 epochs on one NVIDIA Tesla V100 GPU on Google Colaboratory with an initial learning rate of 0.0001 and a batch size of 256. The LR was trained using one Intel(R) Xeon(R) CPU on Google Colaboratory. Appendixes A.4.2 and A.4.3 give more details on the model implementation and training. The code to run the benchmark models is available at https://github.com/cognano/AVIDA-hIL6.
Figure 4(a) shows the prediction performance of the baseline models as a function of the number of IL-6 protein types used for model training. We used the precision, recall, and F1-score as evaluation metrics because the prediction of antibody binders, which are fewer in number than non-binders, is much more important for drug discovery. Figure 4(b) shows the precision-recall curves when 1 and 16 IL-6 protein types were used for training. When the number of antigens was 1—that is, when only the wild-type IL-6 protein was used for training—the recalls of AbAgIntPre, PIPR, MLP, and LR were 67.9, 57.6, 67.2, and  $67.1\%$  respectively. These results indicate that the models failed
Figure 4: (a) Precision, recall, and F1-score as a function of the number of IL-6 protein types used for model training. (b) Precision-recall curves when 1 and 16 antigen types were used for training. The legend shows the area under the curve (AUC) values.
to predict over  $30\%$  of the effective VHHs that bound to mutants in the test set. All the metrics improved as the number of IL-6 protein types used for training increased, and this trend is clearly evident in the precision-recall curves and the area under the curve (AUC) values. After adding 15 mutants for training, the precisions of all baseline models reached about  $95\%$ , but the recalls were still only about  $85\%$ , even for the three neural-network-based models. A possible factor in this result is that the proportion of binding pairs was only about  $3.7\%$ . Hence, techniques to address imbalanced labels, such as oversampling and undersampling, would be useful to improve the model performance.
For drug discovery applications, the construction of a generalized model for unknown mutations from as little antigen-binding information as possible is ideal, because the number of possible mutations in antigens is tremendously large. As shown by the F1 scores and AUCs in Figures 4(a) and (b), respectively, AbAgIntPre outperformed the other three models, but there was still room for improvement. Furthermore, the performance of AbAgIntPre was not significant as compared to that of the simpler MLP. AVIDa-hIL6 differs significantly from the existing datasets used for training by AbAgIntPre and PIPR because it includes cases in which changes of a few amino acids enhance or inhibit antibody binding. Given this difference in properties, AbAgIntPre and PIPR may not have a clear performance advantage over the MLP. Hence, these results indicate the need for research on model architectures that are dedicated to predicting antibody binding to antigen mutants, and AVIDa-hIL6 will be a useful benchmark for evaluating such models.
5.1 Binding Site Prediction
Antibodies recognize specific regions of antigens, called epitopes, and the regions of antibodies that are directly involved in recognition are called paratopes. The antigen-antibody interaction is defined between the epitope and the paratope and is governed by van der Waals forces, electrostatic forces, hydrogen bonding, hydrophobic interactions, and entropic changes at the binding site, which comprise amino acids and their chemical modifications such as phosphorylation, glycosylation, lipidation, methylation, acetylation, or ubiquitylation. Because epitopes and paratopes are crucial for the affinity and specificity of antigen-antibody interactions, many studies have been devoted to predicting epitopes [42, 45] and paratopes [27, 12]. AVIDa-hIL6 has highly sensitive information on changes of a few amino acids in both the antigen and antibody that can significantly affect binding, which should be strongly associated with epitopes and paratopes. Thus, AVIDa-hIL6 may facilitate research on predicting epitopes and paratopes from amino acid sequences.
5.2 Potential Risk
In recent years, VHH technology has rapidly developed not only as a research and diagnostic tool but also as a therapeutic agent. VHHs are known to have low toxicity to humans, and several VHH drugs have been approved to date [4]. Antibody genes are activated in B lymphocytes, and their complementarity-determining regions coding paratopes are the only genes in which unrestricted mutations occur in vivo. Hence, antibodies that happen to bind to their own tissues can be produced. Mammals have an immune tolerance system that minimizes the production of such autoreactive antibodies through several mechanisms [19]. As our dataset was derived from alpacas, even if the risk of autoimmune adverse events is low for alpacas, it may not be for humans. Therefore, a phase I clinical trial cannot be omitted for each clone for the time being.
5.3 Potential Data Biases
Close examination of our data revealed that the probability of the presence of binder VHHs varied dynamically depending on when and where the mother library was collected. Furthermore, because our dataset was produced in an alpaca's body, antibodies that strongly interact with proteins in an alpaca's body were likely to be excluded because they cause autoimmune disease. Therefore, our dataset potentially contains data biases derived from the timing and body site of the library collection and the specific alpaca used in the experiments. To reduce these data biases, it would be beneficial to collect samples at multiple times of immunization, from multiple individuals with different VHH gene sequences, and from multiple animals of different species.
5.4 Limitations and Future Works
We introduce two potential limitations of AVIDa-hIL6 and describe future works to address them. The first limitation is that AVIDa-hIL6 uses artificial mutations. Such mutations offer the advantage of investigating binding to an arbitrary number of mutants; however, natural mutations are more complex, as different sites mutate simultaneously. Furthermore, natural mutants include those that have similar functions and structures even with different amino acid sequences, and point mutations that cause loss or gain of the antigenic protein's function. Because simple artificial mutants have little chance of reproducing these rare properties, the efficiency of data collection for predicting antigen-antibody interactions of mutants with these properties is low. The second limitation is the lack of antigen diversity: specifically, AVIDa-hIL6 only has the IL-6 protein as an antigen. Our experimental scenario is to predict antibody binding to unknown mutants of a known antigen. In drug discovery applications, there is also a need to find effective antibodies against new emerging antigens. These limitations lead to the narrow applicability of a model trained on AVIDa-hIL6.
An essential approach to overcome these limitations will be to accumulate labeled data for a wider variety of antigens and their mutants. Because our data generation method described in section 3.1 is applicable to any target antigen, it can be a fundamental technology for establishing a more comprehensive database of antigen-antibody interactions. In fact, we used the same approach to generate a dataset for SARS-CoV-2 variants and successfully found effective antibodies [29]. In the future, we plan to generate and release datasets for various antigens, which should be more practical for building models to predict antigen-antibody interactions. Furthermore, we will explore combining AVIDa-hIL6 with other data sources, such as those listed in Table 1.
In this paper, we have described AVIDa-hIL6, a large-scale dataset of IL-6 protein-VHH pairs containing amino acid sequence information and reliable labels for binding or non-binding pairs. By introducing artificial mutations into the IL-6 protein used as an antigen, we generated an interaction dataset for 30 types of mutants in addition to wild-type IL-6. This design enabled AVIDa-hIL6 to include many sensitive cases in which point mutations in the IL-6 protein enhance or inhibit antibody binding, thus providing researchers with valuable insights into the effects of antigen mutations on antibody binding. We envision that AVIDa-hIL6 will help democratize antibody discovery and serve as a valuable benchmark for machine learning research in the growing field of predicting antigen-antibody interactions.
We thank Tomohisa Oda for developing the AVIDa-hIL6 website.
[40] Schneider, C., Raybould, M.I., Deane, C.M.: SAbDab in the age of biotherapeutics: updates including SAbDab-nano, the nanobody structure tracker. Nucleic acids research 50(D1), D1368–D1372 (2022)
[41] Shen, W., Le, S., Li, Y., Hu, F.: SeqKit: a cross-platform and ultrafast toolkit for FASTA/Q file manipulation. PloS one 11(10), e0163962 (2016)
[42] da Silva, B.M., Myung, Y., Ascher, D.B., Pires, D.E.: epitope3D: a machine learning method for conformational B-cell epitope prediction. Briefings in Bioinformatics 23(1), bbab423 (2022)
[43] Smith, G.P.: Filamentous fusion phage: novel expression vectors that display cloned antigens on the virion surface. Science 228(4705), 1315-1317 (1985)
[44] Szklarczyk, D., Morris, J.H., Cook, H., Kuhn, M., Wyder, S., Simonovic, M., Santos, A., Doncheva, N.T., Roth, A., Bork, P., et al.: The STRING database in 2017: quality-controlled protein-protein association networks, made broadly accessible. Nucleic acids research 45(D1), D362–D368 (2017)
[45] Tubiana, J., Schneidman-Duhovny, D., Wolfson, H.J.: ScanNet: an interpretable geometric deep learning model for structure-based protein binding site prediction. Nature Methods 19, 730–739 (2022)
[46] Wilman, W., Wróbel, S., Bielska, W., Deszynski, P., Dudzic, P., Jaszczyszyn, I., Kaniewski, J., Młokosiewicz, J., Rouyan, A., Satlawa, T., et al.: Machine-designed biotherapeutics: opportunities, feasibility and advantages of deep learning in computational antibody discovery. Briefings in Bioinformatics 23(4), bbac267 (2022)
[47] Wilton, E.E., Opyr, M.P., Kailasam, S., Kothe, R.F., Wieden, H.J.: sdAb-DB: the single domain antibody database. ACS Synthetic Biology 7(11), 2480–2484 (2018)
A.1 Ethics Statement for Animal Experiments
All animal experiments on an alpaca were conducted in accordance with the KYODOKEN Institute for Animal Science Research and Development (Kyoto, Japan) and the ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines. Veterinarians performed breeding, health maintenance, and immunization by adhering to the published Guidelines for Proper Conduct of Animal Experiments by the Science Council of Japan. The KYODOKEN Institutional Animal Care and Use Committee approved the protocols for these studies (KYODOKEN protocol number 20190216).
Our data generation method uses animal models immunized with a target protein that could potentially harm the animals, such as a particular toxin, pathogen, or allergen. Hence, the risk to the animal should be minimized by treating the immune source to inactivate or detoxify it.
A.2 Dataset Generation
Here, we describe the detailed experimental procedures and conditions for each step.
Step 1. Immunization We immunized a single alpaca with purified recombinant human IL-6 protein and several single-amino-acid mutants. Specifically, the gene encoding the human IL-6 protein was codon-optimized, synthesized, and sub-cloned in the pcDNA3.1(+) vector (Thermo Fisher Scientific K.K., Tokyo, Japan). The amino acid sequence of the wild-type IL-6 protein with a C-terminal  $6 \times$  His-tag was "MNSFSTSAFGPVAFSLGLLLVLPAAFPAPVPPGEDSKDVAAPHRQPLTSSERIDKQIRY-ILDGISALRKETCNKSNCMCESSKEALAENNLNPKMMAEKDGFQSGFNEETCLVKIITGLLEFEVYLEYLQNRESSEEEQARAVQMSTKVLIQFLQKKAKNLDAITTPDPPTTNASLLTLQAQNQWLQDMTTHLILRSFKEFLQSSLRALRQMHHHHHH." We introduced a site-directed mutation with alanine at intervals of three to six amino acids, like the alanine scanning technique [15], which is used in molecular biology to determine the contribution of a specific amino acid. A total of 30 single-amino-acid mutants was prepared: P42A, Q45A, T48A, E51A, D54A, I57A, I60A, G63A, K69A, C72A, C78A, S81A, E87A, L90A, P93A, D99A, F102A, G105A, E108A, T117A, L120A, L126A, L129A, S135A, E138A, Q144A, F153A, D162A, T165A, and D168A. Here, for example, P42A means that an amino acid in the wild type was substituted from proline to alanine at position 42. The antigen cocktail mixture was emulsified in Titermax (Funakoshi, Tokyo, Japan) adjuvant at a dose of  $600 \mu \mathrm{g}$  and subcutaneously injected into an alpaca four times at about two-week intervals. Lymph nodes and blood samples were each collected four times, resulting in a total of 12 libraries.
Step 2. Phage Library Construction Peripheral blood mononuclear cells (PBMCs) were obtained from blood samples by sucrose density gradient centrifugation using Ficoll (Nacalai Tesque, Kyoto, Japan). The lymph nodes and PBMC samples were washed with phosphate-buffered saline (PBS, Nacalai Tesque) and suspended in an RNAlater solution (Thermo Fisher Scientific K.K., Tokyo, Japan). Total RNA was isolated from these samples by using Direct-Zol RNA MiniPrep (Zymo Research, Irvine, CA). Complementary DNA was synthesized from  $1\mu \mathrm{g}$  of total RNA as a template by using random hexamer primers and SuperScript II reverse transcriptase (Thermo Fisher Scientific K.K.). The coding regions of the VHH domain were amplified using LA Taq polymerase (TAKARA Bio Inc., Shiga, Japan) with two PAGE-purified primers (CALL001,  $5^{\prime}$ -GTCCTGGCTGCTCTTCTACAAGG-3' and CALL002,  $5^{\prime}$ -GGTACGTGCTGTTGAACTGTTCC-3'), and they were separated on a  $1.5\%$  low-melting-temperature agarose gel (Lonza Group AG, Basel, Switzerland). Approximately 700 base-pair bands were extracted using a QIAquick Gel Extraction Kit (Qiagen K.K., Tokyo, Japan). Nested PCR was performed to amplify the VHH genes by using two primers that contained flanking PstI (forward) and BstEII (reverse) restriction sites to enable cloning into the pMES4 phagemid vector with a C-terminal His-tag. Electroporation-competent Escherichia coli TG1 cells (Agilent Technologies Japan, Ltd., Tokyo, Japan) were transformed with the ligated plasmids under chilled conditions (Bio-Rad Laboratories, Inc., Hercules, CA). The library densities were monitored and maintained at  $>10^{7}$  colony-forming units per microliter with limiting dilution. Colonies from  $8\mathrm{mL}$  of cultured cells were harvested, pooled, and reserved in frozen glycerol stock as a mother library. Thus, the 12 phagemid libraries were designated as the mother libraries.
2ARRIVE guidelines: https://arriveguidelines.org
Step 3. Affinity Selection One round of biopanning was performed using each target protein-coated magnet beads in  $50\mathrm{-mM}$  phosphate buffer  $(\mathrm{pH}7.4)$  containing  $0.1\%$  Triton X-100 (Nacalai Tesque),  $0.3\%$  (w/v) bovine serum albumin (BSA, Nacalai Tesque), and  $500~\mathrm{mM}$  of NaCl. Every IL-6 mutant was used at  $1.2\mathrm{mL}$  bead slurry, which was saturated with  $240~\mu \mathrm{g}$  of protein, except for P93A  $(90~\mu \mathrm{g})$ , E108A  $(190~\mu \mathrm{g})$ , and L126A  $(180~\mu \mathrm{g})$ . To distinguish nonspecific signals, a negative control sample that did not contain any IL-6 protein was also used. The wild-type IL-6 protein libraries were obtained in triplicate to confirm the reproducibility. After three washes with the same buffer, the remaining phages bound to the beads were eluted with a trypsin-ethylenediaminetetraacetic acid (EDTA, Nacalai Tesque) solution at room temperature for 30 minutes. The eluate was neutralized with a PBS-diluted protein inhibitor cocktail (cComplete, EDTA-free, protease inhibitor cocktail tablets, Roche Diagnostics GmbH, Mannheim, Germany) and used to infect electroporation-competent cells. The infected cells were cultured in LB Miller broth containing  $100~\mu \mathrm{g / mL}$  of ampicillin (Nacalai Tesque) at  $37^{\circ}\mathrm{C}$  overnight. The genes of the phagemids selected by biopanning were collected with a QIAprep Miniprep Kit (Qiagen), amplified by PCR, and purified using AMPure XP beads (Beckman Coulter, High Wycombe, UK). Then, dual-indexed libraries were prepared and sequenced on an Illumina MiSeq (Illumina, San Diego, CA) by using a MiSeq Reagent Kit v3 with paired-end 300-bp reads (Bioengineering Lab. Co., Ltd., Kanagawa, Japan).
Step 4. Sequence Analysis Approximately 100,000 paired reads for each library were generated by NGS analysis. The raw read data were trimmed to remove the adaptor sequence by using cutadapt v1.18 [30] and to remove low-quality reads by using Trimmomatic v0.39 [9]. The remaining paired reads were merged using fastq-join [5], and then the VHH coding sequences were extracted using seqkit v0.10.1 [41]. The DNA sequences were translated to amino acid sequences with EMBOSS v6.6.0.0 [36], and the VHH sequences were cropped from start to stop codon. Finally, each phagemid library was converted to a FASTA file containing tens of thousands of VHH sequences.
A.3 Label Reliability
A.3.1 Experimental Procedures