AVIDa-hIL6: A Large-Scale VHH Dataset Produced from an Immunized Alpaca for Predicting Antigen-Antibody Interactions
Authors: Hirofumi Tsuruta
Hiroyuki Yamazaki
Ryota Maeda
Ryotaro Tamura
Jennifer N. Wei
Zelda E Mariet
Poomarin Phloyphisut
Hidetoshi Shimokawa
Joseph R. Ledsam
Lucy J Colwell
Akihiro Imura
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: antibody
antigen-antibody interaction
drug discovery
VHH
immunization
machine learning
TL;DR: We introduce AVIDa-hIL6, an antigen-variable domain of heavy chain of heavy chain antibody (VHH) interaction dataset produced from an immunized alpaca.

Abstract: Antibodies have become an important class of therapeutic agents to treat human diseases.
To accelerate therapeutic antibody discovery, computational methods, especially machine learning, have attracted considerable interest for predicting specific interactions between antibody candidates and target antigens such as viruses and bacteria.
However, the publicly available datasets in existing works have notable limitations, such as small sizes and the lack of non-binding samples and exact amino acid sequences.
To overcome these limitations, we have developed AVIDa-hIL6, a large-scale dataset for predicting antigen-antibody interactions in the variable domain of heavy chain of heavy chain antibodies (VHHs), produced from an alpaca immunized with the human interleukin-6 (IL-6) protein, as antigens.
By leveraging the simple structure of VHHs, which facilitates identification of full-length amino acid sequences by DNA sequencing technology, AVIDa-hIL6 contains 573,891 antigen-VHH pairs with amino acid sequences.
All the antigen-VHH pairs have reliable labels for binding or non-binding, as generated by a novel labeling method.
Furthermore, via introduction of artificial mutations, AVIDa-hIL6 contains 30 different mutants in addition to wild-type IL-6 protein.
This characteristic provides opportunities to develop machine learning models for predicting changes in antibody binding by antigen mutations.
We report experimental benchmark results on AVIDa-hIL6 by using machine learning models.
The results indicate that the existing models have potential, but further research is needed to generalize them to predict effective antibodies against unknown mutants.
The dataset is available at https://avida-hil6.cognanous.com.

AVIDa-hIL6: A Large-Scale VHH Dataset Produced from an Immunized Alpaca for Predicting Antigen-Antibody Interactions
Hirofumi Tsuruta', Hiroyuki Yamazaki'; Ryota Maeda'; Ryotaro Tamura', Jennifer N. Wef", Zelda Mariet?, Poomarin Phloyphisut", Hidetoshi Shimokawa",
Joseph R. Ledsam2 Lucy Colwell', Akihiro Imura'*
'COGNANO Inc., Goooge LLC
(tsuruta, yamazaki, maeda, ryotarotamra,, akthirooimura@ooonaano co. (weijennifer, zmariet, poomarin, simokawa, jledsam, 1colwell)@googler
Abstract
Antibodies have become an important class of therapeutic agents to treat human diseases. To accelerate therapeutic antibody discovery, computational methods, especially machine learning, have attracted considerable interest for predicting specific interactions between antibody candidates and target antigens such as viruses and bacteria. However, the publicly available datasets in existing works have notable limitations, such as small sizes and the lack of non-binding samples and exact amino acid sequences. AVIDa-hIL6, large-scale dataset for predicting antigen-antibody interactions in the variable domain of heavy chain of heavy chain antibodies (VHHs), produced from an alpaca immunized with the human interleukin-6 (IL-6) protein, as antigens. By leveraging the simple structure of VHHs, which facilitates identification of full- length amino acid sequences by DNA sequencing technology. AVIDa-hIIl contains 573,891 antigen- VHH pairs with amino acid sequences. All the antigen- VHH pairs have reliable labels for binding non-binding, as generated by novel labeling method. Furthermore, via introduction artificial mutations, AVIDa-hlLo contains 30 different mutants in addition to wild-type IL-6 protein. This characteristic provides opportunities to develop machine learning models for predicting changes antibody binding by antigen mutations. We report experimental benchmark results on AVIDa-hlL6 by using machine learning models. The results indicate that the existing models have potential, but further research is needed to generalize them to predict effective antibodies against unknown mutants. The dataset is available at httpss//avida /at:://vvviaaaa hil6. cognanous. com
overcome these limitations, we have developed
1 Introduction
Antibodies are proteins that play an essential role in the immune system. When antigens such as viruses and bacteria invade the body, the immune system protects the body by producing large numbers of antibodies that bind to the antigens to inhibit their function mark them for removal. Antibodies have become an important class of therapeutic agents to treat human diseases because of their high target specificity and binding affinity 22. An essential step in therapeutic antibody discovery is the identification of specific interactions between antibody candidates and target antigens, which has traditionally relied heavily on expensive, time-consuming experiments [24]. Therefore,
Equull contribution.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

computational approaches are increasingly used to complement and accelerate traditional processes
for therapeutic antibody discovery [46 25
In particular, there is growing interest in using machine learning to predict antigen-antibody interac- tions [39 20), which can be used to virtually screen binding antibodies against specific target antigens. Schneider et al. [39] developed the structure-based deep learning for antibodies virtual screening (DLAB-VS) by using the structural antibody database (SAbDab) [17]. which contains collections of antigen-antibody complex structures. Lim et al. (28) generated datasets of antibody sequences from mice immunized with cytotoxic lymphocyte-assoccin
antigen 4 (CTLA-4) and programmed cell death protein 1 (PD-1); then, they built deep learning models to predict binder and non-binder antibodies to CTLA-4 and PD-1. Huang et al. [20] proposed AbAgIndPre, deep Searing-assisste prediction method that was trained using only the amino acid sequences in two public databases: SAbDab and the coronavirus antibody database (CoV-AbDab) [35].
Despite these promising developments, progress in therapeutic antibody discovery has lagged behind progress in other areas of drug discovery. major reason for this is the lack of availability of high-quality, large-scale datasets of antigen-antibodly interactions. First, most existing datasets have small sample sizes. For example, as of May 2023, SAbDab and Lim et al. 's datasets 28 contain 5,737 and 3,064 binder samples, respectively. In addition, SAbDab only has samples for binding antigen-antion antibody pairs. In previous studies [3 20 using SAbDab, antigens and antibodies were randomly paired to form non- bindiinni pairs. CoV-AbDab contains 12,021 entries, from which more than 30,000 antigen-anttibddy pairs, including non- non-binding pairs, are available. However, Cob- provides only the variant name and not the amino acid sequence. As the variant name is defined by representative mutations, the exact amino acid sequence may vary between publications, thus making it difficult to use CoV-AbDab for antibody discovery because single amino acid change can be critical for an antigen-antibodly interaction.
To overcome these limitations, we have developed AVIDa-hlL6, an antigen-variabl domain of heavy chain of heavy chain antibody (VHH) interaction dataset produced by an alpaca immunized with the human interleukin-6 (IL-6) protein. IL-6 is relatively small protein, simply structured, well- characterized cytokine that exists as a monomer in the body and is associated with many inflammatory diseases and cancers. To ensure a wide variety of antibody sequences, we used VHHs, whose simple structures enable much easier identification of full-length amino acid sequences by DNA sequencing technologies such as next-generation sequencing (NGS) than for conventional antibodies. By leveraging these advantages, AVIDa-hlL6 contains 573,891 antigen-VHH VHH pairs, including 20,980 binding pairs, with their amino acid sequences. In addition, we have developed a novel labeling method to obtain reliable labels for binding and non-binding.
Furthermore, AVIDa-hIL6 contains information on the interaction of diverse VHHs with 30 different mutants produced by artificial point mutations, in addition the wild-type IL-6 protein. As the COVID-19 pandemic has shown, viruses continuously evolve through mutation to evade the immune system. Because emerging mutations involving amino acid substitutions can lead to profound changes in antibody binding, prediction of their effects is critical in the development of therapeutic antibodies. Notably, AVIDa-hll6 contains antibody sequences that are positive for most IL.-6 mutants but negative for specific IL-6 mutants, or vice versa, thus providing important insights for understanding how antigen mutations affect antibody binding.
The main contributions of this paper are summarized as follows.
We release AVIDa-hIL6, which the largest existing dataset for predicting antigen-antiboon interactions ( (10 times larger than any other public dataset) and contains amino acid sequences of antigens and antibodies and binary labels for binding and non- non-binding pairs.
AVIDa-hILL hIL6 has the wild type and 30 mutants of the IL-6 protein as antigens, and it includes many sensitive cases in which point mutations in IL-6 enhance or inhibit antibody binding.
We have designed a novel data generation method, including data labeling, by using the immune system of live alpaca. This method can be applied to any target antigen, in addition to 1I.6.
We report benchmark results for the prediction of antigen-antibon interactions by using machine-learning laaeee-eernnnnnbeeeeee based baseline models. These results confirm that AVIDa-hIL6 provides valuable benchmarks for assessing a model's performance in capturing the impact of antigen mutations on antibody binding.
2

---

Table 1: Characteristics of public datasets for predicting antigen-antibot interactions. The numbers of samples were obtained in May 2023. v* denotes that relevant information was missing or not available for part of the dataset.
#Samples
Sequence
Datasct
Binder
Non-binder
Antibody
Antigen
Structure
Antibody type
Data collection
SAbDab-nano
40
1,114
VHH
Curation
sdAb-DB
1,452
Curation
SAbDab
17
5,737
Conventional, VHH
Curation
Lim et ol. 28
3,064
12,055
/
Conventional Conventional. VHH
Experiment
CoV-AhDab B5
32,064
5,303
Curation
AVIDa-hIL6
20,980
552,911
Experiment
Related Work
In this section, we put our work into context with public datasets for predicting antigen-antibody interactions. Recent advances in NGS technology now enable the construction of large-scale databases of antibody sequences, such as the observed antibody space (OAS) [31] and iReceptor [14]. However, those databases cannot be directly used as training data for predicting antigen-antiboue interactions the antigen corresponding
because of the lack of information databases are primarily used to build antibody-speccif language models [37 32 26] and to generate
each antibody. Thus, those
new antibody sequences via deep generative models [3]. Here, we focus on datasets with information on antigen-antibode interactions, as summarized in T
The datasets listed in Table I contain two types of antibodies: conventional and VHH. A conventional antibody comprises two pairs of heavy and light chains. A conventional antibody acts a single functional unit by combining the heavy and light chains encoded on separate chromosomes. For example, humans, the heavy chain is encoded on chromosome 14, and the light chain is encoded chromosome 2 or 22. Therefore, purification of a single-cell lymphocyte is essential for DNA sequencing of the
Antibody Type.
regions on an antibody. Such cell cloning comprises several steps and is time-consuming, making analysis the order of 10 the third power or more virtually impossible. In contrast, a VHH, found in camelids such as alpacas and Ilamas, comprises only heavy chains. The heavy-chain antibodies are derived from single gene, e.g., they are encoded on chromosome 4 in alpacas; moreover, a VHH is the smallest functional unit of heavy-chain antibodies, and the sequencing for the aatigen-determining cell cloning. Thus, we can perform exhaustive analysis on the order of six powers of 10 or more by simply extracting DNA from a bulk sample of lymphocytes. Additionally, VHHs have recently gained interest as therapeutic agents because of their small size, high stability, good human tolerability, and relative ease of production [22 21. SAbDab-nam [40 and sdAb-DB [47] are public databases that collect only VHHs, but they both have too few samples for machine learning drug discovery or design applications. Hence, we use immunized alpacas as data source to generate a large amount of VHH sequence data.
regions does not require
Sequence and Structure Information. SAbDab [17 and its sub-database, SAbDab-nano 40) collect all the available antigen-antibody complex structures in the Protein Data Bank (PDB) g8] Also, some data in sdAb-DB [47] and CoV-AbDab 35 include structural information from the PDB. Because accurate knowledge of antibody structures is important for understanding the antigen- binding function of antibodies. SAbDab is increasingly used for antibody structure prediction via deep learning 38 11. However, experimental methods for antibody structure deternination, such as X-ray crystallography and cryo-electron microscopy, are relatively expensive and time-consuming, making it difficult to increase the amount of data. More recently, machine learning methods such as AlphaFold (23) and Rose (6), which accurately predict a protein's 's structure from the amino acid sequence, have greatly accelerated progress in the biological sciences. AVIDa-hIL6 focuses on amino acid sequences of antigens and antibodies to generate sufficient training data for machine learning.
Number Labeled Samples. Some existing datasets only have samples for binding antigen- antibody pairs. One reason is that the identification of non-binding antigen-antibody pairs has little clinical significance. previous studies (39 20 using SAbDab, antigens and antibodies were randomly paired to form non-binding pairs This process was based on the assumption that
3

---

Step Immunization
Phage Library
Step Affinity Selection
Step Sequence Analysis
Step Data Labeling
Construction
be artibody-arget pairs
lirrries
subiibraries
420 read-court
motiere
Binneinndigg phagee
Before parning
rcchera
After pansing
1-6. PIDA
Propertion of Vitl
2nd
phagersd
reagest beaiss
34 bargt meleaato
<3d layst
Casusis p-okee and seed (BaTAT pretu
moksle
regatin
I-4 DISSA
roh-ignalarr
Wesh
Tras
4h bloo
rothard
Counr expurencee each Vieit MS8
Pute
navdrnder
Inmunniee don wibkipp and rurans I.
beder
Raduse biss poethee sngilio
Figure
Overview of the data generation process for AVIDa-hIL.6
randomly sampled pairs are unlikely bind because of antibodies' high target specificity. Lim al. (28) generated a dataset of antibody sequences labeled as "binder" and "non-binder" through an experimental approach using mice immunized with CTLA-4 and PD-1. The number of samples in i the total for CTLA-4 and PD-1. As with AVIDa-hIL6, Lim et al. created their dataset from their original experiments, thus revealing the potential to increase the amount and diversity of data by the same approach using arbitrary antigens. Co collects antibodies that bind to at least one beta coronavirus and currently contains 12,021 entries. Because each entry has "Binds to" and "Doesn't 't Bind to" columns and contains zero to multiple antigens for specific antibody, the number of samples in Table T was counted for each possible antigen/antibody pair. However, ab only provides variant names, e.g., SARS-CoV 11111mmmmoonBBAA or SARS-CoV2_AIpp and each antigen's exact amino acid sequence is only available if was provided in the original publication.
3 AVIDa-hILó: Antigen-V VHH Interaction Dataset Produced from Alpaca Immunized with Human IL-6 Protein
AVIDa-hIL6 is dataset of antigen- VHH interactions with amino acid sequences and binary labels for binding and non- binding. In this section, we introduce the dataset generation process, dataset statistics, and verification of label reliability. A labeled dataset and the data available httts://avida-Dai hil6. cognanous... com The dataset is released under a CC BY-NC 4.0 license.
3.1 Dataset Generation
Figure [ shows an overview of the data generation process. The KYODOKEN Institutional Animal Care and Use Committee approved the protocols for the experiments (see Appendix I. Appendix A.2 gives the detailed experimental procedures and the amino acid sequences of the IL-6 proteins.
Step 1. Immunization To ensure the diversity of antibodies binding to the IL-6 protein that we used as an antigen, we used the immune system of live alpaca. We introduced site-directed mutation with alanine at intervals of three to amino acids, like the alanine scanning technique [15], which is used in molecular biology to determine the contribution of specific amino acid; result, we obtained 30 types of mutants addition to the wild-type IL-6 protein. A mutant is denoted, for example, as 1L6 P42A, which means that an amino acid in the wild type is substituted from proline to alanine at position 42. We immunized a single alpaca with a cocktail of 31 different IL-6 proteins four times at about two week intervals. After each immunization, one blood sample and one or more lymph nodes from different body sites were collected, yielding a total of 12 libraries. We provide additional information about the collection process at https: / //tavv//vvii aviida hil6. cognanous. com

---

Step 2. Phage Library Construction We used phage display 43 to identify VHHs that bind to the IL-6 protein. Phage display is technique for displaying the target proteins on a phage surface in a form that allows them to bind to other molecules. This versatile technique enables generation of protein libraries containing up to 10th different variants and is often used for affinity screening of antibodies for their binding partners 77. We cloned the VHH genes obtained from each library into the PMES4 phagemid vector to display the VHHs on the phagemid surface. As a result, 12 phage libraries corresponding to each of the above libraries were generated and designated as the mother libraries.
Step Affinity Selection Affinity selection by biopanning using the mother libraries can enrich a phage with displayed VHHs that bind to the target molecule. For the target molecules, we used the wild type and 30 mutants of IL-6 and negative control sample that did not contain any IL-6 protein. Only experiments targeting the wild-type IL-6 protein were performed in triplicate to ensure reproducibility. The mother library was added the container and incubated with target-coated magnet beads. Then, non-binding phages were washed away. and the remaining phages that bound to the beads were eluted. Consequently, by performing one round of biopanning on each of the 12 mother libraries, we generated a total of 408 sublibraries.
Step 4. Sequence Analysis The amino acid sequences of VHHs displayed on a phage surface can be identified by analyzing the phage genome's ' DNA with NGS technology. Approximately 100,000 paired reads were generated for each library by NGS, and singletons were removed to avoid sequencing errors. The DNA sequences were translated into amino acid sequences. We counted the number of occurrences of each unique VHH amino acid sequence from the paired reads, which reflected the concentration of each VHH in the library. For each of the 12 mother libraries before panning and 408 sublibraries after panning, we created a table with the VHH amino acid sequences and their read counts.
Step 5. Data Labeling We designed a labeling method to distinguish whether a VHH binds to each IL-6 protein type by applying a statistical test for differences in the proportions of cach VHH in library before and after panning. Here, we focused examining the binding between a specific VHH and a specific target molecule. Let p1 and pa denote the population proportions of a specific VHH in the libraries before and after panning. We identified some of the VHH sequences in the libraries by NGS analysis. Let n1 and n2 denote the libraries' total read counts before and after panning, respectively, and let Ii and 22 denote the read counts of specific VHH in the libraries. Then, the respective sample proportions of a specific VHH in each library are pl = Σn and f2 d2 Given that the minimum value of all possible n, and n2 was over 10,000, we assumed that Pí and f22 follow normal distributions with mean Pl and P2 and variance (1-1p11) and pa(1-pa), respectively, A2
n1 according to the central limit theorem. Furthermore, the difference in the proportions in fr2 can also be approximated by a normal distribution due to the reproductive property of the normal distribution. Thus, the test statistic Z under null hypothesis Ho P1 = P2 was calculated as follows.
P1 P2
Z
p(1 p)( - + m22
21222 The p-value of Z was calculated using where p is the pooled proportion calculated as p ni ni+nnn the standard normal distribution. In the same way, p-values were calculated for all VHIH-ttrget pairs in the sublibraries with respect to the 12 corresponding mother libraries. Because we had 12 sublibraries associated with the same target molecule, we adopted the smallest p-value, indicating the most significant difference in proportion, among identical VHH-target pairs. If a specific VHH's proportion sublibrary increased from the proportion in the corresponding mother library and the p-value was 0.05 or less (our chosen significance level), the VHH-target pair was labeled with "binder." Similarly, if the proportion decreased and the p-value was 0.05 or less, the pair was labeled with "non-binder." Finally, if the p-value exceeded 0.05, the pair was labeled with "non-significant" Our dataset contains 1,998, 127 samples labeled non-significant, including 325,865 unique VHH sequences that are present in the alpaca body. These labels were not used for supervised learning to predict antigen-antibody interactions. However, these samples may be helpful for pre-training via self-supervised learning used by existing antibody-speciff language models [37 32 26).
5

---

0.10
bnew
tack!
15A00
0.05
0.82
0.00
20
40
60
100
Sequence lcerity (55)
Figure 2: (a) Number of samples for each antigen type. (b) Visualization of the labels for pairs of 100 VHH sequences and each antigen type. Each cell represents unique VHH-antiggn pair. White cells denote non-significant noise labels. (c) Distribution of the pairwise identities of the VHH sequences.
The results of biological experiments always contain background noise, such as binding to contami- nating proteins. Therefore, we developed novel noise reduction algorithm to avoid false positives and improve label reliability. We reconfirmed VHHs labeled as "binder" to any of the IL-6 proteins by comparing the labels to negative control samples under the following conditions.
1. If the VHH was a non-binder the negative control sample, the label remained "binder."
2. If the VHH was binder to the negative control sample, the label was reassigned from "binder" to "noise" because of possible false positives.
3. If the VHH was "non-significant" with respect the negative control sample, the ratio of the p-value of the negative control sample to that of the IL-6 protein was compared to 20 This value was empirically determined by an author (a biologist) according feedback from biological experiments in our previous studies [29].
(a) the ratio of p-values was below 10 the label was reassigned from "binder" to "non-significant" because of possible false positives.
(b) If the ratio of p-values was 102.5 more, the label remained "binder."
We carefully verified the reliability of our labels, as discussed in section 3.3 The code for data labeling is available at at/ttps:///gthhit coaam//oggaoo////// AVID
3.2 Dataset Statisties
AVIDa-hlL6 contains 573,891 data samples, comprising 20,980 binding pairs and 552,911 non- binding pairs. The proportion of binding pairs is about 3.7 %. Figure 2 a) shows the number of samples for each antigen type. Although the number of samples varied for each IL-6 protein type, we successfully generated least 10,000 samples for the wild type and 30 different mutants. Furthermore. at least 250 binder VHH sequences existed for each IL-6 protein type. Because we labeled the VHH sequences the mother library for each IL-6 protein type, AVIDa-hIL6 has information on whether the same VHH sequence binds to each of multiple targets. The number of unique VHH sequences in AVIDa-hIL6 is 38,599, including 4,425 sequences that bind at least one IL-6 protein type. Importantly, 650 VHH sequences, about 14.7 % of the VHH binders, show binding to specific IL-6 protein types but non-binding to others. We visualized whether 100 sequences extracted randomly from these 650 VHH sequences bound to each antigen type, as shown in Figure 2bb). These samples have valuable information on which mutations enhance or inhibit antibody binding, which should be strongly associated with the IL-6 protein's binding site. Furthermore, when focusing the same VHH sequence, i.c., one row, we observe that mutants with mutations at closer positions tend to have the same label.
To gain a better understanding the distribution of VHH sequences, we compared it to the distri- butions in the existing VHH datasets SAbDab-nan nano and sdAb-DB. We used only the binders from AVIDa-hIL6 because the existing datasets only contain binders. The numbers of unique VHH binders in SAbDab-nano, sdAb-DB, and AVIDa-h1L6 are 828, 1,414, and 4,425, respectively. To mitigate the computational complexity, we randomly sampled 700 unique VHH sequences from each dataset and calculated all pairwise sequence identities with Biopython v1.81 [13]. Figure ( shows the

---

(a) Binder (ONU14)
(b) Non-binder (ONU82)
(c) Noise (ONU250)
H2
Mergo
Marge
Marge
Figure 3: Immunoftuorescence staining analysis using three VHHS labeled with (a) binder, (b) non-binder, and (c) noise. HA-tagged IL-6 proteins were introduced into cells and stained with His-tagged VHHs. Subsequently, HA-tags were visualized in green and His-tags in red. The ONU is our identifier to distinguish the VHH sequence.
Table 2: Numbers of samples in the training and test sets.
*Samples
Daraser
Hinder
Non-binder
Antigen types
Training
10,364
282,279
292,845
Wild-type, Q45A, D54A, G63A, C72A, L9DA, P93A, D99A, FIGZA, GIOSA, EIO8A, TI1ZA, S $135A, ELSSA, FIS3A, DI68A P4ZA, T48A, ESIA, 160A, KO9A, C78A, E87A, L120A, L126A, L129A, QI+A, DI62A, TI65A
Test 10,416
270.632
281,048
distributions of sequence identities for these datasets. The results indicate that AVIDa-hIL6 has peaks at regions of higher sequence identity than the other datasets. Interestingly, AVIDa-hIL6 has a peak at 97 % sequence identity, which is absent for the others. As living organism's 's immune response progresses, effective antibodies with high binding affinity the antigen are selected through a process called affinity maturation and are further mutated by repeated exposure to the antigen. Thus, these results may reflect that a live alpaca's immune system selects VHH sequences with high sequence identity that specifically bind to target IL-66 proteins through affinity maturation.
3.3 Label Reliability
To verify our label reliability, we tested the antibody binding ability by immunoffuorescence staining. Because the number of VHHs that could be verified was limited by the time and cost of biological experiments, VHHs were selected under the following conditions to verify label reliability efficiently. First, we examined only the wild-type IL-6 protein as target antigen. Next, the amino acid sequences of all labeled VHHS with higher than 93 % identity were clustered by the UCLUST algorithm [8] to validate diverse sequences. When two or more VHHs with the same label were in the same cluster, the one with the highest read count was selected. We know empirically that if all the VHHs cluster have the same label, these labels are likely to be true. Therefore, such VHHs were excluded from the candidates. Then, VHHS with suspect labels were selected in order of their read counts. Finally, we tested 10 bander,, six non-binder-labeled and four noise-labeled VHHs for validation.
Immunofuuorescence analysis showed that all 10 binder-labeled VHHS actually bound to the wild-type IL-6 protein, whereas the six non- non-binder-lad bibder-eabeeeedd and four ppise-labeled VHHs did not. Figure B shows the results for a representative clone of the three types of labeled VHHs. Appendix(A.3.. A.3.2 gives the results for all the tested clones and their amino acid sequences. We could observe the overlapping of IL-6 signals and VHH signals in the binder group, whereas the VHH signals were lost in the non-binder group. The VHH signals did not coincide with the IL-6 signals in the noise group, which can be interpreted as labeled VHHs binding nonspecifically to cells. These results indicate that our noise reduction algorithm contributed to reducing false positives. In addition. these results were also confirmed by kinetic assay via biolayer interferometry (BLI), as described in Appendix AA.32 As result, we could ensure that AVIDa-hlI.d has highly reliable labels.
A...2
4 Benchmarks
4.1 Benchmark Task
To demonstrate the use of AVIDa-hIL6 for antibody discovery. we performed an experiment on binary classification of whether a given antigen-antibody pair binds. By leveraging information on the binding of diverse antibodies to antigen mutants, we defined benchmark task to assess the model performance in capturing the impact of antigen mutations on antibody binding. First, we randomly selected 15 mutants and reserved the data samples for those mutants test set. The remaining 15
7

---

mutants and the wild type were reserved for model training. Table lists the numbers of samples in the training and test sets. As we used artificial point mutations, each mutant's sequence identity with respect to the wild type was the same, differing only in the position where the alanine was introduced. Next, we trained models by using only the wild-type II.-6 protein and evaluated their performance in predicting antibody bindings in the test set. Then, we randomly selected one mutant from the remaining 15 mutants outside the test set and added it to the training set to evaluate each model's predictive performance. By repeating this process, we tracked the model's predictive performance for unknown mutants contained only in the test set. Because the order of adding mutants to the training set affected the model performance, we the same experiment five times in shuffled order, and we report the averaged results here. For all model training, we randomly selected 10 % of the training set for model validation. This experimental scenario assumes that antigen mutants emerge one after another to evade the immune system, as in the COVID-19 19 pandemic. In such a scenario, we evaluated the model's performance in predicting antibody candidates that will bind to future emerging mutants according to the binding information of antigens that have already been observed.
4.2 Baseline Models
We adopted three neural network- -based models and one classical machine learning model as baselines. The model inputs were the amino acid sequence of an IL-6 protein with a length of 218 and a VHH with a maximum length of 152.
AbAgIntPre 20 state-of-the-art model designed for aatigen-antibody interactions
based on amino acid sequences. It combines the composition of k-spaced amino acid pairs (CKSAAP) [IT] encoding and convolutional neural network (CNN) model with a
Siamese- like architecture. We used the model parameters reported in the original paper [20].
PIPR [10] is a residual recurrent convolutional neural network (RCNN) for protein- protein interaction (PPI) prediction. Following the PIPR strategy. we used an amino acid encoding that combined a five-dimensional vector obtained from pretrained skip-gram model using the STRING database 44 and a seven-dimensional vector describing the categorization of electrostaticity and hydrophobicity. We changed the number of RCNN units from five to three because our sequence length was more than nine times shorter than the protein input in the original PIPR. The other model parameters were the same as in the paper [10]. Although PIPR was not specifically designed for antigen-antibody interactions, such interactions that ignore non-protein antigens can be considered subset of PPI, meaning that models designed for PPI can also apply to antigen-antiboud interactions.
A Multi-Layer Perceptron (MLP) with one hidden layer of 512 neurons was used as a simpler neural network- based model than the above two models. We used one-hot encoding to represent amino acid sequences. One-hot vectors of the VHHs and IL-6 proteins were flattened and concatenated for input to the MLP.
Logistic Regression (LR) was used as classical machine learning model that is commonly used for binary classification tasks. As with the MLP, we used flattened and concatenated one-hot vectors of the VHH and II.-6 proteins as input.
In this experiment, the three neural-nettork- NVIDIA Tesla V100 GPU on Google Colaboratory with an initial learning rate of 0.0001 and a batch size of 256. The LR was trained using one Intel(R) Xeon(R) CPU on Google Colaboratory. Jlive Appendixes A.4.2 and A.4.3) give more details on the model implementation and training. The code to run the benchmark models is available at htttttt///// hnttpa://githut tht. http:hhhhh.ooooocoooooeooeeooeee www.w////oo///////oo///o.oo.o/ AAt//AAIIDDaaIII
models were trained for 100 epochs on one
4.3 Results
Figure 4 shows the prediction performance of the baseline models as a function of the number of IL-6 protein types used for model training. We used the precision, recall, and F1-score as evaluation metrics because the prediction of antibody binders, which are fewer in number than non non-binders, is much more important for drug discovery. Figure(4it) shows the precision-recall curves when 1 and 16 I1-6 protein types were used for training. When the number of antigens was 1-that is, when only the wild-type II.-6 protein was used for training- the recalls of AbAgIntPre, PIPR, MLP, and LR were 67.9, 57.6, 67.2, and 6 %, respectively. These results indicate that the models failed
8

---

(8)
0.00
Phodaze
0:70
0,00
0.5
0.75
0.0
010
0.70
Marigen Type
Morkyen Typee
MArigen Typee
(b)
MArrigen Types
#Anigenn Typee
0.6
Neggethh AUTDE EXC
0.2
18 257
1R A3C--0300 3.300
00
5.0
Figure 4: (a) Precision, recall, and Fi-score as function of the number of IL-6 protein types used for model training. (b) Precision-recall curves when 1 and 16 antigen types were used for training. The legend shows the area under the curve (AUC) values.
to predict over 30 % of the effective VHHs that bound to mutants in the test set. All the metrics improved as the number of IL-6 protein types used for training increased, and this trend is clearly evident in the precision-reces recaal curves and the area under the curve (AUC) values. After adding 15 mutants for training, the precisions of all baseline models reached about 95 %, but the recalls were still only about 85 %, even for the three neural-network. based models. possible factor in this result is that the proportion of binding pairs only about 3.7 %. Hence, techniques to address imbalanced labels, such as oversampling and undersampling, would be useful to improve the model performance.
For drug discovery applications, the construction of generalized model for unknown mutations from as little antigen-binding information as possible is ideal, because the number of possible mutations in antigens is tremendously large. As shown by the Fl scores and AUCs in Figures a) and (b), respectively, AbAgIntPre outperformed the other three models, but there was still room for improvement. Furthermore, the performance of AbAgIntPre was not significant compared that of the simpler MLP. AVIDa-hIL6 differs significantly from the existing datasets used for training by AbAgIntPre and PIPR because it includes cases in which changes of few amino acids enhance inhibit antibody binding. Given this difference in properties, AbAgIntPre and PIPR may not have clear performance advantage over the MLP. Hence, these results indicate the need for research on model architectures that are dedicated to predicting antibody binding to antigen mutants, and AVIDa-hlL6 will be useful benchmark for evaluating such models.
Discussion
5.1 Binding Site Prediction
Antibodies recognize specific regions of antigens, called epitopes, and the regions of antibodies that are directly involved recognition are called paratopes. The antigen-antibon interaction is defined between the epitope and the paratope and is governed by van der Waals forces, electrostatic forces, hydrogen bonding, hydrophobic interactions, and entropic changes at the binding site, which comprise amino acids and their chemical modifications such as phosphorylation, glycosylation, lipidation, methylation, acetylation, or ubiquitylation. Because epitopes and paratopes are crucial for the affinity and specificity of predicting epitopes [42 45 and paratopes 227 [2]. AVIDa-hILó has highly sensitive information on changes of a few amino acids in both the antigen and antibody that can significantly affect binding, which should be strongly associated with epitopes and paratopes. Thus, AVIDa-hlI. may facilitate research predicting epitopes and paratopes from amino acid sequences.
antibody interactions. many studies have been devoted to

---

5.2 Potential Risk
In recent years, VHH technology has rapidly developed not only as a research and diagnostic tool but also therapeutic agent. VHHs are known to have low toxicity humans, and several VHH drugs have been approved to date [] Antibody genes are activated in B lymphocytes, and their complementarit-ddlerminnn
regions coding paratopes are the only genes which unrestricted mutations occur in vivo. Hence, antibodies that happen to bind to their own tissues can be produced. Mammals have an immune tolerance system that minimizes the production of such autoreactive antibodies through several mechanisms [9]. As our dataset was derived from alpacas, even if the risk of autoimmune adverse events low for alpacas, it may not be for humans. Therefore, a phase I clinical trial cannot be omitted for each clone for the time being.
5.3 Potential Data Biases
Close examination of our data revealed that the probability of the presence of binder VHHs varied dynamically depending on when and where the mother library was collected. Furthermore, because our dataset was produced in an alpaca's body, antibodies that strongly interact with proteins in an alpaca's 's body were likely to be excluded because they cause autoimmune disease. Therefore, our dataset potentially contains data biases derived from the timing and body site of the library collection and the specific alpaca used in the experiments. To reduce these data biases, it would be beneficial to collect samples at multiple times of immunization, from multiple individuals with different VHH gene sequences, and from multiple animals of different species.
5.4 Limitations and Future Works
We introduce two potential limitations of AVIDa-hlL6 and describe future works to address them. The first limitation is that AVIDa-hIL6 uses artificial mutations. Such mutations offer the advantage of investigating binding to an arbitrary number of mutants; however, natural mutations are more complex, as different sites mutate simultaneously. Furthermore, natural mutants include those that have similar functions and structures even with different amino acid sequences, and point mutations that cause loss or gain of the antigenic protein's 's function. Because simple artificial mutants have little chance of reproducing these properties, the efficiency of data collection for predicting aatigen-antiboon interactions of mutants with these properties is low. The second limitation is the lack of antigen diversity: specifically, AVIDa-hIL6 only has the IL-6 protein as antigen. Our experimental scenario is to predict antibody binding to unknown mutants of known antigen. In drug discovery applications, there is also need to find effective antibodies against new emerging antigens. These limitations lead to the narrow applicability of a model trained on AVIDa-hIL6.
An essential approach to overcome these limitations will be to accumulate labeled data for wider variety antigens and their mutants. Because our data generation method described section [. is applicable to any target antigen, it can be a fundamental technology for establishing more comprehensive database of antigen-antibody interactions. In fact, we used the same approach to generate a dataset for MARS-CoV-2 variants and successfully found effective antibodies [29]. In the future, we plan to generate and release datasets for various antigens, which should be more practical for building models to predict antigen-antibod, interactions. Furthermore, will explore combining AVIDa-hIL6 with other data sources, such as those listed in Table
Conclusion
In this paper, we have described AVIDa-hIL6, a large-scale dataset of IL-6 protein- pairs containing amino acid sequence information and reliable labels for binding or non-binding pairs. By introducing artificial mutations into the IL-6 protein used as an antigen, we generated an interaction dataset for 30 types of mutants in addition to wild-type IL-6. This design enabled AVI6o to include many sensitive cases in which point mutations in the IL-6 protein enhance or inhibit antibody binding, thus providing researchers with valuable insights into the effects of antigen mutations on antibody binding. We envision that AVIDa-hlI6 will help democratize antibody discovery and serve as a valuable benchmark for machine learning research in the growing field of predicting antigen-antibody interactions.
10

---

Acknowledgments and Disclosure of Funding
We thank Tomohisa Oda for developing the AVIDa-hIL6 website.
References
[1] Abanades, B., Georges, G., Bujotzek, A., Deane, C.M.: ABlooper: fast accurate antibody CDR loop structure prediction with accuracy estimation. Bioinformatics 38(7), 1877-1880 (2022)
[2] Al Ojaimi, Y.. Blin, T., Lamamy, J., Gracia, M., Pitiot, A., Denevault-Sabourin, C., Joubert, N., Pouget, J.P., Gouilleux-Gruart V., Heuzé- Vourc'h, N., et al.: Therapeutic antibodies-natura and pathological barriers and strategies to overcome them. Pharmacology & Therapeutics 233, 108022 (2022)
[3] Amimeur, T., Shaver, J.M., Ketchem, R.R., Taylor, J.A., Clark, R.H., Smith, J., Van Citters, D., Siska, C.C., Smidt, P., Sprague, M., et al.: Designing ceature--cntrollle humanoid antibody discovery libraries using generative adversarial networks. bioRxiv (2020)
[4] Arbabi-Ghahrout M.: Camelid single-domain antibodies: Promises and challenges lifesav- ing treatments. International journal of molecular sciences 23(9), 5009 (2022)
[5] Aronesty, E.: Comparison sequencing utility programs. The open bioinformatics journal 7(1)
(2013)
[6] Baek, M., DiMaio, F,, Anishchenko, I., Dauparas, J., Ovchinnikov, S., Lee, G.R., Wang, J., Cong, Q.. Kinch, L.N., Schaeffer, R.D., et al.: Accurate prediction of protein structures and interactions using three-track neural network. Science 373(6557), 871-876 (2021)
[7] Bazan, J., Calkosinski, I., Gamian, A.: Phage display powerful technique for immunotherapy:
1. introduction and potential of therapeutic applications. Human vaccines & immunotherapeuties 8(12), 1817-1828 (2012)
[8] Berman, H.M., Westbrook, J., Feng, Z.. Gilliland, G., Bhat, T.N., Weissig, H., Shindyalov, I.N., Bourne, P.E.: The Protein Data Bank. Nucleic acids research 28(1). 235-242 (2000)
[9] Bolger, A.M., Lohse, M., Usadel, B.: Trimmomatic: a flexible trimmer for illumina sequence data. Bioinformaties 30(15), 21 2114-2120 (2014)
[10] Chen, M., Ju, C.J.T., Zhou, G., Chen, X., Zhang, T., Chang, K.W., Zaniolo, C., Wang. W.: Multi- faceted protein-protein interaction prediction based on Siamese residual RCNN. Bioinformatics 35(14), i305-i314 (2019)
[11] Chen, Y.Z., Tang, Y.R., Sheng, Z.Y., Zhang, Z.: Prediction of mucin-type O-glycosylation sites in mammalian proteins using the composition of k-spaced amino acid pairs, BMC Bioinformat- ics 9(1,, 1-12 (2008)
[12] Chinery, L., Wahome, N., Moal, I., Deane, C.M.: Paragraph Paragraph-amiiodyy antibody paratope prediction using graph neural networks with minimal feature vectors. Bioinformatics 39(1), btac (2023)
[13] Cock, P.J., Antao, T., Chang, J.T., Chapman, B.A., Cox, C.J., Dalke, A., Friedberg, I., Hamel- ryck, T., Kauff, F., Wilezynski, B., et al.: Biopython: freely available Python tools for computa- tional molecular biology and bioinformatics. Bioinformaties 25(1 1), 1422-1423 (2009)
[14] Corrie, B.D., Marthandan, N., Zimonja, B., Jaglale, J., Zhou, Y., Barr, E.,, Knoetze, N., Breden, F.M., Christley, S., Scott, J.K., et al.: iReceptor: a platform for querying and analyzing antibody/B-cell and T-cell receptor repertoire data across federated repositories. Immunological reviews 284( 1), 24-41 (2018)
[15] Cunningham, B.C., Wells, J.A.: High-resolution epitope mapping of hGH-rcceptor interactions by alanine-ssanning mutagenesis. Science 244(4908), 1081-1085 (1989)
[16] Davydova, E.K.: Protein Engineering: Advances in phage display for basic science and medical research. Biochemistry (Moscow) 87(Suppl 1), $146-S167 (2022)
[17] Dunbar, J., Krawezyk, K., Leem, J., Baker, T., Fuchs, A., Georges, G., Shi, J., Deane, C.M.: SAbDab: the structural antibody database. Nucleie acids research 42(D1), D1140-D1146 (2014)
[18] Edgar, R.C.: Search and clustering orders of magnitude faster than BLAST. Bioinformatics 26(19), 2460-2461 (2010)

---

[19] Goodnow, C.C., Sprent, J. de St Groth, B.F., Vinuesa, C.G.: Cellular and genetic mechanisms of self tolerance and autoimmunity. Nature 435(7042), 590-597 (2005)
[20] Huang, Y., Zhang, Z., Zhou, Y.: AbAgImtPre: A deep learning method for predicting antibody-
antigen interactions based on sequence information. Frontiers in Immunology 13 (2022)
[21] Jin, B.k., Odongo, S., Radwanska, M., Magez, S.: Nanobodies: A review of generation, diagnostics and therapeutics, International Journal of Molecular Sciences 24(6), 5994 (2023)
[22] Jovčevska, I., Muyldermans, S.: The therapeutic potential of nanobodies. BioDrugs 34(1), 11-26 (2020)
[23] Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K.. Bates, R., Zidek, A., Potapenko, A., et al.: Highly accurate protein structure prediction with AlphaFold. Nature 596, 583-589 (2021)
[24] Kandari, D., Bhatnagar, R.: Antibody engineering and its therapeutic applications. International
Reviews of Immunology pp. 156-183 (2021)
[25] Kim, J. McFee, M., Fang. Q., Abdin, O., Kim, P.M.: Computational and artificial intelligence-
based methods for antibody development. Trends in Pharmacological Sciences 44(3), 175-189 (2023)
[26] Leem, J., Mitchell, L.S., Farmery, J.H., Barton, J., Galson, J.D.: Deciphering the language of antibodies using self-supervised learning. Patterns 3(7), 100513 (2022)
[27] Liberis. E., Veličković, P., Sormanni, P., Vendruscolo, M., Liò, P:: Parapred: antibody paratope
prediction using convolutional and recurrent neural networks. Bioinformatics 34(17), 2944- 2950 (2018)
[28] Lim, Y.W., Adler, A.S., Johnson, D.S.: Predicting antibody binders and generating synthetic antibodies using deep learning. mAbs 14(1), 2069075 (2022)
[29] Maeda, R., Fujita, J., Konishi, Y., Kazuma, Y., Yamazaki, H., Anzai, I., Watanabe, T., Yam- aguchi, K., Kasai, K., Nagata, K., et al.: A panel of nanobodies recognizing conserved hidden clefts of all SARS-CoV-2 spike variants including Omicron. Communications Biology 5, 669 (2022)
[30] Martin, M.: Cutadapt removes adapter sequences from high-throughpe sequencing reads. EMBnet. journal 17(1), 10-12 (2011)
[31] Olsen, T.H., Boyles, F., Deane, C.M.: Observed Antibody Space: A diverse database of cleaned, annotated. and translated unpaired and paired antibody sequences. Protein Science 31(1), 141-146 (2022)
[32] Olsen, T.H., Moal, L.H., Deane, C.M.: AbLang: an antibody language model for completing antibody sequences. Bioinformaties Advances 2(1), vbac046 (2022)
[33] Paszke, A., Gross, S., Massa, F,, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (2019)
[34] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: Machine learning in python. the Journal of machine Learning research 12, 2825-2830 (2011)
[35] Raybould, M.I., Kovaltsuk, A., Marks, C., Deane, C.M.: CoV-AbDab: the coronavirus antibody database. Bioinformatics 37(5), 734-735 (2021)
[36] Rice, P., Longden, I., Bleasby, A.: EMBOSS: the european molecular biology open software suite. Trends in genetics 16(6),
(2000)
[37] Ruffolo, J.A., Gray, J.J., Sulam, J.: Deciphering antibody affinity maturation with language models and weakly supervised learning. arXiv preprint arXiv:21 (2021)
[38] Ruffolo, J.A., Sulam, J., Gray, J.J.: Antibody structure prediction using interpretable deep learning. Patterns 3(2), 100406 (2022)
[39] Schneider, C., Buchanan, A., Taddese, B., Deane, C.M.: DLAB: deep learning methods for
structure-based virtual screening of antibodies. Bioinformatics 38(2), 377-383 (2022)
12

---

[40] Schneider, C., Raybould, M.I., Deane, C.M.; SAbDab in the age of biotherapeutics: updates including SAbDab-nano, the nanobody structure tracker. Nucleic acids research 50(D1), D1368- D1372 (2022)
[41] Shen, W., Le, S., Li, Y., Hu, F.: Seq cross-platform and ultrafast toolkit for FASTA/Q file manipulation. PloS 11(10), e01 63962 (2016)
[42] da Silva, B.M., Myung, Y., Ascher, D.B., Pires, D.E.: epitope3D: machine learning method for conformational B-cell epitope prediction. Briefings in Bioinformatics 23(1), bbab423 (2022)
[43] Smith, G.P.: Filamentous fusion phage: novel expression vectors that display cloned antigens on the virion surface. Science 228(4705), 1315-1317 (1985)
[44] Szklarczyk, D., Morris, J.H., Cook, H., Kuhn, M., Wyder, S., Simonovic, M., Santos, A., Doncheva, N.T., Roth, A., Bork, P., al.: The STRING database in 2017: quality-controlled association networks, made broadly accessible. Nucleic acids research 45(D1), D362-D368 (2017)
[45] Tubiana, J., Schneidman-Duueen Duhovny, D., Wolfson, HJJ.: ScanNet: an interpretable geometric deep learning model for structure- based protein binding site prediction. Nature Methods 19, 730-739 (2022)
[46] Wilman, W., Wróbel, S., Bielska, W., Deszynski, P., Dudzic, P., Jaszezyszyn, I., Kaniewski, J., Młokosiewicz, J., Rouyan, A., Satlawa, T., et al.: Machine-designe biotherapeutics: opportuni- ties, feasibility and advantages of deep learning in computational antibody discovery. Briefings in Bioinformatics 23(4), bbac267 (2022)
[47] Wilton, E.E., Opyr, M.P., Kailasam, S., Kothe, R.F. Wieden, HJJ.: sdAb-DB: the single domain antibody database. ACS Synthetic Biology 7(11),
(2018)
13

---

A Appendix
A.1 Ethics Statement for Animal Experiments
All animal experiments on an alpaca were conducted in accordance with the KYODOKEN Institute for Animal Science Research and Development (Kyoto, Japan) and the ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines Veterinarians performed breeding, health maintenance, and immunization by adhering to the published Guidelines for Proper Conduct of Animal Experiments by the Science Council of Japan. The KYODOKEN Institutional Animal Care and Use Committee approved the protocols for these studies (KYODOKEN protocol number 20190216).
Our data generation method uses animal models immunized with a target protein that could potentially harm the animals, such as particular toxin, pathogen, or allergen. Hence, the risk to the animal should be minimized by treating the immune source to inactivate or detoxify it.
A.2 Dataset Generation
Here, we describe the detailed experimental procedures and conditions for each step.
Step 1. Immunization We immunized a single alpaca with purified recombinant hu- man IL-6 protein and several single-amino-acit mutants. Specifically, the gene en- coding the human IL-6 protein was codon-optimized, synthesized, and sub-cloned in the PcDNA3.1(+) vector (Thermo Fisher Scientific K.K., Tokyo, Japan). The amino acid sequence of the wild-type IL-6 protein with a C-terminal 6x His-tat was WWW.SS/SAAASSSNALAAAAAAAAEEEEAAS LLDGUSSLLARTTTNNNSMCCSSSEEEEEEEA STG.://SSSESSSAANNAAAAAAAAAACCCC WWW.ATTLLLLLLNEEEETTTTTAAAAAAALA QAQNQWI DAQNOWLQQDMTHHI WWWTTTIIIIIIIIRI WIW.IIISSSSEEELLSSSS WESSSLRAARQMHHHHHIII mutation with alanine at intervals of three to six amino acids, like the alanine scanning technique (15). which is used in molecular biology to determine the contribution of a specific amino acid. A total of 30 singlc-amino-acid mutants was prepared: P42A, Q45A, T48A, ESIA D54A, 157A, I60A, G63A, K69A, C72A, C78A, S81A, E87A, L90A, P93A, D99A, F102A, G105A, E108A, T117A, L120A, L126A, L129A, S135A, E138A, Q144A, F153A, D162A, T165A, and D168A. Here, for example, P42A means that an amino acid in the wild type was substituted from proline to alanine at position 42. The antigen cocktail mixture was emulsified in Titermax (Funakoshi, Tokyo, Japan) adjuvant at a dose of 600 ng and subcutaneously injected into an alpaca four times at about two-week intervals. Lymph nodes and blood samples were each collected four times, resulting total of 12 libraries.
We introduced a site-directed
Step 2. Phage Library Construction Peripheral blood mononuclear cells (PBMCs) were obtained from blood samples by sucrose density gradient centrifugation using Ficoll (Nacalai Tesque, Ky- oto, Japan). The lymph nodes and PBMC samples were washed with phosphate-bufferts saline (PBS, Nacalai Tesque) and suspended in an RNAlater solution (Thermo Fisher Scientific K.K., Tokyo, Japan). Total RNA was isolated from these samples by using Direct-Zol RNA MiniPrep (Zymo Research, Irvine, CA). Complementary DNA was synthesized from μg of total RNA as template by using random hexamer primers and SuperScript II reverse transcriptase (Thermo Fisher Scientific K.K.) The coding regions of the VHH domain were amplified using LA Taq polymerase (TAKARA Bio Inc., Shiga, Japan) with two purrffed primers (CALL001, 5'- CECCCGGGGGCGCCTCCTCCGGGG
and CALL002, 5' -.GGGTCCGTGCGGGTGGCACCCCCCC 3'), and they were separated on 1.5 % low-melting meetting-temperaure agarose gel (Lonza Group AG, Basel, Switzerland). Approximately 700 base-pair bands were extracted using a QIAquick Gel Extraction Kit (Qiagen K.K., Tokyo, Japan). Nested PCR was performed to amplify the VHH genes by using two primers that contained flanking Pstl (forward) and BstEll (reverse) restriction sites to enable cloning into the pMES4 phagemid vector with a C-terminal His-tag. Ellecroporation-ccopeeenn Escherichia coli TG1 cells (Agilent Technologies Japan, Ltd., Tokyo, Japan) were transformed with the ligated plasmids under chilled conditions (Bio-Rad Laboratories, Inc., Hercules, CA). The library densities were monitored and maintained at >10" colony-forming units per microliter with limiting dilution. Colonies from 8 mL of cultured cells were harvested, pooled, and reserved in frozen glycerol stock as mother library. Thus, the 12 phagemid libraries were designated as the mother libraries.
2 ARRIVE guidelines; httpps://arrivvguudelliin org
14

---

Step 3. Affinity Selection One round of biopanning was performed using each target protein- coated magnet beads 50-mM phosphate buffer (pH 7.4) containing 0.1 % Triton X-100 (Nacalai Tesque), 0.3 % (w/v) bovine serum albumin (BSA, Nacalai Tesque), and 500 mM of NaCI. Every 11-6 mutant was used 1.2 ml. bead slurry, which was saturated with 240 jrg of protein, except for P93A (90 jlg), E108A (190 jrg), and L.16AA (180 μg). To distinguish nonspecific signals, a negative control sample that did not contain any IL-6 protein was also used. The wild-type IL-6 protein libraries were obtained in triplicate to confirm the reproducibility. After three washes with the same buffer, the remaining phages bound to the beads were eluted with pppsii-tttyllnediiminettttt
acid (EDTA, Nacalai Tesque) solution at room temperature for 30 minutes. The eluate was neutralized with a PBS-diiltedd protein inhibitor cocktail (cOmplete, EDTA-free, protease inhibitor cocktail tablets, Roche Diagnosties GmbH, Mannheim, Germany) and used to infect cettrooorraiinccoopetttoo cells. The infected cells were cultured in LB Miller broth containing 100 /g//ml. of ampicillin (Nacalai Tesque) at 37 "C overnight. The genes of the phagemids selected by biopanning were collected with a QIAprep Miniprep Kit (Qiagen), amplified by PCR, and purified using AMPure XP beads (Beckman Coulter, High Wycombe, UK). Then, dual-indexed libraries were prepared and sequenced on an Illumina MiSeq (Ilummina, San Diego CA) by using a MiSeq Reagent Kit v3 with paired-end 300-bp reads (Bioengineering Lab. Co., Lid., Kanagawa, Japan).
Step 4. Sequence Analysis Approximately 100,000 paired reads for each library were generated by NGS analysis. The raw read data were trimmed to remove the adaptor sequence by using cutadapt v1.18 [30] and remove low-quality reads by using Trimmomatic v0.39 (9] The remaining paired reads were merged using fastq-join [5], and then the VHH coding sequences were extracted using seqkit 10.10.1 [T. The DNA sequences were translated to amino acid sequences with EMBOSS v6.6.0.0 (36), and the VHH sequences were cropped from start to stop codon. Finally, each phagemid
library was converted to a FASTA file containing tens of thousands of VHH sequences.
A.3 Label Reliability
A.3.1
Experimental Procedures
The gene sequences encoding each selected VHH clone, which were con- nected with 4X(GGGGS linker for expression as a tandem dimer, were codon-optimized and synthesized (Eurofins Genomies Inc., Tokyo, Japan). The synthesized genes were subcloned into the pMES4 vector to express N-terminal PelB signal conjuat-e and C-terminal 6 His- tagged VHHs. BL21 (DE3) E. coli cells transformed with the plasmids were plated on LB agar with ampicillin and incubated at 37 "C overnight. Grown colonies were picked and cultured at 37 "C to reach an OD of 0.6 AU, and the cells were then cultured at 37 "C for three hours with 1 mM of IPTG (isopropyl-95 -eerppyl-----tiiee waw.thiogallatooyyannsssee
VHH Substantiation
Nacalai Tesque). Lastly, the cultured cells were pelleted by centrifugation and stored in a freezer until use. VHHs were eluted from the periplasm by soaking in TES buffer (200 mM Tris, 0.5 mM EDTA, 125 mM sucrose, and pH 8.0) at 4 "C for one hour. They were further incubated with a 2x volume of 0.25 diluted TES buffer with trace amount of benzonase nuclease (Merek) OC for 45 minutes. The supernatants were centrifuged (20,000 xg. 4 OC for 10 minutes), sterilized by adding gentamicin (Thermo), and passed through a 0.22 um filter (Sartorius AG, Gottingen, Germany). The filtered supernatants were then applied to a HisTrap HP nickel column (Cytiva) on an ÁKTA pure HPLC system, and the bound His-tagged VHHS were eluted with 300 mM of imidazole. The eluted fraction was collected and concentrated with VIVAspin 1000-molecute weeighlar-wwighh cutoff filter column (Sartorius) and applied to a Superdex75 10/300 GL gel-filtratin column (Cytiva) on an ÄKTA pure HPLC system. Finally, the protein purity was measured via Coomassie brilliant blue (CBB) staining (Rapid Stain CBB Kit, Nacalai Tesque).
Immunoftuoresscence Staining Analysis HЕК293Т cells were transiently transfected with a Dasmid-enccdding C-terminally HA-tagged wild-type IL-6 protein by using Lipofectamine 3000 (Thermo) according to the manufacturer's ' instructions. The next day, the cells were seeded on colla- gen type-I-coated culture plates (IWAKI, AGC TECHNO GLASS CO., LTD., Shizuoka, Japan) and cultured for 24 hours before being fixed with 2 % paraformaldehyde (PFA) at 4 "C overnight. After three washes with PBST (PBS with 0.005 % Tween 20), the cells were blocked with PBST containing 2 % goat serum (blocking solution) at room temperature for one hour. Each well was soaked with 100 ,ll of the blocking solution containing 100 ng of purified VHH at CC overnight. After washing with PBST, 1:3000-diluted anti-His-tag rabbit antibodics and I: 100-uut -nti-HA 7C9 mouse
15

---

Table 3: Amino acid sequences of the VHHs used for label verification.
VHII
Aminu and SeqAZZEE
UNU?
MKYLLFIAA WAA.AANVVOOOOSSCCCACOCCCOCO WOP.OEGGLOOOGGOOOOOOOOOOO SAALSE SESSYAMS WWR.RR.GGGGGGLLL EW SSSSSTSSS
FITYLDSY W XPEDTAY CYCAESRORY
MKYIIPAAMMEEI AM/ SUCANNN RESAASSIISNNN TROMPIICCCIII
ONLI14
hinder
NYA.WWKCH///SNNNNNNYYY LMMNN KPIDIAV' YCNAKRWUNDY waggETyyTYSSHHnn
ONUS4
hinder
GNDYEDS WEWTTTSSS...ETTLLLAAANNNO WANSLKPEE KAW.EAYYECECCCCCCCC BWYYCCVVSKYYCCCCC
HKYLLPTAAGELLLAAAPP WAW.LLLLAAAAAANOOOOOOOOOOOO MOW.///GGLLLEGGNNGGGOOOOOOOOOOOO OPCCSLRLSCCC VESSSFSSDY WWT.STTTSD/AASOOROOOOOOO W.KSRERRAAPKKKKAEESSSSSS SCISSSG
ONIMS
hinder
A.A.LL..AAMLLE LAAAAAAAAAAAAU MAAAANNNOEESSSAO DISSIO KI AAS.SSESNYAX, PROAIIIEE-HPPP SCG
STYYADSVKGRETSS TISRINAENTVYLL.CC WWW.NTIVYYLOONSSPEEEEE WWW.ONNSSEEEEEAAYYCCCCCCC AAAAEAAAAAAAAAAAAAAAAAA ENDPAYYYYEYEE YNDDEEY
ONUS
binder
PAAGGLAAAAAAPAAA WWW.M//OOOOEOOOOTOOOOOOOOOO AAA.RRLAAAASSSSSSAAAAAA WMCCSSSOAAAOKOOOOWAAAAAA SOGT
LAYGESVKGRE TEARDNADNT MONTTWWLQMMT WWW.ONLLLLEEEEOOO KPEPEWYYY TACYAS IN COGTTOVVSSSEREEEE
PIAAAAI AA AAGISMAAIVVO VOI QESCOX VOLDIES RI SEAASSUL I- PERYIIMAN AANEERRAEEEEEAAAAAAA SELS
hinder
TEDY QEDDEQEVVRRRFEEEE
SL WEE.GNNNNNLE WWW.TTAAYYCCAACCC SCAAASTTRALLLAA LAALIIGGYOO DY WCG.CTTVVVVSSSIE SISTEEIIE
ONU174
bisder
STYYADSY CEC.RISSDNAATTIV OONSSL AAP
AAYELPEAAGGLLL AAAAAAAAAAAOAAAAAAOAAA DESOOOGLLEEGGSSSOOOG WWW.TNGAAAPSCTIICCCCCCCC INT.GEEYYYOOOOOOOOOOO
ILAY AYADDAAERRRROOOO WWW.EEEEESSSSSSSIEREEEC
ONL119
MKYLL AAAAAAMO SI VONGUST
ACATAAAAAYYYAAAAA AYRSKYGCAREEE
AKYLLPTAAACUE AOAAMAAAOAOOOOAAOLLO OESOOGLOOAGGGLLT SCASSSCCO ANOI WRO.FOOORRLLOLAAAAAAOO
ONUZ20
IMYADSY KORF WERFFIISSDNANTTTT WELLEEEEETYLLLLLLIELL KSPEEEA TYCAAAAS
SADY WOOCOOV WWW.OTOTTTSHHHHBOOITOT ANG TIFTIN
ONL1455
hinder
SKVLIIPARAMI AUTT AAQPAMOONO DESOCCE VORGUSI
TNYANSLE KGRRPTIS PERFISSINN CERFIISENAANTT OLLNNSS KPEDTAY YCAAGIW DRRDDY POOOTTTTTSSSSEEIN
VIRGAIIC COREEEERIIIIVV VASIT
ONU1100
bisder
MKYLLPIAA AGELL LLLLLA ACCAMM AUJOU QUECOGEE NOPIEESI RI SUTTASGFFT LEDVAATU ROGIPUK WWU.IUIIVS ISSSSS
STYYLDSV ANGEETSSDNNNNEEE
YL SWNSSMNNSSEEEE SLNVEDTGGYYCC.CCO YYCAARRSS WAYSDY W WGRGGOOTT SSHHHERI
MKYLIPIAAK TI AACEA MAAOYO SEEGGE V VOIC KS SC AASSPPT IT AY YOR I FIRCIAA PUR VRPEE VING
ONU1301
binder
STY KKRFTISRRNNNEE
YI OANNSI KPEDIAIY Y ACATEHTT TITAAAA SATAAATTSSSII SISSSSSGGOD DYROOOOROOODOOO COVTTVVVVVIIIEIIII
ONU57
DOS binder
MKYLLPIAAKE AAAAAAAAAALLLLAAA LLLA ALA. ADV QUE GEGGGUEN SOP 7T ACA CAA ASCSIL DID DIDII YR QRE
TIYRES SIDEINNEE KORFTISRD FISRD YL FLONNNS XPED DAVY TC RECOAENY Y VNNDCORFREEE WFE.GGGGOOOO OOO.OEOGGOOOOOOOOOOOOOOO TVSSHHBEE
MEVIIIIAAAC A TTT SAMATVO SEGGRI VOP SSST DI SE A.A AAS IFTT IN YLAAW FRRAIN KESTEW VSCREESVI
GNUNI
ton-hinddrr
ATE SEWEEEKGRFFFOOOOE KNTV YI OMNNK KPEDTAVY YCAPS EE SCY TKU HD SRSCTTOTTVVVOST TVVTSSSHEEEE
ONUS2
biider
MKYLLPIANNLLI AACEAMM
SGGGLN UPOCSE BI SASGRMONIN OW ROAPLKORE VATTNNET RYCDDS SECRDIS KNSAW YL KPEDTAYYYCN ANRIOPEKIME MED COOTOOVVVSHEOEOOOOOO
TOn- non-binder
STRRGDDS
KNTV
KIEDDAVV ATTSPIYY
DEWCOOGTY wwTTTOTSSISIIIII
ONU17I
Dns binder
THVVDSVKGRRTT WEN.EEIANTTEEENNNNT YLOMS KPEDTA TYYYYCAAA CAAASSSGGYYHY CSCSSYY IRV WWK.TOVTVSSSSITIOOO
MKYLLPPRAK AAQ POY QL CESOUGIVE VUNCOSS
GDY PROAPURE
ONUZZZ
TOR- binde
MKYLIPIA II AACE AQ ORNGOE VORTEES RI SI TEANE TIPS
NAYYADSY WAWAADSSKKRRPISSSIS
QUNNSI KPED VYCATDRR
TIDOOSS COGTOM TYSHRRRIIIE
RR RIJIN
NSTIGD
WEYLLPPAAAGLLLL ROLAROO DESEGGEIV GEST SC KESGK TUSRY TROUPGRAREE REIVSSSSWING
ONU2
LTHYADEV SARAEEVRERRAIIIANAAAAAAN
YLOMMSS EEEEEAADY YCAAAY YDGISRLE PKVTTIN WOGOOOVTH SYSHEREEEE
UNUB
STYYADSV BERDDVKRRFTSSRDE PEETSRDDNAANNI
SKKYEL PTA AGI JJ J.
AA AQRAMAOO MF KOGT SQUGEESI RI SY SCHASSCTT STE
QMNS KIPESTAS AAAAAAAAAAA AEGIDPYHENY
SESSSE VEANE /T FR RRAAN KFRRR
FIRI GEYTHY WTTOCTOVTVSSHIEET
VAAISHENT VARIRRRAAAASSE
SKYLLPIS PIA VIGLEAAA LOPAANN MANNNN QU QUQESCCKE AELLY SUGUSS KL SLACCSGESS SLDDY Y YNTOW REEA DREOVSCISSSC
ONL4
B062
SINYADSY WERCRRRRRRRRRR RAFFFRRNNANTV IEEENAAKTVYLOONE TVYL VYLOONSSL SL KELKEEETAAY YYC V' EODGEGGCTAARRT WEW.LYYYYOOOTOOOOOOOOO WOO.TOVTV SSHEEEEE
ONUS
MKYLJ APAAAGU LI ALA.GAAM MA AQVQI QESGGGI VOACK SUST SE AAASSSSISSS TEAME W/ AOFRRQQA ROAAPK PRREERRRR VAT
NEYYADD PAADIVVIE GAR REFTIRODM DNNANNA QANNN FIED TAYY CATTI STTHEER AREV W WOGGGOOTY TW
ONU250
SSTNYADPY WWW.OOPPRRRRFESDNNNNNOE CETTTTYOMME LONNNNOONSLLEE
SKKYL PAALEEAAGEEE PAOPAN OLOESCCKE COSOUS AASGEELDYY YNIGW ROOAPOODEEEESSSSSO WWW.YYCCCOGOGCCEEACCCCOC LYDDY WWO.OOOOTTTTSSIIIII TYSHEIEEEEE
monoclonal antibodies (ChromoTek GmbH, -lateng-Martiisriee, Germany) in blocking buffer were added and reacted room temperature for one hour. Finally, after washing, plee---luor-comjugaaee anti-rabbit IgG (594 nm emission) antibodies at 1:3000 dilution and Alcxa-Fluor-comj mouse IgG (488 nm emission) antibodies at 1: 1:3000 dilution in blocking buffer were added to the wells, and the fixed cells were labeled at room temperature for one hour before washing three times with PBST. The cell nuclei were visualized with 4' -awaamiiiio---heylliiie cells were imaged with an 8-ms exposure time (594 nm emission), a 40- -ms exposure time (488 nm emission), or an automatically adjusted exposure time (DAPI) by using an IX7ISIF-3 microscope (Olympus Corporation, Tokyo, Japan) with the cellSens Standard 1.11 application (Olympus). Each full observed field corresponding to a 165 jum 220 jum square was photographed.
anti-
(DAPI). The stained
Kinetic Assays via Biolayer Interferometry (BLI) Real-time binding experiments were per- formed using an Octet Red96 instrument (forteBIO, Pall Life Science, Portsmouth, NH). Each purified VHH clone was biotinylated with EZ-Link Solfo--NS-LC-BBiie
(Thermo) according to the manufacturer's ' protocol: uncoupled biotin was excluded with a size exclusion spin column (PD Spin'rrap G-25, Cytiva) in PBS (pH 7.4). Assays were performed at 30 OC with shaking at 1000 rpm. Biotin-conjugateed clones at 10 /ug/ml. were captured on a streptavidin-coate fortèBIO) to reach the signals at nm. One unrelated VHH P17-coated sensor chip was monitored as a baseline. The loaded concentration of the wild-type IL-6 was 200 jug, corresponding to 0.625 J.M Assays were performed with PBS containing 0.005 % Tween 20 (Nacalai Tesque). After baseline equilibration for 180 s in the buffer, association and dissociation were each performed for 180 s. The data were then subtracted from the baseline data and analyzed with forteBIO data analysis software 9.0.
sensor chip (SA,
A.3.2 Additional Results
As listed in Table we selected 12 binder-labeled, six non-binder-labeled, and five noise-labeled clones for substantiation. Of the 12 binder-labeled clones, two could not be isolated by the E. coli
16

---

(a) Binder
ONU7
ONU14
HA
Merge
ONU54
ONU65
ONU88
ONU90
ONU191
ONU290
ONU455
ONU1160
(b) Non-binder
ONU57
ONU60
ONU84
ONU171
ONU232
(c) Noise
ONU2
ONU3
ONUS
ONU250
Figure 5: Immunoftuorescence staining analysis using (a) 10 bander-labeled, (b) six non- non-binder- labeled, and (c) four noise-labeled VHHs. HA-tagged IL-6 proteins were introduced into the cells and stained with His-tagged VHHSs Subsequently, HA-tags were visualized in green and His-tags in red.

---

(a) Binder
(b) Non-binder
Association Phase
Dissociation Phase
Association Phase
Dissociation Phase
URUU57
Resporsa Uni pam)
0.1
ONU54
ONU14
ONUT
Resporte Unit (rm)
0.1
ONU54
DRU117
ONUI0
ONUt0
OKU222
0.0
ONU80
0.0
-0.1
-0.1
ONU11100
0.22
-02
50
100
150
200
250
300
$50
50
100
150
200
250
300
350
(c) Nolse
Association Phase
Dissociation Phase
ONU2
Rospenze (nt
0.1
50
100
150
200
250
300
350
Time (6)
Figure 6: Biolayer interferometry analysis using (a) 10 binder-labeled, (b) six pon-binder-labeled and (c) four noise-labeled VHHs. Each line represents the association and dissociation curve of each VHH on the wild-type IL-6. When line moves from the base point in the association phase, the VHH is considered binder.
protein synthesis system because of the limitations of the phage display method. Even if protein of interest functioned as fusion protein with the g3p protein on a phage, it was not always possible to express the protein alone in soluble form with function (16). However, observed in the mother library, then the clones must have been truly present as heavy-chain antibodies, least in the alpaca body. The remaining 10 clones all showed binding to the wild-type IL-6 protein, as shown in Figure ().). The immunoftuorescencc staining analysis showed strong to weak signals, which probably reflected avidity differences between the clones. Note that the calculated p-values did not correlate with the staining intensity by simple inverse relationship. The biolayer interferometry (BLI) analysis revealed that the clones positively associated with the wild-type II.- protein with different association curves (Kon), dissociation curves (Kof f), and KDs (Kof as shown in Figure although the sensitivity was relatively lower than that of the immunostaining analysis. All six bon-binder-labeled clones showed negative results in both the immunostaining and BLI analyses, as shown Figures b) and [[bb, respectively. Of the five noise-labeled clones, one could not be isolated. The immunostaining analysis showed that the remaining four clones likely had nonspecific binding, as shown in Figure 5Jc), but not to the wild-type IL-6 protein, as confirmed by the BLI analysis results shown in Figure ()c) Accordingly, the sensitivity and specificity of our labeling method can be considered sufficiently high.
sufficient signal was
A.4 Benchmarks
A.4.1 Data Splitting
For set, we randomly selected 15 mutants: P42A, T48A, E5IA, 157A, I60A, K69A, C78A. S81A, E87A, L120A, L126A, L129A, Q144A, D162A, and T165A. The remaining 15 mutants and the wild type were used for model training. First, we trained the models by using only the wild-type 1L-6 protein and evaluated the model performance on the test set. Then, we randomly selected one mutant from the mutants not contained in the test set and added it to the training set to evaluate each model's 'ss predictive performance on the test set. By repeating this procedure, we tracked each model's predictive performance for unknown mutants contained only the test set. Because the order of adding mutants to the training set affected the model performance,
the same experiment five
18

---

times in shuffled order, and we report the averaged results in section 4. Table 4 summarizes the order in which mutants were added and the number of samples in each set.
A.4.2 Model Implementations
We adopted AbAgIntPre because it is oate-of-the art model designed for the same task setting as ours of predicting interactions solely from antigen and antibody sequences. At present, fewer studies have focused on developing machine learning models for predicting intigen-antibody interactions based only on amino acid sequences, as compared to PPI. However, antigen-antibody interactions that ignore non-protein antigens can be considered a subset of PPI, meaning that models designed for PPI are also applicable to antigen-antibody interactions. Thus, we adopted PIPR as a representative model designed for PPI. In addition, we used MLP as simpler, shallower neural network model than AbAgIntPre and PIPR. Lastly, we chose LR as a classical machine learning model other than neural networks. The implementations of all the benchmark models are available at https: //gttbub.../.../// com/ http///ggaa/////////////cooo.../ Cotiaano/AVDaa--. AVIG
AbAgIntPre (20]. We used the implementation that is provided by AbAgIntPre's 's devel- opers and released under Apache License 2.0 for the composition of k-spaced amino acid pairs (CKSAAP) [I]] encoding. The calculation was performed using k = 0, 1, 2, 3, thus yielding I 1600-dimensional vector for each amino acid sequence. We also used the original PyTorch [33 implementation released under Apache License 2.0 for the AbAgIntPre model. We used the model parameters reported in the original paper [20].
PIPR [10]. We reimplemented PIPR by using PyTorch with reference to the original amplemmentaioorrrlleesee under Apache License 2.0. We changed the number of RCNN units from five to three, while the other parameters were the same as in the original paper [10]. Each RCNN unit had a one-dimensional max pooling with a kernel size of three, which shortened the sequence length by a third. Our dataset's 's maximum sequence length is 218, and the application of five RCNN units would have resulted in a sequence length shorter than one; thus, we reduced the number of units. In addition, we used the pretrained embeddings published by PIPR's s developers in their original implementation as Wiec5_CTC..m
Multi-Layer Perceptron (MIP). We implemented one-hot encoding and an MLP with one hidden layer of 512 neurons and the rectified linear unit (ReLU) activation function by using PyTorch. The one-hot vectors of the VHHs and IL-6 proteins were flattened and concatenated for input to the MLP. We used zero padding to match the dimensions, thus yielding 8000-dimensional vectors for each VHH and IL-6 protein pair.
Logistic Regression (LR). We implemented LR by using scikit-learn (34. We used the default settings of scikit-learn v1.2.2 as model parameters. As with the MLP, the one-hot vectors of the VHHs and IL-6 proteins were flattened and concatenated for input to the LR model. We used zero padding to match the dimensions, thus yielding 8000-dimensional vectors for each VHH and II-66 protein pair.
A.4.3 Model Training
All three neural network-based models were trained on one NVIDIA Tesla V100 GPU on Google Colaboratory. The models were trained for 100 epochs with a batch size of 256. We used the Adam optimizer with fixed learning rate of 0.0001 and no weight decay. Also, we used the binary cross-entropy with sigmoid activation as the loss function. During the model training, the GPU memory consumptions for AbAgIntPre, PIPR, and MLP were approximately 1.4, 1.6, and 1.1 GB, respectively. When using 16 IL protein types for training, the training times for AbAgIntPre, PIPR, and MLP were approximately 0.5, 1, and 0.5 hours, respectively.
The LR model, with L2 penalization, was trained using one Intel(R) Xeon(R) CPU @ 2.20 GHz on Google Colaboratory. The solver was the LBFGS algorithm, with 1,000 as the maximum number of iterations for convergence. When using 16 II. protein types for training, the training time for LR was approximately 0.5 hours. More detailed training information is available at https: //gi //github.com//oeee /owpa//oooo/ooonao///aa------
3 AbAgIntPre: http://///ggthhh....m/////////rr PIPP: htttps://gee http:://gttttbtooooo http://twww..cmmmmmmccceeeeeeppp
19

---

Table 4: Details of the data splitting.
#Samples Non-binder
Experiment
#Antigen types
Added antigen
Binder
Total
Wild-type
540
12,480
13,020
G63A
1,074
39,629
40,703
F153A
2,232
49,940
52,172
T117A
2,967
64.508
67,475
Q45A
3,799
80.574
84,373
Run
E108A
4,424
99,729
104,153
C72A, F102A
5,503
148.208
153,711
10
193A, E138A
7.249
178,334
185,583
12
D54A, S135A
8,737
222,447
231,184
14
D168A, 1999
9,452
257,265
266,717
16
190A, G105A
10,564
282,279
292,843
Wild-type
540
12,480
13,020
2
E138A
1,942
22,052
23,994
C72A
2,406
47,106
49,512
L90A
2,656
59,150
61,806
F153A
3,814
69,461
73,275
Run 2
6
E108A
4,439
88,616
93,055
D99A, G105A
5,737
124,168
129,905
10
DI68A, D54A
6,509
165.155
171,664
12
G63A, P93A
7.387
212,858
220,245
14
Q45A, S135A
9,214
244,286
253,500
16
F102A, T117A
10,564
282,279
292,843
Wild-type
540
12,480
13,020
2
G63A
1,074
39,629
40,703
3
L90A
1,324
51,673
52,997
4
GI05A
2,186
64,643
66,829
F102A
2,801
88,068
90,869
Run 3
6
Q45A
3,633
104,134
107,767
8
D99A, D54A
4,562
155,467
160,029
10
E138A, E108A
6,589
184,194
190,783
12
$135A, C72A
8,048
224,610
232,658
14
P93A, DI68A
8,671
257.400
266,071
16
FIS3A, T117A
10,564
282,279
292,843
Wild-type
540
12,480
13,020
2
F102A
1,155
35,905
37,060
D99A
1,591
58,487
60.078
4
L90A
1,841
70.531
72,372
5
G105A
2,703
83,501
86,204
Run 4
6
DS4A
3,196
112,252
115,448
E138A, P93A
4,942
142,378
147,320
10
C72A, E1O8A
6,031
186,587
192,618
12
F153A, G63A
7,723
224,047
231,770
14
Q45A, T117A
9,290
254,681
263,971
16
D168A, S135A
10,564
282,279
292,843
Wild-type
540
12,480
13,020
2
D99A
976
35,062
36,038
3
E108A
1,601
54,217
55,818
4
F102A
2,216
77,642
79,858
5
G63A
2,750
104,791
107,541
Run 5
6
D168A
3,029
117,027
120,056
8
FI53A, Q45A
5,019
143,404
148,423
10
T117A, G105A
6,616
170,942
177,558
12
D54A, C72A
7,573
224,747
232,320
14
P93A, S135A
8,912
260,663
269,575
16
L90A, E138A
10,564
282,279
292,843
20


Reviews:
[{"review_id": "xV4htlYR3f", "reviewer": "Reviewer_E2ji", "summary": "", "questions": "", "limitations": "It would be great if the authors could incorporate more machine learning models into the experiment, especially the state-of-the-art deep learning methods, e.g., transformers. MLP is a little out-of-date in this area. ", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- This paper Introduces AVIDa-hIL6, the most extensive dataset for predicting antigen-antibody interactions, surpassing any other public dataset by tenfold. This dataset contains amino acid sequences of both antigens and antibodies, along with binary labels indicating binding and non-binding pairs. The dataset is of significance to the community. \n- AVIDa-hIL6 encompasses the wild type and 30 mutants of the IL-6 protein as antigens, offering numerous sensitive cases where point mutations in IL-6 either enhance or inhibit antibody binding. \n- The authors have devised a groundbreaking data generation method, including data labeling, by leveraging the immune system of a live alpaca. This innovative approach can be applied to any target antigen, in addition to IL-6.\n- The authors also present benchmark results for predicting antigen-antibody interactions using neural network-based baseline models. These results validate that AVIDa-hIL6 serves as an invaluable benchmark for assessing a model's capability to capture the impact of antigen mutations on antibody binding. \n\n\n\n\n\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "0T9bHy3ywr", "reviewer": "Reviewer_CS39", "summary": "", "questions": "", "limitations": "The authors have acknowledged several limitations of their work, which is commendable. However, there are opportunities to provide more comprehensive discussions on some of these limitations and their potential societal impacts:\nAddressing Imbalance in Labels: The strong imbalance in the dataset, with only about 3.7% of the data samples being binding pairs, impacts the performance of machine learning models. The authors could elaborate on the techniques used to handle this class imbalance, such as oversampling, undersampling, or using class weights, to prevent the models from being heavily biased toward the majority class.\n\n\nLack of Antigen Diversity: The dataset's narrow focus on a single protein (IL-6) as the antigen limits the applicability of the model to predict interactions with new emerging antigens. The authors could discuss potential strategies for increasing antigen diversity in future work, such as immunizing alpacas with multiple antigens or exploring other sources of data to incorporate a wider variety of antigens.\n\n\nConsideration of Natural Mutations: While the authors introduced artificial mutations for the IL-6 protein, they also acknowledged that natural mutations can be more complex, involving simultaneous mutations at different sites. It would be beneficial to discuss the potential impact of this limitation on the model's ability to predict interactions with naturally occurring mutations.\n\n\nEthical and Societal Implications: Given the potential use of the dataset and machine learning models for drug discovery and antibody development, it is essential to address any potential negative societal impacts. For example, the authors could discuss in more detail the importance of ensuring ethical considerations in the use of animal models and the potential risks associated with the development and use of therapeutic antibodies.\n\n\nPotential Bias in Data Generation: The method of generating the dataset using alpaca immunization may introduce biases in the dataset. The authors should acknowledge and discuss these potential biases and their implications on the generalizability of the model's predictions.\n\n\nAddressing Computational Challenges: As the dataset size is substantial, the authors could elaborate on the computational challenges faced during model training and how they addressed these challenges, such as using distributed computing or parallelization techniques. \nAdditionally, could some basic models like LRs, and trees be used for comparison?\n", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "\n1. The proposed technology presents a novel way of generating antibody-binding data of significantly increased volume, which could enhance the development of therapeutic antibodies. \n\n2. The dataset contains antibody sequences that are positive for some IL-6 mutants and negative for others, thus can provide important insights for understanding how antigen mutations affect antibody binding.\n\n3. Additionally, the authors developed a noise reduction algorithm using the negative control to avoid false positives and improve label reliability. \n\n4. The code and the data are truly easily accessible online, well documented, and clearly organized.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ZHnv7HoXKc", "reviewer": "Reviewer_Bn86", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The data size is much larger than before\n- The data is extracted from wet experiments", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Overall Assessment:
The benchmark paper introduces AVIDa-hIL6, a groundbreaking and expansive dataset for predicting antigen-antibody interactions in the VHHs (variable domain of heavy chain antibodies) using human interleukin-6 (IL-6) proteins as antigens in immunized alpacas. The dataset sets a new standard in the field, offering 573,891 labeled antigen-VHH pairs, 30 different IL-6 mutants, and novel data labeling methods. The paper is highly significant for its innovative approach and its potential to accelerate advancements in therapeutic antibody development.

Strengths:
Extensive Dataset: AVIDa-hIL6 is the most extensive dataset of its kind, exceeding existing public datasets by tenfold.

Novel Data Generation Method: The authors leveraged a live alpaca's immune system to create a groundbreaking data generation and labeling method, with broad applicability beyond IL-6.

Mutant Variability: The dataset includes 30 different mutants of the IL-6 protein, allowing nuanced understanding of the effects of mutations on antibody binding.

Benchmark Models: Benchmark results are provided using neural network-based models, validating the dataset as a valuable resource for the community.

Documentation and Accessibility: The code and dataset are well-documented and easily accessible online, making the research highly reproducible.

Opportunities for Improvement:
Machine Learning Models: The paper would benefit from incorporating more modern machine learning methods, especially transformer models, as the current MLP models are considered outdated.

Empirical Determinations: Clarity could be enhanced by providing explicit references for empirical values and decisions.

Benchmarking Details: More information about the selection criteria for baseline models would improve the paper’s interpretability.

Antigen Diversity: The dataset focuses only on IL-6, limiting its applicability to other antigens. Future work could aim to include more antigen types.

Limitations:
Class Imbalance: With only 3.7% of the data samples being binding pairs, the class imbalance could impact machine learning model performance.

Computational Challenges: Given the dataset’s size, elaboration on computational methods employed during model training would be valuable.
Decision: Accept (Poster)