Immunogenicity Prediction with Dual Attention Enables Vaccine Target Selection
Authors: Song Li
Yang Tan
Song Ke
Liang Hong
Bingxin Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: protein language model
protein representation learning
immunogenicity prediction
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose a protocol for deep learning-based immunogenicity prediction, covering dataset preparation, model construction, and evaluation through quantitative assessments and post-hoc analysis.

Abstract: Immunogenicity prediction is a central topic in reverse vaccinology for finding candidate vaccines that can trigger protective immune responses. Existing approaches typically rely on highly compressed features and simple model architectures, leading to limited prediction accuracy and poor generalizability. To address these challenges, we introduce VenusVaccine, a novel deep learning solution with a dual attention mechanism that integrates pre-trained latent vector representations of protein sequences and structures. We also compile the most comprehensive immunogenicity dataset to date, encompassing over 7000 antigen sequences, structures, and immunogenicity labels from bacteria, viruses, and tumors. Extensive experiments demonstrate that VenusVaccine outperforms existing methods across a wide range of evaluation metrics. Furthermore, we establish a post-hoc validation protocol to assess the practical significance of deep learning models in tackling vaccine design challenges. Our work provides an effective tool for vaccine design and sets valuable benchmarks for future research. The implementation is at \url{https://github.com/songleee/VenusVaccine}.

Published as a conference paper at ICLR 2025
IMMUNOGENICITY PREDICTION WITH DUAL ATTEN- TION ENABLES VACCINE TARGET SELECTION
Song Li!.3~ Yang Tan Song Ke
Liang Hong Bingxin Zhou'
1 SSchool of Physics and Astronomy, Shanghai Jiao Tong University.
2 School of Information Science and Engineering, East China University of Science and Technology.
3 Shanghai Matwings Technology Co., Ltd.
ABSTRACT
Immunogenicity prediction is a central topic in reverse vaccinology for finding candidate vaccines that can trigger protective immune responses. Existing ap- proaches typically rely on highly compressed features and simple model architec- tures, leading limited prediction accuracy and poor generalizability, To address these challenges, we introduce VENUS VACCINE, a novel deep learning solution with dual attention mechanism that integrates pre-trained latent vector represen- tations of protein sequences and structures. We also compile the most comprehen- sive immunogenicity dataset to date, encompassing over 7000 antigen sequences, structures, and immunogenicity labels from bacteria, virus, and tumor. Exten- sive experiments demonstrate that across wide range of evaluation metrics. Furthermore, establish a post-hoc validation protocol to assess the practical significance of deep learning models tackling vaccine design challenges. Our work provides effective tool for vac- cine design and sets valuable benchmarks for future research. The implementation is at hattps://github
outperforms existing methods
INTRODUCTION
Immunogenicity prediction is a pivotal step in reverse vaccinology. It aims at identifying specific proteins peptide segments (protective antigens) that can produce humoral and/or cell-mediated immune responses and lead to memory cells production in the host organism (Adu-Bobie et al. 2003 Doneva et al. 202T. Protective antigens pathogenic origin are potential vaceine candidates (PVCs) (Arnon 2011). Rapid and accurate prediction of protective antigens can reduce research and development costs, minimize vaccine development risks, and provide safe and effective prevention strategies against emerging and reemerging infectious diseases that pose ongoing threats (Pizza &
M. 2000 Bagnoli et al. 2011 Carvalho et al. 202T).
Current machine learning approaches commonly exploit the physicochemical properties of amino acid residues, as characterized by E-descriptors (Venkatarajan & Braun] 200T) or Z-descriptors (Hellberg et al. 1987), and transform these descriptors into uniform vector representation ing auto- and cross-covariance (ACC) methods. This vectorized format serves as input for machine learning models, which are then trained to predict whether antigens protective or non-protective. (Doytchinvva et al., 2007] Rawal et al. 2022). Due to the limited quantity of labeled data and the high complexity of the prediction problem, these methods excessively compress protein information in order to fit relatively simple model, making it difficult to capture the complex relationship be- tween antigens and their immunogenicity. As result, these models often exhibit limited prediction accuracy and inadequate generalizability (e.g., in cross-species immunogenicity predictions).
This study proposes VENUS deep learning method that interprets immunogenicity based on the multimodal encoding of antigens, including their sequences, structures, and picscceee cal properties. To supplement the inherent amino acid (AA) sequence information, two levels of structural tokens are employed to comprehensively capture antigen construction patterns, and the phsshcmclll properties are hand-crafted based on Z-descriptor and E-descriptor. When allo- cating protein representations from different modalities, we establish a dual-attentin mechanism
"Equal contribution first authors. f Corresponding author Saingxin.zhou@sstu.cee

---

Published as a conference paper at ICLR 2025
Piprade
Wariic Stractonn
Entedding
Five-Level Cruss Atlention
DDeersen Lewe Cruss Attention
Atlendion Poskin
Pryplca henncel
The model encodes sequence and structural Figure 1: Illustrative framework of mechanism, followed by aggregation layers to incorporate representations using a dual-attention global physicochemical attributes and perform binary classification of immunogenicity prediction.
(Figure I to facilitate communication between neighboring AAs at different scales. The of tokenized structures at the atomic level (Van Kempen et al. 2023 and peptide level (Hayes et al. 2024) aligns with two biological intuitions: protein binding affimity primarily depends on atomic- level interactions, and local protein structures determine their functionality. Subsequently, we apply attention pooling AA-level hidden representations to integrate them into protein-leve vector rep- resentations, which incorporate the hand-crafted features that are commonly used to characterize the immunogenicity of proteins to further enhance the model's capacity for immunogenicity prediction.
The reliability of deep learning models requires thorough empirical validation, such as statistically significant tests on high-quality datasets and comparisons with similar baseline methods. While wet-lab experiments are considered the most reliable validation due to the inherent noise of high- throughput labels, the significant time and financial costs make impractical for most research works to implement. For this purpose, we establish two types of evaluation protocols. First, we compile an immunogenicity database (ImmunoDB) with 7, 216 labeled antigens from bacteria, virus, and tumor. This benchmark enables large-scale validation of prediction methods and pro- vides valuable training data for developing new models. Secondly, we propose two post-hoc valida- tion protocols involving significant subjects (Helicobacter pylori and SARS-CoV-2), comprehensive evaluation datasets, and biologically meaningful evaluation criteria.
We verify the effectiveness of VENUS using the two established types of validation ap- proaches and provide extensive analysis. Based on ImmunoDB, we designed corresponding varia- tions of training and test datasets, and performed comprehensive comparisons with existing methods across various evaluation metrics, focusing on prediction accuracy and generalizability of the trained model. The results demonstrate significant superiority of the proposed VENUS from differ- ent perspectives. In post-hoc validations, we emphasize the practical significance of VENUSVAC- CINE by demonstrating that its assigned probability to protective antigens effectively helps users identify potential vaccine candidates for vaccine design.
In summary, this study addresses the key challenges of immunogenicity prediction in vaccine de- velopment by (i) introducing VENUS VACCINE, deep learning supervised learning scheme with dual attention mechanism; (ii) presenting the most comprehensive dataset of labeled antigens from different sources for training and testing immunogenicity prediction models; and (iii) constructing assessment protocols for evaluating prediction performance using benchmark datasets and post-hoc analyses. Empirically, challenges of vaccine target selection by providing accurate and robust predictions. Morcover, the correlate with antigenic epitopes, suggesting more potential
provide extensive evidence to validate that VENUSVACCING meets the
key AAs identified by of our model to assist in vaccine development. Our work not only offers a reliable tool for advancing vaccine design but also provides erucial resources and insights for further research.
IMMUNOGENICITY PREDICTION AND PROTEIN LANGUAGE MODELS
Problem Formulation Reverse vaccinology is an extensively utilized methodology for identifying
prospective vaccine candidates by computationally screening the proteome of a pathogen (Dalsass)
2

---

Published as a conference paper at ICLR 2025
NCBI
BLAST VaxiJen 2.0
Pub Med
IEDB
Protegen
UniProt
Turor
4F1
positive samples
process negative samples
construct Immuno
Figure 2: Data collection, redundancy processing, and dataset construction steps of ImmunoDB.
al. 2019.. This research is concerned with binary classification task aimed at predicting the immunogenicity of proteins from three pathogenic sources: bacteria, virus, and tumor. The pri- mary goal is develop model proficient in discerning whether a protein or peptide segment is protective antigen or non-protective antigen. Similar to most protein property prediction tasks, immunogenicity prediction also suffers from the limitation of scarce labeled data, making it crucial to develop powerful and generalizable model. To achieve this goal, feasible solution is to adopt a two-step training approach, where pre-trained models are first used to extract protein embeddings, followed by training the output module specifically for the prediction task.
Pre-trained Protein Language Model for AA-level Encoding Protein language models (PLMs)
have become the dominant method for learning representations of protein sequences (Zou et al.
2024a). The two primary pre-training strategies are masked language model (Meier et al. 2021 Rao
al. 202T) and autoregressive generation (Ferruz et al. 2022 Madani al. 2023). The former
involves predicting masked tokens in the input sequence while capturing co-evolutionary relation- ships between AAs within the same sequence. In contrast, the latter learns to iteratively generate the next token based on previous tokens, which is particularly useful for tasks like sequence design. As we focus on property prediction tasks, we adopt the first strategy. During training, random tokens in the input sequence are masked, and the objective is to optimize the model's parameters, denoted as 0, to minimize the error between the predicted and actual AAs at the masked positions, i.e.,
arg min Ex X EEM
Σ REM
log P(a.|a./M 0).
The conditional probability P( [//a) for the ith token I; in the sequence is determined by the un- masked tokens z //M- The model learns capture the interactions between AAs within the protein sequence. Notably, such sequential representation extraction method is not only applicable to AA sequence tokenization but can also extended capture protein structural features. A straightfor- ward approach to achieve this is by expanding the output tokens to include not only AA types but
also discretized structural tokens (Su al. 2023 Heinzinger et al. 2023 Li al. 2024
IMMUNODB: NEW BENCHMARK FOR TRAINING AND EVALUATION
We curate three datasets with immunogenic (positive) and non-immunogenic (negative) labels from bacterial, viral, and human tumor sources. All positive samples and a part of negative samples are compiled through an exhaustive literature search with manual validation. We check publicly available sources, including PubMed, UniProt, NCBI, Protegen, IEDB, and other publications up to October 2023. The search is strictly limited to proteins proven to be immunogenic in humans. For all three datasets, the preparation steps include collecting positive samples, collecting negative samples, and filtering redundant and tail-region samples. An illustrative workflow is visualized in Figure total, we have 913/15 1562 positive/negative instances in Immuno- -aatnni,,, 2078 1186 in Immuno- Mamuno-Virus, and 300/ 477 in
Positive Samples All positive samples are from the Protegen database and previous publications. For Immuno-Bacteria, Bacteria, data are from Dimitrov et al. (2020); Edison et al. y Rawal et al.
2022). For Immuno-Viree Virus, positive samples are gathered from Rawal et al. (2022): 0 Doneva & mmmuno-Tumor, Dimitrov 22024. For Immuno- Tumor, data soureed from Sotirov & Dimitrov (2044)
4) Notably,
for Immuno- Tumor, both positive and negative samples are collected in this step. Protein or peptide
sequences are from NCBI (Wheeler al. 2007) and UniProtKB (Consortium) 2019)
3

---

Published as a conference paper at ICLR 2025
Negative Samples Due the scarcity of data on non-immunogenic proteins in human studies,
part of the negative sets are collected from previous studies (Rawal et al. 2022; /otirov & Dimitrov
2024 Doneva & Dimitrov, 2024 Dimitrov et al. 2020 Li et al.) 2023; Edison et al. (020), and IEDB which the protein were validated by more than two B cell assays and identified as negative antigen in Homo sapiens, and others are curated following established methods reported in prior search (Dimitrov et al. (2020). For instance, when processing Humuno-Bacteed
we first download the proteomes of all of the bacteria involved from the UniprotKB, then conduct a BLAST search to identify sequences with less than 30% sequence identity to the protective antigens (positive sam- ples). To further improve the possibility that the selected sequences are true-negan negatiie samples, we scored these sequences with VAXIJEN and kept the sequence where it is predicted to be P Probable Non-Ant i gen. Finally, we exclude sequences shorter than 25 AAs or longer than 1024 AAs, and the remaining sequences are the negative samples for approach is applied to the preparation of the negative dataset for Immuno- Virus. For Immuno- Tumor, only perform the redundancy removal step, as the negative instances have already been provided by the literature and source database. Note that BLAST and VAXIJEN are used here to ensure that these selected sequences are true-negative as possible, but this approach cannot be directly extrapolated to predict the immunogenicity of antigens, because it will predict the vast ma- jority of sequences to be positive, which is a clear deviation from the empirical reality.
Baatnnnaaa A similar preprocessing
In summary, the new benchmark dataset for immunogenicity prediction is distinguished by its rig- orous quality control, comprehensive data sourcing, and cross-species diversity. ImmunoDB un- dergoes a series of rigorous quality control measures, which include data cleaning and preprocess- ing steps. It helps in minimizing missing information, erroneous entries, and inconsistencies in data structure, which are critical for reducing noise and enhancing reliability. Meanwhile, Immun- oDB provides the largest and most comprehensive benchmark datasets in terms of size and species. This richness in data enhances the benchmark's applicability for developing and testing immuno- genicity prediction models across various contexts. Moreover, including data from bacteria, viruses, and humans positions our dataset unique resource for cross spossss analysis, thus supporting potential contributions to the robustness and generalizability of future models trained on them.
VENUS MODEL TRAINING AND INFERENCE
4.1
INPUT FEATURES
Sequence Embedding and Structure Tokenization We use a pre-trained PLM to extract embed- dings for protein sequences and structures. The AA sequence embedding can be extracted from, an encoder-only model (e.g., ESM2: Lin et al. (2023) or an encoder-decoder model (e.g., ANKH: EE haggar et al. (2233) For instance, given a protein sequence of length L. ESM2-650M delivers an L L 1280 representation, with each AA being represented as a -280-dimensional vector. For protein structures, we use two tokenizers to encode the backbone predicted by ESMFOLD (Lin al. 2023
The first atomic structure tokenization uses FOLDSEEK (Van Kempen et al. 2023) to analyze the spatial relationships between atoms of neighboring AAs at a finer scale and summarize them into 20- dimensional tokens. The second peptide lokenization uses the ESM3 (Hayes et al., 2024 structure tokenizer to map coarser-scale substructures to 4, 096-dimensional structure tokens.
Physicochemical Descriptors We incorporate five hand-crafted E-descriptors (Venkatarajan & Braun 200T) and three Z-deseriptors (Hellberg et al., 1987) to quantitatively characterize protein
sequences. These descriptors provide an AA-level summary of physicochemical properties, such
as hydrophobicity and secondary structure. They have been validated to assist in immunogenicity
prediction (Doytchinova et al. 2007 Rawal et al. 2022) Details are in Appendix A
4.2 DUAL ATTENTION
A protein's 's properties are determined by its sequence and structure. For immunogenicity prediction, focusing on local topological structures and molecular interactions may help identify exposed epi- hand- crafted features with intrinsic sequence and structural information derived from pre-trained models, as described above. These different categories of protein features are integrated using a dual atten- tion mechanism, which guides the model to focus important local regions of the proteins. The
topes, thereby enhancing the extraction of critical information from the antigen. We combine
4

---

Published as a conference paper at ICLR 2025
features then passed through attention pooling and propagated to the prediction head to obtain the predicted labels. The overall architecture of the model is shown in Figure [ In the following sections, we detail the propagation rules of the dual attention and attention pooling modules.
Finer-lever and Coarser-level Attention Module We construct the dual attention mechanism with two components, including finer-level attention and coarser-level aftention. Both components construct a multi-head cross-attention framework to integrate keys and values from the sequence and queries from the structure. Following Lin et al. (222), we replace the relative positional encoding of queries and keys with Rotary Position Embeddings (RoPE) (Su et al., 2024). For a protein sequence of I. AAs, i.e,, x {I11, (01,:22..... ..... ,.,., we employ pre-trained language model (e-g., ESM2) to extract the sequence encoding Eseq E Rixdd Structural encodings are extracted by structure tokenizers of different scales. We define the finer-level structural encoding from FOLDSEEK as Eine E RLxd and the coarser-level structural encoding from ESM3 as Eccarse E R&Ldd Taking the finer-level attention as an example, the module's input is EEsyy Eise J. and the output is Hine- The queries Q, keys K, and values V for the attention layers are defined as follows:
Q = Efne Wo, K = Ese WK. Ese Wv.
(2)
where Wo: WK. Wv E Rdxdd are learnable projection matrices, and dg is the hidden dimension of queries and keys. Next, the positional information is attached to Q and K with RoPE, which captures relative positional relationships by applying a rotation to the embeddings based on their positions. The modified queries and keys are:
Q: Ro, K, RoPE(K,,)))))
where i and denote the positions in the respective sequences, and RoPE(-) is the RoPE function. Finally, the cross-attention Attu(-) is formulated as:
QK Th
Attn( Attn(Eseqy Eine) = softmax
V
(4)
where Q E Lxdd and K E RLxd2, The coarser-level attention applies the same construction. The only difference is that the input queries are replaced with Ecoarser from ESM3.
Attention Pooling The processed AA-level features H concat Esgq: Hine Hocarser: EO7) are compressed using attention pooling and fed into the classifier for label prediction. Here, H for a protein consists of four components: the original sequence embedding, the joint sequence and structure representations processed by coarser-level and finer-level attention, and the handcrafted E and Z descriptors. Consider a processed feature sequence H E RL is the sequence length, and D is the dimension of concatenated features. The attention pooling H is defined as:
AttnPool(H] (H)
aq H.,
where softmax((MelkecCCon11((((]
(5)
To obtain the final representation for the sequence, the model first computes L x 1 attention scores using a one-dimensional masked convolution with a kernel size of (Yang al. 2023). then applies an input mask and passes it through a softmax function to obtain the attention weights a. These attention weights not only guide the pooling process but also indicate the importance of each posi- tion for the final prediction. In subsequent analysis, we extract these weights as epitope markers. Relevant analysis can found in the post-hoc analysis section of the Experiments.
Prediction Head The output of the attention pooling AttnPool H) is sent directly to the final prediction head to produce the classification logits. This process involves projecting the pooled representation through a linear transformation, followed by dropout layer, a ReL.U activation layer, and final linear layer: Logits( H) = Linear (Ree..e (Linear (Linear(AttnPooll (H)))).
EXPERIMENTS
5.1 EXPERIMENTAL PROTOCOL
Setup We built and trained the model according to the framework introduced in Section For the sequence embedding, consider three pre-trained PLMs, including ESM2 (Lin et al. 2023
5

---

Published as a conference paper at ICLR 2025
Table 1: Average performance of baseline methods in three immunogenicity prediction tasks from 10 random splits. The complete results with standard deviation can be found in Appendices.
Model
Bacteria
Virus
Tumor
T30
MCC
ACC
MCC
KS
ACC
MCC
Random Forest
81.1
77.7
57.1
69.7
0.60
88.3
98.7
76.7
98.5
0.50
66.8
63.5
27.0
46.4
0.42
Gradient Boosting
75.9
56.8
71.2
0.62
86.0
72.3
86.5
0.74
65.4
53.1
0.38
XGBoost
80.8
76.1
57.0
71.1
0.61
89.1
98.8
78.2
89.2
0.80
69.2
62.2
33.7
56.1
0.40
SGD
78.6
72.4
51.0
65,3
0.56
77.3
88.0
56.0
74.0
0.66
52.6
50.9
29.6
62.2
0.31
Logistic Regression
65.2
67.3
1.8
0.5
0.47
61
82.9
35.6
71.9
0.61
60.4
45.2
MLP
78.1
71.6
51.5
68.1
0.57
85.0
90.6
70.5
85.7
0.71
60.4
39.1
0.0
0.0
0.26
SVM
57.3
18.9
53.0
0.33
61.6
88.7 7
25,1
67.3
0.53
51.4
57.3
KNN
80.4
73.8
55.3
68.6
0.52
87.4
86.8
74.8
87.3
0.75
57.4
49.6
10.8
45.3
0.12
Vaxi-DL
68.1
55.5
41.7
66.9
0.42
65.3
62.2
33.4
72.9
0.29
54,9
41.3
8.5
47.7
0.10
Vaxilen2.0
75.7
62.2
54.6
72.1
0.57
82.0
74.8
66.6
84.2
0.64
Vaxilen3.0
83.3
58.7
63.2
75.1
0.63
68.1
62.5
42.3
75.4
0.35
39.0
35.7
0.0
55.9
0.00
Viruslmma
0.17
VENUSVACCINE-Anlt
82.3
78.5
60.3
73.3
0.64
92.2
99.7
84.3
92.3
0.85
76.9
73.5
55.0
73.5
0.61
VANUNNACCIN--EMMM
80.6
77.0
57.0
71.7
0.66
90.3
99.5
80.6
90.6
0.82
74.0
61.7
46.1
68.2
0.58
VENUS VAccINE-PePt
84.5
84.5
65.9
77.0
0.66
91.4
97.4
82.8
91.2
0.84
71.5
68.7
44.7
67.6
0.54
top are highlighted by First, Second, Third.
Burtera
Vims
Tomor
0.5
AUC
0.6
0.6
0.5
0.g
Tomoe
Barteria
1.0
Vinus
1.0
Tumor
Virss
Recall
500
150
Kidh
200
Figure 3: Generalizability of VENUS by the AUC and Recall of cross-test evaluations. For instance, the top left figure reports the test performance on Immuno-Bacteo by different models trained on Immuno- Virus (yellow) and Immuno-Tune (orange).
Ankh-large (Elnaggar et al. 2023). and ProtBert (Elnaggar et al. 2021 The source are listed in Appendix The model is optimized using ADAMW @Loshchllvv et al. 2017) with learning rate of 0.0005 and a weight decay of 0.01. In the attention pooling layer, dropout was set to 0.1. To ensure stable GPU memory usage during training. we limit the maximum number of tokens per batch to 4000, with a gradient accumulation of 1. The maximum training epoch was set to 50, with early stopping based on validation accuracy and patience of 5. All implementations were done using Pyforch (version 1.7.0), and experiments were run on an NVIDIA @ RTX 3090 GPU with 24GB VRAM, mounted on Linux server. The training process tracked using WanDB. All source data and the implementation to reproduce the results will be publicly available upon acceptance.
Baseline Methods The performance of the proposed VENUS is compared with existing machine learning methods and web server prediction tools. We select commonly used machine learning methods from the literature, including Random Forest, Gradient Boosting, XGBoost, SGD, Logistic Regression, MLP, SVM, and KNN, and follow the usual feature engineering approach by using ACC-transformed datasets to classify proteins as protective antigens (positive class) or non- protective antigens (negative class). We also include four popular web server methods specialized in antigen and vaccine candidate prediction: VAXIJEN2.0 (Doytchinova et al. 2007b, VAXIJEN3.0 Dimitrov et al. 2020), VAXI-DL (Rawal et al.) 2022), and VIRUSIMMU (Li et al. 2023). VAXI- JEN2.0 employs an ACC transformation and PLS algorithm for antigen prediction. VAXIJEN3.0 ex- tends it to include machine learning for broader biological entities. VAXI-DL DL utilizes deep learning models tailored for different caasase-causing agents, and VIRUSIMMU applies an ensemble machine learning method with soft voting for viral immunogenicity prediction.

---

Published as a conference paper at ICLR 2025
5.2 RESULTS ANALYSIS ON IMMUNODB
We evaluate the overall performance of the models on VENUSVACCINE, In light of the problems with existing methods, we focus the accuracy and generalization of each method's predictive performance. The performance of the models is assessed using a range of metrics for comprehen- sive evaluation, including AUC-ROC, accuracy. precision, recall, F1 score, MCC, Top K accuracy, cross-entropy, and KS statistics. Details are provided in Appendix C
5.2.1 ACCURACY
We first investigate the overall performance of each model on the three datasets in ImmunoDB. For each dataset, we randomly split the data in a 7 1 2 ratio and select the model with the highest accuracy on the 10% validation set for evaluation. We repeat the assessment ten times for each test set, randomly selecting 50% of the test data to calculate the scores for each metric, and report the See Tables
average scores in Table 0
6 of the Appendix for additional details.
All machine learning and PLM methods retrained ensured non-overlapping among the training, validation, and test sets. However, the four web servers do not have open-source code available for retraining each task, which may introduce some unfair advantages to them. Additionally, during data preparation, we follow the convention in the field by using VAXIJEN3.0 0 to check negative data labels, giving this method natural advantage in predicting true negative data. Furthermore, since VAXI-DL is not available for predicting antigens from tumors, we did not test its performance on the Immuno-tumer dataset. Similarly, VIRUSIMMU is intended only for testing viruses, so we did not report its performance on Bacteria and Tumors for fairness consideration.
is clearly demonstrated by the performance comparison of baseline methods across different met- ries and datasets that VENUS has a significant advantage in prediction accuracy (indicated by top ACC). It achieves high accuracy for both positive and negative samples (indicated by high MCC and F1 scores) and effectively distinguishes the distribution differences between positive and negative samples (indicated by high KS scores). Notably, we specifically report the positive rate of top predictions (T30), i.e., the ratio of the top 30 samples with the highest predicted positive probabilities that are true positives. This metric is crucial in predictive tasks involving biological ex- periments since, in real-world scenarios, the samples are often highly imbalanced, and researchers usually conduct experimental validations on the samples with the highest model prediction confi- dence. Therefore, a model should provide high confidence for true positive samples. However, is also important to note that if model has a high T30 but considerably low accuracy, it may indicate that it occasionally predicts positive samples as negative, risking the omission of pivotal vaccine candidates. To avoid biased evaluations, we advocate assessing models based on their overall per- formance across multiple metrics rather than overly focusing on single measure.
We also investigate the influence of protein structure quality on model prediction with an ablation study comparing structures predicted by AlphaFold2 and ESMFold (see Table 9 performance metrics, detailed in the appendix, conclusively show that AlphaFold2 predicted struc- tures lead to superior model performance, underscoring the importance of structural accuracy. While ESMFold's 's rapid prediction capability is noteworthy, AlphaFold2's 's higher precision in structure pre- diction significantly boosts model performance. This implics that when time is not a critical factor, it is advisable to opt for the more accurate structures provided by AlphaFold2. Unfortunately, the searcity of available crystal structures limits our ability to contrast their performance with predicted ones, highlighting need for more high-quality structural data in future studies.
Appendix). The
5.2.2 GENERALIZABILITY
We next conduct cross-testing and compare the generalizability of different models (Figure S Here, the training and test sets are from different sources. The four web server-based methods are removed as we could not re-train them to avoid data leakage. We focus on AUC and recall to assess the model's 's overall learning ability and its capacity to predict true positives. Generally, models trained on the Virus dataset demonstrate higher generalizability compared to those trained the Bacteria dataset, and those trained on Tumor performed the worst. This may be related to the dataset size, as the number of antigens in Virus is larger than in Bacteria, and both are larger than Tumor.
7

---

Published as a conference paper at ICLR 2025
Venne Varcine
Random Forss
Gradival Booeling
XGBoost
2.0
1,0
1L.000
pobabbling
0,0175
0.5
0.5
0.5-
0.5
410150
0.0
0.0
5001
3000
1500
1500
800
1000
1200
500
1000
1500
SG3
Locistic Reprossion
MLP
KNN
0,003
gnbbbbiity
0.0030
0.5
0.3
0.5
0,0025
0.0
30C0
0.0
0.-
length 1500
0.0
0,,000
kmgth
kngth
Figure 4: KDE of predicted immunogenicity scores on Helicobacter pylori candidates. The experimentally determined immunogen are highlighted by red dots. Only VENUS simul- tancously identifies all positive samples while providing a reasonable overall distribution.
Moreover, excluding models with prediction issues (c.g., logistic regression, where a 100% re- call and relatively small AUC suggest it trivially predicts all samples negative), VENUSVAC- CINE consistently achieved top performance across the six test sets for both metrics. Also, the vari- ation in
performance the same test set when trained different datasets was relatively small, which further suggests the model's robustness in learning intrinsic patterns instead of overfitting local variations in the data, thereby demonstrating stronger generalizabilities.
5.3 POST-HOC ANALYSIS
This section demonstrates the practical significance of VENUS VACCINE addressing vaccine de- sign challenges, as a supplement of the quantitative comparisons on well-established datasets and standard measurements. Two significant research subjects are selected design corresponding ad- hoc analyses, including Helicobacter pylori and SARS-CoV-2. Specifically, in the first case, examine whether the potential vaccine candidates identified by VENUSVACCINE effectively cover immunogens, which is frequently assessed by the enrichment of the model's predictions. In the second case study, we evaluate whether the 'likely-efeective' sequences identified by VENUSVAC- CINE are consistent with experimental evidence, i.e., if the model assigns the top confidence the experimentally validated vaccine development sequences.
5.3. IDENTIFICATION OF POTENTIAL VACCINE CANDIDATES FOR HELICOBACTER PYLORI
In the first challenge, use VENUSVACCINE to predict immunogens within the Helicobacter py- lori proteome. Helicobacter pylori is a Group carcinogen that can persist in the gastric mucosa, causing various gastrointestinal diseases (Maltertheiner al. 2023). Identifying effective vaccine targets and developing corresponding vaccines can prevent Helicobacter pylori infections, thus sig- nificantly reducing the incidence of gastric diseases and lowering the rate of gastric cancer (Dos) Santos Viana et al. (021). To this end, we extract the proteome data from the UniProtKB database (ver 2024_04) (Consortium 2019). A total number of 858 sequences are extracted after redun- dancy removal. Although the majonit of these 1, 858 sequences lack immunogenicity ground truth labels, we referr to the work of Dalsass al. (2019) which record 11 of these sequences as pro- tective antigens. We use the model trained on Immuno-Bacted Bacteria for inference since Helicobacter pylori is bacterium. Note that there is no overlap between the training and prediction datasets.
When evaluate the prediction performance, two measurements are considered. (1) Recall (19) in Appendix measures the fraction of true protective antigens identified within the set of PVCs predicted by VACUSVA It is critical indicator of the model's ability accurately retrieve known protective antigens from larger set of candidates.
(2) Sold-Enrichmen (Dalsass et al, 15 in Appendix C measures the ratio of the true positive rate the overall prevalence of the condition, characterizing the model's ability to enrich for positive samples.
2019
To evaluate the performance of the model, compare the results of VENUSVACCINE with other machine learning approaches. Table [6 compares the recall and enrichment of baseline methods, and Figure visualizes the predicted kernel density estimation (KDE) of immunogenic probabilities
8

---

Published as a conference paper at ICLR 2025
for candidates of varying sequence lengths, with the 11 immunogens highlighted in red dots. We exclude the Vaxilen series as they only provide web servers, in which case we could not confirm the absence of data leakage issues in the training data. Among the 1, 858 candidates, VENUSVAC CINE identifies 123 PVCs. Notably, all the 11 true protective antigens are included within these 123 PVCs. A recall rate of 100% demonstrates VENUSVACCING protective antigens from the proteome data. In comparison, Random Forest, Gradient Boosting, and XGBoost also achieve 100% recall rate, while other baseline models do not perform as well. Fur- thermore, VENUSVACCING
exceptional ability to extract known
fold-enrichment is 15.11 11, while the highest cold-enrichment among Random Forest, Gradient Boosting, and XGBoost is only 8.81. This indicates that VENUS VANU CINE has stronger predictive ability to enrich for positive samples, which is particularly beneficial when experimental testing is limited to a finite number of samples.
Figure 0 demonstrates three observations: (1) The methods in the second row have trouble correctly labeling all the protective antigens. (2) The overall probability distribution found by VENUSVAG CINE aligns more closely with reality, with most samples clustering around probabilities close to 0, implying the majority of candidates are non-immunogenic. (3) VACCINE retains fewer PVCs from the candidates than baseline methods, allowing for the identification of the 11 protective antigens with fewer experimental efforts. The superior performance of VENUSVACCINE could be attributed to its effective integration of multimodal features by the dual-attention mechanism, which captures the complex interplay between sequence, structure, and physicochemical information. In summary, our model exhibits strong robustness and considerable potential to enhance the discovery of immunogenic vaccine targets in addressing the challenges posed by Helicobacter pylori.
5.3.2 UNRAVELING SARS-COV. PROTEOME FOR VACCINE TARGETING & EXPLANATION
In the second challenge, we aim to investigate if our VANUSVACCINE could effectively identify clinically val- idated vaceine targets from the SARS-CoVe The candidate test set consists of 38 sequences from the SARS-CoV-2 Data Hub (Wheleer et al. 2007). For the ground truth vaccine targets, we refer to the nine vac- cines approved by the World Health Organization (Sar-) avanan et al. 2024). five of which use the surface gly- coprotein (spike protein) of the novel coronavirus as the vaccine target, while the other four which also includes the spike protein. The overall predic- tion performance is in Table 17 in the Appendix, where VENUSVACCINE ranks the most important protein (spike protein) as the top among the 38 proteins. The model also identifies some new potential vaccine targets, e.g., non-structural protein 9 (Nsp9).
439-478
proteome.
150-164
291 291-320
the whole virus,
Figure 5: Epitope marker the surface glycoprotein (NCBI ID: YP-009724390.1) by the attention score identifies vaccine targets.
To further validate and understand our predictions, we visualize the prediction results. Figure 5 maps the model's 's attention scores to the antigenic epitopes of the sequences and compares them with experimentally obtained epitopes (Shrock al., 2020 Polyiam al. 2021; Braun et al. 2024). shows clearly that the model not only demonstrates excellent predictive performance but also possesses promising interpretability. For clear presentation, we display the chain trace of the spike protein and color the amino acids based on the attention scores from the attention pooling layer, with darker positions indicating higher attention scores. Overall, the model shows higher attention scores around position 400, and the spike protein contains three immunodominant linear B cell epitopes from position 319 to 541. Additionally, there three regions with deeper colors that overlap with the SARS-Co spike receptor- receptor-binding domain (RBD), including positions 150-164, 291-320 and 439-478. This visualization provides deeper understanding of the model's prediction process and enhances the credibility of our approach in identifying potential vaccine targets.
RELATED WORK
Developing machine learning models has become a popular approach for many predictive problems in vaccine design. One of the most widely used web servers for identifying potential vaccine can-
9

---

Published as a conference paper at ICLR 2025
didates is VAXIJEN (Doytchinova et al. 2007), which employs an alignment- free method based on auto cross-covariance (ACC) transformation to identify antigens without relying on sequence align- ment. Subsequent methods have introduced additional imput features (He et al. 2010) or improved prediction models (Edison et al. 2020; Zhang et al., 2023] Rawal et al. 2022). Compared to other reverse vaccinology tools, these advanced techniques demonstrate superior performance in predict- ing bacterial protective antigens. Recently, Dimitrov et al. explored the immunogenicity of antigens from other sources, such as viruses (Doneva & Dimitrov 2024) and tumors (Sotirov & Dimitrov 2024. and evaluated various popular machine learning methods.
These machine learning approaches for predicting immunogenicity have often relied on limited datasets, typically confined to a specific type of pathogen, such as bacteria or viruses, and have been relatively small in scale. For instance, Doytchinova et al. (2007) trained and validated their models using only few hundred samples. The scarcity of data not only impedes the development of robust predictive models but also restricts the ability to capture the complex immunological responses elicited by diverse antigens. To date, few studies have managed to amalgamate comprehensive dataset that encompasses antigens from multiple pathogen classes, which is essential for training models capable of predicting immunogenicity with high fidelity across varied biological contexts.
Despite the rapid development of deep learning, the advancement of methodologies for protein prop- erty prediction, including immunogenicity prediction, has been relatively slow due to the limited amount of labeled data. Recently, BERT-style PL.Ms (Lin et al. 2023: Li et al. 2024) have gained increasing attention with an increasing number of prediction tasks fine tne-tuning models to better fit the specific targets (Schmirler et al. 2023: Tan et al., 2024bb] 2025a), or attaching additional propagators
to process sequence embeddings, such as geometric encoders (Yang al. 2023
Tan al. 20256
Zhou et al. 2023 2024b) or cross-modality aggregators (Tan et al. 2024a C).
7 CONCLUSION AND DISCUSSION
Despite the rapid advancement of deep learning in providing novel solutions to numerous problems in scientific fields such as protein engineering, the application of advanced models remains largely limited to well-defined engineering problems with relatively mature and abundant data. However, many other issues with significant scientific and practical value have not received sufficient attention, and the methods used for these problems are considerably outdated. In addition to the oversight of important problems due to lack of interdisciplinary cross-talk between research fields, another ma- jor reason is the absence of high-quality datasets and adequate evaluation methods for many specific issues. Preparing and processing datasets for model training and evaluation requires significant ef- fort and domain knowledge, and validating the developed methods also requires feasible evaluation protocols that fit practical requirements. Ideally, these protocols should neither be disconnected from practical needs as current computational metrics (e.g., perplexity for sequence generation) nor as resource-intensive as wet lab validation processes in terms of time, money, and technical barriers.
As an exploratory work, this study takes immunogenicity prediction in vaccine development as an entry point to investigate how prepare data, build models, and design evaluation strategies for scientific problem that seeks powerful deep learning solutions. Although protein representation can be encoded by many pre-trained models, and immunogenicity prediction is inherently one super- vised learning task, we emphasize that utilizing deep learning to solve scientific problems should not only focus on algorithm design but also on dataset construction and the practicality of evalua- tion methods. Here, demonstrate the overall performance of the model using standard datasets and conventional quantitative evaluation methods, and analyze the model's reliability in specific applications using post-hoc analysis without conducting entirely new wet lab experiments.
ACKNOWLEDGMENTS
This work was supported by the grants from the National Natural Science Foundation of China (Grant Number 62302291; 12104295), the Computational Biology Key Program of Shanghai Sci- ence and Technology Commission (23JS1400600) Shanghai Jiao Tong University Scientific and Technological Innovation Funds (2 (21X010200843), the Student Innovation Center at Shanghai Jiao Tong University, and Shanghai Artificial Intelligence Laboratory.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jeannette Adu-Bobie, Barbara Capecchi, Davide Serruto. Rino Rappuoli, and Mariagrazia Pizza. Two years into reverse vaccinology. Vaccine, 2107-8):605-610, 2003.
Ruth Arnon. Overview of vaceine strategies. In Vaccine Design. Innovative Approaches and Novel Strategies; Rappuoli, R., Ed.; Caister Academic Press: Norfolk, UK, 2011.
F. Bagnoli, N. Norais, I. Ferlenghi, M. Scarselli, and R. Rappuoli. Designing vaccines in the era of genomics. Vaccine Design: Innovative Approaches and Novel Strategies, 2011.
Asolina Braun, Louise C Rowntree, Ziyi Huang, Kirti Pandey, Nikolas Thuesen, Chen Li, Jan Pe-
tersen, Dene R Littler, Shabana Raji, Thi HO Nguyen, al. Mapping the immunopeptidome of seven sars-cov-2 antigens across common hla haplotypes. Nature Communications, 15(1):7547, 2024.
Thiago Carvalho, Florian Krammer, and Akiko Iwasaki. The first months of covid-19: timeline of immunological insights. Nature reviews. Immunology, pp. 1-12, 2021.
Uni Prot Consortium, Uniprot: worldwide hub of protein knowledge. Nucleic Acids Research, (1D::500--D115. 2019.
Mattia Dalsass, Alessandro Brozzi, Duccio Medini, and Rino Rappuoli. Comparison of open-source reverse vaccinology programs for bacterial vaccine antigen discovery. Frontiers in Immunology, 10, 2019.
Ivan Dimitrov, Nevena Zaharieva, and Irini Doytchinova. Bacterial immunogenicity prediction by machine learning methods. Vaccines, 8(4):709, 2020.
Nikolet Doneva and Ivan Dimitrov. Viral immunogenicity prediction by machine learning methods. International Journal of Molecular Sciences, 25(5), 2024.
Nikolet Doneva, Irini Doytchinova, and Ivan Dimitrov. Predicting immunogenicity risk in biophar-
maceuticals. Symmetry, 13(3), 2021.
Irineu Dos Santos Viana, Maria Luisa Cordeiro Santos, Hanna Santos Marques, Vinicius Lima de Souza Goncalves, Breno Bittencourt de Brito, Filipe Antonio Franca da Silva, Natalia Oliveira Silva, Fillipe Dantas Pinheiro, Adriano Fernandes Teixeira, Davi Tanajura Costa, et al. Vaccine development against helicobacter pylori: From ideal antigens to the current landscape. Expert Review of Vaccines, 20(8):989-999, 2021.
Irini A Doytchinova, Darren Flower, Irini A Doytchinova, and Darren R Flower. Vaxijen: server for prediction of protective antigens, tumour antigens and subunit vaccines. BMC Bioinformatics, 2007.
Ong Edison, Wang Haihe, Mei U Wong, Seetharaman Meenakshi, Valdez Ninotchka, and He Yongqun. Vaxign-ml: Supervised machine learning reverse vaccinology model for improved prediction of bacterial protective antigens. Bioinformatics, (10):10, 2020.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Toward un- derstanding the language of life through self-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):7112 7127, 2021.
Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Char- lotte Rochereau, and Burkhard Rost. Ankh: Optimized protein language model unlocks general- purpose modelling. w.Xiv:2301.06566 2023.
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. ProtGPT2 is deep unsupervised language model for protein design. Nature Communications, 13(1):4348, 2022.
Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with language model. bioRxiv, pp. 2024-07, 2024.

---

Published as a conference paper at ICLR 2025
Yongqun He, Zuoshuang Xiang, and Harry L Mobley. Vaxign: The first web-based vaccine design program for reverse vaccinology and applications for vaccine development. Biomed Research International, 2010 2010.
Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Martin Steinegger, and Burkhard Rost. ProstT3: Bilingual language model for protein sequence and structure. bioRxiv, pp. 2023-07, 2023.
Sven Hellberg, Michael Sjoestroem, Bert Skagerberg, and Svante Wold. Peptide quantitative structure-activity relationships, multivariate approach. Journal of Medicinal Chemistry, 30(7): 1126-1135, 1987.
Jing Li, Zhongpeng Zhao, Chengzheng Tai, Ting Sun, Lingyun Tan, Xinyu Li, Wei He, Hongjun Li, and Jing Zhang. Virusimmu: a novel ensemble machine learning approach for viral immuno- genicity prediction. bioRxiv, pp. 2023-11, 2023.
Mingchen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Huiqun Yu, Ziyi Zhou, Wanli Ouyang, Bingxin Zhou, Pan Tan, and Liang Hong. ProSST: Protein language modeling with quantized structure and disentangled attention. Advances in Neural Information Processing Systems, 37:35700-35726, 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, (19966637:1123-1111 2023.
Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam, arXiv:1711.05100 5, 2017.
Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology. 1099) 1106, 2023.
Peter Malfertheiner, M Constanza Camargo, Emad El-Omar, Jyh-Ming Liou, Richard Peek, Chris- tian Schulz, Stella I Smith, and Sebastian Suerbaum. Helicobacter pylori infection. Nature Re- views Disease Primers, 91):19 2023.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems, 34:29287-29303, 2021.
Pizza and M. Identification of vaccine candidates against serogroup b meningococeus by whole- genome sequencing. Science, 18765459) 2000.
Kanokporn Polyiam, Waranyoo Phoolcharoen, Namphueng Butkhot, Chanya Srisaowakarn, Arunce Thitithanyanont, Prasert Auewarakul, Tawatchai Hoonsuwan, Marasri Ruengiithaatchwwaly Phenjun Mekvichitsaeng, and Yaowaluck Maprang Roshorm. Immunodominant linear cell epitopes in the spike and membrane proteins of sars-cov-2 identified by immunoinformatics pre- diction and immunoassay. Scientific Reports, 11(1):20383, 2021.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Picter Abbeel, Tom Sercu, and Alexander Rives. MSA transformer. In International Conference on Machine Learning, pp- 8856. PMLR, 2021.
Kamal Rawal, Robin Sinha, Swarsat Kaushik Nath, Preeti, Priya Kumari, Srijanee Gupta, Trapti Sharma, Ulrich Strych, Peter Hotez, and Maria Elena Bottazzi. Vaxi-dl: A web-based deep learning server to identify potential vaccine candidates. Computers in Biology and Medicine, 145:105401, 2022.
Venkatesan Saravanan, Bharath Kumar Chagaleti, and Pavithra Lakshmi Narayanan Vijay Babu AnandamHaritha ManoharanG.N V. AnjanaRamalingam PeramanS. Karthik Raja NamasivayamM. KavisriJesu ArockiarajKathiran velopment of covid-19 vaccine from laboratory to clinic. Chemical Biology and Drug Design, 103(1), 2024.
Moovendhan. Discovery and de-
12

---

Published as a conference paper at ICLR 2025
Robert Schmirler, Michael Heinzinger, and Burkhard Rost. Fine- Fine-tuning protein language models boosts predictions across diverse tasks. bioRxiv, pp. 2023-12, 2023.
Ellen Shrock, Eric Fujimura, Tomasz Kula, Richard Timms, and Stephen J. Elledge. Viral epitope profiling of covid-19 patients reveals cross-reactivity and correlates of severity. Science, 2020.
Stanislav Sotirov and Ivan Dimitrov. Application of machine learning algorithms for prediction of tumor t-cell immunogens. Applied Sciences, 14(10), 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 1670637 2024.
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: protein language modeling with structure- vocabulary. In The Twelfth International Conference Learning Representations, 2023.
Yang Tan, Mingchen Li, Bingxin Zhou, Bozitao Zhong, Lirong Zheng, Pan Tan, Ziyi Zhou, Huiqun Yu, Guisheng Fan, and Liang Hong. Simple, efficient, and scalable structure-aware adapter boosts protein language models. Journal of Chemical Information and Modeling, 2024a.
Yang Tan, Ruilin Wang, Banghao Wu, Liang Hong, and Bingxin Zhou. Retrieval-enhanced mutation mastery: Augmenting zero-shot prediction of protein language model. arXiv:2410.21127, 2024b. 2024b.
Yang Tan, Jia Zheng. Liang Hong. and Bingxin Zhou. ProtSolM: Protein solubility prediction with multi-modal features. In 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 223 232. IEEE, 2024c.
Yang Tan, Chen Liu, Jingyuan Gao, Banghao Wu, Mingchen Li, Ruilin Wang, Lingrong Zhang, Huiqun Yu, Guisheng Fan, Liang Hong, et al. VenusFactory: A unified platform for protein engineering data retrieval and language model fine-tuning. arXiv:2503. 15438, 2025a.
Yang Tan, Bingxin Zhou, Lirong Zheng, Guisheng Fan, and Liang Hong. Semantical and geo- metrical protein encoding toward enhanced bioactivity and thermostabillity elife, 13:RP98033, 2025b.
Michel Van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein struc- ture search with foldseck. Nature Biotechnology. pp. 1-4, 2023.
Mathura S. Venkatarajan and Werner Braun. New quantitative descriptors of amino acids based on multidimensional scaling of a large number of physical-chemical properties. Molecular Modeling Annual, 7122):45-453, 2001.
David L Wheeler, Tanya Barrett, Dennis Benson, Stephen Bryant, Kathi Canese, Vyacheslav Chetvernin, Deanna M Church, Michael Dicuccio, Ron Edgar, and Scott Federhen. Wheeler, d.I. et database resources of the national center for biotechnology information. nucleic acids res. 33 database Issue: D39-d45. Nucleic Acids Research, 35(ttabbese Dssue):DS-12, 2007.
Kevin K Yang, Niccolò Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation learning. Protein Engineering, Design and Selection, 36:gzad015, 2023.
Yuhan Zhang, Anthony Huffman, Justin Johnson, and Yongqun He. Vaxign-DL: deep learning- based method for vaccine design and its evaluation. bioRxiv, 2023.
Bingxin Zhou, Lirong Zheng, Banghao Wu, Yang Tan, Outongyi Lv, Kai Yi, Guisheng Fan, and Liang Hong. Protein engineering with lightweight graph denoising neural networks. Journal of Chemical Information and Modeling. 2023.
Bingxin Zhou, Yang Tan, Yutong Hu, Lirong Zheng, Bozitao Zhong, and Liang Hong. engineering in the deep learning era. mLife, 3(4))477-491, 2024a.
Protein
Bingxin Zhou, Lirong Zheng, Banghao Wu, Kai Yi. Bozitao Zhong, Yang Tan, Qian Liu, Pietro Liò, and Liang Hong. A conditional protein diffusion model generates artificial programmable endonuclease sequences with enhanced activity. Cell Discovery, 10(1):95, 2024h.
13


Reviews:
[{"review_id": "FC7EFi9odq", "reviewer": "Reviewer_Xkx7", "summary": "The authors present the ProVaccine model for immunogenicity prediction, as well as the Immuno datasets of antigens from bacteria, viruses and tumors, which is balanced across protective and nonprotective antigens. They show that ProVaccine is more accurate compared with both baseline ML and hosted models.", "questions": "- I don't understand the approach to measuring accuracy described in the first paragraph of section 5.2.1. Are you doing hyperparameter optimization on the validation set for each split? Why randomly select \"50% of the test data to calculate the scores for each metric\" since you already have a validation set in your split?\n- It's not clear what the generalizability of section 5.2.2 is measuring. Should a immunogenicity predictor generalize across bacteria, viruses and humans?\n- For the three conclusions about ProVaccine in the final paragraph of subsection 5.3.1, I don't understand how the authors concluded the second and third - could you please explain these in greater detail?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- novel model architecture\n- contribution of the Immuno datasets\n- ProVaccine achieves mostly top performance as reported in Table 1", "weaknesses": "- It's not obvious what the contribution of any of the representations (sequence, fine, coarse, and descriptors) is to model performance. What if only the sequence representation is used as input? Doing ablation studies is essential for understanding these contributions, and should be included in this work.\n- Figure 4 is hard to interpret, it would be easier to understand if instead the likelihood stats of the 11 determined immunogens under each model were reported.\n- The appendix is missing:\n  - page 7 in section 5.2.1 refers to Tables 4-6 in the appendix\n  - page 8 in section 5.3.1 refers to recall and fold-enrichment metrics that do not appear in any table. Also reference is made to Table 10, which does not appear to exist.\n  - page 9 in section 5.3.2 refers to Table 11 in the appendix, which does not exist", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bzjuV6BBAW", "reviewer": "Reviewer_G2xM", "summary": "The paper presents PROVACCINE, a deep learning model designed to predict the immunogenicity of proteins, which is a crucial task in vaccine development. The authors introduce a dual-attention mechanism within the model, enabling it to process multimodal representations of protein sequences, structures, and physicochemical properties. They also curate Immuno, an extensive dataset of over 9,500 antigen sequences and structures from various sources, to facilitate robust training and validation. The experimental results show that PROVACCINE outperforms traditional methods across multiple evaluation metrics, with specific validation studies on Helicobacter pylori and SARS-CoV-2. The post-hoc analyses further confirm PROVACCINE’s efficacy in identifying potential vaccine targets.", "questions": "- Was there any homology cut-off applied between the training and testing datasets for each class of the antigens? High sequence similarity b/w training and testing might over-inflate performance.\n\n- Most of  the figures have a generic caption. The captions should explain the figures clearly and the results being shown in the figure (especially the figures showing the results and comparison). The results from the figures should also be discussed in the main text in detail. \n\n- In the post-hoc analysis, it it possible to check how well will the model perform for non-immunogenic antigens in Helicobacter pylori (Fig 4). The sequences of the 11- experimentally verified immunogens can be randomized and then it can be checked where does the prediction for the randomized-sequence lie in terms of probability. The authors also need to describe how 'the probability of identifying known protective antigens through random guessing' was calculated? It's not clear from the current description.\n\n-  Model interpretability plays an important role for any downstream analysis and experimental work (especially in screening vaccine candidates or selecting antigens for vaccine development). Can the weights of the attention modules be used for identifying the immunogenic region in the antigen? This could possibly be leveraged to identify epitopes that can elicit protective antibodies when developing vaccines. This could be an useful addition to the manuscript.\n\n- How would the model handle variations in immunogenicity features across less common or novel pathogens that are not represented in the current dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- In my knowledge, the ProVaccine tool is the first method to utilize all three modalities of sequence, structure, and amino-acid descriptors to predict immunogenicity with deep-learning.\n- The authors have tried multiple PLMs to encode the sequences (ESM2, ANKH, and ProtBert) and structural embedding methods (ESM3, FoldSeek) to encode the structures.\n- They have compiled a new dataset called IMMUNO (stratified by antigen type- viral, bacterial, and tumour), which should serve as a valuable resource for training and benchmarking new methods for researchers in the field.\n- They have carried out a very comprehensive benchmarking and compared their method to appropriate baselines and also previous methods in literature. They also carried-out a cross-testing benchmarking of viral, bacterial, and tumour-specific models across datasets to assess the generalizability of the method.\n- The post-hoc analyses on specific pathogens (Helicobacterpylori and SARS-CoV-2) provide additional evidence of the model's usability in applied research. The method clearly improves upon immunogenicity prediction accuracy (against previous SOTA) and thus provides a reliable framework for future research in computational vaccinology.", "weaknesses": "-  Availability of negative dataset is a known problem in this field. The authors have used VAXIJEN to classify sequences as non-antigens and then subsequently filtered the sequences based on sequence-homology to compile the negative dataset. This will bias the negative dataset for certain features which might coincide with the features used in other tools (not just VAXIJEN, AA descriptors used for immunogenicity predictions are generally common across various methods). This might overinflate the recall for the negative class. An alternative approach could be to randomize the order of the amino-acids in the sequence while making sure that the sequence-identity of the randomized sequence isn't similar to any sequence in the positive class. The authors can also look at the IEDB database for compiling the negative dataset as it has sequences of non-antigens from experimental assays (one can chose sequences which were negative in atleast two independent studies)\n\n- For the structural embeddings, the authors have relied on ESMFold for the structure of the antigenic proteins. ESMFold, while much faster than AlphaFold, lags behind in accuracy (in the backbone atoms as well) to AlphaFold/RosettaFold. There isn't any justification provided for choosing these models. There needs to be more analysis done to show how does the structural model quality affect the performance of ProVaccine. The comparison can be done against experimental structures (from PDB), AlphaFold2/3 structures, and other structure prediction methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FGQMqbGshg", "reviewer": "Reviewer_dRH5", "summary": "This paper introduced PROVACCINE, a novel deep-learning solution for immunogenicity prediction in reverse vaccinology, designed to identify candidate vaccines that trigger protective immune responses. The authors also constructed a comprehensive immunogenicity dataset with over 9,500 antigen sequences from various pathogens. PROVACCINE employs a dual attention mechanism, integrating pre-trained representations of protein sequences and structures, and outperforms existing methods. The paper provided a comprehensive overview of the problem, related work, and the proposed methodology.", "questions": "The authors need to prove their methods on existing datasets. They also need to dig out why the model shows the largest improvement on the smallest dataset. The case study also needs to discuss about reason, instead of the performance only.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposed a deep learning supervised learning framework with a dual attention mechanism to address the key challenges of immunogenicity prediction in vaccine development.\n2. This paper provided a valuable benchmark for future research in vaccine development and immunogenicity prediction.", "weaknesses": "1. The work is innovative to construct a new benchmark dataset, but they didn't show novelty in algorithm development. The authors need to prove their methods on existed dataset, in addition to their own dataset.\n2. The dual-attention mechanism, while sophisticated, may increase the computational cost and complexity of the model significantly. The paper does not thoroughly compare this method against simpler architectures to show that the added complexity justifies the performance gains.\n3. The model’s performance is similar to XGBoost on the largest dataset, Immuno-Virus (1952/1762 positive/negative instances). However, the main improvement over XGBoost is on the smallest tumor dataset (300/477 positive/negative instances). Usually, DL is strong for big dataset and weak for small dataset, in contrary to this study. This discrepancy raised questions about the reliability and robustness of the model's performance.\n3. A major contribution of this work is the construction of a data set for immunogenicity prediction and vaccine development. It’s insufficient to discuss data quality, noise, or variability, particularly for immunogenic labels sourced from different repositories. Moreover, authors should demonstrate that the new dataset is superior to existing datasets. \n4. The authors should include ablation comparisons of sequence and structural features?\n5. This paper only compares web server methods, lacking comparisons with other deep learning frameworks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ZVmZGoqo8", "reviewer": "Reviewer_BQLB", "summary": "In the paper, the authors proposed a protein-language-model-based method called PROVACCINE for predicting the antigen from the amino acid sequence. They constructed the largest and most comprehensive antigen dataset through collecting published sequences. They further conducted cross-validation and some post-hoc analysis to show the effectiveness of the proposed model.", "questions": "See weakness above", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper conducted generalization experiments, which is good.\nThe comprehensive datasets, especially the positive sequence may work as the foundation for other work.", "weaknesses": "1. What I am mostly concerned with is the dataset construction. It seems that the authors construct the positive dataset from the literature, which is great, but the negative data are constructed with some other computational tools. This will lead to a serious problem: if the golden standard is created using BLAST+VAXIJEN, then if we use the combination of the two will be enough, why do we bother to train a model on top of it? \n\n2. Continuing from 1, the authors should add the baseline method using their way to construct the training dataset, they will get a perfect score for all metrics. The logic for building negative samples is very weird and then making the model unreliable.\n\n3. In section 5.2.1, the authors should also consider splitting the data based on the sequence similarity. If the training and testing data share very high sequence similarity, it's very likely that the model just memorizes the training and can not generalize.\n\n4. I see XGboost and KNN are good from Figure 4 -- suggesting the novel antigens share high similarity with known ones. Actually XGBoost seems to be perfect and much better than Provaccine. The proposed model did not show superiority than the traditional similarity-based method. Moreover, as majority of these 1, 858 sequences lack immunogenicity ground truth labels, how can we ensure the enrichment is a good metric?\n\n5. In section 5.3.2, it's unclear whether the spike protein already exist in the training dataset\n\n6. Continuing from 6, section 5.3.2 is too rough -- it's very easy to identify the spike protein as well as the RBD as the antigen, the results can not support the effectiveness of the proposed model.\n\nTaken together, the proposed method did not show enough novelty, and the performance is not as good as the similarity-based method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)