Immunogenicity Prediction with Dual Attention Enables Vaccine Target Selection

## Abstract

Abstract Immunogenicity prediction is a central topic in reverse vaccinology for finding candidate vaccines that can trigger protective immune responses. Existing approaches typically rely on highly compressed features and simple model architectures, leading to limited prediction accuracy and poor generalizability. To address these challenges, we introduce ProVaccine , a novel deep learning solution with a dual attention mechanism that integrates pre-trained latent vector representations of protein sequences and structures. We also compile the most comprehensive immunogenicity dataset to date, encompassing over 9 , 500 9 500 9,500 9 , 500 antigen sequences, structures, and immunogenicity labels from bacteria, viruses, and tumors. Extensive experiments demonstrate that ProVaccine outperforms existing methods across a wide range of evaluation metrics. Furthermore, we establish a post-hoc validation protocol to assess the practical significance of deep learning models in tackling vaccine design challenges. Our work provides an effective tool for vaccine design and sets valuable benchmarks for future research.

## 1 Introduction

Immunogenicity prediction is a pivotal step in reverse vaccinology. It aims at identifying specific proteins or peptide segments (protective antigens) that can produce humoral and/or cell-mediated immune responses and lead to memory cells production in the host organism (Adu-Bobie et al., ; Doneva et al., ). Protective antigens of pathogenic origin are potential vaccine
candidates (PVCs) (Arnon, ). Rapid and accurate prediction of protective antigens can reduce research and development costs, minimize vaccine development risks, and provide safe and effective prevention strategies against emerging and reemerging infectious diseases that pose ongoing threats (Pizza & M., ; Bagnoli et al., ; Carvalho et al., ).

Current machine learning approaches commonly exploit the physicochemical properties of amino acid residues, as characterized by E-descriptors (Venkatarajan & Braun, ) or Z-descriptors (Hellberg et al., ), and transform these descriptors into a uniform vector representation using auto- and cross-covariance (ACC) method. This vectorized format serves as input for machine learning models, which are then trained to predict whether antigens are protective or non-protective. (Doytchinova et al., ; Rawal et al., ). Due to the limited quantity of labeled data and the high complexity of the prediction problem, these methods excessively compress protein information in order to fit a relatively simple model, making it difficult to capture the complex relationship between antigens and their immunogenicity. As a result, these models often exhibit limited prediction accuracy and inadequate generalizability (e.g., in cross-species immunogenicity predictions).

Figure: Figure 1: Illustrative framework of ProVaccine. The model encodes sequence and structural representations using a dual-attention mechanism, followed by aggregation layers to incorporate global physicochemical attributes and perform binary classification tasks of immunogenicity prediction.
Refer to caption: extracted/5899440/figure/architecture.png

This study proposes ProVaccine, a deep learning method that interprets immunogenicity based on the multimodal encoding of antigens, including their sequences, structures, and physicochemical properties. To supplement the inherent amino acid (AA) sequence information, two levels of structural tokens are employed to comprehensively capture antigen construction patterns, and the physicochemical properties are hand-crafted based on Z-descriptor and E-descriptor. When allocating protein representations from different modalities, we establish a *dual-attention mechanism* (Figure 1) to facilitate communication between neighboring AAs at different scales. The use of tokenized structures at the atomic level (Van Kempen et al., ) and peptide level (Hayes et al., ) aligns with two biological intuitions: protein binding affinity primarily depends on atomic-level interactions, and local protein structures determine their functionality. Subsequently, we apply attention pooling to AA-level hidden representations to integrate them into protein-level vector representations, which incorporate the hand-crafted features that are commonly used to characterize the immunogenicity of proteins to further enhance the model’s capacity for immunogenicity prediction.

The reliability of deep models requires thorough empirical validation, such as statistically significant tests on high-quality, large-scale datasets and comparisons with similar baseline methods. Due to the scarcity and noise of high-throughput labels, wet-lab experiments are considered the most reliable validation, but the significant time and financial costs make it impractical for most research works to conduct. To address this, we prepare data for both types of validation methods and establish corresponding protocols. For the first type, we compile Immuno with over $9,500$ labeled antigens from bacteria, viruses, and tumors. This benchmark not only enables large-scale validation of new prediction methods but also provides valuable training data for developing new models. Meanwhile, for the second type of assessment, we propose two post-hoc validation protocols, involving significant subjects (Helicobacter pylori and SARS-CoV-2), comprehensive evaluation datasets, and clear, biologically meaningful evaluation criteria.

We verify the effectiveness of ProVaccine using the established two types of validation approaches and provide extensive analysis. Based on Immuno, we designe corresponding variations of training and test datasets, and perform comprehensive comparisons with existing methods across various evaluation metrics, focusing on prediction accuracy and generalizability of the trained model. The results demonstrate significant superiority of the proposed ProVaccine from different perspectives. In post-hod validations, we emphasize the practical significance of ProVaccine by demonstrating that its assigned probability to protective antigens effectively helps users identify potential vaccine candidates for vaccine design.

In summary, this study addresses the key challenges of immunogenicity prediction in vaccine development by (i) introducing ProVaccine, a deep learning supervised learning scheme with a dual attention mechanism; (ii) presenting the most comprehensive dataset of labeled antigens from different sources for training and testing immunogenicity prediction models; and (iii) constructing assessment protocols for evaluating prediction performance using benchmark datasets and post-hoc analyses. Empirically, we provide extensive evidence to validate that ProVaccine meets the challenges of vaccine target selection by providing accurate and robust predictions. Moreover, the key AAs identified by ProVaccine correlate with antigenic epitopes, suggesting more potential of our model to assist in vaccine development. Our work not only offers a reliable tool for advancing vaccine design but also provides crucial resources and insights for further research.

## 2 Immunogenicity Prediction and Protein Language Models

##### Problem Formulation

Reverse vaccinology is an extensively utilized methodology for identifying prospective vaccine candidates by computationally screening the proteome of a pathogen (Dalsass et al., ). This research is concerned with a binary classification task aimed at predicting the immunogenicity of proteins from three pathogenic sources: bacteria, viruses, and tumors. The primary goal is to develop a model proficient in discerning whether a protein or peptide segment is a protective antigen or a non-protective antigen.
Similar to most protein property prediction tasks, immunogenicity prediction also suffers from the limitation of scarce labeled data, making it crucial to develop a powerful and generalizable model. To achieve this goal, a feasible solution is to adopt a two-step training approach, where pre-trained models are first used to extract protein embeddings, followed by training the output module specifically for the prediction task.

##### Pre-trained Protein Language Model for AA-level Encoding

Protein language models (PLMs) have become the dominant method for learning representations of protein sequences. The two primary pre-training strategies are masked language model (Meier et al., ; Rao et al., ) and autoregressive generation (Ferruz et al., ; Madani et al., ). The former involves predicting masked tokens in the input sequence while capturing co-evolutionary relationships between AAs within the same sequence. In contrast, the latter learns to iteratively generate the next token based on previous tokens, which is particularly useful for tasks like sequence design. As we focus on property prediction tasks, we adopt the first strategy. During training, random tokens in the input sequence are masked, and the objective is to optimize the model’s parameters, denoted as ${\bm{\theta}}$, to minimize the error between the predicted and actual AAs at the masked positions, i.e.,

$$ $\operatorname*{arg\,min}_{{\bm{\theta}}}\mathbb{E}_{{\bm{x}}\sim{\bm{X}}} \mathbb{E}_{M}-\sum_{i\in M}\log{\rm P}({\bm{x}}_{i}|{\bm{x}}_{/M};{\bm{\theta }}).$ (1) $$

The conditional probability ${\rm P}({\bm{x}}_{i}|{\bm{x}}_{/M})$ for the $i$th token ${\bm{x}}_{i}$ in the sequence is determined by the unmasked tokens ${\bm{x}}_{/M}$. The model learns to capture the interactions between AAs within the protein sequence. It is notable that such a sequential representation extraction method is not only applicable to AA sequence tokenization but can also be extended to capture protein structural features. A straightforward approach to achieve this is by expanding the output tokens to include not only AA types but also discretized structural tokens (Su et al., ; Heinzinger et al., ; Li et al., ).
inputs used in our method in Section 4.

Figure: Figure 2: Data collection and processing steps of constructing the Immuno dataset.
Refer to caption: extracted/5899440/figure/datasetPreparation.png

## 3 Immuno : New Benchmark for Training and Evaluation

We curate three datasets with immunogenic (positive) and non-immunogenic (negative) labels from bacterial, viral, and tumor sources. Positive samples are compiled through an exhaustive literature search with manual validation. We check publicly available sources including PubMed, UniProt, NCBI, Protegen, and other publications up to October 2023. The search is strictly limited to proteins proven to be immunogenic in humans. For all three datasets, the preparation steps include collecting positive samples, collecting negative samples, and filtering redundant and tail-region samples. An illustrative workflow is visualized in Figure 2. In total, we have $913/1413$ positive/negative instances in Immuno-Bacteria, $1952/1762$ in Immuno-Virus, and $300/477$ in Immuno-Tumor.

##### Positive Samples

All positive samples are obtained from the Protegen database and previous publications. For Immuno-Bacteria, we collect data from Dimitrov et al. (); Edison et al. (); Rawal et al. (). For Immuno-Virus, positive samples are gathered from Rawal et al. (); Doneva & Dimitrov (). For Immuno-Tumor, data are sourced from Sotirov & Dimitrov (). Notably, for Immuno-Tumor, both positive and negative samples are collected in this step. The sequences of protein or peptide instances are generated from NCBI (Wheeler et al., ) and UniProtKB (Consortium, ).

##### Negative Samples

Due to the scarcity of data on non-immunogenic proteins in human studies, the negative sets are curated following established methods reported in prior research (Dimitrov et al., ; Sotirov & Dimitrov, ; Doneva & Dimitrov, ), aiming to roughly balance positive and negative samples in the three datasets. For instance, when processing Immuno-Bacteria, we first conduct a BLAST search to identify sequences from the same bacterial species as the protective antigens (positive samples) but with less than $30\%$ sequence identity. Samples predicted as Probable Non-Antigen by VaxiJen are subsequently passed to the next step of redundancy removal. We perform pairwise sequence alignment on the remaining samples to ensure no redundant sequences with $100\%$ identity left in the dataset. Finally, we exclude sequences shorter than $25$ AAs or longer than $1024$ AAs. A similar preprocessing approach is applied to the preparation of the negative dataset for Immuno-Virus. For Immuno-Tumor, we only perform the redundancy removal step, as the negative instances have already been provided by the literature and source database.

## 4 ProVaccine : Model Training and Inference

As shown in Figure 1, ProVaccine consists of three main components, including extracting features from protein sequences and structures, processing and integrating features from different modalities based on a dual attention mechanism, and finally compressing and outputting AA-level features to obtain epitope markers and immunogenicity predictions.

### 4.1 Input Features

##### Sequence Embedding and Structure Tokenization

We use a pre-trained PLM to extract embeddings for protein sequences and structures. The *AA sequence embedding* can be extracted from, an encoder-only model (e.g., ESM2; Lin et al. ()) or an encoder-decoder model (e.g., Ankh; Elnaggar et al. ()). For instance, given a protein sequence of length $L$, ESM2-650M delivers an $L\times 1280$ representation, with each AA being represented as a $1280$-dimensional vector. For protein structures, we use two tokenizers to encode the backbone predicted by ESMFold (Lin et al., ). The first *atomic structure tokenization* uses FoldSeek (Van Kempen et al., ) to analyze the spatial relationships between atoms of neighboring AAs at a finer scale and summarize them into $20$-dimensional tokens. The second *peptide tokenization* uses the ESM3 (Hayes et al., ) structure tokenizer to map coarser-scale substructures to $4,096$-dimensional structure tokens.

##### Physicochemical Descriptors

We incorporate five hand-crafted E-descriptors (Venkatarajan & Braun, ) and three Z-descriptors (Hellberg et al., ) to quantitatively characterize protein sequences. These descriptors provide an AA-level summary of physicochemical properties, such as hydrophobicity, molecular size, polarity, and secondary structure. They have been validated to assist in immunogenicity prediction (Doytchinova et al., ; Rawal et al., ). For more detailed information, please refer to Appendix A.

### 4.2 Dual Attention

A protein’s properties are determined by its sequence and structure. For immunogenicity prediction, focusing on local topological structures and molecular interactions may help identify exposed epitopes, thereby enhancing the extraction of critical information from the antigen. We combine handcrafted features with intrinsic sequence and structural information derived from pre-trained models, as described above. These different categories of protein features are integrated using a dual attention mechanism, which guides the model to focus on important local regions of the proteins. The features are then passed through attention pooling and propagated to the prediction head to obtain the predicted labels. The overall architecture of the model is shown in Figure 1. In the following sections, we detail the propagation rules of the dual attention and attention pooling modules.

##### Finer-lever and Coarser-level Attention Module

We construct the dual attention mechanism with two components, including *finer-level attention* and *coarser-level attention*. Both components construct a multi-head cross-attention framework to integrate *keys* and *values* from the sequence and *queries* from the structure. Following Lin et al. (), we replace the relative positional encoding of queries and keys with *Rotary Position Embeddings* (RoPE) (Su et al., ). For a protein sequence of $L$ AAs, i.e., ${\bm{x}}=\{{\bm{x}}_{1},{\bm{x}}_{2},\dots,{\bm{x}}_{L}\}$, we employ a pre-trained language model (e.g., ESM2) to extract the sequence encoding ${\bm{E}}_{\rm seq}\in\mathbb{R}^{L\times d}$. Structural encodings are extracted by structure tokenizers of different scales. We define the finer-level structural encoding from FoldSeek as ${\bm{E}}_{\rm fine}\in\mathbb{R}^{L\times d}$ and the coarser-level structural encoding from ESM3 as ${\bm{E}}_{\rm coarse}\in\mathbb{R}^{L\times d}$. Taking the finer-level attention as an example, the module’s input is $\{{\bm{E}}_{\rm seq},{\bm{E}}_{\rm fine}\}$, and the output is ${\bm{H}}_{\rm fine}$. The queries ${\bm{Q}}$, keys ${\bm{K}}$, and values ${\bm{V}}$ for the attention layers are defined as follows:

$$ ${\bm{Q}}={\bm{E}}_{\rm fine}{\bm{W}}_{Q},\quad{\bm{K}}={\bm{E}}_{\rm seq}{\bm{ W}}_{K},\quad{\bm{V}}={\bm{E}}_{\rm seq}{\bm{W}}_{V},$ (2) $$

where ${\bm{W}}_{Q},{\bm{W}}_{K},{\bm{W}}_{V}\in\mathbb{R}^{d\times d_{k}}$ are learnable projection matrices, and $d_{k}$ is the hidden dimension of queries and keys.
Next, the positional information is attached to ${\bm{Q}}$ and ${\bm{K}}$ with RoPE, which captures relative positional relationships by applying a rotation to the embeddings based on their positions. The modified queries and keys are:

$$ $\tilde{{\bm{Q}}}_{i}={\rm RoPE}({\bm{Q}}_{i},i),\quad\tilde{{\bm{K}}}_{j}={\rm RoPE }({\bm{K}}_{j},j),$ (3) $$

where $i$ and $j$ denote the positions in the respective sequences, and $\mathrm{RoPE}(\cdot)$ is the RoPE function.
Finally, the cross-attention ${\rm Attn}(\cdot)$ is formulated as:

$$ ${\rm Attn}({\bm{E}}_{\rm seq},{\bm{E}}_{\rm fine})={\rm softmax}\left(\frac{ \tilde{{\bm{Q}}}\tilde{{\bm{K}}}^{\top}}{\sqrt{d_{k}}}\right){\bm{V}},$ (4) $$

where $\tilde{{\bm{Q}}}\in\mathbb{R}^{L\times d_{k}}$ and $\tilde{{\bm{K}}}\in\mathbb{R}^{L\times d_{k}}$. The coarser-level attention applies the same construction. The only difference is that the input queries are replaced with ${\bm{E}}_{\rm coarser}$ from ESM3.

##### Attention Pooling

The processed AA-level features ${\bm{H}}={\rm concat}({\bm{E}}_{\rm seq},{\bm{H}}_{\rm fine},{\bm{H}}_{\rm
coarser
},{\bm{E}}_{\rm ez})$ are compressed using attention pooling and fed into the classifier for label prediction. Here, ${\bm{H}}$ for a protein consists of four components: the original sequence embedding, the joint sequence and structure representations processed by coarser-level and finer-level attention, and the handcrafted E and Z descriptors. Consider a processed feature sequence ${\bm{H}}\in\mathbb{R}^{L\times D}$, $L$ is the sequence length, and $D$ is the dimension of concatenated features. The attention pooling on ${\bm{H}}$ is defined as:

$$ $\displaystyle{\rm AttnPool}({\bm{H}})$ $\displaystyle=\sum_{i=1}^{L}\alpha_{i}\cdot{\bm{H}}_{i},$ (5) $\displaystyle{\rm where}\quad\alpha$ $\displaystyle={\rm softmax}({\rm MaskedConv1D}({\bm{H}})).$ $$

To obtain the final representation for the sequence, the model first computes $L\times 1$ attention scores using a one-dimensional masked convolution with a kernel size of $1$ (Yang et al., ), then applies an input mask and passes it through a softmax function to obtain the attention weights $\alpha$. These attention weights not only guide the pooling process but also indicate the importance of each position for the final prediction. In subsequent analysis, we extract these weights as epitope markers. Relevant analysis can be found in the post-hoc analysis section of the Experiments.

##### Prediction Head

The output of the attention pooling ${\rm AttnPool}({\bm{H}})$ is sent directly to the final prediction head to produce the classification logits. This process involves projecting the pooled representation through a linear transformation, followed by a dropout layer, a ReLU activation layer, and a final linear layer:

$$ ${\rm Logits}({\bm{H}})={\rm Linear}({\rm ReLU}({\rm DropOut}({\rm Linear}({\rm AttnPool }({\bm{H}}))))).$ (6) $$

## 5 Experiments

### 5.1 Experimental Protocol

##### Setup

We built and trained the model according to the framework introduced in Section 4. For the sequence embedding, we consider three pre-trained PLMs, including ESM2 (Lin et al., ), Ankh-large (Elnaggar et al., ), and ProtBert (Elnaggar et al., ). The source are listed in Appendix B. The model is optimized using AdamW (Loshchilov et al., ) with a learning rate of $0.0005$ and a weight decay of $0.01$. In the attention pooling layer, dropout was set to $0.1$. To ensure stable GPU memory usage during training, we limit the maximum number of tokens per batch to $4000$, with a gradient accumulation of $1$. The maximum training epoch was set to $50$, with early stopping based on validation accuracy and patience of $5$. All implementations were done using PyTorch (version 1.7.0), and experiments were run on an NVIDIA^® RTX 3090 GPU with 24GB VRAM, mounted on a Linux server. The training process was tracked using WanDB. All source data and the implementation to reproduce the results will be publicly available upon acceptance.

**Table 1: Average performance of baseline methods in three immunogenicity prediction tasks from $10$ random splits. The complete results with standard deviation can be found in Appendix.**
| Model | Bacteria | Virus | Tumor |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ACC | T30 | MCC | F1 | KS | ACC | T30 | MCC | F1 | KS | ACC | T30 | MCC | F1 | KS |  |
| RF | 83.9 | 86.1 | 65.3 | 77.3 | 0.67 | 90.0 | 99.2 | 80.0 | 90.2 | 0.81 | 67.2 | 62.2 | 28.6 | 50.1 | 0.36 |
| GBT | 82.5 | 83.6 | 62.5 | 76.2 | 0.65 | 89.9 | 98.7 | 79.9 | 90.3 | 0.81 | 62.6 | 54.8 | 20.4 | 49.8 | 0.35 |
| XGBoost | 82.7 | 84.5 | 63.3 | 77.1 | 0.66 | 91.4 | 100.0 | 82.9 | 91.7 | 0.84 | 64.0 | 61.7 | 24.3 | 53.3 | 0.33 |
| SGD | 68.8 | 80.0 | 31.7 | 34.4 | 0.59 | 83.5 | 91.7 | 67.2 | 83.2 | 0.71 | 59.9 | 52.2 | 0.0 | 0.0 | 0.28 |
| Logistic Regression | 62.2 | 68.3 | 3.4 | 0.9 | 0.45 | 61.2 | 85.9 | 34.6 | 72.3 | 0.65 | 59.9 | 44.8 | 0.0 | 0.0 | 0.22 |
| MLP | 79.3 | 79.4 | 55.0 | 70.1 | 0.60 | 86.9 | 93.2 | 74.2 | 87.7 | 0.76 | 59.9 | 40.0 | 0.0 | 0.0 | 0.21 |
| SVM | 62.6 | 64.5 | 25.9 | 57.4 | 0.38 | 75.2 | 98.0 | 50.4 | 76.0 | 0.62 | 55.8 | 58.3 | 23.4 | 60.6 | 0.30 |
| KNN | 81.4 | 80.7 | 59.7 | 72.9 | 0.57 | 88.7 | 88.9 | 77.3 | 88.9 | 0.77 | 56.8 | 45.2 | 11.2 | 47.6 | 0.13 |
| Vaxi-DL | 68.1 | 55.5 | 41.7 | 66.9 | 0.42 | 65.4 | 62.2 | 33.4 | 72.9 | 0.29 | - | - | - | - | - |
| VaxiJen2.0 | 75.7 | 62.2 | 54.6 | 72.1 | 0.57 | 82.0 | 74.8 | 66.6 | 84.2 | 0.84 | 54.9 | 41.3 | 8.5 | 47.7 | 0.10 |
| VaxiJen3.0 | 85.7 | 84.4 | 68.8 | 79.1 | 0.66 | 68.2 | 62.5 | 42.3 | 75.4 | 0.35 | 39.0 | 35.7 | 0.0 | 55.9 | 0.00 |
| VirusImmu | - | - | - | - | - | 58.8 | 59.6 | 18.1 | 63.6 | 0.17 | - | - | - | - | - |
| ProVaccine-Ankh | 86.2 | 88.7 | 70.3 | 80.9 | 0.71 | 92.5 | 99.1 | 88.5 | 94.2 | 0.89 | 77.7 | 77.4 | 55.8 | 74.5 | 0.62 |
| ProVaccine-ESM2 | 85.0 | 85.2 | 67.9 | 80.0 | 0.70 | 94.6 | 99.0 | 85.2 | 92.8 | 0.87 | 76.3 | 70.2 | 50.6 | 70.4 | 0.58 |
| ProVaccine-ProtBert | 85.8 | 86.5 | 69.2 | 80.2 | 0.69 | 92.8 | 98.7 | 85.6 | 92.9 | 0.88 | 71.3 | 67.8 | 43.9 | 67.1 | 0.52 |
| $\dagger$ The top three are highlighted by First, Second, Third. |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Figure: Figure 3: Generalizability of ProVaccine by the AUC and Recall of cross-test evaluations.
Refer to caption: extracted/5899440/figure/ood.png

##### Baseline Methods

The performance of the proposed ProVaccine is compared with existing machine learning methods and web server prediction tools. We select commonly used machine learning methods from the literature, including Random Forest, Gradient Boosting, XGBoost, SGD, Logistic Regression, MLP, SVM, and KNN, and follow the usual feature engineering approach by using ACC-transformed datasets to classify proteins as protective antigens (positive class) or non-protective antigens (negative class). We also include four popular web server methods specialized in antigen and vaccine candidate prediction: VaxiJen2.0 (Doytchinova et al., ), VaxiJen3.0 (Dimitrov et al., ), Vaxi-DL (Rawal et al., ), and VirusImmu (Li et al., ). VaxiJen2.0 employs an ACC transformation and PLS algorithm for antigen prediction. VaxiJen3.0 extends it to include machine learning for broader biological entities. Vaxi-DL utilizes deep learning models tailored for different disease-causing agents, and VirusImmu applies an ensemble machine learning method with soft voting for viral immunogenicity prediction.

### 5.2 Results Analysis on Immuno

We evaluate the overall performance of the models on ProVaccine. In light of the problems with existing methods, we focus on the accuracy and generalization of each method’s predictive performance. The performance of the models is assessed using a range of metrics for a comprehensive evaluation, including AUC-ROC, accuracy, precision, recall, F1 score, MCC, Top K accuracy, cross-entropy, and KS statistics. Details are provided in Appendix C.

#### 5.2.1 Accuracy

We first investigate the overall performance of each model on the three datasets in Immuno. For each dataset, we randomly split the data in a $7:1:2$ ratio and select the model with the highest accuracy on the $10\%$ validation set for evaluation. We repeat the assessment ten times for each test set, randomly selecting $50\%$ of the test data to calculate the scores for each metric, and report the average scores in Table 1. Complete averages and standard deviations can be found in Tables Tables 4-6 of the Appendix.

All machine learning and PLM methods we retrained ensured non-overlapping among the training, validation, and test sets. However, the four web servers do not have open-source code available for retraining on each task, which may introduce some unfair advantages to them. Additionally, during data preparation, we follow the convention in the field by using VaxiJen3.0 to check negative data labels, giving this method a natural advantage in predicting true negative data. Furthermore, since Vaxi-DL is not available for predicting antigens from tumors, we did not test its performance on the Immuno-tumor dataset. Similarly, VirusImmu is intended only for testing viruses, so we did not report its performance on Bacteria and Tumors for fairness consideration.

It is clearly demonstrated by the performance comparison of baseline methods across different metrics and datasets that ProVaccine has a significant advantage in prediction accuracy (indicated by top ACC). It achieves high accuracy for both positive and negative samples (indicated by high MCC and F1 scores) and effectively distinguishes the distribution differences between positive and negative samples (indicated by high KS scores). Notably, we specifically report the positive rate of top predictions (T30), i.e., the ratio of the top $30$ samples with the highest predicted positive probabilities that are true positives. This metric is crucial in predictive tasks involving biological experiments, since in real-world scenarios the samples are often highly imbalanced, and researchers usually conduct experimental validations on the samples with the highest model prediction confidence. Therefore, it is desirable for a model to provide high confidence for true positive samples. However, it is also important to note that if a model has a high T30 but considerably low accuracy, it may indicate that it occasionally predicts positive samples as negative, risking the omission of pivotal vaccine candidates. To avoid biased evaluations, we advocate assessing models based on their overall performance across multiple metrics rather than overly focusing on a single measure.

#### 5.2.2 Generalizability

We further conduct cross-testing and compare the generalizability of different models and visualize the results in Figure 3. Here, the training and test sets are from different sources. For example, in the left column (titled “Bacteria”), we report the test performance on Immuno-Bacteria by different models trained on Immuno-Virus (yellow) and Immuno-Tumor (orange). We exclude the four web server-based methods as we could not re-train them to avoid data leakage. We focus on AUC and recall to assess the model’s overall learning ability and its capacity to predict true positives. Generally, models trained on the Virus dataset demonstrate higher generalizability compared to those trained on the Bacteria dataset, and those trained on Tumor performed the worst. This may be related to the dataset size, as the number of antigens in Virus is larger than in Bacteria, and both are larger than Tumor.

Moreover, excluding models with prediction issues (e.g., logistic regression, where a $100\%$ recall and a relatively small AUC suggests it trivially predicts all samples as negative), ProVaccine consistently achieved top performance across the six test sets for both metrics. Additionally, the variation in ProVaccine’s performance on the same test set when trained on different datasets was relatively small, which further suggests the model’s robustness in learning intrinsic patterns instead of overfitting to local variations in the data, thereby demonstrating stronger generalizabilities.

Figure: Figure 4: KDE of predicted immunogenicity scores on Helicobacter pylori candidates. The $11$ experimentally determined immunogen are highlighted by red dots.
Refer to caption: extracted/5899440/figure/case1.png

### 5.3 Post-hoc Analysis

This section demonstrates the practical significance of ProVaccine in addressing vaccine design challenges, as a supplementary of the quantitative comparisons on well-established datasets and standard measurements. Two significant research subjects are selected to design corresponding ad-hoc analyses, including Helicobacter pylori and SARS-CoV-2. Specifically, in the first case, we examine whether the potential vaccine candidates identified by ProVaccine effectively cover immunogens, which is frequently assessed by the enrichment of the model’s predictions. In the second case study, we evaluate whether the ‘likely-effective’ sequences identified by ProVaccine are consistent with experimental evidence, i.e., if the model assigns the top confidence to the experimentally validated vaccine development sequences.

#### 5.3.1 Identification of Potential Vaccine Candidates for Helicobacter pylori

In the first challenge, we use ProVaccine to predict immunogens within the *Helicobacter pylori* proteome. Helicobacter pylori is a Group I carcinogen that can persist in the gastric mucosa, causing various gastrointestinal diseases (Malfertheiner et al., ). Identifying effective vaccine targets and developing corresponding vaccines can prevent Helicobacter pylori infections, thus significantly reducing the incidence of gastric diseases and lowering the rate of gastric cancer (Dos Santos Viana et al., ). To this end, we extract the proteome data from the UniProtKB database (ver 2024_04) (Consortium, ). A total number of $1,858$ sequences are extracted after redundancy removal. Although the majority of these $1,858$ sequences lack immunogenicity ground truth labels, we referr to the work of Dalsass et al. (), which record $11$ of these sequences as protective antigens. We use the model trained on Immuno-Bacteria for inference, since Helicobacter pylori is a bacterium. Note that there is no overlap between the training and prediction datasets.

When evaluate the prediction performance, two measurements are considered. (1) *Recall* measures the fraction of true protective antigens identified within the set of PVCs predicted by ProVaccine. It is a critical indicator of the model’s ability to accurately retrieve known protective antigens from a larger set of candidates. (2) *Fold-enrichment* compares the number of true protective antigens observed in the set of PVCs predicted by the model to the number expected by random chance, drawn from the proteome as a random sample of the same size as the PVC set. It provides a measure of the model’s enrichment capability, highlighting how much more effective it is compared to random guessing.

Among the $1,858$ candidates, ProVaccine identifies 123 PVCs. Notably, all $11$ true protective antigens are included within these $123$ PVCs. A recall rate of $100\%$ demonstrates ProVaccine’s exceptional ability to extract known protective antigens from the proteome data. In contrast, the probability of identifying known protective antigens through random guessing is only $72.8\%$ (^1^11Here, we follow the commonly used calculation method in the literature (Dalsass et al., ), which involves randomly selecting from the $123$ candidates (The candidate number matches the ProVaccine predicted positive counts of $123$ for fairness) to find experimentally determined protective antigens. The base recall is calculated by $11*123/1858=0.728$, or $72.8\%$.) Additionally, ProVaccine’s fold-enrichment is $15$ times that of random guessing. Although our method did not incorporate any human intervention during the screening process, it still shows significant superiority over non-strategic, random approaches.

In addition to the commonly used random selection method, we also compare the results of ProVaccine with other machine learning approaches. Table 10 compares the recall and enrichment of baseline methods, and Figure 4 visualizes the predicted kernel density estimation (KDE) of immunogenic probabilities for candidates of varying sequence lengths, with the $11$ immunogens highlighted in red dots. We exclude the VaxiJen series as they only provide web servers, in which case we could not confirm the absence of data leakage issues in the training data. From this result, three observations could be concluded: (1) The methods in the second row have trouble correctly labeling all the protective antigens. (2) The overall probability distribution found by ProVaccine aligns more closely with reality with most samples clustering around probabilities close to 0 0, implying the majority of candidates are non-immunogenic. (3) ProVaccine retains fewer PVCs from the candidates than baseline methods, allowing for the identification of the $11$ protective antigens with fewer experimental efforts.
In summary, our model exhibits strong robustness and considerable potential to enhance the discovery of immunogenic vaccine targets in addressing the challenges posed by Helicobacter pylori.

Figure: Figure 5: Epitope marker on the surface glycoprotein (NCBI ID: YP_009724390.1) by the attention score identifies vaccine targets.
Refer to caption: extracted/5899440/figure/spike.png

#### 5.3.2 Unraveling SARS-CoV-2 Proteome for Vaccine Target Discovery and Explanation

In the second challenge, we aim to investigate if our ProVaccine could effectively identify clinically validated vaccine targets from the SARS-CoV-2 proteome. The candidate test set consists of $38$ sequences from SARS-CoV-2 Data Hub (Wheeler et al., ). For the ground truth vaccine targets, we refer to the nine vaccines approved by the World Health Organization (Saravanan et al., ), five of which use the surface glycoprotein (spike protein) of the novel coronavirus as the vaccine target, while the other four use the whole virus, which also includes the spike protein. The overall prediction performance of the model is shown in Table 11 in Appendix, where ProVaccine ranks the most important protein (spike protein) as the top among the 38 proteins. The model also identifies some new potential vaccine targets, e.g., non-structural protein 9 (Nsp9).

To further validate and understand our predictions, we visualize the prediction results. Figure 5 maps the model’s attention scores to the antigenic epitopes of the sequences and comparing them with experimentally obtained epitopes (Shrock et al., ; Polyiam et al., ; Braun et al., ). It shows clearly that the model not only demonstrates excellent predictive performance but also possesses promising interpretability. For clear presentation, we display the chain trace of the spike protein and colored the amino acids based on the attention scores from the attention pooling layer, with darker positions indicating higher attention scores. Overall, the model shows higher attention scores around position $400$, and the spike protein contains three immunodominant linear B cell epitopes from position $319$ to $541$. Additionally, there are three regions with deeper colors that overlap with the SARS-CoV-2 spike receptor-binding domain (RBD), including positions $150$-$164$, $291$-$320$, and $439$-$478$. This visualization provides a deeper understanding of the model’s prediction process and enhances the credibility of our approach in identifying potential vaccine targets.

## 6 Related Work

Developing machine learning models has become a popular approach for many predictive problems in vaccine design. One of the most widely used web servers for identifying potential vaccine candidates is VaxiJen (Doytchinova et al., ), which employs an alignment-free method based on auto cross-covariance (ACC) transformation to identify antigens without relying on sequence alignment. Subsequent methods have introduced additional input features (He et al., ) or improved prediction models (Edison et al., ; Zhang et al., ; Rawal et al., ). Compared to other reverse vaccinology tools, these advanced techniques demonstrate superior performance in predicting bacterial protective antigens. Recently, Dimitrov et al. explored the immunogenicity of antigens from other sources, such as viruses (Doneva & Dimitrov, ) and tumors (Sotirov & Dimitrov, ), and evaluated various popular machine learning methods.

These machine learning approaches for predicting immunogenicity have often relied on limited datasets, typically confined to a specific type of pathogen, such as bacteria or viruses, and have been relatively small in scale. For instance, (Doytchinova et al., ) trained and validated their models using only a few hundred samples. The scarcity of data not only impedes the development of robust predictive models but also restricts the ability to capture the complex immunological responses elicited by diverse antigens. To date, few studies have managed to amalgamate a comprehensive dataset that encompasses antigens from multiple pathogen classes, which is essential for training models capable of predicting immunogenicity with high fidelity across varied biological contexts.

Despite the rapid development of deep learning, the advancement of methodologies for protein property prediction, including immunogenicity prediction, has been relatively slow due to the limited amount of labeled data. Recently, BERT-style PLMs (Lin et al., ; Li et al., ) have gained increasing attention with an increasing number of prediction tasks fine-tuning models to better fit the specific targets (Schmirler et al., ; Li et al., ; Zhou et al., ; ), or attaching additional propagators to process sequence embeddings, such as geometric encoders (Yang et al., ; Tan et al., ) or cross-modality aggregators (Tan et al., ; ).

## 7 Conclusion and Discussion

Despite the rapid advancement of deep learning in providing novel solutions to numerous problems in scientific fields such as protein engineering, the application of advanced models remains largely limited to well-defined engineering problems with relatively mature and abundant data. However, many other issues with significant scientific and practical value have not received sufficient attention, and the methods used for these problems are considerably outdated. In addition to the oversight of important problems due to a lack of interdisciplinary cross-talk between research fields, another major reason is the absence of high-quality datasets and adequate evaluation methods for many specific issues. Preparing and processing datasets for model training and evaluation requires significant effort and domain knowledge, and validating the developed methods also requires feasible evaluation protocols that fit practical requirements. Ideally, these protocols should neither be as disconnected from practical needs as current computational metrics (e.g., perplexity for sequence generation) nor as resource-intensive as wet lab validation processes in terms of time, money, and technical barriers.

As an exploratory work, this study takes immunogenicity prediction in vaccine development as an entry point to investigate how to prepare data, build models, and design evaluation strategies for a scientific problem that seeks powerful deep learning solutions. Although the field of protein representation learning has developed numerous pre-trained models, and immunogenicity prediction is inherently a standard supervised learning task, we emphasize that utilizing deep learning to solve scientific problems should not only focus on algorithm design but also on dataset construction and the practicality of evaluation methods. Based on this, we demonstrate the overall performance of the model using standard datasets and multiple conventional quantitative evaluation methods, and analyze the value of the model’s prediction results in specific applications using post-hoc analysis without conducting entirely new wet lab experiments.

## Appendix A Details on Physicochemical Descriptors

To provide protein-level properties, this study follows the convention and employs E-descriptors and Z-descriptors to quantitatively characterize protein sequences.

Venkatarajan & Braun () proposed the E-descriptors in 2001. For each of the 20 naturally occurring amino acids, five numerical values are derived based on the principal component analysis (PCA) of 237 physicochemical properties. The first component E1 strongly correlates with the hydrophobicity of amino acids. E2 provides information about molecular size and steric properties. Components E3 and E5 describe the propensity of amino acids to occur in $\alpha$-helices and $\beta$-strands, respectively. E4 takes into account the partial specific volume, the number of codons, and the relative frequency of amino acids in proteins.

The Z-descriptors, defined by Hellberg et al. (), summarize the principal physicochemical properties of amino acids. These descriptors are derived by PCA of a data matrix consisting of 29 molecular descriptors such as molecular weight, pKas, NMR shifts, etc. The first component Z1 reflects the hydrophobicity of amino acids, Z2 their size, and Z3 their polarity. The E and Z descriptors have been widely used for the characterization and classification of proteins and are good predictors of immunogenicity Doytchinova et al. (); Rawal et al. ().

**Table 2: Description of E-descriptors and Z-descriptors.**
| Name | Description |
| --- | --- |
| E1 | hydrophobicity of amino acids |
| E2 | molecular size and steric properties |
| E3 | the propensity of amino acids to occur in $\alpha$-helices |
| E4 | the partial specific volume, the number of codons, and the relative frequency of amino acids in proteins |
| E5 | the propensity of amino acids to occur in $\beta$-strands |
| Z1 | the hydrophobicity of amino acids |
| Z2 | size of amino acids |
| Z3 | polarity of amino acids |

## Appendix B Baseline Methods

Here we list the sources for running the comparisons in the Experiment section. The machine learning methods are implemented using sklearn, the web server prediction tools are obtained from their official websites, and the pre-trained PLMs (for sequence embedding) are sourced from their respective open-source GitHub repositories. Table 3 provides details for accessing these methods.

**Table 3: Source of baseline methods.**
| Name | version | Source |
| --- | --- | --- |
| Vaxi-DL | - | [https://vac.kamalrawal.in/vaxidl/](https://vac.kamalrawal.in/vaxidl/) |
| VaxiJen2.0 | - | [http://www.ddg-pharmfac.net/vaxijen/VaxiJen/VaxiJen.html](http://www.ddg-pharmfac.net/vaxijen/VaxiJen/VaxiJen.html) |
| VaxiJen3.0 | - | [https://www.ddg-pharmfac.net/vaxijen3/home/](https://www.ddg-pharmfac.net/vaxijen3/home/) |
| VirusImmu | - | [https://github.com/zhangjbig/VirusImmu](https://github.com/zhangjbig/VirusImmu) |
| ESM2 | t33_650M_UR50D | [https://huggingface.co/facebook/esm2_t33_650M_UR50D](https://huggingface.co/facebook/esm2_t33_650M_UR50D) |
| Ankh | large | [https://huggingface.co/ElnaggarLab/ankh-large](https://huggingface.co/ElnaggarLab/ankh-large) |
| ProtBert | - | [uniref&https://huggingface.co/Rostlab/prot_bert](uniref&https://huggingface.co/Rostlab/prot_bert) |

## Appendix C Evaluation Metrics

We employ a list of evaluation metrics to provide a comprehensive comparison of different prediction methods.

- •
AUC-ROC is a performance metric for classification models that measures the ability of the model to distinguish between classes. The ROC curve plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold settings. The AUC represents the area under this curve and provides an aggregate measure of performance across all thresholds.
$\text{AUC}=P(\text{rank}(X_{\text{positive}})>\text{rank}(X_{\text{negative}}))$
(7)
- •
Accuracy measures the proportion of correct predictions made by the model out of all predictions.
$\text{Accuracy}=\frac{\text{True Positives}+\text{True Negatives}}{\text{Total
Predictions}}=\frac{TP+TN}{TP+TN+FP+FN}$
(8)
- •
Precision quantifies the accuracy of positive predictions made by the model. It measures the proportion of true positives among all instances that were predicted as positive. High precision indicates that the model has a low rate of false positives.
$\text{Precision}=\frac{\text{True Positives}}{\text{True Positives}+\text{
False Positives}}=\frac{TP}{TP+FP}$
(9)
- •
Recall also known as Sensitivity or True Positive Rate, quantifies the ability of the model to identify all relevant instances.
$\text{Recall}=\frac{\text{True Positives}}{\text{True Positives}+\text{False
Negatives}}=\frac{TP}{TP+FN}$
(10)
- •
F1 score is the harmonic mean of Precision and Recall, providing a balance between the two metrics, especially when dealing with imbalanced datasets.
$\text{F1 Score}=2\times\frac{\text{Precision}\times\text{Recall}}{\text{
Precision}+\text{Recall}}$
(11)
- •
MCC is a balanced measure for binary classifications, even if the classes are of different sizes. It considers true and false positives and negatives to produce a coefficient between -1 and +1.
$\text{MCC}=\frac{(TP\times TN)-(FP\times FN)}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+
FN)}}$
(12)
- •
Top-K accuracy measures the fraction of times the correct label is among the model’s top $K$ predictions.
$\text{Top-K Accuracy}=\frac{\text{Number of correct predictions in top-K}}{
\text{Total Predictions}}$
(13)
- •
Cross-entropy quantifies the difference between two probability distributions: the true distribution and the predicted distribution by the model. $y$ is the true label (1 for positive class, 0 for negative class). $p$ is the predicted probability of the positive class.
$\text{Cross-Entropy Loss}=-[y\log(p)+(1-y)\log(1-p)]$
(14)
- •
KS statistics measures the maximum difference between the cumulative distribution functions (CDFs) of the positive and negative classes. It’s used to evaluate the discriminatory power of a scoring model.
$\text{KS}=\max_{x}|F_{\text{positive}}(x)-F_{\text{negative}}(x)|$
(15)

## Appendix D Complete Baseline Comparison on Immuno

Tables 4-6 present the detailed performance of all baseline methods, as well as our ProVaccine, on the three benchmark datasets in Immuno. The evaluation procedure follows the same methodology as described in the main text. We repeated the assessment on the test set for $10$ times, with each time randomly selecting $50\%$ of the samples from the test set, and reported the average and standard deviation across these $10$ runs in the tables. Due to limited space, in the main text we only report the average scores of the most important evaluation metrics. Here, we provide a more comprehensive summary, including additional measurements and confidence intervals, to allow for a more thorough comparison of the baseline models.

**Table 4: Average performance of baseline methods in bacteria immunogenicity prediction tasks from $10$ random splits, with the standard deviation reported after $\pm$.**
| Model | AUC-ROC | Accuracy | Precision | Recall | F1 | MCC | TopK-30 | CE | KS statistic |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RF | 89.5$\pm$0.9 | 83.9$\pm$1.2 | 83.0$\pm$2.7 | 72.4$\pm$1.8 | 77.3$\pm$1.8 | 65.3$\pm$2.6 | 86.1$\pm$2.8 | 0.43$\pm$0.01 | 0.67$\pm$0.02 |
| GBT | 87.9$\pm$1.5 | 82.5$\pm$1.3 | 78.8$\pm$2.4 | 73.9$\pm$1.8 | 76.2$\pm$1.6 | 62.5$\pm$2.5 | 83.6$\pm$3.5 | 0.42$\pm$0.02 | 0.65$\pm$0.03 |
| XGBoost | 89.4$\pm$1.2 | 82.7$\pm$1.2 | 77.5$\pm$2.6 | 76.8$\pm$2.1 | 77.1$\pm$1.9 | 63.3$\pm$2.5 | 84.5$\pm$3.9 | 0.47$\pm$0.04 | 0.66$\pm$0.03 |
| SGD | 84.3$\pm$1.6 | 68.8$\pm$1.5 | 85.1$\pm$6.1 | 21.6$\pm$3.3 | 34.4$\pm$4.4 | 31.7$\pm$4.3 | 80.0$\pm$3.5 | 0.84$\pm$0.19 | 0.59$\pm$0.03 |
| Logit Reg | 77.4$\pm$1.9 | 62.2$\pm$2.3 | 40.0$\pm$49.0 | 0.50$\pm$0.6 | 0.9$\pm$1.1 | 3.4$\pm$4.1 | 68.3$\pm$4.5 | 0.65$\pm$0.01 | 0.45$\pm$0.04 |
| MLP | 84.4$\pm$1.5 | 79.3$\pm$1.6 | 77.2$\pm$4.1 | 64.3$\pm$3.7 | 70.1$\pm$3.1 | 55.0$\pm$4.0 | 79.4$\pm$4.1 | 0.47$\pm$0.02 | 0.60$\pm$0.03 |
| SVM | 71.0$\pm$2.3 | 62.6$\pm$2.3 | 50.6$\pm$3.2 | 66.3$\pm$2.9 | 57.4$\pm$2.8 | 25.9$\pm$4.5 | 64.5$\pm$3.9 | 0.65$\pm$0.01 | 0.38$\pm$0.03 |
| KNN | 84.2$\pm$1.0 | 81.4$\pm$1.1 | 81.3$\pm$2.6 | 66.1$\pm$2.8 | 72.9$\pm$1.9 | 59.7$\pm$2.3 | 80.7$\pm$2.7 | 3.73 $\pm$0.38 | 0.57$\pm$0.02 |
| Vaxi-DL | 71.2$\pm$1.8 | 68.1$\pm$1.9 | 55.4$\pm$3.2 | 84.4$\pm$2.2 | 66.9$\pm$2.7 | 41.7$\pm$3.5 | 55.5$\pm$4.1 | 11.51$\pm$0.68 | 0.42$\pm$0.04 |
| VaxiJen2.0 | 78.5$\pm$1.5 | 75.7$\pm$1.7 | 61.0$\pm$2.3 | 88.3$\pm$1.8 | 72.1$\pm$1.9 | 54.6$\pm$2.8 | 62.2$\pm$3.0 | 8.76$\pm$0.61 | 0.57$\pm$0.03 |
| VaxiJen3.0 | 83.2$\pm$1.8 | 85.7$\pm$1.5 | 85.8$\pm$3.5 | 73.5$\pm$2.8 | 79.1$\pm$2.7 | 68.8$\pm$3.6 | 84.4$\pm$4.2 | 5.15$\pm$0.55 | 0.66$\pm$0.04 |
| ProVaccine-Ankh | 89.8$\pm$1.7 | 86.2$\pm$0.8 | 84.5$\pm$1.3 | 77.7$\pm$2.3 | 80.9$\pm$1.5 | 70.3$\pm$1.7 | 88.7$\pm$4.0 | 0.44$\pm$0.04 | 0.71$\pm$0.03 |
| ProVaccine-ESM2 | 89.8$\pm$1.1 | 85.0$\pm$0.8 | 79.1$\pm$2.3 | 80.8$\pm$1.4 | 80.0$\pm$1.1 | 67.9$\pm$1.3 | 85.2$\pm$3.4 | 0.45$\pm$0.03 | 0.70$\pm$0.01 |
| ProVaccine-ProtBert | 89.7$\pm$1.5 | 85.8$\pm$2.0 | 81.8$\pm$3.0 | 77.2$\pm$2.5 | 80.2$\pm$2.2 | 69.2$\pm$3.4 | 86.5$\pm$2.5 | 0.46$\pm$0.06 | 0.69$\pm$0.03 |
| $\dagger$ The top three are highlighted by First, Second, Third. |  |  |  |  |  |  |  |  |  |

**Table 5: Average performance of baseline methods in virus immunogenicity prediction tasks from $10$ random splits, with the standard deviation reported after $\pm$.**
| Model | AUC-ROC | Accuracy | Precision | Recall | F1 | MCC | TopK-30 | CE | KS statistic |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RF | 97.1$\pm$0.5 | 90.0$\pm$1.3 | 89.1$\pm$1.7 | 91.4$\pm$1.5 | 90.2$\pm$1.3 | 80.0$\pm$2.5 | 99.2$\pm$0.6 | 0.27$\pm$0.01 | 0.81$\pm$0.02 |
| GBT | 96.3$\pm$0.6 | 89.9$\pm$1.2 | 88.3$\pm$1.2 | 92.4$\pm$1.6 | 90.3$\pm$1.2 | 79.9$\pm$2.4 | 98.7$\pm$0.8 | 0.26$\pm$0.02 | 0.81$\pm$0.02 |
| XGBoost | 97.8$\pm$0.5 | 91.4$\pm$1.3 | 90.0$\pm$1.8 | 93.5$\pm$1.7 | 91.7$\pm$1.3 | 82.9$\pm$2.7 | 100.0$\pm$0.0 | 0.21$\pm$0.03 | 0.84$\pm$0.03 |
| SGD | 91.2$\pm$1.4 | 83.5$\pm$2.0 | 85.9$\pm$2.0 | 80.7$\pm$2.7 | 83.2$\pm$2.2 | 67.2$\pm$3.9 | 91.7$\pm$2.0 | 0.49$\pm$0.01 | 0.71$\pm$0.03 |
| Logit Reg | 87.7$\pm$1.8 | 61.2$\pm$1.2 | 56.6$\pm$1.3 | 100.0$\pm$0.0 | 72.3$\pm$1.1 | 34.6$\pm$1.2 | 85.9$\pm$2.6 | 0.67$\pm$0.00 | 0.65$\pm$0.04 |
| MLP | 92.8$\pm$1.3 | 86.9$\pm$1.4 | 83.7$\pm$1.6 | 92.2$\pm$1.4 | 87.7$\pm$1.4 | 74.2$\pm$2.7 | 93.2$\pm$1.5 | 0.34$\pm$0.03 | 0.76$\pm$0.03 |
| SVM | 85.9$\pm$1.1 | 75.2$\pm$1.0 | 74.6$\pm$1.5 | 77.5$\pm$1.4 | 76.0$\pm$1.2 | 50.4$\pm$1.9 | 98.0$\pm$1.2 | 0.53$\pm$0.02 | 0.62$\pm$0.02 |
| KNN | 91.2$\pm$0.9 | 88.7$\pm$1.0 | 88.4$\pm$1.5 | 89.3$\pm$1.4 | 88.9$\pm$1.1 | 77.3$\pm$2.1 | 88.9$\pm$2.5 | 2.80$\pm$0.30 | 0.77$\pm$0.02 |
| Vaxi-DL | 64.3$\pm$1.7 | 65.4$\pm$1.7 | 61.4$\pm$1.7 | 89.7$\pm$1.8 | 72.9$\pm$1.7 | 33.4$\pm$3.5 | 62.2$\pm$2.4 | 12.49$\pm$0.61 | 0.29$\pm$0.03 |
| VaxiJen2.0 | 82.1$\pm$1.0 | 82.0$\pm$1.2 | 75.1$\pm$1.8 | 95.7$\pm$1.0 | 84.2$\pm$1.3 | 66.6$\pm$2.1 | 74.8$\pm$3.4 | 6.47$\pm$0.55 | 0.64$\pm$0.02 |
| VaxiJen3.0 | 67.5$\pm$1.2 | 68.2$\pm$1.8 | 62.4$\pm$2.1 | 95.3$\pm$1.5 | 75.4$\pm$1.8 | 42.3$\pm$3.0 | 62.5$\pm$4.2 | 11.48$\pm$0.64 | 0.35$\pm$0.02 |
| VirusImmu | 58.7$\pm$2.2 | 58.8$\pm$2.1 | 56.9$\pm$2.5 | 72.2$\pm$2.1 | 63.6$\pm$2.1 | 18.1$\pm$4.4 | 59.6$\pm$4.6 | 14.86$\pm$0.76 | 0.17$\pm$0.04 |
| ProVaccine-Ankh | 97.7$\pm$0.4 | 92.5$\pm$0.6 | 91.4$\pm$1.3 | 95.5$\pm$1.3 | 92.8$\pm$0.7 | 85.2$\pm$1.2 | 99.5$\pm$0.4 | 0.22$\pm$0.02 | 0.87$\pm$0.02 |
| ProVaccine-ESM2 | 97.9$\pm$0.5 | 94.5$\pm$0.6 | 95.7$\pm$0.7 | 93.6$\pm$1.0 | 94.7$\pm$0.6 | 89.2$\pm$1.2 | 98.9$\pm$0.5 | 0.29$\pm$0.03 | 0.90$\pm$0.01 |
| ProVaccine-ProtBert | 97.5$\pm$0.6 | 91.6$\pm$0.9 | 97.0$\pm$0.8 | 93.5$\pm$1.4 | 92.9$\pm$0.8 | 85.6$\pm$1.5 | 98.7$\pm$0.9 | 0.23$\pm$0.02 | 0.88$\pm$0.02 |
| $\dagger$ The top three are highlighted by First, Second, Third. |  |  |  |  |  |  |  |  |  |

**Table 6: Average performance of baseline methods in Tumor immunogenicity prediction tasks from $10$ random splits, with the standard deviation reported after $\pm$.**
| Model | AUC-ROC | Accuracy | Precision | Recall | F1 | MCC | TopK-30 | Cross Entropy | KS statistic |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RF | 71.1$\pm$4.2 | 67.2$\pm$3.3 | 63.7$\pm$8.9 | 41.6$\pm$6.1 | 50.1$\pm$6.6 | 28.6$\pm$8.3 | 62.2$\pm$8.7 | 0.61$\pm$0.02 | 0.36 $\pm$0.07 |
| GBT | 68.9$\pm$5.2 | 62.6$\pm$2.1 | 53.8$\pm$6.3 | 46.6$\pm$4.3 | 49.8$\pm$4.4 | 20.4$\pm$5.0 | 54.8$\pm$8.3 | 0.64$\pm$0.05 | 0.35$\pm$0.07 |
| XGBoost | 68.7$\pm$6.0 | 64.0$\pm$4.3 | 55.3$\pm$8.9 | 51.9$\pm$7.8 | 53.3$\pm$7.5 | 24.3$\pm$10.2 | 61.7$\pm$9.1 | 0.79$\pm$0.10 | 0.33$\pm$0.08 |
| SGD | 63.8$\pm$3.0 | 59.9$\pm$4.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 52.2$\pm$5.8 | 3.54$\pm$0.66 | 0.28$\pm$0.04 |
| Logit Reg | 59.6$\pm$3.6 | 59.9$\pm$4.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 44.8$\pm$6.7 | 0.67$\pm$0.02 | 0.22$\pm$0.05 |
| MLP | 54.6$\pm$3.9 | 59.9$\pm$4.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 40.0$\pm$6.1 | 0.68$\pm$0.01 | 0.21$\pm$0.07 |
| SVM | 67.0$\pm$5.7 | 55.8$\pm$4.5 | 47.2$\pm$5.1 | 85.3$\pm$5.3 | 60.6$\pm$4.8 | 23.4$\pm$8.8 | 58.3$\pm$9.0 | 0.65$\pm$0.02 | 0.30$\pm$0.06 |
| KNN | 55.8$\pm$3.6 | 56.8$\pm$2.8 | 46.3$\pm$6.0 | 49.6$\pm$7.6 | 47.6$\pm$5.5 | 11.2$\pm$6.7 | 45.2$\pm$8.7 | 10.41$\pm$1.43 | 0.13$\pm$0.06 |
| VaxiJen2.0 | 54.3$\pm$4.3 | 54.9$\pm$3.7 | 44.7$\pm$6.5 | 51.4$\pm$4.6 | 47.7$\pm$5.3 | 8.5$\pm$8.6 | 41.3$\pm$9.2 | 16.27$\pm$1.53 | 0.10$\pm$0.07 |
| VaxiJen3.0 | 50.0$\pm$0.0 | 39.0$\pm$4.6 | 39.0$\pm$4.6 | 100.0$\pm$0.0 | 55.9$\pm$4.8 | 0.0$\pm$0.0 | 35.7$\pm$4.7 | 22.00$\pm$1.66 | 0.00$\pm$0.00 |
| ProVaccine-Ankh | 85.2$\pm$1.9 | 77.7$\pm$2.1 | 70.2$\pm$6.1 | 83.0$\pm$3.8 | 74.5$\pm$3.3 | 55.8$\pm$4.4 | 77.4$\pm$2.6 | 0.48$\pm$0.03 | 0.62$\pm$0.03 |
| ProVaccine-ESM2 | 82.2$\pm$3.5 | 73.7$\pm$3.1 | 67.8$\pm$3.5 | 70.1$\pm$5.4 | 70.4$\pm$4.7 | 50.6$\pm$5.9 | 67.8$\pm$5.6 | 0.61$\pm$0.02 | 0.58$\pm$0.07 |
| ProVaccine-ProtBert | 81.0$\pm$3.2 | 71.3$\pm$3.5 | 59.6$\pm$6.4 | 78.1$\pm$3.9 | 67.1$\pm$5.3 | 43.9$\pm$8.1 | 67.8$\pm$7.3 | 0.58$\pm$0.07 | 0.52$\pm$0.08 |
| $\dagger$ The top three are highlighted by First, Second, Third. |  |  |  |  |  |  |  |  |  |

## Appendix E Ablation Study

This section discusses some of the design choices for our ProVaccine. We evaluate the contributions of different inputs to the final results, as well as the advantages of attention pooling compared to simpler global pooling methods. Similar to the previous section, we conducted a full comparison across the three datasets, with results presented in Tables 7-9.

**Table 7: Average performance of baseline methods in Bacteria immunogenicity prediction tasks from $10$ random splits, with the standard deviation reported after $\pm$.**
| input module | AUC-ROC | Accuracy | Precision | Recall | F1 | MCC | TopK-30 | Cross Entropy | KS statistic |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ankh |  |  |  |  |  |  |  |  |  |
| PLM Encode | 86.7$\pm$1.7 | 81.5$\pm$1.5 | 78.0$\pm$2.8 | 70.7$\pm$2.5 | 74.2$\pm$2.3 | 60.0$\pm$3.3 | 63.1$\pm$3.2 | 0.42$\pm$0.02 | 0.84$\pm$0.04 |
| PLM Encode + Atomic Tokenization | 86.7$\pm$1.9 | 83.1$\pm$1.5 | 82.2$\pm$2.2 | 70.2$\pm$4.1 | 75.7$\pm$2.7 | 63.4$\pm$3.3 | 63.1$\pm$3.6 | 0.45$\pm$0.04 | 0.85$\pm$0.03 |
| PLM Encode + Physiochemical Descriptors | 87.1$\pm$1.7 | 82.2$\pm$1.6 | 77.1$\pm$3.4 | 72.4$\pm$3.5 | 74.6$\pm$2.7 | 61.1$\pm$3.6 | 63.4$\pm$3.9 | 0.42$\pm$0.03 | 0.81$\pm$0.05 |
| ProVaccine | 89.8$\pm$1.7 | 86.2$\pm$0.8 | 84.5$\pm$1.3 | 77.7$\pm$2.3 | 80.9$\pm$1.5 | 70.3$\pm$1.7 | 88.7$\pm$4.0 | 0.44$\pm$0.04 | 0.71$\pm$0.03 |
| ESM |  |  |  |  |  |  |  |  |  |
| PLM Encode | 88.8$\pm$1.4 | 83.0$\pm$1.5 | 78.4$\pm$2.4 | 72.3$\pm$2.8 | 75.2$\pm$2.4 | 62.4$\pm$3.3 | 68.7$\pm$3.0 | 0.39$\pm$0.03 | 0.80$\pm$0.03 |
| PLM Encode + Atomic Tokenization | 89.9$\pm$1.4 | 84.0$\pm$1.0 | 81.0$\pm$2.3 | 74.8$\pm$3.0 | 77.8$\pm$2.0 | 65.4$\pm$2.3 | 68.8$\pm$2.9 | 0.41$\pm$0.03 | 0.85$\pm$0.03 |
| PLM Encode + Physiochemical Descriptors | 88.2$\pm$1.1 | 82.0$\pm$1.5 | 76.5$\pm$1.7 | 75.2$\pm$3.4 | 75.8$\pm$1.8 | 61.5$\pm$2.8 | 67.1$\pm$2.8 | 0.41$\pm$0.02 | 0.80$\pm$0.03 |
| ProVaccine | 89.8$\pm$1.1 | 85.0$\pm$0.8 | 79.1$\pm$2.3 | 80.8$\pm$1.4 | 80.0$\pm$1.1 | 67.9$\pm$1.3 | 85.2$\pm$3.4 | 0.45$\pm$0.03 | 0.70$\pm$0.01 |
| ProtBert |  |  |  |  |  |  |  |  |  |
| PLM Encode | 89.9$\pm$1.2 | 86.1$\pm$1.5 | 84.6$\pm$1.9 | 78.0$\pm$2.8 | 81.2$\pm$2.3 | 70.3$\pm$3.4 | 70.2$\pm$3.4 | 0.39$\pm$0.03 | 0.88$\pm$0.02 |
| PLM Encode + Atomic Tokenization | 89.7$\pm$1.9 | 85.2$\pm$1.5 | 82.6$\pm$2.8 | 75.0$\pm$3.9 | 78.6$\pm$2.7 | 67.6$\pm$3.6 | 68.7$\pm$3.1 | 0.40$\pm$0.05 | 0.85$\pm$0.04 |
| PLM Encode + Physiochemical Descriptors | 88.4$\pm$1.7 | 82.2$\pm$1.3 | 76.9$\pm$2.3 | 74.9$\pm$3.8 | 75.8$\pm$1.8 | 61.9$\pm$2.7 | 66.1$\pm$3.7 | 0.41$\pm$0.03 | 0.84$\pm$0.03 |
| ProVaccine | 89.7$\pm$1.5 | 85.8$\pm$2.0 | 81.8$\pm$3.0 | 77.2$\pm$2.5 | 80.2$\pm$2.2 | 69.2$\pm$3.4 | 86.5$\pm$2.5 | 0.46$\pm$0.06 | 0.69$\pm$0.03 |

**Table 8: Average performance of baseline methods in Virus immunogenicity prediction tasks from $10$ random splits, with the standard deviation reported after $\pm$.**
| input module | AUC-ROC | Accuracy | Precision | Recall | F1 | MCC | TopK-30 | Cross Entropy | KS statistic |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ankh |  |  |  |  |  |  |  |  |  |
| PLM Encode | 94.6$\pm$0.9 | 84.8$\pm$1.5 | 81.3$\pm$2.3 | 91.0$\pm$1.3 | 85.8$\pm$2.5 | 70.1$\pm$2.9 | 75.4$\pm$2.5 | 0.34$\pm$0.02 | 0.98$\pm$0.01 |
| PLM Encode + Atomic Tokenization | 97.9$\pm$0.6 | 93.7$\pm$0.6 | 95.5$\pm$0.7 | 92.2$\pm$1.1 | 93.8$\pm$0.7 | 87.5$\pm$1.2 | 88.9$\pm$1.8 | 0.20$\pm$0.03 | 0.99$\pm$0.00 |
| PLM Encode + Physiochemical Descriptors | 96.1$\pm$0.7 | 88.1$\pm$1.0 | 85.2$\pm$1.6 | 92.6$\pm$2.0 | 88.8$\pm$2.1 | 76.4$\pm$2.0 | 81.3$\pm$2.1 | 0.28$\pm$0.02 | 0.99$\pm$0.01 |
| ProVaccine | 97.7$\pm$0.4 | 92.5$\pm$0.6 | 91.4$\pm$1.3 | 95.5$\pm$1.3 | 92.8$\pm$0.7 | 85.2$\pm$1.2 | 99.5$\pm$0.4 | 0.22$\pm$0.02 | 0.87$\pm$0.02 |
| ESM |  |  |  |  |  |  |  |  |  |
| PLM Encode | 96.0$\pm$0.8 | 90.0$\pm$1.1 | 89.0$\pm$1.7 | 91.6$\pm$2.1 | 90.3$\pm$2.1 | 80.0$\pm$1.9 | 81.5$\pm$2.0 | 0.26$\pm$0.03 | 0.99$\pm$0.01 |
| PLM Encode + Atomic Tokenization | 97.5$\pm$0.6 | 93.7$\pm$0.9 | 95.8$\pm$0.8 | 91.7$\pm$1.0 | 93.7$\pm$0.9 | 87.4$\pm$1.7 | 89.1$\pm$1.8 | 0.29$\pm$0.05 | 1.00$\pm$0.01 |
| PLM Encode + Physiochemical Descriptors | 96.9$\pm$0.5 | 91.2$\pm$0.8 | 91.5$\pm$1.2 | 90.9$\pm$1.7 | 91.2$\pm$1.7 | 82.3$\pm$1.8 | 84.6$\pm$1.3 | 0.23$\pm$0.02 | 0.99$\pm$0.00 |
| ProVaccine | 97.9$\pm$0.5 | 94.5$\pm$0.6 | 95.7$\pm$0.7 | 93.6$\pm$1.0 | 94.7$\pm$0.6 | 89.2$\pm$1.2 | 98.9$\pm$0.5 | 0.29$\pm$0.03 | 0.90$\pm$0.01 |
| ProtBert |  |  |  |  |  |  |  |  |  |
| PLM Encode | 97.2$\pm$0.7 | 91.4$\pm$1.2 | 89.3$\pm$1.4 | 94.2$\pm$2.3 | 91.7$\pm$2.3 | 82.9$\pm$2.3 | 84.8$\pm$2.3 | 0.23$\pm$0.03 | 0.99$\pm$0.01 |
| PLM Encode + Atomic Tokenization | 96.8$\pm$0.5 | 90.8$\pm$0.9 | 90.4$\pm$0.9 | 91.7$\pm$1.5 | 91.0$\pm$1.7 | 81.6$\pm$1.9 | 82.9$\pm$1.7 | 0.23$\pm$0.02 | 0.99$\pm$0.01 |
| PLM Encode + Physiochemical Descriptors | 96.9$\pm$0.5 | 91.5$\pm$1.0 | 89.4$\pm$1.5 | 94.6$\pm$2.1 | 91.9$\pm$2.1 | 83.0$\pm$2.1 | 85.3$\pm$2.1 | 0.24$\pm$0.03 | 0.99$\pm$0.01 |
| ProVaccine | 97.5$\pm$0.6 | 91.6$\pm$0.9 | 97.0$\pm$0.8 | 93.5$\pm$1.4 | 92.9$\pm$0.8 | 85.6$\pm$1.5 | 98.7$\pm$0.9 | 0.23$\pm$0.02 | 0.88$\pm$0.02 |

**Table 9: Average performance of baseline methods in Tumor immunogenicity prediction tasks from $10$ random splits, with the standard deviation reported after $\pm$.**
| input module | AUC-ROC | Accuracy | Precision | Recall | F1 | MCC | TopK-30 | Cross Entropy | KS statistic |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ankh |  |  |  |  |  |  |  |  |  |
| PLM Encode | 65.9$\pm$3.3 | 60.9$\pm$2.0 | 60.9$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$3.3 | 33.4$\pm$3.3 | 0.67$\pm$0.00 | 0.51$\pm$0.06 |
| PLM Encode + Atomic Tokenization | 83.1$\pm$3.0 | 75.9$\pm$2.9 | 75.9$\pm$4.5 | 66.9$\pm$5.5 | 80.5$\pm$4.0 | 72.9$\pm$6.1 | 52.3$\pm$6.2 | 0.51$\pm$0.05 | 0.81$\pm$0.07 |
| PLM Encode + Physiochemical Descriptors | 67.0$\pm$6.3 | 62.7$\pm$3.2 | 62.7$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$8.3 | 33.5$\pm$0.0 | 0.66$\pm$0.01 | 0.49$\pm$0.09 |
| ProVaccine | 85.2$\pm$1.9 | 77.7$\pm$2.1 | 70.2$\pm$6.1 | 83.0$\pm$3.8 | 74.5$\pm$3.3 | 55.8$\pm$4.4 | 77.4$\pm$2.6 | 0.48$\pm$0.03 | 0.62$\pm$0.03 |
| ESM |  |  |  |  |  |  |  |  |  |
| PLM Encode | 81.5$\pm$4.3 | 70.4$\pm$3.6 | 70.4$\pm$4.7 | 60.7$\pm$3.4 | 75.7$\pm$6.8 | 66.8$\pm$9.1 | 41.8$\pm$4.0 | 0.51$\pm$0.04 | 0.73$\pm$0.07 |
| PLM Encode + Atomic Tokenization | 83.4$\pm$3.4 | 70.6$\pm$4.2 | 70.6$\pm$5.5 | 58.0$\pm$5.6 | 81.7$\pm$5.0 | 67.7$\pm$8.0 | 44.3$\pm$7.6 | 0.51$\pm$0.05 | 0.74$\pm$0.07 |
| PLM Encode + Physiochemical Descriptors | 78.4$\pm$2.0 | 68.6$\pm$2.6 | 68.6$\pm$5.1 | 59.8$\pm$4.0 | 73.9$\pm$3.3 | 65.9$\pm$4.8 | 38.0$\pm$3.5 | 0.55$\pm$0.02 | 0.72$\pm$0.07 |
| ProVaccine | 82.2$\pm$3.5 | 73.7$\pm$3.1 | 67.8$\pm$3.5 | 70.1$\pm$5.4 | 70.4$\pm$4.7 | 50.6$\pm$5.9 | 67.8$\pm$5.6 | 0.61$\pm$0.02 | 0.58$\pm$0.07 |
| ProtBert |  |  |  |  |  |  |  |  |  |
| PLM Encode | 69.7$\pm$3.0 | 59.9$\pm$3.8 | 59.9$\pm$6.5 | 51.8$\pm$5.4 | 64.7$\pm$5.2 | 57.3$\pm$7.8 | 20.9$\pm$4.7 | 0.61$\pm$0.02 | 0.61$\pm$0.07 |
| PLM Encode + Atomic Tokenization | 81.4$\pm$3.0 | 73.5$\pm$3.3 | 73.5$\pm$6.0 | 60.6$\pm$6.1 | 77.4$\pm$4.0 | 67.8$\pm$6.0 | 46.9$\pm$5.2 | 0.53$\pm$0.05 | 0.67$\pm$0.06 |
| PLM Encode + Physiochemical Descriptors | 75.7$\pm$3.3 | 69.5$\pm$3.7 | 69.5$\pm$6.5 | 58.9$\pm$5.9 | 76.6$\pm$5.2 | 66.5$\pm$7.5 | 40.4$\pm$7.8 | 0.61$\pm$0.04 | 0.66$\pm$0.07 |
| ProVaccine | 81.0$\pm$3.2 | 71.3$\pm$3.5 | 59.6$\pm$6.4 | 78.1$\pm$3.9 | 67.1$\pm$5.3 | 43.9$\pm$8.1 | 67.8$\pm$7.3 | 0.58$\pm$0.07 | 0.52$\pm$0.08 |

## Appendix F Additional Information on the assessing dataset in Case Study

Tables 10-11 summarize the prediction results on Helicobacter pylori and SARS-COV-2 targets for the two case studies in the post-hoc analysis. The relevant analysis can be found in the main text.

**Table 10: Model prediction performance on the first case study of Helicobacter pylori.**
| Model | Predicted Positive | True Positive | Random Selection | Recall (%) | fold-enrichment |
| --- | --- | --- | --- | --- | --- |
| Random Forest | 211 | 11 | 1.25 | 100.00 | 8.81 |
| Gradient Boosting | 273 | 11 | 1.62 | 100.00 | 6.81 |
| XGBoost | 287 | 11 | 1.70 | 100.00 | 6.47 |
| SGD | 25 | 0 | 0.15 | 0.00 | 0.00 |
| Logistic Regression | 0 | 0 | 0.00 | 0.00 | 0.00 |
| MLP | 227 | 4 | 1.34 | 36.36 | 2.98 |
| KNN | 284 | 10 | 1.68 | 90.91 | 5.95 |
| ProVaccine | 123 | 11 | 0.73 | 100.00 | 15.11 |

**Table 11: Prediction Details of SARS-COV-2 Targets by ProVaccine.**
| NCBI ID | Protein Name | predicted label | predicted probability |
| --- | --- | --- | --- |
| YP_009724390.1 | surface glycoprotein | 1 | 1.00E$+$00 |
| YP_009742617.1 | nsp10 | 1 | 1.00E$+$00 |
| YP_009742616.1 | nsp9 | 1 | 1.00E$+$00 |
| YP_009742609.1 | nsp2 | 1 | 9.99E-01 |
| YP_009742612.1 | 3C-like proteinase | 1 | 9.98E-01 |
| YP_009725312.1 | nsp11 | 1 | 9.97E-01 |
| YP_009724396.1 | ORF8 protein | 1 | 9.97E-01 |
| YP_009742610.1 | nsp3 | 1 | 9.97E-01 |
| YP_009724397.2 | nucleocapsid phosphoprotein | 1 | 9.95E-01 |
| YP_009742614.1 | nsp7 | 1 | 9.94E-01 |
| YP_009724395.1 | ORF7a protein | 1 | 9.41E-01 |
| YP_009725307.1 | RNA-dependent RNA polymerase | 1 | 8.64E-01 |
| YP_009725310.1 | endoRNAse | 1 | 7.68E-01 |
| YP_009742611.1 | nsp4 | 0 | 4.94E-01 |
| YP_009724391.1 | ORF3a protein | 0 | 4.63E-01 |
| YP_009742608.1 | leader protein | 0 | 1.50E-01 |
| YP_009725308.1 | helicase | 0 | 1.40E-01 |
| YP_009742613.1 | nsp6 | 0 | 1.07E-01 |
| YP_009742615.1 | nsp8 | 0 | 4.69E-02 |
| YP_009725309.1 | 3’-to-5’ exonuclease | 0 | 2.15E-02 |
| YP_009725311.1 | 2’-O-ribose methyltransferase | 0 | 2.59E-04 |
| YP_009724393.1 | membrane glycoprotein | 0 | 6.56E-05 |
| YP_009724392.1 | envelope protein | 0 | 1.20E-05 |
| YP_009724394.1 | ORF6 protein | 0 | 4.20E-06 |
| YP_009725318.1 | ORF7b | 0 | 2.73E-06 |
| YP_009725255.1 | ORF10 protein | 0 | 1.21E-06 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 3
presentation: 3
questions: - I don't understand the approach to measuring accuracy described in the first paragraph of section 5.2.1. Are you doing hyperparameter optimization on the validation set for each split? Why randomly select "50% of the test data to calculate the scores for each metric" since you already have a validation set in your split?
- It's not clear what the generalizability of section 5.2.2 is measuring. Should a immunogenicity predictor generalize across bacteria, viruses and humans?
- For the three conclusions about ProVaccine in the final paragraph of subsection 5.3.1, I don't understand how the authors concluded the second and third - could you please explain these in greater detail?
soundness: 3
strengths: - novel model architecture
- contribution of the Immuno datasets
- ProVaccine achieves mostly top performance as reported in Table 1
summary: The authors present the ProVaccine model for immunogenicity prediction, as well as the Immuno datasets of antigens from bacteria, viruses and tumors, which is balanced across protective and nonprotective antigens. They show that ProVaccine is more accurate compared with both baseline ML and hosted models.
weaknesses: - It's not obvious what the contribution of any of the representations (sequence, fine, coarse, and descriptors) is to model performance. What if only the sequence representation is used as input? Doing ablation studies is essential for understanding these contributions, and should be included in this work.
- Figure 4 is hard to interpret, it would be easier to understand if instead the likelihood stats of the 11 determined immunogens under each model were reported.
- The appendix is missing:
  - page 7 in section 5.2.1 refers to Tables 4-6 in the appendix
  - page 8 in section 5.3.1 refers to recall and fold-enrichment metrics that do not appear in any table. Also reference is made to Table 10, which does not appear to exist.
  - page 9 in section 5.3.2 refers to Table 11 in the appendix, which does not exist

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: - Was there any homology cut-off applied between the training and testing datasets for each class of the antigens? High sequence similarity b/w training and testing might over-inflate performance.

- Most of  the figures have a generic caption. The captions should explain the figures clearly and the results being shown in the figure (especially the figures showing the results and comparison). The results from the figures should also be discussed in the main text in detail. 

- In the post-hoc analysis, it it possible to check how well will the model perform for non-immunogenic antigens in Helicobacter pylori (Fig 4). The sequences of the 11- experimentally verified immunogens can be randomized and then it can be checked where does the prediction for the randomized-sequence lie in terms of probability. The authors also need to describe how 'the probability of identifying known protective antigens through random guessing' was calculated? It's not clear from the current description.

-  Model interpretability plays an important role for any downstream analysis and experimental work (especially in screening vaccine candidates or selecting antigens for vaccine development). Can the weights of the attention modules be used for identifying the immunogenic region in the antigen? This could possibly be leveraged to identify epitopes that can elicit protective antibodies when developing vaccines. This could be an useful addition to the manuscript.

- How would the model handle variations in immunogenicity features across less common or novel pathogens that are not represented in the current dataset?
soundness: 3
strengths: - In my knowledge, the ProVaccine tool is the first method to utilize all three modalities of sequence, structure, and amino-acid descriptors to predict immunogenicity with deep-learning.
- The authors have tried multiple PLMs to encode the sequences (ESM2, ANKH, and ProtBert) and structural embedding methods (ESM3, FoldSeek) to encode the structures.
- They have compiled a new dataset called IMMUNO (stratified by antigen type- viral, bacterial, and tumour), which should serve as a valuable resource for training and benchmarking new methods for researchers in the field.
- They have carried out a very comprehensive benchmarking and compared their method to appropriate baselines and also previous methods in literature. They also carried-out a cross-testing benchmarking of viral, bacterial, and tumour-specific models across datasets to assess the generalizability of the method.
- The post-hoc analyses on specific pathogens (Helicobacterpylori and SARS-CoV-2) provide additional evidence of the model's usability in applied research. The method clearly improves upon immunogenicity prediction accuracy (against previous SOTA) and thus provides a reliable framework for future research in computational vaccinology.
summary: The paper presents PROVACCINE, a deep learning model designed to predict the immunogenicity of proteins, which is a crucial task in vaccine development. The authors introduce a dual-attention mechanism within the model, enabling it to process multimodal representations of protein sequences, structures, and physicochemical properties. They also curate Immuno, an extensive dataset of over 9,500 antigen sequences and structures from various sources, to facilitate robust training and validation. The experimental results show that PROVACCINE outperforms traditional methods across multiple evaluation metrics, with specific validation studies on Helicobacter pylori and SARS-CoV-2. The post-hoc analyses further confirm PROVACCINE’s efficacy in identifying potential vaccine targets.
weaknesses: -  Availability of negative dataset is a known problem in this field. The authors have used VAXIJEN to classify sequences as non-antigens and then subsequently filtered the sequences based on sequence-homology to compile the negative dataset. This will bias the negative dataset for certain features which might coincide with the features used in other tools (not just VAXIJEN, AA descriptors used for immunogenicity predictions are generally common across various methods). This might overinflate the recall for the negative class. An alternative approach could be to randomize the order of the amino-acids in the sequence while making sure that the sequence-identity of the randomized sequence isn't similar to any sequence in the positive class. The authors can also look at the IEDB database for compiling the negative dataset as it has sequences of non-antigens from experimental assays (one can chose sequences which were negative in atleast two independent studies)

- For the structural embeddings, the authors have relied on ESMFold for the structure of the antigenic proteins. ESMFold, while much faster than AlphaFold, lags behind in accuracy (in the backbone atoms as well) to AlphaFold/RosettaFold. There isn't any justification provided for choosing these models. There needs to be more analysis done to show how does the structural model quality affect the performance of ProVaccine. The comparison can be done against experimental structures (from PDB), AlphaFold2/3 structures, and other structure prediction methods.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The authors need to prove their methods on existing datasets. They also need to dig out why the model shows the largest improvement on the smallest dataset. The case study also needs to discuss about reason, instead of the performance only.
soundness: 2
strengths: 1. This paper proposed a deep learning supervised learning framework with a dual attention mechanism to address the key challenges of immunogenicity prediction in vaccine development.
2. This paper provided a valuable benchmark for future research in vaccine development and immunogenicity prediction.
summary: This paper introduced PROVACCINE, a novel deep-learning solution for immunogenicity prediction in reverse vaccinology, designed to identify candidate vaccines that trigger protective immune responses. The authors also constructed a comprehensive immunogenicity dataset with over 9,500 antigen sequences from various pathogens. PROVACCINE employs a dual attention mechanism, integrating pre-trained representations of protein sequences and structures, and outperforms existing methods. The paper provided a comprehensive overview of the problem, related work, and the proposed methodology.
weaknesses: 1. The work is innovative to construct a new benchmark dataset, but they didn't show novelty in algorithm development. The authors need to prove their methods on existed dataset, in addition to their own dataset.
2. The dual-attention mechanism, while sophisticated, may increase the computational cost and complexity of the model significantly. The paper does not thoroughly compare this method against simpler architectures to show that the added complexity justifies the performance gains.
3. The model’s performance is similar to XGBoost on the largest dataset, Immuno-Virus (1952/1762 positive/negative instances). However, the main improvement over XGBoost is on the smallest tumor dataset (300/477 positive/negative instances). Usually, DL is strong for big dataset and weak for small dataset, in contrary to this study. This discrepancy raised questions about the reliability and robustness of the model's performance.
3. A major contribution of this work is the construction of a data set for immunogenicity prediction and vaccine development. It’s insufficient to discuss data quality, noise, or variability, particularly for immunogenic labels sourced from different repositories. Moreover, authors should demonstrate that the new dataset is superior to existing datasets. 
4. The authors should include ablation comparisons of sequence and structural features?
5. This paper only compares web server methods, lacking comparisons with other deep learning frameworks.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: See weakness above
soundness: 2
strengths: The paper conducted generalization experiments, which is good.
The comprehensive datasets, especially the positive sequence may work as the foundation for other work.
summary: In the paper, the authors proposed a protein-language-model-based method called PROVACCINE for predicting the antigen from the amino acid sequence. They constructed the largest and most comprehensive antigen dataset through collecting published sequences. They further conducted cross-validation and some post-hoc analysis to show the effectiveness of the proposed model.
weaknesses: 1. What I am mostly concerned with is the dataset construction. It seems that the authors construct the positive dataset from the literature, which is great, but the negative data are constructed with some other computational tools. This will lead to a serious problem: if the golden standard is created using BLAST+VAXIJEN, then if we use the combination of the two will be enough, why do we bother to train a model on top of it? 

2. Continuing from 1, the authors should add the baseline method using their way to construct the training dataset, they will get a perfect score for all metrics. The logic for building negative samples is very weird and then making the model unreliable.

3. In section 5.2.1, the authors should also consider splitting the data based on the sequence similarity. If the training and testing data share very high sequence similarity, it's very likely that the model just memorizes the training and can not generalize.

4. I see XGboost and KNN are good from Figure 4 -- suggesting the novel antigens share high similarity with known ones. Actually XGBoost seems to be perfect and much better than Provaccine. The proposed model did not show superiority than the traditional similarity-based method. Moreover, as majority of these 1, 858 sequences lack immunogenicity ground truth labels, how can we ensure the enrichment is a good metric?

5. In section 5.3.2, it's unclear whether the spike protein already exist in the training dataset

6. Continuing from 6, section 5.3.2 is too rough -- it's very easy to identify the spike protein as well as the RBD as the antigen, the results can not support the effectiveness of the proposed model.

Taken together, the proposed method did not show enough novelty, and the performance is not as good as the similarity-based method.

Meta-review: The paper presents ProVaccine, a deep learning model for predicting immunogenicity using a dual-attention mechanism and a comprehensive dataset of over 9,500 antigens. It demonstrates superior performance compared to traditional methods, particularly in identifying potential vaccine candidates.

Strengths include the novel model architecture, the extensive Immuno dataset, and comprehensive benchmarking against existing methods. The dual-attention mechanism effectively integrates multimodal data, enhancing prediction accuracy.

Weaknesses involve potential biases in the negative dataset construction.

The decision to accept is based on the innovative approach to immunogenicity prediction and the potential impact of the comprehensive dataset.

Decision: Accept (Poster)