dataesr/scientific-paragraphs-categorization huggingface dataset:
We used a whole genome sequencing dataset of H9 hESCs to identify informative genomic SNPs along the X chromosome which can be used for allelic expression analysis (GSM1227088). Reads were aligned using bowtie2 and PCR duplicates were filtered out using MarkDuplicates from picard (2. 22. 0) as previously described. For proper tags formatting of BAM files, AddorReplaceReadGroups from picard was used with the following options: SO ="coordinate", RGID="id", RGLB="library", RGPL="platform", RGPU="machine", RGSM="sample". BAM files were further processed using GATK (4. 1. 2. 0)  (McKenna et al. , 2010)  for SNP identification. BaseRecalibrator and ApplyBQSR were used to generate recalibrated BAM using high confidence SNPs referenced in Mills_and_1000G_gold_standard. indels. hg38. vcf. gz (annotation of indels) and dbsnp_146.
ATAC-seq data analysis ATAC-seq raw data were analyzed using the atacseq pipeline (1. 2. 1) (https://github. com/nfcore/atacseq/) from nf-core  (Ewels et al. , 2020)  using the default parameters. Briefly, reads were aligned to the human genome (hg38) using BWA (0. 7. 17). Duplicates were marked by picard and reads mapping to mitochondrial DNA and blacklisted regions were removed. BigWig files were generated using deeptools as described above.
The accurate prediction of peptide-MHC class I binding probabilities is a critical endeavor in immunoinformatics, with broad implications for vaccine development and immunotherapies. While recent deep neural network based approaches have showcased promise in peptide-MHC prediction, they have two shortcomings: (i) they rely on hand-crafted pseudo-sequence extraction, (ii) they do not generalise well to different datasets, which limits the practicality of these approaches. In this paper, we present PerceiverpMHC that is able to learn accurate representations on full-sequences by leveraging efficient transformer based architectures. Additionally, we propose IMGT/RobustpMHC that harnesses the potential of unlabeled data in improving the robustness of peptide-MHC binding predictions through a self-supervised learning strategy.
Predicting peptide-MHC binding probability is a challenging computational task due to the vast combinatorial space of possible peptide sequences and the subtle nuances that govern their interactions with HLA molecules. Traditional approaches, such as scoring matrices and structural modeling, have made valuable contributions to this field. However, they often struggle to capture the intricate, high-dimensional relationships and the sequence patterns that are vital for accurate predictions. In recent years, machine learning, especially deep learning techniques have emerged as powerful tools for addressing this challenge. These methods leverage vast datasets of experimentally measured peptide-HLA binding probabilities to learn complex patterns and relationships. Nonetheless, existing models still face limitations in effectively encoding and representing the structural and sequential features of both peptide and MHC sequences.
In our study, instead of focusing solely on peptide sequences, we take a broader approach by considering the full Major Histocompatibility Complex (MHC) sequences. By encompassing the entirety of the MHC molecule, including both the peptide-binding groove and the surrounding regions, our model gains a nuanced understanding of the structural context in which peptides bind. This holistic approach not only accounts for polymorphism within the MHC gene but also captures the dynamic interplay between residues that may influence peptide presentation on the surface of MHC molecule. therefore, our model is poised to provide more accurate and contextually relevant predictions of peptide-HLA binding probabilities. While recent advances in deep neural networks have shown promise in peptide-HLA binding predictions, their performance tends to drop when confronted with datasets having slightly different distribution of peptide lengths which limits the applicability of these approaches.
Therefore, we present a Perceiver IO based architecture (PerceiverpMHC) for predicting pMHC bindings on full HLA sequences. In contrast to existing approaches  [2, 10, 31]  that require hand-designed pseudo-sequences, we show that attention mechanisms can learn efficient representations enabling us to work with full sequences. Secondly, we observe that the performance of existing approaches significantly drops on datasets with slightly different distribution of peptide lengths. To mitigate this issue, we propose RobustpMHC that leverages self-supervised learning strategy resulting in superior performance over the state-of-the-art approaches across six different datasets. Thirdly, we introduce a new dataset CrystalIMGT dataset, a set of mass spectrometry observed peptide-MHC pairs, which presents a challenge to the existing approaches due to significantly different data distribution.
Additionally, we propose a transfer learning technique that can substantially enhance performance using only 10% of the available data for the CrystalIMGT dataset. Furthermore, we demonstrate the capability of the features acquired through our method to transfer to the prediction of immunogenicity data for neoantigens. This transfer is assessed through a comparison with the nine state-of-theart approaches. In conclusion, we present a robust pMHC binding prediction strategy and an exhaustive evaluation benchmark, which we believe will advance the field of pMHC binding prediction. Once this paper is accepted, we will make our implementation, trained models, and datasets available, to facilitate reproducibility. Additionally, we have deployed a user-friendly web interface for predicting peptide-MHC binding.
Results We designed our experiments to investigate the following research questions: 1. Do we need to depend on hand-crafted pseudo-sequence extraction, or can neural networks autonomously discern crucial features? 2. How does PerceiverpMHC compare with other Transformer architectures, and how does RobustpMHC's learning strategy compare against other self-supervised learning approaches such as data augmentation and contrastive learning? 3. To what extent do the introduction of mutations contribute to enhancing the robustness of RobustpMHC? How many mutations are sufficient, or does a large number of mutations degrade the performance? 4. Does RobustpMHC generalize well across diverse datasets, enhancing its practical applicability while effectively accommodating diverse peptide lengths in sequences? To answer (1), we compare PerceiverpMHC with recent pseudo-sequence approaches and DeepAttentionPan (the single full sequence base approach ) on 6 datasets in section 1.
1. In section 1. 3 we show that the learned representations of RobustpMHC can be transferred to datasets that are even significantly different than the training dataset. Finally, we present ablation studies in section 1. 4 to answer the research questions  (2) (3) (4). With regard to (4), we show that across 8 different datasets RobustpMHC is either state-of-the-art approach or at par with state-of-the-art approach for that dataset in section 1. 2.
Full sequence evaluation We initially evaluate whether neural networks can inherently learn which amino acids are crucial for binding given the full MHC sequence or if we need to design hand-crated pseudo-sequences. To this end, we evaluate the performance of PerceiverpMHC on four publicly available benchmarks: independent and external set from Anthem  [33]  dataset, Neoantigen  [10]  and HPV  [8]  datasets and compare with the state-of-the-art pseudo-sequences based approaches like TransPHLA  [10] , capsNet  [21] , NetMHCpan 4. 1  [44]  and BigMHC  [2]. The independent set from Anthem dataset  [33]  contains 112 types of HLA alleles, whereas the external set contains five HLA alleles. The Neoantigen data  [10]  comprises non-small-cell lung cancer, melanoma, ovarian cancer and pancreatic cancer from recent works including 221 experimentally verified pHLA binders.
The HPV dataset  [10]  278 experimentally verified pHLA binders from HPV16 proteins E6 and E7, consisting of 8-11-mer peptides. Following TransPHLA  [10]  we report four metrics Area Under Curve (AUC), Accuracy, Matthews correlation coefficient (MCC) and F1 scores on independent set and external set. For the datasets such as Neoantigen and HPV that have only experimentally verified binders, we report the true positives and the false negatives during prediction. Furthermore, we compare our PerceiverpMHC approach with deep-AttentionPan  [19] , which is also a full sequence based deep pHLA binding prediction approach. It can be seen from Figure  1a -c that even on full sequences PerceiverpMHC is at par with the state-of-the-art pseudo-sequence based approaches on three evaluation datasets. While PerceiverpMHC is slightly inferior on the HPV dataset (refer Figure  1d ) as compared to TransPHLA and CapsNet, it is superior to other approaches including NetMHCPan4.
Robustness evaluation We next evaluate how well our self-supervised robust training generalises across different datasets for peptide-HLA-I binding prediction. To this end, we evaluate RobustpMHC on six datasets, independent and external splits form Anthem datasets, HPV and Neoantigen dataset. We further compare them on Covid dataset and IEDB binary evaluation set. The current coronavirus disease 2019 (COVID-19) outbreak is a worldwide emergency, as its rapid spread and high mortality rate has caused severe disruptions. Understanding the immune patterns of COVID-19 is very crucial for mechanisms of SARS-CoV-2-induced immune changes, their effect on disease outcomes, and their implications for potential COVID-19 treatments  [51]. The COVID dataset consists of 29 experimentally verified positive and 89 negative samples from different sources  [32]   [46].
The IEDB dataset  [12]  contains three different measurements IC50, T1/2 and binary. Here we consider the binary set which consists of 3845 experimentally verified EL(Mass spectrometry data include naturally presented MHC ligands, referred to as eluted ligands)  [5]  binding samples and 3656 negative samples. Similar to Anthem, IEDB has been one of the prominent publicly available benchmarks which contains slightly different distribution as compared to Anthem therefore it provides a diverse set for evaluating the generalization of the approaches. It can be seen from Figure  2  that for independent set (Figure  2a ), which has very similar peptide length distribution as training data, the performance of Robust-pHMC is slightly lower as compared to CapsNet. In contrast, on external dataset (Figure  2b ) RobustpHMC shows superior performance.
Even on Neoantigen data RobustpHMC shows slightly better performance as shown in Figure  2c. On HPV dataset (Figure  2d ) RobustpHLA is slightly behind TransPHLA but significantly better than CapsNet and NetMHCpan-4. 1. On COVID dataset (Figure  2e ) RobustpHLA is again superior, marginally outperforming NetMHCpan-4. 1, and significantly outperforming TranspHLA and CapsNet. While on IEDB dataset (Figure  2f ), RobustpHMC performs marginally better than CapsNet and TranspHLA and notably better than NetMHCpan-4. 1. In conclusion, RobustpMHC obtains state-of-the-art results on four out of six datasets. Furthermore, RobustpHMC demonstrates consistent performance across all six dataset which showcases the robustness of our approach. It is worth noting that RobstpMHC and PerceiverpMHC work on full sequences while others baselines in Figure  2  require psuedo-sequences.
Transfer learning evaluation While the learning based approaches have shown promising results for pHLA binding prediction, their performance steeply deteriorates for prediction on data with significantly different distribution. In Figure  3a -e, we plot the distribution of peptide length across different datasets. It can be seen that distribution of Anthem independent set is very similar to the train set, thus the performance of end-to-end approaches are better as can be seen in Figure  2a. On the other hand, for HPV dataset where the distribution of peptide length is different as can be observed by comparing Figure  3d  with Figure  3a , the performance of all approaches significantly deteriorates as depicted in Figure  2d. Hence, for an extensive evaluation of out-of-domain (OoD) generalisation we select two datasets: (i) Neopeptide dataset presented by a very recent method BigMHC  [2].
This dataset comprises 198 positive instances and 739 negative instances, and it has served as a benchmark for transfer learning. (ii) We compile a dataset collected from IMGT/3Dstructure-DB dataset  [32] , which we refer to as CrystalIMGT is an experimental verified Dataset comprising of 14997 crystallographic experimental samples. The peptide length distributions of these datasets are shown in Figure  3e ,f respectively. It can be seen from Figure  3f -h that existing approaches suffer from significant performance degradation on these datasets and especially CrystalIMGT. To tackle such large distribution shift, we present a transfer learning paradigm for RobustpMHC. The CrystalIMGT we select 10% of the data randomly and consider them as an independent training set for transfer learning, while for the Neopeptide dataset, we follow the BigMHC and use the independent training and testing sets provided by them for fair evaluation.
Ablation studies Comparison with data augmentation and contrastive learning: We next evaluate the contribution of robust training, for we compare RobustpMHC without robust training i. e. PerceiverpMHC. Furthermore, it can be argued that whether equation (  1 ) is the most optimal way of robust training? To answer this question we compare our robust training pipeline with two popular self-supervised learning approaches: (i) data augmentation  [53]  and contrastive learning  [36]. In data augmentation the existing dataset is augmented with a synthetic data by transforming the original sample. Here we consider random mutation as a transformation and retrain PerceiverpMHC on the combined dataset, we refer to this model as PerceiverpMHC-Re. (ii) In contrastive learning (specifically noise constrastive learning) for each mini-batch of the training data each sample is passed through a transformation.
Then infoNCE loss  [36]  is used to learn representations robust to a given transformation. Here again considering random mutations as transformation, we train a variant of our model that minimizes infoNCE loss instead of the proposed loss 1. We refer to this variant as PerceiverpMHC-CE. From supplementary Figure  S3 , it can be seen that our approach (RobustpMHC) consistently outperforms PerceiverpMHC, PerceiverpMHCRe and PerceiverpMHC-CE, which validates the contribution of our loss function in obtaining better generalisation across diverse datasets. Other transformer architectures for full sequence pMHC binding prediction: To validate the selection of Perceiver IO as our base module for encoding peptide-MHC sequence we train the vanilla transformer  [54]  on full MHC sequence and compare with PerceiverpMHC and another popular efficient transformer architecture Reformer  [26]  which we call ReformerpMHC.
Here we explicitly study the effect of peptide length on performance of RobustMHC for Anthem external dataset. it can be seen from supplementary Figure  S5 , that RobstMHC's performance drops for peptides with lengths greater than 9, since maximum amount of training data belongs to peptides with lengths 8,9. It is a common issue of leaning based approaches and similar observations have been reported by TranspHLA and CapsNet.
Method 2. 1 Datasets Anthem dataset: The Anthem dataset  [33]  comprises of three types of dataset the training set for model training and model selection, the independent test set and the external test set for model evaluation and methods comparison. The data sources for the training and independent test set are the same (i) four public HLA binders databases (  [12] ,  [43] ,  [28]  and  [41] ), (ii) allotype-specific HLA ligands identified by mass spectrometry in previously published studies as described in TransPHLA  [10]. The training dataset consists of 112 types of HLA alleles, including 3,59,166 EL and 17,95,830 negative instances. Peptides of negative data are sequence segments that are randomly chosen from the source proteins of IEDB HLA immunopeptidomes. The independent and external set consists of 112 types of HLA alleles with 85,876 positive and 85,562 instances.
Because the source of the independent test set and the training set are the same, the data distributions for the training set and independent test set are very similar, TranPHLA  [10]  used an additional external set consisting of 5 types of HLA alleles with 51,984 positive and 51,881 negative instances was used to perform a fairer evaluation. Neoantigen dataset: The Neoantigen dataset presented by the Chu et al. [10] consists of 250 Neoantigen samples was compiled from experimentally verified nonsmall-cell lung cancer, melanoma, ovarian cancer and pancreatic cancer pHLA binders. HPV dataset: The human papilloma virus (HPV) dataset presented by Bonsack et al. [8]  studies one of the most common sexually transmitted disease. It comprises of 278 experimentally verified pHLA binders from HPV16 proteins E6 and E7, consisting of 8-11-mer peptides.
Following  [8] , we also consider 'binder' according to IC50<100µM for the HPV vaccine data. COVID dataset: The coronavirus disease 2019 (COVID-19) belongs to family SARS-CoV-2-induced pneumonia, named by World Health Organization as coronavirus disease 2019 (COVID-19), has been declared a pandemic on the 11th of March 2020 since its first appearance in Wuhan, China, in December 2019  [13]. Coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, became a worldwide pandemic affecting millions of people  [47]. Several efforts were made to identify HLA-related susceptibility to SARS-CoV-1 also belong to this family  [6]  to understand the variation in HLA affect both susceptibility and severity of COVID-19 infection could help identify risk and may support future vaccination strategies.
Experimental Dataset collected from IMGT/3Dstructure-DB structure database  [32]  contains crystallographic experimental data consists of 42 experimentally verified samples. CrystalIMGT dataset: We utilized the IMGT/3Dstructure-DB database  [14]  to provide a unique resource of expertise with detailed specific annotations on structural data of IG, TR, MHC and RPI, from human and other vertebrate species, extracted from the Protein Data Bank PDB  [7]. The IMGT/3Dstructure-DB is publicly available structure database  [32]  comprise experimentally verified 3D data contains 15395 pHLA binders. IMGT/3Dstructure-DB integrates data from sequence and structural sources and provides identification of IMGT genes and alleles expressed in the IG, TR and MHC with known 3D structures. Neoepitope dataset: For evaluating the transfer learning capability of our approach, we follow the BigMHC  [2].
We use the training (positive = 1,407; negative = 4,778), and validation (positive = 173; negative = 515) split for transfer learning our projection block keeping the weights of Perceiver IO block frozen. The transfer learning evaluation dataset consisted of a total of 198 immunogenic and 739 non-immunogenic neoepitopes after removing the intersection with all other pMHC data, were compiled from different sources such as NEPdb  [58] , Neopepsee  [24] , TESLA  [56] , and a data collected from 16 cancer patients using the MANAFEST Albert et al. [2]. They only kept peptides of length at least 8 and at most 11 and peptides with dummy amino acid 'X' were removed. MHC-I database: IPD-IMGT/HLA da-tabase  [4]  is the official repository for the WHO Nomenclature Committee for Factors of the HLA System that receives submissions from laboratories in over 46 countries and active website uses from users in over 150 countries.
Compared methods We exhaustively evaluate our approach by conducting two types of case studies, on one hand we compare our approach on six different datasets for validating robustness and on the other hand, we evaluate transfer learning capability of our approach on two datasets. During these studies we compared with the state-of-the-art methods such as TranspHLA, CapsNet, the IEDB recommended method (NetMHCpan-4. 1), over fifteen IEDB baseline methods and transfer learning based method (BigMHC) published very recently, RobustMHC achieves most consistent performance across all experiments. We here explain few selected recent approaches and highlight their contrast with RobustMHC. It is worth noting that aprat from our approach only DeepAttentionPan can take full sequences, other approaches require pseudo sequences.
HLAthena  [48]  uses three single-layer neural networks trained on mass spectrometry data to predict presentation on short peptides with length in the range  [8, 11]. Each of the three networks trained on a separate peptide encoding: one-hot, BLOSUM62  [3] , and PMBEC40. In addition, the networks consumed peptide-level characteristics, and also amino acid physiochemical properties. The outputs of these networks were used to train logistic regression models that also accounted for proteasomal cleavage, gene expression and presentation bias. HLAthena also saw performance gains when considering RNA-seq as a proxy for peptide abundance. CapsNet  [21]  proposes a capsule neural networks architecture for learning the representation of Anthem's datasets. CapsNet-MHC architecture, is built upon four major units: a) data encoder, b) feature extractor, c) binding dependencies extractor, and binding predictor.
We use efficient tranformer models based on latent cross-attention to efficiently extract better features, we further propose selfsupervision to make the models generalise better across diverse datasets. BigMHC presents a transfer learning paradigm to predict binding on immunology datsets, we further show that the features learned by RobustpMHC also transfers well to immunology datatset (we showcase better performance as compared to BigMHC on one of the two metrics). Moreover, on other datasets BigMHC's performance is far inferior as compared to ours. Hence, our approach can tackle full sequences and is able to show consistently better results across diverse datsets.
Notably, our model maintains robust performance even in the presence of mutations, which is vital for real-world applications. Collectively, the combination of these techniques propels our Perceiver IO-based model to the forefront of peptide-HLA binding prediction, offering a promising tool for immunotherapy and vaccine design. We believe that our model highlights the importance of harnessing advanced deep learning techniques in tackling complex biological problems. Limitations: While IMGT/RobustpMHC c-onsistently demonstrates improved performance compared to the state-of-the-art approaches across a wide range of datasets, it still relies on additional transfer learning data for CrystalIMGT and Neopeptide datasets. Notably, the performance of all approaches remains suboptimal on these specific datasets.
Consequently, we are convinced that the field of peptide-MHC (pMHC) prediction could benefit from extensive pre-training and domain adaptation techniques to bridge this performance gap, and IMGT/RobustpMHC represents a crucial step in this direction. Furthermore, a common issue we encounter across all datasets is the generation of negative samples for training these approaches, which is typically done randomly without experimental verification. Consequently, these negative samples may contain false negatives. We acknowledge that this is one of the shortcomings of existing approaches, including our own, and we intend to address this issue in our future research endeavors. Webserver availability. The webserver is available at https://www. imgt. org/RobustpMHC/.