dataesr/scientific-paragraphs-categorization huggingface dataset:
DiffRBM discriminates specific receptors We tested the power to discriminate receptors specific to the 4 epitopes under consideration (GILGFVFTL, NLVPMVATV, GLCTLVAML, YLQPRTFLL) from generic sequences drawn from a bulk repertoire (background dataset). These can be seen as a proxy for non-specific receptors, since the large majority of them is not expected to respond to a specific epitope. We measured the performance at discriminating specific from generic receptors by the AUC (Figure  6A ) and the results for the 4 epitope-specific models are reported in Figures  6B-E, S16. We observed the same trend described for the models of immunogenicity, whereby the AUC of discrimination for the diffRBM units is consistently higher than the one for the full RBM. Also in this context, singling out the sequence features associated to epitope specificity, as the differential units do, enhances the model's predictive performance compared to the full RBM, where the information on those features is added to the background constraints.
Any discrimination power is lost when using the background RBM, as it should, since it has no information on epitope specificity. To further check the robustness of our model's performance, we also repeated the training and testing constructing the background model from a different set of bulk-repertoire TCRβ sequences from healthy donors (the dataset from  [Dean et al. , 2015] ). All the performance metrics are almost unaffected by this change of the background dataset (Figure  S17 ). The average AUC attained by the diffRBM units across the 4 peptide-specific models is 0. 83 ± 0. 01 with the background dataset from  [Emerson et al. , 2017]  and 0. 82 ± 0. 01 with the background dataset from  [Dean et al. , 2015]  (uncertainties are estimated over several training-test partitions, see Methods).
Discussion We have introduced a framework, based on the probabilistic graphical model of Restricted Boltzmann Machines, aimed at learning what we called 'differential' models, which capture differences of selected subsets of sequences with respect to the full set of sequences of similar type (the 'background'). More specifically, the 'differential' part of the model (the diffRBM units) is designed to learn such differences (Figures  1, S1 ). First, we applied this framework to the problem of modeling antigen immunogenicity, the ability to trigger a positive T-cell response. We showed that the diffRBM units are able to encode several features relevant to immunogenicity. More precisely, we used the parameters associated to the diffRBM units to estimate which positions are more likely to be in contact with the TCR and we showed that they can distinguish immunogenic from non-immunogenic peptides better than a full RBM, their performance being comparable to a supervised model (the 'optimal classifier') that we built and optimized for classification.
To work with samples of sufficient size, the diffRBM models of immunogenicity discussed here were trained on IEDB peptide data, which are predominantly pathogenic. The diffRBM approach is in principle applicable also to cancer neoantigen discovery, but the particular models may need to be retrained on neoantigen-specific training sets, given that substantial variation in method performance and in the immunogenicity-related predicted features between pathogenic and cancer data has been 16 evidenced  [Buckley et al. , 2022]. We did an additional analysis as a preliminary check on the diffRBM performance in the cancer setting: we assigned scores to the HLA-A*02:01-presented neoantigens from the TESLA dataset  [Wells et al. , 2020]  (11 immunogenic and 227 non-immunogenic peptides) and we obtained an AUC of immunogenic vs non-immunogenic discrimination differing significantly from the random expectation (0.
6 for the diffRBM units' scores and 0. 69 for the difference of diffRBM units' scores). We also learnt differential models from datasets of epitope-specific T-cell receptors to capture the sequence features underlying their ability to bind to a given epitope, and we have successfully tested the models' power to discriminate epitope-specific from generic receptors and to identify CDR3β residues in contact with the antigen. We looked at the task of distinguishing a small number of antigen-specific receptors from the bulk repertoire (forming the background), a task which reflects many typical scenarios of application. Indeed, antigen-specific receptors constitute a minority of the repertoire, with experimental estimates of the fraction of the TCR repertoire reactive to a given epitope ranging between 10 -6 and 10 -4  [Yates, 2014].
It is at this task that diffRBM performs as well as the state-of-the-art methods. The performance by diffRBM decreases when we consider as negatives receptors with a different antigen specificity  [Meysman et al. , 2022] , because the differential units can learn features shared between antigen-specific sequences (e. g. , being expressed by CD8+ T cells). More fine-tuned choices of the background (e. g. , with only CD8+ T-cell receptors) would be needed to improve performance for this particular task. The diffRBM model of specific TCR response can be formulated in an equivalent way for the CDR3 region on the other chain of TCRs (α chain). In this case, one uses CDR3α sequences from healthy TCR repertoires and samples of antigen-specific CDR3α as respectively the background and the selected dataset.
The entropy values of background and selected sequences obtained through our models of immunogenicity and TCR specificity highlight a reduction of diversity in selected sequences stemming from the enrichment in distinctive sequence patterns (Figure  S20 ). Such a reduction has substantial variability across epitopes in the case of TCR specificity models (Figure  S20B ), reflecting the existence of more or less well-defined sequence motifs underlying epitope specificity. In this paper, the application of our approach has been clearly limited by the scarcity of experimentally confirmed immunogenic peptides for the majority of HLA alleles, as well as TCR specific to many other epitopes. Even for the HLA alleles and epitopes we considered, available datasets cover only a small fraction of HLA-specific peptides that have the potential to be immunogenic and TCR that can recognize given peptides, and pool together data obtained through a heterogeneous set of T-cell response assays, a source of additional noise in the data (Methods).
Such limitations in the datasets might affect the performance of all the methods we tested. However, we emphasize here the methodological novelty of learning differential models of immunogenicity and antigen specificity. Our study provides robust proof-of-concept results that a differential modeling approach is an effective tool to extract the sequence features underlying the specificity of binding between antigens and T-cell receptors, and to generate insight into the structural basis of this process by identifying putative contact sites from the sequence alone. Furthermore, relying on unsupervised learning has the advantage that negative samples, whose construction is somewhat arbitrary, are not needed. In addition our model is generative, i. e. , it could a priori be used to produce new, putative sequences of antigens or TCR with desired properties, testable in experimental setups.
Nevertheless, in terms of future work, leveraging structural information on the TCR-pMHC complex and its estimated binding energy along with sequences, as investigated by  [Riley et al. , 2019 , Lin et al. , 2021 , Milighetti et al. , 2021 , Karnaukhov et al. , 2022] , will contribute to improving the modeling strategy presented, especially given the expected increase in the number of available crystallographic structures. These tasks will require ad hoc adjustment of the model architecture, to integrate structural and energetic features into its input, and hence goes beyond the scope of the more general and flexible scheme we aimed to proposed here, which relies on a 'background' dataset of large size, and a small subset of selected data with distinctive property that justify their selection.
DiffRBM architecture The core idea of what we refer to as a 'differential' probabilistic model is to learn a distribution for sequence data σ with the parametric form: P (σ) = 1 Z e -H(σ) H(σ) = H b (σ) + H d (σ) (1) where Z is simply a normalization factor, Z = σ e -H(σ). H b specifies the background distribution σ) , where σ). The background distribution P b (σ) is inferred from what we call a 'background' dataset D b , a large-size dataset that enables P b to embed the main statistical properties of the data of interest with high accuracy (thanks to its large size). On the other hand, H d contains the parameters that are learnt on top of the background distribution from a set of 'selected' data D s , picking up their statistical differences with respect to the background distribution. P b through P b (σ) = 1 Z b e -H b ( Z b = σ e -H b ( While this formulation allows for a rather generic choice of both H b and H d , here we assume that they are parametrized in terms of Restricted Boltzmann Machines  [Hinton, 2002, Hinton and Salakhutdinov, 2006].
Thus for an RBM as background one obtains: H b (σ) = -   N i=1 g b i (σ i ) + M b µ=1 Γ b µ I b µ (σ)   (3) where we have set I b µ (σ) = i w b iµ (σ i ) and Γ b µ I b (σ) = log dh b e -U b µ (h b )+h b I b (σ). The parameters g b i and w b iµ , as well as the parameters defining the potentials U b , are learnt from the background dataset D b by maximizing the log-likelihood: 1 |D b | σ∈D b log P b (σ) = log P b (σ) D b (4) After the background distribution has been learnt, we learn the 'differential' units of the architecture that we have named differential Restricted Boltzmann Machine (diffRBM), corresponding to the probability distribution: P (σ, h b , h d ) ∼ exp   N i=1 g d i (σ i ) + g b i (σ i ) - M d µ =1 U d µ (h d µ ) - M b µ=1 U b µ (h b µ ) + i,µ h d µ w d iµ (σ i ) + i,µ h b µ w b iµ (σ i )   (5) see also Figure  S1.
Hence: H d (σ) = -   N i=1 g d i (σ i ) + M d µ =1 Γ d µ I d µ (σ)   (6) where we have defined I d µ (σ) = i w d iµ (σ i ) and Γ d µ I d (σ) = log dh d e -U d µ (h d )+h d I d (σ). The parameters featuring in H d (σ) and defining the 'differential' RBM units are learnt from the dataset D s by maximizing: 1 |D s | σ∈D s log P (σ) = log P (σ) D s with P (σ) = M b µ=1 dh b µ M d µ =1 dh d µ P (σ, h b , h d ) (7) where P (σ, h b , h d ) is given by (  5 ). Let us define a compact notation Ψ d for the set of parameters associated to the differential units, Ψ d = {g d i (σ i ), w d iµ (σ i ), ξ d µ } , where ξ d µ is a shorthand for the parameters specifying the shape of U d µ (in all our implementations, both U b µ and U d µ are set to dReLU potentials, a choice that has been shown to confer high expressivity  [Tubiana et al.
Regularization terms can be added during the training to control the values of the inferred parameters. Here we used a L 1 -type regularization over the weights, which enforces sparsity to prevent overfitting (see  [Tubiana et al. , 2019]  for details); following the convention in  [Tubiana et al. , 2019]  we denote by λ 1 2 the coefficient setting the magnitude of such regularization term. In addition, a L 2 -type regularization on the fields was used for all models to control for large values stemming from undersampling, and following a standard choice  [Cocco et al. , 2018]  its magnitude was set to the inverse of the size of the training dataset (hence to 1/|D b | for background models and 1/|D s | for differential models). The predictions of peptide immunogenicity or epitope specificity rely on the assignment to sequences σ of scores.
Using (1), the score of the full RBM is given by the log-likelihood: L(σ) = log P (σ) = -H b (σ) -H d (σ) + constant (9) where H b (σ) and H d (σ) are given by (  3 ) and (  6 ) and the constant stands for a sequence-independent term coming from the partition function. Analogously, the background RBM score and the diffRBM units' score are respectively: L b (σ) = -H b (σ) + constant (10) L d (σ) = -H d (σ) + constant (11) The entropies of background RBM and full RBM were estimated respectively as: S Background = - σ P b (σ) log P b (σ) S RBM = - σ P (σ) log P (σ) ( 12 ) where log is meant as a natural logarithm, hence the entropy values in Figure  S20  are expressed in nats. In terms of software implementation, diffRBM is coded via a series of additional functions to execute a differential learning on top of the RBM Python implementation from  [van der Plas et al.
Sequence datasets for the immunogenicity model The differential models to describe immunogenicity were trained on sets of immunogenic peptides collected from the Immune Epitope Database (IEDB)  [Vita et al. , 2019] , where the database entries were filtered through the following steps. Firstly, the curated set of HLA ligands tested in T cell assays was downloaded from IEDB (file tcell_full_v3. csv from http://www. iedb. org/database_export_v3. php, accessed in December 2021). We selected from this file linear, human peptides with a given HLA restriction (e. g. HLA-A*02:01), limiting the search to peptides of length 8-11 amino acids like in  [Bravi et al. , 2021b]  and presented by HLA of class I (i. e. , targeted epitopes of killer T cells). Following  [Calis et al. , 2013] , we required the peptide (and not the full protein or the pathogen) to be the first immunogen (by setting the field Antigen Epitope Relation = 'Epitope') and we excluded T-cell response experiments with a restimulation step (by discarding 'Restimulation in vitro' from the field In Vitro Process Type).
We choose only the HLA-I alleles for which the filtering steps just described allowed us to recover at least 200 immunogenic peptides and for which at least one TCR-pMHC structure was available in the Protein Data Bank To train the antigen presentation model, playing here the role of the background model, we relied on the sets of 8-11 amino acid long peptides extracted from IEDB by the RBM-MHC algorithm as described in  [Bravi et al. , 2021b] , choosing the option of peptides from HLA binding affinity assays rather than mass spectrometry, to avoid biases in the amino acid statistics of presented antigens that might be due to mass   [Reynisson et al. , 2020]  are obtained with the option -BA (predictions from the training on binding assay data).
Sequence datasets for the T-cell specificity model Each differential model of specific T-cell response to a given peptide was trained on sets of T-cell receptors experimentally validated to be specific to the peptide, which were collected from the VDJdb database  [Shugay et al. , 2018 , Bagaev et al. , 2020]  (file vdjdb. txt downloaded from https://vdjdb. cdr3. net in July 2021). More precisely, in the database we selected all the human TCRβ chains fully annotated with their V and J segment and labelled to be specific to the given peptide (for example, for the Influenza M1 58 -specific model we set antigen. epitope = 'GILGFVFTL'). We constructed the final training sets from the CDR3β sequence, the V and J segments identity of these entries, removing replicates with the same CDR3β and V/J annotation.
We obtained datasets of epitope-specific receptors of size |D For the background model (here meant as a model of the normal bulk TCRβ repertoire), we considered: the dataset assembled by  [Isacchini et al. , 2021]  pooling together unique TCRβ clones from the 21 743 donors of the cohort in  [Emerson et al. , 2017] , with a total of ∼ 9 × 10 7 sequences; the dataset of TCRβ sequences from healthy donors from  [Dean et al. , 2015] , already preprocessed by  [Isacchini et al. , 2021]  and composed of ∼ 2 × 10 6 sequences. For training the background RBM on these data, we used smaller datasets of 10 6 sequences that could be more easily handled by randomly subsampling from the entire datasets from  [Emerson et al. , 2017]  and  [Dean et al. , 2015].
Data preprocessing and formatting RBM and PWM-based approaches require sequence inputs of fixed length, hence we performed an alignment. Background datasets are aligned to obtain same-length sequences, following the alignment procedures described in  [Bravi et al. , 2021b]  for peptides and  [Bravi et al. , 2021a]  for CDR3β amino acid sequences. The length of the alignment is set to 9 (9 being the typical length of HLA-I ligands) and to 20 in the case of CDR3β. These alignments serve as seeds to learn Hidden Markov Model profiles of length 9 and 20, in such a way that the selected datasets can be aligned against the profile built from the corresponding background dataset (see  [Bravi et al. , 2021b]  for more details). In the models for TCR sequences, as a default, the input combines the aligned CDR3β amino acid sequence to the V segment type and the J segment type, all converted into numerical values varying within an interval of appropriate length (length = 21 for the CDR3β positions, standing for the 20 amino acids + 1 gap; length = 48 for the V type, length = 13 for the J type).
Crystallographic structures from PDB We downloaded, from the Protein Data Bank www. rcsb. org [Berman et al. , 2000]  as of February 2022, the TCR-pMHC crystallographic structures with 9 amino acid-long peptides where the HLA complex is HLA-B*35:01, HLA-A*02:01, or HLA-B*07:02. We excluded the structures with modified/non-peptidic epitopes and with incomplete TCR chains. As a result, we obtained 5 structures for HLA-B*35:01, 56 for HLA-A*02:01, and 1 for HLA-B*07:02. Some of the 56 HLA-A*02:01 structures describe TCRs in contact with the peptides we considered for the differential models of specific TCR response (3 for the Sars-Cov-2 epitope YLQPRTFLL, 3 for the CMV epitope NLVPMVATV, 1 for the EBV epitope GLCTLVAML, and 8 for the Influenza epitope GILGFVFTL). For each structure, we estimated the positions along the peptide in contact with the TCR, using a standard cutoff at 4 Å  [Rossjohn et al.
DiffRBM training and model selection The first step of the diffRBM training consists of training the background model on the background dataset. For the model of immunogenicity, we trained allele-specific presentation models with an RBM architecture by running the RBM-MHC algorithm  [Bravi et al. , 2021b]  with default parameters (10 hidden units, λ 1 2 = 0. 001). The peptide data used by default as training set by RBM-MHC to build allele-specific presentation models are downloaded from IEDB, see  [Bravi et al. , 2021b]. The RBM-MHC algorithm internally aligns peptide sequences of its training dataset to the reference length of 9 amino 22 acids; we utilized the same alignment routine to align the immunogenic peptides against the seed given by the RBM-MHC training data. The second step is training the diffRBM units on the 'selected' datasets, here immunogenic peptides and antigen-specific CDR3βs (section 5.
2). Note that, for the applications discussed in this paper, the training on the background dataset and the one of the selected dataset are implemented consecutively. While the diffRBM architecture allows for a parallel training, the appeal to the sequential training here is justified by the size of the datasets at out disposal (|D b | |D s |) and by the particular problem under consideration (modeling an enrichment of specific features, occurring on top of baseline features, which do not change as feedback from the former). The datasets D s , collected as described in section 5. 2, are divided into a training set with 80% of the data (used for training and for model selection) and a test set with the remaining 20% (used for model validation, see section 5. 5), repeating this split 50 times.
Having fixed the background model, we used the largest available dataset of immunogenic sequences (the one for HLA-A*02:01) to perform model selection by cross-validation, as follows. We further divided randomly each of the 50 training sets into a set actually used for training and a validation set (with respectively 80% and 20% of the training set). We used this training/validation partitions to select optimal hyperparameters for the differential part (number of the number of hidden units and regularization penalty λ 1 2 ), by training diffRBM models on the training set at varying hyperparameters and monitoring the average diffRBM units' score (11) on the validation sets (Figure  S3A-B ). We also performed additional checks on the diffRBM units' AUC of immunogenic vs non-immunogenic discrimination with different hyperparameters (Figure  S3C ) and in a control case (Figure  S3D ).
For the model of T-cell specificity, the background is given by an RBM trained on a random subsample of 10 6 CDR3β sequences either from the bulk repertoires in  [Emerson et al. , 2017]  or the ones in  [Dean et al. , 2015]  (section 5. 2. 2), choosing the optimal RBM architecture (100 hidden units, λ 1 2 = 0. 001) by cross-validation (Figure  S14A-B ). For cross-validation, we used 50 random partitions of into 10 6 data for training and 2 × 10 5 data for validation randomly chosen from the repertoire  [Emerson et al. , 2017]  into training and validation sets (with respectively 80% and 20% of the data). The grid search for optimal hyperparameters for the differential part was then carried out at fixed background model, using the largest dataset of peptide-specific receptors (NLVPMVATV) and partitioning it 50 times at random into training and validation sets (with respectively 80% and 20% of the data, Figure  S14C-F ).
PWM-based approach. We considered a type of differential model based entirely on the information in single-site amino frequency, where both the background model and the differential part are specified only by single-site fields (g b i (σ i ) and g d i (σ i ) respectively). The equations for learning g b i (σ i ) and g d i (σ i ) are: e g b i (σi) σi e g b i (σi) + λ 2 g b i (σ i ) = f b i (σ i ) (13) e (g b i (σi)+g d i (σi)) σi e (g b i (σi)+g d i (σi)) + λ 2 (g b i (σ i ) + g d i (σ i )) = f i (σ i ) (14) where f b i (σ i ) is the frequency of amino acid σ i at position i in the background dataset D b , f i (σ i ) is the frequency of amino acid σ i at position i in the dataset D s and i = 1,. , N. We use the same L 2 23 regularization penalty that we chosen for the RBM fields, and the parameter λ 2 sets its strength (like for the RBM-based approaches, λ 2 = 1/|D b | for background models and λ 2 = 1/|D s | for differential part).
This approach is equivalent to learning from the background and selected datasets Position Weight Matrices (PWMs), probabilistic models of amino acid usage that treat all sequence positions as independent. By defining P b i (σ i ) = e g b i (σi) / σi e g b i (σi) and P i (σ i ) = e (g b i (σi)+g d i (σi)) / σi e (g b i (σi)+g d i (σi)) and given the independent-site assumption of PWMs, the background data PWM probability and the selected data PWM probability are respectively recovered as P b (σ) = i P b i (σ i ) and P (σ) = i P i (σ i ). Since P b i (σ i ) and P i (σ i ) are learnt to closely reproduce the frequencies f b i (σ i ) and f i (σ i ), their predictions are the ones that are inferable from single-site amino acid frequency alone in the respective training datasets (hence in Figures  2, 5 , S4, S5, S15 we refer to such predictions by 'Amino acid frequency').
SONIA We implemented a SONIA model using the software package soNNia from  [Isacchini et al. , 2021]  and setting the option deep=False. For consistency with the background of diffRBM, we first learn a background model from 10 6 randomly assembled from the universal donor repertoire from  [Emerson et al. , 2017] , next we learn a model from epitope-specific repertoires and we take, as SONIA score, the difference between the scores assigned by these two models. The L 2 -type regularization strength is set to the default value (zero); higher regularizations applied to one of our test sets did not provide visible improvement of the log-likelihood.