dataesr/scientific-paragraphs-categorization huggingface dataset:
The (hyper-)parametric search for optimal regularization (λ 2 1 ) and number of hidden units (N h ) was made from the trend of the RBM log-likelihood on a held-out validation dataset, the aim being to achieve a good fit but to avoid overfitting. Supplementary Figs. 3A-B  illustrate such (hyper-)parametric search for a multi-allelic RBM. Low regularizations achieve a better fit of training data (see log-likelihood values for the training dataset); when selecting a low regularization (as λ 2 1 = 0. 001), the log-likelihood over the validation dataset starts to decrease beyond N h = 10, indicating overfitting. Given these trends, we trained the multi-allelic RBM models (Figs. 2C-E ) with N h = 10 and λ 2 1 = 0. 001. This choice is further supported by considering, in Supplementary Fig. 3C , the accuracy of HLA classification (see below for its definition), which reaches an optimal value for λ 2 1 = 0.
The choice of one-hot encoding is justified by the fact that, for the sake of discriminating motifs, we select peptides associated to only 1 HLA type in the database (mainly from mono-allelic sources, see STAR Methods section "Dataset collection from IEDB and preparation"). ŷ(v) is the categorical output predicted by the HLA-I classifier for v, calculated from the softmax activation function: ŷ(v) = softmax(X • I(v) + b) (7) where X are the classifier weights, connecting input to output layer (see Supplementary Fig. 2A ), and b are local biases adjusted during training. Element-wise softmax is defined as: softmax(z) c = e zc c e z c (8) The activation function softmax has the advantage of giving predictions normalized over categories, thus each element ŷc (v) can be interpreted as the probability that sequence v belongs to class c.
Our numerical implementation relies on Theano and Keras Python libraries. Training is performed in minibatches of 64 sequences, by the ADAM optimizer for 1000 epochs, retaining the model that gives the best accuracy on a held-out partition of the ∼30% of the training dataset. The choice of RBM (hyper-)parameters (see above) also ensures a high accuracy of classification (Supplementary Fig. 3C ). Accuracy of classification is measured there as an Area Under the Curve (AUC), which is different from the cross-entropy optimized during training (Eq. 6) and is defined below, in the section "RBM-MHC performance in multi-allelic samples". This AUC value is only minimally affected when reducing the RBM training dataset and is close to 1 (indicating the maximal accuracy) already with the small number of labeled sequences used, i.
Rather, it is precisely learning the classifier on top of the cluster structure discovered in an unsupervised way in the RBM representation space that ensures robust HLA type prediction in the scenario we envision, i. e. , when we have very few labels at our disposal to characterize new datasets. In addition, optimizing Eq. 9 could potentially also imply a loss of the feature discovery power of a fully unsupervised approach, as data features not directly contributing to the HLA specificity would not be internally represented.
Unsupervised clustering K-means algorithm Given a set of points x, K-means  [28, 37]  finds the centroids of c = 1,. , N c clusters and assigns each x to the cluster whose centroid has the minimal distance to x. If we indicate by d c (x) the distance between point x and a cluster c, we can express the probability that x belongs to cluster c as: ŷkm c (x) = e -dc(x) N c c =1 e -d c (x) , (10) as is a common choice in the "soft" version of K-means. We define RBM-km as the application of K-means to sequence representations in the space of inputs to hidden units (i. e. x = I(v)) instead of sequences themselves (i. e. x = v). From such a probabilistic clustering prediction, we define the classification score N c c=1 ŷkm c (x) log(ŷ km c (x)). Our implementation of K-means relies on the routine available in the Python package Scikit-learn  [38].
MixMHCp algorithm Consistently with the other approaches discussed, we assume that the expected number of clusters N c is known, and we implement the clustering by MixMHCp2. 1  [10] [11] [12]. First we build Position Weight Matrices (PWM) for each of the N c clusters found by MixMHCp; each PWM describes the single-site amino acid frequencies f c i (v i ) in cluster c, with c = 1,. , N c. For a sequence v a set of presentation scores (one per cluster) can be defined from the log-likelihood under the corresponding PWM: L M M c (v) = L i=1 log f c i (v i ) c = 1,. , N c , ( 11 ) where the superscript M M stands for "MixMHCp" and L is the length to which all sequences are reduced in MixMHCp (L = 9). The final score of a sequence v is taken as the maximal log-likelihood among the N c clusters, L M M (v) = max c L M M c (v) , which is used jointly to predict the HLA association.
Generation of HLA-specific artificial peptides A RBM is a generative model. As it is based on fitting an entire probability distribution to a given dataset, sampling from this distribution allows one to generate new candidate antigens. The binding specificity to HLA-I of such generated sequences can be controlled by conditioning (fixing) the RBM probability on the values of inputs to hidden units coding for the desired specificity. The search for these values is guided by the HLA-I classifier. This procedure directly builds on the idea of sampling while conditioning on structural, functional, phylogenetic features emerging in RBM representations of protein families  [16]. Schematically, the steps of this HLA-specific sampling are as follows: Supplementary Fig. 2C  shows examples of I n values (2 of its components for simplicity) covered in this search with β = 50 and σ = 0.
01 (dark red points). By this sampling procedure, we generate samples of artificial peptides that would be predicted to be presentable and specifically presented by the selected HLA-I protein complex. Such samples explore a broad diversity of peptides, that are typically 3-4 mutations away from the closest natural ones (Supplementary Fig. 2D ) while preserving the profile of amino acid abundances constrained by a given binding specificity (Supplementary Fig. 2E ). Sequence re-weighting scheme RBM presentation models trained on datasets obtained by mass spectrometry (MS) might underestimate the probability of presentation of ligands that are less frequently detected by MS. The most evident case are ligands containing cysteine, an amino acid that can undergo chemical modifications by oxidation typically not included in standard MS searches.
We introduced a procedure for re-weighting sequences as a general strategy to correct for biases in the training dataset, such as MS detection biases. We defined a weight for each sequence v in the training set: α(v) = N v i=1 f back (v i ) f M S (v i ) , (12) where f M S denotes typical amino acid frequencies in MS (as can be estimated from IEDB data) while f back is a background amino acid frequency distribution that does not suffer from the same biases as the MS one. These weights α(v) are incorporated into the average log-likelihood to maximize during RBM training: L RBM = 1 v α(v) v α(v) log(P (v)). (13) When modeling multiple specificities through mixtures of PWMs (as by MixMHCp and MixMHCpred, see  [11] ), an analogous correction can be implemented by rescaling the amino acid frequency composing each PWM by the background frequency.
Following Ref. [11] , f back can be chosen as the frequency of amino acids in the human proteome or as the frequency of amino acids in IEDB antigens detected by other techniques than MS (we shall refer to these as f non-M S ). More precisely, f non-M S is estimated from frequencies at non-anchor sites (position 4 to position 7 of 9-mers), excluding from the search all alleles that: (i) have fewer than 100 peptides associated to them; (ii) are present in neither MS nor non-MS datasets; (iii) show specificity at positions between 4 and 7. In Supplementary Fig. 4A , the comparison of f non-M S and f M S for the 20 amino acids provides a clear indication of the MS detection bias in relation to the amino acids cysteine (C) and tryptophan (W), the frequency difference between MS detection and other techniques being more than 100% of MS frequency itself.
The C/W frequency in presented antigens is therefore expected to be underestimated by MS, suggesting that ligands containing C/W, whose binding affinity to HLA-I could be successfully tested in vitro, would be missed by MS. For the purpose of illustration, we test how the re-weighting procedure could correct for such bias on a presentation model built from IEDB, MS data for the allele HLA-A*01:01, with f back = f non-M S. We first compute the weights α (Eq. 12) for sequences in the training dataset. In Supplementary Fig. 4B , we show separately their distribution for sequences with and without C/W: for the former, α have particularly high values, in such a way as to weight more their statistical information and to compensate for the underestimation of C/W frequencies in MS. The effect of re-weighting can be then assessed on a validation dataset, in particular on its sequences with C/W, looking at the distribution of their presentation scores relative to the average scores over the full validation set (see Supplementary Fig.
Model predictions in mono-allelic MS datasets To assign scores of presentation by a specific HLA-I, we resorted simply to the RBM to build HLAspecific presentation models (with N h = 5, λ 2 1 = 0. 001). To train them, sets of 8-11-mer peptides documented with the corresponding HLA association were collected from IEDB, restricting to monoallelic-source, mass spectrometry sequence data, as outlined in the section "Dataset collection from IEDB and preparation". We then used the RBM log-likelihood L RBM as probabilistic score of presentation by the HLA under consideration. For the preliminary validity check of the HLA-A*02:01-specific model, we randomly selected a subset with the 80% of these sequences as training dataset and we kept the remaining 20% as test set to evaluate the model's predictions in terms of probabilistic scores of presentation (Fig.
1C ). We randomly selected 5000 peptides from the human proteome (as in UniProt database  [39] ) with length distribution matching the one of presented peptides to serve as a set of peptides predominantly not presented on the cell surface ("generic"). To test our method on mono-allelic MS datasets from Ref. [6] , we trained 92 allele-specific RBM models, for all the A, B, C HLAs analyzed in  [6]. We decided to train a series of allele-specific models to minimize the uneven accuracy across alleles, emerging especially in multi-allelic models, due to the different abundances of peptides with different HLA-I preferences available as training data. (The use of multi-allelic models is intended as tool to characterize unannotated samples through motif reconstruction). For each model, we randomly selected in the corresponding mono-allelic dataset 100 peptides to use for model evaluation.
Since datasets from  [6]  feature in the latest version of IEDB, we carefully excluded the 100 peptides per model from the IEDB-derived training datasets. We produced, as described above, n-fold excess of generic peptides, choosing n in such a way as to consider a proportion between presented and generic peptides that resembles natural conditions of epitope selection. Large-scale experimental and bioinformatic studies on viral peptidomes  [19] [20] [21]  found that ∼ 1% of peptides bind to given MHCs (suggesting the value n=99). Comparing the amount of all 9-long peptides from human proteome (∼ 10 7 ) and the average, allele-specific antigen repertoire size (∼ 10 4 sequences) derived from existing databases as IEDB, Refs. [4, 6]  concluded rather that ∼ 0. 1% of peptides is "presentable" (corresponding to n=999).
These approximate estimates allowed us to fix a threshold for positive prediction at, respectively, the top-scoring 1% and 0. 1% of peptides in order to evaluate model performance by the PPV metric. (The random expectation for the PPV in these conditions is 10 -2 and 10 -3 for the PPV estimated respectively at the top scoring 1% and 0. 1%). AUCs do not vary, apart from noise, with n; the AUC values shown in Fig. 1D  and Supplementary Fig. 5  are the ones obtained with n=99 (i. e. , generic and presented peptides mixed in proportion 100:1). To better rationalize differences between RBM and NetMHCpan4. 0 in connection to different HLAs, we monitored performance as a function of the distance between the query HLA and its nearest neighbor in the NetMHCpan4. 0 training dataset, where such distance is determined from the similarity between the two HLA-I sequences (more precisely, the 34 residues in contact with the peptide).
MS-based model validation for neoantigen discovery To evaluate the predictive power of the HLA-A*02:01-specific RBM model for neoantigen discovery, we acquired the list of missense mutations for ovarian and melanoma cancer cell lines SKOV3, A2780, OV90, HeLa and A375 from the Cosmic Cell Lines Project  [41]. We further filtered them largely following the steps described in Ref. [23] , i. e. we retained only mutations in genes with non-zero expression levels in the corresponding cell line (based on expression data from the Genomics of Drug Sensitivity Center  [42] ) and we excluded common germline variants documented in the Single Nucleotide Polymorphism Database  [43]. As a result, the number of missense mutations considered is as follows: 444 (A375), 521 (SKOV3), 602 (A2780), 280 (OV90), 511 (HeLa).
For each cell line, the list of mutated peptides to score results from adding up all the up to 38 peptides of length 8-11 containing each mutation. Since the MS database for the 5 cancer lines produced by Ref. [23]  was annotated by comparison to the consensus human proteome, it contained only wildtype peptides whose mutated version was in the list of putative neoantigens. As long as a mutation does not affect an anchor site, the mutated version should preserve a high probability of presentation. We followed the authors' choice of not considering the two peptides mutated at anchor positions as presented and we excluded them from the analysis of Fig. 1E. When we consider all neoantigens (including the two neoantigens arising from anchor-site mutations of MS-derived peptides) the mean score percentiles are: RBM-MHC 3.
3% vs NetMHCpan4. 1 1. 6% (among generic peptides) and RBM-MHC 3. 6% vs NetMHCpan4. 1 0. 9% (among all mutated peptides from the same cell line). The studies from which single-patient samples 12T, Mel8, Mel15, Mel5 were retrieved  [26, 27, 31]  listed a total of 11 tumor-presented neoantigens, using techniques that allow for the detection of variants of native proteins by MS. For some of these neoantigens, immunogenicity was also validated in vitro by identification of neoantigen-specific T-cell responses. Their HLA association was predicted by NetMHC software, see  [26, 27, 31]. The prediction was to a large extent confirmed by in vitro validation of immunogenicity, which relied on antigen presenting cells that were positive to the predicted HLA-I. As the 11 neoantigens are 9-10 residues long, following Ref.
[11]  we considered 9-mer and 10-mer peptides overlapping the missense mutations observed in the patient's WES, which resulted in a list of thousands of putative neoantigens (see Supplementary Table  3 ). The RBM-MHC training set per se can thus take into account only 9-10 mers from patients' samples; patients' neoantigens to validate were not included in the training dataset. In this multi-allelic case where antigens may be presented by several HLA types (as opposed to the allele-specific case of the previous paragraph) we used the global score accounting for the probability of presentation (by RBM) as well as the confidence of the HLA assignment (either by the HLA-I classifier in RBM-MHC or by K-means clustering in RBM-km) of Eq. 9.
Model predictions for SARS-CoV-2 epitopes We downloaded the protein-coding regions of SARS-CoV-2 genome from GenBank  [44] , NCBI Reference Sequence: NC_045512. 2. We extracted all the 9-mers contained in the SARS-CoV-2 proteome, giving a list of 9656 candidate cytotoxic T-cell epitopes (HLA-I antigens). Ref. [24]  (available at https://www. immunitrack. com/free-coronavirus-report-for-download/) tested in vitro the 94 top scoring epitopes according to NetMHC4. 0  [7]   [8]  for C*01:02. 159 peptides were identified as high-stability binders (i. e. , with stability above the threshold of 60% of the reference peptide value for the corresponding HLA-I allele). Ref. [24]  also estimated the trend of binding stability (expressed as % of the reference value) vs predicted score (binding affinity in [nM] for NetMHC4.
0, rank percentile for NetMHCpan4. 0). Here we report and quantify these trends in terms of Pearson correlation for the sake of comparison to RBM (Fig. 1F  and Supplementary Fig. 6B ). To probe our method as a predictor for the high-stability binders found, we learned a series of allele-specific RBM presentation models (N h =5 hidden units, λ 2 1 = 0. 001) for the 10 HLA-I alleles considered in this study, prioritizing training datasets from binding assays in IEDB (see STAR Methods section "Dataset collection from IEDB and preparation"). These types of data are almost absent for the 4 HLA-C mentioned, and in these cases models were learned from MS data only but a re-weighting of frequencies, aimed at correcting for MS biases in detection, was applied (see STAR Methods section "Sequence re-weighting scheme").
Final BA datasets used have in average 1269 sequences, MS ones 1678. All peptides tested for binding to a given HLA-I in Ref. [24]  were scored by the corresponding RBM model (see Fig. 1F , where, as a comparison, results achieved by training all RBMs on MS data with re-weigthing are also reported). We next estimated their score percentile relative to scores assigned to all candidate epitopes and we assessed that tested binders were predominantly assigned high scores, able to a good extent to discriminate them from the tested non-binders (see Supplementary Fig. 6A ). As an additional test, we considered the SARS-CoV-2 cytotoxic T-cell epitopes identified as potentially associated to high immune responses by Ref. [25] , who mapped the dominant, experimentally validated SARS-CoV-derived epitopes to the corresponding regions in the SARS-CoV-2 proteome.
Main alignment Alignment refinement (HLA classification) Step 1 Step 3 Step 4  A The probability of presentation P (v) inferred by the RBM consists of potentials g i (v i ), describing single-site amino acid conservation, and terms describing the co-occurrence of motifs through the weights w iµ (v). In the first row, we consider a IEDB, MS-based dataset of antigens specific to one allele, HLA-B*51:01 (mono-allelic dataset). A RBM model is trained on these data (chosen with 5 hidden units and λ 2 1 = 0. 1, to enhance weights' interpretability, see Supplementary Fig. 3G ). With data of the same HLA-I type, the binding motif is essentially captured by conservation alone (logo representation of e i (g i (v i )) , central column). Weights and hidden units embed additional data features (right column).
Hidden unit 1 h 1 encodes the multiple binding modes of HLA-B*51:01 antigens discussed in  [12]  (see also Fig. 2B ): the sequence property "hydrophilic/charge" has a positive projection on W 1 (and hence positive inputs to h 1 ) while the pattern "hydrophobic" has a negative one. In the second row, we train a RBM (10 hidden units and λ 2 1 = 0. 1) on a bi-allelic dataset, i. e. where we added to the mono-allelic dataset IEDB, MS-derived antigen sequences specific to the allele HLA-A*03:01 (a minimal example of a sample mixing antigens of different specificities). In this setting: (i) overall conservation does not allow discrimination of HLA-I motifs (central column), for which the RBM data representation through weights and inputs to hidden units is necessary; (ii) this representation is robust with respect to the mono-allelic case.
A B C E D F G I H Supplementary Figure  7  (related to Fig. 2C ). Performance at predicting antigen HLA association in "synthetic-individual" samples. A: Same plot as in Fig. 2C  for datasets containing peptides of length 8-11 residues, where a sequence alignment step is required (see STAR Methods). With the same representation as in Fig. 2C  and (A   Supplementary Figure  8  (related to Fig. 2D-E ). Sequence motif reconstruction in multi-allelic samples. A: Sample 12T from  [26, 27] , sample size = 9262 sequences (restricted to length 8-11); B: sample Mel8 from  [31] , sample size = 5481 sequences (restricted to length 8-11); C: sample Mel15 from  [31] , sample size = 19304 sequences (restricted to length  [8] [9] [10] [11]. Sequence motifs recovered by RBM-MHC are compared to the ones from mono-allelic data in IEDB and the ones recovered by MixMHCp2.
Figure1: RBM approach to HLA-I antigen presentation. (A) Antigens binding to a given HLA match specific "sequence motifs" represented by logos (example of Tax-HLA-A*02:01 complex structure, PDB-ID:1BD2, Mol* image [33] ). (B) RBM model structure. (C) RBM scores for presented vs generic peptides on HLA-A*02:01 and predictive performance assessed by ROC. (D) RBM and NetMHC performance at recovering MS-detected peptides from [6]  for 92 HLA-I alleles as measured by AUC and PPV metrics. PPVs and AUCs are plotted as a function of the distance between the corresponding HLA and the closest one in NetMHCpan4. 0 training dataset. Bold lines highlight the trend of mean AUC and PPV values (plotted by squares) over subsets of alleles grouped by distance as indicated by vertical dashed lines (63 alleles with distance = 0, 15 alleles with 0 < distance < 0.
Acknowledgment We thank  Benjamin Greenbaum  for suggesting to look at SARS-CoV-2, and helpful exchanges with  Clément Roussel ,  Andrea Di Gioacchino ,  Carlos Olivarez ,  Hannah Carter ,  David G Feller ,  Shelly Kalaora ,  Yardena Samuels ,  David Hoyos ,  Jayon Lihm ,  Anna Paola Muntoni. This work was partially supported by  Stand Up  to Cancer, the  European Research Council  Consolidator Grant n. 724208, the  ANR-17 RBMPro  and  ANR-19  Decrypted  CE30-0021-01 , the  ANR   Flash Covid 19 -FRM, PROJET  " SARS-Cov-2immunRNAs ",  European Research Council Marie Curie-Sklodowska ITN QuanTI  grants. J. T. was supported by a postdoctoral fellowship from the  Human Frontier Science Program Organization  (reference number:  LT001058/2019 ).
1 )Figure 1 :ProjectionFigure 2 : 112 Figure1: RBM approach to HLA-I antigen presentation. (A) Antigens binding to a given HLA match specific "sequence motifs" represented by logos (example of Tax-HLA-A*02:01 complex structure, PDB-ID:1BD2, Mol* image [33] ). (B) RBM model structure. (C) RBM scores for presented vs generic peptides on HLA-A*02:01 and predictive performance assessed by ROC. (D) RBM and NetMHC performance at recovering MS-detected peptides from [6]  for 92 HLA-I alleles as measured by AUC and PPV metrics. PPVs and AUCs are plotted as a function of the distance between the corresponding HLA and the closest one in NetMHCpan4. 0 training dataset. Bold lines highlight the trend of mean AUC and PPV values (plotted by squares) over subsets of alleles grouped by distance as indicated by vertical dashed lines (63 alleles with distance = 0, 15 alleles with 0 < distance < 0.
Contexte et rationnel de l'étude La surexpression du récepteur 2 du facteur de croissance épidermique humain (HER2) est reconnue comme facteur pronostique d'agressivité des cancers du sein, qui conduit à une survie sans maladie (DFS) et une survie globale plus courte que dans les tumeurs où HER2 n'est pas surexprimé  [1, 2]. Réservé jusqu'en 2005 au seul traitement des cancers du sein métastatiques avec surexpression de HER2 (HER2+), l'indication du trastuzumab a été étendue en 2006 en Europe au traitement adjuvant des cancers du sein précoces HER2+  [3]. En France, l'extension de cette indication avait été précédée dès le mois d'octobre 2005 par l'obtention d'un protocole temporaire de traitement sur la base de données concordantes issues de plusieurs essais cliniques présentés à l'ASCO 2005, notamment l'étude HERA (HERceptin Adjuvant)  [4].