dataesr/scientific-paragraphs-categorization huggingface dataset:
C. CLASSIFICATION ACCURACY Here, we report the obtained results using SuperLoss on 8 datasets, and those obtained by previous methods which are those based on the CE, Triplet-loss, N-pair-loss, as well as SupCon  [29]. The results are given in terms of the accuracy score measured on the same balanced test set. Following common practice in contrastive learning, we first study the importance of adding a projection head that maps representations to new space where the supervised contrastive loss is applied. Similar to  [11] ,  [44] , we tested three different MLP architectures: (1) identity mapping; (2) linear projection z = g(h) = W (1) h ∈ R 512 ; (3) non-linear projection with one additional hidden layer as used by several previous approaches z = g(h) = W (2) ReLU (W (1) h) ∈ R 512. Similar to what was found in previous works, we observe that a non-linear architecture is better than both the linear and the identity functions for the projection head network (See table  2 ).
V. ABLATION STUDY We investigate here the effects of different parameters on performance. All experiments have been conducted using Yelp-F dataset. We run each experiment with 10 different seeds, and report the average test accuracy. We consider several BiLSTM encoder-based architectures with growing capacity, particularly the number of hidden layers and the number of hidden units of the model. In Figure  4a  depicts how changing the model's architecture affects the quality of the learnt representations on the downstream task. We found that increasing the number of hidden layers works better for the proposed framework. In this work, we used three layers due to the GPU memory constraint. Our experiments show that SuperLoss surpasses the cross-entropy loss using the same model architecture for all configurations.
To overcome this, we first train the model for a number of epochs (20 epochs on YELP-F dataset) using all negative instances within the batches, then we modify the training strategy by maximizing the distance of anchors with those pairs that have a similarity higher than a fine-tuned threshold. By doing so, our loss will guide the encoder function to produce representations by considering    only truly hard negatives. In our strategy, we simply select these examples by first ordering the similarities for a given an anchor, then we select the most similar instances (hard negatives). Following this new strategy, the simulations show that, indeed, the distances between anchors and the negative examples become higher compared to those obtained using the previous learning strategy. We also noticed that for a given anchor, the selected hard negative examples are those from the closest class (for class 'Very Positive' the majority of the negative examples are from the class 'Positive' which is the most similar class to that of the anchor).
We fine-tune the similarity threshold using the validation set by selecting the top most similar examples to the anchor. Figure  5c  shows the different top-1 accuracy obtained as a function of similarity threshold. We evaluate our negative sampling strategy on several benchmark datasets. Experimental results show that the strategy is beneficial. In table 5, we report the SuperLoss * which refers to the results obtained using the proposed negative sampling strategy. In bold, we report the best performance. As it can be seen in the table, SuperLoss * outperforms SuperLoss in most cases which means that the proposed strategy leads the encoder to learn better representations (relevant features for distinguishing the classes in the latent space).
VII. CONCLUSION We proposed SuperConText, a new framework for learning text representations using a novel supervised contrastive loss. SuperConText encourages an encoder function to learn representations by maximizing the average agreement between the representation of an anchor and those of N positive pairs, determined as elements belonging to the same class as the anchor, while distancing the anchor's representation with those of negative examples. Simulations show that the proposed framework outperforms several methods based on other objective functions on various benchmark datasets. We have conducted a number of experiments to understand the effects of both negative and positive examples on the quality of learned representations. We further introduced a simple yet effective negative sampling strategy to enhance the quality of the representations.
Rare genetic variants can strongly predispose to disease, yet accounting for rare variants in genetic analyses is statistically challenging. While rich variant annotations hold the promise to enable well-powered rare variant association tests, methods integrating variant annotations in a data-driven manner are lacking. Here, we propose DeepRVAT, a set neural network-based approach to learn burden scores from rare variants, annotations and phenotypes. In contrast to existing methods, DeepRVAT yields a single, trait-agnostic, nonlinear gene impairment score, enabling both risk prediction and gene discovery in a unified framework. On 21 quantitative traits and whole-exome-sequencing data from UK Biobank, DeepRVAT offers substantial increases in gene discoveries and improved replication rates in held-out data.
The trained DeepRVAT model gives rise to a single, trait-agnostic gene burden scoring function, which can be used for different genetic analyses, including rare variant association tests and to refine polygenic risk prediction to account for rare variant effects. We offer user-friendly software that provides pre-trained DeepRVAT models for direct application on new data sets, as well as a pipeline for retraining the model, for example to incorporate other annotations. We validate the model using simulations before applying it to 21 quantitative traits and whole-exome-sequencing data from 167,245 UK Biobank (UKBB) individuals, demonstrating a larger number of discoveries compared to existing methods. Additionally, we demonstrate improved replication of DeepRVAT associations in held-out individuals from UKBB.
A deep set network-based RVAT framework DeepRVAT is an end-to-end model (Fig. 1a ) that first accounts for nonlinear effects from rare variants on gene function (gene impairment module) to then model variation in one or multiple traits as linear functions of the estimated gene impairment scores (phenotype (c) Gene discovery. The trained DeepRVAT impairment module can be used as a scoring function to define association tests for gene discovery. This improves both the number of discoveries, as well as their replication rate in held-out data, compared to existing methods. (d) Polygenic risk prediction incorporating rare variants. The DeepRVAT gene impairment module can be used as input to train predictive models for phenotype from genotype, for example, to improve risk stratification based on polygenic risk scores (PRS) with common variants.
Created with BioRender. com. module). The gene impairment module (Fig. 1b ) outputs a gene-and trait-agnostic gene impairment score estimating the combined effect of rare variants, thereby allowing the model to generalize to new traits and genes. Technically, a deep set neural network  32  architecture is employed to aggregate the effects from multiple discrete and continuous annotations for an arbitrary number of rare variants. This architecture captures additive and nonlinear interactions and does not rely on a priori assumptions about the relevance of individual annotations, such as common assumptions about the relationship between allele frequency and variant effects  20, 22  (Methods). For each trait used in model training, a set of informative seed genes that are associated with the given trait are required.
The DeepRVAT software offers an integrated workflow that employs conventional rare-variant association tests to identify seed genes (Methods). Next, the gene impairment module is trained end-to-end together with the phenotype modules on the seed genes. Once trained, the DeepRVAT gene impairment module can be applied to genes not used during training to test for genetic associations using the established principles of conventional burden tests (Fig. 1c ). The estimated gene impairment scores from the trained DeepRVAT module can also be used to predict phenotype from genotype, thereby providing a flexible way to derive polygenic risk scores  11, 33  that account for rare variant effects (Fig. 1d ). The computational complexity of training DeepRVAT scales linearly with the number of individuals, while computational demands of association testing with DeepRVAT are one to two orders of magnitude lower than existing approaches for incorporating multiple annotations  28, 29  (Supp.
Model validation using simulated data To validate DeepRVAT for gene discovery, we considered a semi-synthetic dataset derived from whole exome sequencing (WES) from 167,245 caucasian UK Biobank individuals (November 2020, 200k WES release  34  ; Methods). We annotated all 12,704,497 WES variants using minor allele frequency (MAF), VEP  35  consequences, missense variant impact scores (SIFT 14 and PolyPhen2  16  ), and omnibus statistical deleteriousness scores (CADD  15  and ConDel 36 ), as well as predicted annotations for effects on protein structure (PrimateAI  18  ), and aberrant splicing (AbSplice 17 ). To create synthetic phenotypic data, we first simulated impairment scores for genes designated to be causally associated to the trait. To this end, we stochastically assigned each variant an effect based on its annotations (Supp.
DeepRVAT recovers a larger number of gene-trait associations using rare variants in UK Biobank Next, we applied DeepRVAT to rare variants (MAF < 0. 1%) from whole-exome data from the UK Biobank, considering the same 167,245 caucasian individuals as used in the simulation study. We used a total of 33 variant annotations (Supp. Table  5 ; Supp. Fig. 3. 1), expanding the set considered in the simulation study by variant effect predictions for epigenetic markers in cell types from ENCODE  37  and the Roadmap Epigenomics 38 projects (using principal components of DeepSEA 19 predictions; Supp. Fig. 3. 2; Methods), as well as binding to 6 RNA-binding proteins (selected predictions from DeepRiPe 39 ; Methods). We considered a total of 21 quantitative traits, comprising standing height and 20 blood biomarkers (Supp.
Finally, we tested the generalizability of DeepRVAT gene impairment scores across phenotypes not included in the training set (Fig. 4e-f ). When comparing phenotype predictors derived from a gene impairment module trained on all 21 phenotypes to a model excluding the evaluated phenotype during impairment module training, we found no notable drop in model performance. This suggests that DeepRVAT impairment scores can be effectively applied to diverse phenotypes, demonstrating their robust applicability beyond the training dataset.
Moreover, DeepRVAT improved over polygenic risk scores based on common variants for phenotype prediction, particularly for extreme phenotypes, outperforming conventional burden scores based on single annotations. DeepRVAT is provided as a user-friendly software package that supports both de novo training of gene impairment modules as well as application of pretrained ones, with dramatic improvements in computational efficiency over existing methods. Although we found that DeepRVAT advances the state-of-the-art in two use cases central to genetics, the model is not free of limitations. First, a universal challenge of any method that considers annotations for rare variant aggregation is the question of how to choose the most relevant annotations. DeepRVAT is no exception, yet it can cope with a potentially very large number of annotations, pruning uninformative ones during the learning process.
Acknowledgements The authors wish to thank  Marc Jan Bonder ,  Manu Saraswat ,  Andrea Senacheribbe ,  Kai Ueltzhöffer ,  Danai Vagiaki , and  Hana Susak  for helpful discussions. This research has been conducted using data from UK Biobank, a major biomedical database (project IDs 25214,  44108 , and  81358 ). B. C. , K. M, M. M. , F. M. , & M. W. were supported through state funds approved by the  State Parliament of Baden-Württemberg  for the  Innovation Campus Health + Life Science Alliance Heidelberg Mannheim. E. H. was funded in part by the  Helmholtz Association  under the joint research  school Munich School for Data Science -MUDS.
(c) Gene discovery. The trained DeepRVAT impairment module can be used as a scoring function to define association tests for gene discovery. This improves both the number of discoveries, as well as their replication rate in held-out data, compared to existing methods. (d) Polygenic risk prediction incorporating rare variants. The DeepRVAT gene impairment module can be used as input to train predictive models for phenotype from genotype, for example, to improve risk stratification based on polygenic risk scores (PRS) with common variants. Created with BioRender. com.
In addition, we had previously assessed INTS13-associated proteins in the promyelocytic leukemia cell line HL-60. 23 We reasoned that comparing the interactomes of INTS13 across a diverse array of cell types would increase the chance of identifying specific protein partners of Integrator while decreasing noise from common nuclear contaminants. We calculated iBAQ enrichment scores using MaxQuant and charted normalized intensity values of all identified proteins in a 3D scatterplot (Figures  1A  and 1B ). To test this hypothesis, we took advantage of AlphaFold2 (AF2), a machine-learning algorithm originally designed to predict 3D protein structures from their primary sequence. 27 9] [30] [31]  Therefore, we designed an experimental pipeline to perform an exhaustive search for interactions of INTS15 with all other known components of the Integrator complex (including the PP2A module) using the ColabFold implementation of AF2  34  (see STAR Methods).
Additionally, we compared the results obtained with ColabFold 34 and AF2 multimer methods,  29  both of which differ in their predictive capabilities for protein complexes (Figure  S2A ). For the successfully predicted interactions, we observed almost no differences locally at the predicted interfaces, but globally some structures differed in their long-range geometry. Taken together, analysis of MS/MS data combined with machine-learning structure prediction approaches suggest that INTS15 is an additional component of the Integrator complex that is related to the INTS13/14/10 module.
Importantly, the entire INTS13/14/10 module could not be visualized, leaving many questions unanswered as to how this module fits into the larger structure and interfaces with the core of the Integrator complex. To better understand how Integrator modules associate with one another, we used an innovative combination of proteomics and AF2 structure prediction. 27 F2 was originally designed to predict 3D structures of proteins from their primary sequence, achieving accuracy comparable to experimental methods. 27 9] [30]  This property was further improved by extending the network training dataset to known protein complexes. 29 everal high-throughput studies reported structure predictions of large sets of binary protein complexes using AF2 or RoseTTAfold. 32, 40 Although these studies provide excellent resources, they have several major shortcomings.
Firstly, the RoseTTAfold dataset is restricted to yeast, which is a simplified model organism missing some of the protein complexes found in humans (i. e. , the Integrator complex). Secondly, these predictions are performed for pre-selected targets and do not sample the search space exhaustively for all possible interactions. Finally, predicted interactions are not validated experimentally. To address this issue, we designed a pipeline to perform an exhaustive search for binary interactions on a focused target (i. e. , INTS15) followed by experimental validation. Through AF2, we identified putative protein-protein interfaces within the Integrator Arm module, which were further validated by site-directed mutagenesis, XL-MS, and EM. This, together with benchmark predictions for known structures, provides a proof of concept that our screening approach is suitable for studying the architecture of large and dynamic protein complexes in the absence of sufficient experimental information.
Eluates were prepared for SDS-PAGE and run on a Novex WedgeWell 10% Tris-Glycine Gel (Invitrogen) with Tris-Glycine-SDS buffer (BioRad). The samples were run at 110V for 10 minutes to allow the samples to enter the gel. Following, the gel was stained overnight with the Colloidal Blue staining kit (Invitrogen). Samples were then processed at the Wistar Institute proteomics facility for MS. Each sample was cut from the gel. Following excision, samples were reduced with TCEP, and alkylated with iodoacetamide. The samples were then digested with trypsin and run on a Q-Exactive Plus mass spectrometer and analyzed by LC-MS/MS with the standard 90-minute LC gradient. Following, the MS/MS spectra were searched against the UniProt human database (10/10/2019) with full tryptic specificity with MaxQuant 1.
2% Tween 20 (PBST) for 1h at room temperature. Primary antibodies were added in the following dilutions: a-HA 1:5000 ([HA-7] HRP ab49969, abcam); a-FLAG -1:5000 (Anti-FLAGâ M2-Peroxidase (HRP) sigma A8592-. 2MG); a-INTS1-1:2000 (Anti-INTS1 Antibody, clone 4. 47 Merck Millipore, cat number MABS1984); a-INTS15, -1:5000 (Anti-C7ORF26 antibody SIGMA HPA052175-100UL) in 5% milk/PBST and incubated for 1h at room temperature. The membrane was washed 3 times for 5 min with 20ml of PBST and secondary antibodies were added: anti-rabbit HRP-conjugate, 1:10 000, cat number Anti-Rabbit IgG H&L (HRP) Abcam ab205718 and Anti-Mouse IgG-Peroxidase antibody SIGMA A5906-1ML. The membrane was washed 3 times for 5min with 20ml of PBST and chemiluminescence was detected with HRP substrate kit (Pierce) Pierce TM ECL Western Blotting Substrate in a Chemidoc imager (Bio-Rad).
44 Aligned reads were filtered based on mapping quality (MAPQ >10) using Samtools and rmdup was used to remove PCR duplicates. 42 For spike in normalization, the number of hg19 filtered reads was divided by the lowest number of dm5 filtered reads for the set of experiments to create a down-sampling factor. This downsampling factor was then used to normalize BAM files using samtools -s and normalized bigwig files were generated with bamCoverage -binSize 10 -extendReads 150. 42 MACS2 (2. 1. 2) was used to call peaks using the broad peak option. 45 The UCSC Genome Browser was used to visualize data (http://genome. ucsc. edu/) seqMINER 1. 3. 4. was used to calculate the average read density across genomic intervals (left and right extensions = 5. 0 kb, internal bins = 160, flanking region bins = 20).
44, 49 Read density matrices were exported to calculate average profile plots in RStudio with ggplot2 (v4. 1. 2). Traveling ratio was calculated by taking the of the coverage at the TSS (À150bp/+50bp) and comparing it to the coverage at the gene body (+50bp through TES). computeMatrix from deeptools was used to generate heatmaps from the normalized bigwig files. 43 RStudio with ggplot2 was used to visualize all other data. Motif analysis was performed with Meme-ChIP 54 using standard parameters. The E value is the statistical significance of the motif, as calculated and defined by MEME-ChIP. Antibodies used for ChIP-seq are anti-INTS10 (Proteintech 15271-1-AP), anti-C7ORF26 (INTS15) (Sigma HPA042899), anti-RNA Polymerase II (anti-RNA Polymerase II (raised against the N-terminus of subunit B1  22  ).
The outcome of interaction prediction was considered as positive when the Predicted Aligned Error (PAE) plots showed inter-molecular crosspeaks with values better than 5A ˚for at least 2 out of 5 predicted models. The prediction was performed for INTS10, INTS13, INTS14 and INTS15, paired with all other components of the integrator. For the benchmark dataset as well as for the INTS10/15; INTS5/15 and INTS10/14, predictions were also performed with Deepmind's original implementation of AlphaFold Multimer. 29 No major differences were observed, except for some minor distortions in individual proteins. Buried Surface Area (BSA) was calculated with PDBePISA. Interface pLDDT was calculated as an average pLDDT value for all residues with non-zero BSA that are involved in the interface formation.
following antibodies were used for detection of HeLa and INTS15-FLAG HEK-293T cell lines: anti-INTS1 (Millipore MABS1984), anti-INTS3 (Proteintech 16620-1-AP), anti-INTS5 (Proteintech 14069-1-AP), anti-INTS6 (Santa Cruz sc-676524), anti-INTS8 (Sigma HPA057299), anti-FLAG M2 (Sigma F1804), anti-INTS13 (Bethyl A303-575A), anti-INTS10 (Proteintech 15271-1-AP), anti-C7ORF26 (INTS15) (Sigma HPA042899), anti-Rabbit IgG (Cell Signaling 2729), anti-GAPDH (Cell Signaling 2118), anti-RNA Polymerase II (Santa Cruz sc-56767), anti-INTS14 (Proteintech 25227-1-AP). INTS15 antibodies serum 1 and serum 2.
ACKNOWLEDGMENTS We thank members of the  Gardini and Galej laboratories  for helpful discussions. We also thank the  Genomics Core and the Proteomics Core at The Wistar Institute  ( P30-CA010815 ) for providing excellent technical support;  M. Rettel ,  P. Haberkant , and  F. Stein  from the  EMBL Proteomic Core Facility  for their support with MS experiments; and  M. Pelosse  from  EMBL Eukaryotic Expression Facility  for the support with cell culture. Work from the  Gardini Lab  (A. G. ) was supported by grants from the  American Cancer Society  ( RSG-18-157-01-DMC ), the  NIH  ( R01 HL141326 ), and the  G. Harold & Leila Y. Mathers Charitable Foundation  (A. G. ). S. R. O. and A. Z. are supported by the  NIH  training grant  T32-CA009171. Work from the Galej lab was supported by funding from the  European Molecular Biology Laboratory  (M.
Facing the potential threat raised by quantum computing, a great deal of research from many groups and industrial giants has gone into building public-key post-quantum cryptographic primitives that are resistant to the quantum attackers. Among them, there is a large number of post-quantum key encapsulation mechanisms (KEMs), whose purpose is to provide a secure key exchange, which is a very crucial component in public-key cryptography. This paper presents a formal security analysis of three lattice-based KEMs including Kyber, Saber, and SK-MLWR. We use Maude, a specification language supporting equational and rewriting logic and a high-performance tool equipped with many advanced features, such as a reachability analyzer that can be used as a model checker for invariant properties, to model the three KEMs as state machines.
LWR is a variant of LWE, where one replaces random noise with deterministic rounding in order to increase efficiency. Kyber was selected as a candidate to be standardized for public-key encryption and key exchange in July 2022 by NIST (https:// csrc. nist. gov/Projects/post-quantum-cryptography/selectedalgorithms-2022). It is the unique candidate for public-key encryption and key exchange. We use Maude  [7] , a specification language supporting both equational and rewriting logic, to model the three KEMs as state machines. We first model plenty of common parts in the designs of the three KEMs, such as polynomials and vectors. Then, the complete formal specification of each KEM is achieved by modeling some additional parts that are specific to only that mechanism, followed by modeling its execution.
Section 3 briefly explains the three lattice-based KEMs. Section 4 presents the base specification used for the three KEMs. Then, Section 5 presents how to complete the Maude specification of Kyber from the base specification and reports its model checking experiment. Section 6 briefly describes the complete specifications of Saber and SK-MLWR. Section 7 notes some remarks on the experiments we have conducted. Section 8 discusses some related work and finally, Section 9 summarizes the paper. All of the Maude specifications and checking commands reported in this paper can be downloaded from the webpage (https://github. com/duongtd23/lattice-based-kems-mc).