dataesr/scientific-paragraphs-categorization huggingface dataset:
This approach is significantly less successful in identifying SNP disease associations than COMBI and hence Deep-COMBI, who both perform all substeps on the complete (and therefore statistically more powerful) dataset. Another exemplary statistical method that was shown to be outperformed by COMBI and DeepCOMBI is based on linear mixed models (LMMs) proposed by Lippert et al. (21, 27). Even though they test for pairwise epistatic interactions in addition to the univariate tests and address the issue of population stratification in GWAS, they still test genetic locations and pairs thereof individually instead of simultaneously. In comparison to COMBI and DeepCOMBI, which examine the genomic dataset as a whole, LMMs cannot achieve the same level of power and detection rates. In addition to the main competitor method, COMBI, we also compared DeepCOMBI to the baseline methods of RPVT, raw LRP relevance scores and raw SVM importance scores and showed that only the combination of deep learning and multiple testing show the desired performance increase, which cannot be achieved individually by one of these components.
Therefore, the preprocessing step might dilute such effects and has to be applied carefully and only if necessary. A removed low-impact SNPs might not have reached significance itself if included but could have caused another more high-impact SNP to reach significance. On the other hand, let us point out that selecting a subset of features for DNN training might cause our method to introduce an optimistic bias and produce an increased amount of false positives  (71, 72). When the same dataset is used to select features and train a classifier, so-called feature selection bias can indeed become an issue  (71, 73). The cause of this bias is that the feature selection procedure is part of the training, but not external to the test samples  (74). It is therefore essential to state that the current results might overestimate the real predictive effects of certain SNPs.
Furthermore, the DeepCOMBI method does not account for confounding factors such as relatedness which is studied as one of the most important challenges in statistical genetics  (77, 78)  and GWAS typically involve related individuals. We, therefore, dedicate the following sections to address potential issues with confounding factors and selection biases caused--among other things--by including related individuals, i. e. non-independent and identically distributed (non-i. i. d. ) samples in the datasets under investigation. In general, machine learning and, more specifically, neural networks are formally based on the assumption of i. i. d. random variables  (79, 80). This assumption enables the simple development of efficient theories  (81)  and methods  (53) , causing most ML-based research to focus on learning with i.
i. d. data. In particular, neural networks require independent samples to avoid getting stuck in the training phase, finding local optima or overfitting and not generalizing well to external data  (79, 80). When one datapoint influences and is hence not independent of another datapoint, this violation of the i. i. d. assumption is called an interdependency  (80). In practice, non-i. i. d. samples are often assumed to be only a theoretical issue and included in analyses nonetheless  (82). In part, that is because they can be accounted for by, for example, choosing the architecture and hyperparameters of the network appropriately  (80). On the other hand, in some areas, related samples seem to actually have no effect on the overall performance of a neural network. This is particularly prevalent in the field of image recognition, where datasets often include related images which are rarely addressed as such explicitly  (37).
With large enough datasets, where small clusters of dependent samples (close or distant relatives in our case) are selected at random from the entire data distribution (i. e. population), the effects are often negligible. However, selection bias due to related samples can indeed become an issue when the complete dataset is too small. In addition, interdependencies are very common in computational biology since the corresponding datasets are often composed of related organisms  (27) , phenotypes  (83)  or features  (84). Since ignoring these confounding factors in the corresponding datasets can indeed cause falsepositive discoveries that cannot be replicated on independent data (85), a number of approaches have been developed to account for population structures due to relatedness between samples  (21, 27, 78, 80, 86, 87).
Some of these approaches extend existing algorithms to account for noni. i. d. data  (82, 88). Others have been proposed to address time-structured dependencies  (89). For example, recurrent neural networks are used, in contrast to feedforward networks, to model interdependent data by allowing connec-tions between a neuron and neurons of the same or previous layers  (90). However, this architecture is designed to account for non-i. i. d. data caused by sequentially dependent data (e. g. time series data) and not for dependencies arising from grouped relatedness as found in GWAS. Considering this kind of non-i. i. d. data, a number of methods have been presented to correct for population structures following both statistical  (27)  as well as ML-based approaches  (86). Recently, very large datasets containing related samples have been investigated  (91) (92) (93)  to examine the effect of interdependencies in GWAS using LMMs as one of the most widely used tools to account for relatedness regressing the phenotype measures on a relationship matrix of fixed effects.
LMMs capture confounding factors such as population structures, family structures and relatedness simultaneously, without the exact knowledge of which are present and without the need to tease them apart  (27). The underlying idea is to model the output based on a mix of factors: the functional ones we are trying to identify and the fixed ones that just arise from confounding factors. To evaluate the effect of interdependencies and population structures in the WTCCC datasets under investigation here, we compared the performance of DeepCOMBI to that of the LMM approach proposed by Lippert et al. (21, 27)  in a benchmark analysis and found that DeepCOMBI method yields both more discoveries that were validated in external studies and higher precision than the LMMs as presented by Lippert et al.
(21, 27). The validated discoveries of Deep-COMBI are also more broad covering the whole range of diseases while the validated findings of Lippert et al. (21)  are limited to CD, T1D and T2D. In general, the comparison of the two methods is favorable to DeepCOMBI with a significant P-value of < 0. 0001 (two-tailed Fisher's exact test). Even though the comparison of our method with an approach correcting for population structures was highly favorable to DeepCOMBI, it is not possible to conclude that relatedness within the 2007 WTCCC dataset is limited and non-i. i. d. samples have no impact at all. However, the unique benefit of our method was demonstrated when we revealed biological signals with state-of-the-art methods in datasets of almost 15 years of age that standard GWAS analyses have not been able to capture in any dataset published to this day.
Furthermore, we clearly demonstrate the replicability of our findings, which is a robust way (94) and the gold standard (95) of verifying the biological signals underneath the investigated data. In conclusion, DeepCOMBI, a novel AI-based method, was proposed for the analysis of GWAS data. After training a carefully designed DNN for the classification of subjects into their respective phenotype, the concept of explainable AI is applied by backpropagating the class prediction score to the input layer through the network via LRP. The resulting SNP relevance scores are used to select the most relevant SNPs for multiple testing in combination with a permutation-based thresholding procedure. On both generated, controlled datasets as well as seven real GWAS datasets, DeepCOMBI was shown to perform better than a number of competitor methods in terms of classification accuracy of the DNN and in terms of ROC and PR curves when using either the generated labels or replicability in external studies as a validation criterion.
In addition, two very promising, entirely novel SNP disease associations were discovered. Located on an intron for NEGR1, an important gene many times linked to obesity, body mass index and other correlated factors, rs10889923 on chromosome 1 was found to be significantly linked to hypertension. Another novel location found by DeepCOMBI to be associated with type 1 diabetes is rs4769283. It is part of an intergenic region on chromosome 13 and was previously found to be an eQTL for C1QTNF9B in the pancreas, the affected organ in T1D patients. Future work on the subject of deep learning and explainable AI in the context of analyzing GWAS datasets could focus on one of the three steps of DeepCOMBI. In the first step, DNNs with different architectures or other suitable analysis tools could be investigated.
Future research work considering the third step of DeepCOMBI might examine the effects of replacing the χ 2 test with a different, more sophisticated kind of test, e. g. investigating pairwise hypotheses or other multivariate effects. In addition, fully evaluating our approach on datasets with related samples and integrating DeepCOMBI with LMMs or adapting it to correct for relatedness could be the focus of future research. A dataset much larger with respect to both samples and SNPs would have to be investigated. For example, the UK Biobank dataset could be the focus of such an analysis since it includes about 500 000 subjects for whom hundreds of phenotypes have been registered  (98). Some of the subjects in this dataset are related and it has been the testing ground for several algorithms to account for relatedness  (91) (92) (93).
A combination of the two approaches could be developed to adapt our approach to account for related subjects by employing mixed effect models in convolutional DNNs and applying LRP explanation to identify associated SNPs. Another potential focus of future research including the proposed method could lie in identifying and quantifying the effect of small groups of non-i. i. d. samples (i. e. families) in GWAS datasets. This could be achieved by clustering the LRP explanations of all samples as proposed by Lapuschkin et al. (97)  and identifying the locations of related individuals in the resulting clustering structures. If the subjects of a family end up in the same clusters in such an analysis, the effect of the underlying selection bias could be visualized and it could be shown that SNPs that have an effect in a specific family are not necessarily associated with the trait in the general population.
DATA AVAILABILITY DeepCOMBI source code is available at https://github. com/ AlexandreRozier/DeepCombi. The 2007 WTCCC dataset (  14 ) analyzed during the current study is not publicly available. Access can be requested from the owners at https://www. wtccc. org. uk/info/ access to data samples. html and https://www. sanger. ac. uk/ legal/DAA/MasterController. The code for simulating GWAS datasets is available at https://github. com/AlexandreRozier/DeepCombi.
Generative models emerge as promising candidates for novel sequence-data driven approaches to protein design, and for the extraction of structural and functional information about proteins deeply hidden in rapidly growing sequence databases. Here we propose simple autoregressive models as highly accurate but computationally efficient generative sequence models. We show that they perform similarly to existing approaches based on Boltzmann machines or deep generative models, but at a substantially lower computational cost (by a factor between 10 2 and 10 3 ). Furthermore, the simple structure of our models has distinctive mathematical advantages, which translate into an improved applicability in sequence generation and evaluation. Within these models, we can easily estimate both the probability of a given sequence, and, using the model's entropy, the size of the functional sequence space related to a specific protein family.
INTRODUCTION The impressive growth of sequence databases is prompted by increasingly powerful techniques in datadriven modeling, helping to extract the rich information hidden in raw data. In the context of protein sequences, unsupervised learning techniques are of particular interest: only about 0. 25% of the more than 200 million amino-acid sequences currently available in the Uniprot database  [1]  have manual annotations, which can be used for supervised methods. Unsupervised methods may benefit from evolutionary relationships between proteins: while mutations modify amino-acid sequences, selection keeps their biological functions and their three-dimensional structures remarkably conserved. The Pfam protein family database  [2] , e. g. , lists more than 19,000 families of homologous proteins, offering rich datasets of sequence-diversified but functionally conserved proteins.
The DeepSequence implementation  [33]  was originally designed and tested for predicting the effects of mutations around a given wild type. It currently provides one of the best mutationaleffect predictors, and we will show below that arDCA provides comparable quality of prediction for this specific task. The DeepSequence code has been modified in  [34]  to explore its capacities in generating artificial sequences being statistically indistinguishable from the natural MSA; it was shown that its performance was substantially less accurate than bmDCA. Another implementation of a VAE was reported in  [35] ; also in this case the generative performances are inferior to bmDCA, but the organization of latent variables was shown to carry significant information on functionality. Furthermore, some generated mutant sequences were successfully tested experimentally.
We will show in the following that arDCA performs comparably to bmDCA, and better than DeepSequence, at strongly reduced computational cost. From anecdotal evidence in the works mentioned above, and in agreement with general observations in machine learning, it appears that deep architectures may be more powerful than shallow architectures, provided that very large datasets and computational resources are available  [33]. Indeed, we will show that for the related task of single-mutation predictions around a wild type, DeepSequence outperforms arDCA on rich datasets, while the inverse is true on small datasets.
10GHz CPU. We also tested the Mi3 package  [29] , which is able to learn similar bmDCA models in a time of about 60 minutes for the PF00014 family and 900 minutes for the PF00595 family, while running on two TITAN RTX GPUs, thus remaining much more computationally demanding than arDCA. The positional order matters Eq. (  1 ) is valid for any order of the positions, i. e. for any permutation of the natural positional order in the amino-acid sequences. This is no longer true, when we parameterize the P (a i |a i-1 ,. , a 1 ) according to Eq. (  2 ). Different orders may give different results. In the Supplementary Note 1 we show that the likelihood depends on the order, and that we can optimize over orders. We also find that the best orders are correlated to the entropic order, where we select first the least entropic, i.
Results are shown for the responseregulator Pfam family PF00072  [2]. Other proteins are shown in Table  1  and Supplementary Note 3, Supplementary Figures  5 6. We find that, for these observables, the empirical and model averages coincide very well, equally well or even slightly better than for the bmDCA case. In particular for the one-and two-point quantities this is quite surprising: while bmDCA fits them explicitly, i. e. any deviation is due to imperfect fitting of the model, arDCA does not fit them explicitly, and nevertheless obtains higher precision. In Table  1 , we also report the results for sequences sampled from DeepSequence  [33]. While its original implementation aims at scoring individual mutations, cf. Section Predicting mutational effects via in-silico deep mutational scanning, we apply the modification of Ref.
[34]  allowing for sequence sampling. We observe that for most families, the two-and three-point correlations of the natural data are significantly less well reproduced by DeepSequence than by both DCA implementations, confirming the original findings of  [34]. Only in the largest family, PF00072 with more than 800,000 sequences, DeepSequence reaches comparable or, in the case of the three-point correlations, even superior performance. A second test of the generative property of arDCA is given by Figures  2d-g. Panel d shows the natural sequences projected onto their first two principal components (PC). The other three panels show generated data projected onto the same two PCs of the natural data. We see that both arDCA and bmDCA reproduce quite well the clustered structure of the response-regulator sequences (both show a slightly broader distribution than the natural data, probably due to the regularized inference of the statistical models).
It is now easy to perform an in-silico deep mutational scan, i. e. to determine all mutational scores ∆E(a i → b i ) for all positions i = 1,. , L and all target amino acids b i relative to some reference sequence. In Figure  3a , we compare our predictions with experimental data over a b Figure  3. Prediction of mutational effects by arDCA: Panel a shows the Spearman rank correlation between results of 32 deepmutational scanning experiments and various computational predictions. We compare arDCA with profile models, plmDCA (aka evMutation  [10] ), DeepSequence  [33] , and GEMME  [58] , which currently are considered the state of the art. Detailed information about the datasets and the generative properties of arDCA on these datasets are provided in the Supplementary Note 4. Panel b shows a more detailed comparison between arDCA and DeepSequence, the symbol size is proportional to the sequence number in the training MSA for prokaryotic and eukaryotic datasets (blue dots).
Viral datasets are indicated by red squares. more than 30 distinct experiments and wildtype proteins, and with state-of-the art mutational-effect predictors. These contain in particular the predictions using plmDCA (aka evMutation  [10] ), variational autoencoders (DeepSequence  [33] ), evolutionary distances between wildtype and the closest homologs showing the considered mutation (GEMME  [58] ) -all of these methods take, in technically different ways, the context dependence of mutations into account. We also compare it to the context-independent prediction using the abovementioned profile models. It can be seen that the context-dependent predictors outperform systematically the context-independent pre-dictor, in particular for large MSA in prokaryotic and eukaryotic proteins. The four context-dependent models perform in a very similar way.
Almost all dots are close to the diagonal (apart from few viral datasets), with 15/32 datasets having a better arDCA prediction, and 17/32 giving an advantage to DeepSequence. The figure also shows that arDCA tends to perform better on smaller datasets, while DeepSequence takes over on larger datasets. In Suppelmentary Figure  7 , we have also measured the correlations between the two predictors. Across all prokaryotic and eukaryotic datasets, the two show high correlations in the range of 82% -95%. These values are larger than the correlations between predictions and experimental results, which are in the range of 50% -60% for most families. This observation illustrates that both predictors extract a highly similar signal from the original MSA, but this signal may be quite different from the experimentally measured phenotype.
Many experiments actually provide only rough proxies for protein fitness, like e. g. protein stability or ligand-binding affinity. To what extent such variable underlying phenotypes can be predicted by unsupervised learning based on homologous MSA thus remains an open question. We thus conclude that arDCA permits a fast and accurate prediction of mutational effects, in line with some of the state-of-the-art predictors. It systematically outperforms profile models and plmDCA, and is more stable than DeepSequence in the case of limited datasets. This observation, together with the better computational efficiency of arDCA, suggests that DeepSequence should be used for predicting mutational effects for individual proteins represented by very large homologous MSA, while arDCA is the method of choice for large-scale studies (many proteins) or small families.
However, note that the majority of the false positives for both predictors are concentrated in the upper right corner of the contact maps, in a region where the largest subfamily of response-regulators domains, characterized by the coexistence with a Trans reg C DNA-binding domain (PF00486) in the same protein, has a homo-dimerization interface. One difference should be noted: for arDCA, the definition of effective couplings via epistatic effects depends on the reference sequence (a 1 ,. , a L ), in which the mutations are introduced; this is not the case in DCA. So, in principle, each sequence might give a different contact prediction, and accurate contact prediction in arDCA might require a computationally heavy averaging over a large ensemble of background sequences. Fortunately, as we have checked, the predicted contacts hardly depend on the reference sequence chosen.
,a L P (a 1 ,. , a L ) log P (a 1 ,. , a L ) = E(a 1 ,. , a L ) P , where the second line uses Eq. (  4 ). The ensemble average • P can be estimated via the empirical average over a large sequence sample drawn from P. As discussed before, extracting i. i. d. samples from arDCA is particularly simple due to their particular factorized form. Results for the protein families studied here are given in Table  1. As an example, the entropy density equals S/L = 1. 4 for PF00072. This corresponds to N ∼ 1. 25 • 10 68 sequences. While being an enormous number, it constitutes only a tiny fraction of all q L ∼ 1. 23 • 10 148 possible sequences of length L = 112. Interestingly, the entropies estimated using bmDCA are systematically higher than those of arDCA. On the one hand, this is no surprise: both reproduce accurately the empirical one-and two-residue statistics, but bmDCA is a maximum entropy model, which maximizes the entropy given these statistics  [5].
To illustrate this idea in a simple, but instructive case, we have identified two subfamilies of the PF00072 protein family of response regulators. The first subfamily is characterized by the existence of a DNA-binding domain of the Trans reg C protein family (PF00486), the second by a DNA-binding domain of the GerE protein family (PF00196). For each of the two subfamilies, we have extracted randomly 6,000 sequences used to train subfamily specific profile and arDCA models, with P 1 being the model for the Trans reg C and P 2 for the GerE sub-family. Using the log-odds ratio log{P 1 (seq)/P 2 (seq)} to score all remaining sequences from the two subfamilies, the profile-model was able to assign 98. 6% of all sequences to the correct sub-family, and 1. 4% to the wrong one. arDCA has improved this to 99.
Sample the first residue from P (a 1 ) 2. Sample the second residue from P (a 2 |a 1 ) where a 1 is sampled in the previous step. L. Sample the last residue from P (a L |a L-1 , a L-2 ,. , a 2 , a 1 ) Each step is very fast because there are only 21 possible values for each probability. Both training and sampling are therefore extremely simple and computationally efficient in arDCA. Supplementary Figure 1. Comparison of the Pearson correlation of two-point connected correlations and the log-likelihood between different orders. The value of the random order is the mean of 100 random orders with one standard deviation given as a vertical bar. Results are for Pfam families PF00014 (a), PF00072 (b), PF00076 (c), PF0595 (d), and PF13354 (e).