dataesr/scientific-paragraphs-categorization huggingface dataset:
LRP is a direct way to compute feature importance scores and has been applied very successfully in numerous data science problems to explain decisions of DNNs  (51, 52). Instead of basing the importance score of a SNP on the data of that SNP alone, correlation structures and possible interactions are automatically taken into account. To make LRP applicable as an explanation method for GWAS data, we use a very promising, well-performing machine learning-based method, called COMBI  (31) , as a starting point for our deep learning-based approach. COMBI is a two-step method, which first calculates a relevance score for each SNP by training a support vector machine (SVM)  (53)  for the classification of subjects based on their genetic profile. Using the learned SVM weights as an indicator of importance, COMBI selects the highest scoring SNPs as a subset to put into multiple hypothesis testing.
a statistical testing step where only the SNPs selected in step 2 are tested for statistically significant association with the trait under investigation. The main motivation behind DeepCOMBI is to harness the immense potential of sophisticated, state-of-the-art artificial intelligence (AI) methods to examine complex and potentially nonlinear structures in high-dimensional data by applying the concept of DNNs to GWAS in the first step of the algorithm. Subsequently, in step 2, DeepCOMBI identifies a set of SNPs that have high effects on the classification result of the DNN (either individually or in combination with other SNPs and not due to correlation structures) by calculating an explanation score for each SNP that reflects its contribution to the final classification decision. The third and last step assigns individual P-values to all selected SNPs and quantifies their relevance with a permutation-based significance threshold.
MATERIALS AND METHODS The proposed method applies deep learning and the concept of explainable AI to GWAS data and enables the identification of SNPs that are associated with a given trait with statistical significance. A graphical representation of the method is given in Figure  1. The method is based on a deep learning step that trains a DNN for the classification of GWAS subjects into their respective phenotype class. Using LRP as a post-hoc explanation method, we access the relevancies of all SNPs regarding each individual classification result. The obtained SNP relevance scores are used to select the subset of most important SNPs to test for association in the final multiple testing step. In the following sections, we describe the statistical problem, which is investigated in a GWAS, present the proposed method in detail and specify the experimental setup of performance assessments on generated synthetic data and a real-world application of a known GWAS dataset.
Selecting a DNN architecture is often critical for achieving good performance for a specific, in this case, SNP-based classification task. Montaez et al. (43)  developed a 2-class DNN for the classification of polygenic obesity and have successfully shown its performance to be superior to numerous competitor methods. Romagnoni et al. (28)  have compared the performance of similar architectures and have presented a detailed review of the best design choices for a DNN on a Crohn's disease dataset. Taking inspiration from the conclusions of both of these works and having checked performances on synthetic datasets, we use an architecture of two fully connected layers with 64 neurons and ReLU activations and a dense softmax output layer with two output nodes. To improve validation accu-racy by reducing overfitting, each hidden layer is followed by a dropout layer with a dropout probability of φ.
To overcome limitations due to imbalanced datasets, class weights were calculated according to the class frequencies and used to direct the DNN to balance the impact of controls and cases. Once the parameters w of the DNN have been trained by optimizing the above learning problem, the network is able to predict the phenotype of any unseen genotype x. Regarding this binary classification problem, the output node with the highest score represents the predicted phenotype. In a preprocessing step, the data are centered and scaled by subtracting the global mean and dividing by the global standard deviation. To minimize computational effort and limit the number of model parameters in the DNN, a Pvalue threshold κ can be applied in order to only select SNPs with P-values smaller than κ to be used for training.
This preprocessing step can be applied to large datasets when limited computational resources are available. While all datapoints remain in the dataset, the number of features of the dataset is decreased to only train the DNN on the SNPs with a P-value smaller than the threshold. The set of features remains this size for all further steps (i. e. , DNN training, LRP explanation, SNP selection, etc. ) and all SNPs with a P-value above the threshold automatically get assigned a relevance score of 0 and are no longer candidates for the DeepCOMBI method. For a discussion of the potential effects of this feature selection step and potential adoptions to improve it, please refer to the Discussion section below. The second step of DeepCOMBI--Explanation and SNP selection. To harness the potential of DNNs in the identification of SNP disease associations in GWAS, we now apply the concept of explainable AI.
Baselines In order to evaluate the performance of the proposed DeepCOMBI method in comparison to competitor approaches, we select a set of representative baseline methods. RPVT is chosen as the most widely used traditional, purely statistical testing approach. As a machine learningbased method and the methodological background of DeepCOMBI, we select COMBI as the main competitor method we aim to succeed in terms of performance. Since the COMBI method was shown to outperform other combinatorial machine learning-based approaches  (6, 58, 59 ) and a number of purely statistical analysis tools  (21, 27)  on the same datasets evaluation methods used here, there is no need to compare to all of those methods again. RPVT as a baseline. Raw P-value thresholding (RPVT) is a statistical framework traditionally used in GWAS for identifying significant associations between SNPs and traits.
The Bonferroni correction works well under the assumption that all null hypotheses are independent of each other, which is not the case here. Indeed, since SNPs show high degrees of correlation through LD, the Bonferroni correction can become extremely conservative, leading to a high rate of false rejections, which is why the scientific community mostly applies a fixed threshold that remains constant for multiple GWAS. Here, based on the original publication of the data we are analyzing (WTCCC data, see the Materials and Methods section on validation datasets (  14 )) and the findings of Mieth et al. (31) , we present not only the strong associations at a significance level of t * = 5x10 -7 but also weak associations at t * = 1 × 10 -5. COMBI as a baseline. The COMBI method (31) combines machine learning with multiple hypothesis testing to improve the statistical power of GWAS.
Validation datasets Validation on generated datasets. To create a realistic but controlled environment where the ground truth labels of a dataset, i. e. the SNPs that are indeed linked to the disease, are known, we generate semi-synthetic data for a first evaluation of DeepCOMBI and the baseline methods from above. We follow the instructions for the creation of such GWAS datasets proposed by Mieth et al. (31). The basic concept is to take an ensemble of real genotypes and generate a synthetic phenotype for each subject according to a specific rule. With this method, the underlying architecture of the genome, including, for example, genetic LD and correlation structures, is kept intact while control over the phenotypic labels is gained at the same time. We use the WTCCC dataset (  14 ) described in more detail below and randomly select 300 subjects of the Crohn's disease dataset.
At the same time, the tower structure is limited to those 20 informative positions because there are no correlations of those 20 SNPs with the surrounding 10 000 noise SNPs coming from chromosome 2. The random generation process will also ensure that the datasets will have associations of different strengths to the 20 informative SNPs. The complete data generation process is repeated to generate 1000 datasets. DeepCOMBI and all baseline methods are applied to each dataset with an 80:20 class balanced split in training and test data. The prediction results on the test data are evaluated with the known ground truth of only 20 informative SNPs at the positions 5000-5020 and the corresponding performance can be measured in terms of the number of true and false positives for each method. Since we have adopted the data generation procedure from Mieth et al.
(31) , let us point out that we do not perform a full investigation on its assumptions and its ability to produce realistic GWAS datasets here. Please refer to Mieth et al. (31)  and in particular its Supplementary File where Chapter 3 is dedicated to all necessary experiments and corresponding results can be found. To assess the performance of the data generation procedure here, we include Manhattan plots of three exemplary datasets in Figure  2. To sum up, the experiments performed in Mieth et al. (31) , they investigated the effect size parameter γ and identified γ = 6 to be the value that yields the most realistic tower structures. This investigation was based purely on the RPVT P-values of the resulting datasets and the level of difficulty in the generated datasets was evaluated based on these P-values alone.
In addition, it was shown that an effect size of 6 was the value where the permutation test procedure produced the anticipated family-wise error rate of 0. 05. Mieth et al. (31)  provide analyses assessing the difficulty of the problem under investigation by estimating it with the true-positive rate of RPVT. Extraordinary cases where the generated datasets are either exceptionally hard or exceptionally easy are also investigated. Furthermore, the effect of misspecifications with effect sizes too high or too low is examined. Validation on WTCCC data. For evaluation on real-world genomic data, the performance of DeepCOMBI was assessed on the Wellcome Trust Case Control Consortium phase 1 dataset, released in 2007 14 , featuring the genotypic information on 500 000 SNPs of 17 000 British subjects.
With 3000 shared controls and 2000 case samples for seven major human diseases (Crohn's disease [CD], type 1 diabetes [T1D], type 2 diabetes [T2D], coronary artery disease [CAD], hypertension [HT], bipolar disorder [BD] and rheumatoid arthritis [RA]), it was a landmark study both in terms of sample size and dimensionality at the time of its publication. For our analysis, a case-control dataset for each disease was created, removing all SNPs and samples that did not fulfill the quality control criteria provided in the original WTCCC paper. In agreement with the lack of interchromosomal LD and the findings of Mieth et al. (31) , who showed no significant performance increase with genomewide training, the DeepCOMBI method and all baseline methods were applied to each chromosome separately. For evaluation purposes, the concept of replicability was applied proposed by Mieth et al.
(31). Since the true under-lying genetic architecture of the given traits, i. e. the sets of informative SNPs for each disease, is unknown, an approximation of the truth was created by employing the GWAS catalog  (62)  and examining the results of the 13 years of independent studies after the WTCCC dataset was published. To evaluate the reported finding of a method (DeepCOMBI or competitor), the GWAS catalog (accessed on 30 July 2020) is inquired for that SNP and all SNPs in LD ([R 2 > 0. 2] according to PLINK LD calculations (  63 )) within a 200 kb window around that SNP. If an association with the disease with P-value <10 -5 of the SNP itself or the SNPs in LD was reported by at least one independent GWAS published after the WTCCC study, the reported SNP is counted as a true positive finding.
Parameter selection The application of DeepCOMBI requires the determination of a number of free parameters. In the following sections, we present the selected optimal parameter values and describe the process of finding them for the different datasets under investigation. Parameter selection on generated datasets. For the generation process of semi-synthetic datasets, all parameters were selected according to the information given by Mieth et al. (31). Most importantly, the effect size parameter was set to γ = 6. When applying the DeepCOMBI method to the generated datasets, we studied the effect of all hyperparameters on the performance of the DNN. An accuracy-based random grid search with a stratified split in 90% training and 10% testing data was conducted. Here, we present the selected most successful values and the investigated parameter intervals in parentheses: • number of neurons per dense hidden layer nn = 64  [2, 4, 8, 16, 64] , • L1 regularization coefficient τ = 0.
0001 [0, 1e-6, 1e-5, 1e- 4, 1e-3,1e-2, 1e-1], • L2 regularization coefficient υ = 0. 000001 [0, 1e-6, 1e-5, 1e-4, 1e-3,1e-2, 1e-1], • dropout rate φ = 0. 3 [0. 3, 0. 5], • learning rate η = 0. 01 with learning rate reduction on a plateau with factor 0. 7125 after 50 epochs of no improvement and • number of epochs e = 500 [100, 500, 1000]. A few different parameter values of the αβbackpropagation rule were manually investigated on exemplary datasets. By visually inspecting the resulting LRP vectors and their corresponding DeepCOMBI P-values, the combination of α = 1 [0, 1, 2] and β = 0 [0, 1, 2] was found to be best. For post-processing the global relevance scores and selecting the most relevant SNPs, we assumed that the most successful values found by Mieth et al. (31)  would also be a good choice for our method.
Hence, we set the window size of the moving average filter to l = 35, the norm parameter of the moving average filter to P = 2 and the SNP selection  parameter to k = 30. These values were found to be in agreement with the biological background of the data, e. g. l = 35 reflects the reach of LD along a genetic sequence  (31). Parameter selection on WTCCC data. To choose hyperparameters for the DNN trained on WTCCC data in the first step of DeepCOMBI, a parameter search was run on a single dataset. The Crohn's disease chromosome 3 dataset was selected as a good representative, and an accuracy-based parameter search with a stratified split in 90% training and 10% testing data was conducted. We studied the effect of the hyperparameters on the performance of the DNN and the best performing hyperparameters were as follows (tested intervals in parentheses): • number of neurons per dense hidden layer nn = 64  [2, 4, 8, 16, 64] , • L1 regularization coefficient τ = 0.
001 [0, 1e-6, 1e-5, 1e- 4, 1e-3,1e-2, 1e-1], • L2 regularization coefficient υ = 0. 0001 [0, 1e-6, 1e-5, 1e- 4, 1e-3,1e-2, 1e-1], • dropout rate φ = 0. 3 [0. 3, 0. 5], • P-value threshold κ = 1e-2 [1e-4, 1e-2, 1], • learning rate η = 0. 00001 [1e-7, 1e-6, 1e-5, 1e-4, 1e-3,1e-2, 1e-1], • number of epochs e = 500  [100, 500, 1000]. Detailed results on the classification performance of the final training parameter settings and the corresponding evolution of training and validation measures on an exemplary dataset can be found in the Results section. As before, we visually investigated a few different parameter values of the αβ -backpropagation rule and their influence on both the resulting relevance scores and P-values. On the Crohn's disease chromosome 3 dataset, the combination of α = 2 [0, 1, 2] and β = 1 [0, 1, 2] was found to be optimal.
After manually investigating the global LRP scores and the corresponding DeepCOMBI P-values of the exemplary dataset (Crohn's disease chromosome 3), we found that slightly different settings than for the analysis of the generated datasets should be applied for post-processing the relevance vectors and selecting the most relevant SNPs. Namely, the window size of the moving average filter should be set to l = 21 and the SNP selection parameter should be increased to k = 200. The need for a decreased filter size and an increased number of selected SNPs might be caused by the application of the P-value-based preselection step for limiting the number of model parameters, which is only applied to the real dataset and not the generated datasets. To determine the value of the significance level α to be used in the permutation test procedure of the last steps DeepCOMBI, we follow the recommendations of Mieth et al.
Results on generated datasets Here, we report our results averaged over the 1000 data sets generated in the simulation process described in the Materials and Methods section ('Validation on generated datasets'). We show that, on these data sets, DeepCOMBI performs better than the traditionally used method for analyzing GWAS, RPVT, and its main competitor, the COMBI method. Prediction performance on generated datasets. The first steps of both DeepCOMBI and COMBI consist of training a learning algorithm for the classification of all sub-jects into their respective phenotypic group given their genotypic information. Since all following steps depend on the performance of these classifiers, high prediction accuracy is crucial. On the generated datasets, the SVM (as part of the COMBI method) achieves 59% accuracy on average and 54% balanced accuracy.
In comparison, the DNN (as part of the DeepCOMBI method) performs significantly better with an average of 74% classification accuracy and also avoids negative effects of unbalanced datasets more effectively by applying class weights in the DNN training (74% balanced accuracy). Accuracy scores and additional information are given in Table  1. Following these promising intermediate results, in the next section, we investigate whether the entire workflow of the DeepCOMBI method can also outperform the baseline methods. SNP selection performance on generated datasets. To compare the relevance scores and P-values obtained with the novel LRP-based method to those derived from the SVM weights in the COMBI method, we look at three exemplary synthetic datasets and the corresponding results (see Figure  2 ).
In contrast, DeepCOMBI is successful for both the second and third replication with moderate and strong associations and only misses the very weak association (last column of subfigures). Please note that DeepCOMBI not only precisely identifies the correct informative tower but also filters out a relatively high noise tower at around position 600, which--just by chance--achieved a P-value < 10 -5. The method thus not only increases the probability of finding the correct tower but also, and potentially more importantly, decreases the probability of falsely selecting a noise tower  (31). To investigate whether these exemplary findings represent a general trend, we now examine the results of all competitor methods averaged over all 1000 generated datasets. In Figure  3 , the corresponding ROC and PR curves are shown.
Results on WTCCC data Prediction performance on WTCCC data. In the first step of DeepCOMBI, a DNN is trained and we present the evolution of both training and validation loss and accuracy during training on an exemplary dataset (i. e. the Crohn's disease chromosome 3 dataset) in Figure  4. Overfitting is avoided and both training and validation accuracy increase during training and reach an optimum at the end of the training process (to be seen on the left). Good generalization can also be seen on the right where the model loss decreases with each epoch on both training and validation data. We now investigate the performance of the DNN on all diseases and chromosomes. Figure  5  shows that the DNN of DeepCOMBI performs consistently better than the SVM of COMBI in terms of all four validation metrics described in the Materials and Methods section under 'Performance metrics'.
SNP selection performance on WTCCC data. In Figure  6 , we present the results of the traditional RPVT approach, the COMBI method and the DeepCOMBI applied to the seven diseases of the WTCCC 2007 dataset. In each corresponding Manhattan plot, the negative logarithmic Pvalues of all SNPs at a given position in a chromosome are shown. While RPVT assigns P-values smaller than one (i. e. nonzero in the plots on a logarithmic scale) to all SNPs and, in consequence, produces a lot of statistical noise, both COMBI and DeepCOMBI discard most SNPs (i. e. assign P-value one, i. e. zero in the plot on a logarithmic scale) and hence reduce the level of noise significantly. The COMBI method selects 100 SNPs with high SVM weights per chromosome and DeepCOMBI chooses 200 SNPs with high LRP scores. In all plots, the significance threshold t * is represented by dashed horizontal lines and all statistically significant SNP associations are highlighted.
For comparison, we access this list via Supplementary Table  S2  presented by Lippert et al. (21)  that includes 573 SNPs associated with six diseases (i. e. no reports for CAD). From this list, we exclude any SNPs that were discovered by them with a method other than LMM or that were not included in the dataset analyzed with the DeepCOMBI method (potentially due to different preprocessing steps) to ensure a fair comparison and avoid ascertainment bias. Since a lot of the remaining 110 SNPs are part of small SNP clusterings, we select representative markers for each locus through the LD pruning option in PLINK  (63)  and compute pairwise LD with a sliding window of two SNPs (with steps of 1 SNP at a time). We discard one SNP out of each pair if they are in high LD (R 2 ≥ 0. 8). We run the final list of 72 SNPs, consisting of 1 discovery for BD, 0 for CAD, 19 for CD, 1 for HT, 3 for RA, 39 for T1D and 9 for T2D, through our validation pipeline using the same parameters that we use to evaluate the performance of DeepCOMBI (physical dis-tance to tag-SNP: < 200 kb.
With the increasingly larger amounts of available data, deep learning-based approaches and artificial DNNs are now also being applied to GWAS datasets  (41, 42). However, most of these publications focus on pure classification or regression prediction tasks  (28, (43) (44) (45)  rather than the identification of associated SNPs in the corresponding datasets  (28, 46). To fill this gap and firmly based on the combinatorial approach of the COMBI method  (31) , the proposed Deep-COMBI method uses a deep learning-based phenotype prediction in combination with statistical testing for the identification of SNPs that are associated with the phenotype under investigation. DeepCOMBI could be considered an extension of COMBI, replacing the rather simple prediction tool of a linear SVM with a more sophisticated deep learning method and using the recent concept of explainability to uncover the decision-making process of DNNs and extract SNP relevance scores via LRP  (48) (49) (50).
To our knowledge, Romagnoni et al. (28)  were the first and only scientists to use explainable AI in the context of GWAS and proposed to apply PFI. Even though they were able to identify some novel predictors, the prediction performance of their DNN was not better than that of traditional ma-Table  3. Quantitative summary of the significant findings of RPVT, COMBI and DeepCOMBI. For each of the three competitor methods, the numbers of replicated and unreplicated hits (i. e. the number of true and false positives) as well as precision and error rates are presented. Pairwise tests for the null hypothesis of identical distributions for DeepCOMBI and the two baseline methods are performed and the corresponding P-values are given   DeepCOMBI was shown to compare favorably to its main competitor COMBI on both generated controlled datasets as well as seven real-world GWAS datasets.