dataesr/scientific-paragraphs-categorization huggingface dataset:
The parallel Forward-Backward with pruning algorithm We present the proposed Parallel Forward-Backward with Pruning (PFBP) algorithm, shown in Algorithm 2. To improve readability, several arguments are omitted from function calls. PFBP takes as input a dataset D and the target variable of interest T. Initially the number of Sample Sets ns and number of Feature Sets nf are determined as described in Sect. 5. Then, (a) the samples are randomly assigned to Sample Sets S 1 ,. , S ns , to avoid any systematic biases (see also Sect. S ← BackwardIteration(D, T , S) 26: end while 27: return S in order of occurrence in the dataset, and (d) the dataset D is partitioned into data blocks D i, j , with each such block containing samples and features corresponding to sample set S i and feature set F j respectively.
As p-values tend to get smaller with larger sample sizes (in case the null hypothesis does not hold), they quickly reach the machine epsilon, and will be rounded to zero. If this happens, then sorting and selecting features according to p-values breaks down and PFBP will select an arbitrary feature. This behavior is further magnified in case of combined p-values, as a single zero local p-value leads to a zero combined pvalue no matter the values of the remaining p-values. The R language provides the option to directly compute the logarithm of the p-value (using the option log. p = T ). Next, we give pointers to our implementation, since there was no other implementation available for Spark. For the χ 2 distribution, the p-value can be computed as Γ (k/2,x/2) Γ (k/2) , where x is the test statistic, k the degrees of freedom, Γ (•, •) the incomplete gamma function and Γ (•) the gamma function.
Score tests for the univariate case In the first step of forward selection where no variable has been selected, one can use a score test (also known as Lagrange multiplier test) instead of a likelihood-ratio test to quickly compute the p-value without having to actually fit logistic regression models. The statistic of the Score test equals  (Hosmer et al. 2013 ) Statistic ≡ n j=1 X j (T j -T ) T (1 -T ) n j=1 (X j -X ) 2 where n is the number of samples, T is the binary outcome variable (using a 0/1 encoding), and X is the variable tested for independence. Note that, such tests can also be derived for models other than binary logistic regression, but it is out of the scope of the paper. The score test is asymptotically equivalent to the likelihood ratio test, and in anecdotal experiments we found that a few hundred samples are sufficient to get basically identical results, justifying its use in Big Data settings.
By default, SFO uses a maximum number of variables to select as a stopping criterion. Alternatively, to decide whether R * should be selected a likelihood-ratio test could be used, in which case the test is performed on M and M * , and R * is selected if the p-value is below a threshold α; we used this in our implementation of SFO in the experiments. The parallelization over samples is performed in the map phase, in which one value p j is computed for each sample j, which equals p j = e β•S j 1 + e β•S j where β are the coefficients of S in M and S j are the values of S in the jth sample. The values of p j , the values of the outcome T and all of candidate variables R are then sent to workers to be processed in the reduce phase. Note that, this incurs a high communication cost, as essentially the whole dataset has to be exchanged among workers over the network.
In Big Data settings it can be easily parallelized. However, ITFS methods do not have the same theoretical properties of PFBP, which can be shown to be optimal for distributions that can be faithfully represented by Bayesian networks and maximal ancestral graphs. This stems from the fact that PFBP solves an inherently harder problem, as it conditions on all selected features (creates a model with all selected features) at each iteration in order to select the next feature, while ITFS only conditions on one feature at a time. Furthermore, ITFS methods are not as general as PFBP, which can be applied to various data types as long as an appropriate conditional independence test is available. For example, it is not clear if and how ITFS can be applied to time-to-event outcome variables or time-course data.
7). Conditions for optimal feature selection with Lasso are given in  Meinshausen and Bühlmann (2006). While Lasso is defined as a global optimization problem, it has been proven that many problems can be solved with a greedy Forward-Backward procedure [e. g. , for linear regression  (Efron et al. 2004) ]. Comparing Lasso with algorithms related to PFBP, we note that in extensive simulations it has been shown that causally-inspired feature selection methods are competitive in terms of predictive performance with Lasso on classification and survival analysis tasks on many real datasets  (Aliferis et al. 2010; Lagani and Tsamardinos 2010; Lagani et al. 2013 Lagani et al. , 2017)). Furthermore, the non-parallel version of PFBP (called Forward-Backward Selection with Early Dropping) as well as the standard Forward-Backward Selection algorithm have been shown to perform as well as Lasso when restricted to select the same number of variables  (Borboudakis and Tsamardinos 2017).
(  2014 ) introduced a general, parallel feature selection method for classification tasks that is based on group testing theory. The idea is to select a collection of tests (i. e. , a subset of the input variables), with each variable being present with some probability p. This implicitly creates multiple datasets, one for each test. Those datasets can be processed independently in parallel, producing a score using some scoring function for each dataset corresponding to how predictive the respective variables are of the outcome of interest. Then, variables are ranked based on the scores attained on the datasets they participated in. The authors show that, for specific values of p, if the number of tests is large enough and if the scoring function satisfies some properties (i. e. , it is what the authors call C-separable), the proposed algorithm is able to select the best features with high probability.
However, it is not clear how those results are related to the theoretical properties of other feature selection algorithm (such as PFBP or Lasso), nor how they can be used in practice, especially given that the authors do not propose any way to tune the hyper-parameters of their method. Wang et al. (2016)  proposed a method that also uses vertical partitioning for parallelization. It is method that can be used with any penalized regression method, both for feature selection and for model fitting. Given a dataset, it is partitioned vertically into multiple datasets, and a decorrelation step is performed on each new dataset that tries to minimize the dependencies between all datasets. Then, each dataset can be analyzed independently across multiple workers, and the results are combined into a final one on the master machine.
Experimental setup For the scalability experiments of PFBP (Sect. 9. 2) and the proof-of-concept application on SNP data (Sect. 9. 4) we used a cluster with 5 machines, 1 acting as a master and 4 as workers, connected to a 1 Gigabit network, with each machine having 2 Intel Xeon E5-2630 v3 CPUs with 8 physical cores each and 256 GB of RAM (a total of 64 cores and 1 TB of RAM). For the comparative evaluation of PFBP with other feature selection algorithms we used a cluster with 5 workers, connected to a 14 Gigabit network, with each machine having 4 Intel Xeon E5-4650 v2 CPUs with a total of 80 cores and 400 GB of RAM (a total of 320 cores and 1. 6 TB of RAM). In all cases, 2 cores per worker were left out for other tasks (e. g. the operating system). The first cluster is running Spark 2. 1.
Scalability of PFBP with sample size, feature size and number of workers We investigated the scalability of PFBP on dense synthetic datasets in terms of sample size, variable size and number of workers used. The data were sampled from randomly generated Bayesian networks, which are probabilistic models that can encode complicated dependency structures among features. Such simulated data contain not only features necessary for optimal prediction [strongly relevant in the terminology of  John et al. (1994) ] and irrelevant, but also redundant features [weakly relevant  (John et al. 1994) ]. This is a novelty in the Big Data FS literature as far as we know, making the synthetic tasks more realistic. A detailed description of the data and network generating procedures is given in "Appendix B".
For each experimental setting, we generated 5 Bayesian networks, and sampled one dataset from each. The connectivity parameter C was set to 10 (i. e. , the average degree of each node), the class distribution of T was 50/50, and the variance of the error term was set to 1. To investigate scalability in terms of sample size, we fixed the number of features to 1000 and varied the sample size from 2 to 10 M. Scalability in terms of feature size was evaluated on datasets with 100K samples and varying the feature size from 20 to 100 K, all of which also included the optimal feature set (i. e. the Markov blanket of T ). Finally, scalability in terms of number of workers was investigated on datasets with 10 K variables and 1M samples. The maximum number of variables maxVars to select was set to 50.
Comparative evaluation of PFBP on real datasets We evaluated the PFBP algorithm on 13 binary classification datasets, collected from the LIBSVM dataset repository,  16  with the constraint that each dataset contains at least 500K samples or variables. A summary of the datasets, shown in order of increasing variable size, is shown in Table  2. The first two columns show the total number of samples and variables of each dataset, while the last column shows the average number of non-zero elements of each sample. The maximum number of non-zero elements equals the total number of variables. Except for the first four datasets, all other datasets are extremely sparse. All algorithms were compared in terms of classification accuracy and running time. To estimate the classification accuracy, 10% of the training instances were randomly selected  The fastest algorithms are shown in bold, while algorithms that timed out are indicated with an asterisk.
PFBP significantly outperforms all competitors in terms of running time, and is the only algorithm that is able to terminate for all datasets within the given time limit of 12 h. Furthermore, except for 2 cases (FBS on the epsilon dataset and SFO on the rcv1 dataset; see Table  4 ), none of the competing algorithms were able to select a single variable within 12 h and kept out. The remaining 90% were used by each algorithm to select a set of features and to train a logistic regression model using those features. The maximum number of features to select was set to 50. We note that, for PFBP, the backward phases and the second phase were only executed if the algorithm terminated (i. e. , the remaining variables were empty) before the variable limit was reached. This was done because PFBP would not have terminated otherwise (i.
Overall, all feature selection methods perform similarly, with PFBP and SFO typically having the best predictive performance. PFBP achieves the better or on par performance by selecting fewer variables than its competitors. Furthermore, in most cases, combining models CombLR works as well or better than the logistic regression of MLlib, SparkLR the results of the trivial classifier, which assigns each sample to the most frequent class, and thus attains an accuracy equal to the frequency of the most common class. It can be seen that PFBP outperforms all competing methods in terms of running time. SFO, UFS, and FBS only terminate selecting at least 1 feature in the smallest, first 4 datasets. UFS and FBS reach the timeout limit and do not select a single feature even for the moderately sized rcv1 dataset, which contains only 47K variables and 698K samples, while SFO is able to select only a single feature in 12 h.
17 PFBP is able to terminate for all datasets within 12 h, taking a maximum of 10. 5 h for the kdd2010b dataset which contains 19M samples and 30M variables. In terms of predictive performance, PFBP always produces the best or an equally predictive model. When a competitor produces a better model the difference is in order of 0. 01% of accuracy. Some larger differences are observed only when the final model is fit with the SparkLR. In terms of the number of features, PFBP always selects the lowest number of features to achieve the same or better performance. An exception is when FBS timed-out for the epsilon dataset selecting 10 features versus 50 for PFBP; however, the lower number of features comes with a significant drop in performance of about 10% of accuracy.
Comparison of PBFP with information theoretic feature selection methods Next, we compare PFBP to three algorithms of the family of information theoretic feature selection (ITFS) methods  (Brown et al. 2012 ): the Minimum-Redundancy Maximum-Relevance (MRMR) algorithm  (Peng et al. 2005) , the Joint Mutual Information (JMI) algorithm  (Yang and Moody 2000)  and the Conditional Mutual Information Maximization algorithm  (Fleuret 2004 ). Those methods were chosen as they have been shown to be perform well compared to several other members of the ITFS family  (Brown et al. 2012). For all of the above algorithms, we used existing Spark-based implementations 18   (Ramrez-Gallego et al. 2017). As information-theoretic feature selection methods require discrete data, we performed a simple discretization method on all sparse datasets (i.
In most cases, PFBP produces better methods, often significantly so, having a higher accuracy of up to 5-9% on the rcv1, news20 and webspam datasets. As before, in most cases, combining models works as good or better than the implementation in MLlib treats data as dense and is not specific or optimized for discrete data. In addition, PFBP's current implementation is based on fitting logistic regression models that require iterative techniques, while ITFS methods only require the counts in the contingency tables of the joint distributions. In terms of classification accuracy, PFBP outperforms ITFS methods in most cases. In some cases (rcv1, news20, webspam) the accuracy difference is more than 4%; when PFBP does not produce the best model, the accuracy difference is less than 0. 1%. Thus, overall, if the goal is predictive accuracy, PFBP should be preferred over ITFS methods.
Proof-of-concept application on genetic SNP data Single Nucleotide Polymorphisms (SNPs),  19  the most common type of genetic variation, are variations of a single nucleotide at specific loci in the genome of a species. The Single Nucleotide Polymorphism Database (dbSNP) (build 150)  (Sherry et al. 2001 ) now lists 324 million different variants found in sequenced human genomes. 20 In several human studies, SNPs have been associated with genetic diseases or predisposition to disease or other phenotypic traits. As of 2018-07-17, the GWAS Catalog 21  contains 3471 publications and 65793 unique SNP-trait associations. Large scale studies under way [e. g. , Precision Medicine Initiative  (Collins and Varmus 2015) ] intend to collect SNP data in large population cohorts, as well as other medical, clinical and lifestyle data.
SNP data generation and setup We simulated genotype data containing 500,000 individuals (samples) and 592,555 SNP genotypes (variables), following the procedure described in  Canela-Xandri et al. (2015). As SNP data are dense, they require approximately 2. 16 TB of memory, and thus are more challenging to analyze than sparse data, such as the ones used in the previous set of experiments. The data were simulated with the HAPGEN 2 software  (Chang et al. 2015)  from the Hapmap 2 (release 22) CEU population  (Consortium 2015). A more detailed description of the data generation procedure is given in "Appendix C". We used M = 100 randomly selected SNPs to generate a binary phenotype (outcome), as described in  Canela-Xandri et al. (2015)  (see also "Appendix C"). The optimal accuracy using all 100 SNPs is 81.
Results on the SNP data PFBP was able to select 84 features in 15 h, using a total of 960 core hours. It achieved an accuracy of 77. 62%, which is over 95% of the theoretical optimal accuracy. The results are very encouraging; in comparison the DISSECT software  (Canela-Xandri et al. 2015)  took 4 h on 8400 cores (that is, 33,600 core hours) and using 16 TB of memory to fit a mixed linear model on similar data, and to achieve an accuracy around 86% of the theoretical maximum. The two experiments are not directly comparable because (a) the outcome in our case is binary instead of continuous requiring logistic regression instead of linear regression models favoring DISSECT in terms of computational time, (b) the scenarios simulated in  Canela-Xandri et al. (2015)  had larger Markov blankets (1000 and 10000 instead of 100) favoring PFBP (although, their results are invariant to the size of the Markov blanket).
C: SNP data generation To generate the SNP dataset we followed the procedure described in  Canela-Xandri et al. (2015). We used the HAPGEN 2 software  (Chang et al. 2015)  with the Hapmap 2 (release 22) CEU population  (Consortium 2015)  to simulate 500000 individuals (samples). This population contains 2543887 SNPs, but only 592555 were kept, by filtering out the ones not available in the Illumina Human OmniExpress-12 v1. 1 BeadChip. 22 The final dataset contains 500000.
6. 1), (b) the Sample Sets S 1 ,. , S ns are assigned to Q approximately equal-sized Groups, G 1 ,. , G Q , (c) the features are assigned to feature sets F 1 ,. , F nf , Algorithm 2 Parallel Forward-Backward With Pruning (PFBP) Input: Dataset D, Target T , Maximum Number of Runs maxRuns Output: Selected Variables S 1: //Data Partitioning 2: Randomly assign samples to sample sets S 1 ,. , S ns 3: Assign sample sets S 1 ,. , S ns to equally-sized Groups G 1 ,. , G K 4: Assign features to feature sets F 1 ,. , F n f 5: Partition D to data blocks D i, j containing samples from S i and F j , ∀i, j 6: 7: S ← ∅ //No selected variables 8: run ← 1 //First run 9: 10: //Iterate until (a) maximum number of runs reached, or (b) selected features S did not change 11: while run ≤ max Runs ∧ S changes do 12: S ← OneRun(D, T , S) 13: run ← run + 1 14: end while 15: return S 16: function OneRun(Data Blocks D, Target T , Selected Variables S, Maximum Number of Variables To Select maxVars) 17: R ← F\S //All variables remaining 18: //Forward phase: iterate until (a) maximum number of variables selected or (b) no new variable has been selected 19: while |S| < maxVars ∧ S changes do 20: S, R ← ForwardIteration(D, T , S, R) 21: end while 22: 23: //Backward phase: iterate until no variable can be removed 24: while S changes do 25:.
, the experiments indicate that PFBP scales superlinearly with the available sample size, and linearly with the number of features and available workers. Compared with other algorithms in its class, namely forward selection-based algorithms with map-reduce implementations, under the same conditions (type of test and predictive model), PFBP dominates the alternatives (UFS, FBS, SFO) in terms of execution time, number of selected features, and predictive performance. Against information-theoretic variants specialized for discrete and sparse data with available map-reduce implementations, PFBP performs worse in terms of running time, however, it is still applicable and practical to apply to large datasets. However, PFBP dominates the information-theoretic variants in terms of predicting performance.
Experimental evaluation We performed three sets of experiments to evaluate PFBP. 1. We investigate the scalability of PFBP in terms of variable size, sample size and number of workers on synthetic datasets, simulated from Bayesian networks. 2. We compare PFBP to three competing forward-selection based feature selection algorithms. 3. We compare against three algorithms from the family of information theoretic feature selection methods (Brown et al. 2012), implemented for Big Data ( Ramrez-Gallego  et al. 2017). 4. We performed a proof-of-concept experiment of PFBP on dense synthetic Single Nucleotide Polymorphism (SNP) data. These are important types of very highdimensional data that arise in biology and for which feature selection algorithms that can scale up are needed. Acknowledgements IT and GB have received funding from the  European Research Council  under the European Union's  Seventh Framework Programme  (FP/2007-2013) /  ERC  Grant Agreement n.
Background: Electroencephalography (EEG) is a popular method to monitor brain activity, but it is difficult to evaluate EEG-based analysis methods because no groundtruth brain activity is available for comparison. Therefore, in order to test and evaluate such methods, researchers often use simulated EEG data instead of actual EEG recordings. Simulated data can be used, among other things, to assess or compare signal processing and machine learning algorithms, to model EEG variabilities, and to design source reconstruction methods. New method: We present SEREEGA, Simulating Event-Related EEG Activity. SEREEGA is a free and open-source MATLAB-based toolbox dedicated to the generation of simulated epochs of EEG data. It is modular and extensible, at initial release supporting five different publicly available head models and capable of simulating multiple different types of signals mimicking brain activity.
This paper presents the architecture and general workflow of this toolbox, as well as a simulated data set demonstrating some of its functions. The toolbox is available at https://github. com/lrkrol/SEREEGA. Results: The simulated data allows established analysis pipelines and classification methods to be applied and is capable of producing realistic results. Comparison with existing methods: Most simulated EEG is coded from scratch. The few open-source methods in existence focus on specific applications or signal types, such as connectivity. SEREEGA unifies the majority of past simulation methods reported in the literature into one toolbox. Conclusion: SEREEGA is a general-purpose toolbox to simulate ground-truth EEG data.
Some authors have not implemented their own methods, but have instead relied on commercially available packages (e. g. ,  Yao & Dewald, 2005; Lansbergen, van Dongen-Boomsma, Buitelaar, & Slaats-Willemse, 2011). These packages, however, are not specifically designed for the purpose of simulation, nor are they freely available to the general public or open to scrutiny. One open-source package known to the authors that provides simulation functionality is the Source Information Flow Toolbox (SIFT;  Delorme et al. , 2011). Its main purpose is to investigate structural, functional, and effective connectivity between brain regions and networks, but the toolbox can also be used to simulate scalp EEG using a system of coupled oscillators. Such simulations can serve as effective ground truth for the same connectivity measures that the toolbox investigates, and have been used to evaluate blind source separation methods as well  (Hsu, Mullen, Jung, & Cauwenberghs, 2014).
The SIFT toolbox, however, is not intended for wider-purpose simulation and as such, the methods that can be meaningfully applied to the data it can generate are limited. Another work allows a number of predefined responses (e. g. a P300, an eye blink, a frequency spike) to be configured along a given timeline of events  (Lindgren, Merlini, Lecuyer, & Andriulli, 2018). It focuses on effects specific to brain-computer interface (BCI) applications and is aimed at the batch analysis of BCI classification algorithms. In a more normative approach,  Haufe and Ewald (2016)  proposed a simulation and evaluation framework and made their code publicly available. The framework includes forward modelling using a realistic head model, both source and sensor noise with controlled signal-to-noise ratios, and signal generation based on autoregressive models.