dataesr/scientific-paragraphs-categorization huggingface dataset:
Thus, the rule ĝn,P associated to the path P ∈ Π is formally defined by ∀x ∈ R p , ĝn,P (x) =    1 Nn( Ĥn(P)) n i=1 Y i 1 X i ∈ Ĥn(P) if x ∈ Ĥn (P) 1 n-Nn( Ĥn(P)) n i=1 Y i 1 X i / ∈ Ĥn(P) otherwise , using the convention 0/0 = 0, and where N n ( Ĥn (P)) is the number of observations in the node associated with P. This formal definition can be illustrated with the Titanic dataset presented in the introduction. For the fourth rule, fare is the 6th variable and since q(6) n,4 = 10. 5, the corresponding path is P = {(6, 4, L)}, and the associated rule is thus ĝn,P (x) = 0. 20 if x (6) < 10. 5 0. 50 if x (6) ≥ 10. 5. Finally, a Θ-random tree generates a collection of paths in Π, one for each internal and terminal nodes. In the sequel, we let T (Θ, D n ) be the list of such extracted paths, a random subset of Π.
4 for its tuning procedure. We are thus interested in the set of the extracted paths PM,n,p 0 = {P ∈ Π : pM,n (P) > p 0 }. (3. 2) An important feature of SIRUS algorithm is to stop the growing of the forest with an appropriate number of trees M. Although the right order of magnitude for M is required, no fine tuning is necessary. Indeed, the uncertainty of the importance estimate pM,n (P) of each rule decreases with M , whereas the computational cost linearly increases with M. Thus, to obtain a robust rule extraction, M needs to be high enough to make the uncertainty of pM,n (P) negligible. More precisely, M is set to get the same list of selected rules PM,n,p 0 when SIRUS is run multiple times on the same dataset D n. On the other hand, M should be small enough to avoid useless computations.
Therefore, the growing of the forest is automatically stopped when 95% of the selected rules would be shared by a new run of SIRUS on D n in average, as it is possible to derive a simple stopping criterion based on the properties of the estimates pM,n (P)-all the technical details are provided in Subsection 5. 4. A random forest is usually built with around 500 trees, as the predictive accuracy cannot be significantly increased by adding more trees. SIRUS typically grows 10 times more trees to obtain a robust rule extraction. Besides, we insist that the quantile discretization is critical for the rule selection. The expected value of the rule importance is E[p M,n (P)] = P(P ∈ T (Θ, D n )), but without the discretization, the list of extracted paths from a random tree T (Θ, D n ) takes values in an uncountable space when at least one component of X is a continuous random variable, and therefore the above quantity is null, making the path selection procedure unstable with respect to data perturbation.
We simply average the set of elementary rules {ĝ n,P : P ∈ PM,n,p 0 } that have been selected in the first steps of SIRUS. The aggregated estimate ηM,n,p 0 (x) of η(x) is thus defined by ηM,n,p 0 (x) = 1 | PM,n,p 0 | P∈ PM,n,p 0 ĝn,P (x). (3. 3) Finally, the classification procedure assigns class 1 to an input x if the aggregated estimate ηM,n,p 0 (x) is above a given threshold, and class 0 otherwise. In the introduction, we presented an example of a list of 7 rules for the Titanic dataset. In this case, for a new input x, ηM,n,p 0 (x) is simply the average of the output probability of survival p s over the 7 selected rules. In past works on rule ensemble models, such as RuleFit  (Friedman and Popescu, 2008 ) and Node harvest  (Meinshausen, 2010) , rules are also extracted from a tree ensemble and then combined together through a regularized linear model.
On one hand, numerical discrete variables are left untouched since the number of possible split points is already finite, and the rule definition introduced for continuous variables also applies. On the other hand, we naturally extend the rule definition for categorical variables to "if X (1) is category 1 or 2 then response else default response"-see the Titanic dataset example in the introduction. Originally, categorical variables are efficiently handled in trees by transformation in ordered variables. Such ordering of categories is done with respect to the output mean for each category-see  Breiman et al. (1984) ;  Friedman et al. (2001) , and we follow ranger implementation. Notice that trees are likely to often cut on categorical variables with a high number of categories, as highlighted in  Strobl et al.
, q -1} the empirical r-th q-quantile of {X (j) 1 ,. , X (j) n } is defined by q(j) n,r = inf x ∈ R : 1 n n i=1 1 X (j) i ≤x ≥ r q. (4. 2) Note that, for the ease of reading, (4. 1) is defined for a tree built with the entire dataset D n without resampling. As it is often the case in the theoretical analysis of random forests, we assume throughout this section that the subsampling of a n observations to build each tree is done without replacement to alleviate the mathematical analysis. Recall that the rule selection is based on the probability p n (P) that a Θ-random tree of the forest contains a particular path P ∈ Π, that is, p n (P) = P(P ∈ T (Θ, D n )|D n ), and that the Monte-Carlo estimate pM,n (P) of p n (P) is directly computed using the random forest, and takes the form pM,n (P) = 1 M M =1 1 P∈T (Θ ,Dn).
To do this, we have to show that the CART-splitting criterion is consistent (Lemma 3) and asymptotically normal (Lemma 4) when cuts are limited to empirical quantiles (estimated on the same dataset) and the number of trees grows with n. When cuts are performed on the theoretical quantiles, the law of large numbers and the central limit theorem can be directly applied, so that the proof of Lemmas 3 and 4 boils down to showing that the difference between the empirical CART-splitting criterion evaluated at empirical and theoretical quantiles converges to 0 in probability fast enough. This is done in Lemma 2 thanks to Assumption (A3). Proof of Corollary 1. The first result is a consequence of Theorem 1 since P PMn,n,p 0 = P p 0 ≤ P∈Π P(p Mn,n (P) > p 0 )1 p (P)≤p 0 + P(p Mn,n (P) ≤ p 0 )1 p (P)>p 0.
Experiments We begin this section by providing overall experimental settings. Next, we focus on a case study to illustrate SIRUS with an industrial process example: the semi-conductor manufacturing process SECOM data  (Dua and Graff, 2017). In particular, it shows the excellent performance of SIRUS on real data in a noisy and high-dimensional setting. In Subsection 5. 3, we use 19 UCI datasets  (Dua and Graff, 2017)  to perform extensive comparisons between SIRUS and its main competitors. We show that SIRUS produces much more stable rule lists, while preserving a predictive accuracy and computational complexity comparable to the top competitors. Finally, in Subsection 5. 4, we detail the tuning procedure of the single hyperparameter p 0 , along with a thorough discussion on the design of SIRUS.
Experiment Description Performance metrics. We first introduce relevant metrics to assess the three interpretability properties in the experiments. By definition, the size (i. e. , the simplicity) of the rule ensemble is the number of selected rules, i. e. , | PM,n,p 0 |. estimated by 10-fold cross-validation (repeated 10 times for robustness and standard deviation estimates). With respect to stability, an independent dataset is not available for real data to compute ŜM,n,p 0 as defined in (3. 4) in the Section 3. Nonetheless, we can take advantage of the cross-validation process to compute a stability metric: the proportion of rules shared by two models built during the cross-validation, averaged over all possible pairs  (Guidotti and Ruggieri, 2019). Datasets. We have conducted experiments on the SECOM data, as well as 19 diverse public datasets from the UCI repository  (Dua and Graff, 2017 ; data is described in Table  1 ).
Next, for rule algorithms based on tree ensembles, we evaluate RuleFit and Node harvest. Note that categorical features are transformed in multiple binary variables as it is required by the two software implementations, and RuleFit is limited to rule predictors. For RuleFit, the lasso penalty is tuned by cross-validation as defined in  Friedman and Popescu (2008). As advertised in  Meinshausen (2010)   (Wright and Ziegler, 2017). Besides, notice that for SIRUS experiments, we use the default settings of random forests well known for their excellent behavior, in particular mtry = p 3. We set q = 10 quantiles and tune p 0 as specified in Subsection 5. 4.
Case Study: Manufacturing Process Data SIRUS is run on a real manufacturing process of semi-conductors, the SECOM dataset  (Dua and Graff, 2017). Data is collected from sensors and process measurement points to monitor the production line, resulting in 590 numeric variables. Each of the 1567 data points represents a single production entity associated with a pass or fail output (0/1) for in-house line testing. As it is often the case for a production process, the dataset is unbalanced and contains 104 fails, i. e. , a failure rate p f of 6. 6%. We proceed to a simple pre-processing of the data: missing values (about 5% of the total) are replaced by the median. Figure  2  shows predictivity versus the number of rules when p 0 varies, with the optimal p 0 displayed. Notice that the relation between p 0 and the number of rules is monotone by construction, but also highly nonlinear.
Therefore, we use the number of rules for the xaxis of Figure  2  to improve readability. The 1-AUC value is 0. 30 for SIRUS (for the optimal p 0 = 0. 04), 0. 29 for Breiman's random forests, and 0. 48 for a pruned CART tree. Thus, in that case, CART tree predicts no better than the random classifier, whereas SIRUS has a similar accuracy to random forests. The final model has 6 rules and a stability of 0. 72, i. e. , in average 4 to 5 rules are shared by 2 models built in a 10-fold cross-validation process, simulating data perturbation. By comparison, Node harvest outputs 36 rules with a value of 0. 32 for 1-AUC. Finally, the output of SIRUS may be displayed in the simple and interpretable form of Figure  3 , the output in the R console of the package sirus for the SECOM data. Such a rule model enables to catch immediately how the most relevant variables impact failures.
Improvement over Competitors Overall, we observe that SIRUS provides a high improvement of stability compared to stateof-the-art rule algorithms, while preserving the other properties. For the top competitors, experimental results are gathered in Table  2  for model size, Table  3  for stability, and Table  4  for predictive accuracy. Experiments for additional competitors are provided in Appendix A. 2 in Tables 7, 8 and 9. Standard deviations are made negligible by averaging metrics over 10 repetitions of the cross-validation and are not displayed in the tables to increase readability. Figure  4  provides an example for the dataset "Credit German" of the dependence between predictivity and the number of rules when p 0 varies. In that case, the minimum of 1-AUC is about 0. 25 for SIRUS, 0.
20 for Breiman's forests, and 0. 29 for CART tree. For the chosen p 0 , SIRUS returns a compact set of 22 rules and its stability is 0. 66. Figure  5  provides another example of the good practical performance of SIRUS with the "Heart Statlog" dataset. Here, the predictivity of random forests is reached with 16 rules, with a stability of 0. 83, i. e. , about 13 rules are consistent between two different models built in a 10-fold cross-validation. Thus, the final models are simple, quite robust to data perturbation, and have a predictive accuracy close to random forests. We can draw the following conclusions from the experimental comparisons with competitors, displayed in Tables  2, 3 , and 4. SIRUS produces more stable and predictive rule lists than decision trees, for a comparable simplicity, but at the price of a higher computational complexity since many trees are grown.
To achieve this goal, we proceed as follows. The error 1-AUC is estimated by 10-fold cross-validation for a fine grid of p 0 values, defined such that | PM,n,p 0 | varies from 1 to 25 rules. (We let 25 be an arbitrary upper bound on the maximum number of rules, considering that a bigger set is not readable anymore. ) The randomization introduced by the partition of the dataset in the 10 folds of the cross-validation process has a significant impact on the variability of the size of the final model. Therefore, in order to get a robust estimation of p 0 , the cross-validation is repeated multiple times (typically 10) and results are averaged. The standard deviation of the mean of 1-AUC is computed over these repetitions for each p 0 of the grid search. We consider that all models within 2 standard deviations of the minimum of 1-AUC are not significantly less predictive than the optimal one.
Number of trees. The accuracy, stability, and computational cost of SIRUS increase with the number of trees M. Thus, we simply design a stopping criterion to grow the minimum number of trees which ensures that accuracy and stability are higher than 95% of their maximum asymptotic values with respect to M and conditionally on D n. We empirically observe that the stability requirement is met for a much higher number of trees than the accuracy requirement (about 10 times). Therefore, the stopping criterion is only based on stability. More precisely, we require that 95% of the rules are identical across two runs of SIRUS on a given dataset D n in average. Formally, the mean stability E[ ŜM,n,p 0 |D n ] measures the expected proportion of rules shared by two fits of SIRUS on D n , for fixed n (sample size), p 0 (threshold), and M (number of trees).
Thus, in the experiments, we utilize the following criterion to stop the growing of the forest, with typically α = 0. 05: argmin M 1 | VM,n | p 0 ∈ VM,n ε M,n,p 0 < α. (5. 1) Quantile discretization. In the modified random forest grown in the first step of SIRUS, the split at each tree node is limited to the empirical q-quantiles of each component of X, as described in Section 3. Thus, we check that this modification alone of the forest has little impact on its accuracy. Using the R package ranger, 1-AUC is estimated for each dataset with 10-fold cross-validation for q ∈ {2, 5, 10, 20}. We leave aside datasets with a majority of categorical variables, results are averaged over 10 repetitions of the cross-validation, and displayed in Table  5. Clearly, the decrease of accuracy generated by this discretization is small, and not very sensitive to q, provided that q is not too small.
In the software implementation sirus, the tree depth parameter max. depth is a modifiable input, set to 2 by default to reduce the computational cost while leaving the output list of rules almost untouched as explained above. We conduct experiments where SIRUS is run with a tree depth of 1, 2, and 3, and results are displayed in Table  6. Over the nineteen UCI datasets, rules of three splits appear in SIRUS rule list in only four cases, and a significant accuracy improvement over a tree depth of 2 occurs only once, for the 'Mushrooms' dataset. On the other hand, for all datasets except two, SIRUS outputs rules of two constraints, and predictivity is improved over a tree depth of 1 for half of the datasets. The Titanic example shows how the rule list is drastically simplified by limiting tree depth to 1, lowering the insights provided by SIRUS: Average survival rate p s = 39%.
The principle of SIRUS is to extract rules from a random forest, based on their probability of occurrence in a random tree, and to stop the growing of the forest when the rule selection is converged. Thus, SIRUS inherits the computational complexity of random forests, and has only one tuning parameter. A software implementation, the R/C++ package sirus  (Benard and Wright, 2020) , is available from CRAN. Besides, we believe that the extension of SIRUS to regression is a promising future research direction: the main challenge is the construction of an appropriate rule aggregation framework to accurately estimate continuous outputs without hurting stability. Furthermore, although SIRUS has the ability to handle high-dimensional data, as illustrated with the SECOM dataset (590 inputs), specific variable selection strategies could be used to reduce the number of possible rules and then improve SIRUS performance.
A. 1 Robustness Illustration For the SECOM dataset used in the experimental Section 5 of the article, only three rule algorithms achieve the same predictivity as random forests: RuleFit, Node harvest, and SIRUS (1-AUC of 0. 30, whereas CART and BRL are no better than the random classifier with an error of 1-AUC = 0. 5). SIRUS produces a short and stable list of 6 rules, while RuleFit and Node harvest generate complex, long, and unstable rule lists. Rule algorithms based on tree ensembles are stochastic since they rely on the tree randomness Θ 1 ,. , Θ M. Consequently, RuleFit and Node harvest output different rule lists when run multiple times on the same dataset. Such behavior is a strong limitation in practice, as domain experts become skeptical of the algorithm conclusions. On the other hand, SIRUS is built to have a robust rule extraction mechanism, and the same list of rules is output over multiple repetitions with the same data, as proved in Theorem 2 in the next Section.
To illustrate this, we run each algorithm twice on the SECOM dataset, and display the output models in Figure  6  for SIRUS, Figure  7  for Node harvest, and Figure  8  for RuleFit. We set the regularization parameter of Node harvest and SIRUS as explained in Subsection 5. 3 of the article, to maximize accuracy with the smallest possible model: for Node harvest λ = 4, and for SIRUS p 0 = 0. 04. RuleFit is tuned as defined in  Friedman and Popescu (2008). Figures  7  and 8  show that the rule lists output by RuleFit and Node harvest are quite different across multiple runs with the exact same data, while SIRUS has the same output. We also observe that for the same accuracy, RuleFit and Node harvest models are longer and more complex than SIRUS. In addition, rules are aggregated using weights to generate predictions.
A. 3 Rule Aggregation In Section 3 of the article, ηM,n,p 0 (x) (3. 3) is a simple average of the set of rules, defined as ηM,n,p 0 (x) = 1     To tackle our binary classification problem, a natural approach would be to use a logistic regression and define ln ηM,n,p 0 (x) 1 -ηM,n,p 0 (x) = | PM, P∈ PM,n,p 0 β P ĝn,P (x), (A. 2) where the coefficients β P have to be estimated. To illustrate the performance of the logistic regression (A. 2), we consider again the UCI dataset, "Credit German". We augment the previous results from Figure  4  (in Section 5 of the article) with the logistic regression error in Figure  9. One can observe that the predictive accuracy is slightly improved but it comes at the price of an additional set of coefficients that can be hard to interpret (some can be negative), and an increased computational cost.
B. 1 Experiments The following experiments on the UCI datasets show the good empirical performance of the stopping criterion (5. 1). Recall that the goal of this criterion is to determine the minimum number of trees M ensuring that two independent fits of SIRUS on the same dataset result in two lists of rules with an overlap of 95% in average. This is checked with a first batch of experiments-see next paragraph. Secondly, the stopping criterion (5. 1) does not consider the optimal p 0 , unknown when trees are grown in the first step of SIRUS. Then, another batch of experiments is run to show that the stability approximation 1 -ε M,n,p 0 is quite insensitive to p 0. Finally, a last batch of experiments provides examples of the number of trees grown when SIRUS is fit. Notice that for these experiments, categorical variables are one-hot-encoded.
Experiments 1. For each dataset, the following procedure is applied. SIRUS is run a first time using criterion (5. 1) to stop the number of trees. This initial run provides the optimal number of trees M as well as the set VM,n of possible p 0. Then, SIRUS is fit twice independently using the precomputed number of trees M. For each p 0 ∈ VM,n , the stability metric ŜM,n,p 0 (with D n = D n ) is computed over the two resulting lists of rules. Finally ŜM,n,p 0 is averaged across all p 0 values in VM,n. This procedure is repeated 10 times: results are averaged and presented in Table  10 , with standard deviations in parentheses. Across the considered datasets, resulting values range from 0. 941 to 0. 955, and are thus close to 0. 95 as expected by construction of criterion (5. 1). Experiments 2.
The second type of experiments illustrates that ε M,n,p 0 is quite insensitive to p 0 when M is set with criterion (5. 1). For the "Credit German" dataset, we fit SIRUS and then compute 1 -ε M,n,p 0 for each p 0 ∈ VM,n. Results are displayed in Figure  10. 1 -ε M,n,p 0 ranges from 0. 90 to 1, where the extreme values are reached for p 0 corresponding to very small number of rules, which are not of interest when p 0 is selected to maximize predictive accuracy. Thus, 1 -ε M,n,p 0 is quite concentrated around 0. 95 when p 0 varies. Experiments 3. Finally, we display in Table  11  the optimal number of trees when the growing of SIRUS is stopped using criterion (5. 1). It ranges from 4220 to 20 650 trees. In Breiman's forests, the number of trees above which the accuracy cannot be significantly improved is typically 10 times lower.
08 0. 01 Credit Approval 0. 15 0. 17 0. 15 0. 09 Credit German 0. 37 0. 36 0. 41 0. 25 Diabetes 0. 28 0. 30 0. 28 0. 19 Haberman 0. 46 0. 42 0. 50 0. 35 Heart C2 0. 20 0. 23 0. 19 0. 10 Heart H2 0. 23 0. 23 0. 23 0. 12 Heart Statlog 0. 21 0. 24 0. 20 0. 10 Hepatitis 0. 34 0. 34 0. 39 0. 17 Ionosphere 0. 10 0. 10 0. 13 0. 07 Kr vs Kp 0. 006 0. 008 0. 02 0. 04 Liver Disorders 0. 34 0. 38 0. 50 0. 35 Mushrooms 0. 001 0 6. 10 -5 6. 10 -5 6. 10 -5 6. 10 -4 Sonar 0. 26 0. 26 0. 26 0. 2 Spambase 0. 07 0. 07 0. 12 0. 07 Titanic 0. 20 0. 20 0. 25 0. 17 Vote 0. 04 0. 05 0. 05 0. 02 Wilt 0. 15 0. 17 0. 46 0. 11 Mean stability over a 10-fold cross-validation for UCI datasets (averaged over 10 repetitions). Values within 10% of the maximum are displayed in bold.
In common applications, fit a neural network means use a training dataset to set the connection between the neurons (weights) in order to map the inputs and the outputs in the best way possible. So, the training of a feedforward neural network phase consists of an optimization algorithm based on the back-propagation learning algorithm. It includes forward phase and backward phase  (De Giorgi et al. , 2013). In the Forward phase, the error is obtained as the difference between the target and the actual output with parameters of the network fixed. the Backward phase the weights are adjusted to minimize the error. The Levenberg-Marquardt algorithm is one of the most popular training algorithm for neural network that adopts the gradient descent approach, by which the training can be considered as completed when the performance is minimized to the goal.
Least Squares Support Vector Machines The Least Squares Support Vector Machines represents a noteworthy learning technique with several applications to overcome the overfitting problem successfully. The LS-SVM maps the outputs y i by using a regression function ϕ, applied to the inputs x i as follows: y i = wϕ(x i ) + b i = 1…N (15) where, w is the weight vector, b is the bias and N is the size of the training dataset. The Lagrange function given by: L(w, b, e, α) = J (w, e) - ∑ N i=1 α i {y i [wϕ(x i ) + b] -1 + e i } i = 1…N (16) It is introduced to solve the optimization problem and to minimize The Mercer's theorem introduces a kernel function K and claims that: ϕ T (x i )ϕ ( x j ) = K ( x i , x j ) i, j = 1…N (18) It assumes the Radial Basis Function (RBF) as the kernel function given by: K ( x k , x j ) = exp ( - x k -x j σ 2 ) ( 19 ) with σ the tuning parameter.