dataesr/scientific-paragraphs-categorization huggingface dataset:
Research participants were men and women over 65 years of age. All data have been anonymised. To focus on the normal-borderline PO continuum, four major exclusion criteria were defined: 1) major cognitive impairment or dementia, 2) psychotic symptomatology/psychotic PO and major organic or somatic disorder, which can be a source of significant bias in clinical dynamics and testing, 3) diagnosis of BPD and/or psychiatric history, and 4) disabling visual or praxic impairment to testing and major somatic or psychiatric crisis, or any similar disability that may prevent participants from completing the selfassessment questionnaire. The sample could present minor cognitive disorders that did not significantly affect the participants' thinking and cognitive abilities. Participants needed to have the ability to attend to the survey task.
Participants A total of 444 responses to all questionnaires were collected from participants in three locations. The collected data came from men and women over 65 years old [mean age 77. 8 ± 9. 20 standard deviation (SD), range 65-97 years]. Of the total sample, 211 of 444 participants were male (mean age 77. 7 ± 9. 00 SD, range 65-97 years) and 233 were female (mean age 78. 0 ± 9. 00 SD, range 65-96 years). Concerning co-morbidities (see Table  1 ), 36. 9% of participants had depressive symptoms. Participants with borderline personality organisation had more severe depressive symptoms (18) than neurotic personality organisation (9) or normal personality (6). There were 16. 6% of participants with anxiety symptoms. Also, participants with borderline personality organisation had more moderateto-severe anxiety symptoms (32) than neurotic personality organisation (17) or normal personality (8).
DATA USAGE AND APPLICATION This dataset is useful because its analysis establishes cut-off scores, which facilitate the diagnosis of borderline personality organisation. It helps to distinguish normal from neurotic personality organisations: • These data show an effective alternative to the diagnosis of BPD in ageing only based on DSM-5, the latter having many limitations  (Beatson et al. , 2016). • These data can be easily used as part of a gerontological assessment. They can be used to identify personality organisation (neurotic or borderline). Also, borderline personality organisation can be assessed on a normal-pathological continuum. • These data can be analysed as part of a lifespan approach to borderline personality organisation, integrated with data relating to the IPO in participants over 65 years of age and included in meta-analyses.
Graph-based architectures are becoming increasingly popular as a tool for structure generation. Here, we introduce a novel open-source architecture HyFactor which is inspired by previously reported DEFactor architecture and based on the hydrogen labeled graphs. Since the original DEFactor code was not available, its new implementation (ReFactor) was prepared in this work for the benchmarking purpose. HyFactor demonstrates its high performance on the ZINC 250K MOSES and ChEMBL data set and in molecular generation tasks, it is considerably more effective than ReFactor. The code of HyFactor and all models obtained in this study are publicly available from our GitHub repository:.
This approach allows one to model molecular graphs much more efficiently. Although the results of the benchmarking study reported in the original publication look promising, many important details, such as the number of convolutional layers in the encoder or the batch size, needed for re-implementation of this approach are missing. Therefore, we made an attempt to reimplement and further improve the DEFactor architecture. The improved version of DEFactor is referred here as ReFactor (see the details in the Method chapter). It was tested in ZINC250K and ChEMBL v. 27 data sets as well as MOSES benchmarking, and its efficiency was shown to be comparable to that of other GAE architectures. Moreover, we propose a methodological improvement that reduces the amount of GPU memory required to store the model as well as training time without loss of accuracy.
Based on the above mentioned inventiones, we introduce a new hydrogen-count labelled graphbased GAE architecture named HyFactor. Within this architecture, a DNN is combined with a set of heuristic rules on allowed atoms' valences needed to do a conversion of a regular molecular graph to a hydrogen-count labeled graph (a graph where a certain number of hydrogens is assigned to each heavy atom) and back. It was compared to DEFactor and ReFactor architectures using ZINC250K, ChEMBL v. 27 and MOSES benchmarking data sets in the tasks of reconstruction and generation. It has been shown that ReFactor and HyFactor possess similar reconstruction rates, whereas HyFactor trains faster and requires less GPU memory. Therefore, in this work we present the two new graph-based AutoEncoder architectures that can be used for molecule encoding and generation tasks.
Data and curation In this work, three data sources were used: a subset of ZINC database called ZINC250K  [4] , MOSES (v. 1. 0)  [18]  data set, and ChEMBL database (v. 27)  [19]. All sets were standardized using the following procedures: 1) dearomatization, 2) isotopes removal, 3) stereo marks removal, 4) explicit hydrogens removal, 5) small fragments removal, 6) solvents removal, 7) salts strip, 8) neutralization of charges, 9) functional groups transformation, 10) selection of canonical tautomer form of molecule, 11) aromatization, 12) duplicates removal, 13) dearomatization. All stages of the standardization were done by ChemAxon JChem's utilities  [20]. Once structures were standardized, the order of atoms was defined from the canonical SMILES string produced by ChemAxon JChem. About 1.
7K structures were removed from the ZINC250K data set as the result of the cleaning procedure. The cleaned set was split into training, validation (tuning), and test sets according to the previous article  [21]. The test set was taken as 5K predefined molecules, and the remaining structures were randomly split into training and validation sets in ratio 9:1. Note that several duplicates were found in ZINC250K (see Table  1 ), due to the presence of stereoisomers in the data set. This may cause some overestimation of model performance for architectures reported earlier. The ChEMBL database was standardized by the same workflow as the ZINC250K data set. The initial database consisted of 1. 9M molecules and it was reduced to 1. 6M of standardized structures. The prepared data set was additionally analyzed in terms of the frequency of atom types (Figure  S1 ).
60 unique atomic types (including information on atomic charge) were found in ChEMBL, and only 15 (C, O, N, S, F, Cl, N + , O -, Br, P, I, N -, B, Si, and Se) have been chosen according to the threshold of 1000. Next, compounds with less than 5 and more than 50 heavy atoms have been removed due to their under-representation. The filtrated ChEMBL data set was then split into a training set (80% of data or 1. 3M molecules) and a validation set (20% of data or 327K molecules). It was not divided into a test set because the amount of data in the validation set was sufficient to reliably assess the performance of the model. The MOSES data set was analyzed with the proposed standardization procedure; however, no mistakes were found. Therefore, it was used as it is. The original "training" set was split into training and validation sets in the 4:1 ratio.
All chemical structures from each data set have been kekulized in order to avoid a need to introduce an additional aromatic bond type for ReFactor architecture. The latter would increase the size of the graph-based architecture, slow down the calculations and decrease the number of valid structures in sampling. After standardization of the data sets, they were analyzed in terms of heavy atoms count distribution (Figure  1 ). Both the ChEMBL and ZINC250K data sets have a similar distribution of heavy atom count. However, the MOSES data set differs from both, and most of the structures lie in the range from 16 to 26 heavy atoms.
Reconstruction rate of different graph Autoencoders The performance of Autoencoders' model is measured by the reconstruction rate representing a percentage of correctly reconstructed compounds out of the considered data set. Reconstruction rate values obtainedfor several SMILES-based and graph-based Autoencoders on the ZINC 250K reference set are reported in Table  2. The reconstruction rate for ReFactor, trained and tested on the non-standardized data set, is slightly higher than for the standardized data set. This fact supports our assumption that the performance of Autoencoder may be overestimated since training and test sets partially overlap in the non-standardised data set. Comparing the results of ReFactor with the original DEFactor, we see that on both standardized and non-standardized data sets our implementation outperforms the original architecture.
Generation ability of graph-based Autoencoders The Autoencoders are a class of generative neural networks, which are widely used to generate novel chemical structures. Thus, here we discuss the ability of proposed graph-based Autoencoders to generate molecules. For this purpose, MOSES benchmarking was used  [18]. Original DEFactor architecture could not be benchmarked since the source code is not available and corresponding information is not given in original publication. The metrics included in the MOSES package are adopted for the distribution learning objective (see the supporting information for details). Even though the proposed generation method does not fit into the category of such methods, most of the metrics are applicable to sampled structures. Also worth noting is that distribution learning models generate various structures, while the proposed approach generates molecules around seeds.
Therefore, it was not possible to select a subset of ten thousand structures for calculation of uniqueness, so it was calculated on the entire set. For the generation tests ReFactor and HyFactor architectures were trained on 80% of the original MOSES training set. Both architectures achieved more than 99% of the reconstruction rate on the remaining 20% of the data used as validation set. Then, 10K compounds were randomly selected as seeds for sampling new structures. For each seed compound, 10 virtual structures were generated, so the total number of structures obtained should be 100K. The generation was based on log-normal distribution with a mean equal to zero and a variable standard deviation from 0. 2 to 1. 0 with a step of 0. 2. A preliminary review of the results revealed that MOSES does not correctly handle a molecular graph consisting of several components (mixture of compounds) and some valence errors during metrics calculation.
4 with the results for known Autoencoders. The results of MOSES benchmarking are given in Table  4. In the table, the most important metrics that can help us to analyze the generation ability of the proposed architectures are presented. The results for the all other metrics available in the MOSES package are given in Table  S2. As graphs are discrete objects, it is expected that the latent space of a graph-based Autoencoder should also be discrete. In other words, invalid molecular graphs (for example, disconnected structures or structures with valence mistakes) are present in the space between the valid molecules. With the proposed sampling method, it is not possible to control the validity of the generated molecules, thus validity is lower than for other architectures (Table  4 ). The uniqueness and novelty is also lower than for other architectures, as we generate molecules in the vinicity of seed molecules.
Application on ChEMBL data set The Achilles' heel of graph-based Autoencoders is the reconstruction of molecules with a large number of atoms. Indeed, as the size of a molecule grows, the probability of error in the type of atom or bond increases. Therefore, in the next phase of experiments, the reconstruction ability of Autoencoders was studied on the ChEMBL data set containing molecules bearing up to 50 heavy atoms. The results of training are given in Table  5  and the specifications on training parameters can be found in Table  S1. 6a ). The latter can be explained by small number of heavy molecules present in the training set (see Figure  1 ).
Architecture In order to confirm these suggestions, 460K virtual structures containing >35 atoms have been generated using the SynthI tool  [36]  and then added to the ChEMBL set. The enriched ChEMBL set (from now on referred to as E_ChEMBL) was then divided into training and validation sets in the ratio 4:1 (1. 6M and 400K of molecules) respectively. Distribution of molecular size in the obtained data set is given in Figure  S3  of Supporting Materials. Both architectures trained on the E_ChEMBL training set achieved reconstruction rates above 95% measured on the E_ChEMBL validation set. As a result, the enrichment of the ChEMBL data set to significantly decrease the reconstruction error for heavy molecules (Figure  6b ). By adding new molecules, one can expect the model to reconstruct molecules from the original ChEMBL set more accurately.
Architecture of ZINC250K dataset allowed for a more accurate assessment of graph-based model performance. Moreover, HyFactor and ReFactor were successfully tested in learning of molecules with up to 50 heavy atoms from the ChEMBL data set. While the HyFactor architecture achieved the same reconstruction rate as ReFactor, it was more effective in terms of number of network parameters and training time. The analysis of the results showed that the number of molecules larger than 35 heavy atoms in the ChEMBL dataset was insufficient for the models to learn them. To solve this problem, virtual molecules were added which improved the learning of large molecules. Trained on MOSES dataset, both Autoencoders were used to systematically explore the chemical space around given molecules. From the calculation of MOSES metrics on the generated structures it was concluded that both architectures were able to generate diverse structures with various scaffolds.
this paper two graph-based Autoencoder architectures for molecule encoding and generation are presented. One of the architectures, named ReFactor, is an updated version of the previously published DEFactor architecture. Another Autoencoder, HyFactor, inherits approaches of ReFactor and is based on a new representation of the molecule, the hydrogen-count labelled graph. 40% Fraction of wrongly reconstructed molecules 10% 20% 30% ReFactor HYFactor 0% 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 Number of heavy atoms Both architectures showed the highest performance in reconstruction ability on the ZINC250K dataset comparing to existing SMILES-and graph-based Autoencoders. The standardization.
In practice, data is collected along the manufacturing line, describing both the production environment and its conformity. The retrieved information enables to infer a link between the manufacturing conditions and the resulting quality at the end of the line, and then to increase the process efficiency. Since the quality of the produced entities is often characterized by a pass or fail output, the problem is in fact a classification task, and state-of-the-art learning algorithms can successfully catch patterns of these complex and nonlinear physical phenomena. However, any decision impacting the production process has long-term and heavy consequences, and therefore cannot simply rely on a blind stochastic modelling. As a matter of fact, a deep physical understanding of the forces in action is required, and this makes black-box algorithms inappropriate.
e. , a node) is contained in a randomized tree. The nodes with the highest probability are robust to data perturbation and represent strong patterns. They are therefore selected to form a stable rule ensemble model. Here, we provide a first illustration of SIRUS with a simple and real case: the Titanic dataset  (Piech, 2016). The survival status of 887 passengers are recorded, as well as various personal characteristics: age, sex, class, number of siblings and parents aboard, and the paid fare. SIRUS outputs the following simple set of 7 rules, which enables to grasp at a glance the main patterns to explain passenger survival: Average survival rate p s = 39%.
if sex is male then p s = 19% else p s = 74% To generate the prediction for a new query point x, SIRUS checks for each rule whether the conditions are satisfied to assign one of the two possible p s output values. Let us say for example that x (sex) is female, then x satisfies the condition of the first rule, which returns p s = 74%. Next, the 7 rule outputs are averaged to provide the predicted probability of survival for x. The model is stable: when a 10-fold cross-validation is run to simulate data perturbation, 5 to 6 rules are consistent across two folds in average. The model error (1-AUC) is 0. 17, close to the 0. 13 of random forests, whereas simplicity is drastically increased: 7 rules versus about 10 4 operations for a forest prediction. First, we review the main rule algorithms and present their mechanism principles in Section 2.
Next, Section 3 is devoted to the detailed description of SIRUS. One of the main contributions of this work is the development of a software implementation, via the R/C++ package sirus  (Benard and Wright, 2020)  available from CRAN, based on ranger, a high-performance random forest implementation  (Wright and Ziegler, 2017). In Section 4, we show that the good empirical behavior of SIRUS is theoretically understood by proving its asymptotic stability. Then, in Section 5, we illustrate the efficiency of our algorithm through numerical experiments on real datasets. Finally, Section 6 summarizes the main contributions of the article and provides directions for future research.
Related Work As stated in the introduction, SIRUS has two types of competitors: decision trees and rule algorithms. More precisely, the latter can further be split into three different kinds: classical rule algorithms based on greedy heuristics, those built on top of frequent pattern mining algorithms, and those extracted from tree ensembles. Decision trees may be the most popular competitors of SIRUS because of their simple structure. The main algorithms are CART  (Breiman et al. , 1984)  and C5. 0  (Quinlan, 1992). However, trees are unstable as we have already highlighted. A widespread method to stabilize decision trees is bagging  (Breiman, 1996) , in which multiple trees are grown on perturbed data and aggregated together. Random forests is an algorithm developped by  Breiman (2001a)  that improves over bagging by randomizing the tree construction.
At the end of the 1990's a new type of rule algorithms based on frequent pattern mining is introduced with CBA (Classification Based on Association Rules,  Liu et al. , 1998) , then extended with CPAR (Classification based on Predictive Association Rules,  Yin and Han, 2003). Frequent pattern mining is originally used to identify frequent occurrences in database mining. Since the output Y ∈ {0, 1} is discrete and the input data can be discretized, we can generate candidate rules for classification by identifying frequent patterns associated with each output label. This exhaustive search for association rules is computationally costly (exponential with the input dimension), and efficient heuristics are used, essentially Apriori  (Agrawal et al. , 1993)  and Eclat  (Zaki et al. , 1997). The rule aggregation mechanism is specific to each algorithm.
Notice that SIRUS is also based on this principle. More specifically, RuleFit extracts all the rules of a boosted tree ensemble  (Friedman and Popescu, 2003) , while Node harvest is based on random forests. Then, the extracted rules are linearly combined in a sparse linear model, respectively a logistic regression with a Lasso penalty  (Tibshirani, 1996)  for RuleFit, and a constraint quadratic linear program for Node harvest. These two methods have a computational complexity comparable to random forests and SIRUS, since the main step of all these algorithms is to grow a tree ensemble with a large number of trees. However, both algorithms are unstable, and both output quite complex and long lists of rules. Even running RuleFit or Node harvest multiple times on the same dataset produces quite different rule lists because of the randomness in the tree ensemblessee Appendix A.
1. On the other hand, SIRUS is built to have its structure converged for the given dataset, as explained later in Section 3. To the best of our knowledge, the signed iterative random forest method (s-iRF,  Kumbier et al. , 2018)  is the only procedure that tackles both rule learning and stability. Using random forests, s-IRF manages to extract stable signed interactions, i. e. , feature interactions enriched with a thresholding behavior for each variable, lower or higher, but without specific thresholding values. Therefore, s-IRF can be difficult to operationalize since it does not provide any specific input thresholds, and thus no precise information about the influence of input variables. On the other hand, an explicit rule model identifies specific regions of interest in the input space.
Finally, the most significant rules are retained and are averaged together to form an ensemble model. We describe the four steps of SIRUS algorithm in the following paragraphs: the rule generation, rule selection, rule post-treatment, and the rule aggregation. This section ends with a discussion of SIRUS stability. Rule generation. SIRUS uses at its core the random forest method  (Breiman, 2001a) , slightly modified for our purpose. As in the original procedure, each single tree in the forest is grown with a greedy heuristic that recursively partitions the input space using a random variable Θ. The essential difference between our approach and Breiman's one is that, prior to all tree constructions, the empirical q-quantiles of the marginal distributions over the whole dataset are computed: in each node of each tree, the best split can be selected among these empirical quantiles only.
Algorithm 1 Tree construction 1: Parameters: Number of quantiles q, number of subsampled observations a n , number of eligible directions for splitting mtry. 2: Compute the empirical q-quantiles for each marginal distribution over the whole dataset. 3: Subsample with replacement a n observations, indexed by Θ (S). Only these observations are used to build the tree. 4: Initialize the cell H as the root of the tree. 5: Draw uniformly at random a subset Θ (V ) ⊂ {1,. , p} of cardinality mtry. 6: For all j ∈ Θ (V ) , compute the CART-splitting criterion at all empirical q-quantiles of X (j)  that split the cell H into two non-empty cells. 7: Choose the split that maximizes the CART-splitting criterion. 8: Recursively repeat lines 5 -7 for the two resulting children cells H L and H R. x (1)   x (2)   q(1) n,7 q(1) n,5 q(2) n,4 P 5 = {(2, 4, R), (1, 7, L)} P 6 = {(2, 4, R), (1, 7, R)} P 3 = {(2, 4, L), (1, 5, L)} P 4 = {(2, 4, L), (1, 5, R)} X (2) i < q(2) n,4 X (2) i ≥ q(2) n,4 P 1 P 2 X (1) i < q(1) n,7 X (1) i ≥ q(1) n,7 P 5 P 6 X (1) i < q(1) n,5 X (1) i ≥ q(1) n,5 The main step of SIRUS is to extract rules from the modified random forest.
, q -1} of the corresponding quantile, and a side s k = L if we go the the left and s k = R if we go to the right. The set of all possible such paths is denoted by Π. It is important to note that Π is in fact a deterministic (that is, non random) quantity, which only depends upon the dimension p and the order q of the quantiles. Of course, given a path P ∈ Π one can recover the hyperrectangle (i. e. , the tree node) Ĥn (P) associated with P and the entire dataset D n via the correspondence Ĥn (P) = x ∈ R p : x (j k ) < q(j k ) n,r k if s k = L x (j k ) ≥ q(j k ) n,r k if s k = R , k = 1,. , d. (3. 1) Finally, an elementary rule ĝn,P can be defined from Ĥn (P) as a piecewise constant estimate: ĝn,P (x) returns the empirical probability that the output Y is of class 1 conditional on whether the query point x belongs to Ĥn (P) or not.