dataesr/scientific-paragraphs-categorization huggingface dataset:
Several papers addressed the detection of security issues in IoT systems by means of model checking. The tool IoTSAT  [24]  is a SMT based framework, which analyses the IoT system security. IoTSAT models the device-level interactions as in our approach, but also policy-level behaviours and network-level dependencies. SOTERIA  [5]  is another model checking tool for IoT software. State-models are extracted from source code, then SOTERIA checks whether concrete security properties hold on these models. In comparison to our approach, the works  [14, 23]  go further in the risk assessment by proposing the evaluation of metrics. IoTSAT also goes further in the modelling of the IoT system environment. But all of these approaches require models or formal properties, which have to be manually devised.
MLCA Requirements The capability of MLCA in auditing IoT systems depends on several realistic assumptions made on a system under audit denoted SUA: GP-TM-18: the device software/firmware has the ability to update Over-The-Air (OTA), the update server is secure, the update file is transmitted via a secure connection, it does not contain sensitive data and is encrypted FgetU pdate( f ) ∧ (G(getU pdate( f ) → (encrypted( f ) ∧ ¬sensitive( f )))) ∧ G((begin ∧ Fend) → (¬(getU pdate( f ) ∧ f rom(c)) U(authenticated(c) ∨ end))) GP-TM-19: offer an automatic firmware update mechanism FsearchU pdate ∧ (FsearchU pdate → ¬(FsearchU pdate → (¬searchU pdate U(input ∧ cmdSearchU pdate ∧ ¬searchU pdate)))) GP-TM-22: ensure that weak, null or blank passwords are not allowed G((Request ∧ f rom(c1) ∧ to(c2) ∧ Weakpass(x)) → ((¬(Response ∧ to(c1) ∧ f rom(c2))) U((Response ∧ to(c1) ∧ f rom(c2)) ∧ (errorResponse ∨ (¬validResponse ∧ Response))))) GP- ¬ G( f rom(d) ∧Unavailable) → ¬(¬out put U(out put ∧Unavailable) GP-TM-52(1): ensure a device is not susceptible to XSS G((Request ∧ f rom(c1)∧to(c2)∧XSS(x)) → ¬(¬(Response∧to(c1)∧ f rom(c2)) U(Response ∧ to(c1) ∧ f rom(c2)) ∧ (errorResponse ∨ (¬validResponse ∧ Response)))) GP-TM-52(2): ensure a device is not susceptible to SQL injection G((Request ∧ f rom(c1) ∧ to(c2) ∧ SQLin jection(x)) → ¬(¬(Response ∧ to(c1) ∧ f rom(c2)) U(Response ∧ to(c1) ∧ f rom(c2)) ∧ (errorResponse ∨ (¬validResponse ∧ Response)))) GP-TM-53: avoid security issues when designing error messages G((errorResponse ∨ ¬validResponse) → ¬blackListedWord(x)) Table  1.
RQ1: is MLCA able to detect active security measures? To answer RQ1, we initially experimented the above systems by hands. We gathered a first set of event logs that capture "normal behaviours" and generated first models with CkTail. From these, we identified the testable components, on which we applied a set of penetration testing tools specialised to Web applications. During this testing stage, we collected larger event logs. From them, we generated the final models used for the evaluation. Besides, we manually analysed the source codes of the software used by the devices (sensors, gateways) when available and the event logs to list the ENISA security measures that are active and those that are not (correctly) implemented. This task allowed us to compare the results of MLCA with our observations and hence to measure the sensitivity and specificity of our approach.
The accuracy ensures a high degree of confidence in the generated model. The stability score quantifies the sensitivity of an algorithm to noise, and it allows to evaluate the robustness of the algorithm. The simplicity score could be conceptualized as the ability to easily verify the prediction. A simple model makes it easy to evaluate some qualitative criteria such as justice, ethics and morality. By measuring these three concepts we are therefore able to evaluate the interpretability of several algorithms for a given problem. A similar idea has been proposed in  [28]  in the area of the Logical Analysis of Data (LAD), by introducing the concept of Pareto-optimal patterns or strong patterns. The main part of the LAD is to select the best patterns from the dataset based on a triptych that includes simplicity, selectivity, and evidence.
Datasets We have used publicly available databases from the UCI Machine Learning Repository  [13]  and from  [27]. We have selected six datasets for regression 2  which are summarized in Table  1 , and three datasets for classification which are summarized in Table  2. Covertype 581012 × 54 Classification of forest cover type  [1, 7]  based on cartographic variables  [13]. Speaker 329 × 12 Classification of accent, six possibilities, based on features extracted from the first reading of a word  [15].
Execution For each dataset we perform 10-fold cross-validation. The parameter settings for the algorithm are summarized in Table  3. These parameters were selected according to the author's recommendations to generate models based on rules with equivalent lengths. It means that all rules generated by algorithms have a bounded length. The parameters of the algorithms are not tuned because it is not the purpose of the paper to rank the algorithms. The aim of this section is to illustrate how this score is computed. For each algorithm, a model is fitted on the training set to obtain the simplicity score (4), while we measure the predictivity score (2) on the test set. To obtain the predictivity score we set γ (g; (X, Y )) = (g(X) -Y ) 2 and h n = 1 n n i=1 y i for regression, γ (g; (X, Y )) = 1 g(X) =Y and h n = mode ({y 1 ,.
, y n }) for classification. Then, to obtain the stability score, the training set is randomly divided into two sets of equal length and two models are constructed. The code is a combination of Python and R and it is available on GitHub https://github. com/Advestis/Interpretability. The choice of α's in (  6 ) is an important step in the process of comparing interpretability. For these applications we use an equally weighted average. It means that α 1 = 1/3, α 2 = 1/3, α 3 = 1/3. Another possibility is to set each α to be inversely proportional to the variance of the associated score for each data set. In our application the results were very similar to the equally weighted case (data not shown).
Results for regression The averaged scores are summarized in Table  4. As expected RuleFit is the most accurate algorithm. However, RuleFit is neither stable nor simple. SIRUS is the most stable algorithm and the Covering Algorithm is one of the simplest. For all datasets, SIRUS seems to be the most interesting algorithm among this selection of algorithms and by our score  (6). Figures 1, 2 and 3 are the box-plots of the predictivity scores, q-stability scores and simplicity scores, respectively, of each algorithms on the dataset Ozone. Another interesting result is obtained from the correlation matrix table 5, which was calculated considering all results generated by the 10-fold crossvalidation for all datasets. It shows that the simplicity score is negatively correlated with the predictivity score, which illustrates the well-known predictivity / simplicity trade-off.
Results for classification The averaged scores are summarized in Table  6. All selected algorithms have the same accuracy for all datasets. However, RIPPER and PART are both very stable algorithms, and RIPPER is the simplest of the three algorithms. Therefore, for these datasets and among these three algorithms, RIPPER is the algorithm that is most interpretable according to our measure  (6). Figures  Table  5 : Correlation between the scores for the regressions experiments P n S q n S n P n 1 -0. 1 -0. 27 S q n - 1 -0. 10 S n - - 1 Figure  2 : Box-plot of the q-stability scores for each algorithm for the dataset Ozone  4, 5 and 6 are the box-plots of the predictivity scores, q-stability scores and simplicity scores, respectively, of each algorithms for the dataset Speaker. In contrast to the regression case, the correlation matrix table 7, which was calculated considering all scores generated by the 10-fold cross-validation for all datasets, shows that the scores do not seem to be correlated.
These results should take into account that for the classification part we have tested fewer algorithms on fewer data sets than for the regression part. The accuracy of the models for these datasets was small. If the algorithms are not accurate enough, it may not be useful to look at the other scores. The algorithms appear to be very stable, which may be explained by the fact that they are not complex. Since it is not enough to have a good predictivity score, these algorithms must be tuned to be more complex. - 1 0. 06 S n - - 1 Figure  4 : Box-plot of the prediction scores for each algorithm for the dataset Speaker Figure  5 : Box-plot of the q-stability scores for each algorithm for the dataset Speaker Figure  6 : Box-plot of the simplicity scores for each algorithm for the dataset Speaker.
Conclusion and perspectives In this paper we propose a score that may be used to compare the interpretability of tree-based algorithms and rule-based algorithms. This score is based on the triptych: predictivity (2), stability (3), and simplicity (5), as proposed in  [52, 4]. The proposed methodology seems to provide an easy way to rank the interpretability of a set of algorithms by being composed of three different scores that allow to integrate the main components of interpretability. It may be seen from our applications that the q-stability score and the simplicity score are quite stable regardless of the datasets. This observation is related to the properties of the algorithms; indeed, an algorithm designed for accuracy, stability or simplicity should maintain this property independent of the datasets.
14 1. 0 0. 26 1. 0 Abalone 1. 0 0. 21 0. 86 0. 25 0. 99 Dataset S n RT RuleFit NodeHarvest CA SIRUS Ozone 0. 12 0. 01 0. 04 0. 96 0. 29 Machine 0. 14 0. 02 0. 04 0. 9 0. 25 MPG 0. 15 0. 01 0. 05 0. 98 0. 34 Boston 0. 26 0. 01 0. 07 1. 0 0. 52 Student 0. 37 0. 05 0. 25 0. 91 0. 97 Abalone 0. 58 0. 02 0. 13 0. 66 1. 0 Dataset RT I RuleFit NodeHarvest CA SIRUS Ozone 0. 56 0. 29 0. 54 0. 59 0. 63 Machine 0. 52 0. 41 0. 56 0. 55 0. 57 MPG 0. 63 0. 33 0. 57 0. 61 0. 69 Boston 0. 57 0. 3 0. 52 0. 5 0. 69 Student 0. 47 0. 12 0. 49 0. 43 0. 74 Abalone 0. 66 0. 26 0. 45 0. 43 0. 76.
Networks are powerful tools to represent and investigate biological systems. The development of algorithms inferring regulatory interactions from functional genomics data has been an active area of research. With the advent of single-cell RNA-seq data (scRNA-seq), numerous methods specifically designed to take advantage of single-cell datasets have been proposed. However, published benchmarks on single-cell network inference are mostly based on simulated data. Once applied to real data, these benchmarks take into account only a small set of genes and only compare the inferred networks with an imposed ground-truth. Here, we benchmark six single-cell network inference methods based on their reproducibility, i. e. their ability to infer similar networks when applied to two independent datasets for the same biological condition.
We tested each of these methods on real data from three biological conditions: human retina, T-cells in colorectal cancer, and human hematopoiesis. Once taking into account networks with 100,000 links, GENIE3 results to be the most reproducible algorithm and, together with GRNBoost2, show higher intersection with ground-truth biological interactions. These results are independent from the single-cell sequencing platform, the cell type annotation system and the number of cells constituting the dataset. Finally, GRNBoost2 and CLR show more reproducible performance once a more stringent thresholding is applied to the networks (1000-100 links). In order to ensure the reproducibility and ease extensions of this benchmark study, we implemented all the analyses in scNET, a Jupyter notebook available at https://github.
e. their ability to infer similar networks once applied to two independent datasets for the same biological condition (e. g. two independent scRNA-seq datasets obtained from colorectal tumors). The rationale behind this comparison is that, if the two independent datasets are profiled from the same biological condition (e. g. colorectal cancer) involving the same cell types, we can expect that the regulatory programs underlying them should strongly overlap. As a consequence, a good network inference algorithm should infer highly overlapping networks when applied to single-cell datasets profiled from the same biological condition. We selected six algorithms spanning the main network inference formulations that do not require an ordering of the cells according to pseudo-time, and we tested the reproducibility of the inferred networks in three biological systems: human retina, T-cells in colorectal cancer and human hematopoiesis.
Differently from previous benchmarks, we only applied a soft filtering on genes, thus testing the algorithms based on their performances to infer networks involving from 6,000 to 12,000 nodes/genes. From our benchmark, once an high number of links is taken into account (100,000), GENIE3 results to be the most reproducible algorithm and, together with GRNBoost2, show the highest intersection with ground-truth biological interactions. GRNBoost2 and CLR have instead better performances for low link numbers (1000-100). In order to ensure the reproducibility and ease extensions of this benchmark study, we implemented all the analyses in a Jupyter notebook, called scNET and available at https://github. com/ComputationalSystemsBiology/scNET. Materials and Methods.
g. retina cells in normal state). Testing reproducibility strictly requires the availability of two independent scRNA-seq datasets reflecting the same biological condition and presenting as few as possible technical variations. Indeed, the presence of technical variations due to the sequencing or experimental procedures could drastically impact the outcome of our comparison. In this respect, finding independent scRNA-seq datasets reflecting dynamic transcriptional changes, generated with the same experimental procedure, is really challenging. We thus decided to focus our benchmark study on network inference methods that do not use the pseudo-time information. In addition, only algorithms provided in R or Python code are here taken into account. Six single-cell network inference methods are thus considered in this evaluation: GENIE3  (Huynh-Thu et al.
, 2010) , GRNBoost2  (Moerman et al. , 2019) , PPCOR  (Kim, 2015) , PIDC  (Chan et al. , 2017) , CLR  (Faith et al. , 2007)  and GeneNet  (Opgen-Rhein and Strimmer, 2007). All the methods selected for this benchmark were originally designed for bulk data and they span the main mathematical formulations of network inference, as described in  (The DREAM5 Consortium et al. , 2012). Of note, GENIE3, GRNBoost2 and PIDC are also the best performing in the single-cell benchmark of Pratapa et al. GEne Network Inference with Ensemble of Trees (GENIE3)  (Huynh-Thu et al. , 2010 ) is a tree-based network inference method. For each gene G1 in the expression dataset, GENIE3 solves a regression problem, determining the subset of genes whose expression is the most predictive of the expression of G1. This method was the best performing algorithm in the DREAM4 In Silico Multifactorial challenge  (Greenfield et al.
2. 2 Data acquisition and preprocessing Fourteen public scRNA-seq datasets have been used for this benchmark (Table  1 ): Menon and Lukowski obtained by profiling human retina cells  (Lukowski et al. , 2019; Menon et al. , 2019) ; Zhang and Li profiling T-cells in colorectal cancer (CRC)  (Li et al. , 2017; Zhang et al. , 2019) ; Hay and Setty profiling human hematopoiesis cells  (Hay et al. , 2018; Setty et al. , 2019). See Table  1  for a complete description of these datasets. The hematopoiesis datasets were split according to their cell type of origin. Only those cell types reported in both studies by  Hay et al. And by Setty et al. were considered. We thus obtained a total of ten scRNA-seq datasets in hematopoiesis spanning five cell types: HSC, CLP, Monocyte, Erythroblast and Dendritic Cell.
After downloading the data, we filtered the genes based on their total count number (< 3 *0. 01*number of cells), as well as on the number of cells in which they are detected (>0. 01*number of cells), as described in  (Aibar et al. , 2017). The gene filtering is performed on each dataset independently. Then, for each biological condition (CRC T-cells, retina, hematopoiesis), only the genes retained for both datasets were selected for network inference. The number of genes retained after filtering are reported in the last column of Table  1. Finally, the data were log2-normalised before applying the different network inference algorithms.
2. 3 Indexes employed to measure the reproducibility of the network inference algorithms Percentage of intersection (perINT) and Weighted Jaccard Similarity (WJS) have been employed here to test the reproducibility of the network inference algorithms. The percentage of intersection is used to detect the presence of links shared between two compared networks, while WJS takes into account the similarity of the weights associated with the links shared between the compared networks. Given two networks N1 and N2 inferred respectively from scRNAseq datasets D1 and D2, and indicating as |"| the number of links in the network N, the percentage of intersection (perINT) is computed as: #$%&"'(" ! , " " ) = ) , while the Weighted Jaccard Similarity (WJS)  (Tantardini et al. , 2019) , is defined as |$ ! ∩$ " | &'((|$ ! |,|$ " | ,-.
Results Based on previous works  (Chen and Mar, 2018; Pratapa et al. , 2020) , we selected the six single-cell network inference algorithms that do not require an ordering of the cells according to pseudo-time (GENIE3, GRNBoost2, PPCOR, PIDC, CLR and GeneNet see Materials and Methods) and we evaluated them based on their reproducibility, i. e. their ability to infer similar networks once applied to two independent datasets from the same biological condition (e. g. two independent scRNA-seq datasets of colorectal cancer). The reproducibility is measured based on the Percentage of intersection (perINT) and Weighted Jaccard Similarity (WJS) indexes (see Materials and Methods). In addition, we computed the intersection with two instances of ground-truth, based on the RcisTarget and on Regulatory Circuits scores (see Materials and Methods).
The evaluation is repeated across three biological conditions: human retina, T-cells in colorectal cancer and human hematopoiesis, for a total of fourteen independent scRNAseq datasets. See Figure  1  for an overview of the benchmark workflow. While in previous benchmarks  (Chen and Mar, 2018; Pratapa et al. , 2020) , a low number of highly variable genes had been taken into account (100-1,000 genes), we here tested the ability of the This is a provisional file, not the final typeset article.
Running Title algorithms to infer networks involving all expressed genes (see Materials and Methods for details on the procedure used to filter genes). Indeed, filtering only the top 100-1,000 varying genes is a strong limitation. Restricting the nodes of the inferred network to a low number of genes is reasonable when a manually curated list of relevant genes is available (for example marker genes identified by wet-lab experiments). However, when such a list is not available, working only with the top 100-1,000 varying genes may overlook genes and interactions playing a key role in the regulatory programs of the biological system. We thus tested the various network inference algorithms once applied to scRNAseq datasets containing 6,000-11,000 genes. In our test cases, PIDC failed to reconstruct networks for two main reasons: (i) the algorithms was slow, especially in the discretization step required to infer a network, and (ii) the use of multivariate information measures impose to have a number of genes much lower than the number of cells, thus requiring to drastically filter out the starting set of genes.
Reproducibility in human retina We applied GENIE3, GRNBoost2, PPCOR, CLR and GeneNet to two independent scRNA-seq datasets of human retina, reported in  (Menon et al. , 2019)  and in (Lukowsk et al. , 2019) (see Materials and Methods). After filtering, the two datasets span 6,212 common genes across a comparable number of cells: 20,091 in Menon versus 20,009 in Lukowski. We thus inferred a total of ten networks. For details on the number of links before and after thresholding refer to Supplementary Table  1. We then evaluated the reproducibility of each algorithm by computing the Percentage of intersection (perINT) and the Weighted Jaccard Similarity (WJS) between the networks inferred independently from the two datasets. While perINT is intended to test the amount of common links between the two networks, the WJS takes also into account the similarity of the weights associated with the common links.
Reproducibility in colorectal cancer (CRC) T-cells We further tested the performances of GENIE3, GRNBoost2, PPCOR, CLR and GeneNet in colorectal cancer (CRC) T-cells. The two datasets used in this case are taken from  (Zhang et al. , 2019)  and  (Li et al. , 2017)  (see Materials and Methods), restricting the last dataset to only T-cells (see Materials and Methods). After filtering, we obtained datasets composed of 11,242 common genes and a widely varying number of cells: 10,805 for Zhang, and 35 for Li. We applied GENIE3, GRNBoost2, PPCOR, CLR and GeneNet independently to the two datasets, for details on the number of links before and after thresholding refer to Supplementary Table  1. Of note, PPCOR has been excluded from this comparison, as it produced partial correlation values outside the range [-1;1] for the Li et al.
dataset. After computation of the perINT and WJS (Figure  2B ), GENIE3 (3% perINT and 0. 008 WJS) and GRNBoost2 (3. 4% perINT and 0. 007 WJS) emerged as the best performing methods. The reproducibility indexes are quite low in this test case, probably due to the low number of cells present in the Li dataset (35 cells). Also the RcisTarget and Regulatory Circuits scores reflecting the intersection with a ground-truth are quite low for all algorithms, with GRNBoost2 showing better performances (4% RcisTarget score and 14. 6% Regulatory Circuits score).
3. 3 Reproducibility in human Hematopoiesis Human hematopoiesis has been used as the third biological context for the comparison of GENIE3, GRNBoost2, PPCOR, CLR and GeneNet. The hematopoiesis datasets were split according to the different cell types profiled: HSC, CLP, Monocyte, Erythroblast and Dendritic Cell, obtaining a total of 10 scRNA-seq datasets. Networks were thus inferred on each cell type independently with GENIE3, GRNBoost2, PPCOR, CLR and GeneNet, resulting in a total of 50 networks. Also in this case, details on the number of links before and after thresholding are available in Supplementary Table  1. As for CRC T-cells, PPCOR produced networks composed of links with partial correlation higher than 1 and/or lower than -1 for some CLPs, and Monocytes. For this reason, we did not consider PPCOR in the reproducibility evaluation for these cell types.
3. 4 Stability with respect to link thresholding in the inferred networks In the previous experiments the 100,000 top-ranked links have been taken into account for all methods. Here we test to which extent our conclusions, regarding the reproducibility of the benchmarked methods, are stable with respect to the number (K) of links retained in each network. We thus apply a more stringent filtering, considered a number (K) of top-ranked links of 10,000, 1000 and 100. GeneNet has been excluded from this analysis as the number of its inferred links is lower than 1000 in most of the cases. After thresholding, the intersection between the networks inferred from independent datasets from the same biological condition were evaluated, using the percINT and WJS as above. As shown in Figure  3 , the performances of the various algorithms are quite heterogeneous once different thresholds (K) are considered.