dataesr/scientific-paragraphs-categorization huggingface dataset:
Results on Benchmarks Tables  5  and 6  compare the results of our approach (C-NN) to other approaches presented above as baselines on the four datasets WN18, WN18RR, FB15k, and FB15k-237. The baselines are organized in three groups: naive baseline Freq, latent-based approaches, and rule-based approaches. In each group, approaches are sorted by publication year. The source indicates which paper the results are taken from. The results for AnyBURL are those where 1000s (largest available time) are allocated to the computation of rules. In the results of C-NN, the timeouts (computation of concepts + inference) are 1+1s for WN datasets, and 1. 5+0. 5s for FB datasets. The output logs of C-NN predictions and explanations are available from the companion page. ComplEx-N3 clearly outperforms other approaches on all datasets except WN18 where C-NN outperforms other approaches on the three measures.
C-NN comes second on FB15k, and remains close to the best approaches on WN18RR. On FB15k-237, the MRR delta is -7. 4 with ComplEx-N3, but only -0. 4 with AnyBURL, the best rule-based approach. It is noteworthy that C-NN is competitive with AnyBURL because, whereas AnyBURL rules are computed in a supervised manner (knowing the target relation r k ), C-NN concepts are computed in an unsupervised manner (i. e, only knowing the source entity e i ). This implies that the concepts of neighbours of an entity can be computed once, and used for many different inference tasks, e. g. predicting links for several target relations. Looking at the Freq baseline, it becomes visible how FB15k-237 is difficult because performance improvements over Freq are relatively small while they are huge on other datasets.
In-depth Analysis of Results In this section, we refine the performance evaluation of our approach by analysing results w. r. t two criteria: (a) the kind of rule (by-copy vs by-analogy) that inferred the rank-1 predictions, (b) the characteristics of the relation to be predicted. Prediction by-copy vs by-analogy. We here analyze predictions according to the kind of the best rule, i. e. the rule with highest confidence, which contributed to decide the first entity in the generated ranking. For recall, there are two kinds of rules: by-copy rules and by-analogy rules (see Section 6. 1). Table  7  shows the performance measures, and maximum confidence, on two subsets of the test set for each dataset, depending on the kind of the best rule. Line "percent" gives the relative size of those subsets.
The two percentages may not sum up to 100% because for some test cases, no prediction could be made: e. g. , all predicted entities are already known to hold, and are therefore filtered out. The balance between by-copy and by-analogy can be explained by the nature of each dataset. In WN18 and FB15k it is well known that most test triples can be inferred from an inverse triple, which is well captured with by-analogy rules: e. g. , hypernym(y, x) → hyponym(x, y). In FB15k-237, all inverse triples have been removed, and for many relations, some candidate entities are much more frequent than others: e. g. , US nationality for people. Those frequent entities are easily inferred with by-copy rules. In WN18RR, by-analogy rules remain dominant for two reasons. First, inverse relations have been removed (e.
e. 1-1, n-1, 1-n, and n-n relations. Figure  5  shows a bubble plot of the 224 relations and their inverses, which are found in the test set of FB15k-237, the most challenging dataset. Each buble is positioned according to its degree (ranging from 1 to 4000), and to the MRR of its predictions (ranging from 0 to 1). The bubble size reflects the frequency of the relation, and hence its weight in the global performance measures. Several observations can be made. First, the relations cover a wide range of frequencies and degrees, and there are no obvious correlation between those two dimensions. Second, there is a clear tendency that the higher the degree, the lower the MRR, i. e. multi-valued relations are harder to predict than functional relations. However, many relations stand under and above this tendency.
Some relations have a low MRR despite a low degree (e. g. , place of birth); and some relation succeeds to have a high MRR despite degrees up to 50 (e. g. , inverse of food nutrient with degree=15). Examples of frequent relations with high MRR are: people nationality (degree=1. 1, MRR=79. 3), people gender (degree=1, MRR=90. 4), topic webpage category (degree=1, MRR=100), marriage type of union (degree=1. 1, MRR=100), award winners ceremony (de-gree=12, MRR=71. 4), and its inverse (degree=22, MRR=70. 1). Scalability w. r. t. the size of the knowledge graph. The scalability of the C-NN approach is difficult to evaluate because the runtime is determined by the user-given timeouts. It can only be evaluated by comparing the prediction performance for KGs of different sizes. However, the prediction performance cannot be compared easily between different datasets because inference difficulty varies from one dataset to another.
Even starting from one large KG and using subgraphs of different sizes is tricky if one wants to keep the same inference difficulty. Synthetic KGs may be useful for benchmarking the scalability of query or reasoning engines  [31, 32]  but they may not have enough regularities for the task of link prediction. We propose the following methodology to give some hint on the scalability of our approach. We consider an additional link prediction dataset, YAGO03-10  [9] , that is similar in contents to the Freebase datasets but with about 10 times more entities (123K). According to our discussion at the end of Section 5, the timeout should increase linearly with the number of entities. We therefore ran our approach with timeouts 15+5s, and we checked if C-NN's performance remains competitive with AnyBURL's performance on YAGO3 (available in  [17] , with timeout=1000s like above).
Approach H@1 H@10 MRR AnyBURL 42. 9 63. 9 49 C-NN 42. 4±1. 5 67. 8±1. 4 51. 2±1. 4 The differences are comparable with those of other datasets, hence supporting the linear scaling rule. Now, the consequence of that linear scaling is that our approach cannot yet be applied in practice to very large KGs with millions of entities, like DBpedia or Wikidata, as the timeout per inference would have to be increased to non-practical values (about 15min on DBpedia). In order to achieve this goal, it will be necessary to compute more concepts of neighbours in less time. Optimization will not be enough, and it will be necessary to find robust sampling strategies: to compute C-NNs on a representative subset of entities, to select the relevant part of entity descriptions in the partitioning process, and to estimate the confidence of rules without computing all pattern matchings.
Example Inferences and Explanations In this section, we look at example inferences in order to discuss the explainability of our approach. Note that further work is required to make explanations easier to use, and that our objective is here to show the potential of our approach, and the limits of the current implementation. We here consider inferences for the Mondial datasets because they are easier to read. Indeed, the Freebase and WordNet datasets use opaque identifiers for entities (e. g. , 03735637 in WordNet, 05h43ls in Freebase), while Mondial uses clear URIs (e. g. , http://www. semwebtech. org/mondial/10/country/ANG for Angola). In the next section we discuss inferences on FB15k-237 in comparison to AnyBURL. We describe below four correct and one incorrect inferences. Those are not cherry-picked.
Detailed Comparison with AnyBURL In this section, we detail the comparison with predictions made by AnyBURL  5. Indeed, AnyBURL is the state-of-the-art among rule-based approaches. The main question we answer here is: How predictions differ between C-NN and AnyBURL? Table  8  gives for the four datasets the distribution of test instances into four cases depending on the Hits@1 success of each approach (C-NN and AnyBURL). If the rank-1 predicted entity is correct, then Hits@1 equals 1, otherwise it equals 0. Case 1/1 means that both approaches are correct, case 1/0 means that C-NN is correct but not AnyBURL, etc. The four cases sum up to 100%. We consider the null hypothesis that if one approach has lesser performance, then its set of correct predictions is a subset of the correct predictions of the other approach.
For instance, on FB15k-237 where AnyBURL is better, we expect that case 0/1 is empty. This is rejected in all four datasets. For instance, on FB15k-237, C-NN is the only correct approach for 4. 7% test instances, which amounts to 1918 test instances. Those results demonstrate that C-NN and AnyBURL can complement each other, and that improvement of state-of-the-art performances remains possible. This is made explicit in the last two columns that allow to compare Hits@1 of the best of the two approaches and the best ensemble of the two (union of correct predictions). We now detail a few test instances where C-NN and AnyBURL differ (cases 1/0 and 0/1). The analysis is limited by the fact that AnyBURL's code does not output rules as explanations for predictions (although the code could be modified to do so).
g. SMT  [17]. Running Example. The following example will be used to illustrate the main ideas. Example 1. We consider a textbook example  [66]  [Figure  7. 1, page 289] addressing the classification of a user's preferences regarding whether to read or to skip a given book. For this dataset, the set of features is: { A(uthor), T(hread), L(ength), W(hereRead) } All features take one of two values, respectively {known, unknown}, {new, followUp}, {long, short}, and {home, work}. An example instance is: {(A = known), (T = new), (L = long), (W = home)}. This instance is identified as e 1  [66]  with prediction skips. Figure  1a  shows a possible decision tree for this example  [66]   8. The decision tree can be represented as a set of rules as shown in Fig. 1b    L, T, A and W, as follows. We will associate (L = long) with L and (L = short) with ¬L.
Enumeration of CXps Our experiments demonstrate a novel, unexpected practical use case of CXps enumeration algorithms. In particular, we show that our method gives a new fine-grained view on both global and local standard explanations extracted from ML models. The goal of these experiments is to gain better understanding of existing explainers rather than generate all CXps for a given input. Setup. To perform enumeration of CXps in our first experiment, we use a constraint programming solver, ORtools  [65]. To encode the enumeration problem with ORtools we converted scores of XGBoost models into integers keeping 5 digits precision. We enumerate contrastive explanations in the increasing order by their cardinality. This can be done by a simple modification of Algorithm 1 forcing it to return CXps in this order.
So, we first obtain all minimum size contrastive explanations, and so on. We conduct two sets of experiments. The first experiment, called "real vs fake", distinguishes real from fake images. A dataset contains two classes of images: (a) original MNIST digits and (b) fake MNIST digits produced by a standard DCGAN model  [67]  (see Fig. 2a  and Fig. 2g  for typical examples). The second experiment, called "3 vs 5 digits", uses a dataset that contains digits "3" and "5" from the standard MNIST dataset (these digits are similar in writing) and we distinguish "3" from "5" images. Brief Overview of the SHAP Explainer. Given a classifier f and an explainer model g, SHAP aims to train g be similar to f in the neighborhood of some given point x. The objective function for SHAP is designed so that: (1) g approximates the behavior of the black box f accurately within the vicinity of x, and (2) g achieves lower complexity and is interpretable: ξ(x) = arg min g∈G L(π x , g, f ) + Ω(g), where the loss function L is defined to minimize the distance between f and g in the neighborhood of x using a weight function π x and Ω(g) quantifies the complexity of g; Ω(g) and π x are defined based on game-theoretic notions  [51].
We chose SHAP for our experiments due to its efficiency to generate an explanation (within seconds per input). "Real vs Fake" Experiment. First, we discuss the results of the "real vs fake" experiment in details. We train an XGBoost model  [14]  with 100 trees of depth 6 (accuracy 0. 85/0. 80 on train/test sets). We quantized images so that each pixel takes a value between 0 and 15, image pixels are categorical features in the model. Global and Local Explainers. We start by discussing our results on a few samples (Fig. 2a  and Fig. 2g ). First, we extract important features provided by XGBoost. As these features are global for the model, they are the same for all inputs (Fig. 2b  and Fig. 2h  are identical for real and fake images). Figure  2b  shows that these important features are no very informative for this dataset as these pixels form a blob of pixels that cover an image.
Finally, we compute corr = |S ∩ T |/|S| as the correlation measure. Note that corr = 0. 4 on average, i. e. our method hits 40% of best SHAP features. As the chances of two tools independently hitting the same pixel (out of 784) are quite low, the fact that 40% of |S| are picked indicates a significant correlation. "3 vs 5 Digits" Experiment. Consider our second the "3 vs 5 digits" experiment. We use a dataset that contains digits "3" (class 0) and "5" (class 1) from the standard MNIST (see Fig. 3a  and Fig. 3g  for representative samples). XGboost model has 50 trees of depth 3 with accuracy 0. 98 (0. 97) on train/test sets. We quantized images so that each pixel takes a value between 0 and 15. As before, each pixel corresponds to a feature. So, we have 784 features in our XGBoost model. Global and Local Explainers.
Enumeration of CXps and AXps Datasets. Here, we aim at testing the scalability of explanation enumeration. The results are obtained on the six well-known and publicly available datasets. Three of them were previously studied in  [70]  in the context of heuristic explanation approaches, namely, Anchor  [70]  and LIME  [69] , including Adult, Lending, and Recidivism. These datasets were processed the same way as in  [70]. The Adult dataset  [46]  is originally taken from the Census bureau and targets predicting whether or not a given adult person earns more than $50K a year depending on various attributes, e. g. education, hours of work, etc. The Lending dataset aims at predicting whether or not a loan on the Lending Club website will turn out bad. The Recidivism dataset was used to predict recidivism for individuals released from North Carolina prisons in 1978 and 1980  [73].
Two more datasets were additionally considered including Compas and German that were previously studied in the context of the FairML and Algorithmic Fairness projects  [21, 22, 24, 25] , an area in which the need for explanations is doubtless. Compas is a popular dataset, known  [4]  for exhibiting racial bias of the COM-PAS algorithm used for scoring criminal defendant's likelihood of reoffending. The latter dataset is a German credit data (e. g. see  [22, 25] ), which given a list of people's attributes classifies them as good or bad credit risks. Finally, we consider the Spambase dataset from the UCI repository  [20]. The main goal is to classify an email as spam or non-spam based on the words that occur in this email. Due to scalability constraints, we preprocessed the dataset to keep ten words per email that were identified as the most influential words by a random forest classifier.
The final score summations used in tree ensembles can be encoded into RIA formulas. ) Due to the twofold nature of Algorithm 2, it has to deal with (1) implicit hitting set enumeration and (2) entailment queries with SMT. The former part is implemented using the award-winning maximum satisfiability solver RC2  [37]  written on top of the PySAT toolkit  [36]. SMT solvers are accessed through the PySMT framework  [27] , which provides a unified interface to a variety of stateof-the-art SMT solvers. In the experiments, we use Z3  [17]  as one of the best performing SMT solvers. The conducted experiment was performed in Debian Linux on an Intel Xeon E5-2630 2. 60 GHz processor with 64GByte of memory. Given a dataset, we trained an XGBoost model containing 50 trees per class, each tree having depth 3.
(Further increasing the number of trees per class and also increasing the maximum depth of a tree did not result in a significant increase of the models' accuracy on the training and test sets for the considered datasets. ) All abductive explanations for every instance of each of the six datasets were exhaustively enumerated using the duality-based approach (Algorithm 2 in Algorithm 4). This resulted in the computation of all contrastive explanations as well). Evaluation Results. Table  2  shows the results. There are several points to make. First, although it seems computationally expensive to enumerate all explanations for a data instance, it can still be achieved effectively for the mediumsized models trained for all the considered datasets. This may on average require from a few dozen to several hundred of oracle calls per data instance (in some cases, the number of calls gets up to a few thousand).
Also observe that enumerating all explanations for an instance takes from a fraction of a second to a couple of seconds on average. These results demonstrate that our approach is practical. Second, the total number of AXps is typically lower than the total number of their contrastive counterparts. The same holds for the average numbers of abductive and contrastive explanations per data instance. Third and finally, AXps for the studied datasets tend to be larger than contrastive explanations. The latter observations imply that contrastive explanations may be preferred from a user's perspective, as the smaller the explanation is the easier it is to interpret for a human decision maker. (Furthermore, although it is not shown in Table  2 , we noticed that in many cases contrastive explanations tend to be of size 1, which makes them ideal to reason about the behaviour of an ML model.
Enumeration of AXps (and CXps)Function XpEnum(M π ,C, π)Input: M π : ML model, C: Input cube, π: Prediction Variables: N and P defined on the variables of C1 K = (N , P) ← (∅, ∅) ; // Block AXps & CXps 2 while true do st λ , λ) ← FindMHS(P, N ) ; // MHS of P st N st ρ , ρ) ← SAT(λ ∧ ¬M π ) ← ExtractCXp(M π , ρ, π) ← P ∪ UseLiteralsOf(µ)5 Experimental EvaluationThis section details the experimental evaluation to assess the practical feasibility and efficiency of the enumeration of abductive and contrastive explanations for a few real-world datasets, studied in the context of explainability and algorithmic fairness. To evaluate, we use Algorithm 1 and Algorithm 2 in Sect. 413.
Algorithm 2. 3 ( 4 if ¬st λ then break; 5 ( 6 if ¬st ρ then // entailment holds 7 ReportAXp(λ) 8 N ← N ∪ NegateLiteralsOf(λ) 9 else 10 µ 11 ReportCXp(µ) 12 P 23456789101112 Enumeration of AXps (and CXps)Function XpEnum(M π ,C, π)Input: M π : ML model, C: Input cube, π: Prediction Variables: N and P defined on the variables of C1 K = (N , P) ← (∅, ∅) ; // Block AXps & CXps 2 while true do st λ , λ) ← FindMHS(P, N ) ; // MHS of P st N st ρ , ρ) ← SAT(λ ∧ ¬M π ) ← ExtractCXp(M π , ρ, π) ← P ∪ UseLiteralsOf(µ)5 Experimental EvaluationThis section details the experimental evaluation to assess the practical feasibility and efficiency of the enumeration of abductive and contrastive explanations for a few real-world datasets, studied in the context of explainability and algorithmic fairness. To evaluate, we use Algorithm 1 and Algorithm 2 in Sect.
Motivating trial This work is based on an ongoing first-in-human dose-escalation trial of SAR440234  27  administered as a single agent to patients with relapsed or refractory acute myeloid leukemia, high-risk myelodysplastic syndrome or B-cell acute lymphoblastic leukemia (NCT03594955 28 ). SAR440234 is a novel bispecific T-cell engager antibody that activates and redirects cytotoxic T lymphocytes (CTLs) to enhance the CTL-mediated elimination of CD123-expressing tumor cells. Due to this mechanism of action, the main toxicity expected in this trial is the cytokine release syndrome (CRS), which is a systemic inflammatory response that has been associated with the peak of cytokine, considered as a pharmacodynamic endpoint. 29, 30 5] [6] [7] [8]  Namely, before administrating repeatedly the steady-state dose, defined as the maximum dose the regimen should reach, patients are treated with lower initial increasing doses, defined as the lead-in doses.
PCS of the MTD-regimen We compared the results of the DRtox_indep, DRtox_copula, and DRtox_cond with those of the logistic CRM. The performance of each method was evaluated with respect to the PCS, which is defined as the proportion over the 1000 trials for which the correct MTD-regimen was selected. The proportions of selecting each dose regimen in all scenarios are shown in Table  2. The results of additional scenarios can be found in Web Appendix A. 3 of Data S1. In Scenarios 1 to 4, where the true MTD-regimen is S 4 , the DRtox_indep, DRtox_copula, and DRtox_cond outperform the CRM by 10% in terms of the PCS. All three joint methods achieve very similar results and are robust to an unbalanced distribution of the probabilities of CRS and DLT o , as illustrated in Scenarios 3 and 4. In Scenario 2, defined with a higher association between the CRS and the DLT o , the DRtox_copula and DRtox_cond seem to perform better than the DRtox_indep as they account for the dependence between both toxicities.
However, increasing the prior ESS leads to poorer results when the initial guesses of DLT probabilities overestimate the true DLT probabilities (Scenario 5 where S 6 is the true MTD-regimen). We also evaluated the impact of considering a gamma distribution for the DLT o model and a log-normal distribution for the CRS model, and observed only a limited impact (Web Appendix B. 2 of Data S1). We also evaluated the effect of a noninformative distribution on the DRtox_copula parameter, that is, (0. 1, 0. 1). In this case, the results of the DRtox_copula are similar to the DRtox_indep (Web Appendix B. 3 of Data S1). We also evaluated our methods after an empiric CRM and observed only limited impact on our proposed methods (Web Appendix B. 4 of Data S1). Finally, we studied our methods on another set of dose regimens that is very different from the motivating trial to vary the timing of toxicity occurrences (Web Appendix C of Data S1).
In turn, they also need the ability to ascertain the extent to which such factors are also present in their setting and whether appropriating an approach is likely to engender similar outcomes as in the original setting  (Cartwright, 2013; Moss, 2013). iii) Free and unencumbered access to a robust evidence base. Arguments abound that educators need access to research evidence held in academic journals or research databases as well as access to high-quality research syntheses (e. g. ,  Cordingley, 2000; Goldacre, 2013). Such access enables a more cumulative perspective on topics of teaching and learning. Syntheses (such as those provided by the Education Endowment Foundation or the What Works Clearinghouse) are particularly useful, as they are often written in an accessible language, enabling practitioners to quickly and easily engage.
Applying multiple large-scale ML algorithms, such as multiobjective approximate Bayesian computation and Bayesian optimization, for model calibration and uncertainty quantification 3. Coordinating single-node ML tasks and ordered multinode MPI tasks in a scalable HPC workflow 4. Connecting the system to the real world via observed data and database access We demonstrate the scaling performance of our integrated approach through whole-machine runs on the Argonne Leadership Computing Facility (ALCF) Theta supercomputer. Furthermore, we discuss the potential impact of this work in developing novel use cases for HPC resources beyond COVID-19 to a generally applicable decision-support platform.