dataesr/scientific-paragraphs-categorization huggingface dataset:
Formally, let X be a set of elements, µ its mean, σ its standard deviation, and x an element of X. The z-score of x, denoted z(x), is computed as: z(x) = x -µ σ (31) Each feature score is then replaced with its z-score. To validate our choice of features, we study the pairwise correlation of the features and perform feature selection. For the latter, we use an automatic recursive feature elimination procedure, with a 10-fold cross validation over the entire dataset, and accuracy as a target. The least important features, in terms of the weights learned, are progressively pruned and the accuracy is computed for all resulting feature-sets. The optimal number of features corresponds to the set achieving the best accuracy.
Learning the classifier To train and evaluate our classifier, we asked experts to independently label explorations, by deciding for each query if it contributes (label C) or not (label N) to its exploration. For the sake of simplifying the experts task and to be less sensitive to experts subjectivity, we restrict the annotation task to binary labels. In order to obtain reliable labels, when labels are given by 2 experts, we computed the agreement between experts. We then include in the training dataset only the queries on which the experts agreed. In case of unbalanced dataset (one class of queries being significantly more present in the dataset than the other), we compared several methods on the basis of their respective accuracy and F1-measure, over a 10-fold cross-validation: either random undersampling of majority class, or oversampling of minority class.
In the last case, several heuristics have been tested: random oversampling, 3 variants of SMOTE (with different approaches to sample borderline points between classes) or ADASYN  [18]. We finally favor oversampling to ensure the model is learned over a balanced set of classes while not pruning too much the size of our datasets. To learn our model, we trained a linear SVM classifier on the dataset of queries described by their label and the set of select features, standardized using a z-score. When testing over a single dataset, the model is learned on 80% of the corpus, using 10-fold cross validation to choose its best hyperparameter. We kept the remaining 20% as a test set for model evaluation. When cross-testing with two datasets, the model is learned using 10-fold cross validation over the entire first dataset and then tested over the entire second one.
Experimental setup The experiments have been conducted on three datasets, two with real data and real explorations, and one with artificial data and artificial explorations. Real explorations on open data. The first dataset consists of navigation traces collected in the context of a French project on energy vulnerability. To constitute our corpus, we used three cubes instances built from open data, concerning expenses in mobility and heating. In the main cube, called MobPro, facts represent people trips between home and workplace, and dimensions allows to characterize a trip according to various characteristics of the worker (e. g. age, gender, level of studies), home (e. g. location, family size), job (e. g. location, branch), transport mode, traveled distance, energy used, etc. The cube is organized as a star schema with 19 dimensions, 68 (non-top) levels, and 24 measures.
To explore the cube, the students used Saiku 1 , a web application that allows to navigate OLAP databases in a user-friendly manner, and generates MDX code from graphical manipulations. The students were quite familiar with this OLAP tool, but not necessarily with the data in the cube. From this experiment, we could gather 39 explorations from the system logs. In total, these explorations represent 1608 queries, with an average of 41 queries per exploration. A particularity of some third party OLAP tools, like Saiku, is that their user interface submits a new query for each user action (including intermediate drag-and-drops), resulting in very long explorations in the log. Nevertheless, there were some extremely short explorations (7 explorations counting less than 10 queries), which mainly correspond to incomplete studies.
A subset of 1114 queries were labeled independently by two experts (interns working on OLAP exploration under the supervision of a lecturer), using a labeling tool specifically designed for that purpose. Both experts agreed at 67, 59% on the 1114 queries, which represents a total of 753 queries. On the agreed queries, 649 have been labeled C (contributive) and 104 N (non contributive). In order to avoid any side effect due to the classes distribution, we balanced our dataset using oversampling, as explained in Section 4. Real explorations on enterprise data. The second dataset consists in navigation traces of 14 volunteers of SAP in the context of a previous study on discovering user intents  [19]. Analysts covered a range of skills in data exploration, classed, based on their position in the company, in two expertise groups: beginners and expert users.
We set 10 business needs (named Q 1 to Q 10 ), each corresponding to a specific user interest. Users were asked to analyze some of the 7 available data sources to answer each of the 10 business needs, using a SAP prototype that supports keyword-based BI queries 2. The business needs were grouped in different business cases like: "For each European country, detect which genres of films did not reach the expected sales" or "In which Income Group would you classify a candidate country with a GDP of $6 billion?". As users enter keywords, the BI system suggests, on the fly, further tokens to complete the current ones, letting the user choose among them, as in web search engines. The underlying idea is that a suggestion completes the original BI question in order to obtain a well-formed query over a database.
Then, conversely to Saiku tool, SAP prototype only evaluates final queries, after all keywords were entered and a formal query was selected. In this context, average exploration length is around 5 queries. Another difference on this dataset, is the absence of timestamps in the log. We re-executed the queries in order to compute execution times, but we have no information about consideration time, which had to be excluded from the model. Finally note that elapsed time is only computed in the basis of execution time. All these technical differences, in addition to working with different types of users and different information needs, provide a good opportunity for testing our approach in different configurations. To this aim, our experiments also discuss the ability of our model to adapt to such configurations.
In total, this dataset contains 103 user explorations, accounting for 525 queries. Table  2  describes, for each business need, its difficulty, estimated by an expert (in terms of time, number of queries and exploited sources expected in its resolving), the number of explorations devised for solving it, the number of queries and the number of queries perceived as relevant by users in their own activity. For the sake of confidentiality, the 525 queries were labeled by only one expert, who is one of the authors of this paper. Interestingly, 52% of queries were labeled C (contributive), which results in no need for balancing in this case. Artificial explorations. The third dataset, with artificial data, comes from the Star Schema Benchmark  [20] , and was used with artificial explorations. The Star Schema Benchmark (SSB) is a variation of TPC-H, a popular benchmark from the Transaction Processing Performance Council (TPC).
SSB cube consists of a relational database under the form of a star schema, with one fact table and 4 dimension tables. We used the SSB generator to generate data with a scale factor of 1, corresponding roughly to 1 Gb of data. We used the PDGF data generator  [21]  to generate a realistic instance with skew data. Instead of using the rather limited SSB workload, we generated artificial explorations using CubeLoad  [22] , a tool for generating realistic explorations over star schemas. CubeLoad takes as input a cube schema and creates the desired number of explorations according to templates modeling various user exploration patterns. Templates available in Cubeload simulate: (Goal Oriented ) users with limited OLAP skills pursuing a specific analysis goal, (Slice And Drill and Slice All ) more advanced users navigating with a sequence of slice and/or drill operations, (Exploratory) users tracking unexpected results with exploratory sessions.
We generated a collection of 100 explorations, each one containing between 30 and 70 dimensional queries. Noticeably, explorations were generated before the generation of the cube instance, which enabled us to take advantage of the most frequent selection predicates in the explorations to produce data skew in the most queried zones of the cube. Environment. All experiments were run on a 64 bits Windows 8. 1 Operating System, featuring a Intel(R) Xeon(R) CPU E3-1241 v3 @3. 50GHZ and 16GB of RAM. Cubes are stored using the MonetDB database for SSB and SQL Server for MobPro. Our prototype is written in Java 8 and Python 3, with Scikit-learn and Imbalanced-learn  [23]  packages.
Feature scores computation Our first experiment consists in computing the feature scores for all queries in both open and enterprise datasets. In Tables  3  and 4 , we report for each descriptor its range, average value and standard deviation. In Table  3  we also report the average time needed for the computation. A first observation is that computation remains in interactive time, in the sense that it does not exceed few hundred milliseconds. In average, the computation of all the feature scores for a query is 152 milliseconds, which is low, especially given that the average consideration time for a query result is 29, 371 milliseconds. A second observation is the ranges of values that show important variations from one feature to another, which justifies the standardization done before training the SVM.
Concerning the comparison among datasets, there are some major differences in the range of feature values. First, students use more filters (features NoF , FDepth et NCF ) than analysts. This can be explained by the size of the cubes (bigger in the open dataset) and the nature of informations needs (student analyzed specific phenomena in portions of the cubes while analyst analyzed the overall data). Second, execution times are considerably higher in the open dataset because of the underlying BI system, which impacts features ExecTime, QF and ElapsedTime. Finally, Saiku interface causes explorations to be longer (feature NoQ) as previously discussed.
Feature correlation analysis Our second test aimed at validating the choice of features. We analyzed the pairwise correlation of feature scores, using Pearson's correlation coefficient, over each corpus of queries independently. We report the result in the heat maps of Figure  1  that represents for each pair of features the correlation between them with a specific color code. As expected, correlation is maximal on the diagonal which represents the correlation of a feature with itself. Correlation values range between -1 (maximal negative correlation) and 1 (maximal positive correlation), with 0 indicating no correlation. In the heat map, the more intense the yellow A first general observation is that the vast majority of coefficients show no significant correlation. Interestingly, similar trends of correlation happen in both datasets.
From Figure  2 -left it can be seen that selecting and removing features from the 25 original features decreases the accuracy of the classifier in the case of the open dataset. This trend is clearly less pronounced for the enterprise dataset in Figure  2 -right, that shows a decrease, but has a second high accuracy score for 8 features. This conducts to observe some variability for the enterprise dataset in the number of features to select. As a consequence and as in both datasets, the model with the whole set of features provides a high accuracy, we decide to keep all the features in the model in the experiments hereafter.
Contribution model for the open dataset In the following test, we use the subset of queries that were labeled similarly by the two experts, as explained in Subsection 6. 1. This represents a corpus of 753 queries. We used 10-fold validation, with 80% of the corpus for training and 20% for test. The training set was balanced by oversampling, as explained in Section 4. Table  5  presents the different strategies that we compared to balance the 2 classes of our dataset, either by over-sampling the minority class or under-sampling the majority class. It can be seen that all results are very close, but ADASYN provides the best results along with SMOTE regular. However, ADASYN has the advantage of providing a slightly larger dataset (1057 tuples) for our experiments after resampling. We decide to use ADASYN as an oversampling strategy for the open dataset.
Contribution model for the enterprise dataset The next experiment aims at investigating whether the contribution model is sensible to the application context. To this end, we repeat the previous experiment for the enterprise dataset (corpus of 525 queries), building a second contribution model. Many of the differences between the datasets were introduced in Subsection 6. 1: different types of users (students vs. analysts); different types of information needs (fuzzy information needs to be extended by students vs. predefined information needs); different characteristics of underlying data (large cubes with tens of dimensions and measures vs. small cubes with specific data); different query tools (Saiku vs. SAP prototype). Another important difference is that queries in each dataset were labeled by different experts.
This is a major issue as the concept of query contribution (as the notion of quality itself) is fuzzy and highly dependent on the evaluator. For example, the experts labeling the open dataset indicated that queries providing the same measures, levels and filters (i. e. identical in our query model) but presenting results in a different way (swapping columns-rows, different charts) were highly contributive, because they showed the analysis of a specific phenomenon around these data. On the other hand, the expert labeling the enterprise dataset preferred queries providing new information and in relation to the user need. Repetitive queries were judged as non contributive. The model we obtained is presented in Table  8 , which shows, for each query feature, its importance in the definition of contribution to an exploration.
The model of contribution was evaluated on the test set; results, reported in Table  9 , are similar to the ones obtained for the open dataset. A first general observation is that the importance of the features is substantially different compared to the model obtained on the open dataset. Features that have the highest weights in one model, have negative polarity in the other (ex. RI (relative identity) and IsRefine), and features that have the lowest weights in one model have positive or insignificant weight in the other (ex. FDepth (filter depth), RI (relative identity), RR (relative recall), RP (relative precision), QRP (query relative position)). In addition, 15 out of 24 common features have opposite polarity. The model captures the differences on the underlying datasets. For example, open data cubes being very large, many contributive queries are quite aggregated and have many filters for focusing in a specific portion of a cube.
This explains the substantial weight of NoF (number of filters) and the quite high weights of NoL (number of levels) and NoM (number of measures). On the other hand, enterprise data sources being simpler, both in the number of dimensions and levels, and users tend to analyze the entire dataset (less filters), but at specific data granularity (ex. media shops, countries, years) as required in their information needs. In this context, NoL (number of levels) and LDepth (level depth) had more substantial weights than NoF (number of filters) and FDepth (filter depth). The model also captures labeling differences. For example, on the open dataset, the substantial weight of RI (relative identity) and in general of almost all relative features, reflect the expert's taste of contributive queries being similar to previous ones.
Cross-evaluation of contribution models on real datasets A common objective of machine learning oriented techniques is to be generalizable to any kind of context. The tests reported in previous two sections indicate that, when properly trained on queries related the objective at hand, our method produces good and explainable results. The following test aims at pushing one step further the experiment and at investigating if a model learned for a data set can predict on a completely different data set. It is expected that the results are not as good as previous experiments because of all the differences between datasets and annotation listed in Section 6. 5. Our experimental protocol is as follows. Two tests are conducted: the first one consists of training the model on the open dataset and testing on the enterprise dataset, and the second one consists of performing the other way around.
In each test, 100% of the tuples of the learning dataset are used to train the contribution model, this model being tested on 100% of tuples from the second test dataset. Table  10  presents for each test configuration the classification results observed. Although, the results seem very contrasted, with good results observed in the second test, this experiment shows that our approach is not able, in this context and because of the inherent limits presented before, to learn a single contribution model that can be applied on all our datasets. Indeed, the model learned on the open dataset fails to capture the nature of contributive queries in the enterprise dataset as traduced by a very low recall score. In the second test, the results seem more promising but are biased due to the unbalance of the classes in the open dataset.
Similarly, Exploratory explorations are expected to contain contributive queries because of the diversity in the queries and the increase in information novelty. Table  11  presents, per exploration template, different statistics on the contribution of the queries forming the explorations. The second line corresponds to the average of the contribution of the requests as predicted by the model learned on the real dataset. The next three lines respectively represent the standard deviation, the minimum and the maximum of contribution a query can reach within the pattern. First, it can be observed that queries from the Slice All template are mostly non contributive queries (25% of them are contributive) with a negative average contribution score of -0. 508. This can be explained by the relative simplicity of this pattern which consists solely in making slices on a single level of the cube during the entire exploration.
Model vs. User expertise In general, an analyst with high level of expertise performs explorations of better quality, with more contributing queries and also queries with higher contribution. The objective of this experiment is to verify that our model able to confirm this intuition. To this aim, we conducted this experiment on the open dataset where each exploration has been tagged by professors with A, B or C labels, depending on analyst's skills. Label A corresponds to good explorations, clearly following an information need, investigating it and containing coherent queries. Students producing such explorations are considered to have analysis skills. Contrarily, label C denotes those of the students that produced poor explorations, with less contributive queries, typically switching topics, with no clear information need.
Assessing OLAP explorations The objective of this last experiment is to validate our primary hypothesis: assuming that expert users are more likely to develop better explorations, we claim that the quality of an exploration can be measured by evaluating the skill of writing contributive queries. To this aim, we relate the scores provided by our continuous Knowledge Tracing (KT hereafter) prediction model to the labels given by the experts on the explorations on the open and enterprise datasets (both sets of explorations were labeled as explained in the previous subsection). In order to learn the KT model, we started by learning KT parameters, as explained in Section 5. It can be seen from Table  13  that the initial probability of writing a contributive query P (L 0 ) is very low, which can be directly related to the fact that explorations have been performed by master students who knew very little about the data beforehand.
Interestingly, and expectedly, Table  14  shows a higher P (L 0 ) for enterprise users. However, in both cases users have a sound theoretical background on OLAP exploration which in turn explains the relatively good probability P (T ), i. e. , the probability to acquire the skill at each step of the exploration. Finally, the exploratory nature of OLAP analysis, combined with limited knowledge of the dataset, translates in the explorations by a lot of trials and errors that increased significantly the average probability and standard deviation of P (G) and P (S). P(L0) 0. 085 P(T) 0. 243 mean(P(G)) 0. 320 variation(P(G)) 0. 307 mean(P((S)) 0. 323 variation(P(S)) 0. 273     For the open dataset, it appears clearly that our model provides consistent evaluations, giving users a rating that corresponds to the assessment made by the expert.
The distinction between competent analysts (A and B) and non-competent analysts (C) is clearly marked, in contrast to the distinction between A and B which is less pronounced. This can be explained by the fact that it is difficult for an expert to distinguish between a good analyst and a very good one. However, the distinction between a good and an unqualified analyst is intuitively much easier. The result is more nuanced on the enterprise dataset, even though we retrieve a quite good distinction between competent and noncompetent analysts (C-labeled explorations tend to cluster on the right of the chart, which corresponds to low scores). Interestingly, if only one third of the explorations obtain a score greater than 0. 9 for the open dataset, it is slightly more than 50% for the enterprise dataset, which reflects the average expertise of enterprise users.
In other words, the KT model was able to correctly retrieve that enterprise users were rather skilled compared to students who explored the Mobpro cube. Finally, we evaluate our continuous KT based on a traditional RMSE score. RMSE has been shown to be the strongest performance indicator for binary KT with significantly higher correlation than Log Likelihood and Area Under Curve  [24]. In our case, this RMSE score is computed as the difference between the expected contribution of each query in an exploration and the value predicted by the continuous KT for each of these queries. Our KT model obtains a RMSE score of 0. 291 with a standard deviation of 0. 181 for the open dataset, and 0. 238 with a standard deviation of 0. 270 for the enterprise dataset. Consistently with the literature on KT, we consider that this score are rather good and we conclude that our model is effective at assessing if the skill "writing a contributive query" is acquired.