dataesr/scientific-paragraphs-categorization huggingface dataset:
Now we present the full algorithm for interactive data exploration, as shown in Algorithm 1. The input is the database D, a positive example x + , a negative example x , a user-defined accuracy threshold , and the sampling ratio. First, we extract an evaluation dataset D eval from the database D (line 1). For now, let us assume D eval to be D. Then we initialize data structures, setting the unknown partition of the evaluation dataset to be D eval. The algorithm next goes through iterative exploration. Lines 8-14 update our dsm model. The data space model, R + and R , is updated with the newly labeled example(s) by the user (lines 8-9). This step incrementally updates our convex polytope for R + and the union of convex polytopes for R based on computational geometry  [3]. Afterwards, the corresponding partitions of D eval are incrementally updated (line 10).
In this step, some examples are removed from the unknown partition D u and placed to the positive partition D + or the negative partition D. The accuracy is then estimated using the Three-Set Metric. We also keep track of the labeled and unlabeled examples using D labeled and D unlabeled. We use the labeled examples to train a classifier, that is, the version space model in our dsm. Then lines 15-28 implement uncertainty sampling using dsm. With probability , we perform uncertainty sampling from a pool that is restricted to the unknown partition of the evaluation set, D u. Then the example chosen by uncertainty sampling is labeled by the user. With probability 1 , we perform uncertainty sampling from a pool that is a subsample of all unlabeled examples in the database. Then the example chosen by uncertainty sampling is first run through our data space model, (R + , R ), to see if it falls in the positive or negative region and hence can be labeled directly by the model.
Otherwise, it will be labeled by the user. Then the algorithm proceeds to the next iteration. It repeats until it has met the user accuracy requirement based on the lower bound o↵ered by our Three-Set Metric, or reached the maximum of iterations allowed (line 29). Finally, we run the dsm model over the database to retrieve all tuples predicated to be positive. For the subsample procedures used in the algorithm, we defer their details to Section 3. 4.
Optimization of Sampling Methods Sampling for creating the evaluation set (OPT1). In line 1 of Algorithm 1, we construct an evaluation set D eval to develop a lower bound of the dsm model. Ideally, we want D eval to be as large as the entire database D. For e ciency, we can only a↵ord to have a memory-resident sample. The analysis in Theorem 3. 2 indicates that we can choose a sample of size n from the database, and achieve an approximate lower bound of the F1-score of our dsm algorithm when evaluated on the database. The sample, denoted as Deval , will expedite line 10 of Algorithm 1. For example, the SDSS database used in our experiments contains 190 million tuples. Denote the true lower bound as b, when n = 50k, our approximate lower bound b approximates b by ✏ . 005 with probability 0.
975 or higher. When n = 1. 9 million, b approximates b by ✏ . 001 with probability 0. 994 or higher. Sampling for pool-based active learning (OPT2). Algorithm 1 contains two subsample procedures for poolbased uncertainty sampling, that is, choosing the most uncertain example from the pool for labeling next. The subsample routine in line 17 creates a pool of unlabeled examples from the uncertain partition of the evaluation set, D u , which is memory-resident. This can be implemented using reservoir sampling. The subsample routine in line 21, however, creates a pool, from the unlabeled portion of the entire database, which is large and not materialized. Our work o↵ers a sampling-based optimization to avoid scanning the unlabeled database in each iteration. Due to space constraints the interested reader is referred to  [22]  for details.
Since the user has gone through thinking when deciding the label, specifying a subset of attributes that lead to the negative label can be facilitated via a graphical interface such as in  [11]. Factorization for dsc queries. Formally, factorization concerns a user interest query Q defined on an attribute set A of size d, and can be written in the conjunctive form, Q1 ^. ^Qm. Each Qi, i 2 [1. m], uses a subset of attributes Ai = {Ai1, • • • , A id i }. The family of attribute sets, A = (A1, • • • , Am), is pairwise disjoint with d = P m i=1 di, and is called the factorization structure. In practice, we can derive the factorization structure from available data in a database. It is a partitioning of the database attributes with two properties: 1) Each attribute (a) A query in the dsc family is not convex in either the positive region (green) or the negative region (red).
Therefore, the uncertain region of Q is R u f = R d R + f R f Finally, denote the entire dataset as D and the evaluation set as Deval. The definitions of positive, negative, and uncertain partitions of Deval are the same as in Definition 3. 4 with the three regions replaced by R + f , R f , and R u f. Formal results on factorization. When the true query is conjunctive, taking advantage of the factorization property of the data space proves to be effective for enlarging the positive and negative regions known to the DSM algorithm and hence reducing the uncertain region. Formally, we have the following results. To prove R ✓ R f , it is sufficient to prove that every arbitrary convex cone that R is built upon, is contained by R f. Let e be the apex of a convex cone R e in Definition 3. 2. Since e is negative for Q, there exists I such that its factorization vector e I is negative for QI.
Let I (R e ) be the projection of the cone R e onto the subspace spanned by AI. Then for a point or the set of all possible points enclosed in the object, ⇥ represents the Cartesian product between two sets, and R c represents the complement of set R. Therefore, the uncertain region of Q is R u f = R d R + f R f Finally, denote the entire dataset as D and the evaluation set as Deval. The definitions of positive, negative, and uncertain partitions of Deval are the same as in Definition 3. 4 with the three regions replaced by R + f , R f , and R u f. Formal results on factorization. When the true query is conjunctive, taking advantage of the factorization property of the data space proves to be effective for enlarging the positive and negative regions known to the DSM algorithm and hence reducing the uncertain region.
Therefore, the uncertain region of Q is R u f = R d R + f R f Finally, denote the entire dataset as D and the evaluation set as Deval. The definitions of positive, negative, and uncertain partitions of Deval are the same as in Definition 3. 4 with the three regions replaced by R + f , R f , and R u f. Formal results on factorization. When the true query is conjunctive, taking advantage of the factorization property of the data space proves to be effective for enlarging the positive and negative regions known to the DSM algorithm and hence reducing the uncertain region. Formally, we have the following results. PROOF. If a point x = (x1, • • • , xm) 2 R f , then there exists some subspace spanned by attributes AI such that xI 2 R I. Since all points in R I are negative for QI based on proposition 3.
We note that a positive example with respect to Q implies that its projection in every I th subspace satisfies Qi. However, a negative example with respect to Q implies that there exists at least one subspace, I, where the example's projection is negative for QI , while there may still other subspaces, I 6 = I, where the example's projection satisfies QI. Next we build the positive and negative regions of the query Q from the positive and negative regions in the subspaces: R + f = ⇥ m I=1 R + I , R f = ⇥ m I=1 (R I ) c (1) where each R + I (or R I ) is interpreted equally as a geometric object or the set of all possible points enclosed in the object, ⇥ represents the Cartesian product between two sets, and R c represents the complement of set R. Therefore, the uncertain region of Q is R u f = R d R + f R f Finally, denote the entire dataset as D and the evaluation set as Deval.
Therefore, the uncertain region of Q is R u f = R d R + f R f Finally, denote the entire dataset as D and the evaluation set as Deval. The definitions of positive, negative, and uncertain partitions of Deval are the same as in Definition 3. 4 with the three regions replaced by R + f , R f , and R u f. Formal results on factorization. When the true query is conjunctive, taking advantage of the factorization property of the data space proves to be effective for enlarging the positive and negative regions known to the DSM algorithm and hence reducing the uncertain region. Formally, we have the following results. or the set of all possible points enclosed in the object, ⇥ represents the Cartesian product between two sets, and R c represents the complement of set R. Therefore, the uncertain region of Q is R u f = R d R + f R f Finally, denote the entire dataset as D and the evaluation set as Deval.
Therefore, the uncertain region of Q is R u f = R d R + f R f Finally, denote the entire dataset as D and the evaluation set as Deval. The definitions of positive, negative, and uncertain partitions of Deval are the same as in Definition 3. 4 with the three regions replaced by R + f , R f , and R u f. Formal results on factorization. When the true query is conjunctive, taking advantage of the factorization property of the data space proves to be effective for enlarging the positive and negative regions known to the DSM algorithm and hence reducing the uncertain region. Formally, we have the following results. can appear in only one partition. 2) If several attributes may be used in the same predicate (e. g. , the positional attributes rowc-colc, or the velocity attributes rowv-colv from SDSS), they must be assigned to the same partition.
When e is projected onto z, it lies in the positive region. According to Eq. 2, the new negative region R f extends the convex cone R xy by all possible values of z, resulting in a geometric shape enclosed by the three solid red lines in the figure. Again, the new R f is much larger than R. Formal results. For both dsc and p-dsc queries, we o↵er formal results of the polytope model with factorization. f , R f ), are a superset of (R + , R ) that the original dsm o↵ers. Hence, the lower bound of F1-score built from (R + f , R f ) is higher than that from (R + , R ) based on Def. 3. 4. Testing assumptions of dsmF. Factorization replies on two assumptions, which our system will test when a user is performing data exploration online. The first assumption is that we have a correct factorization structure, A = (A1, • • • , Am), for a database derived from its query trace or schema.
ONLINE FEATURE SELECTION The user exploration task may start with more attributes than those included in the final learned model (user interest query). The attributes that are not included in the final model are noise in data exploration and cause slow convergence of the model. To remove such noisy attributes, we employ both o✏ine dimensionality reduction on the database and online feature selection techniques. We examined dimension reduction methods including PCA for o✏ine compression, and Random forests (RF) and Gradient boosting regression trees (GBRT) for online feature selection. Since these are standard methods, we leave their description and evaluation to  [22]. We found that GBRT works best for reducing dimensions. Therefore, we outline how GBRT is used for online feature (attribute) selection in our work.
EXPERIMENTAL EVALUATION We implemented all of our proposed techniques in a Java-based prototype for data exploration, which connects to a PostgreSQL database. In this section, we evaluate our techniques and compare to recent active learning algorithms  [5, 15] , active search  [16] , and explore-by-example systems, Aide  [12, 13]  and LifeJoin  [9]. Datasets: Our evaluation used two datasets. (  1 ) SDSS (190 million tuples) contains the "PhotoObjAll" table with 510 attributes. By default, we used 1% sample (1. 9 million tuples, 4. 9GB) to create an evaluation set for dsm and for poolbased uncertainty sampling -our formal results in Section 3. 4 allowed us to use the 1% sample for data exploration, yet with bounded di↵erence of ✏ . 001 from the accuracy achieved over the full database with probability 0.
994. (2) Car database (5622 tuples): this small dataset is used for our user study because it is more intuitive for users to perform explorative analytics. We defer a detailed discussion to §6. 3. User Interest Queries: We extracted 8 query templates from the SDSS query release  [38]  to represent user interests. They allow us to run simulations of user exploration sessions, as in  [12, 13] , by using the true query answers as the proxy for user labeling. The 8 templates are shown in Table  1. Each template is instantiated with di↵erent constants to vary query selectivity in [0. 01%, 10%]. In particular, Q1-Q3 represent patterns that are convex in the positive region (Q + c ). Q4 retrieves tuples outside a circle, hence in the Q c class. Q5-Q7 combine these attributes, optionally with irrelevant attributes, for scalability tests.
Dual-Space Algorithm with Factorization We evaluate our Dual-Space Model (dsm) and compare it to two ML techniques: (i) Active Learning (AL) runs uncertainty sampling  [5, 15]  to obtain labeled examples for building a classifier, which is the version space part of dsm. We run AL with an SVM or a standard kNN classifier (kNN + ). (ii) Active Search (AS)  [16]   dsm further o↵ers a lower bound in F1-score. Fig. 5(c ) shows that the lower bound is quite close to the true F1-score. The time cost of dsm depends on the sampling method. Recall from Algorithm 1 that with probability , it samples from the unknown partition of the polytope model, D u ; otherwise, it does so from a subsample of the unlabeled examples in the database, D unlabeled. =0 leads to high time costs because the examples retrieved from D unlabeled may repeatedly fall in the positive or negative region of the polytope model, wasting resources with no new information.
Comparison to Alternative Systems We next compare our system to two state-of-the-art explore-by-example systems: 1) LifeJoin  [9]  uses SVM for classification and active learning for seeking the next example for labeling. But it di↵ers in the SVM implementation from our work. We implemented LifeJoin techniques as additional methods in our system. 2) Aide  [12, 13]  uses decision trees as the classification model and customized methods for seeking the next example for labeling. We obtained the source code from the authors. When comparing these systems, we ex-  clude the overhead to find the first positive example as these systems use di↵erent methods / assumptions to find them. Expt 5: F1-score is reported in Fig. 6 (b) for a 2D query, and in Fig. 6(c ) for 4D and 6D queries. (1) For the 2D query, our system outperforms Aide, which further outperforms LifeJoin.
User Study using a Car Database We conducted a user study by building a car database  3. The database includes 5622 vehicles with 27 attributes such as the model, year, length, height, engine power, and retail price. Additional details on the database are given in  [22]. Our study has two objectives: (1) build a query trace to understand the characteristics of data exploration tasks in this domain; (2) use this trace as the ground truth of user interests to evaluate our system. To develop the query trace, we designed task scenarios with di↵erent price constraints and usage patterns, e. g. , buying a car for everyday city commute, outdoor sports, an elderly in the family, or a small business. The 18 users in our study belong to two groups: the first 11 users are CS professors and graduate students, while the rest of 7 are non-technical people.
We asked each user to find all the cars that meet the requirements of the assigned task so that he can provide a recommendation service to customers who belong to the given scenario. Each user proceeds in three phases: 1) Review the schema: Since this is a familiar domain, most users can understand the schema quickly. 2) Initial exploration: We next show sample data to help the user understand the data and materialize his preference. We also ask the user to come up with the first positive example via (a) naming a car that he already has in mind, or (b) reviewing a sample set precomputed for the given task based on the price constraints, body type, year, etc. , and finding one that appears relevant. Two users chose option (a) while the others chose option (b). 3) Iterative manual exploration: In the third phase, the user is asked to specify his interest precisely by (a) sending a SQL query to the database 4  ; (b) reviewing a subset of the returned tuples; (c) going back to revise the query.
RELATED WORK Data Exploration. Faceted search iteratively recommends query attributes for drilling down into the database, but the user is often asked to provide attribute values until the desired tuple(s) are returned  [26, 34, 35]  or o↵er an "interestingness" measure and its threshold  [10]. Semantic windows  [25]  are pre-defined multidimensional predicates that a user can explore. These methods are di↵erent from our active learning approach. Recent work also supports time series data  [32]  or avoids false discoveries of statistical patterns  [44]  during interactive data exploration. Finding best database objects based on user preferences  [40]  assumes a numeric weight per database attribute, which is di↵erent from the active learning approach to discover the user interest on the fly.
Query by Example is a specific framework for data exploration. Earlier work on QBE focused on a visualization front-end that aims to minimize the user e↵ort to learn the SQL syntax  [23, 33]. Recent work  [31]  proposes exemplar queries which treat a query as a sample from the desired result set and retrieve other tuples based on similarity metrics, but for graph data only. The work  [37]  considers data warehouses with complex schemas and learns the minimal project-join queries from a few example tuples e ciently. It does not consider selection with complex predicates, which is a focus of our work. The work  [24]  helps users construct join queries for exploring relational databases, and  [28]  does so by asking the user to determine whether a given output table is the result of her intended query on a given database.
Query formulation has been surveyed in  [8]. The closest to our work is LifeJoin  [9] , which we compared in Section 6. 2. Query By Output (QBO)  [42]  takes the output of one query on a database, and constructs another query such that running these two queries on the database are instanceequivalent. Dataplay  [2]  provides a GUI for users to directly construct and manipulate query trees. It assumes that the user can specify the value assignments used in his intended query, and learns conjunctions of quantified Horn expressions (with if-then semantics) over nested relations  [1]. Active Learning. Tong and Koller  [41]  provide a theoretical motivation on selecting new examples using the notion of a version space, but with unknown convergence speed. Related to our work is a lower bound on the probability of misclassification error on the unlabeled training set  [7].
CONCLUSION AND FUTURE WORK We presented new algorithms for interactive data exploration in the active learning framework, which leverage the subspatial convex and conjunctive properties of database queries to overcome the slow convergence of active learning. Our results show that our dsm algorithm significantly outperforms active learning  [5, 15] , active search  [16] , and existing explore-by-example systems, Aide  [12, 13]  and LifeJoin  [9] , in accuracy and convergence speed, while maintaining the per-iteration time within 1-2 seconds. In future work, we will address inconsistent labeling by extending our dsm model to a probabilistic model, extend query patterns to multiple disjoint areas using exploration versus exploitation, and leverage data properties to further improve accuracy.
Plane and Equipment Agent The plane agent is a kind of composite agents, which contains the equipment agents. The data for this agent consist of aircraft data (aircraftNumber, aircraft-Type, aircraftState), the flight data (flightNumber, flightHours, city), scheduled maintenance data (ACheckTime, BCheckTime, CCheckTime, DCheckTime) and parts data (type, changeoverTime, repairTime, repairingTime, serialNumber, price, equipmentState). When all the aircraft are loaded from the database, they are in the state "Ready". They are scheduled by departure and arrival events, which are defined in the FA. In the state "SidedTransitCheck", we assume that fault events occur with a specific probability. This state is the only place where the CRA may be triggered. When one aircraft needs to do the scheduled maintenance like "TransitCheck", "ACheck", "BCheck", "CCheck" or "DCheck", the EA will be called to check states of relevant parts.
Experiment Design and Data The performance evaluation of the AMSS is conducted via examining the impacts of the number of aircraft (experiment 1) and technicians (experiment 2) on service level (sl) and maintenance cost. In this model, the key factors are: the probabilities of the occurrence of fault events and uncertain events, and the numbers of aircraft and technicians. We expect to create a severe simulation environment to see the responses of the system. Thus, the probabilities of the occurrence of fault events and uncertain events are amplified, which are assigned as 0. 5 and 1 respectively. In terms of cost analysis, the downtime cost, labor cost, stock cost and parts purchase cost are taken into consideration. The definitions of costs are shown below in equation (  1 )-(  4 ), where the totalRepairTime (shown in table  2 ) and interval are counted by seconds, the salary (i) and time (i) represent the ith aircraft technician's salary counted by hours and the corresponding working time, the number (i) implies the number of the ith kind of parts and the days means the number of storing days, and the cost (j) means the cost of the jth kind of parts.
In addition, all the costs are counted by dollars. It should be noted that the scheduledTime in table 2 means the total repair time of scheduled maintenance. DowntimeCost = (totalRepairT ime -interval) * 2. 8; (1) LaborCosts = n (i=1) salary(i) * time(i); (2) StockCosts = n (i=1) number(i) * 10 * days; (3) P artsP urchaseCosts = n (j=1) cost(j); (4) The industrial relevance of values for data is explained in the following. The data interval (around 1h), inventoryPrepare(1-3h), repairingTime(20-120 mins)(the faulty parts that can be repaired on site) and repairTime(5-30h)(the faulty parts that have to be repaired at workshops) are of a certain level of industrial relevance. The value for data inventoryPrepare is given by the random distribution unif orm discr(min, max). The rest are determined by the data stored in the database.
However, in terms of the data informationDelivery and maintenancePlanning, the values for them are assumed as 2 seconds and 10 minutes respectively, which are of less of industrial relevance. The data of the faultIdentification, exceptionalEventSSA and exceptionalEventSTA are determined depending on knowledge stored in the database. If the solution has been stored in the database, they will take 0 minute. If not, they will take either 30 minutes or one hour. The definition of service level is shown as follow: servicelevel = totalRepairT ime -interval. (5) The relation between the delayed time and service level is shown in table  3. When analyzing the service level for aircraft maintenance, the first two important indicators are the "sl=10" and "sl=0". The indicators of "sl=9 to 6" are the second most important indicators.