dataesr/scientific-paragraphs-categorization huggingface dataset:
Select some performance measures. Performance measures can be classified into effectiveness and efficiency measures. The former focus on assessing the ability of a proposal to discern well between the information to be extracted and the information to be ignored, whereas the latter focus on the computing resources that it requires to do so. We suggest that the researcher should select a number of effectiveness measures that must fulfil the following requirements: (a) they must be global, that is, they must provide an overview of how a proposal behaves regarding every kind of error it can make; (b) they must not be biased in the presence of unbalanced datasets, which are natural in web information extraction; (c) and they must be extensible from a per-slot level to a per-extractor level in an unbiased manner.
Step 2: Create evaluation splits Before using the selected datasets, we need to split them into evaluation splits, that is, pairs of learning and testing sets. Note that learning sets do not actually make sense for proposals that are based on heuristics, so we have to create evaluation splits that are specifically tailored to these proposals. Our proposal is to use the sub-sampling method that is presented in Fig. 3. It takes a collection of datasets as input and returns a set of evaluation splits, i. e. , a set of tuples of the form (ls, ts) in which ls denotes a learning set and ts denotes a testing set. For every dataset, the method first sets h to half its size and then repeats the following steps γ times: it first creates two reservoirs of documents called r 1 , which stores a random half of the documents in the corresponding dataset, and r 2 , which stores the other half.
Step 3: Compute raw experimental data Computing the raw experimental data consists in running every proposal on the appropriate evaluation splits, depending on whether they are rule-based or heuristic-based, and then collecting the nonderived performance measures that the researcher has selected. Fig. 4  shows the method that we propose to use. It works on a collection of proposals, a collection of measures, and a collection of evaluation splits; it computes a map called result in which pairs (p, s) of proposals and evaluation splits are associated with maps d that associate every performance measure with the value that was computed regarding proposal p on evaluation split s. In other words, the experimental data can be interpreted as a map from pairs of proposals and evaluation splits onto the corresponding values of the measures.
Note that not every evaluation split is appropriate to run every proposal: rule-based proposals can work on evaluation splits that provide both a learning and a testing set, whereas heuristic-based proposals must be run on evaluation splits that provide a testing set only. Our experience proves that there are cases in which it is not possible to compute the performance measures regarding a given proposal on a given evaluation split because there is a bug in the implementation or it simply takes too long or consumes too much memory and cannot be executed. Note that such failures must be recorded as missing values; such values shall later be used to compute failure-related derived measures. The method to gather the experimental data is straightforward, but computing confusion matrices or dealing with unsupervised and heuristic-based proposals is, however, a little more involved.
In the following subsections, we provide additional details. Computing confusion matrices. The documents in the input datasets must be annotated; that is, a person must have labelled every piece of information to be extracted with a user-defined slot; the information that is not to be extracted is assumed to be implicitly labelled as belonging to a pre-defined null slot. Given a testing set, it is not difficult to compute the exact matchings, that is, the pieces of information that are correctly or incorrectly extracted as belonging to a given slot. The problem is how to compute inexact matchings. Such matchings are common, for instance, with proposals that work on DOM trees since DOM nodes are very likely not to be perfectly aligned with the information to be extracted. To introduce our proposal, we assume that S = {s 1 , s 2 ,.
If p > m, then the corresponding pm tokens are computed as true positives in the previous case. Dealing with unsupervised and heuristic-based proposals. In the case of supervised proposals, it is not difficult to compute matchings because they learn extraction rules that are specifically tailored to extracting information as belonging to one of the slots that the user has defined in the input datasets. In the case of unsupervised or heuristicbased proposals, the problem is complicated by the fact that they ignore the annotations in the input datasets since they were devised to learn to extract as much information as possible from them, which is assigned to computer-generated slots. It is the user who must analyse the resulting computer-generated slots and map them onto userdefined slots, that is, she or he must assign a meaning to them.
Once the performance knee is computed, the data that do not correspond to evaluation splits whose size is equal to the performance knee can be purged. Note that computing the size of an evaluation split depends on the proposal: if it is rule-based, then the size of the evaluation split is computed as the size of the corresponding learning set; if it is heuristic-based, then it is computed as the size of the corresponding testing set. Note, too, that ARIEX only keeps γ evaluation splits for a proposal that is not purged, all of which are the size of the best-performing evaluation splits found for that proposal. The second step removes the data that correspond to proposals that are very bad. These are the proposals that do not achieve a value for the purging measure above a minimum acceptable value when it is computed on a per-extractor level.
Step 6: Produce a report The last step of ARIEX consists in producing a report in which the results of the previous steps are summarised and commented by the researcher. Below, we present a suggestion regarding how to organise it. Abstract. As usual, the abstract must provide a short overview of the report and highlight the original findings. Experimental environment. The goal of this section is to provide an overview of the experimental environment. Our suggestion is to organise it as follows: Hardware and software. Regarding the hardware, we suggest that the researcher should report on the processors, the motherboard, the memory, the persistent storage, and whether it was virtual or bare metal. Regarding the software, we suggest that the researcher should report on the operating system, the virtual machines, and the libraries used, if applicable.
The report should also make it clear what the selected purging measure is (η) and provide a justification regarding their weights. (Please, recall that we have made some suggestions regarding the most appropriate measures, but our method is open to accommodate new measures as they are proven to be adequate in our context. ) Datasets. For each dataset, the report should list its name and version, the web site from which it was downloaded, the number of documents that it provides, and how large they are in average. We suggest that the datasets should be grouped in categories according to their topic and that the researcher should list the slots that were extracted in each category. Statistics. The researcher must report on the significance level that she or he selected (α) and the number of repetitions set in the method to compute the evaluation splits (γ ).
A case study In this section, we present one of the many case studies that we have conducted to polish our proposal. Below, we present the corresponding report. Abstract. In this case study, we have evaluated, compared, and ranked five proposals to which we refer to as P0, P1, P2, P3, and P4. We keep them anonymous because it is not our intention to contribute with a ranking of a few existing proposals, but to illustrate how ARIEX works in practice so that it can serve as a guideline for other researchers. Our study clearly reveals that analysing the experimental data intuitively can very easily lead to wrong conclusions that are not supported from a statistical point of view. Experimental environment. Next, we report on the experimental environment that we used in our study. Hardware and software.
The experiments were run on a virtual computer that was equipped with four Intel Xeon E7-4807 cores that ran at 1. 87 GHz, had 4 GiB of RAM, and 16 GiB of persistent storage. The motherboard was a Supermicro X8QB6. All of the proposals were run using the Oracle Java Development Kit 1. 7. 9_02. The operating system was Microsoft Windows 7 Pro 64-bit. The regular expression engine required by some proposals was provided by GNU RegEx 1. 1. 4. Some proposals required the input documents to be cleaned before running on them; we used JTidy R938 and JSoup 1. 8. 2. No changes to the default configuration of the hardware or the software were made. Proposals. Table  1  summarises the proposals that we have studied. They all work on semi-structured documents and are closed, but differ significantly regarding the techniques on which they rely, namely: P0 refers to a very simple baseline that uses rules of the form L -R, where L and R are 5-token disjunctive patterns that match the left and the right of the information that has been annotated in the learning sets; P1 is a heuristic-based proposal that compares a number of documents to find the differences amongst them, which are returned as the extracted information; P2 is a rule-based, unsupervised proposal that also finds differences amongst a number of documents and generalises them into a regular expression with variables that capture the differences; P3 is a hybrid proposal that first learns the structure of the information using an automata and then learns transition conditions using a standard machine-learning technique; and P4 is a proposal that learns DOM-based extraction rules using a propositional inductive logic technique.
Performance measures. Table  2  summarises the performance measures that we have used. We have adhered to the suggestions that we have made in the previous section; additional details are provided in Appendix A. Note that we have grouped the measures into effectiveness measures, whose weight is 70%, learning efficiency measures, whose weight is 10%, and extraction efficiency measures, whose weight is 20%. These figures reflect our opinion that it is very important that a proposal be very effective and efficient at extracting information, but the time to learn a rule set, if applicable, is not a concern nowadays; however, it must not be completely neglected because, other things equal, the learning efficiency can make a difference between two proposals. Datasets. Table  3  summarises the datasets that we have used.
We selected 40 datasets from Sleiman and Corchuelo's repository (version 1. 0)  [31]. They were grouped in the following categories: books, movies, cars, events, doctors, jobs, realty, and sport players. From each dataset, we randomly selected a subset of 30 documents and we extracted records of the form Record{attribute 1 , attribute 2 ,. , attribute n }. Statistics. We set the significance level to α = 0. 05 and the number of repetitions in the method to compute the evaluation splits to γ = 10. Experimental data. Next, we report on the measures that we have collected. Non-derived measures. Tables 4-6 report on the mean values that we gathered regarding the non-derived measures (before purging them) and Table  7  presents the best approximations that we have found. Regarding the area under the ROC curve, note that all of the proposals seem to behave logarithmically with respect to the size of the evaluation splits (S), except for P0, which seems to behave linearly.
Regarding the learning time and the extraction time, all of the proposals seem to behave linearly, with small slopes; this confirms that they are very scalable and are then appropriate to deal with large datasets. Furthermore, the learning times range from a few milliseconds to quarter a minute and the extraction times range from a few milliseconds to a few seconds, which is reasonable in this context. Note that proposal P1 is based on heuristics, so it does not have a learning phase; thus the learning times are 0. 00 seconds in every case. Regarding the learning and extraction memory, they also seem to require an amount of memory that evolves linearly as the size of the evaluation splits increases; this is again a good piece of news since it confirms that all of the proposals are scalable in practice.
• Regarding the failure ratio, note that proposals P0, P1, and P2 have failed on some datasets. P0 failed in 13. 00% of the evaluation splits; after working this issue out, we found that the problem was the library that it uses to implement regular expressions, which uses a backtracking parser that fails to match some regular expressions of the form α|αβ. P1 and P2 failed in 6. 00% of the evaluation splits because they cannot work on a single document, so there were many evaluation splits on which they could not work. Rankings. Next, we report on the local rankings and the global ranking that we have computed. Local rankings. Table  9  reports on the local rankings that we have computed. The first column reports on the empirical ranks, as they are computed from the empirical data after purging and normalising them.
Most of the remaining proposals provide enough empirical results, which helps support the conclusions better, but the methods used to evaluate and compare them were not solid enough. Regarding the experimental environment, only a few proposals have paid attention to describing the hardware and the software used in the evaluation process  [11, 31, 37]. Regarding comparing the results, it was surprising that many proposals were not compared to others at all, but to some variations of themselves that resulted from changing the values of their configuration parameters  [4, 15, 36, [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49]. It was also surprising that not many proposals were evaluated on at least 20 datasets, which is the minimum that we recommend  [11, 13, 26, 31, 39, 41, 44, 48, [50] [51] [52] ; a few other proposals were evaluated on 10-20 datasets, which still amounts to a significant number of experiments  [38, 53] ; the others were evaluated on less than 10 datasets, which we do not think is acceptable to draw conclusions.
Furthermore, the results are analysed from a statistical point of view in very few cases  [31, 37]  which makes it difficult to assess if the differences found amongst a number of proposals are statistically significant or not. Finally, no global rankings are computed; the proposals are compared according to different measures in isolation, but no attempt is made to compute a global ranking. The literature on formal methods. We have also surveyed the few existing formal ranking methods. Lehnert and Sundheim  [18] , Chinchor et al. [19] , and Hirschman  [20]  range amongst the first authors who worked on this topic. They worked in the context of the well-known MUC conferences, whose focus was on extracting information from free-text documents; they published a number of datasets so that the proposals that were presented at these conferences could be evaluated using a semi-automatic software tool that computed precision, recall, over-generation, and fallout.
Setting up the experimental environment The papers on web information extraction do not generally put an emphasis on describing the hardware or the software. They generally provide a list with the datasets that were used, but few other details are presented. They commonly commit to precision, recall, and the F 1 scores as effectiveness measures; unfortunately, almost none of them reports on efficiency measures. The specific ranking methods do not put an emphasis on describing the hardware or the software. They all are accompanied with collections of datasets that were devised by a community of researchers. They focus almost exclusively on precision-and recall-based measures; efficiency measures are not taken into account. The common theme in the literature seems to be that describing the hardware and the software is usually paid very little attention and that the performance measures used just provide a partial view of how good an information extractor is.
In ARIEX, we did not forget about describing them. Regarding the performance measures, it is surprising that most of the rankings in the literature focus exclusively on effectiveness measures, and do not provide a clue on efficiency measures. Our method proposes to use both kinds of measures since, otherwise, the resulting rankings would not provide an overall picture of how a proposal performs. We propose to use a small set of orthogonal measures, that is, measures that focus on different performance issues and are complementary to each other. In our survey of the literature, we have also found that precision, recall, and the F 1 score are the most common measures, but, unfortunately, it has also revealed that they are not the most appropriate in our context. The reason is that precision and, therefore, the F 1 score are skewed by unbalanced datasets, which are very common in our context.
To overcome this problem, they proposed to use the well-known bootstrap method in order to compute statistically accurate estimators of the performance measures. The other proposals simply comment on the methods that have been used by other authors and suggest that it is important to mention both the proportions between learning and testing sets and the re-sampling method used to split the datasets. They also encourage researchers to use existing learning and testing sets when using standard datasets and also recommend using as many evaluation splits as possible in order to achieve as much statistical significance as possible. Unfortunately, they do not commit to a specific method or analyse the pros and cons of the existing proposals. The standard to create the evaluation splits seems to be the kfold cross validation method, where k is typically set to 10.
We think that this is not appropriate in the field of web information extraction because the learning sets are k -1 times larger than the testing sets. It is necessary that the testing sets are sufficiently large so as to reduce the possibilities that a proposal seems to work well or bad by chance. The problem is that one must provide 100 annotated documents to have 10 test documents, which is a time-consuming and error-prone task. Furthermore, all of the evaluation splits are the same size, which makes it difficult to analyse how the number of documents may have an impact on the performance of a proposal. Last, but not least, a dataset may include an outlier document, that is, a document that deviates largely from the others and makes it difficult to extract information from it; unfortunately, that document shall be in k -1 learning sets, that is, shall have a negative impact on every experiment, but one.
Furthermore, the method promotes evaluating all of the available proposals on the same evaluation splits, which we think provides a biased view of their performance since some proposals need evaluation splits with only a few learning documents and others need evaluation splits that have a larger number of learning documents to achieve good results. Note that evaluating proposals on different splits does not make the comparison unfair if the appropriate statistical methods are used; however, the datasets on which the splits are computed must be the same because, otherwise, the features of the documents would be different and the comparison would not make sense. Note too, that this method is not appropriate for heuristic-based proposals since it uses as many documents as possible to produce learning sets that these proposals do not require.
Neither think we that n-repeated random sub-sampling is appropriate for our purpose; at least, as it has been instantiated in the literature. The reason is that authors have used this method to create evaluation splits in which, typically, most documents are used for learning purposes, which we have justified is not appropriate in our context. Furthermore, all of the evaluation splits are the same size since the only purpose is to make the choice of documents as random as possible, which we have also justified is not appropriate in our context. In ARIEX, we propose a sub-sampling method that overcomes the problems that we have identified. It splits the datasets into a number of evaluation splits so that there is a set of evaluation splits for each possible size of the learning or the testing sets, as long as the size of the learning set never exceeds the size of the testing set.
Thus, ARIEX ensures that the learning sets are not larger than the testing sets; it can generate a reasonable number of evaluation splits from relatively small datasets; furthermore, the existence of outlier documents does not have a negative impact on every evaluation split, which helps identify them. We run the proposals in their best experimental conditions, which involves computing the size of the evaluation splits on which they perform the best. Thus, every proposal is likely to use a different evaluation split on the same dataset and the comparison will be still fair. Heuristic-based proposals can use testing sets that are larger than in the case of k-fold cross validation and the availability of both learning and testing sets of different sizes allows to compare several proposals regarding their best-performing evaluation splits.
In the literature, effectiveness measures generalised by using macro-, micro-, or weighted averages. Unfortunately, the authors have not commonly paid attention to the problem that micro-and weighted averages are skewed in the context of unbalanced datasets, which makes them of little interest in our context  [64]. In ARIEX, we recommend using macro-averages because they are known not to be skewed in our context; furthermore, we suggest using the area under the ROC curve as an effectiveness measure and we have found an efficient means to extend it to a per-extractor level  [65]. The third issue is regarding how to compute confusion matrices when evaluating an unsupervised or a heuristic-based proposal. If a proposal is supervised, then it is trained to return pieces of text as belonging to a specific user-defined slot; contrarily, if a proposal is unsupervised, then it learns to extract as much information as possible, which is automatically assigned to computer-generated slots; heuristic-based proposals do not learn extraction rules, but assign the information that they extract directly to computer-generated slots.
The problem is that they rely on techniques that are computationally intensive and outdated; furthermore, there are many cases in which they do not lead to a total ranking, but the problem has not been studied further because these methods are not intended to be automated, but require a person to interpret the results and draw conclusions. Furthermore, none of the methods provide a means to compute a global ranking from the experimental data. ARIEX also computes local rankings on a per-measure basis and it also makes sure that the results are statistically sound. The difference is that we rely on Wilcoxon's Rank-Sum test, which is efficient and there are two versions that are specifically adapted to small and large datasets. This is a non-parametric test because it does not assume that the experimental data have a pre-defined distribution and it works on non-paired samples, so that it can be applied to the experimental data computed from the best-performing evaluation splits of each proposal.