dataesr/scientific-paragraphs-categorization huggingface dataset:
INTRODUCTION While there exists a few predominant applications of natural language processing (NLP) regarding finance, for example analyzing sentiment of financial news or reports, many practices remain under-represented. One of these unexplored research topics is hypernym extraction. A hypernym indicates a word/concept that has the highest level of category abstraction and generally has hyponyms, which are terms describing a more specific concept in said category. The task of hypernym extraction generally deals with finding the hypernym-hyponym association that usually belongs to the kind of "is-a" relationship. To the best of our knowledge, FinSim  1  is the first shared task that tackles hypernym extraction in the financial domain. Rather than using news or annual reports, the organizers targeted prospectus  [13] , a type of financial document that describes investment offering to the public, which is mandatory to file and submit to the Securities and Exchange Commission, according to Investopedia 2.
The shared task consists of a list of financial terms extracted from a set of prospectuses that need to be assigned to their corresponding toplevel concepts. These hypernyms are known beforehand, hence the task can be considered as multi-class classification. The remainder of this paper is organized as follows. In Section 2, we present and discuss a selection of works concerning hypernym extraction methods. Then, in Section 3, the dataset explored in this work is presented. The proposed model is detailed in Section 4 and the experiments are described in Section 5. We present and discuss the obtained results in Section 6. Finally, Section 7 concludes this paper and hints at future work.
DATA For training the models, we were provided with 600 terms along with their respective hypernyms/labels, which include ten top-level financial concepts: Bonds, Forward, Funds, Future, MMIs, Option, Stocks, and Swap. Upon observation, we noticed several challenges. First, the training dataset is relatively small, with only 600 entries. While it is much larger than last year (100 terms for the training data), it is still not enough to apply neural-based approaches. As for the characteristic of the terms given in the training set, we realized that, occasionally, these terms contain the top-level concepts to which they belong to. Most of the time, terms usually do not have any hypernyms as part of them. However, there are terms that contain hypernyms of other classes while belonging to different top-level concepts.
In some extreme cases, a term can have both hypernyms that it belongs to and hypernyms that are irrelevant. A number of 119 related hypernyms was observed, while the number of unrelated hypernyms was of 53, and 12 were common to both, in a total of 160 hypernyms. Here are a few examples extracted from the dataset: "Corporate Bonds" contains the "Bonds" hypernym, which is also the category this phrase belongs to. With "Fixed Recovery Swap", while it has the "Swap" concept in it, this term is not a lower concept of said hypernym, but belongs to "Credit Index". Including the "Bond" and "Future" hypernyms, the term "Single Name Bond Future", however, has as top-level ontology term "Future".
METHODS The architecture we proposed is based on a siamese neural network that contains two pre-trained BERT encoders proposed by  [3]  with the same configuration and the same parameters 4 , as presented in Figure  4. This type of architecture allows updating the weights of both encoders such that the produced term embeddings are semantically meaningful and can be compared. We use cosine similarity for comparing the two-term representations, which is defined as 4 The method is implemented and available at https://github. com/UKPLab/sentencetransformers follows: cos(t, e) = te ∥t∥∥e∥ = 𝑛 𝑖=1 t 𝑖 e 𝑖 𝑛 𝑖=1 (t 𝑖 ) 2 𝑛 𝑖=1 (e 𝑖 ) 2 (1) where  is the vector representation of the term and  is the vector representation of the ontology term. The model adds a pooling operation to the output of the BERT encoder using the output of the [CLS]-token and we computed the mean of all output vectors, as this strategy proved the best results in our preliminary experiments.
RESULTS AND DISCUSSION Table  1  describes the results of different methods on the test dataset that was split from the given 600 terms. We used different machine learning techniques as baselines: logistic regression, support vector machine, random forest and decision trees. Unigram TF-IDF along with two pre-trained embeddings, the provided FinSim embeddings and GloVE 7 , were used. Among these baselines, logistic regression and SVM coupled with GloVE yielded the best results, where SVM performed better in the mean rank metric while logistic regression has better accuracy by a small margin. In the results from Table  1  we remark that the chosen baseline models with TF-IDF weighting, GloVe and FinSim embeddings are competitive with the siamese-based models with pre-trained BERT encoders. We also notice that using FinSim pre-trained embeddings obtain in general lower performance scores than those with GloVe pre-trained embeddings, which might indicate that the articles on which the model was trained were not sufficient.
We also evaluated at which hypernyms our proposed methods failed to detect accurately. Table  2  illustrates the performance, using precision, recall and F-1 measure, of our best baseline (SVM using GloVE embeddings) and the siamese network with the best performing architecture, with the pre-trained bert-base-uncased 6 Fine-tuned on the STSbenchmark (semantic textual similarity benchmark) dataset 7 We used the model pre-trained on Wikipedia 2014 and Gigaword 5 (vector size 300). for the encoders. Upon evaluating these metrics as well as the predictions made, we discovered that terms containing their respective hypernym get often miss-classified, with an imbalance between the precision and the recall. "MMIs", has exceptionally poor F1 score. We suspect that since this hypernym is an acronym, its representation might be unclear, hence could appear irrelevant.
CONCLUSIONS AND FUTURE WORK In this paper, we attempt to solve the problem given in FinSim-2 the Shared Task on Learning Semantic Similarities for the Financial Domain by using a siamese network with BERT encoders to compute a similarity score between terms and hypernyms in a ranked manner. Our preliminary experimental results clearly outperformed the baseline, but only ranked around the median in the official scoring. For future improvement, since the dataset was rather small, we plan to approach the task with few-shot learning. Moreover, due to the potential that this type of method could have, another aspect we need to work on is improving the model by focusing the attention mechanism on the contextual words in order to a better disambiguation between the ontology and the terms.
In the last few years, the concept of data lake has become trendy for data storage and analysis. Thus, several approaches have been proposed to build data lake systems. However, these proposals are difficult to evaluate as there are no commonly shared criteria for comparing data lake systems. Thus, we introduce DLBench, a benchmark to evaluate and compare data lake implementations that support textual and/or tabular contents. More concretely, we propose a data model made of both textual and CSV documents, a workload model composed of a set of various tasks, as well as a set of performance-based metrics, all relevant to the context of data lakes. As a proof of concept, we use DLBench to evaluate an open source data lake system we previously developed.
There are several benchmarks for big data systems in the literature, but none of them considers the wide range of possible analyses in data lakes. Hence, we propose in this paper the Data Lake Benchmark (DLBench) to evaluate data management performance in data lake systems. We particularly focus in this first instance on textual and tabular contents, which are often included in data lakes. DLBench is data-centric, i. e. , it focuses on a data management objective, regardless of the underlying technologies  [2]. We also designed it with Gray's criteria for a "good" benchmark in mind, namely relevance, portability, simplicity and scalability  [8]. More concretely, DLBench features a data model that generates textual and tabular documents. By tabular documents, we mean spreadsheet or Comma Separated Value (CSV) files whose integration and querying is a common issue in data lakes.
A scale factor parameter SF allows to vary data size in predetermined proportions. DLBench also features a workload model, i. e. , a set of analytical operations relevant to the context of data lakes with textual and/or tabular content. Finally, we propose a set of performance-based metrics to evaluate such data lake implementations, as well as an execution protocol to execute the workload model on the data model and compute the metrics. The remainder of this paper is organized as follows. In Section 2, we show how DLBench differs from existing benchmarks. In Section 3, we provide DL-Bench's full specifications. In Section 4, we exemplify how DLBench works and the insights it provides. Finally, in Section 5, we conclude this paper and present research perspectives.
Textual Benchmarks In this category, we consider big data benchmarks with a consequent part of text on one hand, and purely textual benchmarks on the other hand. Big-DataBench  [23]  is a good representative of the first category. It indeed includes a textual dataset made of Wikipedia 3  pages, as well as classical information retrieval workloads such as Sort, Grep and Wordcount operations. One of the latest purely textual benchmarks is TextBenDS  [22] , which aims to evaluate performances of text analysis and processing systems. For this purpose, TextBenDS proposes a tweet-based data model and two types of workloads, namely Top-K keywords and Top-K documents operations. Other purely textual benchmarks focus on language analysis tasks, e. g. , Chinese  [25]  and Portuguese  [5]  text recognition, respectively.
Discussion None of the aforementioned benchmarks proposes a workload sufficiently extensive to reflect all relevant operations in data lakes. In the case of structured data, most benchmark workloads only consider SQL operations (TPC-H, TPC-DS, Hi-Bench). More sophisticated machine learning operations remain marginal, while they are common analyses in data lakes. Moreover, the task of finding related data (e. g. , joinable tables) is purely missing, while it is a key feature of data lakes. Existing textual workloads are also insufficient. Admittedly, BigDataBench's Grep and TextBenDS's Top-K documents operation are relevant for data search. Similarly, Top-K keywords and WordCount are relevant to assess documents aggregation  [9, 22]. However, other operations such as finding most similar documents or clustering documents should also be considered.
Thus, our DLBench benchmark stands out, with a broader workload that features both data retrieval and data content analysis operations. DLBench's data model also differs from most big data benchmarks as it provides raw tabular files, inducing an additional data integration challenge. Moreover, DLBench includes a set of long textual documents that induces a different challenge than short texts such as tweets  [22]  and Wikipedia articles  [23]. Finally, DLBench is data-centric, unlike big data benchmarks that focus on a particular technology, e. g. , TPCx-HS and HiBench.
Data Model Data Description DLBench includes two types of data to simulate a data lake: textual documents and tabular data. Textual documents are scientific articles that span from few to tens of pages. They are written in French and English and their number amounts to 50,000. Their overall volume is about 62 GB. Tabular data are synthetically derived from a few CSV files containing Canadian government open data. Although such data are often considered as structured, they still need integration to be queried and analyzed effectively. DL-Bench features up to 5,000 tabular files amounting to about 1,4 GB of data. The amount of data in the benchmark can be customised through scale factor parameter SF , which is particularly useful to measure a system's performance when data volume increases. SF ranges from 1 to 5.
DLBench's data come with metadata catalogues that can serve in data integration. More concretely, we generate from textual documents catalogue information on year, language and domain (discipline) to which each document belongs. Similarly, we associate in the tabular file catalogue a year with each file. This way, we can separately query each type of data through its specific metadata. We can also jointly query textual documents and tabular files through the year field. Eventually, textual documents are generated independently from tabular files. Therefore, each type of data can be used apart from the other. In other words, DLBench can be used to assess a system that contains either textual documents only, tabular files only, or both. When not using both types of data, the workload model must be limited to its relevant part.
Tabular data are reused from an existing benchmark  [13]. These are actually a set of 5,000 synthetic tabular data files generated from an open dataset stored inside a SQLite 6  database. Many of the columns in the tables contain similar data and can therefore be linked, making this dataset suitable to assess structured data integration as performed in data lakes. We apply on this original dataset 7  a script to extract all (or a part) of the tables in the form of raw CSV files. As for textual documents, this second script also generates a metadata catalogue. The script as well as guidelines are available online 7.
Overview of AUDAL To demonstrate the use of DLBench, we evaluate AUDAL  [16] , a data lake system designed as part of a management science project, to allow automatic and advanced analyses on various textual documents (annual reports, press releases, websites, social media posts. ) and spreadsheet files (information about companies, stock market quotations. ). The AUDAL system uses an extensive metadata system stored inside MongoDB 8  , Neo4J 9  , SQLite 8 and ElasticSearch 10  to support numerous analyses. AUDAL provides ready-to-use analyses via a representational state transfer application programming interface (REST API) dedicated to data scientists, and also through a Web-based analysis platform designed for business users.
Setup and Results AUDAL is implemented on a cluster of three VMware virtual machines (VMs). The first VM has a 7-core Intel-Xeon 2. 20 GHz processor and 24 GB of RAM. It runs the API and also supports metadata extraction. Both other VMs have a mono-core Intel-Xeon 2. 20 GHz processor and 24 GB of RAM. Each of the three VMs hosts a Neo4J instance, an ElasticSearch instance and a MongoDB instance to store AUDAL's metadata. The results achieved with DLBench show that AUDAL scales quite well (Figures  1 2 3 4 5 6 ). Almost all task response times are indeed either constant (Tasks #3, #7, #8, #9 and #10) or grow linearly with SF (Tasks #1, #2, and #4 to #6). In addition, we observe that except Task #6 (that takes up to 92 seconds), all execution times are reasonable considering the modest capabilities of our hardware setup.
Eventually, metadata generation time and size scale linearly and almost-linearly, respectively (Figure  7 ). We can also see that metadata amount to about half the volume of raw data, which illustrates how extensive AUDAL's metadata are. We observe some fluctuations in the results, with sometimes negative slopes while SF increases. Such variations are due to external, random factors such as network load or the Java garbage collector starting running. However, the influence on the runtime is negligible (of the order of a tenth of a second) and is only visible on simple queries that run in half a second.
Conclusion In this paper, we introduce DLBench, a benchmark for data lakes with textual and/or tabular contents. To the best of our knowledge, DLBench is the first data lake benchmark. DLBench features: 1) a data model made of a corpus of long, textual documents on one hand, and a set of raw tabular data on the other hand; 2) a query model of twenty query instances across ten different tasks; 3) three relevant metrics to assess and compare data lake implementations; and 4) an execution protocol. Finally, we demonstrate the use of DLBench by assessing the AUDAL data lake  [16] , highlighting that the AUDAL system scales quite well, especially for data retrieval and tabular data querying.
The analysis of most clinical and other empirical datasets allows for alternative strategies, therefore, reported findings of a single analysis may not represent the full range of possible and justifiable estimates derivable from the same data. The multi-analyst approach is a method to systematically assess whether the reported results and conclusions are dependent on the chosen analytical strategy. We present consensus-based guidance and checklist for conducting and reporting multi-analyst studies, in which many analysts independently analyse the same dataset. We discuss why broader adoption of the multi-analyst approach will strengthen the robustness of results and conclusions in clinical and basic sciences.
In the left panel, all plausible paths support conclusion A; in the right panel, most plausible paths support conclusion B. This illustrates that without testing alternative paths, it remains unknown whether the conclusion is robust to justifiable alternative analytical strategies. The analytical variability may be particularly high for datasets that were not initially recorded for research purposes, since we typically only have poor knowledge on the exact measurement procedures and data generation mechanisms. The increasing availability of large routinely collected datasets, for instance through administrative claims and electronic health records, raises hopes and promises of "real-world" evidence and personalized treatment regimes. Yet these hopes and promises are accompanied by a growing realization that the potential analytical variability is much greater in the analysis of these datasets than in the analysis of data from clinical trials or non-randomized studies with prospective data collection.
As an example of analytical variability in medical database research, the journal Surgery published two articles (1,2) a few months apart that used the same dataset to investigate the same question: Does the use of a retrieval bag during laparoscopic appendectomy reduce surgical site infections? Two reasonable, but different, analyses were applied (with notable differences in analytical choices including inclusion and exclusion criteria, outcome measures, sample sizes, and covariates). As a result of the different analytical choices, the two articles reached opposite conclusions, one finding that the use of a retrieval bag reduced infections, and the other that it did not  (3). This and other medical examples  (4) (5) (6)  illustrate how independent analysis of the same data (in this case, unplanned) can reach different, yet justifiable, conclusions.
The analytical robustness of results and conclusions can be studied by evaluating many distinct analysis options simultaneously (e. g. , vibration of effects  (7)  or multiverse analysis (  8 )) or by involving multiple analysts who independently analyse the same data  (9) (10) (11) (12) (13) (14) (15) (16) (17) (18). Rather than exhaustively evaluating all plausible analyses, the multi-analyst method examines analytical choices that are deemed most appropriate by independent analysts. Botvinik-Nezer et al. (13) , for example, asked 70 teams to test the same hypotheses using the same functional magnetic resonance imaging dataset. They found that no two teams followed the same data preprocessing steps or analysis strategies, resulting in substantial variability in their conclusions. This and other initiatives  (9) (10) (11) (12) (13) (14) (15) (16) (17) (18)  demonstrate that the multi-analyst approach is able to explore how research findings vary depending on the applied analytic strategy.
Preregistering the project Before the start of the project, on 11 November 2020, a research plan was compiled and uploaded to a time-stamped repository at https://osf. io/dgrua. During the project, we followed the preregistered plan in all respects except implementing slight changes in the wording of the survey questions to improve comprehension. Although we declared that we would share the R code and codebook of our analyses, the project did not require us to conduct analyses in R.
Consensus survey The expert consensus questionnaire was sent out individually to each expert first on 8 February 2021 in the following Qualtrics survey available at https://osf. io/wrpnq/. The survey contained the 10 recommended practices grouped into the following 5 stages: Recruiting Co-Analysts; Providing the Dataset, Research Questions, and Research Tasks; Conducting the Independent Analyses; Processing the Results; and Reporting the Methods and Results. The respondents were asked to rate each of the 10 recommended practices on a 9-point Likert-type scale ('I agree with the content and wording of this guidance section' ranging from "1-Disagree" to "9-Agree"). Following each section, the respondents could leave comments regarding the given item. The preregistration indicated consensus on the given item, if the interquartile range of the corresponding ratings was 2 or smaller; it defined support for an item if the median of its corresponding ratings was 6 or higher.
g. , authorship). 3. Provide the datasets accompanied with a codebook that contains a comprehensive explanation of the variables and the datafile structure. 4. Ensure that co-analysts understand any restrictions on the use of the data, including issues of ethics, privacy, confidentiality, or ownership. 5. Provide the research questions (and potential theoretically derived hypotheses that should be tested) without communicating the lead team's preferred analysis choices or expectations about the conclusions. Conducting the Independent Analyses 6. To ensure independence, we recommend that co-analysts should not communicate with each other about their analyses until after all initial reports have been submitted. In general, it should be clearly explained why and at what stage co-analysts are allowed to communicate about the analyses (e.
Recruiting co-analysts Depending on the specific goal of the research, the recruitment of co-analysts can happen in several ways. Co-analysts can be recruited before or after obtaining the dataset. With stricter eligibility criteria, co-analysts can be invited individually from among topic experts or statistical experts. Follow-up open invitations can ask experts to suggest others to be invited. Alternatively, the lead team can open the opportunity to anyone to join the project as a co-analyst within the expert community (e. g. , in professional society mailing lists and on social media), where expertise can be defined as the topic requires it. It is important to note that whenever the co-authors' behavior is the subject of the study then they should be regarded similarly to human participants respecting ethical and data protection regulations.
Providing the dataset The lead team can invite the co-analysts to conduct data preprocessing (in addition to the main analysis). If the lead team decides to conduct the preprocessing themselves, showing their preprocessing methods can be informative to the co-analysts, but also has the potential to influence them if the preprocessing reflects some preference of methods or expectations of outcomes. Before providing the dataset, the lead team should ensure that data management will comply with legal (e. g. , the General Data Protection Regulation (GDPR) in the European Union) and ethical regulations applying to all teams (see 22). If the dataset contains personal information, a version should be provided where data can no longer be related to an individual. An alternative is to provide a simulated dataset and ask the co-analysts to provide code to analyse the data  (23, 24).