dataesr/scientific-paragraphs-categorization huggingface dataset:
A. Experimental Methodology We conducted a series of experiments that pursued three objectives: (1) to investigate how the effectiveness of a VGH is subjective to the dataset that will employ it for anonymization; (2) to demonstrate how and why using a dynamic VGH quality assessment scheme, which considers the distribution of the input datasets, is better than a static scheme; and (3) to demonstrate how a rating scale can be applied to classify the quality of VGHs. For this purpose, we created a set of candidate VGHs modeling the same domain for a socio-economic attribute (see Section IV-B). We then evaluated the quality of those VGHs using the d-GSL and s-GSL metrics. s-GSL was chosen as the rival metric for comparison as it represents, to the best of our knowledge, the first quantitative mechanism to assess the quality of VGHs to perform data anonymization.
Later, we conducted the anonymization of datasets (see Section IV-B) using the candidate VGHs and the Datafly algorithm (a popular k-anonymity based algorithm)  [21]. We tested different levels of privacy, varying the k-values ∈ [30. 100]. In this manner, all the anonymity levels of the candidate VGHs were covered, which guaranteed a fair comparison. Finally, we calculated the usefulness of the anonymized datasets using task-independent data utility metrics (see Section IV-C).
B. Evaluated VGHs and Datasets Our testbed consisted of 252 VGHs. Those VGHs were created by perturbing the semantical content of an ideal VGH which was constructed by extracting the minimal taxonomy from WordNet (ontology widely used due to its broad coverage of concepts  [14] ) for our evaluation data. To obtain a varied range of VGH quality scores for our tests, we applied transformations to the ideal VGH, such as: mixing the leaf nodes and replacing the terms of the ancestors with another one selected from a list of candidate terms extracted from WordNet (that are within a semantic similarity boundary). All VGHs were constructed over the same set of leaf concepts. As evaluation data, we used the Insurance dataset  [2]  which contains personal information (in tabular format) that can be of interest to an insurance company for carrying out a risk assessment on potential clients.
From this dataset, we focused on the attribute of occupation, which has the highest diversity of values. To test the generality of our solution, we derived multiple datasets per VGH. This strategy allowed us to considerably diversify the range of evaluated scenarios and to show the impact that a dataset can have over the anonymization performance of the VGHs. To achieve this, for each VGH we computed the semantic loss that each branch (from leaf to root node) incurred in the VGH. We then identified the branches with the minimum and maximum semantic losses. Based on these branches, we generated 10 datasets (each composed of 1,100 records) per VGH. The name assigned to each dataset reflected the frequency distributions of the worst branch, the best branch, and the rest of the terms in the dataset.
C. Evaluation Criteria VGH Quality. The quality of the VGHs is expressed in terms of our d-GSL metric and the s-GSL metric. As both metrics leverage on measures of semantic similarity, in our experiments we used two widely-used path-based metrics: Wu and Palmer (WUP) and Leacock and Chodorow (LCH)  [14]. This strategy allowed us to prove the generality of the d-GSL with respect to the used semantic similarity metric. The WUP metric measures the depth of two given concepts in the taxonomy and the depth of their least common subsumer. The LCH metric measures the length of the shortest path between two concepts considering the depth of the taxonomy. Our implementation used the WS4J library  [3] , which relied on WordNet 3. 0 to compute the semantic similarity between two words. Data Utility. The level of usefulness that remained in the datasets after anonymization was measured using two taskindependent data utility metrics (as not knowing in advance the analysis task is an essential premise of PPDP): Semantic Sum of Squared Errors (SSE)  [8]  and Semantic Information Loss (SemILoss)  [13].
SSE measures the level of intra-group homogeneity in a group of anonymized records. SemILoss measures how semantically different the anonymized values are (on average) compared to the original ones. For both metrics, lower values are better, as it indicates a higher utility of the data. VGH Quality and Data Utility Correlation. To analyze the degree of correlation between the scores representing the quality of VGHs and the utility of the anonymized datasets, we calculated the Spearman's rank order correlation (r Spm ). It measures the strength of a monotonic (but not necessarily linearly related) relationship between paired data. r Spm can take values from -1 to +1; the closer the value is to ±1, the stronger the relationship. VGH Quality Categories. Based on our proposed rating scale, we created clusters based on both VGH quality scores, and compared them against the clusters created based on the data utility scores.
V. EXPERIMENTAL RESULTS Correlation Comparison. This analysis focused on assessing the capacity of the d-GSL and s-GSL metrics to capture (a priori) the effectiveness of the VGHs for anonymizing data. Figs. 3a  and 3b  show the correlation between the VGH quality metrics and the data utility per dataset. Fig. 3a  shows the correlation calculated using WUP to compute the VGH quality scores, and SSE to quantify the data utility. The d-GSL metric obtained stronger correlations (between 0. 91 and 0. 98) than the ones obtained for s-GSL (between 0. 64 and 0. 81). This means that d-GSL was more precise in determining the effectiveness of VGHs. Similar results are observed in Fig. 3b. It shows the correlation calculated using LCH to compute the VGH quality scores, and SemILoss to quantify the data utility.
Results show that d-GSL continued to exhibit a stronger correlation (between 0. 79 and 0. 90) than s-GSL (between 0. 73 and 0. 85). To complement the validation, the highest and the lowest correlations obtained for each quality metric are shown in Figs. 4  and 5  (for the 50w40b10r and 05w90b05r dataset scenarios, respectively). It can be observed how the relationship between the quality of the VGHs and the utility of the anonymized data exhibits a more linear and positive monotonic trend for the d-GSL metric. In conclusion, this analysis demonstrated that d-GSL is a better metric than s-GSL as it provides a more accurate representation of the VGHs' quality. Data Distribution Sensitivity Analysis. This analysis centered on assessing the influence of the input datasets (with different frequency distributions) on the effectiveness of the VGHs.
For the sake of brevity, we only present the comparison of ten sample VGHs (randomly picked from our testbed) that belong to the category 2 (according to their s-GSL score). Fig. 6a  shows the d-GSL and s-GSL scores of the sample VGHs. Since s-GSL is computed using a static scheme, there is one s-GSL score per VGH. For this reason, we use the s-GSL score as the baseline for our comparisons. In contrast, d-GSL uses a dynamic scheme, thus each VGH  has one d-GSL score per input dataset. It can be seen how the s-GSL scores remained within the range of a single quality category (i. e. , [0. 3 to 0. 39] corresponding to cat2), whereas the d-GSL scores fluctuated among four categories (i. e. , [0. 14 to 0. 67] corresponding to cat1 to cat4). Based on their quality scores (per dataset), the VGHs were ranked to identify the best VGH among all candidates.
That is, the VGH that would yield the lower information loss in the anonymized data. These rankings are shown in Fig. 7a. For example, based on d-GSL and the 90w05b05r input dataset, the VGH3 would be the best. Whereas, if the input dataset were 05w90b05r, the best would be VGH2. In summary, Figs. 6a and 7a represent the expected anonymization performance of the VGHs based on their d-GSL and s-GSL quality scores. Figs. 6b and 7b show the real performance of the sample VGHs. That is, the information loss that the datasets suffered after being anonymized with the VGHs. Fig. 6b  shows the data utility magnitudes (in terms of SSE), whereas Fig. 7b  shows the data utility rankings of the VGHs for each dataset scenario. To validate the accuracy of the quality metrics (d-GSL and s-GSL) for anticipating the utility of the anonymized data, we compared the expected rankings (shown in Fig.
7a ) with the real rankings (shown in Fig. 7b ). Overall, the d-GSL metric predicted more accurately the performance of the VGHs than s-GSL. For example in Fig. 7a , s-GSL ranked the VGH2 in 2nd place, however its performance degraded with some datasets (e. g. , 70w20b10r and 90w05b05r) until falling to 9th place. In contrast, this scenario is well-captured by d-GSL, as the d-GSL scores exhibit the same trends in both quality (Fig. 7a ) and utility plots (Fig. 7b ). We also compared the expected magnitude trends (Fig. 6a ) with the real magnitude trends (Fig. 6b ). In this case, we only validated that the trends remained, as the correlation between VGH quality and the utility offered by the VGHs is not linear (as previously discussed in our correlation results). In conclusion, this analysis demonstrated that there is no "best-fit-for-all" VGH for all datasets.
Instead, the best VGH can change depending on the input datasets. Hence, the utility of an anonymized dataset can be improved when the VGH evaluation is performed using a dynamic scheme (i. e. , d-GSL) instead of a static scheme (i. e. , s-GSL). VGH Quality Categories Comparison. This analysis assessed the empirical categories we proposed in Section III-E. Fig. 8  shows the average data utility of the VGHs grouped by quality categories based on their d-GSL score. For the sake of brevity, we only present the results for three datasets, as similar behaviors were obtained for the others. Within each dataset, the highest utility was obtained for the Cat1 as the VGHs belonging to this group reduced the information loss the most, in comparison to the rest of the categories. This demonstrates that a VGH that belongs to a lower category would be more effective to perform the anonymization of data than one that belongs to a higher category.
It can also be noticed that the information losses are generally lower We also grouped the VGHs in categories based on their s-GSL scores; their behaviors were similar to the ones exhibited by the d-GSL categories (shown in Fig. 8 ). To validate which of the two category groups was more accurate, we compared the groups ("clusters") based on the d-GSL and s-GSL scores, against the groups created based on the data utility scores. That is, we evaluated if the VGHs are "classified" in the same way when they are grouped by their quality than when they are grouped by the data utility. Then, we measured the agreement between the groups. Figs. 9a and 9b depict the level of agreement between the data utility In conclusion, this analysis proved that VGHs classified in lower categories are more likely to yield a higher data utility than those in higher categories.
Regarding the privacy implications of our approach, it does not affect the privacy goal set by users. In our experiments, the anonymized datasets improved their utility while still satisfying their corresponding kanonymity levels. Thus, the anonymized datasets kept the same protection and vulnerabilities of the privacy model used. In our experiments, we used k-anonymity. However, our approach is not tied to a specific privacy model or its associated goal. This decision was made not only to make d-GSL independent of the privacy model but also because it is not known a priori how many generalizations will be needed to satisfy a privacy goal (e. g. , k-anonymity level), thus our assumption is that it can be equally satisfied (same probability) at any level of the VGH.
VII. CONCLUSIONS AND FUTURE WORK In this paper, we proposed a dynamic VGH evaluation approach which exploits the frequency distribution of the input datasets. Our approach yields a score (d-GSL) which acts as predictor of the effectiveness of VGHs to perform the anonymization of data. The d-GSL score enables users to effectively compare multiple VGHs for a given domain and select the one that will better retain the usefulness of the original data. We also proposed a rating scale that will help users to classify the VGHs based on their quality (in terms of d-GSL) into categories. Each category includes an interval and a qualitative descriptor to offer practitioners an intuitive interpretation of the d-GSL. Our results demonstrated that the utility of anonymized datasets is improved (without sacrificing the privacy goal) when the selection of the best VGH is based on a dynamic scheme (i.
e. , d-GSL) instead of a static scheme (e. g. , s-GSL). Our results also demonstrated that d-GSL is more accurate than s-GSL, as it was better correlated with the utility of the datasets anonymized with the evaluated VGHs. Furthermore, we showed that VGHs classified in lower categories are more likely to yield a higher data utility than those in higher categories. In our future work, we intend to investigate which other aspects of PPDP might be suitable to extend our VGH evaluation solution. Likewise, we plan to broaden the validation of d-GSL applying task-specific utility metrics (e. g. , data mining). Another interesting idea is to consider the potential vulnerability of the VGHs to different privacy attacks. We plan to use this additional knowledge to develop a global cost evaluation function which can assess VGHs from different perspectives.
Digital humanities scholars strongly need a corpus exploration method that provides topics easier to interpret than standard LDA topic models. To move towards this goal, here we propose a combination of two techniques, called Entity Linking and Labeled LDA. Our method identifies in an ontology a series of descriptive labels for each document in a corpus. Then it generates a specific topic for each label. Having a direct relation between topics and labels makes interpretation easier; using an ontology as background knowledge limits label ambiguity. As our topics are described with a limited number of clear-cut labels, they promote interpretability and support the quantitative evaluation of the obtained results. We illustrate the potential of the approach by applying it to three datasets, namely the transcription of speeches from the European Parliament fifth mandate, the Enron Corpus and the Hillary Clinton Email Dataset.
This will offer to digital humanities scholars an overview of the performance (and the drawbacks) of the different components of the pipeline, and to natural language processing researchers a series of baselines that will guide them towards improving each component of the method proposed. 4 hile the presented solutions could be adopted in various DH tasks 5  , we consider the digital history community as the main target of this work. We illustrate the potential of this approach by applying it to detect the most relevant topics in three different datasets. The first dataset is the entire transcription of speeches from the European Parliament fifth mandate  (1999) (2000) (2001) (2002) (2003) (2004). This corpus (recently updated as Linked Open  Data (van Aggelen et al. 2016 )) has already been extensively adopted for computational political science research  (Hoyland and Godbout 2008; Proksch and Slapin 2010; Høyland et al.
2014)  and holds enormous potential for future political historians. The second dataset is the Enron Thread Corpus, which consists of e-mail threads from the Enron Corpus, a large database of over 600,000 emails generated by 158 employees of the Enron Corporation and acquired by the Federal Energy Regulatory Commission during its investigation after the company's collapse. In the last ten years the natural language processing community has already extensively studied this dataset, conducting network and content-based analyses. Our goal is to examine the quality of our approach on a highly technical and complex dataset of a specific kind of primary source (email) that will become more and more important in future studies in the history domain. Related to that, the third dataset is the Hillary Clinton Email Dataset, which shows a combination of features of the previous two datasets, as the Clinton emails are short correspondences mainly focused on political topics.
Dan  Cohen (2006) , more than a decade ago, anticipated the issue that future political historians will encounter when considering the large abundance of sources 6  that public administration will leave us in the next decades. Our study intends to be a very first experimental attempt to deal with one of these new collections of primary sources, and to provide historians of the digital age with a more fine-grained text exploration solution, compared to traditional LDA. The structure of the paper is as follows: We introduce a series of works related to our study in Section 2 and then describe our approach combining Entity Linking and Labeled LDA in Section 3. We next present the datasets (Section 4) and experiments for each component of our pipeline (Section 5). Finally, we discuss advantages and current limitations of our solution together with future work directions in Section 6.
One of the first papers that presents the task of topic labeling  (Mei, Shen, and Zhai 2007)  addresses the issue as an optimization problem involving minimizing the Kullback-Leibler divergence between word distributions, while maximizing the mutual information between a label and a topic model. A later approach  (Lau et al. 2011)  proposed adopting external knowledge to label topics. The paper that is closest to our topic labeling approach  (Hulpus et al. 2013 ) makes use of structured data from DBpedia 8. The authors hypothesise that words co-occurring in text likely refer to concepts that belong closely together in the DBpedia graph. Using graph centrality measures, they show that they are able to identify the concepts that best represent the topics. Entity Linking. The task of linking textual mentions to an entity 9 in a knowledge base is called entity linking or entity resolution  (Rao, McNamee, and Dredze 2013).
This is an information extraction task that involves being able to recognise named entities in text 7 A completely different strategy  (Chaney and Blei 2012)  to improve the interpretability of topic model results relies on the use of data visualisation techniques. 8 See http://wiki. dbpedia. org/. 9 Some authors  (Chang et al. 2016 ) distinguish between two tasks: First, Entity Linking, where mentions corresponding to named entities are considered. Second, Wikification, where mentions to any term present in Wikipedia (even if they are common nouns) are considered. In this paper we speak of Entity Linking for both cases: We are obviating this possible difference, since it is not essential for the core idea in this work, i. e. that by tagging LDA topics with terms from a knowledge base (in this case Wikipedia), we can improve the understandability and evaluability of LDA topics.
In these tasks, humans have to detect a word or a topic which does not fit with the rest  (Chang et al. 2009). Topic models can be evaluated per se, or their results can be evaluated against a gold standard. A way of doing it is to study the alignment between topic-results and classes in the dataset, or to use topic model outputs as features in a classification task and compare it with other solutions  (Nanni et al. 2016). LDA in Humanities and Social Science. Since the end of the 1990s, and in particular after Blei et al. published their paper on LDA  (Blei, Ng, and Jordan 2003) , the use of LDA topic models has been widespread among the natural language processing community. Interestingly, while applying LDA to humanities and social science datasets has been attempted several times during the last decade by the NLP community (see for example Yang et al.
e. , cherry-picked from the user based on data exploration) or automatically found on the basis of validation datasets or cross-validation techniques. We come back to the issue of selecting a value for k in Section 4. 3 paragraph "Label Selection and Ranking". 3. Entity-based Labeled LDA. In the final phase of our method, we train a vanilla Labeled LDA model  (Ramage et al. 2009 , Section 2) using the top k entity labels from the previous step to set values in the vector of document's labels (λ (d) ) used to constraint the topic prior (α (d) ). Given a model learned from training data, we can then apply inference techniques to test data (i. e. , previously unseen documents) to compute the posterior distribution θ (d) over topics per document, and accordingly rank entity labels in order of relevance for each document.
EuroParl EnronCorpus. The second dataset that was selected is the Enron Thread Corpus  (Jamison and Gurevych 2013). This is based on the Enron dataset, a popular email dataset first presented by  (Klimt and Yang 2004). However, unlike the original Enron dataset, the Enron Thread Corpus contains the emails grouped by email-thread, rather than as isolated emails. The dataset comprises over 600,000 emails generated by 158 employees of the Enron Corporation and acquired by the Federal Energy Regulatory Commission during its investigation after the company's collapse. The email thread reconstruction on the Enron dataset, a non-trivial problem in itself, was carried out by Jamison and Gurevych, who released the corpus publicly 14. The reason why we decided to use email threads instead of the isolated emails is that, for entity linking, precision improves when more context can be taken into account.
(Nanni and Ruiz Fabo 2016) , we represented each party as a single document, containing all speeches by its politicians. The solution chosen for this article will help us understanding the differences in topics in a more fine-grained fashion. 14 https://www. ukp. tu-darmstadt. de/data/text-similarity/email-disentanglement she had used private email servers for official communication. The data was first published on Wikileaks in March 2016 15  and later made publicly available by the US State Department 16. The Hillary Clinton Email Corpus represents a combination of the previous two datasets: On the one hand, it has similar characteristics to the Enron Thread Corpus, as it consists of short correspondences with limited context and highly technical content, such as for example header fields or web links.
Besides, the language in both datasets is sometimes relatively informal. On the other hand, the Clinton Email Dataset contains political topics and terms, and relates therefore to the very formal speeches that can be found in the EuroParl corpus. For these reasons, the third dataset is an interesting complement to the previous two. A SQLite database containing 7,945 emails is available online, hosted by the data science platform Kaggle 17. It consists of four tables, namely Emails, Persons, Aliases, and EmailReceivers. For the present study, the content of the Emails table was extracted with a custom SQL function, which exports given columns of the records of a given table to single text files, one file per entry.
ClintonCorpus By looking at the statistics on the three corpora (see Table  1 ), we notice that the vocabulary of the two email datasets is richer than EuroParl's vocabulary, as the email datasets are characterised by a large variety of informal expressions (e. g. "thx", "fyi", "tl:dr"). At the same time we can see that EuroParl documents are richer in terms of detected entities, given the fact that their subjects (political, economic and social concepts and actors) are largely represented in Wikipedia.