SlowGuess/NAACL_2021_OCR huggingface dataset:
Challenging Distributional Models with a Conceptual Network of Philosophical Terms
Yvette Oortwijn\* Francois Meyer\*
Jelke Bloem\* Wei Zhou\*
Pia Sommerauer* Antske Fokkens*†
\* Institute for Logic, Language and Computation, University of Amsterdam
*Computational Linguistics & Text Mining Lab, Vrije Universiteit Amstersterdam
$^{\dagger}$ Department of Mathematics and Computer Science, Eindhoven University of Technology
{y.oortwijn,j.bloem}@uva.nl,{pia.sommerauer,a.s.fokkens}@vu.nl
francoisrmeyer@gmail.com, c sunset14330@gmail.com
Computational linguistic research on language change through distributional semantic (DS) models has inspired researchers from fields such as philosophy and literary studies, who use these methods for the exploration and comparison of comparatively small datasets traditionally analyzed by close reading. Research on methods for small data is still in early stages and it is not clear which methods achieve the best results. We investigate the possibilities and limitations of using distributional semantic models for analyzing philosophical data by means of a realistic use-case. We provide a ground truth for evaluation created by philosophy experts and a blueprint for using DS models in a sound methodological setup. We compare three methods for creating specialized models from small datasets. Though the models do not perform well enough to directly support philosophers yet, we find that models designed for small data yield promising directions for future work.
Philosophers apply text analysis to understand and delineate the precise meaning of concepts and the relations between them in a given text. This includes comparative research that investigates differences in how concepts are viewed in different philosophical schools or by individual philosophers. Betti and van den Berg (2014) point out that comparative research on concepts should follow a conceptual model approach. This approach states that we should not look at shifts of individual concepts in isolation, but rather address changes of a conceptual model as a whole. In such a system, relations between concepts are made explicit and comparative studies should identify how such relations change. Previous studies have shown that distributional methods can be used to support philosophical research by retrieving passages relevant to concepts in an author's work (e.g., the concept
of grounding within the work of Bernard Bolzano, van Wierst et al., 2016; Ginammi et al., 2020), but can we also generate distributional semantic (DS) models that are precise enough to identify differences in concepts?
This paper takes a first stab at addressing this question. In particular, we address the challenges involved in dealing with highly technical domain-specific terms that are defined in small corpora. As such, our use case has properties difficult for DS modeling, but typical for disciplines working with comparatively limited data. We compare domain-specific embeddings created using Word2Vec (Mikolov et al., 2013a,b) and a count-based SVD model (Levy et al., 2015) to those created by Nonce2Vec (Herbelot and Baroni, 2017), specifically designed for dealing with tiny data. Taking into account previous work criticizing the use of DS models for detecting sense-shift, we construct a data-specific ground truth, apply multiple evaluation metrics and verify whether results are stable across various random initializations. Our results confirm that SVD representations are superior to Word2Vec for small data and show that Nonce2Vec outperforms Word2Vec and, in most cases, SVD. However, results are currently not accurate enough for providing evidence or new insights to philosophers. Nevertheless, we are hopeful that better results can be obtained in the future by optimizing Nonce2Vec to deal with small rather than tiny data and by creating a bigger, more balanced ground truth.
The main contributions of this paper are (1) a new ground truth of philosophical concepts linked to a clean philosophical corpus that is particularly challenging to model; (2) a blueprint for investigating DS models for domain specific research; (3) a comparative study of different approaches of creating embeddings for highly domain-specific terms.
<sup>1</sup>The ground truth, details of results and code can be found on GitHub: https://github.com/YOortwijn/
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2511-2522 June 6-11, 2021. ©2021 Association for Computational Linguistics
After presenting related work, we describe the philosophical context: requirements, corpus and our ground truth. In Section 4, we outline how the DS models we use are created. We then present our evaluation and results in Section 5 which is followed by our conclusions and discussion.
In this section we cover (1) other work related to distributional semantics (DS) for specific concepts and conceptual change (2) critical reflection on evaluation and the methodology involved and (3) work on small datasets and identification of domain specific meaning.
2.1 DS for Concepts and Conceptual Change
A well-known application of DS is the use of diachronic word embeddings to track and analyze changes in the meaning of words over periods of time (Kim et al., 2014; Kulkarni et al., 2015; Mitra et al., 2015; Hamilton et al., 2016b,a; Kenter et al., 2015; Tahmasebi and Risse, 2017; Montariol and Allauzen, 2019; Giulianelli et al., 2020, e.g.). Most of these approaches study what is called sense-shift, which is the change in (dominant) sense of a specific word by comparing the word's meaning representations in different time periods (Kutuzov et al., 2018). DS methods have also been used to study concepts related to gender and intersectionality (Herbelot et al., 2012), studying cultural stereotypes (Lewis and Lupyan, 2019) or harm-related concepts in psychological research papers (Vylomova et al., 2019).
Wevers and Koolen (2020) survey three ways in which distributional semantic representations can help trace concept change. However, none of these methods requires historians of ideas to fix initial and testable hypotheses on the meaning of concepts as Sommerauer and Fokkens (2019) recommend on the basis of Betti and van den Berg (2014). Betti and van den Berg argue that concepts are not isolated, but part of conceptual models. Sommerauer and Fokkens (2019) show that translating conceptual models to words representing them is one of the challenges involved in using DS models for studying conceptual change. They ground their conceptual model of 'Racism' in literature by sociologists, anthropologists and historians, but argue that domain experts would ideally be involved directly, as is done in the current pa
per. Betti et al. (2020) introduce a concept-focused ground truth designed by domain experts, QuiNE-GT, where paragraphs of philosophical text are annotated in terms of their relation to a conceptual model of the concept of naturalized epistemology in Quine's works. We also make use of conceptual modeling methodology to build a ground truth, but our task is to extract knowledge on target term relations rather than to perform an information retrieval task searching for paragraphs relevant to a research question. While QuiNE-GT contains exhaustive lists of words pertaining to a particular research question, we aim for broader coverage of different terms used by Quine and their relations.
2.2 Methodological Challenges
An interdisciplinary collaboration with domain experts can lead to hypotheses about shifts or nearest neighbors of specific terms, which can be tested by methods also used for detecting sense shift. These methods are not without challenges. The meaning representations are affected by random factors such as initialization and order of example (Hellrich and Hahn, 2016a) and frequency effects (Dubossarsky et al., 2017). A major obstacle in addressing these critical points is the lack of high quality evaluation sets (Tahmasebi et al., 2018; Kutuzov et al., 2018) and a tendency to use a single evaluation metric (Gladkova and Drozd, 2016) while each metric has downsides (Bakarov, 2018).
Evaluations on small sets of hand-picked examples that exhibit strong sense-shift (e.g. Hamilton et al. (2016a)) leave it unclear whether they are also suitable for making new discoveries or exploring data. van Aggelen et al. (2019) introduce a large-scale evaluation set derived from a thesaurus and show that performance of distributional methods is much lower on this more challenging set.
These critical findings stress the need for methodologies that allow us to establish the quality of embeddings and to tell the difference between a stable, reliable finding and an artefact of the method. Dubossarsky et al. (2017) propose the use of shuffled and synchronic corpora for verification. Rosenfeld and Erk (2018) use synthetic words that consist of two real words merged together that result in a shift of these words' senses for evaluation. Sommerauer and Fokkens (2019) recommend stress-testing through control words (that should not change) and by comparing results on multiple models. We supplement these proposals for
Challenging_DMs.
diachronic models by providing methods that can be used as a strict test of synchronic model quality, independently of measuring change (so that frequency effects are not a risk). To ground these methods, we introduce a novel, high quality ground truth containing fine-grained meaning distinctions in the philosophical domain. We stress-test our findings by applying multiple evaluation metrics and control for random factors by initializing our models multiple times.
2.3 Dealing with Small Datasets
In addition to the challenges outlined above, we are faced with the issue that domain-specific corpora are typically small, i.e. up to a few million tokens rather than web scale. Learning embeddings from small corpora is not an easy task, where SVD models outperform Word2Vec (W2V) (Sahlgren and Lenci, 2016, on 1M words, Asr et al., 2016, on 8M words), and learning them for rare words presents further difficulty (Luong et al., 2013). Nonce2Vec (N2V) (Herbelot and Baroni, 2017) addresses this issue through 'high-risk' incremental learning with an initially high but decaying learning rate, allowing them to learn embeddings from single sentences (called tiny data). Faruqui et al. (2015) incorporate ontological information from lexical-semantic databases as a postprocessing step, which can be done when training data is sparse. However, when working in a specific domain, such as the texts of a particular philosopher, words may have different and specific uses, and general-purpose evaluation resources or training data do not always reflect these meanings (Betti et al., 2020). Bloem et al. (2019) confirm the domain-specific character of philosophical writings showing that two vectors for the same word, one trained on Wikipedia and one trained on the works of a specific philosopher, can have low similarity, especially for high-frequency terms. Shoemark et al. (2019) find that the top ranked words are domain-specific to the Twitter data they used. Wohlgannt et al. (2019) evaluate DS models trained on two fantasy book series by having domain experts manually compile evaluation datasets addressing the relevant word senses, incorporating domain knowledge in both training and evaluation. Roy et al. (2019) propose incorporating text annotation of in-domain vocabulary and semantic relations into the word embeddings to improve the quality of domain-specific word embeddings learned from relatively small
data sets.
In this paper, we investigate how different approaches for learning embeddings deal with the domain specific concepts we are dealing with. We compare Herbelot and Baroni's N2V to continuing training with W2V and to directly creating SVD models on our corpus.
3 Philosophical Background and Data
The goal of this section is to provide some insights into the process of interpreting philosophical texts and the use case for our experiments. We briefly describe the process and challenges of close-reading and how it could be supported by DS models. Then we present a corpus of philosophical texts and a ground truth for philosophy.
3.1 Philosophical Questions
Many philosophical research questions focus on the interpretation and comparison of philosophical views expressed in writing. These questions revolve around specific concepts and how they are defined and viewed by different philosophers. Often, different philosophers use the same terms to describe different concepts. For example, Quine sees reference as a relation between a singular term and a physical object, where a physical object is not part of reality, but of our ontology (Quine, 1960). This is opposed to many other philosophers, who take what we refer to and what we receive stimulation from as the same thing, i.e., physical objects in reality.
To make solid comparisons between views, it is necessary to determine which concepts are closely related to each other, or which concept pairs stand in similar relations to others. To do this, philosophical experts practice close-reading. The interpretation of only a single passage requires close-reading and expertise of not only the work the passage is in, but often also other works by the same author or even other authors. Conclusions are often drawn on a small subset of the relevant available data. It almost always requires making a selection of sources to consider and thus allows for cherrypicking data. The use of computational linguistic methods, instead, could make it possible to consider all available data as a basis for evidence, and thereby prevent biased source selection.
Differences in the interpretation of a term in different authors' work can be understood as a difference in its relations to other terms. A difference
in how terms can be clustered together would then show a difference in the conceptual relations these terms have to each other. Computational methods that can capture this aspect of meaning can be applied in various stages of philosophical research.
Exploration. In the first stages of research, a philosopher might have a single or a few passages or terms that should be interpreted. At this point, they may want a rough overview of other passages or terms relevant to the one(s) under consideration. DS models may help the researcher to find relevant passages without any of the search terms they may use in key term search. These passages can provide input for more directed searches and be a start for a traditional research path with close-reading of the identified passages. The recall of the method for this application need not be very high: as long as the researcher identifies some new relevant passages without being overloaded with irrelevant ones and the selection is not biased towards a specific interpretation, DS models enrich the philosopher's research.
Testing Hypotheses about the Text. When a researcher already has some competing hypotheses for interpretation based on close-reading of some works or passages, or based on secondary literature, DS models can help to compile evidence for both hypotheses and compare the results. If there are multiple possible interpretations of a term, a DS model could provide insight into which terms are most closely related to this term, giving evidence for the correct interpretation. If the outcome of such a comparison is to be used as direct evidence, it is essential that the DS model is highly accurate and a methodology is applied to distinguish veritable observations from noise. A researcher may however also use these results in a more surveying manner. In this case, more accuracy is needed than in the case of identifying passages, but a certain amount of error is acceptable. In this paper, we aim to investigate the level of accuracy we can obtain on philosophical text with either of these applications in mind (surveying hypotheses or providing evidence for a hypothesis).
3.2 Quine in Context Corpus
We make use of a large corpus that comprises the virtually complete oeuvre in English of Willard V. O. Quine, the QUINE corpus (Version 0.5, Betti et al., 2020),<sup>2</sup> for creating our DS models. The cor
pus includes texts on various topics, from formula-heavy logic works to philosophy of language. Version 0.5 of this corpus consisting of 228 books and articles by Quine, containing 2,150,356 word tokens and 38,791 word types. It is a high quality corpus where scanned page images were OCR-processed and corrected manually.
3.3 A Ground Truth in Philosophy
Establishing a ground truth for philosophical concepts is not trivial (see e.g. van den Berg et al. (2018), Betti et al. (2020)). We address this by building on the methods described by Betti and van den Berg (2014) for building conceptual models. Instead of trying to understand the meaning of a term in isolation, we focus on the interrelations of terms.
We base our ground truth on Quine's Word and Object (Quine, 1960), which encompasses many of the terms and themes that Quine discusses throughout the rest of his work.3. We obtain this book's most important terminology from its index. The philosophical expert on our team established a conceptual network representing the termclusters and relations. The expert categorized each word as either belonging to one of five clusters (LANGUAGE, ONTOLOGY, REALITY, MIND, META-LINGUISTIC) or as a relational term (i.e. part of either the reference or regimentation relation that connects (parts) of clusters to each other). Any two terms in the same cluster can be seen as conceptually related (e.g. noun and verb are conceptually related since they are both linguistic items and are therefore both in the LANGUAGE cluster). The reference relation connects terms from the language and ontology cluster, i.e. elements of language refer to elements of the ontology. Regimentation connects parts of the language and meta-linguistic cluster. So the terms that are clustered together are semantically similar to each other, while the relational terms are related terms that are not necessarily semantically similar. Our conceptual network contains 74 clustered terms and 43 relational terms (overlapping the 74). The conceptual network was checked independently by two other philosophers specialized in
that can show they own the original works. Replication instructions are available here: https://github.com/YOortwijn/QuiNE-ground-truth
<sup>3</sup>A more detailed and accessible explanation of the conceptual network, including further motivation for the categorization of terms can be found at https://github.com/YOortwijn/Challenging_DMs
2The corpus was derived from copyrighted works by Betti et al. (2020). The corpus is available to researchers
Quine. There was a  $100\%$  consensus among the experts on the clustering of the 74 terms and relations of the 43 terms. Since these terms are core terms in the work of Quine for which most experts agree on their coarse interpretation, high consensus was expected. However, differences in interpretations and disagreement between experts is more likely upon more fine-grained analysis and even though consensus was expected, a fourth consulted expert may still disagree with the interpretation.
Even high-quality DS models have certain limitations when it comes to representing words accurately due to their architecture (e.g. expressing very fine-grained differences and polysemy). We identified the following potential challenges prior to examining vector representations from our DS models: First, terms that are related by the reference relation might be closer to each other than to other terms in their respective clusters. For instance, a singular term (cluster LANGUAGE) refers to a physical object (cluster ONTOLOGY). Therefore, they might be closer to each other than to other terms in their clusters (relative clause and class, respectively). Second, the LANGUAGE and METALINGUISTIC clusters are relatively similar. While they can be distinguished in Word and Object by their relation to ontology and regimentation but this is not necessarily the case for all of Quine's works. Examples of terms that could be misplaced due to this are article and noun. Third, there are terms that are comparatively distinct from the other terms in their cluster (but nevertheless clear members of the cluster), such as phoneme in the REALITY cluster. Fourth, the clusters contain some polysemous terms and terms that can be used in both a technical and a non-technical way within Quine's works, e.g., name, particular, context, form. Finally, some terms, such as prelinguistic quality space, might have an extremely low number of occurrences.
Based on these observations, we divide our ground truth in the following subsets: (1) terms that should be assigned to the correct cluster and (2) terms that could be assigned to a wrong but also plausible cluster given the corpus and the first two potential challenges by way of the reference or regimentation relation. The focus will be on (1), but (2) will be used in the first task.
4 Training DS Models
Bloem et al. (2019) noted that reasonable embed
dings for some philosophical terms can be learned from Wikipedia-data. As a baseline, we include a model trained exclusively on a 2019 Wikipedia dump using default Word2Vec (W2V), wikipedia-W2V. Multi-word target terms were linked by underscores to have a single vector per target term and 85 of the 99 target terms are in the vocabulary of this model. We test an SVD count-based model, using the PPMI-SVD approach from Levy et al. (2015) and two predictive approaches for creating our DS models: W2V (Mikolov et al., 2013a,b) in its Gensim implementation (Rehurek and Sojka, 2010), as well as Nonce2Vec (Herbelot and Baroni, 2017, N2V) adapted for small, in-domain data situations (Bloem et al., 2019). To learn an embedding for a specific term, N2V uses the sentences in which this term occurs to map it into a previously learned general-domain semantic background space trained on Wikipedia data. This is done by initializing the vector for the target term to the sum of the background space vectors of words in the in-domain context sentence from the Quine corpus, following Lazaridou et al. (2017). Training then takes place with an initial high learning rate and parameter decay, while the background space is frozen and only the target term is learned. Using W2V, we learn embeddings for specific terms by training only on the in-domain context sentences of our target terms. We test two initialization methods: random initialization, and using the additive model of N2V. We also modify N2V to have a random initialization condition for comparison, giving us four conditions: W2V-random, N2V-random, W2V-additive and N2V-additive (the N2V default).
We carry out various preprocessing steps to ensure that we (1) find the maximum of target term mentions and (2) regularize the contexts so we can exploit the full potential of the small corpus. In part, we make use of the preprocessing steps already performed on the QUINE corpus (v0.5), which was sentence-split and tokenized using UCTO $^{5}$  and lemmatized using Spacy $^{6}$  using its core model for English. $^{7}$  The QUINE corpus features a rather high number of mathematical expressions. Rather than treating them as unique expressions, they were nor
4We used the same Wikipedia dump for wikipedia-W2V.
https://languageemachines.github.io/ucto/
7Spacy English core model: en_core_web_sm
malized by replacing them by the symbol  $XfZ$  for formulas, and  $XsZ$  for symbols. We assume that the specific expressions do not add to the distributional information.