SlowGuess/NAACL_2021_OCR huggingface dataset:
For (1), we need to ensure that all instances of the terms in the evaluation set are identified in the corpus. We search for all morphological variants of the target terms and replace them by the unmarked singular form, by means of a manually created list. Furthermore, many of the target terms consist of two or more words, which should receive a single representation. As with the Wikipedia baseline, we search for all mentions of the target terms in the corpus and join all target terms from the ground truth that consist of multiple words (MWEs) by underscores to turn them into a single token. We did not handle MWEs that were not target terms, so no automatic MWE identification took place.
4.2 Hyperparameter Tuning
We propose a framework for fine-tuning models specifically designed for domain-specific experiments with small data. As the size of our ground truth is comparatively limited (for computational purposes), we do not want to 'waste' portions of it for fine-tuning. Instead, we use 'proxy' terms and a 'proxy' corpus to evaluate and compare models on an artificial task. We aim to select data representative of the target data (inspired by fine-tuning for low-resource languages, Søgaard, 2011).
Terms and Corpus. As target terms we select 20 technical terms from the legal domain. Similar to the philosophical target terms, many technical legal terms have distinct or more specific meanings in legal scholarship as opposed to generic corpora. To select a proxy corpus, we compare the contexts of the target terms to the contexts of the legal terms in four candidate corpora: the British Law Corpus (BLC), the Open Access Journal corpus, Wikipedia, and the British National Corpus (BNC). We compare the contexts in terms of easily computable metrics which characterize properties we expect to have an impact on training a DS model: average relative frequency of all the context words, their average polysemy (in terms of WordNet synsets (Fellbaum, 2010; Miller, 1995)), their entropy (based on unigram frequency), type count, token count, and type/token ratio. We rank each corpus by similarity to the Quine corpus on each metric. Out of the four corpora, Wikipedia and the BNC had an average rank of 1.8, while the BLC
was the least similar with 4. Out of the two equal choices in terms of means, the Wikipedia corpus was more similar to the Quine corpus in terms of variance, so we chose this corpus for extracting contexts of the legal proxy terms.
Task. As we do not have a conceptual ground truth for the legal terms, we rely on an artificial task. We approximate embedding quality in terms of consistency. Bloem et al. (2019) define a model as consistent if "its output does not vary when its input should not trigger variation (i.e. because it is sampled from the same text or domain)". We test whether a model creates consistent representations of a term when trained on only a subset of its contexts using artificial examples in the following way: Our artificial examples consist of contexts of two terms, which are merged to become one pseudo-term. Since the pseudo-term's contexts are split evenly between contexts of term1 and term2, its embedding is expected to be somewhere halfway between the embeddings of the two terms. We train separate vectors  $\vec{t_1}$  and  $\vec{t_2}$  for term1 and term2 on the basis of 100 occurrences of each, as well as  $\vec{t_p}$  for the pseudo-term term1_term2, based on 50 occurrences of each component term. We then compute the vector half-way between  $\vec{t_1}$  and  $\vec{t_2}$ . In a consistent model, the cosine similarity between this vector and  $\vec{t_p}$  should be high. In tuning, we perform a grid search and take the average of this metric computed over 10 random pairs of legal terms for each hyperparameter combination.
The results show that our models can learn vectors for artificial combined terms that are consistent with the middle point between the vectors of the two component terms in vectorial space. There is great variation for different hyperparameter sets. Average cosine similarities varied from 0.08 to 0.87 (N2V-additive) or 0.96 (W2V-random). We found that with the additive initialization, lower learning rates performed better, while with the random initialization, higher number of negative samples had the greatest impact on the consistency scores. For N2V, the lowest parameter decay
Our assumption on the expected position of the pseudoterm embedding oversimplifies the nature of DS models. The structure of semantic spaces and the distances between embeddings are still poorly understood, and it is not guaranteed that the embedding of a merged term should ideally be positioned in between its two constituent terms. However, we only assume that such a middle position is a good approximation when evaluating the consistency of a distributional semantic model using artificial data in tuning, not in testing our models.
The full parameter space can be found in our code repository.
rates performed best, probably because our artificial terms have more occurrences (50 and 100) than N2V was designed for (1-4). The initial high learning rate is a core feature of N2V, so we also include the best setting with a learning rate of 1 as an additional condition (N2V-additive-a1).
The tuned models were evaluated against the ground truth. This section presents multiple evaluation tasks and results to (1) explore different aspects of model quality and (2) stress-test our findings. In these tasks, we use the 74 terms from the conceptual network that were clustered by the experts.
Cluster similarity Our similarity task is defined as follows: Given a target term  $t_t$ , a term from the same cluster  $t_{sc}$  and a term from a different cluster  $t_{dc}$ , we test whether the target term  $t_t$  is closer to  $t_{sc}$  than to  $t_{dc}$ . If the cosine similarity between  $t_t$  and  $t_{sc}$  is higher than the cosine similarity between  $t_t$  and  $t_{dc}$ , it is counted as correct, else as incorrect. We carry out this comparison for all possible term combinations and report the percentage of correct outcomes. We report the proportion of target-terms that are classified in the correct cluster. We also show the proportion of target terms that are clustered incorrectly but plausibly given their relation (via reference or regimentation) to other clusters. We exclude all three terms that are out of vocabulary in any of the models, as a difference in target terms distorts the comparison. This way, we ensure that all models are evaluated on the same terms.
Table 1 shows that N2V outperforms the W2V models in most cases. The best performance is by N2V with additive initialization (standard N2V), pairing  $65.0\%$  correct according to the clusters, and  $72.6\%$  when additional relations between terms are also considered correct. The count-based SVD model performs similarly well. These are the only two models that beat the Wikipedia baseline. The best W2V model (W2V-random), pairs  $56.4\%$  correct according to the clusters, and  $64.3\%$  with additional relations. To evaluate the stability of our best result, we train 25 identically parameterized models as in Hellrich and Hahn's (2016b) reliability metric and obtain similarity scores in a range of  $64.04\% - 65.22\%$  (mean  $64.65\%$ , cf.  $64.95\%$  in testing) indicating high stability.
Model	Sim.	Oth. Rel	Dunn
N2V-additive	64.95%	7.68%	0.56
N2V-additive-a1	56.09%	8.38%	0.24
N2V-random	55.18%	8.39%	0.19
W2V-additive	52.00%	6.44%	0.12
W2V-random	56.44%	7.88%	0.08
SVD	65.19%	7.18%	0.35
wikipedia-W2V	59.58%	6.17%	0.17
Table 1: Outcome Cluster Similarity & Dunn Index
Dunn index The Dunn Index (DI) is a general metric of cluster quality and can be used to measure how well embeddings from the same cluster are clustered in semantic space (Huang et al., 2016). It is the ratio of the minimum inter-cluster distance to the maximum cluster size, and higher values indicate tighter clusters and better separation.
The DI results in table 1 confirm that N2V models outperform W2V models. N2V with additive initialization achieved the highest DI value 0.56, followed by the SVD model (0.35). We can compare this to Huang et al. (2016), who used DI to evaluate word embeddings in the medical domain, using six semantic clusters taken from an expert-defined controlled vocabulary of medical terms using far larger data sources (e.g. PubMed and Wikipedia). In their experiment with 800 terms (we have 99) and six clusters (comparable to our five), their DI scores were 0.16-0.20 for a bag-of-words baseline model, and 0.43 (PubMed) to 0.25 (Wikipedia) for W2V. This is comparable to our wikipedia-W2V condition which scored 0.17 on our clusters and data, indicating that our task is more difficult. In light of this, the 0.56 DI of our N2V-additive model seems quite good, while the 0.08 of W2V-random indicates poor cluster quality. But why did N2V cluster better than SVD while the two did not differ much in the cluster similarity task? DI is determined by both inter and intra distances. We found SVD has greater intra-cluster distances (0.51 inter, 1.47 intra) than N2V (0.55, 0.99) after normalization to unit vectors. This means clusters are more compact in the N2V model, potentially making the cluster similarity task easier.
K-means clustering and Centroids We clustered terms from each model into five clusters using the K-means clustering algorithm from scikit-learn (Pedregosa et al., 2011) and evaluated using three of its performance evaluation metrics: (i) ad
Figure 1: Similarity scores from Table 1 split by term frequency. Low  $\leq 49$  tokens, mid  $= 50 - 750$ , high  $\geq 750$ .
justed Rand index, (ii) adjusted mutual information and (iii) Fowlkes-Mallows index. Results for (i) and (ii) show scores close to zero for all models within the bounded range [-1,1], indicating results close to random. The best model is the SVD model ((i) 0.12, (ii) 0.17). On (iii), with scores in range [0,1], the best N2V model (0.48) outperforms the best SVD and W2V. Manual inspection of the clusters shows that in many cases the majority of terms is put into a single cluster and the other clusters have only a few terms in them.
We also applied a centroid-based approach to evaluate clustering. We calculated the mean of the normalized vectors for each cluster to determine its centroid. We then calculated the F-score by checking for each term whether its cosine distance was closer to its cluster centroid than to another. The best performing model is W2V-random (F-score: 0.10), followed by N2V-random (0.08). All other models perform approximately equally bad (0.04).
K-nearest neighbors In our final evaluation, we classify terms into clusters using K-nearest neighbors (KNNs). We compute the macro-averaged F1 score for each term using leave-one-out cross validation. For both  $k = 3$  and  $k = 1$ , the SVD model performs best with an F-score of respectively 0.45 and 0.42. For  $k = 3$ , the best N2V outperforms W2V, while for  $k = 1$  the best W2V outperforms N2V, scoring almost the same as the SVD model. Manual inspection shows that for all models most of the terms from any cluster are either classified as part of the language or the meta-linguistic cluster, which are the two largest clusters.
Frequency effects To further explore our findings, we performed the cluster similarity evalua
tion task again, but with the target terms split by frequency. This allows us to see how the quantity of training data affects the cluster similarity. We distinguish between low-frequency terms (1-49 occurrences,  $n = 22$ ), medium-frequency terms (50-750 occurrences,  $n = 55$ ) and high-frequency terms (750-6730 occurrences,  $n = 19$ , cutoffs were set to have a reasonable number of terms in the low and high frequency class). For reference, N2V was designed to train on 1-4 occurrences of a term, while for W2V, more is better. We expect additive initialization to outperform random initialization for low frequencies where an informed initialization can make up for a lack of training data.
Figure 1 shows that most models benefit from more data, but N2V clearly outperforms W2V in the low frequency condition, even with random initialization. Secondarily, models with additive initialization outperform their randomly initialized variants, possibly due to the transfer of domain-specific information for low-frequency terms noted by Bloem et al. (2019). N2V-add-a1 forms an exception, where the high learning rate may cause massive changes to the initial vector position after only a few training occurrences, performing worse than random. SVD does not pattern with N2V here, performing very poorly on the low frequency terms. This model performs best in the 50-750 occurrence range. The SVD models cannot benefit from additive initialization and should therefore be compared to the randomly initialized models.
In the high-frequency range, N2V again performs best, probably due to the low rates of parameter decay selected in the hyperparameter tuning. As expected, W2V performs quite well with more data in its standard random initialization condition. Unexpectedly, it performs quite poorly with the additive initialization. This might be an issue with our tuning process: as all our artificial terms had a relatively low frequency of 100, the tuning task may have selected a model that relies too much on the initialization, and learns poorly for the  $w2\nu$ -additive condition. This shows the importance of the tuning data resembling the target data closely.
The results show that, in general, N2V and SVD represent the ground truth clusters better than W2V on this type of data. Furthermore, we see that using N2V or SVD for smaller, domain-specific data outperforms a larger domain-general W2V model
trained on a large corpus. N2V is able to learn higher-quality embeddings than W2V from small texts, as it was designed to, and we confirm previous work showing that the same holds for count-based models to a limited extent.
Clustering methods (centroid and k-means) do not detect anything close to the clusters defined in the ground truth, whereas more fine-grained methods (cluster similarity and KNN) do yield results that are clearly above chance. The evaluation in terms of the Dunn index is also promising. Despite the overall low performance, we take this as an indication that the models group the terms with some systematicity. Furthermore, the rankings of the different models remain consistent across evaluations. Arriving at the same results through various methods can be seen as a fulfillment of Sommerauer and Fokkens's (2019) stress-test requirement.
Manual inspection of the clusters indicates that the imbalance in the (already very small) dataset is problematic for a K-nearest neighbors classifier, which assigned almost all words to the two biggest clusters. We expect that the same may hold for centroids and k-means. In hindsight, we could have controlled for this by extending the dataset beyond the terms in the Index of Word and Object. While this might have provided more accurate insights, we expect that most use-cases that work with small (or even tiny) data are most likely also working with similarly unbalanced data. Standard machine-learning techniques aiming to abstract over examples are most likely not able to pick up (potentially weak) signals based on just a few examples. We therefore consider fine-grained and example-based methods a more promising direction.
Overall, research on small data is still in an early phase. We see that models designed to work with tiny data outperform others on low-frequency terms, but yield only slightly better or comparative results when compared on mid- or high-frequency terms. It has to be considered that these models are overall very similar to the standardly used models. Future research should explore more balanced approaches which combine the strengths of both versions. For instance, by adjusting the settings of N2V based on the frequency of a target term.
From the perspective of a philosopher who may want to make use of DS models to support their work, the results we obtained in this study are not good enough yet. The minimum for exploratory work would be that the vast majority of the terms
is correctly clustered and all categories are exemplified. Currently most terms are placed in the two largest categories, which might even give high accuracy but still does not represent the data well. Thereby, exploration of the data with these models could give a wrong impression about how the terms relate to each other. For hypothesis testing, the required accuracy depends on the hypothesis being tested, but in principle it is possible when the model no longer makes clear mistakes (but it may not be able to always distinguish between conceptual and relational connections and still make errors on clear borderline cases). Unfortunately, this level of accuracy was not yet reached either. More broadly, as studying language change through diachronic word embeddings adds a layer of complexity beyond the synchronic word embeddings we investigated, we expect that diachronic word embeddings trained on small data sets will not be able to reflect actual conceptual change and thus directly support philosophical research at this stage.
We do, however, think that our results have laid the groundwork for using DS models for exploratory purposes. We should keep in mind that the conceptual network based on Word and Object calls for very fine-grained distinctions. While this may still be too challenging, we expect that exploring differences in terms used by different authors could be more realistic.
Oortwijn and Meyer's initial contribution to this research was funded by the Vrije Universiteit's Network Institute. Sommerauer and Fokkens were funded by the Netherlands Organization of Scientific Research (NWO) PGW.17.041 awarded to Pia Sommerauer and NWO VENI grant 275-89-029 awarded to Antske Fokkens. Bloem and Oortwijn were funded by NWO VICI grant 277-20-007 awarded to Arianna Betti. Oortwijn was also funded by NWO grant 314-99-117 awarded to Bettina Speckmann and by the Human(e) AI grant Small data, big challenges funded by the University of Amsterdam. We would like to thank Thijs Ossenkoppele and Arianna Betti for their evaluation of the ground truth. We furthermore thank the eIdeas research group and anonymous reviewers for feedback. All remaining errors are our own.
Fatemeh Torabi Asr, Jon Willits, and Michael Jones. 2016. Comparing predictive and co-occurrence based models of lexical semantics trained on child-directed speech. Cognitive Science.
Amir Bakarov. 2018. A survey of word embeddings evaluation methods. Computing Research Repository, arXiv:1801.09536. Version 1.
Arianna Betti, Martin Reynaert, Thijs Ossenkoppele, Yvette Oortwijn, Andrew Salway, and Jelke Bloem. 2020. Expert concept-modeling ground truth construction for word embeddings evaluation in concept-focused domains. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6690-6702.
Arianna Betti and Hein van den Berg. 2014. Modelling the history of ideas. British Journal for the History of Philosophy, 22(4):812-835.
Jelke Bloem, Antske Fokkens, and Aurélie Herbelot. 2019. Evaluating the consistency of word embeddings from small data. In Proceedings of Recent Advances in NLP, Varna, Bulgaria. INCOMA Ltd. Shoumen, Bulgaria.
Haim Dubossarsky, Daphna Weinshall, and Eitan Grossman. 2017. Outta control: Laws of semantic change and inherent biases in word representation models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1136-1145. Association for Computational Linguistics.
Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1606-1615. Association for Computational Linguistics.
Christiane Fellbaum. 2010. Wordnet. In Theory and Applications of Ontology: Computer Applications, pages 231-243. Springer.
Annapaola Ginammi, Jelke Bloem, Rob Koopman, Shenghui Wang, and Arianna Betti. 2020. Bolzano, Kant and the traditional theory of concepts - A computational investigation [in press]. In *The Dynamics of Science: Computational Frontiers in History and Philosophy of Science*, Pittsburgh. Pittsburgh University Press.
Mario Giulianelli, Marco Del Tredici, and Raquel Fernandez. 2020. Analysing lexical semantic change with contextualised word representations. arXiv preprint arXiv:2004.14118.
Anna Gladkova and Aleksandr Drozd. 2016. Intrinsic evaluations of word embeddings: What can we do
better? In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 36-42, Berlin, Germany. Association for Computational Linguistics.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016a. Cultural shift or linguistic drift? Comparing two computational measures of semantic change. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2116-2121. Association for Computational Linguistics.
William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2016b. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1501. Association for Computational Linguistics.
Johannes Hellrich and Udo Hahn. 2016a. An assessment of experimental protocols for tracing changes in word semantics relative to accuracy and reliability. In Proceedings of the 10th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 111-117. Association for Computational Linguistics.
Johannes Hellrich and Udo Hahn. 2016b. Bad company—neighborhoods in neural embedding spaces considered harmful. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2785–2796. The COLING 2016 Organizing Committee.
Aurelie Herbelot and Marco Baroni. 2017. High-risk learning: Acquiring new word vectors from tiny data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 304-309. Association for Computational Linguistics.
Aurelie Herbelot, Eva von Redecker, and Johanna Müller. 2012. Distributional techniques for philosophical enquiry. In Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 45-54. Association for Computational Linguistics.
Jian Huang, Keyang Xu, and VG Vinod Vydiswaran. 2016. Analyzing multiple medical corpora using word embedding. In 2016 IEEE International Conference on Healthcare Informatics (ICHI), pages 527-533. IEEE, IEEE.
Tom Kenter, Melvin Wevers, Pim Huijnen, and Maarten de Rijke. 2015. Ad hoc monitoring of vocabulary shifts over time. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM '15, pages 1191-1200, New York, NY, USA. Association for Computing Machinery.