SlowGuess/ACL_2020_OCR huggingface dataset:
Ryohei Sasano
sasano@i.nagoya-u.ac.jp
Anna Korhonen
Language Technology Lab
alk23@cam.ac.uk
This paper presents an investigation on the distribution of word vectors belonging to a certain word class in a pre-trained word vector space. To this end, we made several assumptions about the distribution, modeled the distribution accordingly, and validated each assumption by comparing the goodness of each model. Specifically, we considered two types of word classes – the semantic class of direct objects of a verb and the semantic class in a thesaurus – and tried to build models that properly estimate how likely it is that a word in the vector space is a member of a given word class. Our results on selectional preference and WordNet datasets show that the centroid-based model will fail to achieve good enough performance, the geometry of the distribution and the existence of subgroups will have limited impact, and also the negative instances need to be considered for adequate modeling of the distribution. We further investigated the relationship between the scores calculated by each model and the degree of membership and found that discriminative learning-based models are best in finding the boundaries of a class, while models based on the offset between positive and negative instances perform best in determining the degree of membership.
Several studies have been successful in representing the meaning of a word with a vector in a continuous vector space (e.g., Mikolov et al. 2013a; Pennington et al. 2014). These representations are useful for a range of natural language processing (NLP) tasks. The interpretation and geometry of the word embeddings have also attracted attention (e.g., Kim and de Marneffte 2013; Mimno and Thompson 2017). However, little attention has been paid to the distribution of words belonging to a certain word class in a word vector space, though
Figure 1: 2D t-SNE projection of GloVe vectors. The 200 plus symbols  $(+)$  represent the word vectors that can be a direct object of the verb play (positive instances) and the 1000 squares  $(\square)$  represent other word vectors (negative instances).
empirical analysis of such a distribution provides a better understanding of word vector spaces and insight into algorithmic choices for several NLP tasks, including selectional preference acquisition and entity set expansion.
Figure 1 shows a 2D projection of word embeddings. We extracted 200 words that can be a direct object of the verb play (positive instances) and 1000 other words (negative instances) and projected their GloVe vectors (Pennington et al., 2014) into two dimensions using t-distributed Stochastic Neighbor Embedding (t-SNE) (van der Maaten and Hinton, 2008). The plus symbols  $(+)$  represent the positive instances, and the squares  $(\bullet)$  represent the negative instances. This figure shows that the positive instances tend to be densely distributed around their centroid but they are not evenly distributed near the centroid in the 2D spaces. In this study, we aimed to understand how these positive instances are distributed in the pre-trained word vector spaces built by three representative general-purpose models: CBOw, skip-gram (Mikolov et al.,
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3657-3666 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
2013a), and GloVe.
More specifically, we attempted to determine the following: whether or not a simple centroid-based approach can provide a reasonably good model, whether or not considering the geometry of the distribution and the existence of subgroups is useful for modeling the distribution, and whether or not considering the negative instances is essential to achieve adequate modeling. To this end, we first tackled properly modeling the vector distribution to distinguish a possible member of a word class from others when a subset of the class members is given. Note that although various approaches have been proposed to improve word vectors by taking knowledge related to word classes into account (Faruqui et al., 2015; Rothe and Schütze, 2015; Mrkšić et al., 2017), we explored ways to model the distribution of word vectors rather than attempting to improve the word vectors themselves.
We started with a centroid-based model, which is a simple but widely used way of representing a set of word vectors (e.g., Baroni et al. 2014; Woodsend and Lapata 2015) and assumes that how likely a word in the vector space is a member of a word class is proportional to the proximity to the centroid vectors of the class members. We then explored models that take the geometry of the distribution and the existence of subgroups into account. Here, we made two assumptions: vectors of words belonging to a certain word class are distributed with different variances depending on the direction, and most word sets will consist of several subgroups. We then explored the models that also consider negative instances. We assumed that the vectors of the words that do not belong to the target word class can be essential clues to distinguish a possible member of a word class from others. Specifically, we explored a model based on the offset between positive and negative instances and discriminative learning-based models to investigate the impact of negative instances.
Furthermore, we investigated the relationship between the scores calculated by each model and the degree of membership using the Rosch (1975) dataset. The dataset contains typicality ratings for some instances of a category. Through experiments, we found that discriminative learning-based models perform better at distinguishing a possible member of a word class from others, while the offset-based model achieves higher correlations with the degree of membership.
The interpretation and geometry of word embeddings have attracted attention. Mimno and Thompson (2017) reported that vector positions trained with skip-gram negative sampling (SGNS) do not span the possible space uniformly but occupy a narrow cone instead. Mikolov et al. (2013b) showed that constant vector offsets of word pairs can represent linguistic regularities. Kim and de Marneffe (2013) demonstrated that vector offsets can be used to derive a scalar relationship amongst adjectives. Yaghoobzadeh and Schütze (2016) performed an analysis of subspaces in word embedding. These analyses suggest that a certain direction or subspace in the word vector space represents an aspect of the words and the possibility that a word class is distributed with different variances depending on the direction in the vector space.
While we investigated ways to model the distribution of a set of words in pre-trained word vector spaces to validate several assumptions about the distribution, various approaches have been proposed to improve word embeddings by considering knowledge related to word classes into account. For example, Faruqui et al. (2015) proposed a method of refining vector representations using relational information from semantic lexicons by encouraging linked words to have similar vector representations. Mrkšić et al. (2017) proposed an algorithm for improving the semantic quality of word vectors by injecting constraints extracted from lexical resources. Glavaš and Vulić (2018) use the linguistic constraints as training examples to learn an explicit specialization function with deep neural network architecture.
There are also several studies that expand the method for acquiring a word vector to consider the uncertainty of a word meaning via Gaussian models (Vilnis and McCallum, 2015; Athiwaratkun and Wilson, 2017) and word polysemy by introducing several vectors for each word (Chen et al., 2014; Neelakantan et al., 2014; Tian et al., 2014; Athiwaratkun et al., 2018). In this study, we only considered a vector for representing each word, but inspired by these studies, we explored models that can consider the geometry of the distribution and the existence of subgroups.
The problem we tackled is similar to a selectional preference acquisition task. There have been a number of studies on selectional preference acquisition. Resnik (1996) presented an information
(b) GM
(c) GMM
(d) OffSet
Figure 2: Examples of distributions modeled by (a) CENT, (b) GM, (c) GMM, and (d) OffSet.
theoretic approach that inferred selectional preferences based on the WordNet hypernym hierarchy. Erk et al. (2010) described a method that uses corpus-driven distributional similarity metrics for selectional preference induction. Van de Cruys (2014) investigated the use of neural networks for selectional preference acquisition.
An entity set expansion task (Pantel et al., 2009) is also similar to our problem and has been well studied. For example, Sadamitsu et al. (2011) disambiguated entity word senses and alleviated semantic drift by extracting topic information from LDA for entity set expansion. Zhang et al. (2016) proposed a joint model for entity set expansion and attribute extraction. In this study, we seek to understand how these vectors are distributed in the pre-trained word vector space without using contextual or lexical information. A comparison with the state-of-the-art models for selectional preference induction and entity set expansion is beyond the scope of this work.
First, let us introduce the notation.  $W_{c}$  is a subset of words that belong to the target word class  $c$ .  $W_{o}$  is a subset of words that do not belong to the word class.  $w_{t}$  is a target word that can be a member of the word class  $c$  but is not included in  $W_{c}$ .  $v_{w} \in V_{w}$  is a pre-trained vector for word  $w$ . We normalize all the word vectors to unit length. Note that we select the words in  $W_{o}$  to share the same grammatical category as the words in  $W_{c}$ .
Our objective is to distinguish the word  $w_{t}$  from the words in  $W_{o}$ , given  $W_{c}$  and  $V_{w}$ . More specifically, we aim to find a scoring function  $f(w, W_{c})$  that assigns a higher score to  $w_{t}$  and lower scores to the words in  $W_{o}$ . For example, suppose  $c$  is a class of words that can be a direct object of the verb play;  $W_{c}, W_{o}$  and  $w_{t}$  will be as follows:  $W_{c} = \{ \text{role}, \text{part}, \text{game}, \text{golf}, \text{tennis} \}$ ,  $W_{o} = \{ \text{school}, \text{apple}, \text{milk}, \text{arch}, \text{idea} \}$ , and  $w_{t} = \text{basketball}$ . Our objective is to find a scoring function that assigns a higher score to basketball than to school, apple, milk, arch, and idea.
We will start with a centroid-based model (CENT) that measures the score between a word  $w$  and a word set  $W_{c}$  by calculating the cosine similarity between the word vector and the centroid vector of the word vectors in the word set (Figure 2-(a)). The scoring function can be written as:
$$
\mathrm {f} _ {\text {C E N T}} (w, W _ {c}) = \cos \left(v _ {w}, \frac {1}{| W _ {c} |} \sum_ {w _ {c} \in W _ {c}} v _ {w _ {c}}\right). \tag {1}
$$
CENT provides a reasonable baseline, but it does not take the geometry of the distribution of the word vectors into account. Therefore, we introduce a simple Gaussian model (GM) to represent the distribution of word vectors belonging to a word class  $c$  (Figure 2-(b)). The scoring function is as follows:
$$
\mathrm {f} _ {\mathrm {G M}} (w, W _ {c}) = \mathcal {N} (v _ {w} | \mu , \Sigma), \tag {2}
$$
<sup>1</sup>We also performed experiments with original vectors but obtain similar results in most cases.
where mean  $\mu$  and covariance matrix  $\Sigma$  are estimated from  $\{v_{w_c}|w_c\in W_c\}$ . We select the constraint on covariance matrices for Gaussian distribution from  $\{\mathrm{spherical, diagonal, full}\}$  by performing cross-validation on  $W_{c}$ . GM is identical with CENT when the covariance matrix is an identity matrix.
Next, we introduce a Gaussian mixture model (GMM) to take the existence of subgroups in a word class  $c$  into account (Figure 2-(c)). The scoring function can be written as:
$$
\mathrm {f} _ {\mathrm {G M M}} (w, W _ {c}) = \sum_ {k = 1} ^ {K} \pi_ {k} \mathcal {N} \left(v _ {w} \mid \mu_ {k}, \Sigma_ {k}\right), \tag {3}
$$
where weights  $\pi_{k}$ , means  $\mu_{k}$ , and covariance matrices  $\Sigma_{k}$  are estimated from  $\{v_{w_c}|w_c\in W_c\}$ . We select the number of components of a Gaussian mixture  $K$  from  $\{1,2,\ldots ,10\}$  and the constraint on covariance matrices from {spherical, diagonal, full} by performing cross-validation on  $W_{c}$ . GMM can be considered an extension of CENT because it is identical to the CENT when  $K$  is 1 and the covariance matrix is an identity matrix.
Furthermore, we will consider another extension of CENT that only considers the existence of subgroups. Since all word vectors are normalized to unit length,  $\mathrm{f_{CENT}}(w,W_c)$  can also be written as:
$$
\mathrm {f} _ {\text {C E N T}} \left(w, W _ {c}\right) = \frac {\alpha_ {W _ {c}}}{\left| W _ {c} \right|} \sum_ {w _ {c} \in W _ {c}} \cos \left(v _ {w}, v _ {w _ {c}}\right), \tag {4}
$$
where  $\alpha_{W_c}$  is a normalization term depending only on  $W_{c}$  and thus does not affect the ranking. That is, we can consider that CENT takes the average of the cosine similarities between a word vector  $v_{w}$  and all word vectors in the given word set  $W_{c}$ . If the words in the word set consist of several subgroups, it would be more plausible to consider only the top-  $k$  most similar words for scoring. Accordingly, we introduce the  $k$ -nearest neighbor model ( $k\mathbf{NN}$ ), which takes the average of only the top  $k$  similar vectors. The scoring function can be written as:
$$
\mathrm {f} _ {k \mathrm {N N}} (w, W _ {c}) = \frac {1}{k} \sum_ {w _ {c} \in k \mathrm {N N} _ {w} \left(W _ {c}\right)} \cos \left(v _ {w}, v _ {w _ {c}}\right), \tag {5}
$$
where  $k\mathrm{NN}_w(W_c)$  is a function returning a set of words in  $W_{c}$  that take the top-  $k$  highest cosine similarities against the word  $w$ . The number of  $k$  is selected from  $\{1,2,2^2,\dots ,|W_c|\}$  by performing cross-validation on  $W_{c}$ . kNN is identical to CENT when  $|W_{c}|$  is selected as  $k$ .
As the last model without negative instances, we adopt a one-class support vector machine (SVM) (Scholkopf et al., 2001)-based model (1-SVM) to clarify the importance of the negative instances. We select the kernel from {linear, cubic polynomial, RBF} and tune the parameter  $nu \in \{0.05, 0.10, \dots, 0.50\}$  by performing cross-validation. Note that models without negative instances learn a decision function for outlier detection: classifying new data as similar or different to the given positive instances.
Next, we explore models that also leverage negative instances. Here, we introduce a word set  $W_{n}$  as negative instances, where  $W_{n}$  consists of words that are not included in either  $W_{c}$  or  $W_{o}$ . We select the words in  $W_{n}$  to share the same grammatical category as the words in  $W_{c}$  as well as  $W_{o}$ . Both  $W_{o}$  and  $W_{n}$  consist of words that are not included in  $W_{c}$ , but their roles are different. While words in  $W_{o}$  are used as negative instances in the estimation, words in  $W_{n}$  are used as negative instances for modeling the word-class distribution.
As the first model with negative instances, we introduce a model based on the offset between positive and negative instances (Offset). This model is inspired by the Kim and de Marneffe (2013)'s work, which demonstrates that vector offsets can be used to derive adjectival scales. We assume that the vector offset between the centroid of the positive instances and that of the negative instances represents the degree of membership in the vector space (Figure 2-(d)). The scoring function of OffSet is as follows:
$$
\mathrm {f} _ {\text {O f f S e t}} (w, W _ {c}, W _ {n}) = \cos \left(v _ {w}, \frac {v _ {\Sigma c}}{\left| v _ {\Sigma c} \right|} - \frac {v _ {\Sigma n}}{\left| v _ {\Sigma n} \right|}\right), \tag {6}
$$
where  $v_{\Sigma c} = \sum_{w_c \in W_c} v_{w_c}, \quad v_{\Sigma n} = \sum_{w_n \in W_n} v_{w_n}$ .
Now let us move on to discriminative learning-based models. In this study, we chose a support vector machine with a linear kernel  $(\mathbf{SVM}_L)$  or a radial basis function (RBF) kernel  $(\mathbf{SVM}_R)$ . We only used word vectors as the input of these models and regard the decision function as the scoring function. We tuned the parameter  $C \in \{0.1, 0.2, 0.5, 1, 2, 5, 10\}$  and class weight for positive instances  $P \in \{1, 2, 4, 8\}$  for  $\mathbf{SVM}_L$  and the parameter  $C \in \{0.2, 0.5, 1, 2, 5\}$ ,  $\gamma \in \{0.2, 0.5, 1, 2\}$ , and class weight for positive instances  $P \in \{1, 2, 4, 8\}$  for  $\mathbf{SVM}_R$  by performing cross-validation on  $W_c$  and  $W_n$ . Note that
we wanted to determine the usefulness of negative instances in modeling the distribution of word vectors; thus we make no assertions that these are optimal models.
5.1 Word embeddings
We used three publicly available pre-trained word vectors for English: the 300-dimensional embeddings trained on the Google News corpus with the CBOW model (CBOW),<sup>2</sup> the 300-dimensional embeddings trained on Wikipedia with the skip-gram model (SGNS),<sup>3</sup> and the 300-dimensional embeddings trained on Wikipedia and Gigaword with the GloVe model (GloVe).<sup>4</sup> For Japanese, we trained 300-dimensional embeddings on an approximately 1.5 billion word corpus collected from the Web, with the CBOW model (CBOW), the skip-gram model (SGNS),<sup>5</sup> and the GloVe model (GloVe).<sup>6</sup> We also trained 50-, 100-, and 200-dimensional embeddings on the same corpus for each model in order to investigate the effect of the vector size.
For the evaluation, we used two types of datasets for English and Japanese, respectively.
5.2.1 SP dataset
As the first type, we used word sets that consist of words which can be a direct object of a certain verb. For example, suppose a word set consists of  $\{\text{role}, \text{part}, \text{game}, \text{golf}, \text{tennis}, \text{etc.}\}$ , where each word can be a direct object of the verb play. We did not use the verb itself for evaluation but we can regard this as a selectional preference (SP) task.
For the English SP dataset, we extracted pairs of verbs and their direct objects from the Google Books Syntactic N-grams dataset (Goldberg and Orwant, 2013). We first extracted verbs with the POS tag of VBD, VBP or VBZ that have direct objects at a rate of more than  $40\%$ . We decided on a threshold of  $40\%$  empirically to extract transitive verbs only. Then, we listed the extracted verbs in descending order of the number of the different direct objects and chose the top 1,000 of them.
Figure 3: The pair of a synset and a set of its hyponyms in a distance of at most five. The hyponyms are surrounded by a broken line.
For the Japanese SP dataset, we extracted pairs of verbs and their accusative arguments from the predicate-argument data used by Sasano and Okumura (2016). First, we extracted verbs that have accusative arguments at a rate of more than  $70\%$ . Again, we decided on a threshold of  $70\%$  empirically to extract transitive verbs only. Then, we listed the extracted verbs in descending order of the number of the different accusative arguments and chose the top 1,000 of them.
Both datasets consisted of 1,000 verbs with at least 250 unique direct objects. We selected 200 direct objects as  $W_{c}$  from the most frequent 250 direct objects and the other 50 direct objects as  $w_{t}$  for each verb. Thus, the number of tasks  $N$  was 50,000, i.e., 50 tasks for each of the 1,000 verbs. We used 2,000 negative instances against 200 positive instances to build models with negative instances.
5.2.2 WordNet datasets
We used word sets extracted from English and Japanese WordNet (Fellbaum, 1998; Isahara et al., 2008) as the second type. For example, a word set consists of  $\{dog, llama, hedgehog, wolf, etc.\}$ , which are all hyponyms of the same synonym set (synset n01886756, placental). We extracted the pair of a synset ID and a set of words in the synset and its hyponyms in a distance of at most five from the target synset in the WordNet hyponym tree, as shown in Figure 3. We did not use multiword expressions or words whose word vectors are not included in any of the three pre-trained word embeddings.
We extracted synsets that have at least 250 words. There are 109 word sets for English datasets and
$^{2}$ https://code.google.com/archive/p/word2vec/
<sup>3</sup>https://github.com/jhlau/doc2vec
<sup>4</sup> http://nlp.stanford.edu/data/glove.6B.zip
$^5$ https://code.google.com/archive/p/word2vec/. We used the default parameters except for the vector size.
$^{6}$ https://nlp.stanford.edu/projects/glove/. We used the same parameters as demo.sh except for setting the window size to 5 and the vector size to 300.
120 word sets for Japanese datasets. We selected 200 words as  $W_{c}$  and the other 50 words as  $w_{t}$  for each synset. The number of tasks  $N$  was 5,450, i.e., 50 tasks for each of the 109 synsets for English, and 6,000, i.e., 50 tasks for each of the 120 synsets for Japanese. We used 2,000 negative instances against 200 positive instances to build models with negative instances as well as the SP datasets.
5.3 Experimental settings
We compared eight models: CENT, GM, GMM, kNN, 1-SVM, OffSet,  $\mathrm{SVM}_L$ , and  $\mathrm{SVM}_R$ . For each dataset, we made  $W_{o}$  by extracting 999 words from the other word sets; that is, the number of words for scoring was 1,000, including the target word  $w_{t}$ . For OffSet,  $\mathrm{SVM}_L$ , and  $\mathrm{SVM}_R$ , we make  $W_{n}$  by extracting words from the other word sets subject to the constraint  $W_{o} \cap W_{n} = \{\}$ .
We regarded the problem as a ranking task and adopted the mean reciprocal rank (MRR) as the metric for evaluation. The MRR is calculated by the following equation:
$$
\mathrm {M R R} = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {1}{\operatorname {r a n k} \left(w _ {t _ {i}}\right)}, \tag {7}
$$
where  $\mathrm{rank}(w_{t_i})$  is the rank of the target word  $w_{t_i}$  for each task. We tune the parameters to maximize the MRR in parameter tuning.
We measured the statistical significance with an approximate randomization test (Chinchor, 1992) with 99,999 iterations and significance level  $\alpha = 0.05$  after Bonferroni correction. To satisfy the independence assumption, we treated each verb (for the SP datasets) or synset (for the WordNet datasets) as the unit of a randomization test.
5.4.1 Results on the SP datasets
Tables 1 and 2 show the experimental results on the SP dataset for English and Japanese, respectively. In these tables, the best scores for each word embedding model and the scores with no significant difference from the best score are indicated in bold. In addition, the CENT score and the scores with no significant difference from the CENT score are italicized.
The results in these tables indicate that the models considering the geometry of the distribution or the existence of subgroups in the word class outperform the centroid-based model (CENT) for both the English and Japanese SP datasets. In particular,
Model	CENT	GM	GMM	kNN	1-SVM	OffSet	SVML	SVMR
CBOW	.1642	.2539	.2360	.2097	.1726	.2782	.3397	.3905
SGNS	.1887	.2461	.2308	.1918	.2252	.2189	.3365	.3608
GloVe	.1925	.2596	.2462	.2245	.2295	.1150	.3554	.3800
Ave.	.1818	.2532	.2377	.2087	.2091	.2040	.3439	.3771
Table 1: Results on the English SP dataset.
Model	CENT	GM	GMM	kNN	1-SVM	OffSet	SVML	SVMR
CBOW	.2600	.3151	.2947	.2783	.2812	.2516	.4371	.4922
SGNS	.0789	.2231	.2039	.1757	.1249	.2594	.4173	.4510
GloVe	.1643	.2489	.2377	.2016	.1927	.2088	.3264	.3632
Ave.	.1677	.2624	.2454	.2185	.1996	.2399	.3936	.4355
Table 2: Results on the Japanese SP dataset.
a simple Gaussian model (GM) performed the best among the models that only depend on positive instances. This indicates that these word sets are distributed with different variances depending on the direction in the vector space and it is useful to consider the geometry of the distribution.