SlowGuess/ACL_2020_OCR huggingface dataset:
The two discriminative learning-based models with negative instances,  $\mathrm{SVM}_L$  and  $\mathrm{SVM}_R$ , achieved much higher performance, whereas 1-SVM yielded a limited improvement over CENT. This demonstrates that modeling the distribution with only positive instances has an obvious limitation, and it is essential to leverage the negative instances as well. OffSet with CBOW or SGNS achieved a relatively good performance, but OffSet with GloVe did not, which suggests that the usefulness of the offset depends on the word embedding model.
5.4.2 Results on the WordNet datasets
Tables 3 and 4 show the experimental results on the WordNet dataset for English and Japanese, respectively. The meaning of bold and italic fonts is identical to that on the SP dataset.
The two discriminative learning-based models with negative instances and OffSet with CBOw or SGNS achieved a relatively high performance. This demonstrates that the negative instances must be taken into account to model the distribution properly. On the other hand, in contrast with the SP datasets, there were no significant improvements when the geometry of the distribution and the existence of subgroups were considered.
The scores were generally lower than those of the SP datasets. We conjecture that this is because WordNet is developed manually and reflects human
Model	CENT	GM	GMM	kNN	1-SVM	OffSet	SVML	SVMR
CBOW	.1435	.1320	.1460	.1473	.1541	.2263	.2564	.2678
SGNS	.1767	.1679	.1573	.1625	.1704	.1998	.2292	.2357
GloVe	.1792	.1694	.1562	.1744	.1684	.1310	.2075	.2264
Ave.	.1665	.1564	.1532	.1614	.1643	.1857	.2310	.2433
Table 3: Results on the English WordNet dataset.
Model	CENT	GM	GMM	kNN	1-SVM	OffSet	SVML	SVMR
CBOW	.1996	.1991	.1918	.2169	.2082	.2656	.2730	.2961
SGNS	.0466	.0521	.0774	.0768	.0701	.2367	.2686	.2862
GloVe	.1055	.1050	.1021	.0987	.0984	.0681	.2033	.2189
Ave.	.1172	.1187	.1238	.1308	.1256	.1901	.2483	.2671
intuition, whereas the SP datasets are automatically built from the corpus and are highly compatible with the pre-trained word vectors. In addition, we examined which types of words tend to rank low and found that words extracted from a synset corresponding to their infrequent sense such as stock in the sense of livestock tend to rank low. We leave further exploration for future work.
5.4.3 Discussion
It is interesting that although  $\mathrm{SVM}_L$  is effectively just a linear classifier,  $\mathrm{SVM}_L$  achieves a relatively high performance. This is likely due to the relatively large vector size compared to the number of positive instances and indicates that the positive instances occupy a certain span in the vector space though such a span cannot be determined by only using positive instances. We confirmed two desirable properties of the discriminative learning-based models with negative instances for practical applications. One is that since we used simple models, they do not require much training time. The other is that their performance is relatively stable among the different word embeddings and datasets compared to the other models.
We also investigated the effect of the vector size and the number of positive instances on the Japanese SP dataset. Table 5 shows the averaged CBOW, SGNS, and GloVe scores for different vector dimensions, 50, 100, 200, and 300. We found that while CENT and 1-SVM were not affected much by the vector size, the other models, particularly OffSet,  $\mathrm{SVM}_L$ , and  $\mathrm{SVM}_R$ , were significantly affected by the vector size. Table 6 shows the averaged CBOW, SGNS, and GloVe scores for the different number of positive instances, 25, 50, 100,
Table 4: Results on the Japanese WordNet dataset.
Size	CENT	GM	GMM	kNN	1-SVM	OffSet	SVML	SVMR
50	.1686	.2360	.2055	.1909	.1825	.1769	.2842	.3568
100	.1738	.2557	.2177	.2075	.1954	.2189	.3366	.4044
200	.1724	.2697	.2233	.2178	.2005	.2363	.3813	.4340
300	.1677	.2624	.2454	.2185	.1996	.2399	.3936	.4355
Table 5: The average scores of different vector size with the Japanese SP dataset.
|Wc|	CENT	GM	GMM	kNN	1-SVM	OffSet	SVML	SVMR
25	.1563	.1728	.1522	.1635	.1562	.1880	.2326	.2600
50	.1612	.2008	.1779	.1795	.1722	.2144	.2898	.3157
100	.1652	.2388	.2098	.1988	.1880	.2307	.3475	.3790
200	.1677	.2624	.2454	.2185	.1996	.2399	.3936	.4355
Table 6: The average scores of different word set size with the Japanese SP dataset.
and 200. We can conclude that all the models perform at a higher level based on the larger number of positive instances, especially for GM, GMM,  $\mathrm{SVM}_L$ , and  $\mathrm{SVM}_R$ . This is not surprising, since these models have a large number of parameters and can extract a rich variety of information from the large number of positive instances. Similar tendencies were also observed with the other dataset. These results demonstrate that we can obtain relatively high performance by using discriminative learning-based models with a large enough vector and training data size.
5.5 Degree of membership
Rosch (1975) developed the prototype concept and proved that not all members of a category are equally representative of the category. Here, we are interested in the relationship between the scores calculated by each model and the degree of membership. We thus investigated how consistent the score calculated by each model is with human intuition on the degree of membership.
For this experiment, we used the typicality data by Rosch (1975). Rosch asked 209 college students to use a 7-point scale to rate the extent to which each instance represents their idea or image of the meaning of the category term, and reported the rank orders with the mean ratings for ten categories. For example, for the Furniture category, 60 examples are ranked with the mean ratings, chair and sofa are top-ranked with the score of 1.04, and
To test the reliability of ratings, Rosch (1975) obtained Spearman rank-order correlations and Pearson product-moment correlations between sub-groups of students and reported that consistency was extremely high.
Category	Synset ID	|WR|	|Wc|	|WR∩Wc|
Furniture	n03405725	60	89	26
Fruit	n13134947	51	165	41
Vehicle	n04524313	50	346	34
Weapon	n04565375	60	119	19
Vegetable	n07707451	56	102	27
Bird	n01503061	54	330	51
Sport	n00523513	59	106	33
Clothing	n03051540	55	409	31
Table 7: Statistics of the typicality dataset.
Model	CENT	GM	GMM	kNN	1-SVM	OffSet	SVML	SVMR
ρ
CBOW	.1736	.1905	.1706	.2417	.1160	.3224	.3176	.2562
SGNS	.2848	.3194	.4024	.3221	.1924	.2940	.3363	.3121
GloVe	.1458	.1949	.1448	.3204	.1780	.4383	.3367	.2702
Ave.	.2014	.2349	.2393	.2947	.1621	.3516	.3302	.2795
τ
CBOW	.1230	.1373	.1198	.1833	.0728	.2400	.2289	.1855
SGNS	.2101	.2400	.2945	.2355	.1400	.2066	.2390	.2180
GloVe	.1012	.1401	.1080	.2254	.1266	.3038	.2391	.1908
Ave.	.1448	.1725	.1741	.2147	.1131	.2501	.2357	.1981
Table 8: Averaged rank correlation coefficients against the typicality data by Rosch.
stove is ranked as 50th with the score of 5.4.
In this study, we used eight categories that have a corresponding synset in WordNet. Table 7 shows the statistics of the dataset. In the table,  $|W_{R}|$  denotes the number of examples in Rosch's dataset,  $|W_{c}|$  denotes the number of words in the synset and its hyponyms in the WordNet, and  $|W_{R} \cap W_{c}|$  is the number of words included in both  $W_{R}$  and  $W_{c}$ , which we try to rank here.
In this experiment, the objective was not to distinguish a possible member from others but to rank the positive member  $w_{c}$  in  $W_{c}$  according to the degree of membership. That is, we first formed the scoring function by using  $W_{c}$  and  $W_{n}$  and then applied the function to each member of  $W_{R} \cap W_{c}$  to predict the typicality ranking. We evaluated the ranking by calculating Spearman's rank correlation coefficient  $(\rho)$  and Kendall's rank correlation coefficient  $(\tau)$  against the ranking of the goodness-of-example in Rosch's dataset. We computed the average rank correlation coefficient over the eight categories for  $\rho$  and  $\tau$ . Table 8 shows the experimental results.
In contrast with the previous experiments, the highest scores were achieved by OffSet. These results suggest that the vector offsets can be used to derive the degree of membership. We can say that, while discriminative learning-based models, espe
cially  $\mathrm{SVM}_R$ , can find the boundary of a category in a vector space with high accuracy, the vector offset between the centroid of positive instances and that of negative instances can properly represent the degree of membership in a category.
When we focused on each combination of the embedding and distribution models, we found that the highest and second highest scores were achieved by OffSet with GloVe and GMM with SGNS, respectively. In contrast, both achieved relatively low performance in distinguishing a possible member of a word class from others, as shown in Table 3. These results demonstrate that the proper models for finding the boundaries of a class and those for determining the degree of membership are different and that choosing a proper model depending on the task is essential.
We investigated the distribution of words that belong to a certain word class in a pre-trained general-purpose word vector space. The experimental results show that a centroid-based approach cannot provide a reasonably good model and considering the geometry of the distribution and the existence of subgroups is useful for modeling the distribution in some cases. However, the impact is limited, and the negative instances must be taken into account for adequate modeling. The results indicate that just observing the distribution of positive instances is not enough to understand the geometry of word embedding spaces. Furthermore, we investigated the relationship between the score calculated by each model and the degree of membership and demonstrated that, while discriminative learning-based models can distinguish a possible member of a word class from others, the offset-based model achieves higher correlations with the degree of membership.
The investigation in this study leveraged only general-purpose word vectors to represent the meaning of a word. However, several studies have expanded the method for acquiring a word vector to account for the uncertainty of word meanings and word polysemy (e.g., Athiwaratkun et al. 2018). In addition, contextualized word embeddings have been shown to be very effective on a range of NLP tasks (Peters et al., 2018; Devlin et al., 2019). Furthermore, Gong et al. (2018) reported that word embeddings learned in several tasks are biased towards word frequency: the embeddings of high
3664
frequency and low-frequency words lie in different subregions of the embedding space. Thus, in the future, we will take the uncertainty, polysemy, and context sensitivity of the word meanings and the frequency of words into account and explore better ways of modeling the word-class distributions in semantic vector spaces.
This work was supported by JSPS KAKENHI Grant Number 16K16110 and 18H03286.
Ben Athiwaratkun and Andrew Wilson. 2017. Multimodal word distributions. In Proc. of ACL'17, pages 1645-1656.
Ben Athiwaratkun, Andrew Wilson, and Anima Anandkumar. 2018. Probabilistic fastText for multi-sense word embeddings. In Proc. of ACL'18, pages 1-11.
Marco Baroni, Georgiana Dinu, and German Kruszewski. 2014. Don't count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In Proc. of ACL'14, pages 238-247.
Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A unified model for word sense representation and disambiguation. In Proc. of EMNLP'14, pages 1025-1035.
Nancy Chinchor. 1992. The statistical significance of the MUC-4 results. In Proceedings of the 4th Message Understanding Conference (MUC), pages 30-50.
Tim Van de Cruys. 2014. A neural network approach to selectional preference acquisition. In Proc. of EMNLP'14, pages 26-35.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL-HLT'19, pages 4171-4186.
Katrin Erk, Sebastian Padó, and Ulrike Padó. 2010. A flexible, corpus-driven model of regular and inverse selectional preferences. Computational Linguistics, 36(4):723-763.
Manaal Faruqui, Jesse Dodge, Sujay K Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proc. of NAACL-HLT'15, pages 1606-1615.
Christiane Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press.
Goran Glavaš and Ivan Vulić. 2018. Explicit retrofitting of distributional word vectors. In Proc. of ACL'18, pages 34-45.
Yoav Goldberg and Jon Orwant. 2013. A dataset of syntactic-grams over time from a very large corpus of English books. In Proc. of *SEM'13, pages 241-247.
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2018. Frage: Frequency-agnostic word representation. In Proc. of NIPS'18, pages 1334-1345.
Hitoshi Isahara, Francis Bond, Kiyotaka Uchimoto, Masao Utiyama, and Kyoko Kanzaki. 2008. Development of the Japanese WordNet. In Proc. of LREC'08, pages 2420-2423.
Joo-Kyung Kim and Marie-Catherine de Marneffe. 2013. Deriving adjectival scales from continuous space word representations. In Proc. of EMNLP'13, pages 1625-1630.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013a. Distributed representations of words and phrases and their compositionality. In Proc. of NIPS'13, pages 3111-3119.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proc. of NAACL-HLT'13, pages 746-751.
David Mimno and Laure Thompson. 2017. The strange geometry of skip-gram with negative sampling. In EMNLP'17, pages 2873-2878.
Nikola Mrkšić, Ivan Vulic, Diarmuid Řeaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017. Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the Association for Computational Linguistics, 5:309–324.
Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient nonparametric estimation of multiple embeddings per word in vector space. In Proc. of EMNLP'14, pages 1059-1069.
Patrick Pantel, Eric Crestan, Arkady Borkovsky, Ana Maria Popescu, and Vishnu Vyas. 2009. Web-scale distributional similarity and entity set expansion. In Proc. of EMNLP'09, pages 938-947.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proc. of EMNLP'14, pages 1532-1543.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL-HLT'18, pages 2227-2237.
Philip Resnik. 1996. **Selectional constraints: An information-theoretic model and its computational realization. Cognition, 61:127-159.**
Eleanor Rosch. 1975. Cognitive representations of semantic categories. Journal of Experimental Psychology: General, 104:192-233.
Sascha Rothe and Hinrich Schütze. 2015. AutoExtend: Extending word embeddings to embeddings for synsets and lexemes. In Proc. of ACL-IJCNLP'15, pages 1793-1803.
Kugatsu Sadamitsu, Kuniko Saito, Kenji Imamura, and Genichiro Kikui. 2011. Entity set expansion using topic information. In Proc. of ACL-HLT'11, pages 726-731.
Ryohei Sasano and Manabu Okumura. 2016. A corpus-based analysis of canonical word order of Japanese double object constructions. In Proc. of ACL'16, pages 2236-2244.
Bernhard Scholkopf, John C. Platt, John C. Shawe-Taylor, Alex J. Smola, and Robert C. Williamson. 2001. Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443-1471.
Fei Tian, Hanjun Dai, Jiang Bian, Bin Gao, Rui Zhang, Enhong Chen, and Tie-Yan Liu. 2014. A probabilistic model for learning multi-prototype word embeddings. In Proc. of COLING'14, pages 151-160.
Luke Vilnis and Andrew McCallum. 2015. Word representations via gaussian embedding. In Proc. of ICLR'15.
Kristian Woodsend and Mirella Lapata. 2015. Distributed representations for unsupervised semantic role labeling. In Proc. of EMNLP'15, pages 2482-2491.
Yadollah Yaghoobzadeh and Hinrich Schütze. 2016. Intrinsic subspace evaluation of word embedding representations. In Proc. of ACL'16, pages 236-246.
Zhenzhong Zhang, Le Sun, and Xianpei Han. 2016. A joint model for entity set expansion and attribute extraction from web search queries. In Proc. of AAAI'16, pages 3101-3107.
3666