Combining embedding methods for a word intrusion task

We report a new baseline for a Danish word intrusion task by combining pre-trained offthe-shelf word, subword and knowledge graph embedding models. We test fastText, Byte-Pair Encoding, BERT and the knowledge graph embedding in Wembedder, finding fastText as the individual model with the superior performance, while a simple combination of the fastText with other models can slightly improve the accuracy of finding the odd-one-out words in the word intrusion task. In the word intrusion task, see, e.g., (Chang et al., 2009), a cognitive agent is presented with a set of words and is to determine the odd-one-out. Such a test has been used to evaluate unsupervised topic models (Chang et al., 2009) and human subjects in experimental psychology, see, e.g., (Crutch et al., 2008). The test somewhat resembles Test of English as a Foreign Language (TOELF), where the task is to select the semantically most similar one among four words given a query word (Turney, 2006). A convenient method (doesnt_match) is implemented in the distributed semantics models of Gensim (Řehůřek and Sojka, 2010; Wohlgenannt et al., 2019), giving users of this Python package a straightforward way to test trained machine learning models in odd-one-out tasks. (Nielsen and Hansen, 2017) constructed a word intrusion dataset with Danish words and evaluated how well different machine-based methods could identify the intruded word. Explicit semantic analysis and a Word2vec-based word embedding with large corpora performed the best with performances of 73% and 71%, respectively, against a random choice baseline of 25%. Since (Nielsen and Hansen, 2017), new embedding methods have appeared with pre-trained models for non-English languages, e.g., fastText (FT) (Bojanowski et al., 2016; Grave et al., 2018), Byte-Pair Encoding (BPE) (Heinzerling and Strube, 2018), BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2018), and Wembedder (W), a knowledge graph embedding based on the multilingual Wikidata knowledge base (Nielsen, 2017). We note that some of the best performing semantic models have combined corpus-based and explicit lexicon-/knowledge graph-based methods (Turney, 2006; Speer and Lowry-Duda, 2017), and we will also pursue such a combination here. Below we will describe the Danish word intrusion task dataset used for evaluation, the applied new off-the-shelf methods, their results in terms of accuracy of detecting the odd-one-out and finally we discuss what further approaches are needed to handle the remaining misclassified cases. 1 Evaluation dataset The word intrusion dataset comprises 100 sets of 4 words each where one of 4 is the outlier to be detected (Nielsen and Hansen, 2017),1 see the left part of Figure 1 for a small excerpt of the dataset. The dataset contains common and proper nouns (named entities) and other word classes as well as a few numbers, years and phrases. Some sets of “words” require detailed Danish world knowledge, e.g., 1807, 1864, 1940, 1909, — the last being the outlier as the three first years relates to Danish military defeats. The dataset contains also several homographs/polysemous words. Most of the words are common nouns. There are 11 word sets with proper nouns and 11 with verbs. Further sets includes sets of adjective and other word classes. A few of the word sets mix lexical categories, e.g., the set (halvsyg, forkølelse, hoster, vej) corresponding to the English (“half-sick”, flu, the verb “coughs”, road). 1https://github.com/fnielsen/dasem/ blob/master/dasem/data/four_words_2. csv. We use the second version correcting two spelling errors. Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019) Distributed under a CC BY-NC-SA 4.0 license.