Introduction
Traditionally, a word is represented as a sparse vector indicating the word itself (one-hot vector) or the context of the word (distributional vector). However, both the one-hot notation and distributional notation suffer from data sparseness since dimensions of the word vector do not interact with each other. Distributed word representation addresses the data sparseness problem by constructing a dense vector of a fixed length, wherein contexts are shared (or distributed) across dimensions. Distributed word representation is known to improve the performance of many NLP applications such as machine translation BIBREF0 and sentiment analysis BIBREF1 to name a few. The task to learn a distributed representation is called representation learning.
However, evaluating the quality of learned distributed word representation itself is not straightforward. In language modeling, perplexity or cross-entropy is widely accepted as a de facto standard for intrinsic evaluation. In contrast, distributed word representations include the additive (or compositional) property of the vectors, which cannot be assessed by perplexity. Moreover, perplexity makes little use of infrequent words; thus, it is not appropriate for evaluating distributed presentations that try to represent them.
Therefore, a word similarity task and/or a word analogy task are generally used to evaluate distributed word representations in the NLP literature. The former judges whether distributed word representations improve modeling contexts, and the latter estimates how well the learned representations achieve the additive property. However, such resources other than for English (e.g., Japanese) seldom exist. In addition, most of these datasets comprise high-frequency nouns so that they tend not to include other parts of speech. Hence, previous data fail to evaluate word representations of other parts of speech, including content words such as verbs and adjectives.
To address the problem of the lack of a dataset for evaluating Japanese distributed word representations, we propose to build a Japanese dataset for the word similarity task.
The main contributions of our work are as follows:
Related Work
In general, distributed word representations are evaluated using a word similarity task. For instance, WordSim353 2002:PSC:503104.503110, MC BIBREF2 , RG BIBREF3 , and SCWS Huang:2012:IWR:2390524.2390645 have been used to evaluate word similarities in English. Moreover, baker-reichart-korhonen:2014:EMNLP2014 built a verb similarity dataset (VSD) based on WordSim353 because there was no dataset of verbs in the word-similarity task. Recently, SimVerb-3500 was introduced to evaluate human understanding of verb meaning Gerz:2016:EMNLP. It provides human ratings for the similarity of 3,500 verb pairs so that it enables robust evaluation of distributed representation for verbs. However, most of these datasets include English words only. There has been no Japanese dataset for the word-similarity task.
Apart from English, WordSim353 and SimLex-999 Hill:2015:CL have been translated and rescored in other languages: German, Italian and Russian Leviant:2015:arXiv. SimLex-999 has also been translated and rescored in Hebrew and Croatian Mrksic:2017:TACL. SimLex-999 explicitly targets at similarity rather than relatedness and includes adjective, noun and verb pairs. However, this dataset contains only frequent words.
In addition, the distributed representation of words is generally learned using only word-level information. Consequently, the distributed representation for low-frequency words and unknown words cannot be learned well with conventional models. However, low-frequency words and unknown words are often comprise high-frequency morphemes (e.g., unkingly INLINEFORM0 un + king + ly). Some previous studies take advantage of the morphological information to provide a suitable representation for low-frequency words and unknown words BIBREF4 , BIBREF5 . Morphological information is particularly important for Japanese since Japanese is an agglutinative language.
Construction of a Japanese Word Similarity Dataset
What makes a pair of words similar? Most of the previous datasets do not concretely define the similarity of word pairs. The difference in the similarity of word pairs originates from each annotator's mind, resulting in different scales of a word. Thus, we propose to use an example-based approach (Table TABREF9 ) to control the variance of the similarity ratings. We remove the context of word when we extracted the word. So, we consider that an ambiguous word has high variance of the similarity, but we can get low variance of the similarity when the word is monosemous.
For this study, we constructed a Japanese word similarity dataset. We followed the procedure used to construct the Stanford Rare Word Similarity Dataset (RW) Luong-etal:conll13:morpho.
We extracted Japanese word pairs from the Evaluation Dataset of Japanese Lexical Simplification kodaira. It targeted content words (nouns, verbs, adjectives, adverbs). It included 10 contexts about target words annotated with their lexical substitutions and rankings. Figure FIGREF1 shows an example of the dataset. A word in square brackets in the text is represented as a target word of simplification. A target word is not only recorded in the lemma form but also in the conjugated form. We built a Japanese similarity dataset from this dataset using the following procedure.
Comparison to Other Datasets
Table TABREF17 shows how several resources vary. WordSim353 comprises high-frequency words and so the variance tends to be low. In contrast, RW includes low-frequency words, unknown words, and complex words composed of several morphemes; thus, the variance is large. VSD has many polysemous words, which increase the variance. Despite the fact that our dataset, similar to the VSD and RW datasets, contains low-frequency and ambiguous words, its variance is 3.00. The variance level is low compared with the other corpora. We considered that the examples of the similarity in the task request reduced the variance level.
We did not expect SCWS to have the largest variance in the datasets shown in Table TABREF17 because it gave the context to annotators during annotation. At the beginning, we thought the context would serve to remove the ambiguity and clarify the meaning of word; however after looking into the dataset, we determined that the construction procedure used several extraordinary annotators. It is crucial to filter insincere annotators and provide straightforward instructions to improve the quality of the similarity annotation like we did.
To gain better similarity, each dataset should utilize the reliability score to exclude extraordinary annotators. For example, for SCWS, an annotator rating the similarity of pair of “CD” and “aglow” assigned a rating of 10. We assumed it was a typo or misunderstanding regarding the words. To address this problem, such an annotation should be removed before calculating the true similarity. All the datasets except for RW simply calculated the average of the similarity, but datasets created using crowdsourcing should consider the reliability of the annotator.
Analysis
We present examples of a pair with high variance of similarity as shown below:
(e.g., a pairing of “fastUTF8min（速い）” and “earlyUTF8min（早い）”.)
Although they are similar in meaning with respect to the time, they have nothing in common with respect to speed; Annotator A assigned a rating of 10, but Annotator B assigned a rating of 1.
Another example, the pairing of “be eagerUTF8min（懇願する）” and “requestUTF8min（頼む）”. Even though the act indicated by the two verbs is the same, there are some cases where they express different degrees of feeling. Compared with “request”, “eager” indicates a stronger feeling. There were two annotators who emphasized the similarity of the act itself rather than the different degrees of feeling, and vice versa. In this case, Annotator A assigned a rating of 9, but Annotator B assigned a rating of 2.
Although it was necessary to distinguish similarity and semantic relatedness BIBREF7 and we asked annotators to rate the pairs based on semantic similarity, it was not straightforward to put paraphrase candidates onto a single scale considering all the attributes of the words. This limitation might be relaxed if we would ask annotators to refer to a thesaurus or an ontology such as Japanese Lexicon GoiTaikei:1997.
(e.g., a pairing of “sloganUTF8min（スローガン）” and “sloganUTF8min（標語）”.)
In Japanese, we can write a word using hiragana, katakana, or kanji characters; however because hiragana and katakana represent only the pronunciation of a word, annotators might think of different words. In this case, Annotator A assigned a rating of 8, but Annotator B assigned a rating of 0. Similarly, we confirmed the same thing in other parts of speech. Especially, nouns can have several word pairs with different spellings, which results in their IAA became too low compared to other parts of speech.
(e.g., a pairing of “oftenUTF8min（しばしば）” and “frequentlyUTF8min（しきりに）”.)
We confirmed that the variance becomes larger among adverbs expressing frequency. This is due to the difference in the frequency of words that annotators imagines. In this case, Annotator A assigned a rating of 9, but Annotator B assigned a rating of 0. Similarly, we confirmed the same thing among adverbs expressing time.
Conclusion
In this study, we constructed the first Japanese word similarity dataset. It contains various parts of speech and includes rare words in addition to common words. Crowdsourced annotators assigned similarity to word pairs during the word similarity task. We gave examples of similarity in the task request sent to annotators, so that we reduced the variance of each word pair. However, we did not restrict the attributes of words, such as the level of feeling, during annotation. Error analysis revealed that the notion of similarity should be carefully defined when constructing a similarity dataset.
As a future work, we plan to construct a word analogy dataset in Japanese by translating an English dataset to Japanese. We hope that a Japanese database will facilitate research in Japanese distributed representations.
Language Resource References
lrec lrec2018

Question: what is the size of this dataset?
Answer: Unanswerable

Question: did they use a crowdsourcing platform for annotations?
Answer: Yes

Question: where does the data come from?
Answer: Evaluation Dataset of Japanese Lexical Simplification kodaira