Analysis of Italian Word Embeddings
Authors: Rocco Tripodi, Stefano Li Pira (Università Ca' Foscari Venezia, University of Warwick, 2017)
Topic: Distributional Semantics

Abstract
In this work we analyze the performances of two of the most used word embeddings algorithms, skip-gram and continuous bag of words on Italian language. These algorithms have many hyper-parameter that have to be carefully tuned in order to obtain accurate word representation in vectorial space. We provide an extensive analysis and an evaluation, showing what are the best configuration of parameters for specific analogy tasks.

Introduction
The distributional hypothesis of language, set forth by AUTHOR and AUTHOR, states that the meaning of a word can be inferred from the contexts in which it is used. Using the co-occurrence of words in a large corpus, we can observe for example that the contexts in which client is used are very similar to those in which customer occur, while less similar to those in which waitress or retailer occur. A wide range of algorithms have been developed to exploit these properties. Recently, one of the most widely used method in many natural language processing (NLP) tasks is word embeddings AUTHOR. It is based on neural network techniques and has demonstrated to capture semantic and syntactic properties of words taking as input raw texts without other sources of information. It represents each word as a vector such that words that appear in similar contexts are represented with similar vectors AUTHOR. The dimensions of the word are not easily interpretable and, with respect to explicit representation, they do not correspond to specific concepts. In AUTHOR, the authors propose two different models that seek to maximize, respectively, the probability of a word given its context (Continuous bag-of-word model), and the probability of the surrounding words (before and after the current word) given the current word (Skip-gram model). In this work we seek to further explore the relationships by generating word embedding for over 40 different parameterizations of the continuous bag-of-words (CBOW) and the skip-gram (SG) architectures, since as shown in AUTHOR the choice of the hyper-parameters heavily affect the construction of the embedding spaces. Specifically our contributions include: \item Word embedding. The analysis of how different hyper-parameters can achieve different accuracy levels in relation recovery tasks AUTHOR. \item Morpho-syntactic and semantic analysis. Word embeddings have demonstrated to capture semantic and syntactic properties, we compare two different objectives to recover relational similarities for semantic and morph-syntactical tasks. \item Qualitative analysis. We investigate problematic cases.

Conclusion
We have tested two word representation methods: SG and CBOW training them only on a dump of the Italian Wikipedia. We compared the results of the two models using 12 combinations of hyper-parameters. We have adopted a simple word analogy test to evaluate the generated word embeddings. The results have shown that increasing the number of dimensions and the number of negative examples improve the performance of both the models. These types of improvement seems to be beneficial only for the semantic relationships. On the contrary the syntactical relationship are negatively affected by the low frequency of many of its terms. This should be related to the morphological complexity of Italian. In the future it would be helpful to represent the spatial relationship regarding specific syntactical domain in order to evaluate the contribution of hyper-parametrization to syntactical relationship accuracy. Moreover future work will include the testing of these word embedding parametrizations in practical applications (e.g. analysis of patents'descriptions and books' corpora).