# An Empirical Evaluation of Document Embeddings and Similarity Metrics for Scientific Articles
## 4 Embeddings and Measures
### Embeddings  
When talking about embeddings, one of the first ones that may come to mind is _word2 vec_. **word2 vec** (w2v), introduced by [PERSON] et al. [24], trains a network language model in two steps: (i) word learning, and (ii) training \(n-gram\) Neural Network Language Models (NNLM) on the top of the representation of words. They introduced two log-linear model architectures: the Continuous Bag-Of-Words Model (CBOW) (that predicts a current word based on the context and a window of words around the one to be predicted) and the Continuous Skip-gram Model (CSG) (that predicts a window of words around a known word). The authors found that low-dimension vectors, e.g., 300, trained in more data are capable to catch more accuracy than higher-dimension vectors trained in less amount of data. They also found that the training on twice the data during one epoch achieves in many cases better results than iterating over the same data during three epochs, which makes these models clearly interesting to be trained on large amounts of data. **fastText** (fT) model, introduced by [PERSON] et al. [25], is presented as an extension of the CSG model, but instead of taking into account entire words, fT runs over subword information. The idea of this model lies in the fact that languages such as Spanish, which have a lot of non-commonly used conjugations, are hard for learning good word representations because these words are rare in the training sets. To catch these words, the model works with subword units. It was shown that working with subword information, the modeling of the morphologically rich language is improved. They compared the fT model to the basic CSG model for several languages analyzed (Czech, Russian, Spanish, and French). The results found that the prediction rate improves, albeit with different rates depending on the language. **GloVe** (GV), presented by [PERSON] et al. [17], tries to combine two approaches to learn word vectors: the global matrix factorization and the local context window. The global matrix factorization approach is a set of techniques that decompose large matrices containing captured statistical information. The most commonly used is Latent Semantic Analysis (LSA) whose matrix captures the appearances of words in documents. In the end, these types of techniques are statistics-based methods to learn word representations. On the other hand, the local context window-based techniques, such as CSG and CBOW, are methods focused on the context of the words, without taking into consideration the associated corpus statistics. The authors found that in almost all tasks with different corpus, GV outperforms w2v models. Relative to itself, GV seems to be the best one in semantics but not in syntax, but in the end, good in general. Moreover, the authors concrete that the syntactic subtask performance improves when the corpus size increases, but it does not necessarily occur in a semantic task. Introduced by [PERSON] and [PERSON] [15], **doc2 vec** (d2v), also known as Paragraph Vector, extends word2 vec models to use the same base algorithm to learn document vectors and to make the predictions more accurate using their information. As in w2v models, the proposed d2v models map every word to a unique vector (column in a matrix \(W\)), but they also map every sentence, paragraph, or large text to a unique vector (column in a matrix \(D\)). Essentially, the only change from w2v is that the hidden layer is computed by averaging the matrices \(W\) and \(D\), instead of just the matrix \(W\). Using the proposed models in Sentiment Analysis and Information Retrieval tasks, the authors found that both outperform in terms of error rate all the baselines compared. Regarding the window size, they propose a range from 5 to 12. Since doc2 vec has a representation for whole sentences, we cannot use similarity metrics that rely on the use of word representations, such as the SCS or WMD and its derivatives. Consequently, only CS will be tested with this representation.