# An Empirical Evaluation of Document Embeddings and Similarity Metrics for Scientific Articles
## 6 Task I-Validation
### Performance Analysis  
First, we analyze the performance of the different combinations. We measured the average time of execution as well as the standard deviation. For papers, the average is the result of 11 executions, while the Ph.D. theses data used 101 executions.  
In Figure 1, we depict the time cost for the experiment that compares papers from the same research group (third row) and papers from different groups (fourth row). In Table 4, we can see the mean and the standard deviation of the computation times for the different sets of documents previously described. Each one corresponds to papers from the same groups, papers from different groups, and theses, respectively. If we analyze the standard deviations, we can see that it does not explode despite the increase in the number of sentences. On the contrary, the values keep low, which adds confidence to the samples. Average calculation times, which include the embeddings computing when necessary, together with the distance evaluation, show that the embedding used for the same measure does not change the order of magnitude of the times. This happens for the three scenarios. As an example, for SCS, the order of magnitude remains \(10^{0}\) with all the embeddings for papers from the same group and for the papers from different groups; and \(10^{1}\) for the theses. This shows that, in these cases, the impact of the embeddings' computation is not relevant. Average times grow with the increased dissimilarity between the groups. Thus, papers in the same group require a low cost in evaluating their similarity, which is smaller than papers of different groups. Furthermore, in turn, these values are also smaller than for the Ph.D. theses operations.  
SCS is the measure with a lower increment in its order of magnitude for each scenario, i.e., it seems to scale well with the size and the complexity. NRC, instead, increases, but not up to remarkable values. Finally, the cost of WMD and CS + ELMo is so huge, that they could not be analyzed due to the executions that did not finish. Due to the characteristics of WMD, i.e., that is based on a Linear Programming Problem, it seems logical that theses scenario will become untreatable for that distance because in the experiments carried out, the execution of the measure exceeds 8 hours. RWMD, which has one less constraint, relaxes the problem and becomes an option for all the scenarios, being between \(10^{-1}\) and \(10^{1}\). All the experiments were carried out on the same computer. On the other hand, considering that CS is a low-complex measure (\(\mathcal{O}(n)\)), the problem which seems to be in the embeddings' computation with ELMo, is that it becomes a hard problem for large texts.  
The measure with the best scalability is CS combined with d2v, which continues in the \(10^{-1}\) order of magnitude for all the scenarios. The second is NRC, whose order of magnitude is \(10^{-2}\) for papers and \(10^{-1}\) for Ph.D. theses. SCS and RWMD have roughly the same performance, being the inflection point for RWMD (and WMD), the scenario of papers from different groups because the times observed are relatively higher in some cases.