# An Empirical Evaluation of Document Embeddings and Similarity Metrics for Scientific Articles
## 9 Conclusions and Future Work  
In this paper, we have presented a deep and systematic analysis of several word embeddings and distance metrics used in the literature. We covered most of the recently developed techniques, such as ELMo, Fast Text, and Global Vectors, together with more popular techniques such as Word2 vec. Since we wanted to analyze real-world practical scenarios where high performing computing infrastructure might not be available, we discarded some techniques presented recently, which, in most cases, require huge power consumption. As a result, many of these have also a substantial pollution impact for the model training. For example, [PERSON] et al. [28], pretrained BERT for improved results, and used 1024 V100 GPUs for approximately one day. For the experiments, we concentrated on research documents in PDF format. These were processed with the common pipeline used in Natural Language Processing that involves generating the plain text, cleaning strange characters commonly generated by the PDF-to-text translators, and stemming. We carried out two experiments, one intended to more formally evaluate the distance metrics, and a second one intended to mimic a real-world problem, clustering documents according to their similarity. The results show that not all combinations of embeddings and distances are equally suitable. A notable result is a fact that some techniques have a huge cost, which makes them impractical, while others perform equally, even yielding superior results, albeit incurring a lower cost. From the developed techniques, the combination of d2v embedding with CS similarity seems to be the most robust for both problems, except for extremely large documents such as a Ph.D. thesis. \begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline
**Distance** & \multicolumn{3}{c}{**SCS**} & \multicolumn{3}{c}{**WMD**} & \multicolumn{3}{c}{**RWMD**} & \multicolumn{3}{c}{**CS**} & **NRC** \\ \hline
**Embedding** & **w2v** & **fT** & **GV** & **w2v** & **fT** & **GV** & **w2v** & **fT** & **GV** & **d2v** & **ELMo** \\ \hline
**Task 1–Article/thesis comparison** & **+** & **+** & **+** & \(\sim\) & \(\sim\) & \(\sim\) & \(\sim\) & \(\sim\) & \(\sim\) & **+** & **-** & **-** \\ \hline
**Task 2–Clustering** & **+** & **+** & **+** & & **?** & & **\(\sim\)** & \(\sim\)** & \(\sim\) & **+** & **?** & **-** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Summary of the results. “+” means good results, “\(\sim\)” average results, and “-” bad results. Results that were not possible to evaluate (e.g., the calculation took too long) are indicated by “?”. The combination of d2v and CS is labelled as good because the results with the papers are very positive. However, for very long documents, it exhibited a bumpy decrease instead of a monotonic behavior (therefore we marked it with a “**”**. However, for the clustering case, its behavior is superior to Gv + RWMD, though the latter also works acceptably. Though the project presented here has produced several useful insights, there is more work that can be performed from here. First, it would be interesting to curate a larger set of documents that can be used for further experiments. Second, it would be interesting to further investigate the problems of d2v + CS with very long documents. Moreover, other tasks beyond clustering could also be addressed. In this paper, we addressed an empirical evaluation with the most popular and well-known techniques in the literature. However, some other methods of document representations are being presented now, in most cases, only in arXiv (e.g., [38]). In the future, we would like to study these new techniques, especially the ones that achieve a certain degree of popularity, and test them against the other techniques evaluated here. Conceptualization, [PERSON]-P.V. ; methodology, [PERSON] and [PERSON]-P.V. ; software, [PERSON]; validation, [PERSON] and [PERSON]-P.V. ; investigation, [PERSON] and [PERSON]-P.V. ; data curation, [PERSON] and [PERSON]-P.V.