# An Empirical Evaluation of Document Embeddings and Similarity Metrics for Scientific Articles
## 6 Task I-Validation
### Analysis of Results  
Next, we analyze the behavior of the different similarity measures concerning our synthetic test. The expected result should be that, as long as we introduce a larger number of sentences from one document to the other, the similarity should increase. Since we \begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**SCS**} & \multicolumn{3}{c}{**WMD**} & \multicolumn{3}{c}{**RWMD**} & \multicolumn{3}{c}{**CS**} & **NRC** \\ \hline  & **w2v** & **fT** & **GV** & **w2v** & **fT** & **GV** & **w2v** & **fT** & **GV** & **d2v** & **ELMo** & \\ \hline \(\overline{t}\) & 17,986 & 19,680 & 32,810 & \textgreater{}1 h & \textgreater{}1 h & \textgreater{}1 h & 59,137 & 50,750 & 50,653 & 0.648 & \textgreater{}1 h & 0.8489 \\ \hline \(\sigma_{t}\) & 1598 & 0.969 & 0.695 & - & - & - & 4787 & 3288 & 3551 & 0.009 & - & 0.131 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Mean and standard deviation for computation times for the different combinations with Ph.D. theses. Figure 1: Time performance (in seconds) of the different similarity measures when evaluating papers belonging to the same (third row of each table) research group and different ones (fourth row) with the different strategies. The timings include the calculation of the embeddings when necessary and the similarity measurement. measured distances, the charts, ideally, should show a monotonically decreasing distance. First, we analyze the results relative to papers, and afterward, the Ph.D. thesis. Figure 2 shows the results of soft cosine similarity for the different embeddings. There are two notable observations here: First, the distance decreases in all scenarios. Second, most of the papers belonging to the same group (the ones labeled CDIF\(i\)) have smaller distances both at the start and the end than papers from different groups. This follows our intuition that articles written by people of the same research group will likely describe more similar problems and techniques than papers of other research groups. Moreover, the relative position of the papers is mostly preserved (e.g., CDIF2 is the one with a smaller distance at the beginning and the end, and IDEAI is the group with both larger distances), although some differences appear within the papers in the middle range. In Figure 3, we analyze the behavior of Words Mover's Distance. Like in the previous case, the distances decrease as the number of copied sentences from one document to the other increases. Again, most of the documents belonging to the same research group have smaller distances than the distances to papers of other research groups. In this case, the relative positions of the papers at the end are fully preserved with the three embeddings, and all of them share the initial and final rankings. Figure 3: Performance of WMD in the validation task for w2v (**left**), fT (**center**), and gV (**right**). In this case, some papers of the same group (CDIF1, CDIF2, and CDIF5) are always considered as more similar to the base paper (CDIF) than the rest. Furthermore, in all cases, the distance decreases with larger number of phrases passed from one to the other. Figure 2: Behavior of SCS in the validation experiment. Some papers within the same group as the base paper (CDIF) exhibit smaller distances than papers from different research groups (labeled after the group name). More concretely, papers CDIF1 through CDIF4 show higher similarity than the rest of the papers in fT + SCS (**center**) and good performance in w2v + SCS (**left**) and gV + SCS (**right**). Moreover, in all cases, the distance decreases as the shared text between the files grows. The relaxed version of WMD is analyzed in Figure 4 for the different embeddings. Its behavior is very similar to the previous cases: monotonically decrease, most of the relative distances rankings preserved. Furthermore, it is noticeable that the behavior of the papers from the same research group than the base one seems better than in previous cases.