## 6 Details of Experiments and Their Results
### Results of the Experiments  
In this section we first discuss the statistical properties of the experiments to show that our models capture statistically significant geometric correspondences between the graph of recommendation summaries and the graph of conceptual distances between the full document guidelines. Table 6 shows results of the main series of experiments we performed. Additional experiments are reported in Appendix A.  
Table 6 shows the results of the experiments with full text of the guidelines. For our best model, BioASQ_WMD, we found a 69% similarity (top line), or 0.31 distance (distortion). As can be seen, the average distortion of permutations (using the distances produced by BioASQ_WMD) is 38%; however the standard deviation of the distortions is less than 1%. Thus, the distance between our model and the mean is about seven standard  
\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Model** & **Distortion** &
\begin{tabular}{l} **Distortion of** \\ **Permutations** \\ \end{tabular} & **STD** \\ \hline
**BioASQ\_WMD** & 0.31393366 & 0.38137817 & 0.00901798 \\ \hline
**Conceptualized\_WMD** & 0.33504400 & 0.39118512 & 0.00929325 \\ \hline
**NoConcept\_WMD** & 0.34457155 & 0.38822718 & 0.00909964 \\ \hline
**BioASQ\_CosD** & 0.41787106 & 0.59569767 & 0.01572929 \\ \hline
**Conceptualized\_CosD** & 0.53452523 & 0.61350075 & 0.01626678 \\ \hline
**NoConcept\_CosD** & 0.51399564 & 0.59093162 & 0.01538653 \\ \hline
**BioASQ\_CosD’** & 0.39343054 & 0.57170607 & 0.01494240 \\ \hline
**Conceptualized\_CosD’** & 0.47697532 & 0.55849892 & 0.01458596 \\ \hline
**NoConcept\_CosD’** & 0.47889093 & 0.55465835 & 0.01434584 \\ \hline  & \multicolumn{2}{l}{Distance measured as \(1-\)} & \\  & \multicolumn{2}{l}{\"normalized search score\"} & \\ \hline
**Search** & 0.54364717 & 0.61957994 & 0.01753947 \\ \hline \hline \end{tabular}
\end{table}
Table 6: This table shows the values obtained in multiple experiments. Column 2, Distortion, shows the distortions of graphs produced using corresponding models from Column 1. Average distortions per permutation are shown in Column 3. STD is the standard deviation of the distortion per permutation of vertices. Note that the distortion is somewhat depended on how we measure distances; however, the shapes of the distributions are very similar. (The cosine measures are capitalized for readability).  
deviations. Therefore, we conclude that the similarity between the shapes of the two graphs is extremely unlikely to be coincidental. Hence the model represents a non-trivial similarity. Moreover, we performed the same kind of analysis using different models, i.e., different embeddings and different distance measures. The table also shows experiments using the normalized search distance implemented in Solr ([[https://lucene.apache.org/solr/](https://lucene.apache.org/solr/)]([https://lucene.apache.org/solr/](https://lucene.apache.org/solr/)) (accessed on 24 February 2021)) and the distortion using a transformed _cosd_ distance. (Note that there is no natural transformation of WM distance applicable here). Additionally, while the distances and distortions change, the chances of similarities arising by accident are always smaller than 1/1000 (four standard deviations from the mean of distortions). By this standard statistical criterion, no matter what measures of distance we use, the similarity between two graphs, one from human analysis [3] and the other from automated concept modeling, is non-trivial and not accidental. This observation is amplified by the additional experiments reported in Appendix A. We conclude that vector-based representation are capable of detecting conceptual differences, i.e., the types and densities of concepts brought to the writing of medical recommendations.