# Using Graph Embedding Techniques in Process-Oriented Case-Based Reasoning
## 6 Experimental Evaluation
### Experimental Results  
The first experiment aims to answer hypothesis H1 and evaluates the variants of sGEM and sGMN as MAC similarity measures in a scenario of MAC/FAC retrieval. Different combinations of _fs_, i.e., the number of candidates cases of the MAC phase, and \(k\), i.e., the number of retrieval results from the FAC phase, are examined. These values are chosen to be similar to the experiments in previous work [19]. The FAC similarity measure is an A* measure that is used in the same configuration in every retrieval. Table 1 shows the evaluation results, which include the metrics k-NN quality and retrieval time (in milliseconds). Besides the variants of the neural network models, we also evaluate FBM and EBM since these two measures are specifically designed for MAC/FAC applications. The highlighted values represent the maximum quality and minimum time, respectively, for each combination of _fs_ and \(k\). \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c c c} \hline \hline  & & & **sGEM** & **sGEMtree** & **sGMN** & **sGMNtree** & **sGMNconst** & **sGMNtree,const** & **FBM** & & **EBM** \\  & **fs** & **k** & **Quality** & **Time** & **Quality** & **Time** & **Quality** & **Time** & **Quality** & **Time** & **Quality** & **Time** & **Quality** & **Time** & **Quality** & **Time** & **Quality** & **Time** \\ \hline \multirow{8}{*}{**MAC**} & 5 & 5 & 0.508 & 16 & 0.511 & **14** & 0.489 & 1051 & 0.490 & 2067 & 0.489 & 1074 & 0.489 & 2169 & **0.557** & 510 & 0.499 & 17 \\  & 50 & 5 & 0.613 & 100 & 0.600 & **95** & 0.522 & 1137 & 0.523 &  
The results for CB-I show that the two variants of sGEM perform similarly, with similar quality values and retrieval times. The same applies to the quality values of the sGMN variants. The retrieval times of the sGMN variants reveal a negative influence of the variants that use tree encoding, which is likely caused by the more complex embedding process. The sGMN variants and the sGEM variants perform similarly in terms of quality, despite the more expressive embedding process of sGMN. EBM shows quality values that are on par with the values of the sGEM variants. FBM shows the highest quality values across all combinations of _fs_ and \(k\). The retrieval times are higher than those of the sGEM variants and EBM but lower than those of the sGMN variants. The results for CB-II present a similar picture. FBM still outperforms all other measures with regard to quality. However, some sGMN variants, such as sGMN\({}_{\text{tree,const}}\), are performing in a similar range. It is also important to note that the sGMN variants with tree encoding are still the slowest measures but outperform all other automatically learned measures for CB-II. The sGMN variants also outperform the sGEM variants for almost all combinations of _fs_ and \(k\), which shows that the variants of sGMN favor retrieval situations with more complex semantic descriptions of task and data nodes, as present in CB-II. The performance of the sGEM and sGMN measures for retrieving graphs from a rather simple domain, such as in CB-I, is respectable but does not consistently surpass the currently available, automatically learned approach of EBM. When only looking at the automatically learned measures in the results of CB-II, i.e., the variants of sGEM, the variants of sGMN, and EBM, sGEM variants are the most suitable for a MAC/FAC scenario since they show a good combination of very low retrieval times and high quality values. The FBM, with its manually modeled similarity measure, still performs best for both case bases, taking into account the combination of quality and time. However, under conditions of strict time requirements, sGEM or EBM can become better suited than FBM. Due to their speed, the measures allow for the performance of a MAC phase with high values of _fs_, which, in turn, produce better overall quality results for the MAC/FAC retrieval.