# Using Graph Embedding Techniques in Process-Oriented Case-Based Reasoning
## 6 Experimental Evaluation  
We evaluate the Semantic Graph Embedding Model (sGEM) and the Semantic Graph Matching Network (sGMN) by applying them as similarity measures in standard retrieval and MAC/FAC retrieval scenarios. Thereby, we compare different variants of the embedding models and different retrieval scenarios. The variants are marked with subscripts in our terminology, with a total of six different variants:  
\begin{tabular}{l l}
**sGEM** & sGEM used with sequence encoding; \\
**sGEMtree** & sGEM used with tree encoding; \\
**sGMN** & sGMN used with sequence encoding; \\
**sGMNtree** & sGMN used with tree encoding; \\
**sGMNconst** & sGMN used with matching constraints; \\
**sGMNtree,const** & sGMN used with tree encoding and matching constraints. \\ \end{tabular}  
As ground-truth similarities for training the neural networks and comparing their predictions, we use similarities computed by the A*-similarity measure by [PERSON] and [PERSON] [7] (A*M; see Section 2.2). The ground-truth similarities are computed with a configuration of the measure that uses a very high parameter value for MaxPMS, to allow a good solution to be found in the mapping process. Additionally, we also use this measure with a lower value of MaxPMS in the evaluation to measure the deviations from the ground-truth similarities. This allows us to see how much the computed similarities of A*M deviate from the ground-truth similarities if MaxPMS is reduced. Another evaluated measure is the feature-based measure by [PERSON] and [PERSON] [12] (**FBM**), which uses a manually modeled feature representation of NEST graphs, specifically designed to be used in combination with a lightweight similarity measure in MAC/FAC retrieval situations. It is included in order to achieve a comparison between automatically learned measures and manually modeled ones, but comparison with this measure is not the main focus. Furthermore, the embedding-based measure of [PERSON] et al. [19] (**EBM**) is evaluated. It is also designed for MAC/FAC retrieval and utilizes learned graph embeddings and a vector-based similarity (see Section 2.4 for more details). EBM will be the main measure to comparesGEM and sGMN against since it is also automatically learned and embedding-based. We investigate the following hypotheses in two experiments:  
**Hypothesis 1 (H1).**_Using the sGEM variants as MAC similarity measures of a MAC/FAC retrieval leads to improved retrieval results compared to using EBM as a MAC similarity measure;_  
**Hypothesis 2 (H2).**_The sGMN variants can approximate the ground-truth graph similarities better than A*M, considering a reduced MaxPMS value such that the retrieval time of both retrievers is comparable._  
Each hypothesis refers to a dedicated experiment. The first experiment examines the measures in a MAC/FAC setup, where the focus is placed on the suitability of sGEM as a MAC similarity measure. We focus on evaluating sGEM against the other automatically learned similarity measure, EBM, in H1 as it is, by design, more optimized for performance, which fits the requirements of a MAC similarity measure (see Section 5.2). The second experiment examines the degree to which sGMN is capable of approximating the ground-truth A* similarities. As introduced before, A*M can be configured (by adjusting the MaxPMS parameter) to compute solutions faster with lower quality in return. H2 aims at comparing sGMN to a configuration of A*M with a reduced value of MaxPMS where the deviations from the ground-truth similarities are comparable.