# Using Graph Embedding Techniques in Process-Oriented Case-Based Reasoning
## 6 Experimental Evaluation
### Experimental Setup  
We perform our experiments with an implementation of the presented approach in the Process-Oriented Case-Based Reasoning (POCBR) framework _ProCAKE_[24] and the Deep Learning (DL) framework _TensorFlow_[53]. The source code and all supplementary data can be retrieved with the instructions given in the _Data Availability Statement_ on page 22. The neural networks are trained with the help of TensorFlow and the similarity assessment is performed in ProCAKE. Our experiments examine two case bases from different domains that are both represented as semantic NEST graphs. The cooking processes (_CB-I_) contain 800 sandwich recipes with ingredients and cooking steps [3], split into 660 training cases, 60 validation cases, and 80 test cases. The processes of the data mining domain (_CB-II_) are built from sample processes that are delivered with RapidMiner (see [5] for more details), split into 509 training cases, 40 validation cases, and 60 test cases. A single training instance is a pair of graphs with the associated similarity value to learn, leading to \(660^{2}=435,600\) and \(509^{2}=259,081\) training instances, respectively. Thereby, the training case base is used as training input for the neural networks, while the validation case base is used to monitor the training process and to optimize hyperparameter values. Hyperparameter tuning is performed individually per domain with the two base models of sGEM and sGMN. The model variants then use the same hyperparameter settings as the associated base model, e.g., sGEM\({}_{\text{tree}}\) uses the same hyperparameter configuration as sGEM in the same domain. The training of all models is stopped as soon as the validation loss does not further decrease for two epochs (early stopping). The metrics that are used to evaluate our approach cover performance and quality. The performance is measured by taking the retrieval time, which is the entire retrieval time including pre-processing of the data if necessary. Since all variants of sGEM and EBM allow for caching of embedding vectors for the case base in an offline phase, these measures only embed the query during the experiments and do not include the caching time in the results. The quality of the results to evaluate \(RL_{\text{eval}}\) is measured by comparing them to the ground-truth retrieval results \(RL_{\text{true}}\) in terms of Mean Absolute Error (MAE), correctness (see [21; 55] for more details), and k-NN quality (see [19; 20; 21] for more details). The MAE (ranging between 0 and 1) expresses the average similarity error between all pairs of query graph and case graph in \(RL_{\text{true}}\) and the same pairs in \(RL_{\text{eval}}\). The correctness (ranging between \(-1\) and 1) describes the conformity of the ranking positions of the graph pairs in \(RL_{\text{eval}}\) according to \(RL_{\text{true}}\). Given two arbitrary graph pairs \(GP_{1}=(QG,CG_{1})\)and \(GP_{2}=(QG,CG_{2})\), the correctness is decreased if \(GP_{1}\) is ranked before \(GP_{2}\) in \(RL_{\text{eval}}\) although \(GP_{2}\) is ranked before \(GP_{1}\) in \(RL_{\text{true}}\) or vice versa. The k-NN quality (ranging between 0 and 1; see Equation (16)) quantifies the degree to which highly similar cases according to \(RL_{\text{true}}\) are present in \(RL_{\text{eval}}\). \[quality(QG,RL_{\text{true}},RL_{\text{eval}})=1-\frac{1}{\left\lvert RL_{\text{ true}}\right\rvert}\cdot\sum_{CG\in RL_{\text{true}}\setminus RL_{\text{eval}}}sim(QG,CG) \tag{16}\]  
Therefore, the cases from \(RL_{\text{true}}\) are compared with \(RL_{\text{eval}}\). Each case from \(RL_{\text{true}}\) that is missing in \(RL_{\text{eval}}\) decreases the quality, with highly relevant cases affecting the quality more strongly than less relevant cases. The machine that is used for training and testing computations is a PC with an Intel i7 6700 CPU (4 cores, 8 threads), an NVIDIA GTX 3070 GPU, and 48 GB of RAM, running Windows 10.