# CvAMoS--Event Abstraction Using Contextual Information
## 5 Evaluation
### Evaluation on Synthetic Datasets
To interpret the result of the algorithms, and obtain a valuable evaluation, we defined two metric values. To explain the metrics, a parallelism with _precision_ and _recall_ can be made. In the first metric, **Q1**, we identify the verified motifs (precision), i.e., the ability of the algorithm to produce a motif that perfectly fits the given pattern. The second metric, **Q2**, verifies that the number of verified motifs is the expected one (recall), i.e., it checks if the algorithm produces at least one trace that cannot be explained by the pattern. For simplicity reasons, we made Q1 and Q2 produce values of 1 or 0. The combination of these two values, which we call quality rate (**QR**), follows the idea of _accuracy_:  
\[QR=\frac{\sum_{patterns}Q1}{\sum_{patterns}Q1+\sum_{patterns}Q2}.\]  
We designed the metric so that QR has a value of 1 if exactly the expected patterns are identified as verified motifs and no extra ones; a value of 0 would indicate that none of the expected patterns are reflected in the verified motifs. Please note that we tested on all patterns, and for 6 of them, we have their version including noise, thus ending up with 14 total patterns (8 + 6 extended with noise). We also computed the total time required to conduct the experiments. The values have been calculated on an Intel Core i7-10710U 1.61 GHz CPU and 16 GB of RAM, equipped with Windows 10 Professional and Java 16.0.1, and the times are the result of the average of the time taken for each execution (i.e., for each pattern). Once the evaluation criteria were defined, we set three different configurations for the algorithms to be compared (R1-R3). To create fair configurations, we elicited similarities between the parameters of LPM and CvAMoS: the max _number of transitions_ in LPM can be associated with the _motif length_ in CvAMoS; the _min occurrence_ in LPM can be related to the _quorum_ in CvAMoS; the min _non-fitting events_ instead result in the _activity distance_. The _attribute distance_ is not considered in these configurations as the focus is only on the activities. The detailed configurations for R1, R2, and R3 are shown in Table 2.  
In the first run (R1), we used the default settings for LPM, in which the max _number of transitions_ is set to four out of five. In CvAMoS, the _motif length_ is equal to the length of the patterns to be identified. The number of results to produce is limited to one in LPM, in order to investigate the algorithm's capability to identify a perfect match on the first try. For the occurrences we had to make an assumption: in LPM, the value refers to the frequency of occurrence in the entire log, while the _quorum_ refers to the percentage of traces \begin{table}
\begin{tabular}{c l l l l} \hline \hline  & & **R1** & **R2** & **R3** \\ \hline \multirow{4}{*}{_LSTM_} & Max transitions & 4 & 5 & 5 \\  & No. of LPMs & 1 & 5 & 10 \\  & Min occurrences & 4 & 4 & 3 \\  & Min determinism & 0.5 & 0.5 & 0.5 \\  & Min non fitting ev. & F & 2 & 2 \\  & Projection & None & Markov & Markov \\  & Duplicate trans. & F & F & T \\ \hline \multirow{4}{*}{_LSTM_} & Motif length & 6 & 6 & 12 \\  & Quorum & 1 & 1 & 0.8 \\ \cline{1-1}  & Max activity distance & 0 & 1 & 2 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Algorithm configurations for the identification evaluation. in which the pattern appears. Hence, _min occurrences_ is set equal to the amount of traces in each log. The _non-fitting events_ and the _activity distance_ are both set to zero. The following run (R2) foresees relaxation in the settings. In LPM, a higher number of models is produced. In addition, the _projection_ is changed to the Markov clustering as the authors in the paper claim it has better performance in terms of computational complexity. On the other hand, CvAMoS increases the _activity distance_ value. The third run (R3) has parameters set toward optimization for both algorithms. The LPM can now represent duplicate transition, as well as the number of expected models is doubled.