# SMM: Leveraging Metadata for Contextually Salient Multi-Variate Motif Discovery
## 5 Experimental Evaluation
### Evaluation Criteria  
To evaluate the identified motifs, we consider both temporal and variate alignments.  
**Definition 12** (Variate Overlap).: _Given two multivariate subsequences \(S_{1}(v_{1},t_{1})\) and \(S_{2}(v_{2},t_{2})\) from a multivariate time series \(T(v,t)\), where \(v_{1},v_{2}\subseteq v\), the variate overlap between \(S_{1}(v_{1},t_{1})\) and \(S_{2}(v_{2},t_{2})\) is \(VO(S_{1},S_{2})=\frac{|v_{1}\cap v_{2}|}{|v_{1}\cup v_{2}|}\)._  
**Definition 13** (Time Overlap).: _Given two multivariate subsequences \(S_{1}(v_{1},t_{1})\) and \(S_{2}(v_{2},t_{2})\) from a multivariate time series \(T(v,t)\), where \(t_{1},t_{2}\) and \(t\) are set of consecutive temporal intervals such that \(t_{1},t_{2}\subseteq v\), the time overlap between \(S_{1}(v_{1},t_{1})\) and \(S_{2}(v_{2},t_{2})\) is \(TO(S_{1},S_{2})=\frac{|t_{1}\cap t_{2}|}{|t_{1}\cup t_{2}|}\)._  
**Definition 14** (Instance Overlap).: _Given two multivariate subsequences \(S_{1}(v_{1},t_{1})\) and \(S_{2}(v_{2},t_{2})\) from a multivariate time series \(T(v,t)\), the instance overlap between \(S_{1}(v_{1},t_{1})\) and \(S_{2}(v_{2},t_{2})\) is \(IO(S_{1},S_{2})=TO(S_{1},S_{2})*VO(S_{1},S_{2})\)._  
To evaluate the motif extraction algorithms, we use _precision_, _recall_ and \(f\)-score accuracies. Let a time series have \(m\) ground truth motifs (\(M_{1}\) through \(M_{m}\)) inserted as ground truth, each with \(k\) recurring subsequences (\(M_{i}=\{ms_{i,1},\dots,ms_{i,k}\}\)). Let also the motif search identify \(c\) clusters (\(C_{1}\) through \(C_{c}\)), where \(C_{j}=\{cs_{j,1},\dots,cs_{j,|C_{j}|}\}\).  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Values** \\ \hline number of motifs (\(m\)) & **1**, 2**, 3 \\ recurrence count (\(k\)) & 5**, 10**, 15 \\ length flexibility percentage (\(f\)) & 50**, 75**, 100 \\ average displacement scaling percentage & **10**, 50**, 75**, 100 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Experimental settings; default values are highlighted in bold.  
**Definition 15** (Cluster-Motif Precision).: _This is defined as the ratio of subsequences in a cluster \(C_{j}\), corresponding to motif \(M_{i}\):_  
\[prec_{i,j}=\big{(}\sum_{cs\in C_{j}}in(cs,M_{i})\big{)}\,/\,|C_{j}|.\]  
**Definition 16** (Cluster-Motif Recall).: _This is defined as the ratio of subsequences in motif \(M_{i}\) that are in cluster \(C_{j}\):_  
\[recall_{i,j}=\big{(}\sum_{ms\in M_{i}}in(ms,C_{j})\big{)}\,/\,|M_{i}|.\]  
Above, \(in()\) is a function which checks if a given subsequence occurs in a given set of subsequences and returns a value between 0 and 1, depending on how good a match is located (for simplicity of the discussion, we assume that \(in()\in\{0,1\})\). Given these, we define the \(fscore\) for a cluster-motif pair.  
**Definition 17** (F-Score for a Cluster-Motif Pair).: _The f-score for the cluster \(C_{j}\) and the motif \(M_{i}\) is defined as_  
\[fscore_{i,j}=(2\times prec_{i,j}\times recall_{i,j})\,/\,(prec_{i,j}+recall_{i,j}).\]  
Note that our goal is that (a) each motif occurs in one and only one cluster and (b) each cluster contains recurring patterns from one motif. Therefore, given the above, we define the overall accuracy relative to motif \(M_{i}\) as follows:  
\[acc_{i}=\frac{\sum_{C_{j}\text{ s.t. }recall_{i,j}\
eq 0}fscore_{i,j}}{count(\{C_{ j}\mid recall_{i,j}\
eq 0\})}.\]  
The overall accuracy is the average accuracy for all inserted motifs:  
\[acc=\frac{AVG}{1\leq i\leq m}(acc_{i}).\]  
In the results presented in the next subsection, any cluster that does not include _at least one_ multi-variate subsequence with a minimum degree (\(50\%,75\%,90\%,\) or \(100\%\)) of overlap with a true motif pattern is ignored as being a cluster that does not relate to the ground-truth motif.  
Each experiment was executed on 40 random settings and the averages are reported.