## 4 Evaluation
### Evaluation Metrics  
Evaluation metrics from the MIREX discovery of repeated themes and sections task were used for evaluation. This subsection provides a short description and formalization of the definitions found in the MIREX task definition [20]. The establishment measure (precision \(P_{est}\), recall \(R_{est}\) and F score \(F_{1 est}\)) evaluates the algorithm's ability to find at least one occurrence of each pattern shifted in time and pitch. Two occurrence measures \(F_{1 occ}\) evaluate the extent of the model's ability to find all pattern occurrences, where the \(c=\{0.5,0.75\}\) factor represents the inexactness tolerance threshold. [PERSON] [30] proposed an additional three-layer metric (\(P_{3}\), \(R_{3}\), \(TLF_{1}\)) that provides balance between the establishment and the occurrence measures. The exact precision, recall and F score measures (\(P\), \(R\), \(F_{1}\)) show the algorithm's performance in matching the found patterns with the reference annotations in an exact manner. The metrics are formally defined using the following set of symbols:  
* \(n_{P}\): the number of patterns in a ground truth
* \(\Pi=\{\mathcal{P}_{1},\mathcal{P}_{2},\ldots,\mathcal{P}_{n_{P}}\}\): a set of ground truth patterns
* \(\mathcal{P}=\{P_{1},P_{2},\ldots,P_{n_{P}}\}\)--occurrences of pattern \(\mathcal{P}\)
* \(n_{Q}\): the number of patterns in the algorithm's output
* \(\Xi=\{Q_{1},Q_{2},\ldots,Q_{n_{Q}}\}\): a set of patterns returned by the algorithm
* \(\mathcal{Q}=\{Q_{1},Q_{2},\ldots,Q_{n_{Q}}\}\)--occurrences of pattern \(\mathcal{Q}\). * \(k\): the number of ground truth patterns identified by the algorithm  
The standard precision is defined as \(P=k/n_{Q}\), the recall as \(R=k/n_{P}\), and the \(F_{1}\) score as \(F1=2\times P\times R/(P+R)\). Due to the extreme difficulty of discovering strictly exact patterns, more robust versions of the metrics are provided: the occurrence and the establishment scores. First, the cardinality score is used to determine the music similarity between the annotated and the discovered patterns:  
\[s_{c}(P_{i},Q_{j}):|P_{i}\cap Q_{j}|/\max\{|P_{i}|,|Q_{j}|\} \tag{14}\] \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Algorithm** & \(P_{est}\) & \(R_{est}\) & \(F_{test}\) & \(P_{occ(e=0.75)}\) & \(R_{occ(0.75)}\) & \(F_{1 occ(e=0.75)}\) \\ \hline
**SymCHM MIREX 2015** & 53.36 & 41.40 & 42.32 & 81.34 & 59.84 & 67.92 \\ NF1 MIREX 2014 & 50.06 & 54.42 & 50.22 & 59.72 & 32.88 & 40.86 \\ DM1 MIREX 2013 & 52.28 & 60.86 & 54.80 & 56.70 & 75.14 & 62.42 \\ OL1 MIREX 2015 & 61.66 & 56.10 & 49.76 & 87.90 & 75.98 & 80.66 \\ VM2 MIREX 2015 & 65.14 & 63.14 & 62.74 & 60.06 & 58.44 & 57.00 \\ SymCHM JKU PDD & 67.92 & 45.36 & 51.01 & 93.90 & 82.72 & 86.85 \\ SymCHMerge JKU PDD & 67.96 & 50.67 & 56.97 & 88.61 & 75.66 & 80.02 \\ \hline  & \(TLF_{1}\) & \(P_{occ(e=0.5)}\) & \(R_{occ(e=0.5)}\) & \(F_{1 occ(e=0.5)}\) & \(P\) & \(R\) & \(F_{1}\) \\ \hline
**SymCHM MIREX 2015** & 37.78 & 73.34 & 62.48 & 67.24 & 10.64 & 6.50 & 5.12 \\ NF1 MIREX 2014 & 33.28 & 54.98 & 33.40 & 40.80 & 1.54 & 5.00 & 2.36 \\ DM1 MIREX 2013 & 43.28 & 47.20 & 74.46 & 56.94 & 2.66 & 4.50 & 3.24 \\ OL1 MIREX 2015 & 42.72 & 78.78 & 71.08 & 74.50 & 16.0 & 23.74 & 12.36 \\ VM2 MIREX 2015 & 42.20 & 46.14 & 60.98 & 51.52 & 6.20 & 6.50 & 6.2 \\ SymCHM JKU PDD & 51.75 & 78.53 & 72.99 & 75.41 & 25.00 & 13.89 & 17.18 \\ SymCHMerge JKU PDD & 52.89 & 83.23 & 68.86 & 73.88 & 35.83 & 20.56 & 25.63 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluation of SymCHM, SymCHMerge and Music Information Retrieval Evaluation eXchange (MIREX) results of other proposed approaches for the discovery of repeated themes and sections task on the JKU Patterns Development Dataset (PDD) and JKU Patterns Testing Dataset (PTD), denoted as MIREX 2015. A score matrix is calculated based on the similarity as follows:  
\[s(\mathcal{P},\mathcal{Q})=