# Applying Automatic Translation for Optical Music Recognition's Encoding Step
## 4 Results
### Experimental Results  
The results presented in the Table 3 strongly depend on the corpus used for evaluation. As can be seen, for all the three approaches evaluated, the best results were obtained for the Camera-PrIMuS corpus, for which the transformer outperformed the Seq2 Seq-Attn architecture by 1.5% SER, and the SMT approach by more than 1% SER. On the other end of performance, the worst results were obtained for the Zaragoza corpus, with which the SMT approach outperformed the transformer and Seq2 Seq-Attn architectures by 51% SER and 56% SER, respectively. In the case of FMT, the Seq2 Seq-Attn and the SMT approaches obtained almost the same result, and outperformed the transformer by about 5% SER. Regarding the SMT method, it should be noted that the results obtained differ noticeably depending on the type of agnostic notation used in the different corpora. For the Camera-PrIMuS and the Zaragoza corpora, the SER increased dramatically when contextual information was only provided at the beginning of the segment (standard agnostic notation). This result was expected, and confirms the hypothesis that when context is relevant to translating a symbol that appears far from the contextual information (i.e., the clef and key signature), this approach is unable to translate it properly. However, this limitation is overcome in the neural approaches through the attention mechanism. The only exceptions in this regard are the results obtained for the FMT corpus, which showed very similar results for both the contextual and the non-contextual agnostic notations when applying the statistical approach. The explanation for these contradictory results lies in two key aspects related to the addition of contextual information to the agnostic notation: the number of unknown symbols in the test set (see Table 1) and the improvement of the performance for specific types of errors related to contextual information (see Table 4). \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Camera-PrIMuS** & **FMT** & **Zaragoza** \\ \hline SMT & 23.7 \(\pm\) 0.3\% & **9.6 \(\pm\) 3.8\%** & 69 \(\pm\) 12\% \\ Seq2 Seq-Attn & 2.04 \(\pm\) 0.09\% & 9.8 \(\pm\) 3.2\% & 89.5 \(\pm\) 1.6\% \\ The Transformer & **0.50 \(\pm\) 0.06\%** & 15.3 \(\pm\) 3.9\% & 86.3 \(\pm\) 5.1\% \\ \hline SMT Contextual & 1.58 \(\pm\) 0.05\% & 11.1 \(\pm\) 4.3\% & **35 \(\pm\) 16\%** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Results obtained in terms of average SER and standard deviation on the 10-fold cross-validation samples for the three corpora used for evaluation. Results for the three approaches evaluated on the task of translating standard agnostic input into **“_kern_\({}^{*}\) are reported. Additional results are presented in the last row using SMT to translate contextual agnostic input into **“_kern_\({}^{*}\). As discussed in Section 4.3 below, the improvement in the translation accuracy for FMT when using contextual agnostic notation was more modest than for Zaragoza and Camera-PrIMuS. However, the ratio of unknown words in the FMT test set increased from 0.13 to 0.28. As a result, FMT is the only case in which using contextual agnostic notation not only did not improve the final result, but even slightly worsened it. Regarding the two neural approaches evaluated, the transformer clearly obtained the best results on the Camera-PrIMuS corpus. However, none of the neural approaches seemed to outperform the SMT approach for the FMT and Zaragoza corpora. This may be explained by the fact that these two corpora are substantially smaller than Camera-PrIMuS, and neural approaches do not seem to be able to learn enough from small amounts of training data. To test this hypothesis, we ran an additional experiment by taking subsamples of 1000 to 5000 segment pairs from the Camera-PrIMuS corpus and used them to evaluate the SMT approach (the best performance on small corpora) and the transformer approach (the best performance on Camera-PrIMuS).