# Optical Medieval Music Recognition Using Background Knowledge
## 2 Related Work  
The use of Deep Neural Network techniques has reported outstanding results in many areas related to computer vision, including OMR. [PERSON] et al. treated this problem as an object detection task [2; 3]. Their pipeline uses an existing deep convolutional neural network for object detection such as Faster R-CNN [4] or YOLO [5] and yields the bounding boxes of all detected objects along with their most likely class. Their dataset consists of 60 fully annotated pages from the 16 th-18 th century with 32 different classes written in menstrual notation. Typically, for object detection methods, the mean Average Precision (mAP) was used as the evaluation metric. A bounding box is considered as correctly detected if it overlaps 60% of the ground-truth box. Their proposed algorithm yielded 66% mAP. Further observation showed that especially small objects such as dots or rests posed a significant challenge to the network. Alternatively, the challenge can be approached as a sequence-to-sequence task [6; 7; 8; 9]. [PERSON] et al. [6] used a Convolutional Sequence-to-Sequence model to segment musical symbols. A common problem for segmentation tasks is the lack of ground truth. Here, the model is trained with the Connectionist Temporal Classification (CTC) loss function [10], which has the advantage that it is not necessary to provide information about the location of the symbols in the image; just pairs of input scores and their corresponding transcripts are enough. The model was trained on user generated scores from the MuseScore Sheet Music Archive (Musescore: [[https://musescore.org/](https://musescore.org/)]([https://musescore.org/](https://musescore.org/)), accessed on 30 May 2022) to predict a sequence of pitches and durations. In total, the dataset consisted of about 17,000 MusicXML scores, of which 60% were used for training. Using data augmentation, a pitch and duration accuracy of 81% and 94% was achieved, respectively. The note accuracy, where both pitch and duration are correctly predicted, was 80%. [PERSON] et al. [11] used Convolutional Neural Networks for automatic document processing of historical music images. Similar to our task, the corpus used for training and evaluation was rather small, consisting of 10 pages of the Salzinnes Antiphonal ([[https://cantus.simssa.ca/manuscript/133/](https://cantus.simssa.ca/manuscript/133/)]([https://cantus.simssa.ca/manuscript/133/](https://cantus.simssa.ca/manuscript/133/)), accessed on 30 May 2022) and 10 pages of the Einsiedeln ([[http://www.e-codecices.unifr.ch/en/sbe/0611/](http://www.e-codecices.unifr.ch/en/sbe/0611/)]([http://www.e-codecices.unifr.ch/en/sbe/0611/](http://www.e-codecices.unifr.ch/en/sbe/0611/)), accessed on 30 May 2022). Both of them are also written in square notation. The task was addressed as a classification task, where each pixel in the image was assigned one of four selected classes: text, symbol, staff line or background. The input to the network were small patches extracted from the pages centered at the point of interest. Each pixel was evaluated with the \(F_{1}\)-metric for the four classes. This yielded an average \(F_{1}\)-score of around 90%, while the music symbols' specific \(F_{1}\)-score was about 87.46%. Similar to that, [PERSON] et al. [12] also considered this as a segmentation task. A FCN was used to detect staff lines and symbols in documents. The staff line detection achieved a \(F_{1}\)-score of over 99%. For the symbol detection task, they achieved an \(F_{1}\)-score of over 96% if the type is ignored. By using a sequence-to-sequence metric, the so-called diplomatic symbol accuracy rate (dSAR) [12] (see Section 8.2), they reached an accuracy of about 87%. A similar approach can be observed in [13]. Other approaches use handcrafted features or heuristics for the symbol recognition task, such as in [14] using Kalman filters or [15; 16] using projections.