Automatic Mapping of Scanned Sheet Music to Audio Recordings

ABSTRACTSignic ant digitization efforts have resulted in large mul ti-modal music collections comprising visual (scanned sheetmusic) as well as acoustic material (audio recordings). In this paper, we present a novel procedure for mappingscanned pages of sheet music to a given collection of au-dio recordings by identifying musically corresponding au-dioclips. Tothisend,boththescannedimagesaswellastheaudiorecordingsarer sttransformedintoacommonfeaturerepresentation using optical music recognition (OMR) andmethods from digital signal processing, respectively. Basedon this common representation, a direct comparison of thetwo different types of data is facilitated. This allows for asearch of scan-based queries in the audio collection. We re-port on systematic experiments conducted on the corpus ofBeethoven's piano sonatas showing that our mapping pro-cedure works with high precision across the two types ofmusic data in the case that there are no severe OMR errors.The proposed mapping procedure is relevant in a real-worldapplication scenario at the Bavarian State Library for auto-maticallyidentifyingandannotatingscannedsheetmusicbymeans of already available annotated audio material.1 INTRODUCTIONThe last years have seen increasing efforts in building uplarge digital music collections. These collections typicallycontain various types of data ranging from audio data suchasCDrecordingstoimagedatasuchasscannedsheetmusic,thus concerning both the auditorial and the visual modal-ities. In view of multimodal searching, navigation, andbrowsing applications across the various types of data, onerequires powerful tools that support the process of analyz-ing,correlating,andannotatingtheavailablematerial. Inthecase of digitized audio recordings, r st services have beenestablished to automate the annotation process by identify-ing each recording and assigning available metadata such astitle, artist, or lyrics. Here, the metadata is drawn from spe-cialized annotation databases provided by commercial ser-vices such as Gracenote [6] or DE-PARCON [9]. Opposed to acoustic music data, which is increasinglyavailable in digital formats, most sheet music is still pro-duced and sold in printed form. In the last years, dig-ital music libraries have started to systematically digitizetheir holdings of sheet music resulting in a large numberof scanned raster images. To make the raw image dataavailable to content-based retrieval and browsing, meth-ods for automatically extracting and annotating semanti-cally meaningful entities contained in the scanned docu-ments are needed. In this context, optical music recogni-tion (OMR) [3] is a key task. Here, the goal is to convertscannedsheetmusicintoacomputerreadablesymbolicmu-sic format such as MIDI or MusicXML [13]. Even thoughsignic ant progress has been made in the last years, cur-rentOMRalgorithmsaresubstantiallyerror-prone,resultingin systematic errors that require subsequent correction [2]. Similarly, there is still a high demand for reliable solutionsfor the more general task of automatic sheet music annota-tion in the digital library community.In this paper, we present a novel approach for automat-ically annotating scanned pages of sheet music with meta-data. Ourapproachisbasedonanewprocedureformappingthe scanned sheet music pages to an existing collection ofannotated audio recordings. The mapping allows for iden-tifying and subsequently annotating the scans based on themetadata and annotations that are already available for theaudio recordings. In particular, as it is the case in the spe-cic application scenario at the Bavarian State Library, weassume the existence of an audio collection containing an-notated digitized audio recordings for all pieces to be con-sidered in the sheet music digitization process. The con-version of both the audio recordings (by employing l teringmethods) and the scanned images (by employing OMR) toa common feature representation allows for a direct com-parison of the two different types of data. Using the featuresequence obtained from a few consecutive staves or an en-tire page of the scanned sheet music as query, we computethe top match within the documents of the audio database.The top match typically lies within a musically correspond-ing audio recording, which then allows for identifying thescanned page and for transferring all available audio anno-