# Large-Scale Multimodal Piano Music Identification Using Marketplace Fingerprinting +
## 3 Results
### Camera-Based Sheet Music Retrieval  
We compare the results of seven different systems. The first five systems use fixed n-gram fingerprints based on bootleg score features. This approach comes from [27], which proposes using each column of a bootleg score representation as a fingerprint (i.e., a fixed 1-gram), and we also include the straightforward generalization of this strategy to 2-grams, 3-grams, 4-grams, and 5-grams. The sixth system uses dynamic n-gram fingerprints [28; 33] based on bootleg score features, which is the current state of the art for sheet music retrieval. In the dynamic n-gram approach, the database contains multiple types of fixed n-gram fingerprints (e.g., 1-grams, 2-grams, 3-grams, and 4-grams), and at runtime, the size of each individual n-gram (query) fingerprint is selected dynamically to ensure that the fingerprint is sufficiently discriminative to warrant a table lookup. The seventh system is our proposed marketplace fingerprinting system.  
Table 1 shows results on the camera-based sheet music retrieval task. To gain further insight, we evaluate performance under two different conditions. Condition 1 is when the original PDF (that the cell phone image is taken of) exists in the database. Condition 2 is when the original PDF is not in the database but an alternate edition of the same piece exists in the database. We can simulate the second condition by simply removing the exact matching PDF from the database. This means that queries for which there is only a single PDF available in IMSLP are removed from the test set under condition 2. For experiments under the second condition, there are a total of 370 train queries and 1560 test queries. Note that each row of the table corresponds to a single experiment, and the MRRs for the two conditions are determined in the evaluation phase.  
There are three things to notice about Table 1. First, the marketplace system substantially outperforms all other systems in retrieval accuracy. For example, the marketplace approach achieves 0.91 and 0.89 MRR under conditions 1 and 2, respectively, while the dynamic n-gram approach achieves 0.85 and 0.82 MRR. Second, the marketplace system has the lowest average runtime of all systems. The benefit of the marketplace approach is not simply a tradeoff between runtime and retrieval accuracy: it significantly improves retrieval accuracy while simultaneously reducing or maintaining average runtime. It is therefore a unilaterally better solution than the dynamic n-gram approach. Third, there is a  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**System**} & \multicolumn{2}{c}{**MRR**} & \multicolumn{2}{c}{**Runtime**} \\  & **Cond 1** & **Cond 2** & **Avg** & **Std** \\ \hline
1-gram [27] & 0.71 & 0.66 & 21.5 s & 12.5 s \\
2-gram & 0.85 & 0.82 & 2.76 s & 1.11 s \\
3-gram & 0.81 & 0.76 & 1.99 s & 0.36 s \\
4-gram & 0.76 & 0.74 & 1.12 s & 0.25 s \\
5-gram & 0.69 & 0.67 & 1.07 s & 0.13 s \\ Dynamic [33] & 0.85 & 0.82 & 0.98 s & 0.12 s \\ \hline Marketplace & 0.91 & 0.89 & 0.95 s & 0.14 s \\ \hline \hline \end{tabular}
\end{table}
Table 1: Experimental results on the camera-based sheet music retrieval task. Condition 1 assumes that the exact same PDF is in the database, while condition 2 assumes that only alternate printed editions of the same piece exist in the database.  
small decrease in MRR if we remove the original PDF from the database (Condition 2). This indicates that the system is relatively insensitive to differences in various printed editions.