## 3 Evaluation
### Results
For the barbershop quartets, however, the performance increases until Iteration 20, before decreasing slightly until the end of the process. This slight decrease in performance over the final 10 iterations is due to the alto and soprano voices: \(F_{b}\) and \(F_{t}\) each remain stable over the final 10 iterations, while \(F_{a}\) and \(F_{s}\) each decrease. This difference is likely explained by the acoustic model not being able to properly decompose the alto and soprano voices. The barbershop quartets have no true female voices (i.e., each part is sung by a male vocalist), but the template dictionary's alto and soprano voices are sung by female vocalists; thus, the alto and soprano parts must be estimated through a rough approximation of a spectral basis combination of female voices. Such a rough approximation could be the cause of our model's difficulty in decomposing the alto and soprano voices in the barbershop quartets. \begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **Bach Chorales** & **Barbershop Quartets** \\ \hline [PERSON] [49] & 54.62 (3.00) & 48.24 (4.50) \\ [PERSON] and [PERSON] [50] & 49.52 (5.18) & 45.22 (6.94) \\ [PERSON] et al. [4] & 53.58 (6.27) & 51.04 (8.52) \\ [PERSON] and [PERSON] [51] & 67.19 (3.82) & 63.85 (6.69) \\ [PERSON] and [PERSON] [21] & 71.03 (3.33) & 70.84 (6.17) \\ VOCAL4-MP & 63.05 (3.12) & 59.09 (5.07) \\ VOCAL4-VA & **71.76 (3.51)** & **75.70 (6.18)** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Multi-pitch detection results, where standard deviations are shown in parentheses. The post-processing refinement step described in Section 2.1.1 was also run on the output of all cited methods. VOCAL4-MP represents our proposed method with the acoustic model only, while VOCAL4-VA refers to our fully-integrated model. Figure 6 illustrates the output of our proposed system, run on excerpts from both the Bach chorale (a, left) and barbershop quartet (b, right) datasets, for the joint multi-pitch detection and voice assignment tasks. Figure 6(a1),(b1) show the ground truth, using colour to denote vocal part; Figure 6(a2),(b2) show the probabilistic pitch detections from the acoustic model after the 30 th EM iteration, summed over all voices (\(\sum_{p=1}^{5}P_{t}(p|v)\)), where a darker shade of gray indicates a greater probability; Figure 6(a3),(b3) present the final output of the integrated system, again using colour to denote vocal part. As mentioned earlier, the bass voice assignment outperforms all other voice assignments in almost all cases, since false positive pitch detections from the acoustic model often correspond with overtones from lower notes that occur in the same pitch range as the correct notes from higher voices. These overtone errors are most commonly found in the soprano voice, for example at around 105 seconds in the [PERSON] chorale excerpt and around 64.5 seconds in the barbershop quartet excerpt, where Figure 6(a2),(b2) clearly show high probabilities for these overtones. It is clear from Figure 6(a3),(b3) that such overtone errors in the soprano voice also lead to voice assignment errors in the lower voices since our system can now assign the correct soprano pitch detections to the alto voice, alto to tenor and tenor to bass. Another common source of errors (for both multi-pitch detection and voice assignment) is vibrato. The acoustic model can have trouble detecting vibrato, and the music language model prefers voices with constant pitch over voices alternating between two pitches, leading to many off-by-one errors in pitch detection. Such errors are evident throughout the [PERSON] chorale excerpt, particularly in the tenor voice towards the beginning where our system detects mostly constant pitches (both in the acoustic model output and the final output) while the ground truth contains some vibrato. Furthermore, at the end of both excerpts, there is vibrato present, and our system simply detects no pitches rather than the vibrato.