# Phoneme Recognition in Korean Singing Voices Using Self-Supervised English Speech Representations
## 4 Experiments
### Experimental Details
#### 4.2.3 Training Setup  
Two fine-tuning strategies were used. The linear probing mentioned below indicates that freezing the pre-trained HuBERT encoder for the whole training time only updates the additional predictor to make the most of the priors. Correspondingly, the full fine-tuning mentioned below indicates updating the additional predictor with fixed encoder weights on a few earlier epochs, and then updating the encoder weights together with the predictor on the following epochs to make the best of the current data. Essentially, we followed the default training settings provided by the Fairseq toolkit, version 0.12.2. All models were trained with an Adam optimizer [40]. The hyper parameters not mentioned below are consistent with the toolkit defaults. We aimed to maximize the use of priors, and due to the scale of our datasets, the learning rate was set to 0.00002 for all phoneme recognition training processes. The pre-trained encoder was fixed for the first 10,000 updates using tri-stage learning rate scheduling [41] for the phoneme recognition training, except for the training processes for the first two rows of the results in Table 2. The learning rate for linear probing in the second row of the COn results in Table 3 was 0.000001. The GPU model used for computation was RTX 3090 Ti.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Prior**} & \multirow{2}{*}{
\begin{tabular}{c} **Fine-Tuning** \\ **Strategy** \\ \end{tabular} } & \multicolumn{4}{c}{**Error Rate (\%)**} \\ \cline{3-8}  & & **Phoneme** \(\downarrow\) & **Syllable** \(\downarrow\) & **Consonant** \(\downarrow\) & **Vowel** \(\downarrow\) \\ \hline None & Full & 38.63 & 73.44 & 32.60 & 29.86 \\ En & Linear Probe & 21.28 & 49.43 & 17.99 & 18.48 \\ En & Full & **7.44** & **19.66** & **5.77** & **6.34** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of English–Korean adaptation results. The downward arrow signifies that lower values are preferable.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{6}{c}{**Error Rate (\%) \(\downarrow\)**} & \multicolumn{6}{c}{**COn**} \\ \cline{2-10}  & \multicolumn{6}{c}{**CSD Corpus**} & \multicolumn{6}{c}{**NIKLSEOUL Corpus**} & \multicolumn{6}{c}{**CSD Corpus**} \\ \cline{2-10}  & **Phoneme** & **Syllable** & **Consonant** & **Vowel** & **Phoneme** & **Syllable** & **Consonant** & **Vowel** & **Precision** (\%) \(\uparrow\) & **Recall** (\%) \(\uparrow\) & **F1-Score** (\%) \(\uparrow\) & **(\%) \(\uparrow\) \\ \hline A \({}^{1}\) & 17.64 & 40.93 & 15.71 & 15.13 & 34.74 & 74.96 & 26.94 & 31.39 & \(\setminus\) & \(\setminus\) & \(\setminus\) \\ \hline B \({}^{2}\) & 17.76 & 41.08 & 16.36 & 13.69 & **14.30** & **35.80** & **12.09** & **11.17** & **31.60** & 81.28 & 44.93 \\ C \({}^{3}\) & **17.09** & **39.48** & **15.32** & **13.50** & 15.68 & 38.62 & 13.30 & 12.67 & 31.48 & **82.89** & **45.13** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of speech–singing adaptation results. The upward arrow signifies that higher values are preferable, whereas the downward arrow signifies that lower values are preferable.  
Figure 7: Pitch distribution of the CSD corpus.