## 10 Effects of Training Visual Units for Phoneme Classifiers  
Here we present the results of our proposed hierarchical training method (described in Section 4 with two different language models. Figure 9 shows the mean correctness, \(C\), for all 12 speakers over 10 folds. We have plotted four symbols, one for each of the pairings of our HMM unit labels and the language network unit ({visual units and phonemes, visual units and words, phonemes and phonemes, phonemes and words}). Random guessing is plotted in orange. The \(x\)-axis of Figure 9 is the size of the optimal visual unit sets from Figure 5, from 11 to 36. This is the range of optimal number of visual units where phoneme label classifiers do not improve classification. The baseline of visual unit classification with a word network from [31] is shown in blue and is not significantly different from conventionally learned phoneme classifiers. Based on our language network study in Section 9, it is not a surprise to see just by using a phoneme network instead of a word network to support visual unit classification we significantly improve our mean correctness score for all visual unit set sizes for all speakers (shown in pink). We have plotted weighted guessing in orange. More interesting to see is our new weakly trained phoneme HMMs are significantly better than the visual unit HMMs. In the first part of our work here phoneme HMMs gave an all-speaker mean \(C=0.059\) and was not significantly different from the best visual units. Here, regardless of the size of the original visual unit set, \(C\) is almost double. Weakly learned phoneme classifiers with a word network gain 0.031 to 0.040 in mean \(C\), and when these phoneme classifiers are supported with a phoneme network we see a correctness gain range from 0.17 to 0.18. These gains are supported by the all-speaker mean minimum and maximums listed in Table 9. These gain scores are from over all the potential P2V mappings and show there is little difference in which P2V map is best for knowing which set of visual units to initialize our phoneme classifiers. All results are significantly better than guessing. \begin{table}
\begin{tabular}{l l r r} \hline \hline
**Classifier units** & **Network units** & \(C_{w}\) & **1 se** \\ \hline Viseme & Viseme & 0.02 & 0.0063 \\ \hline Viseme & Phoneme & 0.19 & 0.0036 \\ Phoneme & Phoneme & 0.19 & 0.0036 \\ \hline Viseme & Word & 0.09 & 0.0 \\ Phoneme & Word & 0.20 & 0.0043 \\ Word & Word & 0.19 & 0.0005 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Unit selection pairs for HMMs and language network combinations, and the all-speaker mean \(C_{w}\) achieved. \begin{table}
\begin{tabular}{l r r r} \hline \hline  & **Min** & **Max** & **Range** \\ \hline visual units + word net & 0.03 & 0.06 & 0.03 \\ Phonemes + word net & 0.09 & 0.10 & 0.01 \\ Effect of WLT & 0.06 & 0.04 & – \\ visual units + phoneme net & 0.20 & 0.22 & 0.02 \\ Phonemes + phoneme net & 0.26 & 0.24 & 0.01 \\ Effect of WLT & 0.02 & 0.02 & – \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Minimum and maximum all-speaker mean correctness, \(C\), showing the effect of hierarchical training from visual units on phoneme-labeled HMM classification. In Figures 10-13, we have plotted for each of our 12 speakers non-aggregated results showing \(C\pm\) one standard error. While not monotonic, these graphs are much smoother than the speaker-dependent graphs shown in Appendix A. The significant differences between visual unit set sizes (in Figure 5) have now disappeared because the learning of differences between visual units, has been incorporated into the training of phoneme classifiers, which in turn are now better trained (plotted in red and green which improve on blue and pink respectively). An intriguing observation is comparing the use of a phoneme network for visual units and for weakly taught phonemes. For some speakers, the weakly learned phonemes are not always as important as having the right network unit.