# Hierarchical Phoneme Classification for Improved Speech Recognition
## 2 Phoneme Clustering
### Phoneme Clustering Using Confusion Matrix  
We clustered phonemes using the similarities defined by the sum of the off-diagonal elements in the confusion matrix as shown in Figure 1. We use an agglomerative method based on bottom up clustering, which starts from a unique cluster and ties up a pair one at a time [40]. To define the distance between a phoneme pair, we use the confusion error rates, obtained by one of the off-diagonal elements of the confusion matrix. For a newly created phoneme cluster, the new error rate is computed by the average of the merged phonemes. The process stops until the desired number of clusters remain. The results of hierarchical clustering are usually presented in a dendrogram [41]. As shown in Figure 2, the dendrogram is a tree diagram representing clusters obtained from hierarchical clustering. The \(y\)-axis shows phoneme labels and the phonemes are hierarchically paired with the height of the pair connection being the misclassification rate between the phonemes \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**No** & **ARPabet** & **No** & **ARPabet** & **No** & **ARPabet** & **No** & **ARPabet** \\ \hline
1 & iy & 11 & oy & 21 & ng eng & 31 & b \\
2 & ih ix & 12 & aw & 22 & v & 32 & p \\
3 & eh & 13 & ow & 23 & f & 33 & d \\
4 & ae & 14 & er axr & 24 & dh & 34 & dx \\
5 & ax ah & 15 & l el & 25 & th & 35 & t \\
6 & uw ax & 16 & r & 26 & z & 36 & g \\
7 & uh & 17 & w & 27 & s & 37 & k \\
8 & ao aa & 18 & y & 28 & zh sh & 38 & h hv \\
9 & ey & 19 & m em & 29 & jh & 39 & bcl pcl \\  & & & & & & & & & \\  & & & & & & & & \\
10 & ay & 20 & n en nx & 30 & ch & & epi pau \\  & & & & & & & & h \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Mapping 61 phonemes to 39 phonemes for phoneme recognition. The grouping is based on the linguistic similarities between phonemes [33]. in the corresponding pair. The misclassification rate range from 0.0 to 1.0 along the \(x\)-axis. This method is called hierarchical clustering because paired phonemes are considered as a single phoneme after the pair is merged and all phonemes finally converge to a single cluster. The goal of the the proposed method is to improve the performance of phoneme recognition. Therefore, for the confusing phonemes, other classification method that is different from the baseline phoneme classifier is applied to further distinguish the phonemes in the same cluster. Figure 1: Confusion matrix from the results of the baseline phoneme classification. Each row of the matrix represents the instances in a truth and each column represents the instances in a predicted class. Figure 2: Dendrogram of clustering. The \(x\)-axis shows a similarity of the clustered nodes. The higher the \(x\)-value, the more similar they are to each other. The \(y\)-axis is the phoneme label. This dendrogram shows that the consonants and vowels are well distinguished. In Figure 2, if we backtrack the clustering path from the finally merged cluster at the right end, all the phonemes are broadly divided into two clusters. The largest cluster in the upper side is a set of phonemes from [th] to [zh]. The lower largest cluster consists of phonemes from [uh] to [oy], most of which are vowel sounds. However, [l] and [r] are consonants and more specifically, belong to voiced alveolar sounds. The phonemes [w] and [y] are semivowels, belonging to voiced labio-velar and voiced palatal sounds, respectively. These sounds are between vowels and consonants, so we split the lower cluster into \"Vowels\" and \"Mixed\", as shown in the first and second rows of Table 3. Among vowels, the phonetic classification is somewhat grouped according to the pronunciation position similarly to the conventional phonetic classification, but is not accurately distinguished. This is because the pronunciation of vowels vary according to the adjacent phonemes. All of the phonemes in the upper largest cluster are consonants. The individual consonants have different characteristics, so we divided the consonant set into four major groups.