Speech recognition system design based on automatically derived units

In most speech recognition systems today, acoustic modeling and lexical modeling are viewed as separable problems. Currently the most popular approach is to manually define canonical word pronunciations in terms of phonetic units and let the acoustic models capture differences between actual spoken and canonical pronunciations implicitly with Gaussian mixture models. As a result, these models can be very broad, particularly for casual spontaneous speech. An alternative approach, explored in this thesis, is to learn a unit inventory and pronunciation dictionary from training data using a maximum likelihood objective function. In particular, this thesis addresses previously unsolved problems in automatic unit design with three main contributions. First, to make design of a large unit inventory practical, a new approach is described that combines the problems of unit selection and lexicon design. The design of the units is acoustically driven but constrained to guarantee a matched, limited complexity pronunciation model. Instead of using an acoustic unit training algorithm followed by separate pronunciation model design, the algorithm proposed here incorporates a pronunciation constraint within the unit design algorithm. The resulting unit inventory, unit models and lexicon are matched since they are designed by a single joint design step. The second problem addressed involves synthesizing models for unobserved contexts, needed to model contextual variation at word boundaries. As in phone-based systems, decision tree clustering is used, but this requires classes or sets of units that have a similar influence in context. The solution is to learn these classes from data by a parallel context clustering process. Third, the ability to generalize at the word-level, i.e. to handle words not observed in the training data, is provided by a hybrid system design algorithm. In the hybrid system, automatically derived units are designed for the most frequent words, and phonetic units are designed for all words in the vocabulary. Using an estimation step, the word models constructed by the independent automatic and phonetic units are evaluated and the most likely model is included in the lexicon. The new automatic unit design algorithm showed improved performance over phonetic units in experiments on a medium vocabulary (1000 words) task (Resource Management) for both small and large unit inventory systems, outperforming an alternative approach to automatic unit design reported on this task. The algorithm for learning context conditioning groups is successful in that the performance of a system derived by decision tree clustering is equivalent to that of the best unconstrained clustering system and an additional gain is observed when modeling contextual effects across word-boundaries. Finally, when automatically derived units were used in experiments on a large vocabulary (20,000 word) conversational speech task (Switchboard), the recognition accuracy improved over the phonetic unit baseline. In summary, the joint unit and lexicon design algorithm gives higher recognition performance or can be configured to give similar performance at lower cost (lower system complexity) than phone-based units for applications where several examples of each vocabulary word can be provided.