January 20 , 2004 UTML TR 2004 – 001 A Segmental HMM for Speech Waveforms

We present a purely time domain approach to speech processing which identifies waveform samples at the boundaries between glottal pulse periods (in voiced speech) or at the boundaries between unvoiced segments. An efficient algorithm for inferring these boundaries is derived from a simple probabilistic generative model of speech and state of the art results are presented on pitch tracking, voiced/unvoiced detection and timescale modification. A Segmental HMM for Speech Waveforms Kannan Achan, Sam Roweis, Brendan Frey Machine Learning Group, University of Toronto 1 Speech Segments in the Time Domain Processing of speech signals directly in the time domain is commonly regarded to be difficult and unstable, due to fact that perceptually very similar utterances exhibit very large variability in their raw waveforms. As a result, by far the most common preprocessing step for most speech systems is to convert the raw waveform into a time-frequency representation, using a variety of spectral analysis and filterbank techniques. In this paper we pursue a purely time domain approach to speech processing in which we identify the samples at the boundaries between glottal pulse periods (in voiced speech) or at the boundaries between unvoiced segments of similar spectral shape (“colour”). Having identified these segment boundaries, we can perform a variety of important low level speech analysis operations directly and conveniently. For example, we make a voiced/unvoiced decision on each segment by examining the periodicity of the waveform in that segment only. In voiced segments we can estimate the pitch as the reciprocal of the segment length. Timescale modification without pitch or format distortion can be achieved by stochastically eliminating or replicating segments in the time domain directly. More sophisticated operations, such as pitch modification, gender and voice conversion, and companding (volume equalization) are also naturally performed by operating on waveform segments one by one without the need for a cepstral or other such representation. The computational challenge with this approach is in efficiently and robustly identifying the segment boundaries, across silence, unvoiced and voiced segments. In this paper we introduce a segmental Hidden Markov Model, defined on variable length sections of the time domain waveform, and show that performing inference in this model allows us to identify segment boundaries and achieve excellent results on the speech processing tasks described above. 2 A probabilistic generative model of time-domain speech segments The goal of our algorithm is to break the time domain speech signal s1, . . . , sN into a set of segments, each of which corresponds to a glottal pulse period or a segment of unvoiced colored noise. Let bk denote the time index of the beginning of the kth segment and sk = (sbk , . . . , sbk+1−1) denote the waveform in the kth segment, where k = 1, . . . , K indexes segments. Our algorithm searches for the segment boundaries, b1, b2, . . . , bK+1, so that each segment can be accurately modeled as a time-warped, amplitude-scaled and amplitude-shifted version of the previous segment. We denote the transformation used to map segment sk−1 into segment sk by Tk. (A similar idea is used in  to cluster patterns in a way that is invariant to a set of transformations.) Given the segment boundaries b1, . . . , bK+1 and the transformations T1, . . . ,Tk we † Thanks to John Hopfield.