A VERY LOW BIT RATE SPEECH CODER USING
HMM-BASED SPEECH RECOGNITION/SYNTHESIS TECHNIQUES

y

z

y

z

Keiichi Tokuda , Takashi Masuko , Jun Hiroi , Takao Kobayashi , and Tadashi KitamurayyDepartment of Computer Science, Nagoya Institute of Technology, Nagoya, 466 JapanzPrecision and Intelligence Laboratory, Tokyo Institute of Technology, Yokohama, 227 Japan
Email: ftokuda,jun,kitamurag@ics.nitech.ac.jp, fmasuko,tkobayasg@pi.titech.ac.jp
ABSTRACTThis paper presents a very low bit rate speech coder based on HMM(Hidden Markov Model). The encoder carries out phoneme recognition, and transmits phoneme indexes, state durations and pitch
information to the decoder. In the decoder, phoneme HMMs areconcatenated according to the phoneme indexes, and a sequence ofmel-cepstral coefficient vectors is generated from the concatenated
HMM by using an ML-based speech parameter generation technique. Finally we obtain synthetic speech by exciting the MLSA
(Mel Log Spectrum Approximation) filter, whose coefficients are
given by mel-cepstral coefficients, according to the pitch information. A subjective listening test shows that the performance of the
proposed coder at about 150 bit/s (for the test data including 26 %silence region) is comparable to a VQ-based vocoder at 400 bit/s(= 8 bit/frame 50 frame/s) without pitch quantization for bothcoders.



Input Speech
Encoder
Mel-Cepstral Analysis
C1 C2

Phoneme Recognition

Phoneme HMMs
/a/

/a/

Pitch

Phoneme SequenceThis work was partially supported by the Ministry of Education, Science and Culture of Japan, Grant-in-Aid for Encouragement of Young
Scientists, 09750399, 1997.

State Durations
/i/

/a/

1. INTRODUCTION
To code speech at rates on the order of 100 bit/s, phonetic andsegment vocoders are the most popular techniques [1]-[6]. These
coders decompose speech into a sequence of speech units (i.e.,phonetic units and acoustically derived segment units, respectively)by using a speech recognition technique, and transmit the obtained
unit indexes and unit durations. The decoders synthesize speechby concatenating typical instances of speech units according to theunit indexes and unit durations.
This paper presents a phonetic vocoder based on HMM (Hidden Markov Model), in which speech spectra are consistently represented by mel-cepstral coefficients obtained by a mel-cepstral
analysis technique [7], and the sequence of mel-cepstral coefficient
vectors for each speech unit is modeled by phoneme HMM. The
encoder carries out phoneme recognition which adopts advanced
techniques used in the area of speech recognition, and transmits
phoneme indexes and state durations to the decoder by using entropy coding and vector quantization. Pitch information is alsotransmitted to the decoder whereas this paper excludes pitch quantization from consideration. In the decoder, phoneme HMMs areconcatenated according to the phoneme indexes, and the state sequence is determined from the transmitted state durations. Then asequence of mel-cepstral coefficient vectors is determined in sucha way that the likelihood of the sequence of mel-cepstral coefficient vectors is maximized for the concatenated HMM and the

/i/

/i/

/k/

Parameter Generation
~ ~
C1 C2

MLSA Filter

Decoder

Synthetic Speech

Figure 1: A very low bit rate speech coder based on HMM.

state sequence [8]-[10]. Finally speech signal is synthesized by the
MLSA (Mel Log Spectrum Approximation) filter according to theobtained mel-cepstral coefficients [7].
In the following, we summarize the HMM-based speech synthesis technique, and describe the proposed coder in Section 2 and3, respectively. The results of a subjective evaluation test are shownin Section 4.
2. HMM-BASED SPEECH SYNTHESIS
A block diagram of the proposed speech coder is illustrated inFig. 1. The encoder is equivalent to an HMM-based phoneme recognizer, and the decoder does the inverse operation of the encoderusing an HMM-based speech synthesis technique [10], which consists of two techniques: a mel-cepstrum-based vocoding [7] andHMM-based speech parameter generation [8], [9]. This section
summarizes these two techniques.

respect to O = [o01 ; o02 ; : : : ; o0T ]0 under constraints (3) and (4).
The output distribution of each state is assumed to be a single
Gaussian distribution. Thus the logarithm of P (O Q; ) can be
written asjjlog P (O Q; )

2.1. Vocoding Technique Based on Mel-Cepstrum
Since we model speech spectrum using M + 1 mel-cepstral coefficients, i.e., frequency-transformed cepstral coefficients, the minimum phase synthesis filter can be written asD(z) = exp

M
X
m=0

(1)

,1
z̃,1 = 1z, ,z,1 ; j j < 1:

(2)

When the sampling frequency is 10kHz, the phase character= 0:35 gives a
istics of the all-pass transfer function withgood approximation to the mel frequency scale. For a given
the input speech sequence assumed to be zero-mean Gaussianx = [x(0); x(1); : : : ; x(N 1)]0 , we obtain mel-cepstral coefficients c = [c(0); c(1); : : : ; c(M )]0 which maximize P (x c).
It can be shown that P (x c) is convex with respect to c, accordingly the minimization problem can be solved efficiently by an
iterative technique described in [11], [7]. It is noted that maximizing P (x c) with respect to c corresponds to minimizing theenergy of the inverse filter output under a constraint that the gain
factor of D(z ) is unity, i.e., the impulse responses of D(z ) and
1=D(z ) are unity at time 0.
To synthesize speech from the mel-cepstral coefficients, wehave to realize the transfer function of (1), which is not a rational
function. Fortunately the MLSA filter can approximate D(z ) with
sufficient accuracy. The MLSA filter is an IIR filter which has a
special structure shown in [7], and its stability is guaranteed forspeech sounds. The coefficients of the MLSA filter can be obtained
from the mel-cepstral coefficients with M multiply-add operations.
Thus, by using the MLSA filter, we can synthesize speech easily
from the mel-cepstral coefficients.

,

j

j

j

2.2. Speech Parameter Generation from HMM
Let ct be the vector of mel-cepstral coefficients at frame t. Thenthe dynamic features ∆ct and ∆2 ct , i.e., delta and delta-delta melcepstral coefficients at frame t, respectively, were calculated asfollows:
=

∆2 ct

=

L1
X
 =,L1
L2
X
 =,L2

w1 ( ) ct+(3)

w2 ( ) ct+ :(4)

We assume that a speech parameter vector ot at frame t consists ofstatic and dynamic feature vectors, that is, ot = [c0t ; ∆c0t ; ∆2 c0t ]0 ,where 0 denotes matrix transpose.
For a given continuous HMM  and a state sequence Q =q1 ; q2 ; : : : ; qT 0, we0 obtain 0a sequenceof mel-cepstral coefficient
vectors C = [c1 ; c2 ; : : : ; cT ]0 by maximizing P (O Q; ) with



f

g

, 12 log jUj + Const.

(5)

where

c(m) z̃,mwhere z̃ ,1 is an all-pass transfer function defined by∆ct, 12 (O , M)0 U,1 (O , M)

=

j

M
U

=
=

0q ; 0q ; : : : ; 0qT 0diag Uq ; Uq ; : : : ; UqT ;




1

2

1

2

(6)
(7)

and qt and Uqt are the mean vector and the covariance matrix
associated with state qt , respectively. Without dynamic features(i.e., ot = ct ), it is obvious that P (O Q; ) is maximized when
C = M, that is, the sequence of mel-cepstral coefficient vectors isdetermined by the mean vectors, independently of the covariancesU.
On the other hand, under the constraints (3) and (4), the sequence of mel-cepstral coefficient vectors C is determined by a setof linear equations @ log P (O Q; )=@ C = 0, which can easily
be solved by a fast algorithm derived in [8], [9]. It has been shownthat the obtained mel-cepstral coefficient vectors reflect not only
the means of static and dynamic feature vectors but also the covariances of those, resulting in a natural-sounding synthetic speech.

jj

3. VERY LOW BIT RATE SPEECH CODER
BASED ON HMM
In this section, each part of the speech coding system is describedbriefly.
3.1. Speech Recognition
We used phonetically balanced 503 sentences uttered by a malespeaker MHT in the ATR Japanese speech database for training
phoneme HMMs. Speech signals were sampled at 10kHz and windowed by a 25.6ms Hamming window with a 5ms shift, and thenmel-cepstral coefficients were obtained by the mel-cepstral analysis technique. The feature vectors consisted of 13 mel-cepstralcoefficients including the 0th coefficient, and their delta and deltadelta coefficients.
We used 3-state left-to-right triphone models with no skip.
Each state was modeled by a single Gaussian distribution with the
diagonal covariance. Total of 34 phonemes and a silent models
were prepared. Decision-tree based model clustering was applied
to each set of triphone models, and the resultant set of tied triphonemodels has approximately 1,800 distributions.
The speech recognizer of the encoder uses the phoneme pairconstraints in Japanese language. The phoneme recognition rate
for the test data used in the subjective evaluation (Section 4) was
73.68 % (88.7 % when insertion errors are ignored). The averagephoneme rate computed from the transcription data is about 9.5
phoneme/s while the average phoneme rate computed from therecognition results for the test data was 11.7 phoneme/s. It is notedthat the test data includes 26 % of silence region.

3.2. Phoneme Index Coding
The phoneme sequence obtained by the phoneme recognizer istransmitted using entropy coding. The histograms of phonemes
and phoneme pairs were measured from the phoneme recognition
results for the training data. When the Huffman coding based on
the occurrence probability distribution of phonemes was used, thebit rate of phoneme information for the test data was about 54 bit/s.
Further, using the occurrence probability distribution of phoneme
pairs (i.e., phoneme bigram probability) we obtained a bit rate ofabout 46 bit/s.
3.3. State Duration Coding
For transmitting state durations we examined the following threemethods:
Method 1
The histogram of state durations for each phoneme was measured from the phoneme recognition results for the training data.
State durations are transmitted by the Huffman coding based on
the occurrence probability distribution of state duration for thecorresponding phoneme.
Method 2
The histogram of phoneme durations for each phoneme wasmeasured from the phoneme recognition results for the training
data. Each phoneme duration is transmitted using the Huffman
coding based on the occurrence probability distribution of thecorresponding phoneme. In the decoder each phoneme duration
is divided into three state durations using state duration densities
associated with the corresponding phoneme HMM. The state
durations are determined by a method based on the maximum
likelihood criterion [8], that is,

dk = mk + k2
= T,

3
X

k=1

mk

(8)

!

3
X

k =1

k2

(9)

where T is phoneme duration, mk , k2 are the mean and variance of the duration density associated with k-th state of thephoneme HMM, respectively. To obtain the state duration densities, the histogram of state durations was measured from thephoneme recognition results for the training data. Each state
duration density was modeled by a single Gaussian distribution. Regarding state duration densities of a triphone HMM asa three-dimensional Gaussian, we applied decision-tree basedmodel clustering to the three-dimensional Gaussians, and the resultant set of tied state duration models had approximately 1,600
distributions.
Method 3
State durations of each phoneme are regarded as a three-dimensional vector, and vector-quantized. The codebook is trainedby the LBG algorithm based on state durations obtained byphoneme recognition for the training data. Three codebooks
whose sizes are 8, 32 and 1024, respectively, were trained for theexperiment of Section 4. Further the VQ indexes are transmitted
by using the Huffman coding.
3.4. Speech Synthesis
In the decoder, triphone HMMs corresponding to the transmitted
phoneme indexes are concatenated, and from the obtained HMM

0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5Frequency (kHz) Frequency (kHz) Frequency (kHz)

Figure 2: Spectra comparing original (left), proposed 160
bit/s (middle), and vector-quantized 400 bit/s (= 8 bit/frame
50 frame/s) (right).

a sequence of mel-cepstral coefficient vectors is generated using
the algorithm described in Section 2.2. By exciting the MLSA
filter by pulse train or white noise generated according to the pitch
information, speech signal is synthesized based on the generated
mel-cepstral coefficients.
4. EXPERIMENTS
In preliminary experiments we observed the following:
1. In the case where both state durations and phoneme durations
are not transmitted and the decoder determines state durationsof each phoneme based on the state duration densities associated with each phoneme HMM, recognition errors not only
have an impact on the subjective quality of the coded speech
but degrade the intelligibility significantly.
2. When the unquantized state durations are transmitted, recognition errors do not have an impact on the subjective qualityof the coded speech whereas the subjective quality is in proportional to recognition rate.
To evaluate the speech quality of the proposed speech coder,
we conducted a DMOS test. Test utterances were eight senteces
which are not included in the training data. Subjects were eightmales. In this experiment, pitch was not quantized, and original
pitch values were used with Viterbi alignment based on phoneme
HMMs. Fig. 2 exemplifies spectra for original speech, those reconstructed by the proposed coder, and those vector-quantized.
In Fig. 3, DMOS values for the proposed coder were comparedto that for the mel-cepstral vocoder with vector quantization ofmel-cepstral coefficients. The speech database used for trainingProposed Method 1

3.3

Proposed Method 1

160

Proposed Method 2

2.6

Proposed Method 2

100

2.1

68
Proposed Method 3

2.9

Proposed Method 3

Conventional VQ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
3.4,,,,,,,,,,,,,,,,,,
1

95
146

3.4

2

3

400
Conventional VQ ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
4

5

DMOS

60

80

100

120

140

160

180

400

Bit Rate (bit/s)

Figure 3: Subjective performance for the proposed and conventional vocoders measured by DMOS.

Figure 4: Bit rates for the proposed and conventional coders.

phoneme HMMs was also used for training the VQ codebook for
the VQ-based vocoder. The bit rates for both coders are shown in
Fig. 4. The proposed coder uses 46 bit/s for transmitting phoneme
indexes, and the remaining bits are used for transmitting state orphoneme durations.
Fig. 3 and 4 show that the proposed coder with higher biterate achieves better performance. This suggests that inaccurate
reproduction of state durations degrades the coded speech quality.
It can be seen from the figures that the performance of the proposed
coder at about 150 bit/s is comparable to that of the VQ-based
vocoder at 400 bit/s (= 8 bit/frame 50 frame/s) without pitch
quantization for both coders. The proposed coder at about 70 bit/s
degrades its speech quality while it still preserves the intelligibilityof the coded speech.
The coding delay of the proposed coder can be summarized
as follows. The delay which arises in the encoder depends on the
search strategy of the phoneme recognizer. Generally it could beon the order of 100 ms. On the other hand, the delay corresponding
to one phoneme duration; an average of about 100 ms, arises at the
decoder since the decoder needs the next phoneme index to choosea triphone HMM. Additionally the speech parameter generation
algorithm causes a delay of approximately 100 ms at the decoder.

REFERENCES



5. CONCLUSION
We presented a new framework of a very low bit rate speech coder
using HMM-based speech recognition and synthesis techniques.
The HMM-based speech synthesis consists of a vocoding techniquebased on mel-cepstrum and an HMM-based speech parameter generation algorithm. It has been shown that the performance of the
proposed coder at about 150 bit/s (for the test data including 26 %silence region) is comparable to that of a VQ-based vocoder at 400
bit/s (= 8 bit/frame 50 frame/s) in terms of subjective quality
measured by DMOS without pitch quantization for both coders.
The proposed coder at 70 bit/s can still preserve the intelligibilityof the coded speech.
We expect that further improvement of the phoneme recognizer results in better performance of the proposed coder. The
future work will be conducted towards constructing a speakerindependent version of the proposed coder using an HMM-based
voice conversion technique [12] which adopts a speaker adaptation
technique used in HMM-based speech recognition.

[1] S. Roucos, R. M. Scshwartz and J. Makhoul, “A segment
vocoder at 150 b/s,” in Proc. ICASSP-83, 1983, pp.61–64.
[2] F. K. Soong, ”A phonetically labeled acoustic segment(PLAS) approach to speech analysis-synthesis,” in Proc.
ICASSP-89, 1989, pp.584–587.
[3] Y. Shiraki and M. Honda, “LPC speech coding based on
variable-length segment quantization,” IEEE Trans. Acoust.,
Speech, Signal Processing, vol. ASSP-36, no. 9, pp.1437–
1444, Sep. 1989.
[4] Y. Hirata and S. Nakagawa, “A 100bit/s speech coding using
a speech recognition technique,” in Proc. EUROSPEECH-89,
1989, pp.290–293.
[5] C. M. Ribeiro and I. M. Trancoso, “Phonetic vocodingwith speaker adaptation,” in Proc. EUROSPEECH-97, 1997,
pp.1291–1294.
[6] M. Ismail and K. Ponting, “Between recognition andsynthesis —300 bits/second speech coding,” in Proc.
EUROSPEECH-97, 1997, pp.441–444.
[7] T. Fukada, K. Tokuda, T. Kobayashi and S. Imai, “An adaptive algorithm for mel-cepstral analysis of speech,” in Proc.
ICASSP-92, 1992, pp.137–140.
[8] K. Tokuda, T. Kobayashi and S. Imai, “Speech parameter
generation from HMM using dynamic features,” in Proc.
ICASSP-95, 1995, pp.660–663.
[9] K. Tokuda, T. Masuko, T. Yamada, T. Kobayashi and S.
Imai, “An algorithm for speech parameter generation fromcontinuous mixture HMMs with dynamic features,” in Proc.
EUROSPEECH-95, 1995, pp.757–760.
[10] T. Masuko, K. Tokuda, T. Kobayashi and S. Imai, “Speechsynthesis from HMMs using dynamic features,” in Proc.
ICASSP-96, 1996, pp.389–392.
[11] K. Tokuda, T. Kobayashi, T. Fukada, H. Saito and S. Imai,“Spectral estimation of speech based on mel-cepstral representation,” Trans. IEICE, vol. J74-A, pp.1240–1248, Aug.
1991 (in Japanese).
[12] T. Masuko, K. Tokuda, T. Kobayashi and S. Imai, “Voicecharacteristics conversion for HMM-based speech synthesis
system,” in Proc. ICASSP-97, 1997, pp.1611–1614.