Preface
 Automatic Speech Recognition

Applying Non-Audible Murmur(NAM) microphone in speech
recognition systems

 Non-Audible Murmur (NAM) definition

Scenario – Applications
Similar approaches
 NAM recognition based on adaptation

Maximum Likelihood Linear Regression (MLLR)
Maximum A Posteriori (MAP) adaptation
Combination of MLLR and MAP
Experiments and Results

Panikos Heracleous
COE-Postdoctoral Fellow
Speech and Acoustics Processing Laboratory
NAIST

1

5/26/2004

Automatic Speech Recognition
2

5/26/2004

Speech Recognition SystemGoalAdaptation

Convert a speech signal into a text message
Device-, environment-, speaker-independentApplications

NAM microphone

Automation of operator-based tasks
Dictation Customer care, help
 Etc.
Acoustic Hidden Markov Models (HMM)
modelsSpeech receiverInput
speechClose-talking microphone
Head-set
Microphone array
NAM microphone

Featureextraction

Decoding (search)

MFCC

Viterbi search
Language
ModelBigram
trigram3

5/26/2004

Non-Audible Murmur (NAM)

Word
Lexicon4

5/26/2004

Scenario

DEFINITION
Non-Audible Murmur (NAM) is murmur uttered very quietly,
which cannot be heard by other listeners near the talker.

Directly from bodyLess sensitive to noise

ＮＡＭ microphone (Nakajima et al. 2003)

Noise robustness
Privacy

Recognition Engine

Multiple users
Telephony
Etc.

AnalysisAutomatic Speech Recognition
5/26/2004

Word sequence

Speech Transform
5

5/26/2004

NAM

Sound-impaired
After surgery
Disabilities
61

Similar approaches

NAM recognition using adaptation
PARAT

Bone-microphone Adaptation techniques

Maximum Likelihood Linear Regression(MLLR) adaptation
Maximum A Posteriori (MAP) adaptation
Microsoft [ASRU2003]

 Advantages

Norwegian University of Scienceand Technology [ASRU2003]

Require only a small amount of data
Speaker and environment adaptation
High performance-Noise-robustness-Noise-robustness-Unable to recognize normal speech-Unable to recognize inaudible murmur-Unable to recognize inaudible murmur -Practical difficulties (communication)-Complementary usage
The NAM microphone can be used for normal speech, and inaudible murmur recognition.
75/26/2004 experiments show noisePreliminary
noise-robustness. No additional techniques are required.

MLLR adaptation
85/26/2004

MLLR: Transforms sharingAim

Obtain a set of transformations matrices for the model parameters that maximizes
the likelihood of the adaptation data. For the solution, the Expectation-Maximization(EM) technique is used.

Acoustically similar components are transformed together. Splitting algorithm using
Euclidean distance.
Less data for accurate estimation
Unseen parameters can be adapted

W2

Adaptationdata

1µ m , Σm

Speaker-dependent ^parameters

Adaptation^^µ i , Σi

Speaker-independentParametersµ m = Wmξ m
ξ m = [1 µ1 .. µ n ]T
95/26/2004

MAP adaptation

W44

Informative priors are used
Speaker-independent parameters Requires more adaptation data, than MLLR
 Together with MLLR more effective

2N jmN jm + τ_µ jm +τN jm + τ5

74W4Wi : Transformation

105/26/2004

Experimental conditions
Initial models
Normal-speech PTM HMMs (JNAS)
NAM-speech PTM HMMs (1800 utterances)

Adaptation techniques
MLLRMAP MLLR+MAPµ jm : Updated mean
µ jm : Speaker - independent mean

Adaptation data_µ jm : Obtained from adaptation data

710 NAM utterances – Male speaker

Test setN jm : Occupation likelihood (forward - backward

5/26/2004

6

Binary regression tree^

µ jm

573

j : state, m : mixture, τ : weight^

W3Experiments

µ jm =

6W2Wm

W311

5/26/2004

24 NAM uuterances

122

Results using MLLR adaptation
88.9

MLLR (JNAS)

88.1

88.9

89.7

65.7

0123
Iterations ＃

4

5100
90
80
7060
50
4030
20
1006

82.4

89.7

MAP (NAM)

90

89.5

90.3

64.1

0.5
0123
Iterations ＃4592.4
Word Accuracy [%]Word Accuracy [%]86.8

9265.70123
Iterations ＃45610090807060
5040302010081.2

89.7

91.4

92.7

93.5

93.5

123
Iterations ＃45610090807060
5040302010067.5

77.1

82.2

81.6

83.2

456

44.3

0.5
0123
Iterations ＃145/26/2004

Discussion
MLLR adaptation
Requires less amount of data
Initial models do not have serious effect

MLLR + MAP (JNAS)

93

MAP (JNAS)

90.5

65.7

0


92.7

91.6

90!

13

MLLR + MAP (NAM)

86.2-Higher initial performance using SI NAM initial HMMs-No regression tree is used – Unseen components are not transformed-Initial models have significant effect-High performance 93.5% using NAM-speech initial models
Iterative MAP performs much better that single-iteration MAP.
MAP formula very similar to EM formula. Only in priors are different

Results using MLLR + MAP
1009080706050403020
10083.5

6-Higher initial performance using SI NAM initial HMMs
-After few iterations almost equal performance is achieved-The initial models do not have serious effect-Due to the components grouping, unseen components are also transformed
5/26/2004

1009080706050403020100

Word Accuracy [%]89.2

Word Accuracy [%]84.3

Word Accuracy [%]Word Accuracy [%]MLLR (NAM)
1009080706050403020100Results using MAP adaptation

93.5

92.2

Components grouping – Multiple iterations

Critical factor: Number of classes (32 or 128)
MAP adaptation
Performs better when more data are available
Initial models are very important
Critical factor: Scaling factor

0.50123
Iterations ＃456

-Simultaneous MLLR and MAP adaptation (one pass)-Transformed components by MLLR are used as priors-Initial models do have significant effect-High performance 93.5% - Same as MAP adaptation with NAM initial models

5/26/2004

MLLR and MAP adaptation
Very useful when there are not NAM initial models
Equal performance with MAP

155/26/2004

16

175/26/2004

18Conclusion

Brief introduction of Non-Audible Murmur
Automatic Speech Recognition
MLLR adaptation
MAP adaptation
MLLR + MAP



93.5% word accuracy
20k vocabulary
Only 700 utterances
Very promising result
Comparable to normal-speech recognition

On-going work
Silicon NAM microphone

5/26/2004

3