Introduction
NAM microphone is a special device (sensor) which is attachedbehind the talker’
talker’s ear and perceives very quietly uttered speech

Progress in audible (normal) and inaudible
speech recognition based on NAM
microphones

NAM microphone [Nakajima et al., NAIST,2003-2004]

Panikos HeracleousNAM

NAM
NAMRECOGNITIONCONVERSION

NAM plays an important role in Ubiquitous Networking
Speech and Acoustics Processing Lab

2004/12/22.

It extends the speech media applications to cover:- Applications with privacy (e.g. speech recognition, telephony)
私は名義を貸しただけで株購入をしていない。たとえ逮捕されても裁判で徹底的に闘う- Applications under several (e.g. noisy) environments- Applications for specific people groups (e.g. speech-impaired)

NAIST

NAM microphones

Some advantages of NAM

1. Stethoscopic NAM microphone [2003]••••

Based on stethoscope used in medical science
Able to capture audible and inaudible speech
Only low frequency band
Low speech intelligibilityPrivacy in communication (human machine, human-human)
Direct attachment to the skin – Expected noise robustnessSilicon is used to wrap a sensitive microphone
Able to capture audible and inaudible speech
Wide frequency band
Higher speech intelligibility

Able to capture audible and inaudible speechMale

2. Silicon NAM microphone [2004]
•••
•

NAIST

2004/12/22

Music is heard

No additional techniques are required

Only a microphone pre-amplifier is required
Directly connected to PC for recording, or speech recognition

Male

Size, weightFemale

3. First pre-commercial prototype NAM [2004]••

Based on the Silicon NAM microphone
Still under development and investigation
Silicon NAM microphoneNAIST

2004/12/22

Some similar approaches
Bone-microphonePARATA throat microphone
NAIST

2004/12/22

Stethoscopic NAM
Speaker-dependent experiments (Clean)

Microsoft

Norwegian University ofScience and Technology

Word Acuuracy [%]MLLR- Special applications
Military, Police, mobile telephone- Usually normal speech recognition
Integration with other sensors- Big size and uncomfortable usage- Results show high noise robustness
2004/12/22

NAIST

MLLR+MAP

500MLLR
MAP
MLLR+MAP

Throat microphones

MAP

100

00.5
0.5
0.5

1
64.1
44.3
81.2

282.4
67.5
89.7

3489.79077.1
82.2
91.4
92.7
Iterations #589.581.6
93.5

690.3
83.2
92.2

700 training utterances, 24 test utterances
Julius recognizer, 20k vocabulary Japanese dictation task
High performance! Comparable to normal-speech recognition
2004/12/22

NAIST

1

Stethoscopic NAM

Stethoscopic NAM

Speaker-dependent experiments (Noise) - I

Speaker-dependent experiments (Noise) - II

MAP

MLLR

MLLR+MAP
Word Acuura

Word Acuura

MLLR
100
500-50
MLLR
MAP
MLLR+MAP

0-1.4
-1.4
-1.4

163
42.7
65.9

278.1
61.9
83.9

3489.7
89.6
70.2
73.5
9091.9Iterations #590.5
80.8
91.3

0690.88091.3MLLR
MAP
MLLR+MAP

73.5Female

70.0

62.4

MLLR+
MAP
77.8

EM
75.9

58.4

71.1

65.4

Reasonableresults

Recording conditions- 16-bit SONY ZA5ES SUPER BIT MAPPING DAT tape digital recorder- Anechoic room 30dBA, Sampling frequency: 44100kHz -> 16kHz
Recording data
-25 speakers (14 male, 11 female), 100 newspaper, 50 phoneme balanced utterances-3189 training utterances, 48 training utterances
NAIST

2004/12/22

34
89.2
91.6
74.6
78.7
91.4
90Iterations #Intelligibility objective measurement
The intelligibility of speech refers to the accuracy with whicha normal listener can understand a spoken word or phrase

Speaker-dependent experiments (Clean)
MAP

MLLR+MAP

100
500
MLLR
MAP
MLLR+MAP0
4.9
4.9
4.9

1
85.7
69.6
88.4

2
90.3
84.6
92.3

34
91.1
92.1
84.6
88.9
91.8
94.5
Iterations #Analyzer
SIA-Smaart Time Analyzer

TSP signal

miModulation function : Shows the reduction of the modulation index

MLS signal∫ h (τ ). exp(− j.2.π .F .τ ).dτ2momim( F ) =0f∞

2∫ h f (τ )dτ0

2004/12/22

NAIST

Base on impulse response(Schroeder, 1981)

NAM microphonePre-amplifierSound card

Impulse response(example)

ChannelSpeech Intelligibility objective measures∞m( F ) =+ 21.5% relative

NAM microphone measurement

Loud speaker

modulation index

694.8
88.9
94.9

NAIST

30cm

Received signal with smaller modulation

594.6
88.9
93.5

350 training utterances, 24 test utterances
Julius recognizer, 20k vocabulary Japanese dictation task

CEI-IEC standard n. 60268-216
Steeneken and Houtgast

Modulated test signal

691.6
81.4
91.4

Silicon NAM

2004/12/22

Speech Transmission Index (STI): Based on the reduction of590.5
81.1
90.3

NAIST

Word Acuuracy [%]Training method

Male2
82.6
5688.4MLLR

Word Accuracy [%]

MAP

167.8
35.4
69.3

2004/12/22

Stethoscopic NAM
Speaker-independent experiments

MLLR

00.5
0.5
0.5

700 training utterances, 24 test utterances
Room noise is superimposed to clean data
Noise level: About 50dBA – HMMs clean

NAISTSpeaker

MLLR+MAP

50700 training utterances, 24 test utterances
Room noise is superimposed to clean data
Noise level: About 50dBA – HMMs noisy
2004/12/22

MAP100STIRSTI
Clarity% AlCons
2004/12/22

Speech transmission index – Optimal 1.0 / Cannot hear 0.0
Rapid Transmission Index – Data from narrow band
Early to late energy ratio in impulse response
Articulation Loss of consonants
NAIST

2

Conclusions

NAM microphones measurement
Investigating effects of design (e.g. wrapping),
quality of NAM and recording setup correctness
TSP signal

Impulse response

1

NAM microphones introduction
Able to capture audible and inaudible speech
Can be applied in speech recognition and communication
Easy to use – Small, light2
Experiments on audible speech recognition
Stethoscope NAM

Reference microphone (Simple Desktop PC mic)Pre-commercial Silicon NAM microphone

Clean: 93.5% word accuracy
Noisy: 93.3% word accuracy (50dBA)

Silicon NAM
34

Silicon NAM microphone

Clean: 94.86% word accuracy

NAM quality and intelligibility investigation

Data collection for speaker-independent experiments

NAM microphones show similar intelligibility to other sensors

Stethoscope microphone

Silicon microphones do not show significant degradations
2004/12/22

NAIST

2004/12/22

NAIST

2004/12/22

NAIST3