IEEE TRANSACTIONS ON COMPUTERS,VOL. 56, NO. 9,

SEPTEMBER 2007

1153

Guest Editors’ Introduction: Special Section on
Emergent Systems, Algorithms, andArchitectures for Speech-Based
Human-Machine Interaction
Rodrigo Capobianco Guido, Member, IEEE, Li Deng, Fellow, IEEE, andShoji Makino, Fellow, IEEEÇW

ELCOME to this special section of the IEEE Transactions on
Computers (TC) on emergent systems, algorithms, andarchitectures for speech-based human-machine interaction.
Recently, we have witnessed significant progress onsoftware and hardware techniques for human-machineinteraction, especially for speech-based ones. While highly
successful at the core, with many commercial applications
resulting from this progress, the techniques still require agreat deal of improvement, especially in the areas of speech
signal processing, robustness of speech systems, speechmodeling, learning algorithms, real-time decoding algorithms, speech quality enhancement, speech synthesis,
speaker identification, speech source separation, and soon, to further impact speech-centric human-machine interaction. This special section concentrates on the state-of-theart of such techniques. We received many submissions forconsideration. We finally selected nine papers among them,which have been critically peer-reviewed with top-quality
research and/or with more timely information than thepapers that were not selected. The selected papers haveaddressed a considerable number of new ideas andapplications in the field of speech-based human-machineinteraction.
The first paper we present in this issue is entitled“Architecture, User Interface, and Enabling Technology inWindows Vista’s Speech Systems,” by Julian Odell andKunal Mukerjee from Microsoft Corporation. In this paper,
they describe the architecture, user interface, and keytechnologies that make up the speech system incorporated
into Microsoft Windows Vista, which allows the combination of high accuracy and high usability for the end-to-end
speech experience. The key elements of the speech user

. R.C. Guido is with the SpeechLab, University of São Paulo (USP), Instituteof Physics at São Carlos (IFSC), Department of Physics and Informatics(FFI), Avenida Trabalhador São Carlense 400, São Carlos, SP 13560-970,
Brazil. E-mail: guido@ifsc.usp.br.
. L. Deng is with Microsoft Research, One Microsoft Way, Redmond, WA
98052-6399. E-mail: deng@microsoft.com.
. S. Makino is with NTT Communication Science Laboratories, NTTCorporation, 2-4 Hikaridai, Seika-cho, Kyoto 619-0237 Japan.
E-mail: maki@cslab.kecl.ntt.co.jp.
For information on obtaining reprints of this article, please send e-mail to:
tc@computer.org.
0018-9340/07/$25.00 ß 2007 IEEE

interface and how they maintain the user’s ability to control
the system despite limitations in the underlying recognition
technology are clearly and carefully explained.
The second paper of this issue, by Maycel-Isaac Faraj andJosef Bigun, is entitled “Synergy of Lip Motion and AcousticFeatures in Biometric Speech and Speaker Recognition.” In
this work, the authors present a novel and robust audiovisual digit and speaker-recognition system using lip-motion
and speech biometrics, where a verification barrier based on a
person’s lip movement is added to the system to guard
against advanced spoofing attempts. The system is based on a
Support Vector Machine (SVM) and a Gaussian Mixture
Model (GMM) and is tested with a database of about
300 different identities as well as the XM2VTS database.
Roger Moore is the author of the third paper of this issue,“PRESENCE: A Human-Inspired Architecture for SpeechBased Human-Machine Interaction,” where a novel architecture for speech-based human-machine interaction ispresented. It is inspired by recent findings in the neurobiology of living systems. PRESENCE, which is the acronym for
PREdictive SENsorimotor Control and Emulation, blurs thedistinction between the core components of a traditional
spoken language dialogue system, focusing on a recursive
hierarchical feedback control structure. It is founded on amodel of interaction in which the system has in mind theneeds and intentions of a user and vice versa.
“Speaker Verification via High-Level Feature-Based
Phonetic-Class Pronunciation Modeling” is the fourth paperof this issue, authored by Shi-Xiong Zhang, Man-Wai Mak,and Helen Meng. The focus of this paper is to improve theresults obtained with the articulatory feature-based conditional pronunciation models (AFCPMs) for speaker verification by grouping similar phonemes into phonetic classes
and representing background and speaker models asphonetic-class dependent density functions. Evaluations
based on the 2000 NIST SRE show that this phonetic-classapproach effectively alleviates the data spareness problem
encountered in conventional AFCPM, which results inbetter performance when fused with acoustic features.
The next paper of this issue is “Incorporating Knowledge
Sources into a Statistical Acoustic Model for Spoken
Language Communication Systems,” where Sakriani Sakti,
Konstantin Markov, and Satoshi Nakamura introduce aPublished by the IEEE Computer Society

1154IEEE TRANSACTIONS ON COMPUTERS,

general framework to incorporate additional sources ofknowledge into an Hidden Markov Model (HMM)-based
statistical acoustic model. Their method has the advantagesof allowing the probabilistic relationship between information sources to be learned and of facilitating the decomposition of the joint probability density function (PDF) intoa linked set of local conditional PDFs. The results demonstrated by the authors revealed that it improved the word
accuracy with respect to standard HMMs, with or without
additional sources of knowledge.
The field of rich transcription research, which includes
speaker diarization together with the annotation of sentenceboundaries and the elimination of speaker disfluencies, isexplored by José M. Pardo, Xavier Anguera, and Chuck
Wooters in their paper entitled “Speaker Diarization for
Multiple-Distant-Microphone Meetings Using Several
Sources of Information.” The authors analyze the correlation between signals coming from multiple microphones
and propose an improved method for carrying out speakerdiarization for meetings with multiple distant microphones.
In fact, they achieve state-of-the-art performance on theNIST Spring 2006 rich transcription evaluation database,improving the Diarization Error Rate (DER) by 15 percent to20 percent relative to previous systems.
Chien-Lin Huang and Chung-Hsien Wu are the authorsof our next paper, “Generation of Phonetic Units for MixedLanguage Speech Recognition Based on Acoustic andContextual Analysis.” In this work, a novel approach togenerating phonetic units in order to recognize mixedlanguage or multilingual speech is presented, where theHyperspace Analog to Language (HAL) model is adopted
for contextual modeling and contextual similarity estimation and the Multidimensional Scaling (MDS) method isapplied for reducing dimensionality. The created phonetic
set provides a compact and robust set that considers
acoustic and contextual information for mixed-language
or multilingual speech recognition in systems that utilize
speech-based human-machine interaction.
“Unsupervised Speaker Change Detection Using SVM
Training Misclassification Rate” is the next paper, authoredby Po-Chuan Lin, Jia-Ching Wang, Jhing-Fa Wang, andHao-Ching Sung. They present an unsupervised speakerchange detection algorithm based on Support Vector
Machines (SVMs) to detect speaker change in a speechstream. The proposed algorithm can identify speakerchanges with less speech data collection, making it capableof detecting speaker segments with short durations.
According to the experiments on the NIST Rich Transcription 2005 Spring Evaluation (RT-05S) corpus, their systemgives a lowest missed detection rate.
Chi-Chun Hsia, Chung-Hsien Wu, and Jian-Qi Wu are
the authors of the last paper, which is entitled “Conversion
Function Clustering and Selection Using Linguistic andSpectral Information for Emotional Voice Conversion.”
They use a Gaussian Mixture Bi-gram Model (GMBM) as
the conversion function to characterize the temporal andspectral evolution of speech signals and use it for emotional
voice conversion, incorporating linguistic and spectral
information for conversion function clustering and selection. Subjective and objective evaluations with statisticalVOL. 56,

NO. 9,SEPTEMBER 2007

hypothesis testing are conducted to evaluate the quality of
the converted speech, which compares favorably withprevious methods in conversion-based emotional speechsynthesis.
Last, we would like to thank the more than 40 referees
who spent a considerable amount of time reviewing thepapers and also the authors and coauthors of all of thepapers submitted to our special section, including those
whose papers could not be published in this issue due toextensive revision requests as well as the limited pagebudget. Special thanks are due to the Editor-in-Chief,
Dr. Fabrizio Lombardi, for hosting our special issue andto Ms. Joyce Arnold for her excellent support during thesubmission and peer review process. We hope you
appreciate the reading.

Warmest Regards,
Rodrigo Capobianco Guido
Li Deng
Shoji Makino
Guest Editors

Rodrigo Capobianco Guido received the BSc
degrees in computer science and in computerengineering, the MSc degree in electrical engineering, and the PhD degree in computational
applied physics, respectively, from São Paulo
State University (UNESP) at São José do Rio
Preto, Brazil, from the Educational Foundation at
Votuporanga (FEV), Brazil, in 1998, from Campinas State University (UNICAMP), Brazil, in2000, and from the University of São Paulo atSão Carlos (USP), Brazil, in 2003, all focusing on signal processing. Hehas worked with signal processing since 1995. He has alreadyparticipated in two postdoctoral programs in speech processing andwavelets at USP, from 2003 to 2007. He has taught wavelets, speech
and audio processing, programming languages, and electronics since1999 and has published scientific articles in IEEE and Elsevier journals
and magazines plus papers in conferences. He has served or is serving
as as a guest editor for many special issues of IEEE and Elsevier
journals and as an organizer and chairman for IEEE conferences,including IEEE ISM ’05, ISM ’06, ISM ’07, and ICSC ’07. He is a memberof the editorial board of scientific journals, including Pattern Recognition
Letters, Neurocomputing, and the International Journal of Semantic
Computing, and has also served as a reviewer for many IEEE andElsevier journals and conferences (~140 reviewed papers). He received
several grants and awards from the State of São Paulo Research
Foundation (FAPESP), from the National Council of Research andDevelopment (CNpQ), and the Ministry of Education and Culture (MEC)of Brazil. He has supervised many theses in his field and is a member ofthe IEEE. The main objective of his group’s work is concentrated indigital speech and audio processing (analysis, synthesis, patternmatching and recognition, voice morphing, compression, and so on),especially based on wavelets, neural networks, plus other statisticaltechniques.

GUIDO ET AL.: GUEST EDITORS’ INTRODUCTION: SPECIAL SECTION ON EMERGENT SYSTEMS, ALGORITHMS, AND ARCHITECTURES...

Li Deng received the PhD degree from theUniversity of Wisconsin-Madison in electricalengineering. In 1989, he joined the Department
of Electrical and Computer Engineering, University of Waterloo, Ontario, Canada, as an
assistant professor and became a full professor
in 1996. From 1992 to 1993, he conductedsabbatical research at the Laboratory for Computer Science, Massachusetts Institute of Technology, Cambridge, and, from 1997-1998, atATR Interpreting Telecommunications Research Laboratories, Kyoto,
Japan. In 1999, he joined Microsoft Research, Redmond, Washington,
as a senior researcher, where he is currently a principal researcher. Heis also an affiliate professor in the Department of Electrical Engineering
at the University of Washington, Seattle. His research areas include
automatic speech and speaker recognition, statistical methods andmachine learning, neural information processing, machine intelligence,statistical signal processing, digital communication, human speech
production and perception, acoustic phonetics, auditory speech processing, noise robust speech processing, speech synthesis and enhancement, spoken language understanding and systems, multimedia signal
processing, and multimodal human-computer interaction. In theseareas, he has published more than 250 refereed papers in leadinginternational conferences and journals, 12 book chapters, and has given
keynotes, tutorials, and lectures worldwide. He has been granted morethan 20 US or international patents in acoustics, speech and language
technology, and signal processing. He has authored two books: Speech
Processing—A Dynamic and Optimization-Oriented Approach (MarcelDekker Publishers, 2003) and Dynamic Speech Models—Theory,Algorithms, and Applications (Morgan & Claypool Publishers, 2006).
He served on the Education Committee and the Speech Processing
Technical Committee of the IEEE Signal Processing Society (19962000, and was associate editor for the IEEE Transactions on Speech
and Audio Processing (2002-2005). He currently serves on the Society’s
Multimedia Signal Processing Technical Committee and is an area
editor of IEEE Signal Processing Magazine. He was a technical chair ofthe IEEE International Conference on Acoustics, Speech, and Signal
Processing (ICASSP ’04) and is the general chair of the IEEE Workshop
on Multimedia Signal Processing, 2006. He is a fellow of the AcousticalSociety of America and a fellow of the IEEE.

1155

Shoji Makino received the BE, ME, and PhD
degrees from Tohoku University, Japan, in 1979,1981, and 1993, respectively. He joined NTT in
1981 and he is now an executive manager at theNTT Communication Science Laboratories. Heis also a guest professor at Hokkaido University.
His research interests include adaptive filtering
technologies and realization of acoustic echo
cancellation, blind source separation of convolutive mixtures of speech. He received the ICA
Unsupervised Learning Pioneer Award in 2006, the Paper Award of theIEICE in 2005 and 2002, the Paper Award of the ASJ in 2005 and 2002,the TELECOM System Technology Award of the TAF in 2004, the Best
Paper Award of the IWAENC in 2003, the Achievement Award of theIEICE in 1997, and the Outstanding Technological Development Awardof the ASJ in 1995. He is the author or coauthor of more than 200 articles
in journals and conference proceedings and is responsible for more than150 patents. He is a tutorial speaker at ICASSP ’07 and was a panelistat HSCMA ’05. He is a member of both the Awards Board and theConference Board of the IEEE Signal Processing Society. He is anassociate editor of the IEEE Transactions on Speech and Audio
Processing and an associate editor of the EURASIP Journal on Applied
Signal Processing. He is a guest editor of a special issue of the IEEE
Transactions on Audio, Speech and Language Processing. He is amember of the Technical Committee on Audio and Electroacoustics ofthe IEEE Signal Processing Society and the chair-elect of the TechnicalCommittee on Blind Signal Processing of the IEEE Circuits and Systems
Society. He is the chair of the Technical Committee on Engineering
Acoustics of the IEICE and the ASJ. He is a member of the International
IWAENC standing committee and a member of the International ICA
steering committee. He is the general chair of WASPAA ’07 in Mohonk,
was the general chair of IWAENC ’03 in Kyoto and the organizing chairof ICA ’03 in Nara. He is an IEEE fellow, a council member of the ASJ, amember of EURASIP, and a member of the IEICE.