Missing Feature Speech Recognition in a Meeting
Situation with Maximum SNR Beamforming
Dorothea Kolossa∗† , Shoko Araki† , Marc Delcroix† , Tomohiro Nakatani† , Reinhold Orglmeister∗ , Shoji Makino†∗ Electronics and Medical Signal Processing, TU Berlin

10587 Berlin, Germany
Email: d.kolossa@ee.tu-berlin.de† NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
Email: {shoko,marc.delcroix,nak,maki}@cslab.kecl.ntt.co.jp
Abstract— Especially for tasks like automatic meeting transcription, it would be useful to automatically recognize speechalso while multiple speakers are talking simultaneously. For thispurpose, speech separation can be performed, for example byusing maximum SNR beamforming. However, even when goodinterferer suppression is attained, the interfering speech will still
be recognizable during those intervals, where the target speaker issilent. In order to avoid the consequential insertion errors, a newsoft masking scheme is proposed, which works in the time domain
by inducing a large damping on those temporal periods, where
the observed direction of arrival does not correspond to that of
the target speaker. Even though the masking scheme is aggressive,by means of missing feature recognition the recognition accuracy
can be improved signiﬁcantly, with relative error reductions in theorder of 60% compared to maximum SNR beamforming alone,
and it is successful also for three simultaneously active speakers.
Results are reported based on the SOLON speech recognizer,
NTT’s large vocabulary system [1], which is applied here for therecognition of artiﬁcially mixed data using real-room impulse
responses and the entire clean test set of the Aurora 2 database.

I. I NTRODUCTION
Many source separation methods are optimized for situations, in which the number of speakers is ﬁxed, and known
beforehand. However, this assumption is unrealistic for manypotential real-world applications, such as automatic meetingtranscriptions. For such situations, in which the number andposition of active speakers often changes over time, more ﬂexible source separation architectures are needed. In this paper,
the use of maximum SNR beamforming [2] with adaptive
online clustering is investigated for application together with
automatic speech recognition and it is modiﬁed to also include
a time-masking which improves separation performance.
However, such quickly time-varying nonlinear masking is
generally detrimental for speech recognition, therefore, insteadof using a standard HMM speech recognition engine, the
SOLON large vocabulary speech recognizer has been modiﬁed
to be able to include additional uncertainty information in
the recognition process. This information is won from thepreprocessing module, and passed on to the recognizer, as
shown in the overview in Figure 1. Since the recognition
takes places on mel-cepstrum features Ỹ (c, τ ), rather thanthe STFT-coefﬁcients ỹ(f, τ ) which are computed by the

978-1-4244-1684-4/08/$25.00 ©2008 IEEE

Fig. 1.

Overall Structure of the System

preprocessing stage, an additional module for transforming thefeatures ỹ(f, τ ) with associated uncertainties σỹ (f, τ ) fromthe spectrum to the mel-cepstrum features Ỹ (c, τ ) with theiruncertainties σỸ (c, τ ) is necessary. This transformation isachieved here by means of the unscented transform.
II. M IXING M ODELIt is assumed here, that noisy, convolutive mixing of N
speech sources takes place. This means that the M observed
microphone signals xj (t), j = 1 . . . M are composed ofxj (t) =

N 

k=1

hjk (l)sk (t − l) + nj (t),(1)

lwhere hjk stands for the room impulse response from source k
to sensor j, and where nj is the sensor noise at sensor j. Aftera T -point short time Fourier transform, the mixing model (1)simpliﬁes toxj (f, τ ) =Nhjk (f )sk (f, τ ) + nj (f, τ ),

(2)

k=1



where f ∈ 0, . . . , T T−1 fs , fs is the sampling frequency andτ denotes the frame index.
III. M AXIMUM SNR B EAMFORMING
Maximum SNR beamforming attempts to maximize the
Signal-to-Noise-Ratio (SNR) at the beamformer output. In
order to achieve this, it is necessary ﬁrst, to distinguish targetactive periods PTk from target-inactive-periods PIk for eachsource k. As described in [2], this can be achieved by a threestage process.

3218

Fig. 2.

Block diagram of maximum SNR beamformer

First, noise only periods are determined by statistical,
model-based voice activity detection according to [3].
This results in a distinction between speech frames PS
and noise frames PN .
• Secondly, the generalized cross correlation method withphase transform (GCC-PHAT) [4] is applied for determining time-differences of arrival (TDOAs) q in each of
the frames.
• Finally, all speech frames PS are clustered according totheir TDOA by means of an online clustering method,which does not require advance knowledge about the
number of active sources [5].
This ﬁnal clustering yields the target-active periods PTk foreach of the sources k. Interference periods are assumed foreach source, whenever the source is not deemed active, thus
PIk = P − PTk and the set of all frames P is therefore
partitioned into two disjoint sets for each of the sources. Usingtarget-active-periods PTk and source-inactive periods PIk forsource k, the beamformer weights are obtained by optimizing
the ratio between the output energy in target-active periods
and the energy in interference periods for each of the sources
k = 1 . . . N according to•

wk,opt (f ) = arg max

E(|yk (f, τ )|2 )PTk

wk (f ) E(|yk (f, τ )|2 )P k

Fig. 3. Histogram of estimated DOAs and approximation with Gaussian
distribution.

where wk,opt (f ) is the eigenvector associated with the largest
eigenvalue λmax (f ). This generalized eigenvector problem
can also be simpliﬁed to an eigenvalue problem by leftmultiplying both sides with RkI (f )−1 . Finally, the output
signal for source k is obtained byyk (f, τ ) = wk,opt (f )H x(f, τ ).

Here, the signal vector x(f, τ ) is composed of all microphonesignals [x1 (f, τ ) . . . xM (f, τ )]T .
IV. T IME M ASKING
Maximum SNR beamforming by itself already leads to asigniﬁcant suppression of the interfering speaker, for example,in [2], a signal to interference ratio (SIR) of 12dB was attained
for a reverberation time of 350ms. However, for the task ofrobust speech recognition, greater SIR values are needed inorder to avoid insertion errors during those periods, where
the target speaker is inactive. Therefore, in order to improverecognition results, a time-varying damping was applied tothe beamformer output, which introduces additional damping
according toỹk (f, τ ) = Dk (τ ) · yk (f, τ ).

I

wH (f )RkT (f )wk (f )= arg max kH.
wk (f ) wk (f )Rk
I (f )wk (f )(3)

The correlation matrix for target-active periods RkT is calculated by
RkT = E x(f, τ )x(f, τ )H P k
T
1 
= k
x(f, τ )x(f, τ )H
(4)
|PT |
k
τ ∈PT

and RkI is obtained in the same manner via
RkI = E x(f, τ )x(f, τ )H P k
I
1 
= k
x(f, τ )x(f, τ )H .
|PI |
k

(5)

τ ∈PI

Finally, the optimum beamformer weights wk,opt (f ) are obtained by differentiating (3) with respect to wk . This resultsin the eigenvector problem
RkT (f )wk (f ) = λ(f )RkI (f )wk (f ),

(6)

(7)

(8)

The damping factor Dk is obtained by comparing the directionof arrival (DOA) for each time frame, θ(τ ), with a source DOA
model P (θ(τ )|sk ) for source sk . For this purpose, the TDOA
which is already available for each time frame τ , is converted
to a DOA value with the help of the known array geometry[6]. Then, a statistical model of source DOAs can be learned
for each source k by ﬁtting a model to the observed DOAs in
those frames where source k is active, i.e. in which τ ∈ PTk .
As Figure 3 shows, even a simple Gaussian model may be
sufﬁcient. Such a model can be learned online, and it gives
the probability P (θ(τ )|sk ) for each frame τ and each sourcek. This probability was ﬁrst used, to obtain a mask accordingto D1k (τ ) = P (θ(τ )|sk ). However, the probability is very
quickly time-varying, so that low-energy frames within words
might be corrupted by application of the mask. Therefore, asimple smoothing ﬁlter was applied to the intermediate mask
D1 by
L

1
D1k (τ − i),(9)D2k (τ ) =2L + 13219

i=−LFig. 4.

Calculation of Mask from Source Probability

with L = 7 and a ﬁnal compansion
Dk (τ ) = L(D2k (τ ), α, ϑ)(10)with the logistic function
L(D, α, ϑ) = α +1−α1 + exp(10(ϑ − D))

−∞

(11)

ensures that the mask Dk (τ ) remains between a small constant
α > 0, and 1 as an upper limit. The lower bound α and thethreshold ϑ of L were set to α = 0.03 and ϑ = 0.5.
V. M ISSING F EATURE R ECOGNITION
A. Uncertainty Calculation
To estimate uncertainty values, it was assumed that the
maximum SNR beamformer leaves residual interference in thesignal, which is damped with a damping factor d. Thus, theremaining uncertainties are estimated by taking a summation
over all estimated non-target outputs, each damped with thefactor d, according toσỹk (f, τ ) =Nd|ỹi (f, τ )|.

which gives the best match to the feature vector sequence[Y(1), . . . , Y(E)] when each HMM state has an associatedoutput probability distribution p(Y|q). Y(i) denotes the ith
feature vector deﬁned as Y(i) = [Y (1, i), ..., Y (F, i)]T , wherei = 1, ..., E and F and E are the number of features and thenumber of frames, respectively.
1) HMM Variance Compensation: In HMM variance compensation, the computation of state output probabilities is
modiﬁed to incorporate frame-by-frame and feature by feature
uncertainties [9]. The formulation ∞p(Ỹ(τ )|S(τ ))p(S(τ )|q)dS(τ )(13)p(Ỹ(τ )|q) =leads top(Ỹ(τ )|q) = N (Ỹ(τ ); μq , σq 2 + σỸ (τ )2 ),where q denotes the HMM state, with the parameters μq andσq , Ỹ(τ ) is the feature vector at frame τ with the associatedfeature uncertainty σỸ (τ ), and S stands for the clean speechfeature vector.
2) Modiﬁed Imputation: In modiﬁed imputation, the idea
is replacing the imputation equation, originally proposed forcompletely missing features in [10], by an alternative formulation, which also allows for real-valued degrees of uncertainty.
Thus, whereas missing parts of feature vectors are replaced by
the corresponding components of the HMM model mean μq inclassical imputation, modiﬁed imputation ﬁnds the maximum
likelihood estimate(12)Ŷ(τ ) = arg max p(Y(τ )|q, x(τ )).

i=1,i=kY(τ )The damping factor was set to d = 0.1 for all experiments.
B. Uncertainty Transformation
After the preprocessing stage, processed features ỹk (f, τ )and associated uncertainty values σỹk (f, τ ) are available in
the time-frequency domain. However, since speech recognition
can be performed far more robustly in other domains, using
features such as the mel-cepstrum or RASTA-PLPs, a transformation of these uncertain features to the domain of speech
recognition is necessary. This transformation can be calculated
analytically for speciﬁc transformations, or it can be obtainedin a general fashion by means of the unscented transform[7], which is the way chosen for all subsequent experiments.
Then, after the unscented transform, features Ỹk (c, τ ) andassociated uncertainty values σỸk (c, τ ) are available in themel-cepstrum domain. This is the basis for missing-feature
speech recognition.
C. Uncertain Recognition
Speech recognition was performed with the SOLON recognizer, NTT’s large vocabulary recognition system. Two
missing feature approaches, modiﬁed imputation [8] andHMM variance compensation [9], have been implemented for
SOLON and were used in the tests. Both methods are developed for HMM based systems, where recognition takes placeby ﬁnding the optimum HMM state sequence [q1 , . . . , qE ],

(14)

(15)

For a single Gaussian, and assuming a ﬂat prior for Y(τ ),Equation (15) leads to−1 μq σq −1 + Ỹ(τ )σỸ (τ )−1 ,Ŷ(τ ) = σỸ (τ )−1 + σq −1
as shown in [8], but better results are achieved with anadditional uncertainty weight w according to−1 μq σq −1 + Ỹ(τ )(wσỸ (τ ))−1Ŷ(τ ) = (wσỸ (τ ))−1 + σq −1
and with an empirically chosen w = 0.1.
VI. E XPERIMENTS AND R ESULTS
A. Simulations with Recorded Impulse Responses
Recordings of impulse responses, made in a meeting roomwith the reverberation time trev ≈ 100ms and the layout
shown in Figure 5, have been used to generate artiﬁcial
mixtures of the clean test set of the Aurora 2 database. The
clean test set consists of 1081 utterances with 12147 words.
B. Recognizer Parameters
The sampling rate of the data is 8kHz. Before feature extraction, the signal is highpass ﬁltered with a cutoff frequencyof 64Hz and a ﬁrst order FIR preemphasis ﬁlter with the
coefﬁcient a0 =0.97 is applied. The STFT windowsize is 30ms
with 20ms overlap. The acoustic model consists of speakerindependent, word based HMMs trained on clean speech and3220

TABLE IIR ESULTS FOR R ECOGNITION OF AURORA T ESTSET, USING MIXTURES OF
THREE SPEAKERS .

Clean
Mixtures
MaxSNRMaxSNR + Mask
MaxSNR + Mask + MI
MaxSNR + Mask + VC

Fig. 5.

TABLE IR ESULTS FOR R ECOGNITION OF AURORA T ESTSET, USING MIXTURES OFTWO SPEAKERS AT 210 ◦ AND 315 ◦ IN F IGURE 5.

S
0.9
20.9
5.2
8.0
6.9
6.9

D
0.6
2.3
0.8
2.2
3.0
2.9

I
1.1
48.2
39.4
27.5
8.5
9.7

D
0.6
3.6
2.0
2.9
4.7
4.4

I
1.1
71.5
60.4
48.8
28.2
28.7

Word Accuracy
97.4
-9.1
23.5
33.2
52.4
52.5

by means of subsequent uncertain recognition, where theresults of modiﬁed imputation and of variance compensation
are similar, with an overall accuracy of 52.4% respectively
52.5%.

Recording setup for obtaining room impulse responses

Clean
Mixtures
MaxSNRMaxSNR + Mask
MaxSNR + Mask + MI
MaxSNR + Mask + VC

S
0.9
33.9
14.2
15.5
14.7
14.4

Word Accuracy
97.4
28.5
54.4
62.3
81.6
80.5

each state uses a single Gaussian distribution to model the
39-dimensional feature vector consisting of 13 mel frequency
cepstral coefﬁcients, including the zero’th coefﬁcient, deltas
and acceleration features.
C. Recognition Results
Table I shows the results for clean data, artiﬁcial mixtures
and the results of the various intermediate stages of processing.
S stands for the percentage of substitution errors, D and I are
the deletions respectively insertions and the Word Accuracy is
obtained as 100−S −D−I. Using maximum SNR beamforming alone increases the word accuracy from 28.5%, measured
with the artiﬁcial mixtures of two speakers, to 54.4%. Mostof the remaining errors are due to insertions, which, at 39.4%,
make up 86% of the total error rate. Subsequent application of
the masking function reduces the insertion error rate to 27.5%.
Further use of the available uncertainty information leads
to signiﬁcant further improvements. By means of modiﬁed
imputation (MI), the ﬁnal word accuracy is 81.6%, whereas
variance compensation (VC) leads to 80.5% word accuracy.
The situation is similar in the second test setup, where artiﬁcial
mixtures of three speakers are used. The results for this setup
are shown in Table II. Here, again, insertion errors make up
the bulk of the error rate, with 71.5% insertion errors for the
mixtures and 60.4% for the beamformer output, and this maincause of recognition errors can be signiﬁcantly reduced bymeans of time-masking. The lowest rate of errors is achievedVII. C ONCLUSIONS
Maximum SNR beamforming is a promising approach forthe separation of speech in meeting situations, since, unlike forclassical ICA, neither the number nor the location of the speakers is required to be known or constant over time. In order to
use this method for meeting transcription, however, greaterinterference suppression is necessary especially during target
inactive periods. For this purpose, a time-varying damping has
shown good results, for two as well as for three mixed speech
signals. Further improvements can be obtained, and especiallythe number of insertions can be reduced signiﬁcantly, when
this postprocessing is used in conjunction with missing feature
recognition, which has been applied here in the decoding stageof the SOLON large vocabulary speech recognizer.
R EFERENCES[1] T. Hori, “NTT speech recognizer with outlook on thenext generation: SOLON,” in Proc. NTT Workshop on
Communication Scene Analysis, 2004, pp. SP–6. [Online]. Available:
www.kecl.ntt.co.jp/icl/signal/hori/publications/thori_csa2004.pdf[2] S. Araki, H. Sawada, and M. Shoji, “Blind speech separation in a
meeting situation with maximum SNR beamformers,” in Proc. ICASSP,
vol. 1, Hawaii, 2007, pp. 41–44.
[3] J. Sohn, N. Kim, and W. Sung, “A statistical model-based voice activity
detection,” Signal Processing Letters, vol. 6, no. 1, pp. 1–3, 1999.
[4] C. H. Knapp and G. C. Carter, “The generalized correlation method for
estimation of time delay,” IEEE Trans. Acoustics, Speech and Signal
Processing, vol. 24, no. 4, pp. 320–327, 1976.
[5] R. Duda, P. Hart, and D. Stork, Pattern Classiﬁcation, 2nd ed. Wiley
Interscience, 2000.
[6] S. Araki, H. Sawada, R. Mukai, and S. Makino, “DOA estimation for
multiple sparse sources with normalized observation vector clustering,”
in Proc. ICASSP, vol. 5, May 2006, pp. 33–36.
[7] R. Astudillo, D. Kolossa, and R. Orglmeister, “Propagation of statisticalinformation through non-linear feature extractions for robust speech
recognition,” in Proc. MaxEnt2007, 2007.
[8] D. Kolossa and R. Orglmeister, “Separation and recognition of noisy,
convolutive speech mixtures using time-frequency masking and missing
data techniques,” in Proc. WASPAA 2005, ser. Lecture Notes in Computer
Science, vol. 3195. Springer, 2005, pp. 832–839.
[9] L. Deng, J. Droppo, and A. Acero, “Dynamic compensation of HMM
variances using the feature enhancement uncertainty computed from aparametric model of speech distortion,” IEEE Trans. Speech and Audio
Processing, vol. 13, no. 3, pp. 412–421, May 2005.
[10] J. Barker, P. Green, and M. Cooke, “Linking auditory scene analysis androbust ASR by missing data techniques,” in Proceedings WISP 2001,
2001.

3221