A VOICE ACTIVITY DETECTION BASED ON THE ADAPTIVE INTEGRATION OF
MULTIPLE SPEECH FEATURES AND A SIGNAL DECISION SCHEME
Masakiyo Fujimoto, Kentaro Ishizuka, and Tomohiro Nakatani
NTT Communication Science Laboratories, NTT Corporation
2-4, Hikari-dai, Seika-cho, Souraku-gun, Kyoto, 619-0237, Japan
E-mail: {masakiyo, ishizuka, nak}@cslab.kecl.ntt.co.jp
ABSTRACTThis paper addresses the problem of voice activity detection (VAD)
in noisy environments. The VAD method proposed in this paper integrates multiple speech features and a signal decision scheme, namely
the speech periodic to aperiodic component ratio and a switching
Kalman filter. The integration is carried out by using the weighted
sum of likelihoods outputted from each VAD (stream). The stream
weight is decided adaptively each short time frame. The evaluation
is carried out by using a VAD evaluation framework, CENSREC1-C. The evaluation results revealed that the proposed method significantly outperforms the baseline results of CENSREC-1-C as regards VAD accuracy in real environments. In addition, we carriedout speech recognition evaluations by using detected speech signals,
and confirmed that the proposed method contributes to an improvement in speech recognition accuracy.
Index Terms— voice activity detection, periodic to aperiodic
component ratio, switching Kalman filter, adaptive integration
1. INTRODUCTION
Voice activity detection (VAD), which automatically detects a periodof target human speech from a continuously observed signal, is oneof the most important techniques for speech signal processing. VAD
is widely used in various speech signal processing techniques, e.g.,speech enhancement, speech coding for cellular or IP phones, andthe front-end processing of automatic speech recognition.
Usually, VAD consists of two parts: a feature extraction part anda decision part. The feature extraction part extracts acoustic features
for speech / non-speech discrimination, and the traditional features
are the zero-crossing rate and the energy difference between speechand non-speech [1]. However, these parameters are not robust in
the presence of interference noise, thus several noise robust features
have been proposed [2, 3]. These parameters can improve the VAD
accuracy. On the other hand, a statistical model-based VAD technique has been proposed as a robust decision mechanism by Sohn etal. [4]. This method defines a speech / non-speech state transition
model, and calculates the likelihood ratio of a speech state to a nonspeech state by using forward probability estimation. Sohn’s method
provides robust performance in noisy environments. However, thisperformance is restricted to specific environments. Namely, assumptions of stationary noise environments and a priori knowledge ofnoise are indispensable to Sohn’s method.
The applicable noise environment for VAD differs depending on
the feature parameter and decision scheme. It is difficult to copewith all noises observed in the real world by using only one method.
Thus, we investigate the VAD method with wide noise coverage by
combining multiple VAD methods. For the combination methods,
in this paper, we adopt a speech periodic to aperiodic component

Table 1. Comparison of VAD techniques
Clean
ITU-T G. 729B
ETSI ES 202 050
Sohn 1999
Ramirez 2004
PARADESKF
MUSCLE-VAD

°
4
°
°
°
°
°

VAD performance
Noise environments
StationNonary
stationary Burst
×
×
×
°
4
×
°
×
×
°
4
×
°
4
°
°
°
4
°
°
°

RTF
0.06
0.06
0.07
0.05
0.06
0.10
0.13

ratio (PAR)-based VAD [3] and the switching Kalman filter (SKF)based VAD [5]. The combination is carried out by employing theadaptive weighting sum of likelihood outputted from each method
independently. By using this approach, we can improve the VAD
performance compared with when using each method alone.
The proposed method was evaluated on the CENSREC-1-C (Corpora and Environments for Noisy Speech RECognition-1 Concatenated) [6], which is concatenated Japanese noisy speech data for
VAD evaluation. The evaluation results revealed that the proposed
method significantly improves VAD accuracy compared with the
CENSREC-1-C baseline. In addition, we confirmed that the proposed VAD improves the speech recognition accuracy of concatenated utterances.
2. COMPARISON OF EACH METHODThis section compares several VAD methods, and confirms the applicable and inapplicable noise environments for each method. Table
1 shows expected performance in several noise environments and thereal time factor (RTF) of each method. The RTF was measured byusing an Intel Pentium 4 3.6 GHz CPU.
Several VAD methods have been proposed, e.g., ITU-T recommendation G. 729 Annex B. [7], ETSI recommendation ES 202 050[8], the statistical model-based approach proposed by Sohn et al. [4],and the spectral divergence proposed by Ramirez et al. [2]. However, although these methods can be applied to stationary noise environments, it is difficult to apply them to non-stationary ot burst noise
environments.
Here, our proposed methods, PAR-based VAD [3] and SKFbased VAD [5], has cover a wide range of noise environments. As
shown in Table 1, the PAR-based VAD is robust for not only stationary noise but also burst noise, because almost burst noises have no
periodic characteristics. On the other hand, the SKF-based VAD isrobust for stationary and non-stationary noises, because the method
can estimate the time varying noise sequentially. However, eachmethod has inapplicable noise environments, i.e., PAR-based VAD

and SKF-based VAD are not robust for the noise with periodic characteristics and the burst noise, respectively. Consequently, by integrating each method effectively, the resulting method can cope withall noise environments with a practical RTF by compensating each
inapplicable noise environment.
3. PERIODIC TO APERIODIC COMPONENT RATIO
We first provide a short explanation of the PAR calculation (see [3]for details). With this calculation, the dominant harmonic component in the observed signal is referred to as the periodic component,
which is not always the target signal, and the other sound components are referred to as aperiodic components and include both environmental noise and the aperiodic components of target speech. Although the estimated power of the periodic component is affected by
the changes in the aperiodic components, this effect can be mitigatedin the PAR. Therefore, PAR is expected to be insensitive to dynamicchanges in noise power. We define the power of the observed signaloτ within a t-th short-time frame of length T as:
ρt =

T
−1
X

|ôτ |2 =

τ =0

M ˛
˛
1 X ˛ ST F S ˛2
˛Ot,m ˛ ,
M m=1

(1)

ST F S
where ρt is the power of the observed signal, and Ot,mis a shorttime Fourier spectrum (STFS) of ôτ = oτ gτ . gτ is a symmetricshort-time analysis window, and M is the number of STFS bins. Let
us suppose that the fundamental frequency (F0) at the t-th frame isalready obtained as f0t . Then, we assume the following equations todecompose the power ρt into the powers of its periodic and aperiodic
components, ρpt and ρat :ρt = ρpt + ρatρpt = η ·

h=1

η=

2

T
X
τ =1

(2)

Ht ˛
X

˛
˛ ST F S ˛2˛O p
˛˛ t,[hf0 ] ˛

!,

gτ2

(3)

t

T
X

!2
gτ

(4)

τ =1

˛2
Ht ˛
M ˛
˛
1 X ˛˛ ST F S ˛˛
1 X ˛ ST F S ˛2ρat =˛Oat,m ˛ =˛Oat,[hf0 ] ˛ ,
M m=1
Ht
t

(5)

h=1

FS
FS
where OpST
and OaST
are the STFSs of unknown periodic andt,mt,maperiodic components in ôτ , η is a normalization constant that represents the ratio of the power of a sinusoidal component to its power
spectrum, and [hf0t ] and Ht are the function that outputs an STFS
bin index corresponding to the h-th harmonic frequency and thenumber of harmonics defined based on f0t , respectively. In theabove decomposition, we estimate F0 by the autocorrelation method
widely used for estimating F0 [9]. Note that Eq. (3) can be viewed
as a kind of comb filtering, and that Eq. (5) represents the assumption that the power of the aperiodic components is widely distributed
over the entire frequency range in a manner that is independent fromthe frequencies of the periodic components. After certain mathematical manipulations described in [3] based on the assumptions given
by Eqs. (2) to (5), we can obtain the following:˛˛
PHt ˛˛ ST F S ˛˛2− Ht ρ t
h=1 ˛Opt, hf[ 0t ] ˛ρ̂pt = η(6)
1 − ηHt˛˛
P t ˛ ST F S ˛2˛O p˛ρt − η · H
h=1 ˛t,[hf0 ] ˛tρ̂at =,(7)
1 − ηHt

Clean speech model
Noise model

H0

H1

N t-1

Nt

N t+1

Composition

H0
N t-1

H0
Nt

H0
N t+1

H1
N t-1

H1
Nt

H1
N t+1

Noisy speech modelFig. 1. Speech/non-speech state transition model with noise dynamics. The symbols H0 and H1 denote the non-speech and speechstates, respectively. The symbol Nt denotes a noise state sequence.
where ρ̂pt and ρ̂at denote estimated values of ρpt and ρat . PAR canbe calculated by using the ratios of the estimated ρ̂pt and ρ̂at .
4. VAD BASED ON STATISTICAL MODEL4.1. Speech / non-speech state transition model
The proposed method discriminates between speech and non-speech
periods based on the likelihood ratio test (LRT) with a statisticalmodel.
As shown by the clean speech model in Fig. 1, the proposed
method trains Gaussian mixture models (GMMs) of clean speechand silence in advance by using a clean speech corpus.
Next, we assume that noise has non-stationary characteristics,
thus, the noise sequence is modeled by using a sequential state transition model as shown by the noise model in Fig. 1. With this method,
we assume that the noise statistics are not known in advance. Thus,
we estimate the parameters of the noise model sequentially by using
a Kalman filter.
Finally, by composing speech and noise models, we can construct the speech / non-speech state transition model with noise dynamics as shown by the noisy speech model in Fig. 1. Namely,
this model has state transition processes for both speech and noise.
Speech has a discrete state transition process and noise has a sequential process. By using this model, we can construct VAD that isrobust as regards a variety of speech and time varying noise.
4.2. Formulation of likelihood ratio calculation
This section describes the speech / non-speech discrimination method
based on the state transition model shown in Fig. 1.
In the proposed method, O0:t , N0:t , and qt denote the L-dimensional
vector of the log Mel spectra of the observed signal and noise at thet-th short time frame, and the speech or the non-speech state at the tth frame. When O0:t = {O0 , · · · , Ot } and N0:t = {N0 , · · · , Nt }are given, the state is decided with respect to the conditional probability p (qt |O0:t , N0:t ) as follows:p (qt |O0:t , N0:t )=∝p (O0:t , qt , N0:t ) /p (O0:t , N0:t )p (O0:t , qt , N0:t )(8)Here, we assume that qt and Nt are mutually independent, thus,
the recursive formula of p (O0:t , qt , N0:t ) is given by
p (O0:t , qt , N0:t )=X

p (qt |qt−1 ) p (Nt |Nt−1 ) p (Ot |qt , Nt )qt−1

×p (O0:t−1 , qt−1 , N0:t−1 ) .

(9)By defining p (qt = Hj |qt−1 = Hi ) = ai,j (the state transition probability of speech), p (Ot |qt = Hj , Nt ) = bj,Nt (Ot ) (the output
probability), and p (Nt |Nt−1 ) = ct,t−1 (the state transition probability of noise), αj,t = p (O0:t , qt = Hj , N0:t ) (the forward probability) is represented as the following equation from Eq. (9).
αj,t =

1X(ai,j αi,t−1 ) bj,Nt (Ot ) ct,t−1

(10)

i=0

In Eq. (10), ct,t−1 is set at 1, because we assume that the noise has
a continuous state transition process. Thus, Eq. (10) is simplified as
αj,t =

1X(ai,j αi,t−1 ) bj,Nt (Ot ) .

(11)

i=0

In Eq. (11), when t = 0, the current frame is assumed to be a nonspeech frame. Thus, the initial values α0,0 = 1 and α1,0 = 0 are
given.
Finally, the state qt is given by the LRT, namely, the thresholding
likelihood ratio Rt = α1,t /α0,t , as

H0 Rt < Threshold
.
(12)
qt =
H1 Rt ≥ Threshold
On the other hand, when we focus on the state transition noisemodel shown in Fig. 1, it is also given by the following equation.
This equation is completely equivalent to a statistical representationof a Kalman filter [10].
p (O0:t , N0:t ) = p (Nt |Nt−1 ) p (Ot |Nt ) p (O0:t−1 , N0:t−1 )(13)
If the probability (state) variable qt is added to Eq. (13), thestatistical process is equivalent to Eq. (9). This means that Eq. (9)is equivalent to a statistical representation of a switching Kalmanfilter that switches the state-space model of a Kalman filter based ona state variable. The details of the Kalman filter-based noise state
updating are provided in [5].
5. INTEGRATION OF VAD METHODSThis section describes the combination of PAR and SKF mentioned
in sections 3 and 4. The combination is carried out by employing theadaptive weighted sum of likelihoods outputted from each method
independently. Thus, the GMMs of clean speech and silence for PAR
are trained in advance, and the likelihood of PAR bj,P AR (P ARt )is given by the following equation.
bj,P AR (P ARt )
=

K
X

“
”
wP ARj,k N P ARt ; µP ARt,j,k , σP2 ARt,j,k (14)k=1When bj,P AR (P ARt ) is given, the likelihood of SKF bj,Nt (Ot )is added with weight γt as follows:
bj (Ot , P ARt ) = γt bj,Nt (Ot )+ (1 − γt )bj,P AR (P ARt ) , (15)where 0 ≤ γt ≤ 1. The forward probabilities for the LRT are
calculated by using combined likelihood bj (Ot , P ARt ) as follows:
αj,t =

1X(ai,j αi,t−1 ) bj (Ot , P ARt )

(16)

i=0

When γt = 0, the discrimination is carried out by PAR alone,and for γt = 1, the discrimination is carried out by SKF alone.

The adaptive frame by frame decision of the weight γt is givenby the following method.
First, bj,Nt (Ot ) and bj,P AR (P ARt ) are normalized as the total likelihood of non-speech state (j = 0) and speech state (j = 1)
equal to 1. Next, the absolute differences of the likelihoods of thenon-speech state and the speech state DSKF,t and DP AR,t are calculated by the following equations.
DSKF,t = |b0,Nt (Ot ) − b1,Nt (Ot )|(17)

DP AR,t = |b0,P AR (P ARt ) − b1,P AR (P ARt )|(18)

When DSKF,t or DP AR,t has a large value, it shows that thedecision confidence is high regarding whether the current frame belongs to a non-speech state or a speech state. When the value are low,it shows that the decision confidence is low. The weight γt given byusing DSKF,t and DP AR,t as follows:γt = DSKF,t / (DSKF,t + DP AR,t )(19)

Equation (19) shows that the weight of the SKF increases ordecreases according to the confidence of the SKF. Thus, by using
this weight, we can select the most suitable method for the current
frame, frame by frame.
6. EXPERIMENTS
6.1. Experimental setup
The proposed method was evaluated by using the CENSREC-1-C
database [6]. CENSREC-1-C was designed as an evaluation framework for VAD in noisy environments and has two types of evaluation
data set, i.e., simulated data and real recorded data. In this paper, wechose the real recorded data set for the evaluation.
The data was recorded in two real noisy environments (a restaurant (Rest.) and a street (St.)) with two different sound pressurelevels (avg. 60 dBA: high SNR (Hi.) and avg. 70 dBA: low SNR(Lo.)). The data were originally recorded at a sampling rate of 48
kHz (with 16 bit quantization), and were down-sampled to 8 kHz.
There were ten speakers (five males and five females). The recorded
speech consisted of four files per subject. A single file included 8-10
utterances of continuous numbers consisting of 1-12 digit numberswith two-second intervals between each utterance in each noisy environment and for each SNR condition. The correct segment labels
were tagged manually.
The feature parameters for the PAR-based VAD and SKF-based
VAD were 1st order PAR and 24th order log-Mel spectra, respectively, which were extracted by using a Hamming window with a
25 msec frame length and a 10 msec frame shift length. We trained
the silence and clean speech GMMs for PAR-based VAD and SKFbased VAD by using clean speech data for the HMM training of
CENSREC-1 (AURORA-2J) [11]. Each GMM has 32 Gaussian distributions. The training data consisted of 8,440 utterances spoken by
110 speakers. The state transition probabilities of the clean speechmodel were set at ai,j = {0.90, 0.10, 0.45, 0.55}.
6.2. Experimental results of VAD
In the evaluation, we compare the VAD performance of the proposed
method with the CENSREC-1-C baseline, Sohn’s method [4], PARalone (γt = 0), and SKF alone (γt = 1). The baseline VAD technique of CENSREC-1-C is energy-based VAD with adaptive thresholding.
The evaluation criteria are the utterance correct rate and utterance accuracy rate as shown by Eqs. (20) and (21).
Corr = Nc /N × 100 [%]Acc = (Nc − Nf )/N × 100 [%] ,

(20)
(21)

Table 2. VAD results (%)

Baseline
Sohn
PAR
SKF
PAR+SKF

Rest. Hi.
74.20
72.75
70.72
89.57
93.04

Corr(%)
Rest. Lo.
St. Hi.
56.52
39.42
57.10
97.39
57.10
87.25
66.96 100.00
70.72 100.00

Rest. Hi.
45.17
44.16
52.67
37.45
39.76
43.75
46.85

Word accuracy (%)
Rest. Lo. St. Hi.
St. Lo.
1.28
34.43
25.23
18.12
29.96
21.62
29.17
41.25
29.50
-3.81
33.41
29.58
8.89
39.16
24.08
12.50
46.99
33.15
18.67
47.27
33.52

St. Lo.
41.45
78.55
80.58
97.97
97.97

Ave.
52.90
76.45
73.91
88.63
90.43

Rest. Hi.
21.45
45.51
24.35
68.41
72.75

Rest. Lo.
-43.48
-6.38
-6.67
12.46
19.71

Acc(%)
St. Hi.
-15.65
94.49
64.35
97.68
99.13

St. Lo.
-33.91
57.39
54.49
93.62
94.78

Ave.
-17.90
47.75
34.13
68.04
71.60

Table 3. Speech recognition results after VAD (%)

w/o VAD
Baseline
Ideal VAD
Sohn
PAR
SKF
PAR+SKF

where N , Nc , and Nf denote the total number of speech utterances,
the number of correctly detected utterances, and the number of incorrectly detected utterances, respectively.
Table 2 shows the results of an utterance-level evaluation. As
seen in the table, the proposed method “PAR+SKF” significantly
improves both Corr and Acc compared with the baseline. In particular, the average improvement in Acc when compared with thebaseline was approximately 83 %. In addition, the proposed method
improves both Corr and Acc compared with PAR and SKF. Thismeans that the proposed likelihood combination works effectively.
6.3. Experimental results of speech recognition
We also carried out an evaluation of speech recognition with theproposed method. We used the HTK (HMM Tool Kit) [12] forspeech recognition and acoustic model training. The acoustic modelis trained as whole word (digit) HMMs (16 states, 20 Gaussian distributions per state) by using clean training data from CENSREC1. The feature parameters used in this evaluation consisted of 39
MFCCs with 12 MFCCs, log-energy, and their first and second orderderivatives. Cepstral mean normalization was not applied at the feature extraction. A more detailed evaluation scheme for CENSREC-1
is described in [11].
Table 3 shows the speech recognition results in terms of word accuracy. In the table, “w/o VAD” and “Ideal VAD” represent speech
recognition results without VAD and with VAD using hand labeledutterance boundaries, respectively. The table shows that the proposed method “PAR+SKF” improves speech recognition accuracy.
As regards the speech recognition results obtained with the proposed
method, there was an increase in the deleted word and substituted
word errors caused by VAD errors. However, the insertion word errors, especially in the silent periods between utterances, was significantly reduced. Therefore, we can confirm that the proposed method
contributes to an improvement in speech recognition accuracy by reducing insertion word error.
7. CONCLUSION
This paper presented a noise robust VAD technique based on the integration of multiple speech features and a signal decision scheme.
The proposed method combined a speech periodic to aperiodic component ratio and a switching Kalman filter with adaptive weighting. The evaluation results show that our proposed method improves
VAD accuracy compared with the periodic to aperiodic component
ratio or the switching Kalman filter alone. In addition the proposed

Ave.
26.53
28.47
38.15
24.16
27.97
34.10
36.58

Error reduction rate from w/o VAD (%)
Rest. Hi. Rest. Lo. St. Hi.
St. Lo.
Ave.
0.00
0.00
0.00
0.00
0.00
-1.84
17.06
-6.82
-4.83
2.64
13.68
28.25
10.40
5.71
15.82
-13.83
-6.22
-1.46
5.55
-3.31
-9.87
7.71
7.21
-1.54
1.97
-2.59
11.37
19.16
10.59
10.30
3.06
17.62
19.58
11.09
13.68

method also improves speech recognition accuracy. In the future, weare planning to investigate the optimal threshold decision.
8. ACKNOWLEDGEMENTS
The present study was conducted using a CENSREC-1 database anda CENSREC-1-C database developed by the IPSJ-SIG SLP Noisy
Speech Recognition Evaluation Working Group.
9. REFERENCES[1] Rabiner, L. R. et al., “An algorithm for determining the endpoints of isolated utterances,” The Bell System Technical Journal, Vol. 54, No. 2, pp. 297–315, Feb. 1975.
[2] Ramirez, J. et al., “Efficient voice activity detection algorithm
using long-term speech information,” Speech Communication,
Vol. 42, pp. 271–287, Apr. 2004.
[3] Ishizuka, K. et al., “Study of noise robust voice activity detection based on periodic component to aperiodic component
ratio,” Proc. of SAPA ’06, pp.65–70, Sept. 2006.
[4] Sohn, J. et al., “A statistical model-based voice activity detection,” IEEE Signal Processing Letters, Vol. 6, No. 1, pp. 1–3,
Jan. 1999.
[5] Fujimoto, M. et al., “Noise robust voice activity detection
based on switching Kalman filter,” Proc. of Intespeech ’07, pp.
2933-2936, Aug. 2007.
[6] CENSREC-1-C Web site, http://sp.shinshu-u.ac.jp/
CENSREC/en/CENSREC/CENSREC-1-C/[7] ITU-T Recommendation G.729 Annex B., “A silence compression scheme for G.729 optimized for terminals conforming to
Recommendation V.70,” Nov. 1996.
[8] ETSI standard document, “Speech processing, transmission
and quality aspects (STQ), Advanced distributed speech recognition; Front-end feature extraction algorithm; Compression
algorithms,” ETSI ES 202 050 v.1.1.4, Nov. 2005.
[9] Hess, W., “Pitch Determination of Speech Signals,” Springer–
Verlag, New York, 1983.
[10] Arulampalam, M. S. et al., “A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking,” IEEE Trans.
on Signal Processing, Vol. 50, No. 2, pp. 174–188, Feb. 2002.
[11] Nakamura, S. et al., “Data collection and evaluation of
AURORA2-J Japanese corpus,” Proc. of ASRU ’03, pp. 619–
623, Dec. 2003.
[12] HTK Web site, http://htk.eng.cam.ac.uk/