# Multiengine Speech Processing Using SNR Estimator in Variable Noisy Environments

Authors: Ahmad R. Abu-El-Quran; Adrian D. C. Chan; Rafik A. Goubran
Journal: Advances in Acoustics and Vibration
 (2012)

---

## Abstract

We introduce a multiengine speech processing system that can detect the location and the type of audio signal in variable noisy environments. This system detects the location of the audio source using a microphone array; the system examines the audio first, determines if it is speech/nonspeech, then estimates the value of the signal to noise (SNR) using a Discrete-Valued SNR Estimator. Using this SNR value, instead of trying to adapt the speech signal to the speech processing system, we adapt the speech processing system to the surrounding environment of the captured speech signal. In this paper, we introduced the Discrete-Valued SNR Estimator and a multiengine classifier, using Multiengine Selection or Multiengine Weighted Fusion. Also we use the SI as example of the speech processing.
The Discrete-Valued SNR Estimator achieves an accuracy of 98.4% in characterizing the environment's SNR. Compared to a conventional single engine SI system, the improvement in accuracy was as high as 9.0% and 10.0% for the Multiengine Selection and Multiengine Weighted Fusion, respectively.


---

## Body

## 1. Introduction

Speech processing systems, such as Speaker identification (SI) and Automatic Speech Recognition (ASR), have two operating modes: a training mode and a testing mode. The goal of the training mode is for the classifier to learn how to discern between different speech classes, using exemplars whose classes are known. The purpose of the testing mode is to classify speech data, from an unknown class, based on the training.The performance of most speech processing systems is degraded severely when the training and the testing audio are captured in different acoustic environments (e.g., different levels of noise [1–4]). For example, training speech may be captured using a telephone handset in a noise-free environment, while the test speech is captured using a hands-free phone in a noisy environment. The conventional method to improve the performance of speech processing in noisy environments is to perform speech enhancement prior to speech processing [2–7]. In this method, training speech signals are obtained in a clean environment (i.e., no noise). The test speech signals, however, are obtained in environments that may contain noise. Denoising and/or speech enhancement methods may be applied to these signals to reduce the effects of the environment; however, often these methods enhances the speech signal audibility from the human perspective, but ignores the fact that the enhancements may remove or distort information in the speech signal, which may be essential to the speech processing.In [2], the effect of five different speech enhancement techniques on a GMM-based ASR system was examined. These enhancement techniques were (1) perceptual wavelet adaptive denoising (PWAD) (TPS-PWAD), (2) VAD-based noise floor estimation (VAD-NFE), (3) minimum-mean-square-error-based two-dimensional spectral enhancement (MMSE-TDSE), (4) two-pass adaptive noise estimation (TPANE), and (5) combined two-pass spectral and PWAD. The performance of the speech enhancement techniques was evaluated using Perceptual Evaluation of Speech Quality-Mean Opinion Score (PESQ-MOS) [7, 8]. PESQ-MOS predicts the subjective quality of input speech by returning a score between 0 and 4.5; the higher the score, the cleaner the input speech (i.e., higher SNR). These enhancement techniques were used as preprocessing blocks to the ASR engine. The overall performance was evaluated based on the ASR classification accuracy using each enhancement technique.The author of [2] noted the following.(i)
The speech enhancement techniques of the corrupted speech signal with white noise generally improved the classification accuracy of the ASR; however, this improvement is inconsistent for differentSNR values for a single enhancement technique.(ii)
The MMSE-TDSE has the greatest average improvement in the speech quality PESQ-MOS score, overSNR range from 0 to 20 dB, an average improvement of 0.7 points. This improvement, however, did not translate into the best ASR classification accuracy.(iii)
AtSNR=20 dB, the MMSE-TDSE speech enhancement resulted in a decrease in ASR classification accuracy of −0.71%, relative to ASR without speech enhancement techniques.(iv)
The VAD-NFE achieved the bestSNR improvement of 0.5 dB and 3.7 dB at 20 and 15 dB, respectively; however, this did not translate into the best ASR classification accuracy at these SNR values.The results of [2] suggest that optimizing the quality of speech perception or SNR, using speech quality enhancement techniques, does not optimize speech processing performance. Indeed, speech quality enhancement techniques may actually degrade the speech processing performance instead of improving it.In [5], the combination of four speech enhancement techniques have been combined to enhance the ASR in noisy environment. These speech enhancement techniques were (1) spectral subtraction with smoothing of time direction, (2) temporal domain SVD-based speech enhancements, (3) GMM-based speech estimation, and (4) KLT-based comp-filtering. In [5], two combination methods have been introduced to combine the speech enhancement techniques: selection of front-end processor and combination of results from multiple recognitions processors. The speech enhancement techniques are applied on the captured noisy speech. A combination of these speech enhancement techniques is combined using one of the two proposed combination methods. The overall performance was evaluated based on the ASR classification accuracy using the two combination techniques. The authors of [5] reported an ASR accuracy enhancement of 5.4% of the proposed combination methods; compared to best performing standalone speech enhancement technique. This combination has been achieved using a noise environment detector with a detection accuracy of 54%. Using an ideal noise environment detector with detection accuracy of 100% can enhance the ASR accuracy with only 1.9%, compared to the environment detector with detection accuracy of 54%. As indicated in [5] the main factor of enhancing the ASR accuracy is the proposed combination method using different speech enhancement techniques. Meanwhile the authors [5] did not discussed the fact indicated in [2] that speech enhancement techniques may actually degrade the speech processing performance instead of improving it.In this paper, we introduce a speech processing system that adapts to the surrounding environment of the speech signal, rather than trying to adapt the speech signal to the system; in this system, no speech enhancements are applied to the captured speech signal. This speech processing system characterizes its environment from the captured speech signal and adapts the speech processing to better suit the environment, enhancing the overall performance in variable environments. In this paper, speaker identification (SI) is used as the speech processing system.TheSI system proposed in this paper uses multiple SI engines, each trained in a noisy environment but with different SNR levels. Two multiengine fusion SI systems are proposed: Multiengine Selection and Multiengine Weighted Fusion (Figure 1). These proposed systems permit the intelligent fusion of the multiple SI engines using noise information of the surrounding environment. The noise information is determined by a Discrete-Valued  SNR Estimator from the incoming speech signal. The proposed system enhances the SI performance, improving classification accuracy and robustness to different noise conditions.Figure 1
MultiengineSI system.The multiengine fusion system is being proposed for use in a novel security instrument using microphone array. This security instrument is capable of robustly determining the identity and location of a talker, even in a noisy environment. The instrument is also capable of identifying the audio type and location of nonspeech audio signals (e.g., footsteps, windows breaking, and cocktail noise) [9]. This instrument can be incorporated in a variety of applications, such as hands-free audio conferencing systems to perform voice-based security authentication for the conference users [10]. Also, this instrument can be utilized in smart homes and health monitoring for older adults that will support independent living and enable them to continue to age in their own homes [11]. Smart homes can benefit from this instrument by monitoring their occupants through analysis of the audio signals present in the home and recognition of certain sound sources. For instance, respiration, falls to the ground, and water leaking sounds could be detected. The proposed instrument could perform further analyses and make decisions according to the captured audio type, such as extracting the respiration rate, calling emergency help, or alerting the occupant that a plumber may be needed. The analysis of the audio data preserves the privacy of the users better than video monitoring.Previously mentioned applications require high performance to have successful functionality. In addition, the audio processing system needs to be robust even in variable conditions, such as environments with different levels of noise. Poor instrument robustness would limit utility of these systems.The remainder of the paper is organized as follows. In Section2 of this paper, we describe the overall implementation of the proposed approach. This section introduces and discusses the speech/nonspeech classifier, the Discrete-Valued SNR Estimator, the multiengine SI system, and the two methods of multiengine fusion: Multiengine Selection and Multiengine Weighted Fusion. Performance of the multiengine SI system is evaluated using a library of speech signals in noisy environments. In Section 3, results of this performance evaluation are discussed and compared against the baseline method of SI. Major conclusions are summarized in Section 4.


## 2. Proposed System

### 2.1. Instrument Overview

A block diagram of the overall instrument is shown in Figure2. The microphone array captures the audio signal then sends the digitized audio signal to the localization block to determine the location of the sound source [12, 13] in addition to steering the microphone array to reduce undesired acoustic noise.Figure 2
Block diagram of proposed system.A speech/nonspeech classifier is used as a frontend to appropriately select the next stage in audio processing, which is either theSI system or the nonspeech audio classification system [9]. The speech/nonspeech classifier is also adapted to the environment, so that it is robust to varying levels of noise. In this paper, we utilize the Discrete-Valued SNR Estimator (Section 2.4) to adapt the speech/nonspeech classifier to the surrounding environment based on the estimated SNR value. If the detected signal is nonspeech, the audio classification system will determine the audio type of the detected signal (e.g., footsteps, widows breaking, and door opening sounds); otherwise, the speech signal will be sent to both the Discrete-Valued SNR Estimator and the multiengine SI, consisting of ESI engines to identify the talker. Each SI engine is trained in an environment with a specific noise condition. Using the SNR estimate value from the Discrete-Valued SNR Estimator, the system fuses the outputs of the ESI engines.We are introduced to a new fusion method, Multiengine Weighted Fusion. Multiengine Selection selects theSI engine with training environments that best matches the surrounding environment; the multiengine SI system decision is equal to the decision associated with this single SI engine. The Multiengine Weighted Fusion system intelligently weights and combines the output decisions of the ESI engines; the higher the similarity of the testing environment to the engine’s training environment, the higher the weighting that engine receives.The resultant talker identity (or audio type for nonspeech audio), along with the location information, is sent to the data processing system for application specific tasks. For example, these data can be utilized in audio-video hands-free conferencing, where a video camera can be appropriately steered to talkers and not steered toward undesired audio sources.


### 2.2. Speech/Nonspeech Classification

In [9, 14], pitch ratio (PR) was used to perform speech/nonspeech classification. The PR algorithm calculates the presence of human pitch (70 Hz < human pitch < 280 Hz) in multiple small frames in certain audio segment duration as shown in Figure 3. The PR is calculated as follows:(1)PitchRatio(PR)=NPNF,
where(2)NF=⌊(SD-FS)/(1-OL)FS⌋+1,
where ⌊•⌋ is floor operator, NP is the number of frames that have human pitch, and NF is the total number of frames. This PR value is then compared to a certain threshold. The speech/nonspeech decision is made as follows:(3)Decision={Speech,if  PR>Threshold,Nonspeech,if  PR<Threshold.Figure 3
PR parameters.In [9], a threshold was empirically determined as a function of frame size (FS), segment duration (SD), and overlap (OL). Noise was shown to deteriorate the pitch period of sound. This phenomenon reduces the capability of the pitch detection algorithm to detect the presence of the pitch. To remedy this, the threshold of the PR algorithm is adapted to maximize the accuracy of the speech/nonspeech classifier. We have found that the optimal threshold (Figure 4(a)), the threshold corresponding to the maximum speech/nonspeech classification accuracy (Figure 4(b)), reduces as the SNR reduces. This adaptive speech/nonspeech algorithm was termed the adaptive pitch ratio (APR) algorithm [9]. Adaption is performed based on the SNR estimates from the Discrete-Valued SNR Estimator.Relation between thePR threshold and SNR (sampling rate = 8 KHz, FS=60 ms, SD=450 ms, and OL=0.85).
(a)(b)If the decision of the APR is nonspeech, further audio classification is performed to determine the audio type. Nonspeech audio classification will not be discussed here, as it is not the focus of this paper; however, readers can refer to [9] for more details. If the decision of the APR is speech, then an SI algorithm is used to identify the talker.


### 2.3. Discrete-ValuedSNR Estimator

AnSNR estimator that is computationally efficient with low complexity is desirable for this system, as it enables the selection of the SI engine in real time. Output of this estimator should be a discrete value that matches one of the training environments’ SNRs. This estimator provides information about the noise in the surrounding environment by extracting features from the captured audio sounds emanating within this environment.In this paper, the Discrete-ValuedSNR Estimator is implemented using covariance matrix modeling [1, 15, 16]. The feature vector of the noisy environment is the mel-frequency spectrum coefficients (MFSCs) and the delta MFSC. The MFSCs are computed in a similar fashion as mel-frequency cepstrum coefficients (MFCCs) using a filter bank output but without the use of the discrete cosine transform (DCT) [17]. The DCT of the MFSC vector decorrelates the feature vectors’ elements, which leads to a feature vector with low sensitivity to environment change in the audio signal and high sensitivity to the talker (i.e., dependent on the talker) [18]. In our case we desire a feature vector with high sensitivity to the noise but independent of the talker (i.e., provides information about the environments without being affected by who uttered the speech). Thus, the DCT stage has been removed from the feature vector generation process.To implement the Discrete-ValuedSNR Estimator, let Xe represent the covariance matrix of the M training feature vectors for environment e{x⃗j,1≤j≤M} and let Y represent the covariance matrix of the N test feature vectors {y⃗i,1<i<N} from an unknown environment. Feature vectors for both training and testing covariance matrices, have a length L; therefore, Xe and Y are matrices of size L×L and are computed as follows:(4)Xe=1M∑i=1M(x⃗i-x¯)T(x⃗i-x¯),Y=1N∑j=1N(y⃗j-y¯)T(y⃗j-y¯),
where vector x̅ is the mean of the M training vectors x⃗i and vector y¯ is the mean of the N test vectors y⃗j. Using the following property:(5)1N∑j=1N(y⃗j-y¯)TXe-1(y⃗j-y¯)=tr⁡(YXe-1).The distance between training environmente and test environment can be calculated using the arithmetic harmonic sphericity measure as follows [1, 16, 18]:(6)d(e)=log⁡[tr⁡(YXe-1)tr⁡(XeY-1)]-2log⁡(L),1≤e≤E,
where L is the feature vector length. The distance d(e) represents the similarity between training environment e and the test environment; the smaller the distance, the more the similarity between the training and test environment. Then the distance vector is generated as follows:(7)D=[d(1)d(2)⋯d(E)]T,d(e)≥0.The proposed instrument only requires a rough estimation of theSNR value to set the threshold of the APR and to calculate the distance between the surrounding environment and the training environments to fuse the output of the multiengines. Other SNR estimators, such as SNR estimation using high-order statistics [19], have higher resolution (i.e., estimates SNR as continuous scale); meanwhile, they have higher complexity. The increased precision is not needed in the proposed instrument implementation, so the increased complexity can be avoided.


### 2.4. Speaker Identification Engines

Gaussian mixed models (GMM) are used to implement the individualSI engines. GMMs are used commonly for speaker identification [20, 21]. During training, the parameters of each talker GMM is determined from a sequence of M training vectors. First, the vectors are grouped into clusters using Linde et al.’s (LBG) clustering algorithm [22]. Second, the mean and variance of each cluster are then computed. During testing, the probability that a given talker uttered a certain test speech is computed from the GMMs. Using extracted N feature vectors from the test utterance, the SI engine computes a log-likelihood probability vector as follows:(8)L=[l(1)l(2)⋯l(S)]T,
where S is the number of elements in this vector which is equal to the numbers of the talkers. The SI engine selects the talker that uttered the test speech as the talker with the highest log-likelihood value. In this work, a 32-GMM is used as a classifier, with a feature vector of 16 MFCCs.The accuracy of theSI engine can be degraded by a mismatch between the training and test speech. In the context of this work, this mismatch occurs when different levels of noise are present in the training and testing sets. In the proposed multiengine SI system, a set of ESI engines are used, each trained with different levels of noise. For the multiengine SI, E log-likelihood probability vectors are produced, one vector from each engine. Using the SNR estimate from the Discrete-Valued SNR Estimator multiengine fusion is performed.


### 2.5. Multiengine Selection

Each environmente in a set of E training environments is modeled by a single covariance matrix Xe. A test noisy utterance from an unknown environment is modeled by a covariance matrix Y. The distance d(e) is computed for each training environment. The SI engine associated with the minimum distance is selected, and the talker identity determined by this engine is used as the SI system output (Figure 5). The engine selection is calculated as follows:(9)ê=arg⁡min⁡D,1≤e≤E.Figure 5
Multiengine speaker identification using the Engine Selection method.The estimatedSNR value is the SNR of the selected training environment {SNR=SNRê,1≤ê≤E}.


### 2.6. Multiengine Weighted Fusion

Performance ofSI engines varies in different environments. This performance variation depends on the similarity between the test and the training environment. Using this fact, SI system can utilize the output decision of each engine and combine the decisions of these engines (Figure 6).Figure 6
Multiengine speaker identification using the Weighted Fusion method.Assume that we haveS talkers ti(i∈(Ω={1,2,…,S})) and ESI engines trained in E environments. The conditional probability for each talker ti generated from engine e is denoted as p(ti∣enve) and is computed using the following steps. Each SI engine that is trained in environment e generates a maximum log-likelihood vector Le as follows:(10)Le=[le(1)le(2)⋯le(S)]T,
where le(i) is likelihood of the ith talker from the ethSI engine. These likelihoods are normalized as(11)p(tienve)=le(i)-min⁡j⁡[le(j)]∑k(le(k)-min⁡j⁡[le(j)]).This normalization ensures that the values ofp(ti∣enve) have properties consistent with probabilities; that is,(12)∑i=1Sp(ti∣enve)=1,p(ti∣enve)≥0.
The probability that the surrounding environment is environment e is denoted as p(enve) and is computed as follows. First, the distance between the surrounding environment and the training environment is measured using the Discrete-Valued SNR Estimator. Then the distance vector is generated as in (7). The training environment e with the smallest distance d(e) value has the highest similarity with the surrounding environment. The distances are normalized to the probability of the surrounding environments as follows:(13)p(enve)=d(e)-max⁡j⁡[d(j)]∑k(d(k)-max⁡j⁡[d(j)]).
This normalization ensures the values of p(enve) have properties consistent with probabilities; that is:(14)∑e=1Ep(enve)=1,p(enve)≥0.
The probability of talker ti is computed for the SI system as follows:(15)p(ti)=∑e=1Ep(ti∣enve)p(enve).
The decision of the Multiengine Weighted Fusion engines is the talker with the maximum probability. The Multiengine Weighted Fusion method generalizes the solution for the speaker identification systems in a variable environment.


## 2.1. Instrument Overview

A block diagram of the overall instrument is shown in Figure2. The microphone array captures the audio signal then sends the digitized audio signal to the localization block to determine the location of the sound source [12, 13] in addition to steering the microphone array to reduce undesired acoustic noise.Figure 2
Block diagram of proposed system.A speech/nonspeech classifier is used as a frontend to appropriately select the next stage in audio processing, which is either theSI system or the nonspeech audio classification system [9]. The speech/nonspeech classifier is also adapted to the environment, so that it is robust to varying levels of noise. In this paper, we utilize the Discrete-Valued SNR Estimator (Section 2.4) to adapt the speech/nonspeech classifier to the surrounding environment based on the estimated SNR value. If the detected signal is nonspeech, the audio classification system will determine the audio type of the detected signal (e.g., footsteps, widows breaking, and door opening sounds); otherwise, the speech signal will be sent to both the Discrete-Valued SNR Estimator and the multiengine SI, consisting of ESI engines to identify the talker. Each SI engine is trained in an environment with a specific noise condition. Using the SNR estimate value from the Discrete-Valued SNR Estimator, the system fuses the outputs of the ESI engines.We are introduced to a new fusion method, Multiengine Weighted Fusion. Multiengine Selection selects theSI engine with training environments that best matches the surrounding environment; the multiengine SI system decision is equal to the decision associated with this single SI engine. The Multiengine Weighted Fusion system intelligently weights and combines the output decisions of the ESI engines; the higher the similarity of the testing environment to the engine’s training environment, the higher the weighting that engine receives.The resultant talker identity (or audio type for nonspeech audio), along with the location information, is sent to the data processing system for application specific tasks. For example, these data can be utilized in audio-video hands-free conferencing, where a video camera can be appropriately steered to talkers and not steered toward undesired audio sources.


## 2.2. Speech/Nonspeech Classification

In [9, 14], pitch ratio (PR) was used to perform speech/nonspeech classification. The PR algorithm calculates the presence of human pitch (70 Hz < human pitch < 280 Hz) in multiple small frames in certain audio segment duration as shown in Figure 3. The PR is calculated as follows:(1)PitchRatio(PR)=NPNF,
where(2)NF=⌊(SD-FS)/(1-OL)FS⌋+1,
where ⌊•⌋ is floor operator, NP is the number of frames that have human pitch, and NF is the total number of frames. This PR value is then compared to a certain threshold. The speech/nonspeech decision is made as follows:(3)Decision={Speech,if  PR>Threshold,Nonspeech,if  PR<Threshold.Figure 3
PR parameters.In [9], a threshold was empirically determined as a function of frame size (FS), segment duration (SD), and overlap (OL). Noise was shown to deteriorate the pitch period of sound. This phenomenon reduces the capability of the pitch detection algorithm to detect the presence of the pitch. To remedy this, the threshold of the PR algorithm is adapted to maximize the accuracy of the speech/nonspeech classifier. We have found that the optimal threshold (Figure 4(a)), the threshold corresponding to the maximum speech/nonspeech classification accuracy (Figure 4(b)), reduces as the SNR reduces. This adaptive speech/nonspeech algorithm was termed the adaptive pitch ratio (APR) algorithm [9]. Adaption is performed based on the SNR estimates from the Discrete-Valued SNR Estimator.Relation between thePR threshold and SNR (sampling rate = 8 KHz, FS=60 ms, SD=450 ms, and OL=0.85).
(a)(b)If the decision of the APR is nonspeech, further audio classification is performed to determine the audio type. Nonspeech audio classification will not be discussed here, as it is not the focus of this paper; however, readers can refer to [9] for more details. If the decision of the APR is speech, then an SI algorithm is used to identify the talker.


## 2.3. Discrete-ValuedSNR Estimator

AnSNR estimator that is computationally efficient with low complexity is desirable for this system, as it enables the selection of the SI engine in real time. Output of this estimator should be a discrete value that matches one of the training environments’ SNRs. This estimator provides information about the noise in the surrounding environment by extracting features from the captured audio sounds emanating within this environment.In this paper, the Discrete-ValuedSNR Estimator is implemented using covariance matrix modeling [1, 15, 16]. The feature vector of the noisy environment is the mel-frequency spectrum coefficients (MFSCs) and the delta MFSC. The MFSCs are computed in a similar fashion as mel-frequency cepstrum coefficients (MFCCs) using a filter bank output but without the use of the discrete cosine transform (DCT) [17]. The DCT of the MFSC vector decorrelates the feature vectors’ elements, which leads to a feature vector with low sensitivity to environment change in the audio signal and high sensitivity to the talker (i.e., dependent on the talker) [18]. In our case we desire a feature vector with high sensitivity to the noise but independent of the talker (i.e., provides information about the environments without being affected by who uttered the speech). Thus, the DCT stage has been removed from the feature vector generation process.To implement the Discrete-ValuedSNR Estimator, let Xe represent the covariance matrix of the M training feature vectors for environment e{x⃗j,1≤j≤M} and let Y represent the covariance matrix of the N test feature vectors {y⃗i,1<i<N} from an unknown environment. Feature vectors for both training and testing covariance matrices, have a length L; therefore, Xe and Y are matrices of size L×L and are computed as follows:(4)Xe=1M∑i=1M(x⃗i-x¯)T(x⃗i-x¯),Y=1N∑j=1N(y⃗j-y¯)T(y⃗j-y¯),
where vector x̅ is the mean of the M training vectors x⃗i and vector y¯ is the mean of the N test vectors y⃗j. Using the following property:(5)1N∑j=1N(y⃗j-y¯)TXe-1(y⃗j-y¯)=tr⁡(YXe-1).The distance between training environmente and test environment can be calculated using the arithmetic harmonic sphericity measure as follows [1, 16, 18]:(6)d(e)=log⁡[tr⁡(YXe-1)tr⁡(XeY-1)]-2log⁡(L),1≤e≤E,
where L is the feature vector length. The distance d(e) represents the similarity between training environment e and the test environment; the smaller the distance, the more the similarity between the training and test environment. Then the distance vector is generated as follows:(7)D=[d(1)d(2)⋯d(E)]T,d(e)≥0.The proposed instrument only requires a rough estimation of theSNR value to set the threshold of the APR and to calculate the distance between the surrounding environment and the training environments to fuse the output of the multiengines. Other SNR estimators, such as SNR estimation using high-order statistics [19], have higher resolution (i.e., estimates SNR as continuous scale); meanwhile, they have higher complexity. The increased precision is not needed in the proposed instrument implementation, so the increased complexity can be avoided.


## 2.4. Speaker Identification Engines

Gaussian mixed models (GMM) are used to implement the individualSI engines. GMMs are used commonly for speaker identification [20, 21]. During training, the parameters of each talker GMM is determined from a sequence of M training vectors. First, the vectors are grouped into clusters using Linde et al.’s (LBG) clustering algorithm [22]. Second, the mean and variance of each cluster are then computed. During testing, the probability that a given talker uttered a certain test speech is computed from the GMMs. Using extracted N feature vectors from the test utterance, the SI engine computes a log-likelihood probability vector as follows:(8)L=[l(1)l(2)⋯l(S)]T,
where S is the number of elements in this vector which is equal to the numbers of the talkers. The SI engine selects the talker that uttered the test speech as the talker with the highest log-likelihood value. In this work, a 32-GMM is used as a classifier, with a feature vector of 16 MFCCs.The accuracy of theSI engine can be degraded by a mismatch between the training and test speech. In the context of this work, this mismatch occurs when different levels of noise are present in the training and testing sets. In the proposed multiengine SI system, a set of ESI engines are used, each trained with different levels of noise. For the multiengine SI, E log-likelihood probability vectors are produced, one vector from each engine. Using the SNR estimate from the Discrete-Valued SNR Estimator multiengine fusion is performed.


## 2.5. Multiengine Selection

Each environmente in a set of E training environments is modeled by a single covariance matrix Xe. A test noisy utterance from an unknown environment is modeled by a covariance matrix Y. The distance d(e) is computed for each training environment. The SI engine associated with the minimum distance is selected, and the talker identity determined by this engine is used as the SI system output (Figure 5). The engine selection is calculated as follows:(9)ê=arg⁡min⁡D,1≤e≤E.Figure 5
Multiengine speaker identification using the Engine Selection method.The estimatedSNR value is the SNR of the selected training environment {SNR=SNRê,1≤ê≤E}.


## 2.6. Multiengine Weighted Fusion

Performance ofSI engines varies in different environments. This performance variation depends on the similarity between the test and the training environment. Using this fact, SI system can utilize the output decision of each engine and combine the decisions of these engines (Figure 6).Figure 6
Multiengine speaker identification using the Weighted Fusion method.Assume that we haveS talkers ti(i∈(Ω={1,2,…,S})) and ESI engines trained in E environments. The conditional probability for each talker ti generated from engine e is denoted as p(ti∣enve) and is computed using the following steps. Each SI engine that is trained in environment e generates a maximum log-likelihood vector Le as follows:(10)Le=[le(1)le(2)⋯le(S)]T,
where le(i) is likelihood of the ith talker from the ethSI engine. These likelihoods are normalized as(11)p(tienve)=le(i)-min⁡j⁡[le(j)]∑k(le(k)-min⁡j⁡[le(j)]).This normalization ensures that the values ofp(ti∣enve) have properties consistent with probabilities; that is,(12)∑i=1Sp(ti∣enve)=1,p(ti∣enve)≥0.
The probability that the surrounding environment is environment e is denoted as p(enve) and is computed as follows. First, the distance between the surrounding environment and the training environment is measured using the Discrete-Valued SNR Estimator. Then the distance vector is generated as in (7). The training environment e with the smallest distance d(e) value has the highest similarity with the surrounding environment. The distances are normalized to the probability of the surrounding environments as follows:(13)p(enve)=d(e)-max⁡j⁡[d(j)]∑k(d(k)-max⁡j⁡[d(j)]).
This normalization ensures the values of p(enve) have properties consistent with probabilities; that is:(14)∑e=1Ep(enve)=1,p(enve)≥0.
The probability of talker ti is computed for the SI system as follows:(15)p(ti)=∑e=1Ep(ti∣enve)p(enve).
The decision of the Multiengine Weighted Fusion engines is the talker with the maximum probability. The Multiengine Weighted Fusion method generalizes the solution for the speaker identification systems in a variable environment.


## 3. Experiment and Results

### 3.1. Audio Database

A speech/nonspeech audio database containing a total of 1356 audio segments was used to train and test the speech/nonspeech classifier. Audio segments were 0.45 seconds in length, with a sampling rate of 8 kHz. The speech/nonspeech audio database contains nonspeech audio (e.g., windows breaking, wind, fan, and footsteps sounds) and speech audio. A full list of the audio types can be found in Table1. Half of each audio type in the speech/nonspeech audio database was used in training the speech/nonspeech classifier, and the remaining half was used as a test set.Table 1
Speech/nonspeech audio database.
Audio classNumber of Audio filesSpeechMale282Female282NonspeechChair20Cocktail noise20Door61Drawer20Fan noise169Footsteps41Glass breaking145Rain50Thunder40Traffic20Water20Wind186Total1356The KING speech database is used in training and testing theSI system. The speech/nonspeech portion of the system was trained using the previously mention speech/nonspeech audio database. The KING database consists of speech from 51 speakers; each speaker has 10 utterances with the exception of 2 speakers, which only have 5 utterances [23]. Data are sampled at a rate of 8 kHz, with 16-bit resolution, and utterance durations ranging from 30 to 60 seconds, which were all shortened to 15 seconds. The first 3 utterances of each speaker was used to form a training set (153 utterances), and the remaining utterances were used to form a test set (347 utterances). For the speech/nonspeech and KING databases, the training sets and test sets are mutually exclusive.A total of 6 levels ofSNR (−10, 0, 10, 20, and 30 dB and a clean environment) were used for training, using additive white Gaussian noise to adjust the SNR. A total of 11 levels of SNR (−10, −5, 0, 5, 10, 15, 20, 25, 30, and 35 dB and a clean environment) were used for testing; 5 levels of SNR were different from the training SNR levels. Additive white Gaussian noise and pink noise were used to create the different levels of noise; therefore, while the SNR level may be the same in the training and test set, the type of noise may differ.


### 3.2. Speech/Nonspeech Classification Results

The speech/nonspeech classifier was tested over anSNR range of −10 dB to clean environment, using white and pink noise. The average performance using the pink and white noise of APR algorithm is shown in Table 2. The average classification performance with pink noise is similar to the performance with white noise. Also, the classification performance of KING database is similar to the speech in the speech/nonspeech audio database. Due to the deterioration of the pitch period, APR has a poor speech classification performance of 32.5% and 28.2% for white noise and pink noise at low SNR=-10 dB, respectively. APR maintains an overall accuracy above 93.0% for SNR above 0 dB.Table 2
Speech/nonspeech classification results (%).
Audio classSNR (dB)−100102030CleanWhite noiseNonspeech84.095.298.198.398.698.6Speech32.093.296.897.397.397.3KING33.093.798.098.698.698.6Average58.294.397.798.198.398.3Pink noiseNonspeech87.094.097.597.898.498.6Speech27.190.096.095.995.996.7KING29.289.197.598.297.497.5Average57.591.797.197.897.997.9Speech/nonspeech average57.893.097.49898.198.1


### 3.3. Discrete-ValuedSNR Estimator Results

The Discrete-ValuedSNR Estimator was trained using one randomly selected talker from the KING database. Training involved 3 utterances in the training set, associated with this talker, with the 6 training SNR levels. Discrete-Valued SNR Estimator training and testing are conducted using a feature vector of 19 MFSCs and delta MFSCs. The minimum distance is used to classify test cases to one of the training environments as described in Section 2.4. Performance of the Discrete-Valued SNR Estimator is evaluated on the KING database test set, using the 11 testing SNR levels with white and pink noise.Table3 summarizes the results of the Discrete-Valued SNR Estimator classification. The total number of utterances for jthSNR value in the test set is equal to Uj, where Uj = 347 for j=1 (clean environment) and Uj=2×347=694 for  j≠1 (noisy environments). The elements of Table 3Qji are the percentage of the total number of utterances Uj that were classified to the ith training SNR value.Table 3
Discrete-valued noise estimator classification results (%).
Training env.Test env.Clean3020100−10Clean100.00.00.00.00.00.03567.132.90.00.00.00.03034.064.51.50.00.00.02510.170.019.90.00.00.0200.019.5782.50.00.0150.07.262.130.70.00.0100.00.018.678.52.90.050.00.00.029.870.20.000.00.00.00.097.12.9−50.00.00.00.02.997.1−100.00.00.00.00.0100The Discrete-ValuedSNR Estimator output decision is assumed to be correct if the SNR of the selected training environment has the closest value to the SNR of the surrounding test environment or one of the adjacent environments in terms of SNR (these are set in bold in Table 3). For example if the surrounding environment has an SNR=15 dB, then the output decision is assumed to be correct if the selected SNR=10 or 20 dB (accuracy=30.7%+62.1%=92.8%).TheSNR classification is not expected to require high precision, which justifies the inclusion of the adjacent SNR values in the definition of a correct decision. Overall, the average accuracy of the Discrete-Valued SNR Estimator is 98.4% overall accuracy.


### 3.4. Multiengine Selection Results

SixSI engines were trained using the SNR values of the training environments. Figure 7 plots the results of the 6 individual SI engines with two different noise types (white and pink noise) using the KING database. The maximum performance of a single SI occurs when the test data from a particular noisy environment is used with the SI engine trained in that environment. Also, the performance of SI engines trained in similar noise environments is better than those trained in environments that greatly differ.Figure 7
Speaker identification accuracy using the proposed multiengine methods.The estimate of theSNR from the Discrete-Valued SNR Estimator is utilized to select the best performing SI engine for Multiengine Selection method. This method has average classification accuracy of 88.7% in noisy environments over an SNR range from 10 dB to clean environment. Compared to the accuracy using the SI engine trained in a clean environment, the improvement in accuracy was as high as 30.6%. Also the Multiengine Selection achieved an improvement in accuracy as high as 17% over an SNR range of −10 to 10 dB. Due to inaccuracies in the Discrete-Valued SNR estimator, the Multiengine Selection accuracy is slightly lower than the best individual SI engine at each SNR level. The Multiengine Selection maintains an accuracy above 75% for SNR above 10 dB, whereas the clean SI engine only does so for SNR above 30 dB.


### 3.5. Multiengine Weighted Fusion Results

Figure7 plots the Multiengine Weighted Fusion SI engines’ performance using the KING database. The estimate of the SNR from the Discrete-Valued SNR Estimator is utilized to weight and combine the SI decision of the 6 SI engines. Figure 7 shows that this Multiengine Weighted Fusion method has average classification accuracy of 91.3% in noisy environments over an SNR range of 10 dB to clean environment. The identification accuracy of the Multiengine Weighted Fusion method outperforms Multiengine Selection method (88.7%) and the identification accuracy using the SI engine trained in a clean environment (58.1%).ForSNR5≥ dB, the classification accuracy of the Multiengine Weighted Fusion SI engines is superior to all individual SI engines. For example, at SNR=15 dB, the engines trained at SNR=10 and 20 dB have an accuracy of 68% and 87%, respectively, the Multiengine Selection classification accuracy has an accuracy of 81%, and the Multiengine Weighted Fusion has an accuracy of 89%. Naturally, the Multiengine Selection method cannot perform better than all individual SI engines and as previously mentioned is typically worse due to inaccuracies in the Discrete-Valued SNR estimator.The Multiengine Weighted Fusion method has higher complexity than the Multiengine Selection method, albeit only slightly higher complexity associated with (11), (13), and (15). Meanwhile, the Multiengine Weighted Fusion method has higher classification accuracy.For variable environments with a knownSNR set, the SI training environments can be matched with the testing environments. In cases, where the training and testing environments are the same (SNR=30, 20, 10, 0, and −10 dB and clean environment), the Multiengine Selection method has almost similar accuracy as the Multiengine Weighted Fusion method. This makes the Multiengine Selection method perhaps a better solution because of its lower complexity; however, when the testing environments are different from the training environments (SNR=35,25,15,5,-5 dB), the classification performance enhancement of the Multiengine Weighted Fusion is 7.2% compared to the Multiengine Selection. This suggests that the Multiengine Weight Fusion is a more generalizable solution.For high-SNR environments (SNR≥30 dB), it seems there is no benefit for a multiengine system. Indeed, the engine trained at SNR=30 dB has similar accuracy as the multiengine systems (Figure 7); however, it is clearly not robust as there is severe degradation in performance for SNR<30 dB.


### 3.6. Speech/Nonspeech Classification with MultiengineSI

The overall adaptive system accuracy (withSNR estimator and APR) is compared with a nonadaptive system (cleanSIengineandPRthreshold=0.32) (Figure 8) using the KING database. The main contributor of the accuracy enhancement of the introduced instrument at SNR above 10 dB is the Multiengine SI system (from Table 2 the performance of the APR is nearly perfect for SNR above 10 dB). The performance curve of the Multiengine Selection, shown in Figure 8, has a large dip at SNR=5 dB. For SNR below 10 dB the performance curve could be smoothed by decreasing the SNR spacing between SI engines.Figure 8
Overall accuracy of the introduced system.The classification accuracy performance using a perfectSNR estimator with the APR and Multiengine Selection is shown in Figure 8. Using this SNR value, the best-performing SI engine and the APR threshold are selected. Compared to APR with Multiengine Weighted Fusion using the Discrete-Valued SNR Estimator, the improvement in accuracy is only 1.6% over SNR range clean to −10 dB. This result justifies the utility of the rough estimation of the SNR value.


## 3.1. Audio Database

A speech/nonspeech audio database containing a total of 1356 audio segments was used to train and test the speech/nonspeech classifier. Audio segments were 0.45 seconds in length, with a sampling rate of 8 kHz. The speech/nonspeech audio database contains nonspeech audio (e.g., windows breaking, wind, fan, and footsteps sounds) and speech audio. A full list of the audio types can be found in Table1. Half of each audio type in the speech/nonspeech audio database was used in training the speech/nonspeech classifier, and the remaining half was used as a test set.Table 1
Speech/nonspeech audio database.
Audio classNumber of Audio filesSpeechMale282Female282NonspeechChair20Cocktail noise20Door61Drawer20Fan noise169Footsteps41Glass breaking145Rain50Thunder40Traffic20Water20Wind186Total1356The KING speech database is used in training and testing theSI system. The speech/nonspeech portion of the system was trained using the previously mention speech/nonspeech audio database. The KING database consists of speech from 51 speakers; each speaker has 10 utterances with the exception of 2 speakers, which only have 5 utterances [23]. Data are sampled at a rate of 8 kHz, with 16-bit resolution, and utterance durations ranging from 30 to 60 seconds, which were all shortened to 15 seconds. The first 3 utterances of each speaker was used to form a training set (153 utterances), and the remaining utterances were used to form a test set (347 utterances). For the speech/nonspeech and KING databases, the training sets and test sets are mutually exclusive.A total of 6 levels ofSNR (−10, 0, 10, 20, and 30 dB and a clean environment) were used for training, using additive white Gaussian noise to adjust the SNR. A total of 11 levels of SNR (−10, −5, 0, 5, 10, 15, 20, 25, 30, and 35 dB and a clean environment) were used for testing; 5 levels of SNR were different from the training SNR levels. Additive white Gaussian noise and pink noise were used to create the different levels of noise; therefore, while the SNR level may be the same in the training and test set, the type of noise may differ.


## 3.2. Speech/Nonspeech Classification Results

The speech/nonspeech classifier was tested over anSNR range of −10 dB to clean environment, using white and pink noise. The average performance using the pink and white noise of APR algorithm is shown in Table 2. The average classification performance with pink noise is similar to the performance with white noise. Also, the classification performance of KING database is similar to the speech in the speech/nonspeech audio database. Due to the deterioration of the pitch period, APR has a poor speech classification performance of 32.5% and 28.2% for white noise and pink noise at low SNR=-10 dB, respectively. APR maintains an overall accuracy above 93.0% for SNR above 0 dB.Table 2
Speech/nonspeech classification results (%).
Audio classSNR (dB)−100102030CleanWhite noiseNonspeech84.095.298.198.398.698.6Speech32.093.296.897.397.397.3KING33.093.798.098.698.698.6Average58.294.397.798.198.398.3Pink noiseNonspeech87.094.097.597.898.498.6Speech27.190.096.095.995.996.7KING29.289.197.598.297.497.5Average57.591.797.197.897.997.9Speech/nonspeech average57.893.097.49898.198.1


## 3.3. Discrete-ValuedSNR Estimator Results

The Discrete-ValuedSNR Estimator was trained using one randomly selected talker from the KING database. Training involved 3 utterances in the training set, associated with this talker, with the 6 training SNR levels. Discrete-Valued SNR Estimator training and testing are conducted using a feature vector of 19 MFSCs and delta MFSCs. The minimum distance is used to classify test cases to one of the training environments as described in Section 2.4. Performance of the Discrete-Valued SNR Estimator is evaluated on the KING database test set, using the 11 testing SNR levels with white and pink noise.Table3 summarizes the results of the Discrete-Valued SNR Estimator classification. The total number of utterances for jthSNR value in the test set is equal to Uj, where Uj = 347 for j=1 (clean environment) and Uj=2×347=694 for  j≠1 (noisy environments). The elements of Table 3Qji are the percentage of the total number of utterances Uj that were classified to the ith training SNR value.Table 3
Discrete-valued noise estimator classification results (%).
Training env.Test env.Clean3020100−10Clean100.00.00.00.00.00.03567.132.90.00.00.00.03034.064.51.50.00.00.02510.170.019.90.00.00.0200.019.5782.50.00.0150.07.262.130.70.00.0100.00.018.678.52.90.050.00.00.029.870.20.000.00.00.00.097.12.9−50.00.00.00.02.997.1−100.00.00.00.00.0100The Discrete-ValuedSNR Estimator output decision is assumed to be correct if the SNR of the selected training environment has the closest value to the SNR of the surrounding test environment or one of the adjacent environments in terms of SNR (these are set in bold in Table 3). For example if the surrounding environment has an SNR=15 dB, then the output decision is assumed to be correct if the selected SNR=10 or 20 dB (accuracy=30.7%+62.1%=92.8%).TheSNR classification is not expected to require high precision, which justifies the inclusion of the adjacent SNR values in the definition of a correct decision. Overall, the average accuracy of the Discrete-Valued SNR Estimator is 98.4% overall accuracy.


## 3.4. Multiengine Selection Results

SixSI engines were trained using the SNR values of the training environments. Figure 7 plots the results of the 6 individual SI engines with two different noise types (white and pink noise) using the KING database. The maximum performance of a single SI occurs when the test data from a particular noisy environment is used with the SI engine trained in that environment. Also, the performance of SI engines trained in similar noise environments is better than those trained in environments that greatly differ.Figure 7
Speaker identification accuracy using the proposed multiengine methods.The estimate of theSNR from the Discrete-Valued SNR Estimator is utilized to select the best performing SI engine for Multiengine Selection method. This method has average classification accuracy of 88.7% in noisy environments over an SNR range from 10 dB to clean environment. Compared to the accuracy using the SI engine trained in a clean environment, the improvement in accuracy was as high as 30.6%. Also the Multiengine Selection achieved an improvement in accuracy as high as 17% over an SNR range of −10 to 10 dB. Due to inaccuracies in the Discrete-Valued SNR estimator, the Multiengine Selection accuracy is slightly lower than the best individual SI engine at each SNR level. The Multiengine Selection maintains an accuracy above 75% for SNR above 10 dB, whereas the clean SI engine only does so for SNR above 30 dB.


## 3.5. Multiengine Weighted Fusion Results

Figure7 plots the Multiengine Weighted Fusion SI engines’ performance using the KING database. The estimate of the SNR from the Discrete-Valued SNR Estimator is utilized to weight and combine the SI decision of the 6 SI engines. Figure 7 shows that this Multiengine Weighted Fusion method has average classification accuracy of 91.3% in noisy environments over an SNR range of 10 dB to clean environment. The identification accuracy of the Multiengine Weighted Fusion method outperforms Multiengine Selection method (88.7%) and the identification accuracy using the SI engine trained in a clean environment (58.1%).ForSNR5≥ dB, the classification accuracy of the Multiengine Weighted Fusion SI engines is superior to all individual SI engines. For example, at SNR=15 dB, the engines trained at SNR=10 and 20 dB have an accuracy of 68% and 87%, respectively, the Multiengine Selection classification accuracy has an accuracy of 81%, and the Multiengine Weighted Fusion has an accuracy of 89%. Naturally, the Multiengine Selection method cannot perform better than all individual SI engines and as previously mentioned is typically worse due to inaccuracies in the Discrete-Valued SNR estimator.The Multiengine Weighted Fusion method has higher complexity than the Multiengine Selection method, albeit only slightly higher complexity associated with (11), (13), and (15). Meanwhile, the Multiengine Weighted Fusion method has higher classification accuracy.For variable environments with a knownSNR set, the SI training environments can be matched with the testing environments. In cases, where the training and testing environments are the same (SNR=30, 20, 10, 0, and −10 dB and clean environment), the Multiengine Selection method has almost similar accuracy as the Multiengine Weighted Fusion method. This makes the Multiengine Selection method perhaps a better solution because of its lower complexity; however, when the testing environments are different from the training environments (SNR=35,25,15,5,-5 dB), the classification performance enhancement of the Multiengine Weighted Fusion is 7.2% compared to the Multiengine Selection. This suggests that the Multiengine Weight Fusion is a more generalizable solution.For high-SNR environments (SNR≥30 dB), it seems there is no benefit for a multiengine system. Indeed, the engine trained at SNR=30 dB has similar accuracy as the multiengine systems (Figure 7); however, it is clearly not robust as there is severe degradation in performance for SNR<30 dB.


## 3.6. Speech/Nonspeech Classification with MultiengineSI

The overall adaptive system accuracy (withSNR estimator and APR) is compared with a nonadaptive system (cleanSIengineandPRthreshold=0.32) (Figure 8) using the KING database. The main contributor of the accuracy enhancement of the introduced instrument at SNR above 10 dB is the Multiengine SI system (from Table 2 the performance of the APR is nearly perfect for SNR above 10 dB). The performance curve of the Multiengine Selection, shown in Figure 8, has a large dip at SNR=5 dB. For SNR below 10 dB the performance curve could be smoothed by decreasing the SNR spacing between SI engines.Figure 8
Overall accuracy of the introduced system.The classification accuracy performance using a perfectSNR estimator with the APR and Multiengine Selection is shown in Figure 8. Using this SNR value, the best-performing SI engine and the APR threshold are selected. Compared to APR with Multiengine Weighted Fusion using the Discrete-Valued SNR Estimator, the improvement in accuracy is only 1.6% over SNR range clean to −10 dB. This result justifies the utility of the rough estimation of the SNR value.


## 4. Conclusion

This paper confirms that the closer the match between the training and testing environment the higher theSI accuracy. This paper proposed a security instrument that can detect location and identity of talker in noisy environment. To achieve this instrument, an adaptive multiengine speaker identification system and Discrete-Valued SNR Estimator are proposed. The Discrete-Valued SNR Estimator is also utilized to adapt the speech/nonspeech classifier. The proposed instrument is reliable and robust even in variable noise environments.In the multiengineSI system, six speaker identification engines were utilized to accommodate five noisy environments in addition to clean environment. The SI system using the Multiengine Selection and Multiengine Weighted Fusion methods achieved average identification accuracies of 88.7% and 91.3%, respectively; this represents an enhancement of 30.6% and 33.2% over the SI engine trained in a clean environment, respectively, for SNR≥10 dB. Also the Multiengine Selection and the Multiengine Weighted Fusion achieved an improvement in accuracy as high as 17% and 18.5% over an SNR range of −10 to 10 dB. While the Multiengine Weighted Fusion has high accuracy and reliability than the Multiengine Selection, the tradeoff is a slightly higher complexity.In future work, this framework of this system (i.e., the combination of the Discrete-ValueSNR Estimator and a Multiengine Fusion method) can be deployed in other applications, such as biomedical applications. An example of a biomedical application area that can benefit from the proposed instrument is hearing aid devices. Hearing aids could utilize the proposed instrument to customize their parameters to achieve an optimal hearing performance in variable environments. Modern hearing aids allow the user to set their internal equalizer or beam former variables to predefined settings by selecting the surrounding environment. The deployment of the proposed instrument in hearing aids can automate the selection of the predefined settings according to the surrounding environment.



---

*Source: 172625-2012-02-08.xml*