# BPN Based Likelihood Ratio Score Fusion for Audio-Visual Speaker Identification in Response to Noise

Authors: Md. Rabiul Islam; Md. Abdus Sobhan
Journal: ISRN Artificial Intelligence
 (2014)

---

## Abstract

This paper deals with a new and improved approach of Back-propagation learning neural network based likelihood ratio score fusion technique for audio-visual speaker Identification in various noisy environments. Different signal preprocessing and noise removing techniques have been used to process the speech utterance and LPC, LPCC, RCC, MFCC, ΔMFCC and ΔΔMFCC methods have been applied to extract the features from the audio signal. Active Shape Model has been used to extract the appearance and shape based facial features. To enhance the performance of the proposed system, appearance and shape based facial features are concatenated and Principal Component Analysis method has been used to reduce the dimension of the facial feature vector. The audio and visual feature vectors are then fed to Hidden Markov Model separately to find out the log-likelihood of each modality. The reliability of each modality has been calculated using reliability measurement method. Finally, these integrated likelihood ratios are fed to Back-propagation learning neural network algorithm to discover the final speaker identification result. For measuring the performance of the proposed system, three different databases, that is, NOIZEUS speech database, ORL face database and VALID audio-visual multimodal database have been used for audio-only, visual-only, and audio-visual speaker identification. To identify the accuracy of the proposed system with existing techniques under various noisy environment, different types of artificial noise have been added at various rates with audio and visual signal and performance being compared with different variations of audio and visual features.


---

## Body

## 1. Introduction

Biometric authentication [1] has grown in popularity as a way to provide personal identification. Person’s identification is crucially significant in many applications and the hike in credit card fraud and identity thefts in recent years indicate that this is an issue of major concern in wider society. Individual passwords, pin identification, or even token based arrangement all have deficiencies that restrict their applicability in a widely networked society. Biometrics is used to identify the identity of an input sample when compared to a template, used in cases to identify specific people by certain characteristics. No single biometrics is expected to effectively satisfy the needs of all identification applications. A number of biometrics have been proposed, researched and evaluated for authentication applications. Each biometrics has its strengths and limitations, and accordingly, each biometrics appeals to a particular identification application [2]. Biometric characteristics can be divided in physiological and behavioral classes [3]. Physiological characteristics are related to the shape of the body and thus it varies from person to person. Fingerprints, face recognition, hand geometry, and iris recognition are some examples of this type of Biometrics. Behavioral characteristics are related to the behavior of a person. Some examples in this case are signature, keystroke dynamics, voice, and so on.The Audio-Visual speaker identification system combines the speech and face biometric characteristics which mix the physiological and behavioral characteristics. There are different levels where the audio and visual features can be concatenated. Preclassification and post-classification are the two broad categories for information fusion in biometric system [4]. In preclassification, multimodal information is  fused before going to the classifier decision. But, in postclassification, information is combined after the decision of multiple classifiers. In this proposed system, feature level fusion has been performed under preclassification approach. Appearance and shape based facial features are combined for visual identification result. Decision level fusion has been applied in the proposed system where audio and visual speaker identification decisions are concatenated to find out the final identification result. However, both feature level and decision level fusion are performed for the proposed system.The rest of the paper is constructed as follows: Section2 describes the literature review and the proposed system architecture, Section 3 focuses on the audio-only identification performance, visual-only identification has been elaborated in Section 4, Audio and visual reliability measurement techniques are focused on Section 5 and back-propagation learning neural network based likelihood ratio score fusion technique has been shown in Section 6. NOIZEUS speech database has been used to measure the performance of the Audio-Only speaker identification, visual-only performance has been populated using ORL database and overall system performance that is, back-propagation learning neural network score fusion based performance has been counted by applying VALID audio-visual multimodal database.


## 2. Literature Review and Proposed System Architecture

Since human speech is bimodal in nature [5, 6], visual speech information can play a vital role for the improvement of natural and robust human-computer interaction [7–11]. Most published works in the areas of speech recognition and speaker recognition focus on speech under the noiseless environments and few published works focus on speech under noisy conditions [12–15]. Indeed, various important human-computer components, such as speaker identification, verification [16, 17], localization [18], speech event detection [19], speech signal separation [20], coding [21], video indexing and retrieval [22] and text-to-speech [23, 24] have been shown to benefit from the visual channel [25]. Adaptive weighting in decision fusion with acoustic and visual features from a given Audio-Visual speech datum, the recognized utterance class has been proposed [26]. The reliability of each audio and visual modality can be measured in various ways such as average absolute difference of loglikelihood [27], variance of loglikelihood [28], average difference of log-likelihood from the maximum [29], and inverse entropy of posterior probability [30]. Decision level information integration techniques have been developed where each biometric matcher individually decides on the best match based on the input presented to it. Methods like majority voting [31], behavior knowledge space [32], weighted voting based on the Dempster-Shafer theory of evidence [33], AND rule and OR rule [34], and so forth are some of the decision level fusion techniques proposed by different researchers.The proposed architecture of the audio-visual speaker identification system is shown in Figure1. Signal preprocessing and noise removing techniques have been applied after acquisition of the speech utterances. Then features are extracted using various standard speech feature extraction methods such as LPC, LPCC, RCC, MFCC, ΔMFCC, and ΔΔMFCC. Principal Component Analysis (PCA) has been used to reduce the dimensionality of the extracted feature vector. Now the reduced feature vector is feed to Discrete Hidden Markov Model (DHMM) to get the log likelihood of each speech modality. Reliability measurement method has been used to measure the reliability for audio signal. For visual identification, captured faces are preprocessed using different noise removing techniques and Active Shape Model (ASM) is used to extract the appearance and shape based features. These two different types of features are fused after applying feature normalization and PCA based dimensionality reduction techniques. The concatenations of these features are important in the sense when the appearance based feature is captured with noise (i.e., light variations) then shape based features can retain the performance on a satisfied level. This is also true when the shape based feature is captured by noise highly. By combining this approach, the proposed system performs very well especially in various lighting environmental conditions. Finally, log likelihood of visual modality has been evaluated using DHMM classification and reliability has been measured using the same reliability measurement technique like audio modality. Integrated weights of audio and visual reliability measurement are fed to the Back-propagation learning neural network algorithm to calculate the final speaker identification result.Figure 1
Paradigm of the proposed BPN based likelihood ratio score fusion for audio-visual speaker identification system.Rogozan and Deléglise [26] developed a technique for combining different likelihoods of multilevel biometric identification. In this proposed system, BPN algorithm has been used to combine the likelihood of audio and visual modality to enhance the performance of audio-visual speaker identification. This is the main contribution of the proposed system. Experimental results show the superiority of BPN based approach over the Rogozan and Deléglise [26] method in terms of audio-visual speaker identification system.


## 3. Audio-Only Speaker Identification

### 3.1. Speech Signal Preprocessing and Feature Extraction

Speech signal preprocessing plays an important role for the efficiency of speaker identification. After capturing the speech utterances, wiener filter has been used to remove the background noise from the original speech utterances [35]. The wiener filter is a noise removing filter based on Fourier iteration. Its main advantage is the short computational time it takes to find a solution [36].Lets(t) be the smear signal let and r(t) be the known response that causes the convolution. Then s(t) is related to u(t) by
(1)s(t)=∫-∞∞r(t-τ)u(τ)dτ
or
(2)S(f)=R(f)U(f),
where S, R, U are Fourier Transform of s, r, and u. Consider the following.The second source of signal corruption is the unknown background noisen(t). Therefore the measured signal c(t) is a sum of s(t) and n(t)(3)c(t)=s(t)+n(t).To deconvolves to find u, simply divide S(f) by R(f) that is, U(f)=S(f)/R(f) in the absence of noise n. To deconvolve c where n is present then one needs to find an optimum filter function ϕ(t) or ϕ(f) which filters out the noise and gives a signal u~ by
(4)U~(f)=C(f)ϕ(f)R(f),
where u~ is as close to the original signal as possible.Foru~ to be similar to u, their differences square is as close to zero as possible; that is,
(5)∫-∞∞|u~(t)-u(t)|2dt
or
(6)∫-∞∞|u~(f)-u(f)|2df
is minimized.Substituting the above three equations, the Fourier version becomes:(7)∫-∞∞|R(f)|-2|S(f)|2|1-ϕ(f)|2+|N(f)|2|ϕ(f)|2df
after rearranging. The best filter is one where the above integral is a minimum at every value of f. This is, when
(8)ϕ(f)=|S(f)|2|S(f)|2+|N(f)|2.
Now, |S(f)|2+|N(f)|2≈|C(f)|2, where |C(f)|2, |S(f)|2, and |N(f)|2 are the power spectrum of C, S, and N. Therefore,
(9)ϕ(f)≈|S(f)|2|C(f)|2.Figure2(a) shows a sample signal with background noise and Figure 2(b) shows the signal after applying the wiener filter.Effects of filtering technique (a) signal with noise (b) signal after applying Wiener filtering technique.
(a)
(b)Speech end points detection and silence part removal algorithm have been used to detect the presence of speech and to remove pulse and silences in the speech utterance [37, 38] which is shown in Figure 3.(a) Speech with silence parts (b) Results after applying silence parts removal algorithm.
(a)
(b)To detect word boundary, the frame energy is computed using the short-term log energy equation [39]
(10)Ei=10log∑t=nini+N-1S2(t).Preemphasis has been used to balance the spectrum of voiced sounds that have a steep roll-off in the high frequency region [38]. The transfer function of the FIR filter in the z-domain is [40]
(11)H(Z)=1-α·z-1,0≤α≤1,
where α is the preemphasis parameter.Frame blocking has been performed with an overlapping of 25% to 75% of the frame size. Typically a frame length of 10–30 milliseconds has been used. The purpose of the overlapping analysis is that each speech sound of the input sequence would be approximately centered at some frame [41].From different types of windowing techniques, a hamming window has been used for this system. The purpose of using windowing is to reduce the effect of the spectral artifacts that results from the framing process [42]. The hamming window can be defined as follows [43]:
(12)w(n)={0.54-0.46cos2πnN,-(N-12)≤n≤(N-12)0Otherwise.To extract the audio features, RCC, LPCC, MFCC, ΔMFCC, and ΔΔMFCC based various standard speech feature extraction techniques [44, 45] have been used to enhance the efficiency of the system because the quality of the system depends on the proper feature extracted values.


### 3.2. Experimental Results according to NOIZEUS Speech Database

NOIZEUS speech corpus [46, 47] has been used to calculate the accuracy of the audio-only speaker identification system which contains 30 IEEE sentences (produced by three male and three female speakers) corrupted by eight different real-world noises at different SNRs. The noise was taken from the AURORA database and includes suburban train noise, babble, car, exhibition hall, restaurant, street, airport and train-station noise. The sentences were originally sampled at 25 kHz and downsampled to 8 kHz [48].To measure the performance of the system according to NOIZEUS speech database, one clean speech utterance has been used for learning and four different noisy speeches ranging from 0 dB to 20 dB with 5 dB interval are used for testing purpose. Tables1, 2, 3, 4, 5, 6, 7, 8, and 9 show the results of audio-only speaker identification rate at different types of noisy environments with different SNRs.Table 1
Airport Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
60.67
63.33
53.67
53.67
56.67
10 dB
53.33
58.67
43.33
43.67
53.33
5 dB
50.00
53.33
30.00
33.33
43.33
0 dB
17.67
17.67
10.67
12.33
15.00
Average
45.42
48.25
34.42
35.75
42.08Table 2
Babble  Noise  Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
62.67
65.33
53.67
56.67
60.00
10 dB
56.33
60.33
43.33
43.67
56.67
5 dB
43.33
53.33
33.33
43.33
50.00
0 dB
17.67
18.00
11.33
13.33
12.00
Average
45.00
49.25
35.42
39.25
44.67Table 3
Car Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
56.67
60.33
43.33
50.00
53.33
10 dB
56.67
56.67
33.33
40.67
53.33
5 dB
46.33
46.67
30.00
40.33
42.33
0 dB
15.33
17.33
10.00
12.33
15.00
Average
43.75
45.25
29.17
35.83
41.00Table 4
Exhibition Hall Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
65.67
67.33
50.00
53.33
60.33
10 dB
60.00
63.33
43.33
46.67
56.33
5 dB
53.67
56.67
33.33
43.33
50.00
0 dB
18.33
18.33
10.33
13.33
16.33
Average
49.42
51.42
34.25
39.17
45.75Table 5
Restaurant Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
60.00
63.33
43.33
53.33
56.67
10 dB
56.67
60.00
40.00
46.67
53.33
5 dB
53.33
56.67
33.33
40.00
46.67
0 dB
17.33
18.67
13.67
15.00
15.67
Average
46.83
49.67
32.58
38.75
43.09Table 6
Street Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
63.33
67.67
53.67
56.67
60.00
10 dB
53.33
60.00
46.67
43.33
50.00
5 dB
50.00
50.00
30.00
43.33
43.67
0 dB
17.33
18.67
11.33
13.67
16.00
Average
46.00
49.09
35.42
39.25
42.42Table 7
Train Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
63.33
67.67
50.00
53.33
58.33
10 dB
60.00
63.33
43.33
46.67
53.33
5 dB
46.67
50.00
30.00
43.33
43.67
0 dB
18.33
18.67
12.33
13.67
15.33
Average
47.08
49.92
33.92
39.25
42.67Table 8
Train Station Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
63.67
63.33
48.33
50.00
58.33
10 dB
60.67
56.67
43.33
46.67
53.33
5 dB
56.00
53.33
33.33
46.67
46.67
0 dB
16.67
18.33
12.67
13.33
18.00
Average
49.25
47.92
34.42
39.17
44.08Table 9
Overall Average Speaker Identification Rate (%) for NOIZEUS Speech Corpus.
Various Noises
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
Airport Noise
45.42
48.25
34.42
35.75
42.08
Babble Noise
45.00
49.25
35.42
39.25
44.67
Car Noise
43.75
45.25
29.17
35.83
41.00
Exhibition Hall Noise
49.42
51.42
34.25
39.17
45.75
Restaurant Noise
46.83
49.67
32.58
38.75
43.09
Street Noise
46.00
49.09
35.42
39.25
42.42
Train Noise
47.08
49.92
33.92
39.25
42.67
Train Station Noise
49.25
47.92
34.42
39.17
44.08
Average Identification Rate (%)
46.59
48.85
33.70
38.30
43.22Table9 shows the overall average speaker identification rate for NOIZEUS speech corpus. From the table, it is easy to compare the performance among MFCC, ΔMFCC, ΔΔMFCC, RCC and LPCC methods for DHMM based audio-only speaker identification system. It is shown that ΔMFCC has greater performance 48.85% than any other methods such as MFCC, ΔΔMFCC, RCC and LPCC. It also shows that ΔMFCC feature can perform better than any other feature extraction method in all of eight different environmental conditions.


## 3.1. Speech Signal Preprocessing and Feature Extraction

Speech signal preprocessing plays an important role for the efficiency of speaker identification. After capturing the speech utterances, wiener filter has been used to remove the background noise from the original speech utterances [35]. The wiener filter is a noise removing filter based on Fourier iteration. Its main advantage is the short computational time it takes to find a solution [36].Lets(t) be the smear signal let and r(t) be the known response that causes the convolution. Then s(t) is related to u(t) by
(1)s(t)=∫-∞∞r(t-τ)u(τ)dτ
or
(2)S(f)=R(f)U(f),
where S, R, U are Fourier Transform of s, r, and u. Consider the following.The second source of signal corruption is the unknown background noisen(t). Therefore the measured signal c(t) is a sum of s(t) and n(t)(3)c(t)=s(t)+n(t).To deconvolves to find u, simply divide S(f) by R(f) that is, U(f)=S(f)/R(f) in the absence of noise n. To deconvolve c where n is present then one needs to find an optimum filter function ϕ(t) or ϕ(f) which filters out the noise and gives a signal u~ by
(4)U~(f)=C(f)ϕ(f)R(f),
where u~ is as close to the original signal as possible.Foru~ to be similar to u, their differences square is as close to zero as possible; that is,
(5)∫-∞∞|u~(t)-u(t)|2dt
or
(6)∫-∞∞|u~(f)-u(f)|2df
is minimized.Substituting the above three equations, the Fourier version becomes:(7)∫-∞∞|R(f)|-2|S(f)|2|1-ϕ(f)|2+|N(f)|2|ϕ(f)|2df
after rearranging. The best filter is one where the above integral is a minimum at every value of f. This is, when
(8)ϕ(f)=|S(f)|2|S(f)|2+|N(f)|2.
Now, |S(f)|2+|N(f)|2≈|C(f)|2, where |C(f)|2, |S(f)|2, and |N(f)|2 are the power spectrum of C, S, and N. Therefore,
(9)ϕ(f)≈|S(f)|2|C(f)|2.Figure2(a) shows a sample signal with background noise and Figure 2(b) shows the signal after applying the wiener filter.Effects of filtering technique (a) signal with noise (b) signal after applying Wiener filtering technique.
(a)
(b)Speech end points detection and silence part removal algorithm have been used to detect the presence of speech and to remove pulse and silences in the speech utterance [37, 38] which is shown in Figure 3.(a) Speech with silence parts (b) Results after applying silence parts removal algorithm.
(a)
(b)To detect word boundary, the frame energy is computed using the short-term log energy equation [39]
(10)Ei=10log∑t=nini+N-1S2(t).Preemphasis has been used to balance the spectrum of voiced sounds that have a steep roll-off in the high frequency region [38]. The transfer function of the FIR filter in the z-domain is [40]
(11)H(Z)=1-α·z-1,0≤α≤1,
where α is the preemphasis parameter.Frame blocking has been performed with an overlapping of 25% to 75% of the frame size. Typically a frame length of 10–30 milliseconds has been used. The purpose of the overlapping analysis is that each speech sound of the input sequence would be approximately centered at some frame [41].From different types of windowing techniques, a hamming window has been used for this system. The purpose of using windowing is to reduce the effect of the spectral artifacts that results from the framing process [42]. The hamming window can be defined as follows [43]:
(12)w(n)={0.54-0.46cos2πnN,-(N-12)≤n≤(N-12)0Otherwise.To extract the audio features, RCC, LPCC, MFCC, ΔMFCC, and ΔΔMFCC based various standard speech feature extraction techniques [44, 45] have been used to enhance the efficiency of the system because the quality of the system depends on the proper feature extracted values.


## 3.2. Experimental Results according to NOIZEUS Speech Database

NOIZEUS speech corpus [46, 47] has been used to calculate the accuracy of the audio-only speaker identification system which contains 30 IEEE sentences (produced by three male and three female speakers) corrupted by eight different real-world noises at different SNRs. The noise was taken from the AURORA database and includes suburban train noise, babble, car, exhibition hall, restaurant, street, airport and train-station noise. The sentences were originally sampled at 25 kHz and downsampled to 8 kHz [48].To measure the performance of the system according to NOIZEUS speech database, one clean speech utterance has been used for learning and four different noisy speeches ranging from 0 dB to 20 dB with 5 dB interval are used for testing purpose. Tables1, 2, 3, 4, 5, 6, 7, 8, and 9 show the results of audio-only speaker identification rate at different types of noisy environments with different SNRs.Table 1
Airport Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
60.67
63.33
53.67
53.67
56.67
10 dB
53.33
58.67
43.33
43.67
53.33
5 dB
50.00
53.33
30.00
33.33
43.33
0 dB
17.67
17.67
10.67
12.33
15.00
Average
45.42
48.25
34.42
35.75
42.08Table 2
Babble  Noise  Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
62.67
65.33
53.67
56.67
60.00
10 dB
56.33
60.33
43.33
43.67
56.67
5 dB
43.33
53.33
33.33
43.33
50.00
0 dB
17.67
18.00
11.33
13.33
12.00
Average
45.00
49.25
35.42
39.25
44.67Table 3
Car Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
56.67
60.33
43.33
50.00
53.33
10 dB
56.67
56.67
33.33
40.67
53.33
5 dB
46.33
46.67
30.00
40.33
42.33
0 dB
15.33
17.33
10.00
12.33
15.00
Average
43.75
45.25
29.17
35.83
41.00Table 4
Exhibition Hall Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
65.67
67.33
50.00
53.33
60.33
10 dB
60.00
63.33
43.33
46.67
56.33
5 dB
53.67
56.67
33.33
43.33
50.00
0 dB
18.33
18.33
10.33
13.33
16.33
Average
49.42
51.42
34.25
39.17
45.75Table 5
Restaurant Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
60.00
63.33
43.33
53.33
56.67
10 dB
56.67
60.00
40.00
46.67
53.33
5 dB
53.33
56.67
33.33
40.00
46.67
0 dB
17.33
18.67
13.67
15.00
15.67
Average
46.83
49.67
32.58
38.75
43.09Table 6
Street Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
63.33
67.67
53.67
56.67
60.00
10 dB
53.33
60.00
46.67
43.33
50.00
5 dB
50.00
50.00
30.00
43.33
43.67
0 dB
17.33
18.67
11.33
13.67
16.00
Average
46.00
49.09
35.42
39.25
42.42Table 7
Train Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
63.33
67.67
50.00
53.33
58.33
10 dB
60.00
63.33
43.33
46.67
53.33
5 dB
46.67
50.00
30.00
43.33
43.67
0 dB
18.33
18.67
12.33
13.67
15.33
Average
47.08
49.92
33.92
39.25
42.67Table 8
Train Station Noise Average Identification Rate (%) for NOIZEUS Speech Corpus.
SNR
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
15 dB
63.67
63.33
48.33
50.00
58.33
10 dB
60.67
56.67
43.33
46.67
53.33
5 dB
56.00
53.33
33.33
46.67
46.67
0 dB
16.67
18.33
12.67
13.33
18.00
Average
49.25
47.92
34.42
39.17
44.08Table 9
Overall Average Speaker Identification Rate (%) for NOIZEUS Speech Corpus.
Various Noises
Method
MFCC
ΔMFCC
ΔΔMFCC
RCC
LPCC
Airport Noise
45.42
48.25
34.42
35.75
42.08
Babble Noise
45.00
49.25
35.42
39.25
44.67
Car Noise
43.75
45.25
29.17
35.83
41.00
Exhibition Hall Noise
49.42
51.42
34.25
39.17
45.75
Restaurant Noise
46.83
49.67
32.58
38.75
43.09
Street Noise
46.00
49.09
35.42
39.25
42.42
Train Noise
47.08
49.92
33.92
39.25
42.67
Train Station Noise
49.25
47.92
34.42
39.17
44.08
Average Identification Rate (%)
46.59
48.85
33.70
38.30
43.22Table9 shows the overall average speaker identification rate for NOIZEUS speech corpus. From the table, it is easy to compare the performance among MFCC, ΔMFCC, ΔΔMFCC, RCC and LPCC methods for DHMM based audio-only speaker identification system. It is shown that ΔMFCC has greater performance 48.85% than any other methods such as MFCC, ΔΔMFCC, RCC and LPCC. It also shows that ΔMFCC feature can perform better than any other feature extraction method in all of eight different environmental conditions.


## 4. Visual-Only Speaker Identification

### 4.1. Facial Feature Extraction and Dimensionality Reduction

After acquisition of a face image, Stams [49] Active Shape Model (ASM) has been used to detect the facial features. Then the binary image has been taken. The Region Of Interest (ROI) has been chosen according to the ROI selection algorithm [50, 51]. Lastly the background noise has been eliminated [52] and finally appearance based facial feature has been found. The procedure of the facial image pre-processing parts is shown in Figure 4 where Figures 4(d) and 4(e) shows the shape based and appearance based facial feature respectively.Figure 4
Facial image preprocessing for the proposed system (a) original image: (b) Output taken from Stams Active Shape Model (c) extracted facial edges (d) shape based features (e) region Of interest (ROI) selection with background noise, and (f) appearance based facial features.To improve the performance of the face recognition system and since we want to compare the proposed technique with the appearance and shape based feature fusion method, we have to combine the appearance and shape based features. The concatenation procedure of two different features is shown in Figure5. Initially raw 5000 dimension appearance based features and 176 dimension shape based features are extracted. The Principal Component Analysis method [53, 54] has been used to reduce the dimension of appearance and shape based features into 192 and 14, respectively. Two different features are added and produced 206 dimension features. Finally, PCA has been used again to resize from 206 dimensional to 130 dimensional appearance-shape based facial feature vector.Figure 5
Process of appearance and shape based feature fusion.


### 4.2. Experimental Results according to ORL Facial Database

Olivetti Research Laboratory (ORL) face database [55] produced by AT&T Laboratories has been used for measuring the performance of the proposed system. The database contains 10 different images of 40 distinct subjects. For some of the subjects, the images were taken at different times, varying lighting slightly, facial expressions (open/closed eyes, smiling/nonsmiling), and facial details (glasses/no-glasses). All the images are taken against a dark homogeneous background and the subjects are in upright, frontal position (with tolerance for some side movement). The size of each face image is 92 × 112 and 8-bit grey levels. Experiment results are evaluated according to various dimensions such as optimum value selection of the number of hidden states of DHMM, response of the system based on noisy facial images and the system accuracy based on appearance, shape and combined appearance and shape based facial features.


#### 4.2.1. System Response for Noisy Facial Images

The facial identification performance has been tested with the variations of different noises. Filtering is used for modifying or enhancing an image. To emphasize certain features or remove other features from an image, different filtering techniques are used. Filtering is a neighbourhood operation in which the value of any given pixel in the output image is determined by applying some algorithm to the values of the pixels in the neighbourhood of the corresponding input pixel. A pixel’s neighbourhood is some set of pixels defined by their locations relative to that pixel. To remove the noise from the facial images, wiener filtering technique has been used. Wiener filtering technique has been used to remove or reduce white Gaussian noise from the facial image. Wiener filter can be used adaptively to an image where the variance is large, wiener filter performs little smoothing and where the variance is small, wiener filter performs more smoothing. Wiener filtering technique performs selective operation compared with other filters, preserving edges and other high-frequency parts of an image.For measuring the accuracy of the face system, noise has been added in various rates for appearance based, shape based and appearance-shape based feature fusion technique with PCA based dimensionality reduction where Euclidian distance has been used as a classifier. Table10 shows the response of applying Wiener filtering technique.Table 10
Results after Applying Wiener Filter.
Noise addition rate (in Variance)
Recognition rate based on PCA
Appearance based feature
Shape based feature
Appearance-shape based feature fusion
0.01
89%
91%
93%
0.05
88%
88%
90%
0.08
82%
85%
87%
0.1
80%
81%
82%
0.4
75%
77%
80%


#### 4.2.2. Performance Measurements between Single and Multiple Feature Fusion Based Techniques

Facial identification performance has been measured according to individual feature based technique such as appearance based feature, shape based feature and appearance-shape based feature fusion based technique. Receiver Operating Characteristics (ROC) curve is generated for the above mentioned techniques where a tradeoff is made between security and user friendness. The performance graph is shown in Figure6. From the graph, it is shown that the appearance-shape based feature fusion can achieve compared with highest accuracy individual appearance based and shape based technique. For example, at a FRR = 30%, the appearance based, shape based and appearance-shape feature fusion FAR are 42%, 30%, and 28% respectively.Figure 6
Performance comparison among individual modality and multimodal feature fusion techniques.


## 4.1. Facial Feature Extraction and Dimensionality Reduction

After acquisition of a face image, Stams [49] Active Shape Model (ASM) has been used to detect the facial features. Then the binary image has been taken. The Region Of Interest (ROI) has been chosen according to the ROI selection algorithm [50, 51]. Lastly the background noise has been eliminated [52] and finally appearance based facial feature has been found. The procedure of the facial image pre-processing parts is shown in Figure 4 where Figures 4(d) and 4(e) shows the shape based and appearance based facial feature respectively.Figure 4
Facial image preprocessing for the proposed system (a) original image: (b) Output taken from Stams Active Shape Model (c) extracted facial edges (d) shape based features (e) region Of interest (ROI) selection with background noise, and (f) appearance based facial features.To improve the performance of the face recognition system and since we want to compare the proposed technique with the appearance and shape based feature fusion method, we have to combine the appearance and shape based features. The concatenation procedure of two different features is shown in Figure5. Initially raw 5000 dimension appearance based features and 176 dimension shape based features are extracted. The Principal Component Analysis method [53, 54] has been used to reduce the dimension of appearance and shape based features into 192 and 14, respectively. Two different features are added and produced 206 dimension features. Finally, PCA has been used again to resize from 206 dimensional to 130 dimensional appearance-shape based facial feature vector.Figure 5
Process of appearance and shape based feature fusion.


## 4.2. Experimental Results according to ORL Facial Database

Olivetti Research Laboratory (ORL) face database [55] produced by AT&T Laboratories has been used for measuring the performance of the proposed system. The database contains 10 different images of 40 distinct subjects. For some of the subjects, the images were taken at different times, varying lighting slightly, facial expressions (open/closed eyes, smiling/nonsmiling), and facial details (glasses/no-glasses). All the images are taken against a dark homogeneous background and the subjects are in upright, frontal position (with tolerance for some side movement). The size of each face image is 92 × 112 and 8-bit grey levels. Experiment results are evaluated according to various dimensions such as optimum value selection of the number of hidden states of DHMM, response of the system based on noisy facial images and the system accuracy based on appearance, shape and combined appearance and shape based facial features.


### 4.2.1. System Response for Noisy Facial Images

The facial identification performance has been tested with the variations of different noises. Filtering is used for modifying or enhancing an image. To emphasize certain features or remove other features from an image, different filtering techniques are used. Filtering is a neighbourhood operation in which the value of any given pixel in the output image is determined by applying some algorithm to the values of the pixels in the neighbourhood of the corresponding input pixel. A pixel’s neighbourhood is some set of pixels defined by their locations relative to that pixel. To remove the noise from the facial images, wiener filtering technique has been used. Wiener filtering technique has been used to remove or reduce white Gaussian noise from the facial image. Wiener filter can be used adaptively to an image where the variance is large, wiener filter performs little smoothing and where the variance is small, wiener filter performs more smoothing. Wiener filtering technique performs selective operation compared with other filters, preserving edges and other high-frequency parts of an image.For measuring the accuracy of the face system, noise has been added in various rates for appearance based, shape based and appearance-shape based feature fusion technique with PCA based dimensionality reduction where Euclidian distance has been used as a classifier. Table10 shows the response of applying Wiener filtering technique.Table 10
Results after Applying Wiener Filter.
Noise addition rate (in Variance)
Recognition rate based on PCA
Appearance based feature
Shape based feature
Appearance-shape based feature fusion
0.01
89%
91%
93%
0.05
88%
88%
90%
0.08
82%
85%
87%
0.1
80%
81%
82%
0.4
75%
77%
80%


### 4.2.2. Performance Measurements between Single and Multiple Feature Fusion Based Techniques

Facial identification performance has been measured according to individual feature based technique such as appearance based feature, shape based feature and appearance-shape based feature fusion based technique. Receiver Operating Characteristics (ROC) curve is generated for the above mentioned techniques where a tradeoff is made between security and user friendness. The performance graph is shown in Figure6. From the graph, it is shown that the appearance-shape based feature fusion can achieve compared with highest accuracy individual appearance based and shape based technique. For example, at a FRR = 30%, the appearance based, shape based and appearance-shape feature fusion FAR are 42%, 30%, and 28% respectively.Figure 6
Performance comparison among individual modality and multimodal feature fusion techniques.


## 4.2.1. System Response for Noisy Facial Images

The facial identification performance has been tested with the variations of different noises. Filtering is used for modifying or enhancing an image. To emphasize certain features or remove other features from an image, different filtering techniques are used. Filtering is a neighbourhood operation in which the value of any given pixel in the output image is determined by applying some algorithm to the values of the pixels in the neighbourhood of the corresponding input pixel. A pixel’s neighbourhood is some set of pixels defined by their locations relative to that pixel. To remove the noise from the facial images, wiener filtering technique has been used. Wiener filtering technique has been used to remove or reduce white Gaussian noise from the facial image. Wiener filter can be used adaptively to an image where the variance is large, wiener filter performs little smoothing and where the variance is small, wiener filter performs more smoothing. Wiener filtering technique performs selective operation compared with other filters, preserving edges and other high-frequency parts of an image.For measuring the accuracy of the face system, noise has been added in various rates for appearance based, shape based and appearance-shape based feature fusion technique with PCA based dimensionality reduction where Euclidian distance has been used as a classifier. Table10 shows the response of applying Wiener filtering technique.Table 10
Results after Applying Wiener Filter.
Noise addition rate (in Variance)
Recognition rate based on PCA
Appearance based feature
Shape based feature
Appearance-shape based feature fusion
0.01
89%
91%
93%
0.05
88%
88%
90%
0.08
82%
85%
87%
0.1
80%
81%
82%
0.4
75%
77%
80%


## 4.2.2. Performance Measurements between Single and Multiple Feature Fusion Based Techniques

Facial identification performance has been measured according to individual feature based technique such as appearance based feature, shape based feature and appearance-shape based feature fusion based technique. Receiver Operating Characteristics (ROC) curve is generated for the above mentioned techniques where a tradeoff is made between security and user friendness. The performance graph is shown in Figure6. From the graph, it is shown that the appearance-shape based feature fusion can achieve compared with highest accuracy individual appearance based and shape based technique. For example, at a FRR = 30%, the appearance based, shape based and appearance-shape feature fusion FAR are 42%, 30%, and 28% respectively.Figure 6
Performance comparison among individual modality and multimodal feature fusion techniques.


## 5. Audio and Visual Reliability Measurements

Since DHMM learning and testing models have been adopted for the audio and visual system, an ergodic discrete HMM (DHMM),θk [56], has been built in DHMM training phase for each face k. The model parameters (A,B, and θ) have been estimated to optimize the likelihood of the training set observation vector for the kth face by using the Baum-Welch algorithm. The Baum-Welch reestimation formula has been considered as follows [57, 58]:
(13)Π-i=γ1(i),a-ij=∑t=1T-1ξt(i,j)∑t=1T-1γt(i),b-j(k→)=∑t=1(∑t=1Tγt(j)s,t,o→t=v→k)Tγt(j)∑t=1Tγt(j).In the DHMM testing phase, for each unknown face to be recognized which includes(i)
measurement of the observation sequence,O={o1,o2,…,on}, via a feature analysis of the corresponding face,(ii)
transformation of the continuous values ofO into integer values,(iii)
calculation of model likelihood for all possible models,P(O∣θk), 1≤k≤K,(iv)
declaration of the face ask* person whose model likelihood is highest—that is,
(14)k*=argmax1≤k≤K[P(O∣θk)].In this work, the probability computation step has been performed using Baum’s Forward-Backward algorithm [58, 59]. By applying HMM as a learning phase, the log likelihood of each appearance and shape based feature of each person face have been captured. After getting the log likelihood of each modality separately, their outputs are combined by a weighted sum rule to produce the final decision. In this work, match score level is used to combine the appearance and shape based outputs. For a given appearance-shape test datum of OA and OS, the final recognition C* is given by [60]
(15)C*=argmaxi{γlogP(OAλAi)+(1-γ)logP(OSλSi)},
where λAi and λSi are the appearance and the shape HMMs for the ith utterance class, respectively, and logP(OA/λAi) and logP(OS/λSi) are their log-likelihood against the ith class.Among various types of score fusion techniques, baseline reliability ratio-based integration has been used to combine the appearance and shape recognition results. The reliability of each modality can be measured from the outputs of the corresponding HMMs. The reliability of each modality can be calculated by the most appropriate method which is the best in performance [61],
(16)Sm=1N-1∑i=1N(maxjlogP(Oλj)-logP(Oλi)).Which means the average difference between the maximum log-likelihood and the other ones andN is the number of classes being considered to measure the reliability of each modality, m∈{A,S}.Then the integrated weight of appearance based reliability measureγA can be calculated by [62]
(17)γA=SASA+SS,
where SA and SS are the reliability measures of the outputs of the appearance and shape HMMs, respectively.The integrated weight of shape modality measure can be found as(18)γS=(1-γA).


## 6. BPN Based Likelihood Ratio Score Fusion

A Back-propagation learning feed-forward neural network [63] with tan-sigmoid transfer functions has been used in both the hidden layer and the output layer which is shown in Figure 7. Three-layer Back-propagation learning neural network algorithm has been used to classify the visual speech features [64].Figure 7
Architecture of Back-propagation neural network.If the input vector isI=[p1,p2,…,pn], then the output of hidden layer has been calculated as follows:
(19)n=IW+b,a=f(n)=2(1+e(-2*n))-1,
where, W is weight vector and b is bias input. The error is calculated as the difference between the target output and the network actual output. The goal is to minimize the average of the sum of these errors. Consider the following:
(20)mse=1M∑k=1Me(k)2=1M∑(t(k)-a(k))2.Here, mse means mean square error,t(k) represents the target output, and a(k) represents the network output. The weights and bias values are updated based on the goal average error value.In the proposed audio-visual system, the final weights and bias values are calculated in the training stage. In test phase, the output of the network has been calculated for the new input and compared with the target output to select the class of the input. The numbers of input layers, hidden layers and output layers nodes are 2, 100, and 8, respectively. The overall procedure for the proposed system with likelihood ratio based score fusion with Back-propagation learning neural network is shown in Figure8.Figure 8
BPN based score fusion technique for proposed audio-visual speaker identification system.The major drawbacks of Back-propagation learning neural network algorithm are  the training time and local minima. Convergence time of the Back-propagation algorithm is inversely proportional to the error tolerance rate. In learning, effective use of error rate can decrease the convergence time. At first, select the final error rate. Then converge the weights such that all the patterns overcome some of the percentage error of the total system (the error must be higher than the final error rates). Finally, converge the system to the next lower error rate until crossing the final targeted error. For example, if the error rate of the system is 0.001, first the converged error rate for all of the patterns is 0.009, then 0.005, 0.003, and finally 0.001. This process is known as SET-BPL [65].100 speech utterances are trained in Back-propagation learning neural network and the effects of applying SET-BPL of the proposed system areshown in Figure9. Sometimes local minima problem occurs in a Back-propagation learning neural network algorithm. As a result, some precautions such as addition of internal nodes and  lowering the gain term have been considered to set the learning parameters. The addition of internal nodes and lowering the gain term can increase the convergence time. To overcome these learning difficulties, a momentum term has been used to speed up the convergence process for this proposed speaker identification system.Figure 9
Effects of applying SET-BPL process of the proposed system.


### 6.1. Experimental Evaluation according to VALID Audio-Video Database

VALID audio-visual multimodal database [66] has been used to measure and compare the accuracy between the proposed and existing system. For visual features, database contains 106 subjects each with four office lighting conditions, gathered periodically over one month giving some temporal variation and one studio session with controlled lighting. The 576 × 720 stills were extracted from the video segments. Three sets of these are offered, the 1st, 10th, and 50th frames for each of the 106 × 5 sessions. The five sessions were recorded over a period of one month, allowing for variation in the voice, clothing, facial hair, hairstyle and cosmetic appearance of the subjects and also variation of the visual background, illumination, and acoustic noise. The first session was recorded under controlled acoustic/illumination conditions, that is, controlled environment. The database is designed to be realistic and challenging; hence the other four sessions were recorded in noisy real-world scenarios with no control on illumination or acoustic noise that is, uncontrolled environment. Some processed facial images of VALID database are shown in Figure 10.Figure 10
Some processed facial images of VALID database for the proposed system.For the speech wave, the database contains 106 subjects with one studio and four office conditions recordings for each person corresponding to the facial images where the following two different speech utterances are found:Uttered sentence 1:“Joe Took Father’s Green Shoe bench Out,”Uttered sentence 2: “5 0 6 9 2 8 1 3 7 4.”From the above two sentences, second sentence has been used for learning and testing operations. Out of five facial images and speeches, the neutral face and corresponding speech utterance have been used for learning and other four official noisy images and speeches have used for testing.


### 6.2. Performance Analysis between Existing and Proposed Method

To evaluate the performance of BPN based likelihood ratio based score fusion technique, different variations of audio and visual features are combined and results are taken according to various SNRs of audio signal which are shown in the following subsections.


#### 6.2.1.  Experiment of Appearance Based Facial Feature with Audio Feature

Appearance based facial features are concatenated with MFCC based audio feature to populate the performance of the proposed score fusion based speaker identification. Results are shown in Figure11 where the  highest speaker identification rate has been found to be 95% at SNR of 30 dB for proposed BPN score fusion approach compared with existing Rogozan and Deléglise method of 91.33%.Figure 11
Results of score fusion for appearance based facial feature with MFCC based feature of audio feature.


#### 6.2.2. Experiment of Shape Based Facial Feature with Audio Feature

Figure12 shows the results of shape based facial feature and MFCC based audio feature. At SNR of  30 dB, the speaker identification rate of Rogozan and Deléglise method and proposed BPN score fusion approach has been achieved with 93.33% and 96.33%, respectively.Figure 12
Shape based facial feature and MFCC based audio features for score fusion technique.


#### 6.2.3. Experiment of Combined Appearance-Shape Based Facial Feature with Audio Feature

Results of appearance and shape based facial features with MFCC based audio feature for score fusion technique are shown in Figure13. The highest speaker identification rate of 98.67% has been found at SNR of 30 dB with proposed BPN score fusion approach where existing Rogozan and  Deléglise method achieves 95% at the same SNR.Figure 13
Result of the combinations of audio features with combined appearance and shape based facial feature for score fusion technique.Form the above experimental results, it has been shown that Back-propagation learning network based score fusion approach gives greater performance than any combination of audio and visual features compared with existing Rogozan and Deléglise method of score fusion. Here, it has also been focused on that combined appearance and shape based facial feature achieves higher accuracies than any individual facial feature based technique which is shown in Table11.Table 11
Performance Comparison of Various Combination of Facial Features with Audio Feature for Proposed BPN Approach.
SNRs
Type
Appearance and ΔMFCC based feature with BPN approach (in %)
Shape and ΔMFCC based feature with BPN approach (in %)
Appearance, shape and ΔMFCC based feature with BPN approach (in %)
0 dB
21.67
26.67
33.33
5 dB
29.33
40.25
43.67
10 dB
45.33
56.25
59.67
15 dB
73.25
79.00
83.25
20 dB
84.67
90.67
92.33
25 dB
90.00
93.00
96.00
30 dB
95.00
96.33
98.67


## 6.1. Experimental Evaluation according to VALID Audio-Video Database

VALID audio-visual multimodal database [66] has been used to measure and compare the accuracy between the proposed and existing system. For visual features, database contains 106 subjects each with four office lighting conditions, gathered periodically over one month giving some temporal variation and one studio session with controlled lighting. The 576 × 720 stills were extracted from the video segments. Three sets of these are offered, the 1st, 10th, and 50th frames for each of the 106 × 5 sessions. The five sessions were recorded over a period of one month, allowing for variation in the voice, clothing, facial hair, hairstyle and cosmetic appearance of the subjects and also variation of the visual background, illumination, and acoustic noise. The first session was recorded under controlled acoustic/illumination conditions, that is, controlled environment. The database is designed to be realistic and challenging; hence the other four sessions were recorded in noisy real-world scenarios with no control on illumination or acoustic noise that is, uncontrolled environment. Some processed facial images of VALID database are shown in Figure 10.Figure 10
Some processed facial images of VALID database for the proposed system.For the speech wave, the database contains 106 subjects with one studio and four office conditions recordings for each person corresponding to the facial images where the following two different speech utterances are found:Uttered sentence 1:“Joe Took Father’s Green Shoe bench Out,”Uttered sentence 2: “5 0 6 9 2 8 1 3 7 4.”From the above two sentences, second sentence has been used for learning and testing operations. Out of five facial images and speeches, the neutral face and corresponding speech utterance have been used for learning and other four official noisy images and speeches have used for testing.


## 6.2. Performance Analysis between Existing and Proposed Method

To evaluate the performance of BPN based likelihood ratio based score fusion technique, different variations of audio and visual features are combined and results are taken according to various SNRs of audio signal which are shown in the following subsections.


### 6.2.1.  Experiment of Appearance Based Facial Feature with Audio Feature

Appearance based facial features are concatenated with MFCC based audio feature to populate the performance of the proposed score fusion based speaker identification. Results are shown in Figure11 where the  highest speaker identification rate has been found to be 95% at SNR of 30 dB for proposed BPN score fusion approach compared with existing Rogozan and Deléglise method of 91.33%.Figure 11
Results of score fusion for appearance based facial feature with MFCC based feature of audio feature.


### 6.2.2. Experiment of Shape Based Facial Feature with Audio Feature

Figure12 shows the results of shape based facial feature and MFCC based audio feature. At SNR of  30 dB, the speaker identification rate of Rogozan and Deléglise method and proposed BPN score fusion approach has been achieved with 93.33% and 96.33%, respectively.Figure 12
Shape based facial feature and MFCC based audio features for score fusion technique.


### 6.2.3. Experiment of Combined Appearance-Shape Based Facial Feature with Audio Feature

Results of appearance and shape based facial features with MFCC based audio feature for score fusion technique are shown in Figure13. The highest speaker identification rate of 98.67% has been found at SNR of 30 dB with proposed BPN score fusion approach where existing Rogozan and  Deléglise method achieves 95% at the same SNR.Figure 13
Result of the combinations of audio features with combined appearance and shape based facial feature for score fusion technique.Form the above experimental results, it has been shown that Back-propagation learning network based score fusion approach gives greater performance than any combination of audio and visual features compared with existing Rogozan and Deléglise method of score fusion. Here, it has also been focused on that combined appearance and shape based facial feature achieves higher accuracies than any individual facial feature based technique which is shown in Table11.Table 11
Performance Comparison of Various Combination of Facial Features with Audio Feature for Proposed BPN Approach.
SNRs
Type
Appearance and ΔMFCC based feature with BPN approach (in %)
Shape and ΔMFCC based feature with BPN approach (in %)
Appearance, shape and ΔMFCC based feature with BPN approach (in %)
0 dB
21.67
26.67
33.33
5 dB
29.33
40.25
43.67
10 dB
45.33
56.25
59.67
15 dB
73.25
79.00
83.25
20 dB
84.67
90.67
92.33
25 dB
90.00
93.00
96.00
30 dB
95.00
96.33
98.67


## 6.2.1.  Experiment of Appearance Based Facial Feature with Audio Feature

Appearance based facial features are concatenated with MFCC based audio feature to populate the performance of the proposed score fusion based speaker identification. Results are shown in Figure11 where the  highest speaker identification rate has been found to be 95% at SNR of 30 dB for proposed BPN score fusion approach compared with existing Rogozan and Deléglise method of 91.33%.Figure 11
Results of score fusion for appearance based facial feature with MFCC based feature of audio feature.


## 6.2.2. Experiment of Shape Based Facial Feature with Audio Feature

Figure12 shows the results of shape based facial feature and MFCC based audio feature. At SNR of  30 dB, the speaker identification rate of Rogozan and Deléglise method and proposed BPN score fusion approach has been achieved with 93.33% and 96.33%, respectively.Figure 12
Shape based facial feature and MFCC based audio features for score fusion technique.


## 6.2.3. Experiment of Combined Appearance-Shape Based Facial Feature with Audio Feature

Results of appearance and shape based facial features with MFCC based audio feature for score fusion technique are shown in Figure13. The highest speaker identification rate of 98.67% has been found at SNR of 30 dB with proposed BPN score fusion approach where existing Rogozan and  Deléglise method achieves 95% at the same SNR.Figure 13
Result of the combinations of audio features with combined appearance and shape based facial feature for score fusion technique.Form the above experimental results, it has been shown that Back-propagation learning network based score fusion approach gives greater performance than any combination of audio and visual features compared with existing Rogozan and Deléglise method of score fusion. Here, it has also been focused on that combined appearance and shape based facial feature achieves higher accuracies than any individual facial feature based technique which is shown in Table11.Table 11
Performance Comparison of Various Combination of Facial Features with Audio Feature for Proposed BPN Approach.
SNRs
Type
Appearance and ΔMFCC based feature with BPN approach (in %)
Shape and ΔMFCC based feature with BPN approach (in %)
Appearance, shape and ΔMFCC based feature with BPN approach (in %)
0 dB
21.67
26.67
33.33
5 dB
29.33
40.25
43.67
10 dB
45.33
56.25
59.67
15 dB
73.25
79.00
83.25
20 dB
84.67
90.67
92.33
25 dB
90.00
93.00
96.00
30 dB
95.00
96.33
98.67


## 7. Conclusions and Observations

In this work, proposed system performance has been evaluated in various levels with various dimensions. Two different types of facial features are combined with audio feature with various artificial noise addition rates. NOIZEUS speech database has been used to evaluate the performance of the Audio-Only speaker identification system whereas ORL facial database has been used for visual-only identification system. Finally, overall performance that is, audio-visual speaker identification has been measured according to VALID audio-visual database. Noise removing techniques have used to reduce or eliminate the noises from speech utterances and facial images. Experimental results and performance analysis shows the versatility of the proposed BPN score fusion approach over the existing Rogozan and Deléglise method for audio-visual speaker identification system which can be effectively used in various real life access control and authentication purposes.



---

*Source: 737814-2014-01-08.xml*