# An Objective Approach to Identify Spectral Distinctiveness for Hearing Impairment

Authors: Yeou-Jiunn Chen; Jiunn-Liang Wu
Journal: Mathematical Problems in Engineering
 (2013)

---

## Abstract

To facilitate the process of developing speech perception, speech-language pathologists have to teach a subject with hearing loss the differences between two syllables by manually enhancing acoustic cues of speech. However, this process is time consuming and difficult. Thus, this study proposes an objective approach to automatically identify the regions of spectral distinctiveness between two syllables, which is used for speech-perception training. To accurately represent the characteristics of speech, mel-frequency cepstrum coefficients are selected as analytical parameters. The mismatch between two syllables in time domain is handled by dynamic time warping. Further, a filter bank is adopted to estimate the components in different frequency bands, which are also represented as mel-frequency cepstrum coefficients. The spectral distinctiveness in different frequency bands is then easily estimated by using Euclidean metrics. Finally, a morphological gradient operator is applied to automatically identify the regions of spectral distinctiveness. To evaluate the proposed approach, the identified regions are manipulated and then the manipulated syllables are measured by a close-set based speech-perception test. The experimental results demonstrated that the identified regions of spectral distinctiveness are very useful in speech perception, which indeed can help speech-language pathologists in speech-perception training.


---

## Body

## 1. Introduction

Hearing loss would seriously degrade a subject’s speech perception, thereby affecting the development of articulation ability. It then reduces speech intelligibility and affects speech and language development, learning, and communication. Recently, assistive listening devices such as hearing aids or cochlear implants could help subjects with hearing loss utilize their residual hearing and develop their speech perception [1–7]. To facilitate this process, speech-language pathologists (SLPs) have to provide speech-perception training that could increase subject’s ability to distinguish one syllable from another. In clinical practice, SLPs manually enhance the distinguishable acoustic cues and extensively use them in speech-perception training. However, it is a time-consuming and expensive process. Thus, it is beneficial for SLPs in speech-perception training if the regions of spectral distinctiveness between two syllables can be identified automatically.When a speech wave propagates on the basilar membrane, it is characterized as time-spectral patterns. Then, unique perceptual cues, which are the basic units for speech perception, can be identified. Therefore, the relation between the acoustic cues and perceptual units is a key problem for speech perception [8–10]. In the last decade, the relation had been examined [11–18] and the results show that the main factors of acoustic cues are duration, stress, and spectral distinctiveness.SLPs generally increase the duration and stress of a syllable to teach a subject how to distinguish one syllable from another. Duration and stress can be simply manipulated by speech techniques [19]. However, the spectral distinctiveness between two syllables is very difficult to be identified. In clinical practice, SLPs have to repeatedly pronounce a syllable by enhancing the volume of part segment of a syllable. However, it is a complicated task for SLPs to manually enhance spectral distinctiveness of a syllable. In order to identify the regions of spectral distinctiveness, Li et al. proposed a psychoacoustic method in three dimensions: time, frequency, and intensity [20]; still it is a time-consuming process. Moreover, it is difficult to apply to other languages. Hence, to automatically identify the regions of spectral distinctiveness is very important for hearing impairment in speech-perception training.In this study, an objective approach to identify the regions of spectral distinctiveness is proposed. The mel-frequency cepstrum coefficients (MFCCs) are selected as analytical parameters and used to represent the characteristics of acoustic signal. The mismatch between two syllables in time domain is handled by dynamic time warping; thereby, an optimal matching condition could be obtained. To accurately estimate the spectral similarity, filter bank is applied to find the spectral components of different frequency bands. For the speech signal in each frequency band, the MFCCs are also extracted to represent the acoustical characteristics. According to the optimal matching condition, the spectral distinctiveness of each frequency band between two syllables can be estimated easily by using Euclidean metrics. Finally, the morphological gradient operator is developed to automatically identify the regions of spectral distinctiveness. Moreover, in order to evaluate the accuracy of identified regions of spectral distinctiveness, an acoustic cue manipulation is proposed in this study.The rest of this paper is organized as follows. Section2 describes the objective approach to identify spectral distinctiveness including feature extraction, spectral distinctiveness estimation, and spectral distinctiveness identification. Besides, the acoustic cue manipulation is introduced. Section 3 then describes a series of experiments to examine the performance of our approach. Finally, conclusions are drawn in Section 4, along with recommendations for future research.


## 2. Materials and Methods

In this section, the proposed objective approach to identify the regions of spectral distinctiveness between two syllables (as shown in Figure1) is introduced. Firstly, the MFCCs are extracted from the input speech signals and the filtered speech signals. Secondly, the distance between the MFCCs of two syllables is computed and used to find the consonant-vowel boundary. This approach also adopts the dynamic time warping to find an optimal matching condition between two input syllables. Thirdly, according to the optimal matching condition, the spectral distinctiveness of each frequency band can be easily estimated by using Euclidean metric. Finally, a morphological gradient operator is applied to automatically identify the regions of spectral distinctiveness. To examine the proposed approach, an acoustic cues manipulation is also proposed to manipulate the regions of spectral distinctiveness. These procedures are illustrated serially in the following subsections.Figure 1
The flowchart of the objective approach to identify the regions spectral distinctiveness.


### 2.1. Feature Extraction

Analytical parameters which can accurately represent a speech signal play an important role in objective measurement of spectral distinctiveness. Since, MFCCs had been widely used in speech processing, especially speech recognition [21], they are very suitable for accurately representing not only a speech signal but also speech signals in different frequency bands. The procedure to extract MFCCs from a speech signal is illustrated as follows:(1)
taking the Fourier transform of frames windowed from input speech signal;(2)
mapping the powers of the spectrum onto the mel scale which is defined as(1)m=2595log10(1+h700),
where h is the frequency (Hz) in linear domain;(3)
using triangular overlapping windows to get the power spectrum in mel scale;(4)
taking the logs of the powers at each of the mel frequencies which is denoted asf;(5)
taking the discrete cosine transform of the mel log powers which is defined as(2)c(u)=α(u)∑x=0N-1f(x)cos(π(2x+1)u2N),
where N is the length of window size and α(u) is defined as
(3)α(u)={1N,foru=0;2N,foru≠0;(6)
finally, MFCCs are composed of amplitudes of the resulting spectrum,c(u).In Mandarin, a syllable can be decomposed into an INITIAL and a FINAL. INITIALs consist of consonants or semivowels, and FINALs consist of vowels or vowels plus one of the two nasal sounds. Thus, two syllables represented assi=civ and sj=cjv are used to estimate the regions of spectral distinctiveness. Let Oi=oi1oi2oi3,…,oini and Oj=oj1oj2oj3,…,ojnj, respectively, represent the MFCCs of si and sj, in this study.In addition, a filter bank is adopted to separate the input signal into multiple components, which are the acoustical characteristics of frequency bands. For thekth frequency band, the corresponding MFCCs of si and sj are also extracted and denoted as Oik=oi1koi2koi3k,⋯,oinik and Ojk=oj1koj2koj3k,⋯,ojnjk, respectively.


### 2.2. Spectral Distinctiveness Estimation

In order to estimate the spectral distinctiveness between two syllables, the mismatch should be dealt; thereby, the difference in frequency bands can be easily estimated by using Euclidean metric. Therefore, the dynamic time warping algorithm is adopted to compare two sequencesOi, and Oj; thereby, a plane spanned by Oi and Oj is considered as a distance matrix D, which is written as
(4)D(Oi,Oj)=[d11⋯d1nj⋮⋱⋮dni1⋯dninj],
where dqr is the Euclidean distance between oiq and ojr. The matching condition indicating the correspondence between the time axes of Oi and Oj can be represented a sequence of lattice points on the plane D and written as
(5)H={(h1i,h1j),(h2i,h2j),(h3i,h3j),…,(hni,hnj)},
where hrq is the rth matching pair in sq. The H is the best matching condition and then the dynamic time warping algorithm is described as shown in Algorithm 1.Algorithm 1:The dynamic time warping algorithm.
Step 1. Initialization
DTW(1, 1) =d11
Path(1, 1) = (0, 0)
Fora=2 to ni
DTW(a, 1) = da1 + DTW(a-1, 1)
Path(a, 1) = (a-1, 1)
End For
Forb=2 to nj
DTW(1,b) = d1b + DTW(1, b-1)
Path(1,b) = (1, b-1)
End For
Step 2. Iteration
Fora=2 to ni
Forb=2 to nj
DTW(a,b)=dab+minv∈{((a-1),b),(a,(b-1)),((a-1),(b-1))}{dv}
Path(a,b)=argminv∈{((a-1),b),(a,(b-1)),((a-1),(b-1))}{dv}
End For
End For
Step 3. Backtracking and Termination
The optimal (minimum) distance is DTW(ni, nj).
The optimal matching pathH is found by simple backtracking from Path(ni, nj).
H=⌀
lp = (ni, nj)
Whilelp≠(0,0) Do
H=H∪lp
lp = Path(lp)
End WhileIn the dynamic time warping algorithm, the variable is used to store a path, which reaches at lattice point (a, b) with minimum accumulative distance. The minimum accumulative distance is stored in variableDTW(a,b). Since the durations are quite different for each INITIAL, the boundary condition should be ignored in this study. Hence, the path of the lattice point at first row and column goes through its previous lattice point at the step 1. In order to stop the backtrack for finding the optimal path, path(1, 1) is set to be (0, 0). In addition, monotony and continuity condition is applied to be concerned in the matching condition at step 2. It means that the search space of lattice point (a,b) includes three lattice points: (a-1,b), (a-1,b-1) and (a,b-1). At step 3, a simple method is implemented to find an optimal matching condition H by backtracking from lattice point (ni,nj).According to the optimal matching conditionH and a distinguishable matrix for syllable si, M(si), can be estimated as
(6)M(si)=[m1Fm2F⋯mni-1FmniFm1F-1m2F-1⋯mni-1F-1mniF-1⋮⋮⋱⋮⋮m12m22⋯mni-12mni2m11m21⋯mni-11mni1],
where F is the number of filter bands and mtf is the distance for frame t at fth frequency band. mtf can be defined as
(7)mtf=1#c∑c∋(t,c)∈HDist(oitf,ojcf),
where #c is the number of frames in Oj which is matched with frame t in Oi.


### 2.3. Spectral Distinctiveness Identification

The spectral distinctiveness between two syllables is estimated; thereby, the regions of spectral distinctiveness should be identified from the distinguishable matrixM. Recently, the grayscale morphological gradient operator is a powerful and fast technique for both contour detection and region based segmentation [22]. Thus, it can be successfully used to detect the regions of spectral distinctiveness. To obtain the regions of spectral distinctiveness from M, the morphological gradient operator is used and defined as
(8)MG(f)=1n∑i=1n(((f⊕bi)-(fΘbi))⊕bi-1),
where n is the scale of morphological gradient operator and bi denotes the group of square structuring elements.In (8), the symbols ⊕ and Θ are the grayscale dilation and grayscale erosion, which are defined as follows:
(9)(M⊕b)(s,t)=max(M(s-x,t-y)-b(x,y)(s-x)HHHHHHH∣(t-y)∈Df,(s,y)∈Dbb(x,y)(s-x)),(MΘb)(s,t)=max(M(s-x,t+y)-b(x,y)(s+x)HHHHHHH∣(t+y)∈Df,(s,y)∈Dbb(x,y)(s+x)),
where b is a flat structuring element. Df and Db are the domains of M and b, respectively. According to (9), the grayscale opening and closing then can be derived as
(10)f∘b=(fΘb)⊕b,f•b=(f⊕b)Θb.


### 2.4. Acoustic Cues Manipulation

The accuracy of identify regions of spectral distinctiveness was examined in this subsection. Therefore, the power of the spectrogram of these regions should be manipulated to examine that a syllable is converted to another. Thus, a speech modification procedure based on short-time Fourier transform (STFT) is proposed to analyze a speech sound and then synthesize an enhanced speech sound [23].Letx[n] denote the speech signal at sample times n. A Hamming window w[n] is used to divide x[n] into N point overlapping frames, which can be written as
(11)x[m,n]≡w[n]x[mR-n],
where R is the step size and is defined as N/4. Therefore, the resulting STFT coefficients X[m,k] can be derived as
(12)X[m,k]=∑n=0N-1x[m,n]e-j2πkn/N.
To improve the accuracy of modification, the windowed speech is zero-padded before performing the Fourier transform.The region of spectral distinctiveness is then modified by multiplying a specific gainM[m,k]. Specifically, M[m,k]=0 indicates feature removal, 0<M[m,k]<1 corresponds to feature attenuation, and M[m,k]>1 represents feature enhancement. Thus, the modified speech spectrum can be written as
(13)X~[m,k]=X[m,k]•M[m,k].
Generally, the gain is expressed in dB as
(14)G=20log10(M[m,k]).According toX~[m,k], the single frame signal is recovered by applying an inverse Fourier transform, which is defined as follow:
(15)x~[m,n]=1N∑k=0N-1X~[m,k]ej2πkn/N.
Finally, an overlap-add synthesis is used to generate the modified speech X~[n] in time domain, which can be written as
(16)X~[n]=∑m=-M00X~[mR,n].


## 2.1. Feature Extraction

Analytical parameters which can accurately represent a speech signal play an important role in objective measurement of spectral distinctiveness. Since, MFCCs had been widely used in speech processing, especially speech recognition [21], they are very suitable for accurately representing not only a speech signal but also speech signals in different frequency bands. The procedure to extract MFCCs from a speech signal is illustrated as follows:(1)
taking the Fourier transform of frames windowed from input speech signal;(2)
mapping the powers of the spectrum onto the mel scale which is defined as(1)m=2595log10(1+h700),
where h is the frequency (Hz) in linear domain;(3)
using triangular overlapping windows to get the power spectrum in mel scale;(4)
taking the logs of the powers at each of the mel frequencies which is denoted asf;(5)
taking the discrete cosine transform of the mel log powers which is defined as(2)c(u)=α(u)∑x=0N-1f(x)cos(π(2x+1)u2N),
where N is the length of window size and α(u) is defined as
(3)α(u)={1N,foru=0;2N,foru≠0;(6)
finally, MFCCs are composed of amplitudes of the resulting spectrum,c(u).In Mandarin, a syllable can be decomposed into an INITIAL and a FINAL. INITIALs consist of consonants or semivowels, and FINALs consist of vowels or vowels plus one of the two nasal sounds. Thus, two syllables represented assi=civ and sj=cjv are used to estimate the regions of spectral distinctiveness. Let Oi=oi1oi2oi3,…,oini and Oj=oj1oj2oj3,…,ojnj, respectively, represent the MFCCs of si and sj, in this study.In addition, a filter bank is adopted to separate the input signal into multiple components, which are the acoustical characteristics of frequency bands. For thekth frequency band, the corresponding MFCCs of si and sj are also extracted and denoted as Oik=oi1koi2koi3k,⋯,oinik and Ojk=oj1koj2koj3k,⋯,ojnjk, respectively.


## 2.2. Spectral Distinctiveness Estimation

In order to estimate the spectral distinctiveness between two syllables, the mismatch should be dealt; thereby, the difference in frequency bands can be easily estimated by using Euclidean metric. Therefore, the dynamic time warping algorithm is adopted to compare two sequencesOi, and Oj; thereby, a plane spanned by Oi and Oj is considered as a distance matrix D, which is written as
(4)D(Oi,Oj)=[d11⋯d1nj⋮⋱⋮dni1⋯dninj],
where dqr is the Euclidean distance between oiq and ojr. The matching condition indicating the correspondence between the time axes of Oi and Oj can be represented a sequence of lattice points on the plane D and written as
(5)H={(h1i,h1j),(h2i,h2j),(h3i,h3j),…,(hni,hnj)},
where hrq is the rth matching pair in sq. The H is the best matching condition and then the dynamic time warping algorithm is described as shown in Algorithm 1.Algorithm 1:The dynamic time warping algorithm.
Step 1. Initialization
DTW(1, 1) =d11
Path(1, 1) = (0, 0)
Fora=2 to ni
DTW(a, 1) = da1 + DTW(a-1, 1)
Path(a, 1) = (a-1, 1)
End For
Forb=2 to nj
DTW(1,b) = d1b + DTW(1, b-1)
Path(1,b) = (1, b-1)
End For
Step 2. Iteration
Fora=2 to ni
Forb=2 to nj
DTW(a,b)=dab+minv∈{((a-1),b),(a,(b-1)),((a-1),(b-1))}{dv}
Path(a,b)=argminv∈{((a-1),b),(a,(b-1)),((a-1),(b-1))}{dv}
End For
End For
Step 3. Backtracking and Termination
The optimal (minimum) distance is DTW(ni, nj).
The optimal matching pathH is found by simple backtracking from Path(ni, nj).
H=⌀
lp = (ni, nj)
Whilelp≠(0,0) Do
H=H∪lp
lp = Path(lp)
End WhileIn the dynamic time warping algorithm, the variable is used to store a path, which reaches at lattice point (a, b) with minimum accumulative distance. The minimum accumulative distance is stored in variableDTW(a,b). Since the durations are quite different for each INITIAL, the boundary condition should be ignored in this study. Hence, the path of the lattice point at first row and column goes through its previous lattice point at the step 1. In order to stop the backtrack for finding the optimal path, path(1, 1) is set to be (0, 0). In addition, monotony and continuity condition is applied to be concerned in the matching condition at step 2. It means that the search space of lattice point (a,b) includes three lattice points: (a-1,b), (a-1,b-1) and (a,b-1). At step 3, a simple method is implemented to find an optimal matching condition H by backtracking from lattice point (ni,nj).According to the optimal matching conditionH and a distinguishable matrix for syllable si, M(si), can be estimated as
(6)M(si)=[m1Fm2F⋯mni-1FmniFm1F-1m2F-1⋯mni-1F-1mniF-1⋮⋮⋱⋮⋮m12m22⋯mni-12mni2m11m21⋯mni-11mni1],
where F is the number of filter bands and mtf is the distance for frame t at fth frequency band. mtf can be defined as
(7)mtf=1#c∑c∋(t,c)∈HDist(oitf,ojcf),
where #c is the number of frames in Oj which is matched with frame t in Oi.


## 2.3. Spectral Distinctiveness Identification

The spectral distinctiveness between two syllables is estimated; thereby, the regions of spectral distinctiveness should be identified from the distinguishable matrixM. Recently, the grayscale morphological gradient operator is a powerful and fast technique for both contour detection and region based segmentation [22]. Thus, it can be successfully used to detect the regions of spectral distinctiveness. To obtain the regions of spectral distinctiveness from M, the morphological gradient operator is used and defined as
(8)MG(f)=1n∑i=1n(((f⊕bi)-(fΘbi))⊕bi-1),
where n is the scale of morphological gradient operator and bi denotes the group of square structuring elements.In (8), the symbols ⊕ and Θ are the grayscale dilation and grayscale erosion, which are defined as follows:
(9)(M⊕b)(s,t)=max(M(s-x,t-y)-b(x,y)(s-x)HHHHHHH∣(t-y)∈Df,(s,y)∈Dbb(x,y)(s-x)),(MΘb)(s,t)=max(M(s-x,t+y)-b(x,y)(s+x)HHHHHHH∣(t+y)∈Df,(s,y)∈Dbb(x,y)(s+x)),
where b is a flat structuring element. Df and Db are the domains of M and b, respectively. According to (9), the grayscale opening and closing then can be derived as
(10)f∘b=(fΘb)⊕b,f•b=(f⊕b)Θb.


## 2.4. Acoustic Cues Manipulation

The accuracy of identify regions of spectral distinctiveness was examined in this subsection. Therefore, the power of the spectrogram of these regions should be manipulated to examine that a syllable is converted to another. Thus, a speech modification procedure based on short-time Fourier transform (STFT) is proposed to analyze a speech sound and then synthesize an enhanced speech sound [23].Letx[n] denote the speech signal at sample times n. A Hamming window w[n] is used to divide x[n] into N point overlapping frames, which can be written as
(11)x[m,n]≡w[n]x[mR-n],
where R is the step size and is defined as N/4. Therefore, the resulting STFT coefficients X[m,k] can be derived as
(12)X[m,k]=∑n=0N-1x[m,n]e-j2πkn/N.
To improve the accuracy of modification, the windowed speech is zero-padded before performing the Fourier transform.The region of spectral distinctiveness is then modified by multiplying a specific gainM[m,k]. Specifically, M[m,k]=0 indicates feature removal, 0<M[m,k]<1 corresponds to feature attenuation, and M[m,k]>1 represents feature enhancement. Thus, the modified speech spectrum can be written as
(13)X~[m,k]=X[m,k]•M[m,k].
Generally, the gain is expressed in dB as
(14)G=20log10(M[m,k]).According toX~[m,k], the single frame signal is recovered by applying an inverse Fourier transform, which is defined as follow:
(15)x~[m,n]=1N∑k=0N-1X~[m,k]ej2πkn/N.
Finally, an overlap-add synthesis is used to generate the modified speech X~[n] in time domain, which can be written as
(16)X~[n]=∑m=-M00X~[mR,n].


## 3. Results and Discussions

To evaluate the proposed approach, a close-set based speech-perception test on stop consonants was performed in this study. The speech stimuli, including the syllables /da, ga, ka, ba, pa, ta/, were chosen from the University of Pennsylvania’s Linguistic Data Consortium (LDC) LDC2005S22. The detailed experimental results are shown as follows.


### 3.1. Results of Manipulating /ta/ and /ka/

In this subsection, the syllables /ta/ and /ka/ were used to illustrate the results of the proposed approach. First, the regions of spectral distinctiveness were manually identified to check the results of our approach. The spectrograms of /ta/ and /ka/ were shown in Figure2. It is obvious that /ta/ has high-frequency burst above 4 k Hz (marked as black rectangle) and /ka/ has a low-frequency burst about 1 k Hz (marked as black rectangle). These two regions should be very important to distinguish /ta/ from /ka/.The spectrograms of (a) /ta/ and (b) /ka/.
(a)
(b)Second, the results of distance matrix and dynamic time warping algorithm are examined here. The distance matrixD estimated from /ta/ and /ka/ is shown in Figure 3(a). In this figure, the FINALs of /ta/ and /ka/ are the same; then the distances between the speech segments of FINALs are very small. The distances between the speech segments of FINALs and those of INITIALs are much large. Further, the speech segments of INITIALs for /ta/ and /ka/ are also quite different. Obviously, there exists an optimal path in 45-degree. In Figure 3(b), the optimal matching path can be successfully detected by dynamic time warping algorithm. It demonstrates that the mismatch between two syllables in time domain can be handled in our approach. Besides, the first frame and the third frame of /ka/ match with three frames of /ta/; thus, the first frame and the third frame of /ka/ should be duplicated three times when /ka/ is manipulated as /ta/. It also shows that our approach can be applied to correctly increase the duration of a syllable.The results of (a) distance matrix and (b) optimal matching condition.
(a)
(b)Third, the spectral distinctiveness between /ta/ and /ka/ measured by our approach is validated with manually identified results. Therefore, with the optimal matching path (shown in Figure3(b)), the MFCCs of /ta/ and /ka/ in different frequency bands were adopted to estimate the spectral distinctiveness. The results were normalized in time domain and shown in Figure 4. Comparing Figures 4 and 2, it is obvious that the differences in spectrogram are precisely estimated. By selecting a suitable threshold in morphological gradient operator, the regions of spectral distinctiveness for /ta/ and /ka/ can be identified (shown in Figure 5), which are very similar to the expected regions (shown in Figure 2).The results of distinguishable matrix for (a) /ta/ and (b) /ka/.
(a)
(b)The identified regions of spectral distinctiveness for (a) /ta/ and (b) /ka/.
(a)
(b)Finally, the regions of spectral distinctiveness should be manipulated to examine the accuracy by a subject. According to the identified regions in Figure5, the acoustic cues manipulation was applied to modify /ta/ and /ka/. /ta/ and /ka/ were converted to /ka/ (denoted as /ta → ka/) and /ta/ (denoted as /ka → ta/), respectively. Then, the spectrograms of /ta → ka/ and /ka → ta/ were shown in Figure 6. Comparing Figure 6(a) with Figure 2(b), the spectral energy about 1 k Hz had been relatively decreased. Comparing Figure 6(b) with Figure 2(a), the spectral energy above 4 K increases. Therefore, /ta/ and /ka/ are heard as /ka/ and /ta/, respectively. So, the identified regions really play an important role in distinguishing one syllable from another.The spectrograms of manipulated syllables (a) /ta→ka/ and (b) /ka→ta/.
(a)
(b)


### 3.2. Experimental Results of Subject Evaluation

In this subsection, the results of manipulated syllables are used to examine the identified regions of spectral distinctiveness. Seven males and three females (college students, age about 30 years) were asked to participate in this study. Each token with and without manipulation was randomly presented to each subject 5 times. The speech stimuli were played at the most comfortable level (around 70 dB SPL) for the listeners. The parameters of gainG in (14) were set to be 3 dB, 6 dB, 9 dB, and 12 dB. After each presentation, subjects responded to the stimulus by clicking on one of two buttons labeled with syllables. The detailed results in recognition rate (%) are shown in Table 1. The experimental results show that the recognition rate is over 86%. Moreover, the average recognition rates of manipulated syllables are 89.53%, 91.27%, 92.87%, and 92.80% for G which is 3 dB, 6 dB, 9 dB, and 12 dB, respectively. When the gain G is set to be 9 dB, it can achieve the best recognition rate. Then, speech intelligibility is distorted for the larger gain.Table 1
The recognition rates (%) of syllables with identified regions enhanced by (a) 3 dB, (b) 6 dB, (c) 9 dB, and (d) 12 dB.
(a)
Manipulated syllable
Target syllable
da
ga
ka
ba
pa
ta
da
—
88
90
90
88
90
ga
92
—
94
84
86
88
ka
88
86
—
90
92
96
ba
96
96
90
—
92
92
pa
84
88
88
86
—
90
ta
90
90
88
86
88
—
(b)
Manipulated syllable
Target syllable
da
ga
ka
ba
pa
ta
da
—
88
90
90
88
90
ga
94
—
94
92
96
88
ka
88
96
—
90
92
96
ba
96
96
90
—
92
94
pa
90
88
88
90
—
90
ta
96
90
88
90
88
—
(c)
Manipulated syllable
Target syllable
da
ga
ka
ba
pa
ta
da
—
90
92
92
90
92
ga
94
—
94
92
96
92
ka
94
96
—
90
94
96
ba
96
96
90
—
94
94
pa
90
94
92
90
—
92
ta
96
90
94
90
94
—
(d)
Manipulated syllable
Target syllable
da
ga
ka
ba
pa
ta
da
—
90
92
92
90
92
ga
94
—
94
92
96
92
ka
94
96
—
90
92
96
ba
96
96
92
—
92
94
pa
90
94
92
90
—
92
ta
96
90
94
90
94
—To objectively compare these results, the syllables without manipulation were also used and the results are shown in Table2. The average of recognition rate is 94.93% which is very similar to that of syllables with manipulation (G=9dB). It means that a syllable can be heard as another syllable by manipulating these regions of spectral distinctiveness. Hence, the identified regions of spectral distinctiveness really play an important role in speech perception. SLPs then can apply the identified regions of spectral distinctiveness to help a subject with hearing loss increase his/her ability to distinguish one syllable from another; thereby, the process of speech-perception training then can be facilitated.Table 2
Recognition rates (%) of syllables without manipulation.
Testing syllable
Competitive syllable
da
ga
ka
ba
pa
ta
da
—
94
92
94
92
92
ga
96
—
96
96
94
96
ka
94
96
—
94
96
94
ba
94
94
92
—
98
96
pa
92
96
96
94
—
92
ta
96
94
92
92
94
—


## 3.1. Results of Manipulating /ta/ and /ka/

In this subsection, the syllables /ta/ and /ka/ were used to illustrate the results of the proposed approach. First, the regions of spectral distinctiveness were manually identified to check the results of our approach. The spectrograms of /ta/ and /ka/ were shown in Figure2. It is obvious that /ta/ has high-frequency burst above 4 k Hz (marked as black rectangle) and /ka/ has a low-frequency burst about 1 k Hz (marked as black rectangle). These two regions should be very important to distinguish /ta/ from /ka/.The spectrograms of (a) /ta/ and (b) /ka/.
(a)
(b)Second, the results of distance matrix and dynamic time warping algorithm are examined here. The distance matrixD estimated from /ta/ and /ka/ is shown in Figure 3(a). In this figure, the FINALs of /ta/ and /ka/ are the same; then the distances between the speech segments of FINALs are very small. The distances between the speech segments of FINALs and those of INITIALs are much large. Further, the speech segments of INITIALs for /ta/ and /ka/ are also quite different. Obviously, there exists an optimal path in 45-degree. In Figure 3(b), the optimal matching path can be successfully detected by dynamic time warping algorithm. It demonstrates that the mismatch between two syllables in time domain can be handled in our approach. Besides, the first frame and the third frame of /ka/ match with three frames of /ta/; thus, the first frame and the third frame of /ka/ should be duplicated three times when /ka/ is manipulated as /ta/. It also shows that our approach can be applied to correctly increase the duration of a syllable.The results of (a) distance matrix and (b) optimal matching condition.
(a)
(b)Third, the spectral distinctiveness between /ta/ and /ka/ measured by our approach is validated with manually identified results. Therefore, with the optimal matching path (shown in Figure3(b)), the MFCCs of /ta/ and /ka/ in different frequency bands were adopted to estimate the spectral distinctiveness. The results were normalized in time domain and shown in Figure 4. Comparing Figures 4 and 2, it is obvious that the differences in spectrogram are precisely estimated. By selecting a suitable threshold in morphological gradient operator, the regions of spectral distinctiveness for /ta/ and /ka/ can be identified (shown in Figure 5), which are very similar to the expected regions (shown in Figure 2).The results of distinguishable matrix for (a) /ta/ and (b) /ka/.
(a)
(b)The identified regions of spectral distinctiveness for (a) /ta/ and (b) /ka/.
(a)
(b)Finally, the regions of spectral distinctiveness should be manipulated to examine the accuracy by a subject. According to the identified regions in Figure5, the acoustic cues manipulation was applied to modify /ta/ and /ka/. /ta/ and /ka/ were converted to /ka/ (denoted as /ta → ka/) and /ta/ (denoted as /ka → ta/), respectively. Then, the spectrograms of /ta → ka/ and /ka → ta/ were shown in Figure 6. Comparing Figure 6(a) with Figure 2(b), the spectral energy about 1 k Hz had been relatively decreased. Comparing Figure 6(b) with Figure 2(a), the spectral energy above 4 K increases. Therefore, /ta/ and /ka/ are heard as /ka/ and /ta/, respectively. So, the identified regions really play an important role in distinguishing one syllable from another.The spectrograms of manipulated syllables (a) /ta→ka/ and (b) /ka→ta/.
(a)
(b)


## 3.2. Experimental Results of Subject Evaluation

In this subsection, the results of manipulated syllables are used to examine the identified regions of spectral distinctiveness. Seven males and three females (college students, age about 30 years) were asked to participate in this study. Each token with and without manipulation was randomly presented to each subject 5 times. The speech stimuli were played at the most comfortable level (around 70 dB SPL) for the listeners. The parameters of gainG in (14) were set to be 3 dB, 6 dB, 9 dB, and 12 dB. After each presentation, subjects responded to the stimulus by clicking on one of two buttons labeled with syllables. The detailed results in recognition rate (%) are shown in Table 1. The experimental results show that the recognition rate is over 86%. Moreover, the average recognition rates of manipulated syllables are 89.53%, 91.27%, 92.87%, and 92.80% for G which is 3 dB, 6 dB, 9 dB, and 12 dB, respectively. When the gain G is set to be 9 dB, it can achieve the best recognition rate. Then, speech intelligibility is distorted for the larger gain.Table 1
The recognition rates (%) of syllables with identified regions enhanced by (a) 3 dB, (b) 6 dB, (c) 9 dB, and (d) 12 dB.
(a)
Manipulated syllable
Target syllable
da
ga
ka
ba
pa
ta
da
—
88
90
90
88
90
ga
92
—
94
84
86
88
ka
88
86
—
90
92
96
ba
96
96
90
—
92
92
pa
84
88
88
86
—
90
ta
90
90
88
86
88
—
(b)
Manipulated syllable
Target syllable
da
ga
ka
ba
pa
ta
da
—
88
90
90
88
90
ga
94
—
94
92
96
88
ka
88
96
—
90
92
96
ba
96
96
90
—
92
94
pa
90
88
88
90
—
90
ta
96
90
88
90
88
—
(c)
Manipulated syllable
Target syllable
da
ga
ka
ba
pa
ta
da
—
90
92
92
90
92
ga
94
—
94
92
96
92
ka
94
96
—
90
94
96
ba
96
96
90
—
94
94
pa
90
94
92
90
—
92
ta
96
90
94
90
94
—
(d)
Manipulated syllable
Target syllable
da
ga
ka
ba
pa
ta
da
—
90
92
92
90
92
ga
94
—
94
92
96
92
ka
94
96
—
90
92
96
ba
96
96
92
—
92
94
pa
90
94
92
90
—
92
ta
96
90
94
90
94
—To objectively compare these results, the syllables without manipulation were also used and the results are shown in Table2. The average of recognition rate is 94.93% which is very similar to that of syllables with manipulation (G=9dB). It means that a syllable can be heard as another syllable by manipulating these regions of spectral distinctiveness. Hence, the identified regions of spectral distinctiveness really play an important role in speech perception. SLPs then can apply the identified regions of spectral distinctiveness to help a subject with hearing loss increase his/her ability to distinguish one syllable from another; thereby, the process of speech-perception training then can be facilitated.Table 2
Recognition rates (%) of syllables without manipulation.
Testing syllable
Competitive syllable
da
ga
ka
ba
pa
ta
da
—
94
92
94
92
92
ga
96
—
96
96
94
96
ka
94
96
—
94
96
94
ba
94
94
92
—
98
96
pa
92
96
96
94
—
92
ta
96
94
92
92
94
—


## 4. Conclusions

In this study, an objective approach is proposed to identify the regions of spectral distinctiveness between two syllables. The MFCCs are appropriate to represent not only the speech signal but also the speech components in different frequency bands. In addition, the use of the dynamic time warping overcomes the mismatch between two speech signals in time domain. According to the optimal matching condition, the spectral distinctiveness of each frequency band between two syllables is easily estimated by using Euclidean metrics. The regions of spectral distinctiveness are precisely identified by morphological gradient operator. The experimental results demonstrate that the identify regions play an important role in distinguishing one syllable from another. In the future, the regions of spectral distinctiveness should be automatically enhanced and extensively used in speech-perception training; then it can efficiently reduce the loading of SLPs and facilitate the process of developing speech perception.



---

*Source: 760258-2013-11-28.xml*