FRICATIVE PHONEME DETECTION WITH ZERO DELAY

1 FRICATIVE PHONEMES AND HIGH-FREQUENCY HEARING LOSS . High-frequency hearing loss is a particular type of hearing loss that is mostly age related . People with high-frequency hearing loss have difficulties with word recognition in conversations . This is often caused by the loss of discrimination or detection ability of their ear for high-frequency sounds . Fricative phonemes are particularly hard to recognize for the affected people because these phonemes contain high-frequency information that is crucial to be able to discriminate them . The problem can persist if the affected people wear modern hearing aids . Researchers suggested frequency lowering approaches to make fricative phonemes audible by the hearing impaired . These techniques shift the high frequency parts of these sounds into lower frequency bands . For example , Robinson et al . ( 2009 ) show that a frequency transposition technique leads to improved detection and recognition ability in high frequency hearing impaired people for the sounds /s/ and /z/ , which are fricative sounds . In this paper , we use the standard phonetic codes when we refer to the fricative phonemes as listed in Table 1 . In order to employ such techniques in hearing aids , one must be able to reliably detect fricative phonemes with zero delay . This means that the algorithm needs to be able to detect that a fricative phoneme is coming next , before this has happened , so that the necessary alterations to the sound can be made in real time . In this paper we show how to use deep learning to solve this problem . 2 DATASET , NETWORK ARCHITECTURE , AND TRAINING . We employ a fully convolutional neural network with residual connections ( He et al. , 2016 ) . The network takes raw speech segments as inputs , and calculates the posterior probability for the event that the sample at the end of the segment belongs to a fricative phoneme . The detection procedure is explained in Figure 1 where the raw speech segment is in blue , the input segment of the network is highlighted in green and the dashed green lines show the borders of the phonemes as marked in the TIMIT dataset ( Garofolo et al. , 1993 ) . Based on the input segment , the network detects the identity ( belongs to a fricative phoneme or not ) of the sound sample represented with the red vertical line at the end of the segment . It is important to stress that the red line is at the end of the green segment , and not in the middle of the green segment . This means that we are not using non-causal information to detect the identity of the red sample . In other words , our method introduces zero statistical delay . Observe that in Figure 1 the green segment spans four phonemes : /tcl/ , /w/ , /ix/ , /z/ ( see Table A.1 for an explanation of these phonetic codes ) . The red sample belongs to the phoneme /z/ , which is a fricative phoneme , so that the correct detection in this case is fricative . Note that the fricative phoneme /z/ is barely contained in the green segment . Essentially , the network must learn to make accurate inferences about future phonemes based on the currently observed phonemes . Statistically , this is harder than making a detection about , for example , the middle point of the segment : think about the difference between interpolation and extrapolation . 2.1 DATASET AND DATA PREPROCESSING . The TIMIT Speech Corpus consists of raw speech sentences ( at 16 kHz sampling rate ) and their corresponding time aligned phonetic labels ( Garofolo et al. , 1993 ) as illustrated in Figure 1 . It contains a total of 6300 sentences , 10 sentences spoken by each of 630 speakers from 8 major dialect regions of the United States . We used the entire training set of the TIMIT dataset for training ( 462 speakers ) , and the core test set for testing ( 24 speakers ) . We created the validation set ( 50 speakers ) following the methodology in Halberstadt ( 1998 ) . In the TIMIT dataset , there are two sentences that are read by all the speakers ; these are called SA sentences . We excluded these two sentences from all of the speakers in all sets ( training , validation and test ) to prevent a possible bias in the evaluation . Specifically , if SA sentences are not excluded , the model might conceivably memorize the locations of the phonemes in these sentences using the training set , and then make near perfect detection for these sentences in the test set . This would make the model appear to perform better than it does in reality . After the exclusion of SA sentences , there were 8 sentences per speaker left . To keep the training and validation set balanced , we ensured a ( approximately ) 50/50 fricative-tonon-fricative sample ratio . To do this , we extracted 16 randomly placed 192 ms ( 3072 in samples ) long speech segments from each sentence : 8 segments labeled as fricative phoneme and 8 segments labeled as non-fricative phoneme . Recall that the label for a segment is determined based on the identity of the last sample of the segment . If a sentence did not have a fricative phoneme sample , we generated 16 non-fricative segments ; this procedure did not affect the ( approximate ) 50/50 fricativeto-non-fricative sample ratio , since there were not many sentences in training and validation set that had no fricative phoneme sample in them . For each epoch , our training set consisted of 59136 ( 462 speakers x 8 sentences x 16 random segments ) speech segments placed randomly as explained above . One such segment is highlighted in green in Figure 1 . From epoch to epoch the training set of 59136 segments was randomly regenerated from the training set of the TIMIT dataset . We applied the same procedure to form our validation set , but we kept it fixed for different training runs and also for different epochs in same training run . Our validation samples consisted of 6400 ( 50 speakers x 8 sentences x 16 segments ) speech segments . For each segment x = [ x1 , . . . , x3072 ] from training , validation and test sets , we applied standard deviation normalization x← x/ √ Var ( x ) , where Var ( x ) is the variance of the segment . 2.2 NETWORK ARCHITECTURE . Table 2 shows the layer structure of the model we used . The input layer of the network takes raw speech segments of 3072 samples ( 192 ms at 16 kHz sampling rate ) . The input layer is followed by 5 stages . Stage 1 consists of one convolutional layer , stages 2–5 consist of 6 convolutional layers each . For each convolutional layer , in Table 2 , the first number in the square brackets indicates the kernel size of the layer , the number after “ / ” indicates the stride value in the convolution , and the last number indicates the number of filters used in the layer . The strides are only applied to the first convolutional layer of each stage . All of the layers have ReLU non-linearity , except the single neuron at the output layer that has the sigmoid non-linearity . In stages 2–5 we used residual connections ( He et al. , 2016 ) . If the filter dimensions do not match between two layers for residual connection , we applied zero padding to match the filter dimensions ( some residual connections had to be applied among two layers that had different number of filters , since we increased the number of filters as we went deeper in the model ) . The residual connections were applied around pairs of convolutional layers , connecting the input to the first layer of the pair to the output of the linear filter of the second layer of the pair , immediately before the non-linearity of the second layer . After the convolutional stages , 1D global average pooling layer was used . The total number of parameters of this model is 1.1 million . See Figure A.1 in Appendix for details of the model . 2.3 ALTERNATIVE ARCHITECTURES . We experimented widely with the network architecture : the input segment size , the number of layers , the number of features . The network in Table 2 achieved the best performance in our experiments . As can be seen in Figure 2 below , the input size of 3072 samples corresponds to 2-3 phonemes . Intuitively , this provides abundant information to infer the phoneme that follows . Performance of more shallow networks and networks with fewer features is discussed in Section 4 . Our network receives the raw speech signal as input . The use of raw waveform ( see also Palaz et al . ( 2013 ) ; Zeghidour et al . ( 2017 ) ) enables us to learn problem-specific low-level features . Nevertheless , in audio processing it is more common to first transform the raw signal into a time-frequency representation , for example , by computing the Mel-Frequency Cepstrum Coefficients ( MFCC ) , and then to feed the resulting features into a classifier . Because of the time-frequency uncertainty principle , the computation of MFCC features requires a window of certain size , often 25 ms. To compensate for the delay incurred by processing the signal in the window , the classifier must learn to extrapolate detections into the future . While this is possible , as explained in Section 5 below , in general it is a harder task that to make predictions about the present sample . In a separate line of work , we used a proprietary high-quality time-frequency filterbank , followed by a recurrent neural network , to solve the fricative detection problem . After extensive experimentation , we were able to get within a few percent of the performance reported in this paper , however , we were not able to surpass the performance of the model in Table 2 . The advantage of the time-frequency transformation followed by a recurrent neural network , is that that approach leads to a much more computationally efficient implementation . The value of this work is to push the boundary of what is possible in terms of detection accuracy . We plan to report the somewhat inferior but computationally more efficient results in a separate publication . 2.4 TRAINING . We used binary crossentropy loss and Adam optimizer starting with 0.001 learning rate . We halved the learning rate if the validation loss did not decrease within 10 epochs , and we applied early stop- ping if the loss computed on the validation set did not decrease within 40 epochs . We applied 0.0001 l2 weight decay to all of the weights in the convolutional layers , and applied batch normalization after each linear filtering operation and before each non-linearity . See Figure A.1 in Appendix for details on batch normalization . For the results reported in Tables 3 , 4 and 5 and Figures A.3 and A.4 we trained the network 10 times and selected the network that performed best on the validation set ; for the results reported in Table 7 we trained the network once only . 3 EVALUATION . We evaluated the performance of our detection algorithm with the test set ( the core test of the TIMIT dataset ) that contains 192 sentences with 24 different speakers ( each speaker has 8 sentences since we excluded two SA sentences as explained in Section 2.1 ) . The total duration of the test set is 545 seconds . Figure 2 explains how we use the trained network to make inferences . The raw speech is displayed in blue in the top display . We take a sliding window that consists of 3072 samples ( 192 ms ) , represented in black in the top display . The window is moved from left to right , one sample at a time . For each location of the sliding window , we apply the network to the speech segment marked by the window . This gives us the posterior probability for the event that the sample at the end of the sliding window belongs to a fricative phoneme . The posterior probabilities are displayed in blue in the bottom display . When we compare the network ’ s detections with the ground truth displayed in orange , we observe that the network was able to detect the beginning and the end of the fricative phoneme regions accurately , even though it did not have access to the future context . To make the final decision based on the posterior probabilities produced by the network , we apply thresholding as follows : Decision = { Fricative , if p ( x ) > threshold Non-fricative , otherwise . ( 1 ) Above , p ( x ) ( the blue line in the lower display in Figure 2 ) is the posterior probability for the event that the last sample of x ( segment in black box in the upper display in Figure 2 ) belongs to a fricative phoneme . We used the validation set to decide a threshold that maximized the Unweighted Average Recall ( UAR ) of our network . UAR is the average recall of the two classes ( fricative and non-fricative phonemes ) : UAR = Recallf +Recalln 2 , Recallf = TP TP + FN , Recalln = TN TN + FP . Above , TP , TN , FP , and FN is the number of true positive , true negative , false positive , and false negative detections of fricative phonemes , respectively ; Recallf and Recalln is the recall rate for fricative and non-fricative phonemes , respectively . We used UAR as our metric since the test set was highly unbalanced ( see # of Samples column in Table 3 ) . We also report the precision rates and the F1-scores ( i ∈ { f , n } ) : Precisionf = TP TP + FP , Precisionn = TN TN + FN , F1 , i = 2× Precisioni ×Recalli Precisioni +Recalli . Table 3 summarizes the detection results with the selected threshold ( 0.42 ) . Overall 94.76 % UAR was achieved . To the best of our knowledge , this is the best UAR for fricative phoneme detection on the TIMIT dataset .

Summary: This paper apples supervised deep learning methods to detect exact duration of a fricative phoneme in order to improve practical frequency lowering algorithm. A major challenge compared to existing work is to have an algorithm with nearly zero delay while preserving detection accuracy. A deep convolutional neural network is trained for this purpose and it is validated on TIMIT dataset.