FRICATIVE PHONEME DETECTION WITH ZERO DELAY
Authors: Metehan Yurt
Alberto N. Escalante B.
Veniamin I. Morgenshtern
Venue:  (2020)
Keywords: fricative detection
phoneme detection
speech recognition
deep learning
hearing aids
zero delay
extrapolation
TIMIT

Abstract: People with high-frequency hearing loss rely on hearing aids that employ frequency lowering algorithms. These algorithms shift some of the sounds from the high frequency band to the lower frequency band where the sounds become more perceptible for the people with the condition. Fricative phonemes have an important part of their content concentrated in high frequency bands. It is important that the frequency lowering algorithm is activated exactly for the duration of a fricative phoneme, and kept off at all other times. Therefore, timely (with zero delay) and accurate fricative phoneme detection is a key problem for high quality hearing aids. In this paper we present a deep learning based fricative phoneme detection algorithm that has zero detection delay and achieves state-of-the-art fricative phoneme detection accuracy on the TIMIT Speech Corpus. All reported results are reproducible and come with easy to use code that could serve as a baseline for future research.


Under review as a conference paper at ICLR 2020
FRICATIVE PHONEME DETECTION WITH ZERO DELAY
Anonymous authors Paper under double-blind review
ABSTRACT
People with high-frequency hearing loss rely on hearing aids that employ fre- quency lowering algorithms. These algorithms shift some of the sounds from the high frequency band to the lower frequency band where the sounds become more perceptible for the people with the condition. Fricative phonemes have an impor- tant part of their content concentrated in high frequency bands. is important that the frequency lowering algorithm activated exactly for the duration of a fricative phoneme, and kept off at all other times. Therefore, timely (with zero delay) and accurate fricative phoneme detection a key problem for high quality hearing aids. In this paper we present deep learning based fricative phoneme detec- tion algorithm that has zero detection delay and achieves sfte-of--hhe-rt fricative phoneme detection accuracy on the TIMIT Speech Corpus. All reported results are reproducible and come with easy to use code that could serve baseline for future research.
FRICATIVE PHONEMES AND HIGH-FREQUENCY HEARING LOSS
High-frequenee hearing loss is a particular type of hearing loss that is mostly age related. Peo- ple with high-frequency hearing loss have difficulties with word recognition in conversations. This is often caused by the loss of discrimination or detection ability of their ear for high-frequency sounds. Fricative phonemes phonemes contain high-frequency information that crucial to able to discriminate them. The problem persist if the affected people wear modern hearing aids. Researchers suggested fre- quency lowering approaches to make fricative phonemes audible by the hearing impaired. These techniques shift the high frequency parts of these sounds into lower frequency bands. For example, Robinson et al. (2009) show that a frequency transposition technique leads to improved detection and recognition ability in high frequency hearing impaired people for the sounds /s/ and /z/. which are fricative sounds. this paper, we use the standard phonetic codes when we refer to the fricative phonemes as listed in Table T In order to employ such techniques in hearing aids, one must be able reliably detect fricative phonemes with zero delay. This means that the algorithm needs be able to detect that fricative phoneme is coming next, before this has happened, so that the necessary alterations to the sound can be made in real time. In this paper we show how use deep learning to solve this problem.
particularly hard recognize for the affected people because these
Table 1: Fricative phonemes of English.
PHONETIC CODE EXAMPLE WORD
/s/
/sh/
/f/
/th/
/z/
/zh/
/v/
/dh/ then
sea
she
fin
thin
zone
azure
van
2 DATASET, NETWORK ARCHITECTURE. AND TRAINING
We employ a fully convolutional neural network with residual connections (He et al.) 2016). The network takes raw speech segments as inputs. and calculates the posterior probability for the event that the sample at the end of the segment belongs to a fricative phoneme. The detection procedure is explained in Figure T where the raw speech segment is in blue, the input segment of the network is highlighted in green and the dashed green lines show the borders of the phonemes as marked in the TIMIT dataset (Garofolo al. 1993). Based on the input segment, the network detects the identity

---

Under review as a conference paper at ICLR 2020
(belongs fricative phoneme not) of the sound sample represented with the red vertical line the end of the segment.
Phonetic codes
ih
tel
W
ix
Amplitude
0.02
0.00
-0.0
50
100
150
200
250
300
350
Time in ms
Figure Speech segment pronouncing "it was" with the network input æ highlighted in green.
is important to stress that the red line is at the end of the green segment, and not in the middle of the green segment. This means that of the red sample. In other words, our method introduces zero statistical delay.
not using non-causal information to detect the identity
Observe that in Figure T the green segment spans four phonemes: /tll, /w/, /ix/, /z/ (see Table A.I for an explanation these phonetic codes). The red sample belongs to the phoneme /z/, which is fricative phoneme, so that the correct detection in this case is fricative. Note that the fricative phoneme /z/ barely contained in the green segment. Essentially, the network must learn to make accurate inferences about future phonemes based on the currently observed phonemes. Statistically, this is harder than making detection about, for example, the middle point of the segment: think about the difference between interpolation and extrapolation.
2.1 DATASET AND DATA PREPROCESSING
The TIMIT Speech Corpus consists of raw speech sentences (at 16 kHz sampling rate) and their corresponding time aligned phonetic labels (Garofolo et al.] 1993) as illustrated in Figure [ contains total of 6300 sentences, 10 sentences spoken by each of 630 speakers from 8 major dialect regions of the United States. We used the entire training set of the TIMIT dataset for training (462 speakers), and the core test set for testing (24 speakers). We created the validation set (50 speakers) following the methodology in Halberstadt (1998).
In the TIMIT dataset, there are two sentences that are read by all the speakers; these called SA sentences. We excluded these two sentences from all of the speakers in all sets (training, validation and test) to prevent possible bias in the evaluation. Specifically, if SA sentences are not excluded, the model might conceivably memorize the locations of the phonemes in these sentences using the training set, and then make near perfect detection for these sentences in the test set. This would make the model appear to perform better than does in reality. After the exclusion of SA sentences, there were 8 sentences per speaker left.
To keep the training and validation set balanced, ensured (approximately) 50/50 fricative-to- non-fricative sample ratio. To do this, we extracted 16 randomly placed 192 ms (3072 in samples) long speech segments from each sentence: 8 segments laheled fricative phoneme and 8 segments labeled as non-fricative phoneme. Recall that the label for segment is determined based on the identity of the last sample of the segment. If sentence did not have fricative phoneme sample, generated non-fit non-fricative segments; this procedure did not affect the (approximate) 50/50 fricative- -f-nnnffricatve sample ratio, since there were not many sentences in training and validation set that had no fricative phoneme sample in them.
For each epoch, our training set consisted of 59136 (462 speakers x 8 sentences 16 random seg- ments) speech segments placed randomly as explained above. One such segment is highlighted in green in Figure [] From epoch to epoch the training set of 59136 segments was randomly regener- ated from the training set of the TIMIT dataset.
We applied the same procedure to form our validation set, but we kept it fixed for different training runs and also for different epochs in same training run. Our validation samples consisted of 6400 (50 speakers x sentences x 16 segments) speech segments.
For each segment [ from training, validation and test sets, we applied standard deviation normalization i var(aa) where Var(æ:) is the variance of the segment.
2

---

Under review as a conference paper at ICLR 2020
2.2 NETWORK ARCHITECTURE
Table 2] shows the layer structure of the model we used. The input layer of the network takes raw speech segments of 3072 samples (192 ms at 16 kHz sampling rate). The input layer is followed by 5 stages. Stage consists of one convolutional layer, stages 2-5 consist of convolutional layers each. For each convolutional layer, in Table 3 the first number in the square brackets indicates the kernel size of the layer, the number after "7" indicates the stride value the convolution, and the last number indicates the number of filters used in the layer. The strides are only applied to the first convolutional layer of each stage. All of the layers have ReLU non-linearity, except the single neuron at the output layer that has the sigmoid non-linearity. In stages 2-5 we used residual connections (He al. 2016). If the filter dimensions do not match between two layers for residual connection, we applied zero padding to match the filter dimensions (some residual connections had be applied among two layers that had different number of filters, since we increased the number of filters as we went deeper in the model). The residual connections were applied around pairs of convolutional layers, connecting the input to the first layer of the pair the output of the linear filter of the second layer of the pair, immediately before the non-linearity of the second layer. After the convolutional stages, ID global average pooling layer was used. The total number of parameters of this model is 1.1 million. See Figure A.I in Appendix for details of the model.
Table 2: Network architecture
LAYER
DESCRIPTION
Input
Raw speech segment with shape [3072, 1] [32/6, 48]
Convolutional Stage 1
Convolutional Stage 2
[8/3, 64] 6
Convolutional Stage 3
[8/3, 64] 6
Convolutional Stage 4
[8/2, 80] 6
Convolutional Stage 5
[8/2, 96] x 6
Global Average
ID global average pooling
Output
Single sigmoid neuron
2.3 ALTERNATIVE ARCHITECTURES
We experimented widely with the network architecture: the input segment size, the number of layers, the number of features. The network in Table 2 achieved the best performance in our experiments. As can be seen in Figure below, the input size of 3072 samples corresponds to 2-3 phonemes. Intuitively, this provides abundant information to infer the phoneme that follows. Performance of more shallow networks and networks with fewer features is discussed in Section
Our network receives the raw speech signal as input. The use of raw waveform (see also Palaz et al. 2013); Zeghidour al. (2017)) enables us learn problem-specific low-level features. Neverthe- less, in audio processing it is more common to first transform the raw signal into a time-frequency representation, for example, by computing the Mel-Frequency Cepstrum Coefficients (MFCC), and then to feed the resulting features into a classifier. Because of the time-freyuency uncertainty prin- ciple, the computation of MFCC features requires a window of certain size, often 25 ms. To com- pensate for the delay incurred by processing the signal in the window, the classifier must learn to extrapolate detections into the future. While this is possible, as explained in Section 5 below, in general it is a harder task that to make predictions about the present sample.
separate line of work, we used a proprietary high-quality time-frequency filterbank, followed by recurrent neural network, to solve the fricative detection problem. After extensive experimentation, we were able to get within a few percent of the performance reported in this paper. however, we were not able to sur- pass the performance of the model in Table 2 The advantage of the fine-freuenccy transformation followed by recurrent neural network, is that that approach leads to much more computationally efficient implementation. The value of this work is to push the boundary of what is possible in terms of detection accuracy. We plan to report the somewhat inferior but computationally more efficient results in a separate publication.
2.4 TRAINING
We used binary crossentropy loss and Adam optimizer starting with 0.001 learning rate. We halved the learning rate if the validation loss did not decrease within 10 epochs, and we applied early stop-
3

---

Under review as a conference paper at ICLR 2020
ping if the loss computed on the validation set did not decrease within 40 epochs. We applied 0.0001
the weights in the convolutional layers, and applied batch normalization after each linear filtering operation and before each non-linearity. See Figure [A.]] in Appendix for Figures and
12 weight decay
details on batch normalization. For the results reported Tables and
we trained the network 10 times and selected the network that performed best on the validation set; for the results reported in Table we trained the network once only.
EVALUATION
We evaluated the performance of our detection algorithm with the test set (the core test of the TIMIT dataset) that contains 192 sentences with 24 different speakers (each speaker has 8 sentences since we excluded two SA sentences as explained in Section 2.1)). The total duration of the test set 545 seconds.
Figure[Jexplains how we use the trained network to make inferences. The raw speech is displayed in blue in the top display. We take sliding window that consists of 3072 samples (192 ms), represented in black in the top display. The window is moved from left to right, one sample at a time, For each location of the sliding window, we apply the network to the speech segment marked by the window. This gives us the posterior probability for the event that the sample at the end of the sliding window belongs to fricative phoneme. The posterior probabilities are displayed in blue in the bottom display.
When we compare the network's detections with the ground truth displayed in orange, we observe that the network was able detect the beginning and the end of the fricative phoneme regions accurately, even though did not have access to the future context.
ih
k
pau
eh
y
k
tel
ix
q
kel
w del
r
kel
w
del
Figure 2: Top: Raw speech in blue and input to the network in the black box. Bottom: Posterior probabilities calculated by the network blue and the ground truth in orange.
To make the final decision based on the posterior probabilities produced by the network, we apply thresholding follows:
Decision Fricative, if p(x) > threshold
(1)
Non-fricative, otherwise.
Above, p(z) (the blue line in the lower display in Figure 2 is the posterior probability for the event that the last sample of (segment in black box in the upper display in Figure (2) belongs to fricative phoneme. We used the validation set to decide a threshold that maximized the Unweighted Average Recall (UAR) of our network. UAR is the average recall of the two classes (fricative and non-fricative phonemes):
Recall, + Recall,
TP
TN
Recall, TN + FP FP. and FN is the number of true positive, true negative, false positive, and false negative detections of fricative phonemes, respectively; Recall, and Recall, is the recall rate for fricative and non-fricative phonemes, respectively. We used UAR as our metric since the test set was highly unbalanced (see # of Samples column in Table 3.
UAR
Recally = TP + FN'
2
Above, TP, FP,
We also report the precision rates and the Fi-scores (i E (f,n)): TN
TP
Precision; Recall, Fi, ===22 Precision; + Recall,
Precision, TP FP'
Precision,
TN + FN
Table 3 summarizes the detection results with the selected threshold (0.42). Overall 94.76% UAR was achieved. To the best of our knowledge, this is the best UAR for fricative phoneme detection on the TIMIT dataset.
4

---

Under review as a conference paper at ICLR 2020
Table 3: Precision, Recall and FI-Score in % for our phoneme detection algorithm. Number of Samples indicates the total number of time samples (at 16 kHz) in the test set.
CLASS
PRECISION
RECALL
FI-SCORE
# OF SAMPLES
Non-fricative Phonemes (n)
98.98
94.71
96.80
7344071
Fricative Phonemes (f)
77.15
94.80
85.07
1382200
Unweighted Averages
88.06
94.76
90.93
3.1 COMPARISON WITH THE LITERATURE
In this section we compared our algorithm with other results in phoneme detection literature. We first summarized the literature that focused only fricatives phoneme detection (as in this work), then we briefly summarized the literature that focused on generie phoneme identification. Finally, summarized all the results in one table.
3.1.1 FRICATIVE PHONEME DETECTION
Fricative phoneme detection is the task of detecting all occurrences of fricative phonemes in contin- uous speech. Vydana Vuppala (2016) investigated the fricative phoneme detection using a com- bination of S S-Transform and Linear Prediction Coefficients (LPC) approach. In their evaluation, the authors used post-processing rule to remove spurious detections: if the duration of the detected evidence was shorter than 10 ms, then this evidence was not considered as fricative phoneme. The authors applied majority voting to each phoneme, so that single detection made only at the end of each phoneme. Further, the authors evaluated their algorithm by using 40 sentences from the TIMIT dataset, without excluding the SA sentences (see Section 2.I). This, we believe, might have introduced a bias in the reported detection rate making it too optimistic, since the SA sentences are same for all speakers in the dataset and must have appeared both in the training set and in the test set. The authors reported 83.81% % recall rate for the fricative phoneme class and did not report the recall rate for the non-fricative class.
Ruinskiy & Lavner (2014) investigated the fricative phoneme detection by combining time domain and frequency domain features that were fed into linear classifier. The detections were post- processed by using median filtering to avoid short gaps within a sequence of positive detections, and to eliminate detections that were too short to be considered as whole phonemes. The authors applied majority voting for each phoneme and only evaluated their algorithm with unvoiced fricative phonemes (/sl, /sh/, /f/, /th/) and not with all 8 fricative phonemes from Table I In the evaluation stage, the authors did not exclude the SA sentences, which, as stated above, might have introduced a bias in the reported detection rate making it too optimistic. To the best of our knowledge, this reached the best fricative phoneme detection rate reported on the TIMIT dataset date: 94.08% UAR.
The non-causal post-processing that was used in Vydana & Vuppala (2016) and Ruinskiy & LLvner (214)) might be suitable for some applications, such as for speech-to-tex conversion. However is not suitable for signal processing in hearing aids, because, as explained in Section these devices must work in real time with minimal delay.
3.1.2 GENERIC PHONEME IDENTIFICATION
The generic phoneme identification task is similar to the fricative phoneme detection task, with the only difference that the aim is to identify all phonemes individually. This task is more challeng- ing than the fricative phoneme detection since the number of classes is higher (fotal-number-of- phonemes-mand many classes vs. two classes).
It was widely believed that the processing of the input context forward and backward in time from the speech frame to be identified is crucial for success in phoneme identification (Graves Schmid- huber, (2005). Hence, almost all of phoneme identification studies relied on system where the speech frame to be identified was located the middle of a larger input context used by the algo- rithm to make the decision. This comes with a delay in the phoneme identification. For example, if the input to the algorithm consists of N successive overlapping short speech frames (typically 25 ms long each with overlap of 10 ms) and the detection about the middle frame is made, a delay of (N - 1)/2 22ix, is accumulated, assuming N is odd. This is illustrated in Figure A.5
5

---

Under review as a conference paper at ICLR 2020
Siniscalchi et al. (2011) and Yu et al. (2012) utilized single hidden layer Multi Layer Perceptron (MLP) and a Deep Neural Network (DNN), respectively, where the input context had 11 frames and the frame in the middle was identified. Therefore, there were 5 frames delay in the identification. Unfortunately, the authors did not report the frame length in seconds. The authors achieved 93. 17% and 96.2% recall rates for fricative phonemes on the Nov92 dataset. SSiniscalchi et al. (013) utilized a DNN, using 310 ms (total time span from Frame to Frame N in Figure A.5 input context that was centered around the frame being detected leading to a 155 ms delay. The authors reported 95,4% recall rate for fricative phonemes on the Nov92 dataset. [Chn et 2014) employed a DNN, where the input context consisted of 9 frames that was centered around the frame being detected. The authors reported 88.2% recall rate for fricative phonemes on the Switchboard part of the NIST 2000 Hub5 dataset. All these methods introduce significant detection delays and the reported rates directly comparable
provided for the TIMIT dataset. For convenience of the reader we assembled a summary of these results in Table B. in the appendix. Zhang et al. (2019;; Chorowski et al. (2013): Töth (2015) addressed the task of generic phoneme detection on TIMIT using neural networks, unfortunately comparison with this work is not possible since fricative-specife results are reported.
ours because the rates
Table 4: Fricative phoneme detection rates. Recally, Recall, and U UR are given %.
RELATED WORK
RECALLF
RECALLN UAR
DELAY
Ruinskiy & Lavner 2014)
89.86
98.29
94.08
80 ms
Vydana & Vuppala
2016
83.81
80 ms?
Our Result
94.80
94.71
94.76 0
Majority voting prot-processing leads to a phoneme-length detection delay. Average length of phoneme the TIMIT dataset 80 ms. If there would be no majority voting post-processing, the rates of these two contributions would have been worse than reported the table.
Vydana & Vuppala 6 and Ruinskiy & Lavner 2014) aimed at fricative phoneme detection and evaluated their al algorithm on the TIMIT dataset as this work. comperes the reported results in these two approaches to ours. RRuinkky & Lavner (2014) is paper that is most closely related to ours, which calls for detailed comparison provided next.
The authors of Ruinskiy & Lavner (2014) only attempted to detect unvoiced fricative phonemes: /s/, /sh/, /f/ and /th/ and applied majority voting in their evaluation. To make a fair comparison, we also tested our detection with unvoiced fricative phonemes only and also applied majority voting. Table
summarizes the results. A further comparison in terms of Receiver Operating Characteristic (ROC) curves between our results and those of Ruinskiy & Lavner 2014) is reported in Figure A.3 in Appendix.
Table 5: Comparison between this work and Ruinskiy & Lavner (2014). First row: the rates Ru- inskiy & Lavner (2014) for unvoiced fricative phonemes with majority voting. Second row: the rates for our basic approach that does not rely on any post-processing introducing delay and also is evaluated with all 8 fricative phonemes. Third row: the rates for our basic approach evaluated only with unvoiced fricative phonemes in |Ruinskiy Lavner (2014). Forth row: the rates for our basic approach followed by majority voting (introducing delay as in Ruinskiy & Lavner (2014)) evaluated with unvoiced fricative phonemes. All the rates given in %. Our rates reported in Forth row were obtained using the same evaluation strategy as in Ruinskiy & Lavner (2v1er](2014) except the median filtering (since the details for this step were not given in their paper).
RELATED WORK
RECALLP
RECALL
UAR
Ruinskiy & Lavner (2014)
89.86
98.29
94.08
Our result: no post processo.,,,,gg, 8 fricatives
94.80
94.71
94.76
Our result: no proseprocessing. unvoiced fricatives
95.92
94.71
95.32
Our result:
majority voting, unvoiced fricatives
96.94
95.87
96.41
Based the table, observed the following.

---

Under review as a conference paper at ICLR 2020
3.1.3 DETECTION OF /V/ AND /DH/
Our detection performance increased when fricatives /v/ and /dh/ excluded from consideration as was done in Ruinskiy Lavner (2014), since these two phonemes are voiced fricative phonemes. The reason for this is well-known: the fricative phonemes /v/ and /dh/ have vowel-like structure, and can easily confused with vowels, reducing Recally. Figure 3 shows the resemblance of the fricative phonemes /v/ and /dh/ with the vowel phonemes /ty/ in beer and /ux/ in toot in time- frequency plain.
(a) Spectrogram of /v/
(b) Spectrogram of /iy/
(c) Spectrogram of /dh/
(d) Spectrogram of /ux/
Figure Phonemes in time-frequency plain: /v/ resembles /ty/ and /dh/ resembles /ux/.
Figure demonstrates the effect of this resemblance in the detections of our network: Figure
4a
the posterior probability for fricativity of phoneme /dh/ is not high as that for fricativity of
phoneme /f/. Same problem exists for /v/ as compared to /z/ Figure
dh
pel
iy
kel
m
pel
1.0
1.0
0.5
0.5
0.0
0.0
ch
CT
uw
gel
V
k
ix
ix
(a) Poorly detected /dh/
(b) Poorly detected /v/
Figure Posterior probability produced by our network for phonemes /dh/, /fl, /v/ and /zl.
3.1.4 MAJORITY VOTING
Majority voting the evaluation makes single decision for a phoneme based on all detections for the individual samples constituting that phoneme: if more than half of the samples in the phoneme are classified as fricative by the network, then the phoneme itself classified as fricative.
Majority-voting based poot-proc introduces delay equal to the duration of phoneme mak- ing useless for applications in hearing aids. We applied this post-processing our network to make a direct and fair comparison with Ruinskiy & Lavner 42014). As expected, with the post-processing the performance increased 96. 41% UAR rate. This about 39% decrease in error rate as compared to 94.08% UAR of Ruinskiy & Lavner (2014), a substantial improvement. We that even without any post-processing and evaluated with all 8 fricative phonemes our result (94.76% UAR rate) is still better than those of Ruinskii & Lavner (2014) that does use post- processing and evaluate only with unvoiced fricative phonemes, albeit by much.
the detections of
4 COMPUTATIONAL CONSIDERATIONS
In this paper when we claim that our method introduces zero delay, we make statistical statement: Detection for the current sample is based only on the past signal. So far, we ignored the computa- tional aspect of the problem: Making inference using a neural network takes time for computation. This introduces a certain delay that depends primarily on how powerful the computer is, how large the neural network is, and how optimized the software
On the one hand, the processor in a hearing aid is much weaker than that in a modern powerful GPU, On the other hand, envision installing dedicated low-power chip for neural network-based

---

Under review as a conference paper at ICLR 2020
inference into future hearing aids; modern phones already contain such chips. No matter what the computational infrastructure is, a smaller model always more efficient. The basic model described in consisss of 25 convolutional layers; it is called Net25. We constructed two smaller models as follows. The model Net25h obtained from Net25 by reducing the number of features in each layer by factor of two (h stands for half). The model Net19 is obtained from Net25 by removing the Convolutional Stage 5; details of these networks are given in Figure A.2 the Appendix for reference. Table [ summarizes the detection rates of the three models, the number of parameters in each model, and the processing time per sample for the Keras implementation executed on Nvidia Quadro P P1000 GPU. One can see that Net25h has more than three times fewer parameters as Net25, about 35% faster processing speed, and its UAR rate is about 6% worse than that of Net25.
Table 6: Fricative phoneme detection rates and computational time for different model sizes, Ne125 trained 10 times, Net19 and Net25h are trained times. The reported results for the networks that perform best on the validation set.
MODEL
RECALLF
RECALLN
UAR
#F PARAMETERS
ms SAMPLE
Net25
94.80
94.71
94.76
1.1M
1.25
Net19
92.8T
95.95
94.38
0.7M
1.04
Net25h
94.82
94.03
94.42
0.3M
0.82
Still, there will always a nonzero time that necessary to perform the computation. Since we are targeting zero delay, we must accommodate the extra time needed. A natural solution is to try to extrapolate our detections into the future. This is done in the next section.
EXTRAPOLATING DETECTIONS INTO THE FUTURE
In this section, we use our network to extrapolate fricative phoneme detections into the future. In Figure the green highlighted segment represents the input to the network. The goal of the network is to use this input to calculate the probability that a sample (marked in orange) located a couple of ms in the future, beyond the right boundary of the green segment (marked in red), belongs to fricative phoneme.
To accomplish this goal retrain the network with the data that is relabeled accordingly. For example, the label for the segment highlighted in green Figure 3 would be based on the identity of the orange sample, not on the identity of the red sample as was done previously.
We considered four different values for the extrapolation gap: detecting 1 mss 2ms, 3ms, and 4 ms ahead into the future. This corresponds to the time difference between the orange line (future) and the red line (present time) in Figure
Phonetic codes
ix
z
Amplitude
0.02
4 ms extrapolation gap
0.00
25
50
75
100
125
Time in ms
Figure 5: Input segment in green and sample to be detected in orange. Extrapolation gap is 4 ms.
Except for the relabeling of the training set, the training is exactly the same as before. The inference also unchanged, except that make a detection for the sample at orange line, we fed the net- work the green highlighted segment in Figure 5 Table 7 reports the rates for the fricative phoneme detection as function of the extrapolation gap.
Table 7 clearly shows that the performance decays very gracefully with the increasing extrapolation gap. With the extrapolation gap of 1-2ms we already have enough computational time for inference. Even at 3ms extrapolation gap and solving a harder problem of detecting all 8 fricative phonemes method still outperforms the method of Ruinskiy & Lavner (204)) in UAR.
8

---

Under review as a conference paper at ICLR 2020
Table 7: Rates for fricative phoneme detection as function of extrapolation gap reported in %.
Extrapolation gap
RECALLP
RECALLN
UAR
0 ms
94.80
94.71
94.76
ms
95.04
93.68
94.36
ms
93.25
95,47
94.36
ms
94.33
94.21
94.27
ms
95.01
92.76
93.89
CONCLUSION
In this work, provided state-of-the-ar baseline for fricative phoneme detection and demon- strated that this problem may be solved with zero delay. Further, we demonstrated that it is feasible to extrapolate fricative phoneme detections into the future, creating sufficient computational time for inference. Among other things, these results are important for building the future generation of hearing aids. Our results are fully reproducible and we make the complete source code available. We also include all the trained models that were used in evaluations this paper.
The following interesting research directions remain. Restructure the computational graph of the evaluation pipeline take advantage the seme-series-like structure of the data and avoid redundant computations, Reduce the size of the network to make it more suitable for a low-power device (hear- ing aids). This may be achieved, for example, using deep learning compression techniques: Prun- ing, compression through quantization, knowledge transfer through distillation or teacher-student approach, using memory efficient models such fire module depth wise convolutions.
REFERENCES
Yossi Adi, Joseph Keshet, Olga Dmitrieva, and Matt Goldrick. Automatic measurement of voice onset time and prevoicing using recurrent neural networks. In INTERSPEECH, pp. 3152-3155, 2016.
I-Fan Chen, Sabato Marco Siniscalchi, and Chin-Hui Lee. Attribute based lattice rescoring in spon- taneous speech recognition. In Proc. IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3325-3329, Florence, Italy, 2014.
Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In Proceedings of the 28th International Con- ference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 577-585, 2015.
Joerg Franke, Markus Mueller, Fatima Hamlaoui, Sebastian Stueker, and Alex Waibel. Phoneme boundary detection using deep bidirectional LSTMs. In Speech Communication; 12. ITG Sympo- sium, pp. 377-381. VDE, October 2016.
John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett. Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1. NASA STI/Recon technical report n1 93, 1993.
Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional Istm and other neural network architectures. Neural networks, 18(5-6):602-615 610, 2005.
Andrew King Halberstadt. Heterogeneous acoustic measurements and multiple classifiers for speech recognition. PhD thesis, Massachusetts Institute of Technology. November 1998.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, Las Vegas, USA, 2016.
Paul Michel, Okko Rasanen, Roland Thiollière, and Emmanuel Dupoux. Blind phoneme segmen- tation with temporal prediction error.. In Proceedings of ACL 2017, Student Research Workshop, pp. 62-68, 2017.

---

Under review as a conference paper at ICLR 2020
Dimitri Palaz, Ronan Collobert, and Mathew Magimai-Doss, Estimating phoneme class conditional probabilities from raw speech signal using convolutional neural networks. In INTERSPEECH, 2013.
Okko Johannes Räsänen, Unto Kalervo Laine, and Toomas Altosaar. An improved speech segmen- tation quality measure: the R-value. In INTERSPEECH, pp. 1851-1854, 2009.
Joanna Robinson, Thomas Baer, and Brian CJ Moore. Using transposition to improve consonant discrimination and detection for listeners with severe high-frequeen hearing loss: La utilización de la transposición para mejorar discriminación consonántica y la detección en oyentes con hipoacusia severa para frecuencias agudas. International Journal of Audiology, 46(6):293-308, July 2009.
Dima Ruinskiy and Yizhar Lavner. multistage algorithm for fricative spotting. In Proc. IEEE XXI Annual Pacific Voice Conference (PVC), Krakow, Poland, April 2014. doi: 10.1109/PVC. 2014.6845421.
Sabato Marco Siniscalchi, Torbjørn Svendsen, and Chin-Hui Lee. bottom-up stepwise knowledge- integration approach to large vocabulary continuous speech recognition using weighted finite state machines. In Proc. 12th Annual Conference of the International Speech Communication Associ- ation (INTERSPEECH), pp. 901-904, Florence, Italy, August 2011.
Sabato Marco Siniscalchi, Torbjørn Svendsen, and Chin- Hui--Hui Lee. A bottom-up modular search ap- proach large vocabulary continuous speech recognition. JEEE Transactions on Audio, Speech, and Language Processing, 21(4):786-777, April 2013.
László Tóth. Phone recognition with hierarchical convolutional deep maxout networks. EURASIP Journal on Audio, Speech, and Music Processing, 2015(1):25, September 2015.
Hari Krishna Vydana and Anil Kumar Vuppala. Detection of fricatives using s-transform. Journal of the Acoustical Society of America, 140(5):38963399) 3907, November 2016.
Dong Yu, Sabato Marco Siniscalchi, Li Deng, and Chin-Hui Lee. Boosting attribute and phone estimation accuracies with deep neural networks for detection- based speech recognition. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4169- 4172, Kyoto, Japan, 2012.
Neil Zeghidour, Nicolas Usunier, Iasonas Kokkinos, Thomas Schatz, Gabriel Synnaeve, and Em- manuel Dupoux. Learning filterbanks from raw speech for phoneme recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5509-5513, 2017.
Wei Zhang, Minghao Zhai, Zilong Huang, Chen Liu, Wei Li, and Yi Cao. Towards end-to-end speech recognition with deep multipath convolutional neural networks. In Intelligent Robotics and Applications, pp. 332-341. Springer International Publishing, 2019.
10

---

Under review as a conference paper at ICLR 2020
A
SUPPLEMENTARY DETAILS
Table A.I: Phonemes in Figure
PHONETIC CODE EXAMPLE WORD
/ih/ bit
/tel/ it
/w/
/ix/
/z/
way
debit
zone
Input [3072.1)
Convolution Stage
ooery 64
coav, 64
Convolution Stage
conv, 64
64
copv.64 64
cont, 64
Weight layer
conv, 64
onery 64
Convolution Stage
coev, 64
Weight layer
bodty 64
Batch normulication
80
conv, 80
ReL.
Convolution Stage
oxerv, 801
coav,,
coev, 96
Convolution Stage
96
cont,
96
Globbll Average
Chipui
Figure A.1: Left: Detailed network architecture used the main paper consists of 25 convolutional layers; it is called Net25. The dashed lines show the downsampling for residual connections and Global Average represents 1D Global Averaging. Right: Illustration residual connection; weight layer represents the convolutional layer without non-linearity.
11

---

Under review as a conference paper at ICLR 2020
Apart from the main network displayed in Figure A.1 considered smaller model sizes displayed in Figure A.2
Exput (3072.11
Input [3072.1]
Convolution Stage
32 Mew.weteee.......
Convolution Stage
coay, 48
8 conv, 32
cony, 64
9 conv, 32
coity, 64
8 conv, 32
coity 64
Convolution Stage 2
Convolution Stage
8 conv, 32
X cony, 64
8 conv, 32
coity 64
8 conv, 32
cony, 64
8 conv, 32
8 conv,32 32
cony, 64
8 conv, 32
X conv,
Convolution Stage 3
Convolution Stage
32
8 conv, 32
cony, 64
8 conv, 32
64
10
40
40
Convolution Stage
Convolution Stage
10
9 conv, 40
cotty, 80
10
conv, 48
Global Average
48
Output
48
Convolution Stage 5
48
8 conv, 4%
48
Average
Curput
Figure A.2: Left: Network architecture with half the number of filters in each convolutional layer as Net25; is called Net25h. Right: Network architecture with the same number of filters as in Net25 obtained by removing the Convolutional Stage from Net25 consists of 19 convolutional layers; it is called Net19.
12

---

Under review as a conference paper at ICLR 2020
100
95
90
TP Rate %)
85
80
75
Our result: no post-processing. fricatives
Our result: post-processing. unvoiced fricatives
Our resulf: majority voting, unvoiced fricatives
Ruinskiy & Lavner (2014) Result 1
65
Ruinskiy Lavner (2014) Result
20
40
60
80
100
FP Rate (in %)
Figure A.3: Comparison of our results with Ruinskiy & Lavner 2014) using ROC curves. To obtain the ROC curve, we applied the decision rule in Equation [ with thresholds in the range [0. 1) with increment of 0.01. Note that there are two rates given in the figure for Ruinskiy & Lavner (2014). since they reported two different evaluations. In the main part of the paper we only used Result I of Ruinskiy & Lavner (2014) (in red) in comparisons since it has better UAR than the Result 2 (in dark blue). The zoomed version of ROC curves is on next page.
90.5
90.0
TP Rate (in %6)
89.5
89.0
Our result: no post- post-processing. fricatives
88.5
Our result: no post-processing unvoiced fricatives
Our result: majority voting, unvoiced fricatives
Ruinskiy & Lavner (2014) Result 1
88.0
Ruinskiy & Lavner (2014) Result
FP Rate (in %)
Figure A.4: Zoomed version Figure A.3
Table A.2: Recall, in % fricative phoneme detection task for related approaches. In all these contributions the primary task is generic phoneme identification, but the results are evaluated on the fricative phoneme detection task. Delays given in frames if they were not stated in the original work in seconds. Note that these contributions are not evaluated on the TIMIT dataset.
RELATED WORK
RECALLE
DELAY 5 frames
DATASET
Siniscalchi et al. (011
93.17
Nov92
Yu al. (2012)
96.2
5 frames
Nov92
Siniscalch chi el al.
2013
95.4
155 ms
Nov92
Chen et 2014
88.2
4 frames
NIST 2000 Hub5
13

---

Under review as a conference paper at ICLR 2020
Frame
Frame 2
Frame K
Frame N-1
Frame N
Algorithm
Figure A.5: Detecting phonemes in the middle of larger context. Input context {Frame 1, Frame 2,...: Frame N} the detection algorithm consists of overlapping succes- sive speech segments. The speech segment to be identified is in the middle of the input context.
B DETECTING PHONEME BOUNDARIES
Our networks optimized to solve the two class classification problem the level of samples: does the current sample belong to a fricative phoneme or not? A related problem is to precisely identify the boundaries of fricative phonemes. A good metric to measure the quality of boundary identification has been proposed in Räsänen et al. (2009). The metric is called R-value and it is defined as follows:
Reat = V(1 R)² + OS
Recall 11111-1-000 1-08
v2
Recollys
OS
(2)
Precision,s.
where Recall and Precisionys denote the recall and the precision in identifying the fricative boundary correctly. The boundary identified correctly if it lies no further than threshold, mil- liseconds from a fricative boundary in the ground truth data. Each ground truth boundary may only match single boundary in the data.
Interesting work on identifying phoneme boundaries precisely (not specializing fricative phonemes) has been done in Franke et al. (2016); Michel et al. (2017); Adi et al. (2016). In the table below we report the performance of Net25 on this task for fricative phonemes. We note that
Table B.I: Rates Net25 on detection of fricative boundaries task. The boundaries for the begin- ning and the end of a phoneme are detected and compared to the ground truth separately. We report the average scores over beginning of phoneme/end of phoneme boundaries.
thresholdg, ms
PRECISIONER
RECALLFR
FIFB
R- R-VALER
20
0.58
0.77
0.66
0.60
25
0.61
0.81
0.70
0.63
30
0.63
0.83
0.72
0.64
network trained for this task and the results do not to be state-of-the-art. We leave to future work to identify and remove the dominant sources of errors in this task and to understand how critical the precise boundary detection is for the applications in hearing aid devices.
14


Reviews:
[{"review_id": "rke_KL2hFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper apples supervised deep learning methods to detect exact duration of a fricative phoneme in order to improve practical frequency lowering algorithm. A major challenge compared to existing work is to have an algorithm with nearly zero delay while preserving detection accuracy. A deep convolutional neural network is trained for this purpose and it is validated on TIMIT dataset.\n\nAfter a careful preprocessing of the data, long segments of raw audio are given as input to the convolutional net. It is trained as a binary classification problem. Therefore, for each different phenome, a different network is needed. To improve the accuracy, Majority voting is also adopted. This however increases the computational cost. To address this issue, an extrapolation detection problem is formulated to predict the fricative phoneme a few ms in advance. Extensive numerical results show that the approach still outperforms the method of Ruinskiy & Lavner (2014) in Unweighted Average Recall. \n\nI find the accuracy attained by the neural nets quite impressive, although more insights would be favored to understand what is going on. This is yet an interesting application of deep learning useful for real-life problems. If the method could be tested on another dataset, the result would be more convincing. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1esJzqhFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Updated review\n\nI have read the rebuttal. I have some concerns about the new version of the paper: The addition of section 2.3 about the MFCCs is welcome but feels a bit out of place. The first part about the MFCC is interesting and relevant, but it could be in the introduction as a motivation. The second part about the \"proprietary high-quality time-frequency filterbank\" is not clear at all. Firstly, results are discussed so this part should be in the Evaluation section. Secondly, why using proprietary filterbanks and not the standard Mel filterbanks ?\n\nGiven that the rebuttal and the new version of the paper didn't address my major concerns, I am keeping my original rating.\n\n## Original review\n\nThis paper presents an approach to detect fricative phoneme in speech with as little delay as possible, in the context of hearing aids improvement. The model is based on CNN and is trained to detect fricative given the past context. The model is evaluated in terms of recall and compared with recent published works. The results show that the proposed approach outperforms the baselines and yields state-of-the-art performance with no delay. The paper concludes with some analysis on the computational cost and draw possible future work.\n\nThis paper should be rejected for the following reasons:\n- The novelty is very limited: this work applied a well-known architecture (CNN) to a common problem, phoneme recognition. This only novelty is the zero-delay constraint, which is probably not sufficient for ICLR.\n- The significance is also limited given the very specialized application.  \n- Some references are missing (see below).\n- The presented results are not very clear.\n- The computational considerations section is interesting but is missing some important elements.\n\nDetailed comments:\n- The authors selected raw speech signal as input to the CNN, which is not trivial and should be motivated and discussed in the paper. For instance, using the standard features like Mel filterbanks or MFCC will introduce a delay as they are computed on an overlapping window of 25ms. Phoneme recognition using raw speech as input to a CNN has been presented before, the authors should cite [1] and [2] for instance.\n- Table 4 is confusing as the first four lines are not actually evaluated on TIMIT, so I don't see the point of adding these numbers to the table, as they cannot be compared anyway. I would remove these four lines from the Table.\n- In terms of previous works, phoneme recognition on TIMIT is a very popular task, and many others could be cited, such as [3-5].\n- On the computation consideration, the analysis is interesting, but a discussion on the size (i.e. number of parameter) of the network is missing: one way to decrease computation time is to have a smaller network, which is in line with the application: hearing aids probably do not have gigabytes of ram available. \n- Question about the network: the input segment seems to be of size 3072 samples, why ? any motivation for this particular input size ?\n\nMy review can seem to be a bit harsh, I actually enjoyed the paper, but I don't think ICLR is the right conference for it, and I would advise the authors to improve it and submit it to a speech conference.\n\nReferences:\n[1] Palaz, D., Magimai Doss, M. and Collobert, R.. \"Estimating Phoneme Class Conditional Probabilities from Raw Speech Signal using Convolutional Neural Networks.\" Proceedings of Interspeech 2013.\n[2] Zeghidour, N., Usunier, N., Kokkinos, I., Schaiz, T., Synnaeve, G., & Dupoux, E. \"Learning filterbanks from raw speech for phone recognition\". Proceedings of ICASSP 2018.\n[3] Zhang, Ying, Mohammad Pezeshki, Philémon Brakel, Saizheng Zhang, Cesar Laurent, Yoshua Bengio, and Aaron Courville. \"Towards end-to-end speech recognition with deep convolutional neural networks.\" arXiv preprint arXiv:1701.02720 (2017).\n[4] Chorowski, Jan K., et al. \"Attention-based models for speech recognition.\" Advances in neural information processing systems. 2015.\n[5] Tóth, László. \"Phone recognition with hierarchical convolutional deep maxout networks.\" EURASIP Journal on Audio, Speech, and Music Processing 2015.1 (2015): 25.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1x0i7iTFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "====================================== Updated Review =====================================\nI would like to thank the authors for providing more experiments and details regarding their work.\nHowever, after reading the authors rebuttal, I still think that there is more work to do in terms of comparison to prior work. That way it would be much clearer what is contribution of this work, and how it can be used for future research in that field. \n\nHence, I would like to keep my score as is. \n==========================================================================================\n\nThis paper describes a method for fricative phonemes boundary detection with zero delays. The authors suggest optimizing a convolutional based neural network with a binary cross loss function to detect such events. \nThe authors provide results on the TIMIT dataset and compare the proposed model to several baselines. \n\nThe task of phoneme boundary detection was well studies under different setups and is very important for various applications, including the one proposed in this paper.\n\nHowever, I have some major concerns regarding this paper, which I would like the authors to clarify. Without these, it is hard to understand the contribution in this paper.\n\n1) the authors chose to model the problem from the raw wave. Although it is getting popularity in several speech processing tasks, it is not clear why not using magnitude/MFCC, for example. In case the authors claim that learning from the waveform is better, I suggest providing a comparison to other features.  \nAdditionally, did the authors experience with simpler architectures Maybe more shallow models? Regarding supervision, did the authors tried comparing to the method proposed by [2] but with a unidirectional RNN? Similar to [3].\n\n2)  If I understand it correctly, the motivation for this task was: accurate detection of fricatives boundary can be used to shift into lower frequency bands in hearing aids. It seems like the boundaries are more important than other phoneme parts such as a mid phoneme, for example. \nIn that case, a better metric might be Presicion + Recall + F1 + R-val next to the boundaries (for instance, with a tolerance level of 10-20ms). Those metrics were suggested on several studies of phoneme segmentation, [1], [2]. \n\n3) The comparison in Table 3 is very strange. Results are reported on different datasets. Although the authors mentioned it in the caption, it is still misleading. I suggest the authors to compare either obtain results on the same benchmark or compare to other baselines.\n\nMinor comments: \n\"If for the majority of the samples in a phoneme our network’s output is greater than the threshold we set\" -> not a clear sentence. \n\n[1] Franke, Joerg, et al. \"Phoneme boundary detection using deep bidirectional lstms.\" Speech Communication; 12. ITG Symposium. VDE, 2016.\n[2] Michel, Paul, et al. \"Blind phoneme segmentation with temporal prediction errors.\" arXiv preprint arXiv:1608.00508 (2016).\n[3] Adi, Yossi, et al. \"Automatic Measurement of Voice Onset Time and Prevoicing Using Recurrent Neural Networks.\" INTERSPEECH. 2016.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate the importance of the problem, and one reviewer particularly appreciated the gains in performance. However, two reviewers raised concerns about limited novelty and missing comparisons to prior work. While the rebuttal helped address these concerns, the novelty is still limited. The authors are encouraged to revise the presentation to clarify the novelty.
Decision: Reject