2020 RISP International Workshop on Nonlinear Circuits,Communications and Signal Processing (NCSP 2020)
Honolulu, Hawaii, USA, February 28 - March 2, 2020

Voice Recognition Using Surrogate Data Methodwith Neural Network
Tomiyuki Furugaki, Tomoya Takata, Yoko Uwate, and Yoshifumi Nishio
Dept. of Electrical and Electronic Engineering, Tokushima University
2-1 Minami-Josanjima, Tokushima 770-8506, Japan
Email: {furugaki, tomoya, uwate, nishio}@ee.tokushima-u.ac.jp

Abstract
Voices are often classified by using Recurrent Neural Network and 1-Dimensional Convolutional Neural Network. In
general, Recurrent Neural Network and 1-Dimensional Convolutional Neural Network learn original data. In this study,
original data is replaced with surrogate data. Test accuracies
at that time are compared. In this way, we search which partof time series data is effective for using Recurrent Neural Network and 1-Dimensional Convolutional Neural Network.
1. Introduction
Neural Network (NN) is a system model on neurons of thehuman brain nervous system. Among them, Recurrent Neural Network (RNN) and 1-Dimensional Convolutional Neural
Network (1D-CNN) are used for voice classification [1]-[2].
They need to learn voice waveform in advance to classify
voice. There are various characteristics depending on eachsound source. They classify voice by finding them. However, the specific judgment part is unknown. Therefore, it isnecessary to find these.
In this study, surrogate data method is used. The surrogate data method is creating surrogate data. Surrogate data isdata that preserves some of the statistical properties of timeseries data and destroys other properties. After that, it is indicated that there is a significant difference between the statistical properties of time series data and the surrogate data.
In this way, the method proves the importance of destroyedproperties.
In this study, the data learned by RNN and 1D-CNN arereplaced from original data with surrogate data. It can befound out which part of the voice waveform is important bycomparing the test accuracy at that time.

The network structure of NN is divided into an input layer,
an intermediate layer and an output layer. The intermediate
layer of CNN includes convolution layers, pooling layers andfully connected layers. Features of inputs are extracted in theconvolution layer, and position invariance is acquired in thepooling layer. Next, it becomes the 1-Dimensional array infully connected layers and it changes to probability. Finally,CNN outputs classification results by the probability. However, in recent years, CNN has also been use to time series
data. In this study, CNN is used for time series data that is
one-dimensional data to classify voice.
3. Surrogate Data Method
The surrogate data method was proposed in 1992 for chaostime series analysis [7]. There are no necessary and sufficientconditions for chaos. Therefore, the only way to determine
chaos is to find out that there is chaoticity. In many cases,chaos is determined by spectral continuity, strange attractors,
Lyapunov exponents, bifurcations, and so on. However, it
has been pointed out that even with random noise alone. The
Lyapunov exponent is positive and noise and chaos cannot bedistinguished. Therefore, the surrogate data method is proposed to test whether it is noise or data generated from a deterministic system. With hypothesis testing, it is difficult tosay that it is noise if a data passes the test. However, it cannot
be asserted that it is chaos because the surrogate data method
is based on hypothesis testing in statistics [8]-[10]. In thisstudy, surrogate data is created and compared the accuracy oflearning surrogate data with the accuracy of learning original data. In this way, what characteristics of original data areimportant can be found.

4. Dataset
2. Neural Network
In this study, time series data of voices are classified. Three
The research on NN was established as an academic field in types of voice are prepared. In this study, they are called
1958 [3]. Since then, it has repeated the ice ages and booms voices 1, 2 and 3. Forty pieces of data for 6 second each aremany times and now reaches the present. Currently, NN is prepared. Each time series data is sampled at a sampling freused in medical field, car field, home electronics field and so quency 3000 [Hz]. Next, the data is augmentationed. Thereon [4]-[6]. NN has two famous classification models. They are three types of augmentation. Figure 1 shows the examare RNN and CNN. RNN is used in fields related to time se- ples of original time series data. Figures 2, 3 and 4 show theries data. CNN is used in fields related to image recognition. examples of the time series data about the data is added white307

noise, time shift and time stretch for augmentation.

5. Proposed Method
Four types of surrogate data are created. Surrogate data is a
destruction of some information of original data. The following explanations (a), (b) , (c) and (d) describe how to create
four types of surrogate data.
(a) Random Shuffle Surrogate Data (RSSD)
x(n) means time function. n means time. It is RSSD datathat changes the order of n at random. The correlation of original data is broken by converting the data into RSSD. Figure
5 shows RSSD of original data in Fig. 1.

Figure 1: Original data

Figure 5: RSSD(b) Fourier Transform Surrogates Data (FTSD)

Figure 2: White noise datax(n) =n∑x(n)e−i N

(1)

2πkn
1X(ω)ei N
N n=1(2)X(ω) =2πkn

n=1n∑Equations (1) and (2) show discrete Fourier Transform(DFT) and Inverse Discrete Fourier Transform (IDFT). kmeans frequency. N (= 15000) means the number of the samples.
Step 1. Calculate DFT X(ω) of x(n).
Step 2. Randomize the phase of X(ω).
Step 3. Calculate IDFT randomized X(ω).
Figure 3: Time shift data

FTSD is made in this way. The frequency distribution of original data is broken by converting the data into FTSD. Figure
6 shows FTSD of original data in Fig. 1.

Figure 4: Time stretch data

Figure 6: FTSD

308

(c) Amplitude Adjusted Fourier Transform Surrogates 6. Architecture
Data (AAFTSD)
RNN and 1D-CNN are used for convention architecture.
Step 1. Prepare random numbers R(n) according to the Figures 8, 9 show the structure of RNN and 1D-CNN westandard normal distribution.
used.
Step 2. Sorting R(n) in the same size relation as x(n).
Step 3. Create R′ (n) which is FTSD of sorted R(n).
Step 4. Sorting x(n) in the same size relation as R′ (n).
AAFTSD is made in this way. The correlation of original
data is broken by converting the data into AAFTSD. However
AAFTSD has similar correlation than that of RSSD. Figure 7
shows AAFTSD of original data in Fig. 1.

Figure 9: structure of RNN

Figure 7: AAFTSD(d) Iterated Amplitude Adjusted Fourier Transform Surrogates Data (IAAFTSD)
Step 1. Prepare s(0) which is RSSD of original data as theinitial value.
Step 2. Calculate DFT S (i) of s(i) .
Step 3. Replace amplitude of S (i) with amplitude of origi(i)nal. Put it as S(i)Step 4. Calculate IDFT s(i) of S .
Step 5. Sorting s(i) in the same size relation as original data.
Step 6. Add 1 to i.
Step 7. Repeat until i = 7.
IAAFTSD is made in this way. IAAFTSD saves the frequency distribution and has similar correlation than that ofAAFTSD. Figure 8 shows IAAFTSD of original data inFig. 1.

Figure 10: structure of 1D-CNN

RNN has 1024 nodes in the intermediate layer. RNN canstore past information.
Two convolutional layers and two maxpooling layers wereused for 1D-CNN. Drop out layer was prepared to prevent
over learning. We derived the probability to calculate classification results by the softmax activation function. Equation(3) shows softmax function.

exp(xj )ρ(x) = ∑ni=1 exp(xi )(3)

ρ(x) is the probability of being classified as j. n is the total
number of classes.

Figure 8: IAAFTSD

7. Simulation Results
Forty pieces of data are prepared. Each data was augmentationed to three types. After sampling them, surrogate data iscreated. Furthermore, each data is divided into three pieces.
In this way, the number of data became four hundred. Table 1
shows the number of the train data and test data. Table 2 and3 show the results of our research.

309

[2] Abhishek Sehgal, Nasser Kehtarnavaz, ”A Convolutional Neural Network Smartphone App for Real-Time
Voice Activity Detection”, Access IEEE, vol. 6, pp.
9017-9026, 2018.

Table 1: The number of the train data and test data
voice 1voice 2
voice 3train data
360
360
360

test data909090[3] F. Rosenblatt,“The Perceptron：a Probabilistic Modelfor Information Storage and Organization in the Brain”,
Psychol. Rev, Vol. 65, no. 6, pp. 386-408, 1958.

Table 2: Test accuracy of RNN
original data
RSSD
FTSD
AAFTSD
IAAFTSD[4] Shih-Chung B. Lo, 1 Heang-Ping Chan, 2 Jyh-Shyan
Lin, 1 Huai Li, 1 Matthew T. Freedman and Seong
K. Mun 1, “Artificial Convolution Neural Network for
Medical Image Pattern Recognition”, Neural Networks,
Vol. 8, No. 7/8, pp. 1201-1214, 1995.

test accuracy [%]
74.1
43.6
39.5
42.5
45.0

[5] S. Draghici. “A neural network-based artificial visionsystem for license plate recognition”. Int. J. Neural Syst.
January, pp. 113-126, 1997.

Table 3: Test accuracy of 1D-CNN
original data
RSSD
FTSD
AAFTSD
IAAFTSD

test accuracy [%]
87.9
56.9
75.9
57.2
78.7[6] Ameena Saad al-sumaiti, Mohammed Hassan Ahmed
Magdy M. A. Salama. “Smart Home Activities: A Literature Review”. Electric Power Components and Systems, pp. 294-305, Februaly 2013.
[7] J. Theiler, S. Eubank, A. Longtin, B. Galdrikian, and J.
D.Farmer, Physica (Amsterdam) 58D, pp. 77 1992.

We investigate the average of ten times of test accuracy. [8] James Theiler, Stephen Eubank, Andre Longtin, Bryan
Galdrikian, and J. Doyne Farmer. “Testing for nonlinTables 2 and 3 show that test accuracies of surrogate data
earity in time series: the method of surrogate data”
are lower than test accuracies of original time series data.
Physica D, Vol. 58, pp. 77-94, 1992.
The decay rate of test accuracy of RNN of RSSD is lower
than that of FTSD. The decay rate of test accuracy of 1D[9] Dean Prichard and James Theiler. “Generating SurroCNN of FTSD is lower than that of RSSD. Test accuracies
gate Data for with Several Simultaneously Measuredof IAAFTSD of RNN and 1D-CNN are higher than those ofVariables”. Physical Review Letters, Vol. 73, No. 7, pp.
the other surrogate data. FTSD did not store frequency dis951-954, August 1994.
tribution of original data. RSSD did not store correlation oforiginal data. Therefore, it was understandable that correla- [10] Thomas Schreiber and Andreas Schmitz. “Improved
Surrogate Data for Nonlin- earity Tests”. Physical Retion is more important than frequency distribution for RNN.
view Letters, Vol. 77, No. 4, pp. 635-638, 1996.
On the other hand, it was understandable that frequency distribution is more important than correlation for 1D-CNN.
8. Conclusion
In this study, three types classification were carried out
with surrogate data. Then, we understood that correlation ismore important than frequency distribution for RNN. On theother hand, it was understandable that frequency distribution
is more important than correlation for 1D-CNN.
In the future, we will find that relationship between similarity of correlation or frequency distribution and test accuracy.
References[1] Gelly, Gregory / Gauvain, Jean-Luc, “Minimum word
error training of RNN-based voice activity detection”,
In Interspeech-2015, pp. 2650-2654.

310