2023 RISP International Workshop on Nonlinear Circuits, Communications and Signal
Processing (RISP NCSP 2023), Feb. 28 - Mar. 3, 2023

Time Series Classification Using Neural Networks
with Chaotic Feature Extraction For Multiple Data
Ryosuke Shimizu, Yoko Uwate and Yoshifumi Nishio
Dept. of Electrical and Electronic Engineering, Tokushima University
2-1 Minami-Josanjima, Tokushima 770-8506, Japan
E-mail: {shimizu, uwate, nisho}@ee.tokushima-u.ac.jp
Abstract

2. Convolutional Neural Network

Time series classification is an important and challenging
problem in data analysis. Recently, time series analysis using neural networks (NN) has attracted much attention. However, the analysis of time series data with complex oscillations is difficult. Therefore, it is important to search for effective features of the data. In this study, we transform thedimensionality of the data and search for features suitable for
NN classification. The effect of the appropriate dimension onclassification accuracy and the characteristics of the data arebeing investigated.
1. Introduction
In recent years, there has been a great deal of research on
the analysis of time series data. This real-world data has various characteristics, such as not only periodic oscillation but
also random oscillation. Among them, chaos theory deals
with data that obey deterministic laws. However, in order toobserve chaos, it is necessary to transform the data. One of
the methods is the time-delay coordinate system, which is akind of chaos theory [1]. Using this method, one dimensional
data (1d-data) can be transformed into multi-dimensional data
called attractors. Many phenomena in the real world are represented by attractors, which can be treated as mathematicalmodels. Attractors are also applied to systems, physical phenomena, and economic phenomena in the living body [2]. In
recent years, there is a lot of research on time series data analysis using neural networks (NN) such as 1-dimensional convolutional neural networks (1d-CNN) and recurrent neuralnetworks (RNN). The advantage of NN is that they can capture features by themselves. However, even with these models, it is difficult to learn numerical and time-series featuresof time series data with very complex oscillations. Therefore, data preprocessing is important in NN. In this research,the data are extended to multi-dimensional data using a timedelay coordinate system and compressed to 1d-data using adimensionality reduction method. The goal is to find featuresthat are easy for NN to learn from time series data with complex oscillations by selecting the optimal dimension.

Convolutional Neural Network (CNN) is one of the NN
and is mainly used in image recognition [3]. In 1982,
Fukushima and Miyake proposed ”New Recognition TRON”,
the predecessor network of CNN [4]. However, CNN are required high computational power. To compensate for theseshortcomings, in 2015, Kiranyaz proposed the first compact and adaptive 1d-CNN that works directly with patientspecific ECG data [5]. Today, 1d-CNN have quickly achieved
state-of-the-art performance levels in several applications
such as biomedical data classification and early diagnosis,
structural health monitoring, anomaly detection and identification in power electronics and electrical [6]-[8]. CNN
is a model with a hierarchical structure that overlaps layers
called convolutional layers, and the accuracy has been improved by deepening this hierarchical structure. In 1998, Le
Cun Laba LeNet-5 has five layers. Categorizing handwrittencharacters using CNN. In 2012, eight-layer AlexNet won theILSVRC image classification competition In the 2014 competition, VGG Net 19. GoogleNet 22 layers have been layered to further improve accuracy. In the 2015 competition,
the Residual Network (ResNet) layered 152 layers and won
the competition of the year. Now ResNet is popular as oneof the standard CNN models [9]. In this study, 1d-ResNet isused for time series classification.

384

Figure 1: An example of CNN structure.

2023 RISP International Workshop on Nonlinear Circuits, Communications and SignalProcessing (RISP NCSP 2023), Feb. 28 - Mar. 3, 2023

3. Dataset

4.1 Attractor construction

The UCR repository [10] data used in this study is theEMG (EMG) of hand movements [11]. The participants wereasked to repeat the following three movements, which areconsidered basic hand movements. Data (a) is an action forgrasping with the palm of the hand. Data (b) are movements
for holding thin and flat objects. Data (c) is for holding acylindrical tool. An example of the EMG time series data forthese three movements is shown in Fig. 2.

Time-delayed coordinate systems are commonly known asdimensional dilation methods by Takens Embedding Theorem. It is a method of dilation of dimensions. Let the valueof the data at a certain time be x(n). Further, let the value oftime delay be τ . This system is represented by Eq. (1) and isshown in Fig. 4 [12]. In this study, we extended the data tothree dimensions.
f (x) = [x(n), x(n + τ ), x(n + 2τ ) ...]

(1)

Figure 4: Time delay coordinate system.
4.2 False Nearest Neighbor
The attractor has arbitrary values of dimension D and timedelay τ . By setting each of these to an appropriate value, it ispossible to extract chaotic features. This study calculates theFigure 2: Three hand movements.
optimal dimension D using False Nearest Neighbor (FNN).
The basic idea underlying the estimation of embedding dimension using FNN was proposed by Kennel. Suppose twodata points in the one-dimensional time series are close to4. Proposed Method
gether then they are neighbors. Their difference in magnitude
In this study, we propose a method of feature translation for provides us with the distance of those neighbors. If we embedusing multidimensional features of time series data. Figure 3 the time series once using some time delay τ , then we can use
shows the flow of the proposed method of this study.
the coordinates of those data points to examine whether thedistance between them has changed appreciably. If embedding changes the distance between the neighbors appreciably,
1. Calculate the optimal dimension.
then they are dubbed false neighbors, and this indicates thatthe data need to be embedded further. Usually, this method
2. Transformed into three dimensional attractor.
requires a threshold, but Cao’s proposed averaging method
does not threshold [13].
3. Compressed to extract the necessary features.
E1(τ ) =E b (τ ) =Figure 3: Procedure of the proposed method.

E a (τ + 1)
E a (τ )N−τ∑
1b(t, τ )N − τ + 1 t=0

(2)

(3)

A point that satisfies either of these two Eq. (2) and Eq. (3)are an FNN. we choose a value for D at the point where thenumber of FNN drops to 0, or subsequent embeddings do
not change the number of FNN or the point before which thenumber of FNN starts to increase again.
385

2023 RISP International Workshop on Nonlinear Circuits, Communications and Signal
Processing (RISP NCSP 2023), Feb. 28 - Mar. 3, 2023

4.3 Dimensionality reduction

Table 1 shows the batch size, epochs, and learning rate inthisstudy. Batch size is the amount of data processed at once.
Dimensionality reduction refers to the reduction of highTheepoch is the number of the learnings. Simulation terdimensional feature vectors to low-dimensional vectors whileminates
when val loss no longer decays during 1500 epochs.
retaining the distribution information of the original dataset.
Thelearning
rate is the degree of learning progress. By setThe procedure of Principal Component Analysis (PCA)
tingasshown
in Table 1, the training accuracies reach 100%.
method is shown in the following steps.
1. Determine the axis that can maximize the variance

Table 1: Learning parameters.

2. Determine the axis of the second that is orthogonal tothe first
3. Get the subspace as eigenvectors by singular value decomposition
4. Compute lower-dimensional embedding1d-ResNet

Batch size
42

Learning rate
0.0001

EpochEarlystopping

6. Simulation Results

The results of the FNN method at τ = 10 and 25 are shown
PCA is one of the most widely used low-dimensional em- in the following fig. 7.
bedding methods. As the name implies, it is an algorithm
that searches for the important element principal component
in the observed data.
5. Verification Structure
In this study, the 1d-ResNet is used for classification
model.
Figure 7: Relationship between FNN ratio and dimension.

Figure 5: 1d-ResNet structure used in this study.
ResNet is commonly used for the classification of image
data. ResNet is the winner network of ILSVRC 2015, and itis the network that enable to have deep layered networks without vanishing gradient problem by using residual learning thatis shown in Fig. 5. By inserting the shortcut connection thatis expressed by the longest arrow over 3 convolutional layers,previous information is learned, and it avoid vanishing gradient problem. In Fig. 6, Convolution (1 × 1, 64) indicatesconvolution layer with 1 × 1 filter and 64 channel.

It is appropriate for the FNN method to deal with dimensions where the FNN ration (FNNR) decrease begins to converge. The figure 7 shows that at τ = 10, dimensions 5 and6 are appropriate, and at τ = 25, dimensions 7 and 8 areappropriate.
The test accuracy is the average of five test accuracies. 1dResNet is used to investigate the accuracy of 3-value classification. Table 2 shows the accuracy of the original data without processing and the test accuracy of the proposed methodwith different dimensions.
Table 2: Test accuracy of the proposed method.
originaldimension
τ = 10τ = 25Figure 6: The structure of residual learning.

test accuracy
0.871
D=3
D=50.849
0.839
0.868
0.905

This is consistent with the appropriate value of 8 dimensions by the FNN method. On the other hand, the appropriate
values at τ =10 are 5 and 6 dimensions, but the accuracy did
not increase. In this study, τ = 25 is considered suitable.

386

2023 RISP International Workshop on Nonlinear Circuits, Communications and Signal
Processing (RISP NCSP 2023), Feb. 28 - Mar. 3, 2023

7. Conclusions[8] I. Mitiche, A. Nesbitt, S. Conner, “Philip Boreham, Gordon Morison, “1D-CNN based real-time fault detection
In this study, we extended time series data to Multi Dimensystem for power asset diagnostics”, Generation Transsional space using time-delay coordinate method and commission Distribution IET, vol. 14, no. 24, pp.5766-5773,
pressed the data. We checked how the accuracy of time series
2020.
classification by 1d-ResNet changes by optimizing the number of parameters during dimensionality expansion. We also [9] K. He, X. Zhang, S. Ren, J. Sun, “Deep residual learning for image recognition”, InProceedings of the IEEE,
compared the accuracy of 1d-ResNet without processing andconference on computer vision and pattern recognition,with the proposed method. The results showed that dimenpp.770-778, 2016.
sionality expansion using the appropriate dimension calculated by FNN extracted important features and improved the[10] H. A. Dau, A. Bagnall, K. Kamgar, C. M. Yeh,accuracy of the test accuracy. In addition, by appropriately
Y. Zhu, S. Gharghabi, C. A. Ratanamahatana, E.Keogh,
compressing the higher dimensional features of the attractors,“The UCR Time Series Archive”, IEEE/CAA J. Autom.
the features could be used in low dimensional classification.
Sinica, vol. 6, no. 6, pp.1293-1305, Nov. 2019.
We will use these results to investigate whether they are valid
for other data as well. We will also use models such as RNN [11] C. Sapsanis, G. Georgoulas, A. Tzes and D. Lymthat are specialized for analyzing the flow of time series orberopoulos, “Improving EMG based Classification of baconstruct optimal extended dimensions.
sic hand movements using EMD”, 2013 35th Annual
International Conference of the IEEE Engineering inMedicine and Biology Society (EMBC), Osaka, Japan,
References
pp.5754-5757, 2013.
[1] F. Takens,“Detectingstrange attractors intur bulance”, [12] K. Aihara, T. Ikeguchi, Y. Yamada and M. Komuro,LectureNotes in Mathematics, Vol.898(Springer“Fundamentals and applications of chaotic time seriesVerlag.), pp.366-381, 1981.
analysis”, Industrial book, p.23, 2000.
[2] C. Oestreicher,“A history of chaos theory”, doi: [13] C. Li, “Practical method for determining the minimum
10.31887/DCNS.2007.9.3/coestreicher, Dialogues Clinembedding dimension of a scalar time series”, PVLD
Neurosci, pp.279289, 2007.
110, pp.43-50, 1997.
[3] M. Matsugu, K. Mori, Y. Mitari and Y. Kaneda, “Subject independent facial expression recognition with robust face detection using a convolutional neural network”, Neural Networks16, vol.5, pp.555-559, 2003.
[4] K. Fukushima, S. Miyake,“A new algorithm for patternrecognition tolerant of deformations and shifts in position”, Pattern Recognition, Neocognitron15, pp.455-469,
1982.
[5] S. Kiranyaz, T. Ince, R. Hamila, M. Gabbouj, “Convolutional Neural Networks for patient specific ECG classification”, in Proc. Annu. Int. Conf. IEEE Eng. Med. Biol.
Soc. EMBS, pp.2608-2611, 2015.
[6] S. R. Bajare, V. V. Ingale, “ECG Based Biometricfor Human Identification using Convolutional NeuralNetwork”, Computing Communication and NetworkingTechnologies (ICCCNT) 2019 10th International Conference on, pp.1-7, 2019.
[7] P. Sharma, S. Chandan, B. P. Agrawal, “Vibration Signalbased Diagnosis of Defect Embedded in Outer Race ofBall Bearing using 1-D CNN”, Computational Performance Evaluation (ComPE) 2020 International Conference on, pp.531-536, 2020.
387