2022 International Symposium on Nonlinear Theory and Its Applications,NOLTA2022, Virtual, December 12-15, 2022

Time Series Classification by Neural Network Using Features of Attractors
after Smoothing Process
Ryosuke Shimizu† , Yoko Uwate‡ , and Yoshifumi Nishio⋆†Dept. of Electrical and Electronic Engineering, Tokushima University
2-1 Minami-Josanjima, Tokushima 770–8506, Japan
Email: shimizu@ee.tokushima-u.ac.jp, uwate@ee.tokushima-u.ac.jp, nishio@ee.tokushima-u.ac.jp
Abstract—Time series classification is an important andchallenging problem in data analysis. Recently, time series
analysis using neural networks (NN) has attracted muchattention. However, the analysis of time series data withcomplex oscillations is diﬃcult. Therefore, it is importantto search for eﬀective features of the data. In this study,
we transform the dimensionality of the data and search forfeatures suitable for NN classification. In this study, we investigate the eﬀect of smoothed data on the classification
accuracy of the dimensionality reduction method and thefeatures of the data.
1. Introduction
In recent years, there has been a great deal of researchon the analysis of time series data. This real-world data hasvarious characteristics, such as not only periodic oscillation
but also random oscillation. Among them, chaos theory
deals with data that obey deterministic laws. However, inorder to observe chaos, it is necessary to transform the data.
One of the methods is the time-delay coordinate system,
which is a kind of chaos theory [1]. Using this method,
one dimensional data (1d data) can be transformed into
multi-dimensional data called attractors. Many phenomenain the real world are represented by attractors, which canbe treated as mathematical models. Attractors are also applied to systems, physical phenomena, and economic phenomena in the living body [2]. In recent years, there isa lot of research on time series data analysis using neuralnetworks (NN) such as 1-dimensional convolutional neuralnetworks (1d-CNN) and recurrent neural networks (RNN).
The advantage of NN is that they can capture features bythemselves. However, even with these models, it is diﬃcult
to learn numerical and time-series features of time series
data with very complex oscillations. Therefore, data preprocessing is important in NN. In this study, to solve thisproblem, this research firstly performs optimal noise reduction of biological signals. Next, the data is extended to 3D
data using a time-delay coordinate system and the dimensional reduction methods to compress the data into 1D data.
The objective is to find features that can be easily learned
by NN from time series data with complex oscillations.
ORCID iDs Ryosuke Shimizu:
0000-0001-9684-7334, Yoko Uwate:0000-0002-2992-8852, Yoshifumi Nishio:
0000-0002-0247-0001

2. Convolutional Neural Network
Convolutional Neural Network (CNN) is one of theNN and is mainly used in image recognition [3]. In
1982, Fukushima and Miyake proposed ”New Recognition
TRON”, the predecessor network of CNN [4]. However,
CNN are required high computational power. To compensate for these shortcomings, in 2015, Kiranyaz proposed
the first compact and adaptive 1d-CNN that works directlywith patient-specific ECG data [5]. Today, 1d-CNN havequickly achieved state-of-the-art performance levels in several applications such as biomedical data classification andearly diagnosis, structural health monitoring, anomaly detection and identification in power electronics and electrical [6]-[8]. CNN is a model with a hierarchical structurethat overlaps layers called convolutional layers, and the accuracy has been improved by deepening this hierarchicalstructure. In 1998, Le Cun Laba LeNet-5 has five layers. Categorizing handwritten characters using CNN. In
2012, eight-layer AlexNet won the ILSVRC image classification competition In the 2014 competition, VGG Net
19. GoogleNet 22 layers have been layered to further improve accuracy. In the 2015 competition, the Residual Network (ResNet) layered 152 layers and won the competitionof the year. Now ResNet is popular as one of the standard
CNN models [9]. In this study, 1d-ResNet is used for timeseries classification.

Figure 1: An example of CNN structure.

3. Dataset
The UCR repository [10] data used in this study is theEMG (EMG) of hand movements. The participants wereasked to repeat the following six movements, which areconsidered basic hand movements. Data (a) is the movement to hold a spherical tool. Data (b) is for holding asmall tool. Data (c) is an action for grasping with the palm

This work is licensed under a Creative Commons Attribution NonCommercial, No Derivatives 4.0 License.

– 638 –

of the hand. Data (d) are movements for holding thin andflat objects. Data (e) is for holding a cylindrical tool. Data(f) is an action to support a heavy object. An example ofthe EMG time series data for these six movements is shownin (Fig. 2).
1007060

80

5060

4030

40202010000200

400

600

800

1000

1200

1400

0200

400600

800

1000

1200

locally-weighted scatterplot smoother. The routine selects
a fixed percentage of all points using the data that are closest neighbors in x-values on either side of the (x, y) point. Aweighted linear regression is performed on each data point.
This will give the point closest to each x-value the greatest
weight in smoothing, limiting the eﬀect of outliers. Parameters can be specified to change the degree of smoothing
and influence from outliers. The weights of the smoothing
parameters can also be specified. The larger the weight, themore the smoothed value follows the data; the smaller theweight, the smoother the pattern of smoothed values.

1400(b) Attractor construction(a) hand movement
100(b) hand movement

Time-delayed coordinate systems are commonly known
as dimensional dilation methods by Takens Embedding
Theorem. It is a method of dilation of dimensions. Let
the value of the data at a certain time be x(n). Further, let
the value of time delay be τ. This system is represented
by Eq. (1) and is shown in Fig. 4 [13]. In this study, weextended the data to three dimensions.

608050604030402020100002004006008001000

1200

14000(c) hand movement

200400600

800

1000

1200

1400(d) hand movement

f (x) = [x(n), x(n + τ), x(n + 2τ) ...]

350(1)

300

2000

250

1500

200
150

1000

10050050000200

400600

800

1000

1200

1400(e) hand movement

0200

400600

800

1000

1200

1400(f) hand movement

Figure 2: Six hand movements

Figure 4: Time delay coordinate system.

4. Proposed method
In this study, we propose a method of feature translation for using multidimensional features of time series data.
Figure 3 shows the flow of the proposed method of thisstudy.

(c) Dimensionality reduction

2. Transformed into three dimensional attractor.

Dimensionality reduction refers to the reduction of highdimensional feature vectors to low-dimensional vectors
while retaining the distribution information of the original dataset. The procedure of Principal Component Analysis (PCA) method is shown in the following steps.

3. Compressed to extract the necessary features.

1. Determine the axis that can maximize the variance

1. The data is smoothed for feature extraction.

2. Determine the axis of the second that is orthogonal tothe first
3. Get the subspace as eigenvectors by singular value decomposition
4. Compute lower-dimensional embedding

Figure 3: 1d-ResNet structure used in this study.
(a) The LOWESS smoothing method
The LOWESS smoothing method is a general methodfor determining smoothing lines. LOWESS stands for

PCA is one of the most widely used low-dimensional embedding methods. As the name implies, it is an algorithm
that searches for the important element principal component in the observed data.

– 639 –

Table 2: Test accuracy of the proposed method.

5. Simulation ModelIn this study, the 1d-ResNet is used for classification
model.

Figure 5: 1d-ResNet structure used in this study.
ResNet is commonly used for the classification of image data. ResNet is the winner network of ILSVRC 2015,
and it is the network that enable to have deep layered networks without vanishing gradient problem by using residual learning that is shown in Fig. 5. By inserting the shortcut connection that is expressed by the longest arrow over 3
convolutional layers, previous informa- tion is learned, andit avoid vanishing gradient problem. In Fig. 6, Convolution(1 × 1, 128) indicates convolution layer with 1 × 1 filterand 128 channel.

PCA(not smoothed)
PCA(p = 10)
PCA(p = 50)
PCA(p = 100)

test accuracy
0.836
0.819
0.861
0.840

The value p is the range parameter of the subset when
performing weighted linear least squares regression inlowess regression. Optimizing this value improved the accuracy. The best accuracy was obtained when p = 50 in thelowess regression. When p = 10, the accuracy was lower
than the accuracy when the smooth process was not used.
Figure 8 shows the image of attractor without the smoothprocess. Figures 9-11 are images of the attractor composedof waveform (a) in Fig. 2 before dimensionality reduction.
The attractors are characteristic, and it is thought that thesmooth process has brought out the characteristics.

Figure 7: not smoothed Attractor.

Figure 6: The structure of residual learning.
Table I shows the batch size, epochs, and learning ratein this study. Batch size is the amount of data processedat once. The epoch is the number of the learnings. Simulation terminates when val loss no longer decays during
1500 epochs. The learning rate is the degree of learning
progress. By setting as shown in Table I, the training accuracies reach 100%.

Figure 8: Attractor at p = 10.

Table 1: Learning parameters.
1d-ResNet

Batch size
42

Learning rate
0.0001Epoch
Earlystopping

6. Simulation results
We investigate the average of 5 times of test accuracy.
1d-ResNet is used to investigate the accuracy of 6-value
classification. Table II shows the accuracy of NN with andwithout the smooth process of the proposed method wascompared.

– 640 –

Figure 9: Attractor at p = 50.

and Networking Technologies (ICCCNT) 2019 10th
International Conference on, pp. 1-7, 2019.
[7] Pragya Sharma, Swet Chandan, B. P. Agrawal, “Vibration Signal-based Diagnosis of Defect Embedded inOuter Race of Ball Bearing using 1-D CNN”, Computational Performance Evaluation (ComPE) 2020 International Conference on, pp. 531-536, 2020.
[8] Imene Mitiche, Alan Nesbitt, Stephen Conner, Philip
Boreham, Gordon Morison, “1D-CNN based real-timefault detection system for power asset diagnostics”,
Generation Transmission Distribution IET, vol. 14, no.
24, pp. 5766-5773, 2020.

Figure 10: Attractor at p = 100.

7. Conclusion
In this study, we extended smoothed time series data to3D space using time-delay coordinate method and compressed the data. we checked how the accuracy of timeseries classification by 1d-ResNet changes by optimizing
the number of subset parameter during smoothing. TheNN accuracy was also compared between the cases without the smoothing process and with the proposed method.
The results show that the accuracy of the test Smoothingwith appropriate values allowed the noisy data to perform
feature extraction well. Image of the the attractors of theproposed method were found to be distinctive. We will use
these results to investigate whether they are valid for otherdata as well. We will also use models such as RNN thatare specialized for analyzing the flow of time series or construct optimal extended dimensions.
References[1] F.Takens, Detectingstrange attractors intur bulance,LectureNotes in Mathematics, Vol.898(SpringerVerlag. 1981) 366-381.
[2] Christian Oestreicher,“A history of chaos theory”, doi:
10.31887/DCNS.2007.9.3/coestreicher.
[3] M. Matsugu, K. Mori, Y. Mitari and Y. Kaneda, “Subject independent facial expression recognition with robust face detection using a convolutional neural network”, Neural Networks 16., vol.5, pp.555-559, 2003.

[9] He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep residual learning for image recognition. InProceedings ofthe IEEE. conference on computer vision and patternrecognition(pp. 770-778).
[10] Yanping Chen, Eamonn Keogh, Bing Hu, Nurjahan
Begum, Anthony Bagnall, Abdullah Mueen and Gustavo Batista (2015). The UCR Time Series Classification Archive.
[11] E. F. Delagi, A. O. Perotto, J. Iazzetti, and D. Morrison, Anatomical guide for the Electromyographer: TheLimbs and Trunk, Charles C Thomas Pub Ltd, 5ht ed.,2011.
[12] T. S. Saponas, D. S. Tan, D. Morris, R. Balakrishnan, J. Turner, and J A. Landay, Enabling AlwaysAvailable Input with Muscle-Computer Interfaces´ in
22nd annual ACM symposium on User interface software and technology, Association for Computing Machinery, New York, USA, pp 167-176, 2009.
[13] K. He, X. Zhang, S. Ren and J. Sun, “Delving Deep
into Rectifiers: Surpassing Human-Level Performanceon ImageNet Classification , IEEE International Conference on Computer Vision and Pattern Recognition,
arXiv:1502.01852, 6 Feb 2015.
[14] K. Aihara, T. Ikeguchi, Y. Yamada and M. Komuro,Fundamentals and applications of chaotic time series
analysis , Industrial book, p.23, 2000

[4] K. Fukushima, S. Miyake, Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position, Pattern Recognit. 15 (1982)
455469. doi 10.1016/0031-3203(82)90024-3.
[5] S. Kiranyaz, T. Ince, R. Hamila, M. Gabbouj,
Convolutional Neural Networks for patient specific ECG classification, in: Proc. Annu. Int.
Conf. IEEE Eng. Med. Biol. Soc. EMBS, 2015.
doi:10.1109/EMBC.2015.7318926.
[6] Shraddha R. Bajare, Vaishali V. Ingale, “ECG Based
Biometric for Human Identification using Convolutional Neural Network”, Computing Communication– 641 –