Journal of Signal Processing, Vol.23, No.4, pp.145-149, July 2019

SELECTED PAPER AT NCSP'19

Application of a Selective Desensitization Neural Network to Concept Drift Problems
Tomoki Ichiba1 , Kazumasa Horie2 , Shoichi Someno1 , Tomohiro Aki1 and Masahiko Morita3
Graduate School ofSystems and Information Engineering,
University of Tsukuba
1–1–1 Tennodai, Tsukuba,
Ibaraki 305–8573, Japan
E-mail: ichiba@bcl.esys.tsukuba.ac.jp
someno@bcl.esys.tsukuba.ac.jp
aki@bcl.esys.tsukuba.ac.jp
12

Center for Computational Sciences,
University of Tsukuba
1–1–1 Tennodai, Tsukuba,
Ibaraki 305–8577, Japan
E-mail: horie@bipl-sdnn.org

Faculty of Engineering,Information and Systems,
University of Tsukuba
1–1–1 Tennodai, Tsukuba,
Ibaraki 305–8573, Japan
E-mail: mor@bcl.esys.tsukuba.ac.jp
3

Abstract
A selective desensitization neural network (SDNN) has a highfunction-approximation ability, low hyperparameter dependence, and suitability for online incremental learning. These
properties suggest that an SDNN can deal well with temporal changes in the characteristics of data, or concept drift, although this has not been verified. In this study, we conducted
experiments on online learning using an artificial dataset generated using a time-varying function and a real-world datasetof a stock prices index, and evaluated the effectiveness of anSDNN to solve concept drift problems. The results show thatusing the SDNN exhibited superior performance over the existing methods for both datasets, suggesting that an SDNN ishighly suitable for certain types of concept drift problems.
1. Introduction
Concept drift is a problem in which the data characteristics, such as the input-output relation and the input distribution, change over time. Examples of data with concept drift
are stock prices, which have often been used as real-world
datasets in previous studies [1, 2]. We focus on online prediction of concept drift problems using machine learning.
Conventional learning algorithms have a certain difficulty
in dealing with concept drift, in that they require hyperparameter optimization, or are unsuitable for online incrementallearning. Recent studies [3, 4] have shown that a selective desensitization neural network (SDNN) has high expression andgeneralization abilities and low hyperparameter dependence.
In addition, we found that an SDNN can fit additional data
without significantly disrupting previously learned information, indicating that it is suitable for online incremental learning, and that it may deal well with concept drift. However, anSDNN has not been applied to concept drift problems to test
its performance.
In the present study, we conducted experiments on online

Journal of Signal Processing, Vol. 23, No. 4, July 2019

Figure 1: Structure of SDNN
prediction and evaluated the effectiveness of an SDNN for
dealing with concept drift.
2. Selective Desensitization Neural Network
An SDNN function approximator is constructed by applying manipulations of pattern coding and selective desensitization to a parallel perceptron (PP), which is the only part thatan SDNN trains. The structure of an SDNN is shown in Fig.
1. Details of pattern coding, selective desensitization, and a
PP are described in this section.
2.1 Pattern coding
Pattern coding converts each analog value into binary (±1)
n-dimensional vectors as code patterns. In pattern coding,
Q code patterns, which correspond to an input space divided
into Q, are created. There are several methods of creatingcode patterns. We explain our method below.

145

First, we create P1 , the first code pattern, by selecting +1
and −1 randomly for each element, such that half of the elements take positive values and the rest take negative values.
Second, we create P2 by inverting the sign of r elements of+1 and r elements of −1 of P1 , which are selected randomly.
Subsequently, we repeat this process to create Pk based on
Pk−1 until P1 to PQ are created.
The results of this method ensure that the correlation between two consecutive patterns is high, and that the correlation between two patterns decreases as the two patterns are
increasingly separated.

(a) 1 ≤ t ≤ 1000(b) t = 1500(c) t = 2000

2.2 Selective desensitization
Selective desensitization integrates two binary code patterns into a single ternary (−1, 0, 1) pattern by modifying one(d) t = 2500(e) t = 2999
(f) 3000 ≤ t ≤ 7000
pattern with the other.
As an example, we consider the case of desensitizing thepattern S = (s1 , ..., sn ) with the pattern C = (c1 , ..., cn ). In
Figure 2: Target function at a certain time step t
desensitization, it is assumed that there is a random one-to3.1 Methods
one correspondence between the elements of S and C. Whencorrespondence exists between si and cj , si is desensitized
An artificial dataset was generated by sampling an artificial
through the following expression.
time-varying function f (x, y; t).
s′i =si (1 + cj )2(1)When cj = −1, si is desensitized, and s′i becomes a neutral
value (= 0). When cj = 1, s′i becomes the value of si . As aresult, half of the desensitized pattern elements are 0, and therest are 1 or −1.
2.3 Parallel perceptron
A PP consists of m simple perceptrons (SPs) with a Heaviside function as an activation function. The output of the PP
is obtained from the number of SPs that output a value of 1.
The p-delta method [5] is used for learning.
As an example, we consider a case in which only k SPs
output 1, although l(> k) SPs should output 1 for the input.
At this time, we apply error correction learning for l − k SPs,
which are selected from the m − k SPs that output 0 in theorder of increasing closeness of the internal potential to thethreshold value (= 0). In the case of k > l, the same error
correction learning is conducted for k − l SPs selected fromthe m − k SPs that output 1.
3. Experiment with an Artificial Dataset
We compared the SDNN and other methods using an artificial dataset. Concept drift is categorized into several patterns,
such as sudden drift, incremental drift, and reoccurring drift[1]. In this experiment, we focus on both sudden and incremental drift.

146


1
(g(x, y; t) > 1)

0(g(x, y; t) < 0)f (x, y; t) = g(x, y; t)(otherwise)k(x, y)(t < 1000)





k(x, y) − t−1000(1000 ≤ t < 1500)

1500


(1500 ≤ t < 2500)
k(x, y) − 2000−t
1500g(x, y; t) =
t−3000 k(x, y) −(2500 ≤ t < 3000)
1500



 l(x, y)(t ≥ 3000)0.5(0.7 ≤ x ≤ 0.95, 0.1 ≤ y ≤ 0.9)221((x − 0.25) + (y − 0.75) < 0.04)
l(x, y) =√ 1+x cos(5π xy 2 ) + 1(otherwise)
42√
11+xcos(2π xy 2 ) +k(x, y) =42(2)The value of this function gradually decreases during1000 < t ≤ 1500, increases during 1500 < t ≤ 2500,and again decreases during 2500 < t ≤ 2999 to the initialvalue (Fig. 2(a)–2(e)), which can be regarded as incremental
drift. It then changes abruptly at t = 3000 (sudden drift); thespatial frequency of the function increases and discontinuous
domains appear (Fig. 2(f)).
At each time step, one sample was obtained randomly fromthe lattice points at 0.01 intervals within the input domain

Journal of Signal Processing, Vol. 23, No. 4, July 2019

Table 1: Average approximation error over all time steps
MLP

RBFN 441

RBFN 81

INGnet

SDNN

0.220

0.064

0.071

0.055

0.047

{(x, y) | x, y ∈ [0, 1]} and was given to each function approximator only once for training (and discarded at the nexttime step). The mean absolute error was then calculated fromthe approximation error at all (101 × 101) lattice points. The
total number of time steps was 7,000. For comparison, wetested other function approximators: a multilayer perceptron(MLP), a radial basis function network (RBFN) [6], and anincremental normalized Gaussian network (INGnet) [7].
MLP One hidden layer of 50 units was applied between
the input and output layers. For the hidden layer, the hyperbolic tangent (tanh) was used as the activation function,
whereas a linear function was used for the output layer. Training was conducted using a backpropagation algorithm. We
repeated the above experiment ten times using different initial synaptic weights.
RBFN We set the lattice points at s intervals in the input
domain, which were the center of the basis functions. We
used a Gaussian function as the basis function. The intervals greatly affects the expression and generalization abilities ofan RBFN. In this experiment, we used two types of RBFN.
One is termed RBFN 441, whose parameters are s = 0.05, astandard deviation of σ = 0.05, and 441 basis functions. The
other is termed RBFN 81, whose parameters are s = 0.125,
a standard deviation of σ = 0.125, and 81 basis functions.
Training was conducted using a gradient descent.
INGnet We used a normalized Gaussian function as thebasis function, which was additionally set during the learning
phase. We added a basis function when the absolute errorof the training was larger than the threshold value emax andwhen the value of all existing basis functions was smaller than
the threshold value amin . The parameters were emax = 0.05,
amin = 0.5, and σ = 0.05. Training was conducted using agradient descent.
SDNN In this experiment, we used LIBSDNN1 . The pattern coding parameters were n = 400, q = 201, and r = 2.
The number of SPs m was set to 280, and the output of theSDNN was calculated using 0.005k − 0.2, where k is thenumber of SPs with outputs of 1.
Training was repeated until the absolute training error
reached below 0.01 for each function approximator.

Figure 3: Temporal changes in approximation error
MLP exhibited the largest error at all time steps, indicatingthat it is not good at online incremental learning. The errorof INGnet did not decrease during 1000 < t < 3000, probably because the number of bases increased with a decrease ingeneralization ability and thus, the given samples were insufficient for INGnet to follow the changes in the target function.
Here, RBFN 81 reduced the error during the early time steps
and adapted to the gradual change at 1000 ≤ t < 3000, but
not to the abrupt change at t ≥ 3000, indicating that it is unable to represent a complex function with 81 basis functions.
However, RBFN 441 was able to fit the complex function butreduced the error only slowly for t ≤ 1000 and could not
adapt to the gradual change during 1000 < t ≤ 3000, indicating that it requires more samples. Thus, RBFN cannot
adapt to both sudden and incremental drifts.
In contrast, the SDNN was able to adapt to both drifts, indicating that it can fit a simple target function with a small
number of samples, and can also represent a complex function.
4. Experiment Using Real-Time Dataset
We also applied an SDNN to the prediction of future closing prices of the Nikkei225 on the Tokyo Stock Exchange2from previous closing prices. This price usually changes
gradually, but occasionally does so extremely rapidly, andthus has been used in previous studies on concept drift [1, 2].
4.1 Methods

We used the same data (from May 19, 1979 to May 15,2017) and procedure as in [1].
A simple method of handling streaming data with con3.2 Results
cept drift is to use a sliding window that keeps the k newest
Table 1 shows the average approximation error over all samples. These k samples are normalized and input into
time steps, in which the error was lowest for the SDNN. Fig. an SDNN trained to predict the value N steps ahead (N =3 shows the temporal changes in the mean absolute error for 1, 3, 5). Different SDNNs are used for different N . The
each function approximator.
dataset consists of 4,906 samples d1 , . . . , d4906 . At time step
1 https://github.com/BIPL-HORIE/LIBSDNN

Journal of Signal Processing, Vol. 23, No. 4, July 2019

2 https://finance.yahoo.com/quote/% 5EN225/147

Table 2: Prediction errors for each method

ARWin
SDNN
OPOSSAM(for reference)

N =1RMSE
MAE

N =3RMSE
MAE

N =5RMSE
MAE

299.45
191.63222.12136.98387.99327.36290.04242.11

467.19425.53351.94317.20200.35142.90347.73252.50444.58331.09

Table 2 shows the prediction errors, which indicate that
the SDNN can predict future prices much better than ARWin. This indicates that the SDNN is adaptable to changes in
the characteristics of the data. Fig. 4 shows the values predicted by ARWin and the SDNN for N = 3. ARWin exhibits
large prediction errors, particularly when significantly changing from a rising to falling trend or vice versa, but the SDNN
does not.
It should be noted that most of the error values of theSDNN were smaller than those of OPOSSAM, which exhibited the best performance in a previous study [1], although
we did not test that method ourselves. We also conducted thesame experiment using other data (from May 21, 1979 to May
19, 1997), and confirmed that the SDNN exhibited almost thesame performance, indicating the generality of the results.
5. Conclusions
We showed that an SDNN has an excellent capability for
online incremental learning for a time-variant function approximation. We also applied an SDNN to a real-world timeseries prediction task and obtained a superior performanceover existing methods. These results suggest that an SDNN
is highly suitable for certain types of concept drift problems.
In future research, we will apply an SDNN to other concept
drift problems to confirm its effectiveness.

Figure 4: Stock price predicted by each approximator for thelast 200 time steps for N = 3t, the SDNN receives k samples dt−N , . . . , dt−k−N +1 , and istrained to predict dt . At t > 1000, it also predicts dˆt+N from
xt , . . . , xt−k+1 , and the prediction error between dt+N anddˆt+N is calculated (we use both RMSE and MAE as metricsof the prediction accuracy).
We used ARWin [2] for comparison.
ARWin ARWin has a sliding window of k in size, andthe sliding window induces k subwindows. Each subwindow
has the i newest samples (i = 1, . . . , M ), which are used tocalculate a predicted value based on linear regression. The
output of ARWin is determined using the weighted sum ofthese values, where the weights are updated through learning.
In this experiment, the window size k was set to 3, 4, . . . , 50,
according to [1].
SDNN We used LIBSDNN with the parameters n =1250, q = 1251, r = 1, and m = 700. The window size
k was fixed to 50 for the SDNN. The 200-day moving average d¯t = (dt + · · · + dt−199 )/200 was used as the baseline, that is, the input variable was given by xi = dt−i+1 /d¯t(i = 1, . . . , 50), and dˆt+N was obtained by multiplying theoutput value by d¯t .
4.2 Results148

Acknowledgment
This work was supported partly by JSPS KAKENHI, Grant
No. JP18H03304.
References[1] A. Yamaguchi, S. Maya, T. Inagi and K. Ueno: OPOSSAM: Online prediction of stream data using selfadaptive memory, IEEE International Conference on Big
Data, 2018.
[2] S. Yoshida, K. Hatano, E. Takimoto and M. Takeda:
Adaptive online prediction using weighted windows, The
IEICE Transactions on Information and Systems, Vol. E94-D, No. 10, pp. 1917-1923, 2011.
[3] K. Nonaka, F. Tanaka and M. Morita: Empirical comparison of feedforward neural networks on two-variable
function approximation, The IEICE Transactions on Information and Systems (Japanese Edition), Vol. J-94-DII, No. 12, pp. 2114-2125, 2011.
[4] K. Horie, A. Suemitsu, T. Tanno and M. Morita: Directestimation of wrist joint angular velocities from surface
EMGs by using an SDNN function approximator, Proceedings of the 23rd International Conference on NeuralInformation Processing (ICONIP), 2016.

Journal of Signal Processing, Vol. 23, No. 4, July 2019[5] P. Auer, H. Burgsteiner and W. Maass: A learning rule
for very simple universal approximators consisting of asingle layer of perceptrons, Neural Networks, Vol. 21,
pp. 786-795, 2008.
[6] D. S. Broomhead and D. Lowe: Radial basis functions,multi-variable functional interpolation and adaptive networks, Complex Systems, Vol. 2, No. 3, pp. 321-355,
1988.
[7] J. Morimoto and K. Doya: Learning dynamic motor sequence in high-dimensional state space by reinforcement
learning: Learning to stand up, The IEICE Transactions
on Information and Systems (Japanese Edition), Vol. J82-D, No. 11, pp. 2118-2131, 1999.

Journal of Signal Processing, Vol. 23, No. 4, July 2019

149