Improving Time Series Prediction by the Selective Desensitization Neural Network
Based on Synaptic Weight Analysis
Shoichi Someno1 , Kazumasa Horie2 , Tomoki Ichiba1 , Tomohiro Aki1 , Masahiro Morita1
1Graduate School of Systems and Information Engineering
University of Tsukuba,
1–1–1 Tennodai, Tsukuba-shi, 305–8573 Japan

2Center for Computational Sciences
University of Tsukuba,
1–1–1 Tennodai, Tsukuba-shi, 305–8577 Japan

Abstract
The selective desensitization neural network (SDNN) is afunction approximator with high learning ability. It is suitable for time-series prediction problems with concept drift.
However, it is not very effective at dealing with a long timewindow, because the computational costs increase with thesquare of the number of input variables. In this paper, wepresent a method for reducing the computational costs of theSDNN for time-series prediction and eliminating trials anderrors by analyzing the synaptic weights of the SDNN aftertraining. We confirmed, using an artificial problem, that thevariance of synaptic weights reflects the significance of thecorresponding input variable. We also applied this analysis
to a real-world problem and improved prediction accuracy byexpanding the time window without increasing computationalcosts. This method may be applied to other kinds of problems
and thus, extend the application range of the SDNN.
Figure 1: Structure of an SDNN with n dimensional input
1. Introduction
The selective desensitization neural network (SDNN) is afunction approximator with high expression and generalization abilities and numerous other merits such as low hyperparameter dependency and suitability for online incremental
learning [1]. In light of these advantages, Ichiba et al. [2]applied the SDNN to predicting time series data with concept
drift and achieved better performance than previous studies.
However, the computational costs (computation time andrequired memory) of the SDNN increase with the square of
the number n of input variables because the basic SDNN calculates the output from the m = n(n − 1) variable pairs,
which limits the size of n. In the case of time series prediction, the length of the time window is restricted, even if alonger time window is desired [3]. Although we can reducethe computational costs by eliminating insignificant variable
pairs [4], prioritization of their significance is difficult; it usually requires prior knowledge of the problem or many trials
and errors.
We have recently found that it is possible to analyze thesignificance of variable pairs based on the synaptic weightsof the trained SDNN. Specifically, we found that in the approximation task of an artificial function with additional input
variables that were irrelevant to the function value, the variance of the synaptic weights for pairs of irrelevant variables
was less than that for pairs of relevant variables. However, it
was not clear whether this analysis is effective at evaluating
the significance of variable pairs and reducing the computational costs in general cases.
In the present study, we analyze the synaptic weights of theSDNN for time series prediction and aim to improve the prediction accuracy with lower computational costs and minimaleffort.
2. Selective Desensitization Neural Network
The SDNN is constructed by introducing two manipulations, pattern coding and selective desensitization into a parallel perceptron in Figure 1.

2.1 Pattern coding
First, the analogue input value is converted into a highdimensional binary vector (code pattern). Specifically, wequantize the input range into q bins and create a lookup tablein which the code patterns P 1 , P 2 , ..., P q are assigned to eachbin. Each code pattern P 1 , ..., P q is the L dimension binary
code pattern (L represents even numbers), consisting of equal
numbers, +1 and −1. Furthermore, it is preferable for thecorrelation between two consecutive patterns to be high andthat between two patterns that are further apart to be closer to0.
There are several methods of creating such code patterns[1]. In this study, the following procedure is used. First, werandomly create a pattern whose +1 and −1 are equal numbers, and designate it P 1 . Then, we randomly select each relements of +1 and −1 from P 1 , and we invert these signs toget P 2 . Likewise, P 3 is created by inverting the signals of 2r
in the elements of P 2 . Similarly, P u is sequentially createdby inverting 2r elements of P u−1 . The expression and generalization ability of the SDNN are influenced by the dimension
number of code pattern L, division number of input range q,
and the number of inversion of the adjacent pattern r. In thisstudy, these parameters were selected based on preliminaryexperiments.
2.2 Selective desensitization
Selective desensitization is an operation that involves integrating two binary code patterns into one ternary (−1, 0, 1)
code pattern by modifying one with the other. For the explanation of the modification procedure, we specifically consider the instance where the code pattern S = (s1 , s2 , ..., sL )is modified with the code pattern C = (c1 , c2 , ..., cL ). It isalso to be assumed that there is a random one-to-one correspondence between the elements of S and the elements of C.
If there is a correspondence between sµ and cν , the modified
element s′µ is represented as follows:s′µ =1 + cνsµ2(1)the simple perceptrons. Error correction learning is used inparallel perceptron learning. For example, let us assume l outof K simple perceptrons should output 1, but only v(< l) ofthem actually outputted 1. At this point, we select l − v perceptrons with internal potentials nearest the threshold value
among the K − v perceptrons that outputted 0 and train them.
In the case of v > l, we select v − l perceptrons with internalpotentials nearest the threshold value among the perceptrons
that outputted 1.
3. Experiment 1
In Experiment 1, we examine the relationship between thevariance of the synaptic weights and the significance of thevariables using artificial time series.
3.1 Methods
We use a chaotic time series {d1 , d2 , . . . , d1000 } generated
by the logistic map [5] and small noise. Specifically, the i-thcomponent di is represented in the following equation:
di = 3.9di−1 (1 − di−1 ) + N (0, 0.022 )(2), where N (0, 0.022 ) denotes Gaussian noise with zero mean
and a standard deviation of 0.02.
At each time step t, the SDNN receives dt−1 , . . ., and dt−6
as the input variables x1 , . . ., x6 and is trained to predict dt .
The training is performed completely online; thus, the input
vector is given only once.
In the case of 499 ≤ t < 1000, the SDNN also receives
dt , . . ., dt−5 and outputs dˆt+1 , the predicted value of dt+1 , tocalculate the mean absolute error (MAE).
Then, we analyze the synaptic weights of the parallel perceptron. First, the input synapses are divided into 30 groups,
corresponding to the 30 variable pairs x1 (x2 ), . . ., x6 (x5 ),where xi (xj ) represents the modification of the variable xi by
xj . For each group, the variance of synaptic weights is calculated. Afterwards, we arrange the variable pairs in descendingorder of variance. Finally, we repeat the above training anderror evaluation using first m pairs only, with m varying from
1 to 29.
The parameters of the SDNN are L = 1000, K = 700,
q = 501, and r = 2.

In the case of cν = +1, the value of sµ is retained as it is.
In the case of cν = −1, it is converted to a neutral value(= 0). Now, half of the elements of C are −1, therefore, halfof the elements of the ternary code patterns (s′1 , s′2 , ..., s′L ) are
0 , while the other half are ±1. This code pattern is called “S 3.2 Results
modified by C” and is represented as S(C). The signals of all
Figure 2 shows the predicted and actual values of the timethe modified code patterns enter into the parallel perceptron.
series for 970 ≤ t ≤ 1000, indicating that the SDNN canefficiently predict this time series. The MAE was 0.037.
2.3 Parallel perceptron
Figure 3 shows the variance of synaptic weights for each
The parallel perceptron is created by paralleling simple variable pair. We see that the value of the variance depended
perceptrons that output either 0 or 1, and determining the mainly on the modified variable. The variance for xi (xj )overall output based on the sum of the output values of all tended to decrease as i and j increased. As x1 is the mostFigure 4: Prediction error when only high variance pairs wereused
Figure 2: Predicted and actual values of the time series (a partof data)

4.1 Methods

Ichiba et al. [2] applied SDNN to the prediction of thefuture closing price of Nikkei 225 on the Tokyo Stock Exchange1 based on previous closing prices. First, we briefly
explain their method.
The dataset is a time series of 4906 components {d1 , ...,d4906 }. The task is to predict the price of Nikkei 225 N days
ahead based on the prices of the last k days (N = 1, 3, 5,k = 30).
At time step t, k data points dt−N , . . ., dt−N −k+1 are inputted in the SDNN in this form: xi = dt−N −i+1 /d¯t−N(i = 1, . . ., 30), where d¯ is the average of dt−N , . . .,dt−N −199 , and was trained to predict dt /d¯t−N . At t > 1000,it also predicted dˆt+N /d¯t based on dt , . . . , dt−k+1 , and calculated the prediction error between dt+N and dˆt+N .
Figure 3: Variance of synaptic weights for each variable pairIn the present experiment, we first analyze the “originalSDNN used in [2] for N = 3 in the same way as in Experiment 1.
important factor for predicting the next value, and prediction
Based on the results of this analysis, we revise the SDNN
from xi (i = 2, ..., 6) becomes more difficult with an increase
byeliminating insignificant variable pairs and extending thein i, because of noise and the chaotic property of the logisticsize,k, of the time-window and adding some new pairs.
map, this result is considered to indicate that the value of theThen,
we test the revised SDNN for N = 1, 3, and 5, andvariance reflects the significance of the variable and variable
examinethe possibility of improving the prediction accuracy
pair for prediction.
withoutincreasing
the computational costs, while keeping triFigure 4 shows the prediction error (MAE) against thealsanderrors
totheminimum.
number m of variable pairs used for training and prediction.
Theparametersofthe SDNN are fixed at L = 1250, K =From this result, we see that removing variable pairs with low700,
q=1251,r=
1.
variance reduced not only the computational costs but also
the prediction error, indicating that these pairs were unnecessary for prediction. We also see that although it is possible 4.2 Results
to base prediction on only one pair in this task, using many
Figure 5 shows the variance of the synaptic weights for all
pairs with large variance improved the prediction accuracy,(870)
pairs xi (xj ) (i, j = 1, . . . , 30, i ̸= j). It is seen that theprobably because the influence of the noises contained in thevariancefor xi (xj ) mainly depends on i and is particularly
different variables was canceled.
large for i ≤ 6, suggesting that the prices within the fiveprevious days are particularly significant. It is also seen that
4. Experiment 2
the prices around 21 days prior might be more significant thanthose around ten days or 28 days prior.
In Experiment 2, we apply the above method to a real1 https://finance.yahoo.com/quote/% 5EN225/
world task of time-series prediction.

Table 1: Prediction errors for the original and revised SDNNs
N =1

Original
RevisedFigure 5: Variance of synaptic weights for each variable pair

N =3

N =5

MAE

RMSE

MAE

RMSE

MAE

RMSE

138.4
138.0

194.1
193.7

245.9
242.4

335.3
329.3

327.7
314.8

439.8
423.5

prediction task and shown that the variance of the synaptic
weights for a variable pair reflects the significance of the pair,and that we can reduce both the prediction error and the computational costs by removing pairs with low variance. We
have also applied this method to a real-world time series prediction task on stock price, and demonstrated that with minimal trials and errors, it can improve prediction accuracy without increasing computational costs.
Furthermore, the method of analyzing and revising theSDNN presented here may be applied to different types ofproblems; thus, it extends the application range of SDNN.
Acknowledgment
This work was supported by JSPS KAKENHI Grant Number
JP18H03304.
References

Figure 6: Prediction error when only high variance pairs wereused[1] K. Nonaka, F. Tanaka and M. Morita: Empirical comparison of feedforward neural networks on two-variable function approximation, IEICE Trans. Inf. & Syst. (JapaneseEdition), Vol. J94-D-II, No. 12, pp. 2114–2125, 2011.

Figure 6 shows the prediction errors (MAE) when only m [2] T. Ichiba, K. Horie, S. Someno, T. Aki, and M. Morita:
Application of the selective desensitization neural netvariable pairs with larger variance were used for training andwork to concept drift problems, RISP International Workprediction. The error tended to decrease as more pairs wereshop on Nonlinear Circuits, Communications and Signal
used, but the decrease was gentle for m > 100.
Processing, 2019 (accepted).
Based on these results, we extended the time window andrevised the SDNN. Specifically, we reduced 770 pairs, ex[3] A. Yamaguchi, S. Maya, T. Inagi, and K. Ueno: OPOScluding the top 100 pairs from the original SDNN. Then, weSAM online prediction of stream data using self-adaptive
selected 28 variables x1 , . . ., x7 , x12 , x17 , . . ., x112 and added
memory, IEEE International Conference on Big Data,756 pairs by combining them. As 38 pairs were included in2018 (accepted).
the top 100 pairs, 818 pairs were used in the revised SDNN.
Table 1 compares the errors of the revised SDNN with [4] K. Horie, A. Suemitsu, T. Tanno, and M. Morita: Directm = 818 and that of the original SDNN with m = 870.
estimation of wrist joint angular velocities from surface
Although the revised SDNN showed almost the same error asEMGs by using an SDNN function approximator, Interthe original for N = 1, it exhibited considerably less error
national Conference on Neural Information Processing,for N = 5, indicating that the extension of the time window2016.
was more effective for longer term prediction.
[5] G. Boeing: Visual analysis of nonlinear dynamical systems: chaos, fractals, self-similarity and the limits of pre5. Conclusions
diction, Systems, Vol. 4, 2016.
We have analyzed the SDNN for an artificial time-series