# Deep Echo State Network with Variable Memory Pattern for Solar Irradiance Prediction

Authors: Qian Li; Tao Li; Jiangang Ouyang; Dayong Yang; Zhijun Guo
Journal: Complexity
 (2022)

---

## Abstract

Accurate solar irradiance prediction plays an important role in ensuring the security and stability of renewable energy systems. Solar irradiance modeling is usually a time-dependent dynamic model. As a new kind of recurrent neural network, echo state network (ESN) shows excellent performance in the field of time series prediction. However, the memory length of classical ESN is fixed and finite, which makes it hard to map sufficient features of solar irradiance with long-range dependency. Therefore, a novel deep echo state network with variable memory pattern (VMP-DESN) is proposed in this brief. VMP-DESN consists of multiple connected reservoirs in series, and there exist different types of memory modes in VMP-DESN. To remember more input history information in the states, the time delay links are added in each reservoir and between every two reservoirs. The VMP-DESN is more flexible to deal with different input signals due to its variable memory modes in the reservoir states. Additionally, the effect of different memory patterns on the VMP-DESN performance is discussed in detail, including the antidisturbance ability, memory capacity, and prediction accuracy. Finally, the effectiveness of VMP-DESN is evaluated by predicting the real solar irradiance task.


---

## Body

## 1. Introduction

In recent years, accurate solar irradiance prediction has played an increasingly important role in the management, dispatch, and security of renewable energy systems [1]. To predict the solar irradiance series, various techniques have been proposed, such as statistical models [2, 3], artificial neural network (ANN) [4, 5], hybrid methods [6, 7], and so forth. Among those approaches, ANN has become the most popular method for solar irradiance prediction due to its strong nonlinear approximation ability [8].Among ANN, one powerful algorithm is the recurrent neural network (RNN) [9]. Compared with feed-forward neural network (FNN), the RNN has dynamic characteristics and memory performance, by introducing connection loops in the hidden layer. Thus, it is more appropriate for RNN to model the time-dependent solar irradiance series than FNN. Echo state network, a simple yet improved variant of RNN [10], adopts a dynamic reservoir as the hidden layer. The dynamic reservoir contains a large number of randomly sparsely connected neurons, which can encode the input signal from the low-dimensional input space to the high-dimensional state space. In addition, only the output connection weights in ESN need to be trained by linear regression algorithms [11], while the input-reservoir and reservoir-reservoir weights are usually initialized randomly and remain unchanged in the training process. Therefore, ESN avoids the drawbacks of high computational complexity and gradient disappearance in traditional RNN. Due to those advantages, ESN has been widely applied to time series prediction [12–14], pattern recognition [15, 16], and control fields [17–19].For those time series modeling with long-term dependency, the traditional ESN still shows limitations. On the one hand, the reservoir state update equation of ESN is fixed, which can only express the relation between partial input history and the current state. However, the input features far away from the current time cannot be preserved in the current states of ESN. If the states cannot contain enough input history, then the expected value will not be accurately fitted in output layer. On the other hand, the traditional ESN only contains one reservoir layer, which makes it difficult to fully extract input features. Thus, different variants of ESN have been developed for the performance improvement, for instance, expanding single-reservoir to multiple reservoirs [20–23], changing the state update rules of the reservoir [24–26], and so on. The authors in [20, 21, 23] propose deep ESN (DESN) with multiple reservoirs in series array, which can process the input signal layer by layer. The experimental results show that the DESN has better prediction accuracy than traditional ESN due to the strong feature extraction ability of its deep topology. In [24, 25], some improved ESNs with variable state update equation are proposed, by adding the leaky integrator units in the reservoir. For these ESNs, more input and state history are preserved in the current state. Therefore, they have higher memory capacity than ESN, which also has been illustrated in some experiments. In addition, some other variants of ESN have been established and have been successfully applied to solar irradiance prediction, such as the chain-structure ESN (CESN) with multiple independent ESN modules [27, 28], multi-timescale ESN (MTS-ESN) with multiple reservoirs in parallel array [29], and so forth. These ESNs mainly focus on the topology changes for performance improvement, whose memory patterns are single and fixed.It can be noticed that the aforementioned ESNs modify the topology or neuron model to improve the performance. Although some effective results have been achieved, they still face some challenges in practical complex application. Firstly, the memory capacity of DESN is finite and continuous in spite of the strong deep learning ability. The single memory pattern will affect the flexibility of DESN to process different kinds of input signals. Furthermore, the leaky ESNs mainly focus on single-reservoir topology although they have larger and variable memory capacity. The single-reservoir ESN has limited feature extraction capability, which will influence the modeling accuracy.Therefore, a novel deep echo state network with variable memory pattern (VMP-DESN) is proposed in this paper in order to handle aforementioned challenges. To encode the input feature into a richer state space, VMP-DESN adopts multiple subreservoirs in series as the hidden layer. To remember more input history information, the time delay links are added in each reservoir and every two reservoirs. The VMP-DESN can be regarded as an extension of the DESN. Unlike the DESN, the memory capacity of VMP-DESN is variable due to the selective memory mode. Compared with single-reservoir leaky ESNs, the hidden layer of VMP-DESN consists of multiple subreservoirs, which can hierarchically process the input information. Therefore, it is more flexible and advantageous for VMP-DESN to deal with different input signals.In summary, the main contributions of this paper are summarized as follows:(i)
A novel VMP-DESN model is proposed for solar irradiance prediction, which has multiple reservoirs topology and selective memory patterns. The VMP-DESN is studied in terms of mathematical model, pattern classifications, and stability analysis, respectively.(ii)
The time delay links are added in each reservoir and between every two reservoirs in order to preserve more input history in the states. It is more flexible for VMP-DESN to process different input signals.(iii)
The effect of different memory patterns on VMP-DESN is quantitatively and qualitatively analyzed, including the prediction accuracy, antidisturbance ability, and memory capacity.The remainder of this paper is organized as follows. Section2 introduces the proposed VMP-DESN methodology in detail. In Section 3, the performance of VMP-DESN is comprehensively evaluated in terms of prediction accuracy, antidisturbance ability, and memory capacity. Finally, some conclusions are summarized in Section 4.


## 2. Methodology of a Novel VMP-DESN

The memory length of traditional ESN is usually finite and fixed, which cannot extract sufficient features of input signals with long-term dependency. Therefore, a novel deep echo state network with variable memory pattern (VMP-DESN) is proposed in this study to preserve more input history features. The developed VMP-DESN is more flexible to deal with different kinds of input signals, as a result of variable memory capacity. This section will describe the principle of the proposed VMP-DESN methodology.


### 2.1. VMP-DESN Model

As shown in Figure1, VMP-DESN is composed of an input layer, a hidden layer with multiple subreservoirs in series and time delay links, and an output layer. On the one hand, the input signals can be encoded into a richer state space as a result of the hierarchical topology in VMP-DESN. On the other hand, it is more flexible to deal with various input signals as different types of memory patterns are included in VMP-DESN. To remember more input history features from the states, the time delay links are added in both each reservoir and between every two reservoirs. Therefore, VMP-DESN has selectively variable memory modes, which will be helpful for the reservoirs to map more characteristics of input signals with long-term dependency.Figure 1
Structure of VMP-DESN.Assume VMP-DESN hasM subreservoirs with the same neurons N for simplification, K and L are the number of input and output neurons, respectively. Denote the input and output signals at time step t as ut=u1t,u2t,…,uKtT and yt=y1t,y2t,…,yLtT. The global reservoir states are given as Xt=x1t,x2t,…,xMtT, where xlt=xl1,xl2,…xlN represents the states of reservoir layer l, l=1,2,3,…M. The input signal and reservoir states satisfy the compact sets through this paper. Win∈ℝN×K is the input connection weights matrix, Wlres∈ℝN×N denotes the internal connection weights matrix for reservoir l, and Wext/ll=2M∈ℝN×N is the external connection weights matrix between every two adjacent subreservoirs. τ1 denotes the delayed time between every two adjacent reservoirs, while τ2 is the delayed time in each reservoir.The reservoir states of VMP-DESN are updated according to(1)x1t=tanhWinut+W1resx1t−τ2,x2t=tanhW2extx1t−τ1+W2resx2t−τ2,x3t=tanhW3extx2t−τ1+W3resx3t−τ2,⋮xMt=tanhWMextxM−1t−τ1+WMresxMt−τ2.It can be noticed that the input signal of layerll>1 is the delayed state value of reservoir layer l−1. The above equations can be generalized as follows:(2)xlt=tanhWinut+W1resx1t−τ2,ifl=1,tanhWlextxl−1t−τ1+Wlresxlt−τ2if1<l≤M.The network outputs of VMP-DESN are expressed as follows:(3)yt=WoutXt,where Wout∈ℝL×MN is the output connection weights matrix.The method of training VMP-DESN is similar to classical ESN. That is to say, only the output connection matrixWout needs to be learned in VMP-DESN, while other connection matrices remain fixed after proper initialization. The ridge regression training mechanism [30] is adopted to compute Wout by minimizing the cost function as follows:(4)Wout=argminWMW−D22+γW22.The solution of equation (4) is given by(5)WoutT=MTM+γE−1MTD,where M is a Ltr-by-MN reservoir state matrix, D is a Ltr-by-L teacher signal vector matrix, and Ltr is the training length. E denotes a MN-by-MN identity matrix, and γ is the regulation parameter. During the training stage, M and D are collected as follows:(6)M=x11x21…xM1x12x22…xM2⋮⋮⋮⋮x1Ltrx2Ltr…xMLtrLtr×MN,(7)D=d11d21…dL1d12d22…dL2⋮⋮⋮⋮d1Ltrd2Ltr…dLLtrLtr×L,where the t-th column of M includes the state signal of all reservoir layers at time t and the t-th column of D denotes the corresponding expected output signal.The training procedure of VMP-DESN is given in Algorithm1. According to Algorithm 1, the model parameters are firstly initialized by trial and error. Then, the states are updated according to equation (1), when the system is driven by input signal. At the same time, the reservoir states matrix and corresponding expected signal are collected according to equations (6)-(7), respectively. The output connection weights are finally computed as equation (5).Algorithm 1:Training procedures of VMP-DESN.
(1)
Input: u;(2)
Output: Wout;(3)
Initialization:Win,Wlext,Wlres,K,N,L,M,γ,τ1,τ2;(4)
fort←1toLtr(5)
forl←1toM(6)
ifl=1then(7)
xlt←tanhWinut+W1resx1t−τ2;(8)
else(9)
xlt←tanhWlextxl−1t−τ1+Wlyesxlt−τ2;(10)
end(11)
end(12)
yt←ydesiredt; Xt←x1t,x2t,…,xMt;(13)
end(14)
M←X1,X2,…,XLtr;(15)
D←y1,y2,…,yLtr;(16)
WoutT←MTM+γE−1MTD;Remark 1.
When the delayed time are set toτ1=0 and τ2=1, the DESN can be obtained. On this basis, if the number of reservoir is further set to 1 (i.e., M = 1), the DESN model will degrade to the standard ESN. In addition, the variable memory pattern ESN (VMP-ESN) can be obtained when τ1=0, τ2>1, and M=1. DESN, ESN, and VMP-ESN are special cases of VMP-DESN, which will be taken as benchmarks to verify the effectiveness of VMP-DESN in later comparative simulation.Remark 2.
The proposed VMP-DESN includes multiple reservoirs in series array and different types of time delay links. On the one hand, the input signals can be encoded into a richer state space representation due to the hierarchical structure of VMP-DESN. On the other hand, the reservoir state update equation of VMP-DESN is variable and designable, which can deal with different input signals more flexibly.


### 2.2. Pattern Classification of VMP-DESN

From the dynamics of VMP-DESN in equation (2), it can be noticed that the VMP-DESN mainly includes two types of delayed links, i.e., the delay of internal state in each reservoir and the delay of state information transmission between every two subreservoirs. In order to explore the impact of different memory patterns on the network performance, three various VMPs-DESN are discussed in this current study, as shown in Table 1.Table 1
Different pattern cases of VMP-DESN.
PatternModelParameter settingIVMP1-DESNτ1=0, τ2≥1IIVMP2-DESNτ1≠0, τ2=1IIIVMP3-DESNτ1≠0, τ2>1


#### 2.2.1. Pattern I: VMP1-DESN

To test the delay of internal state in each reservoir on network performance, the VMP1-DESN is developed withτ1=0 and τ2≥1. The update rules of VMP1-DESN are rewritten as follows:(8)xlt=tanhWinut+W1resx1t−τ2,ifl=1,tanhWlextxl−1t+Wlresxlt−τ2,if1<l≤M.It can be seen that only one kind of memory mode is retained in VMP1-DESN. VMP1-DESN is a special case of VMP-DESN (equation (2)) which can also be regarded as an extension of DESN. The ridge regression training mechanism in equation (4) can also be used to regulate Wout of VMP1-DESN.


#### 2.2.2. Pattern II: VMP2-DESN

In order to verify the delay of state information transmission between every two subreservoirs on network performance, the VMP2-DESN is established withτ1≠0 and τ2=1. The update equations of VMP2-DESN are expressed as follows:(9)xlt=tanhWinut+W1resx1t−1,ifl=1,tanhWlextxl−1t−τ1+Wlresxlt−1,if1<l≤M.Obviously, the VMP2-DESN is also an extension of DESN. Compared with traditional ESN and DESN, the VMP2-DESN can obtain richer asynchronous state signals, which can express more input history features.


#### 2.2.3. Pattern III: VMP3-DESN

VMP3-DESN includes two kinds of delayed links, i.e., the delay of internal state in each reservoir and the delay of state information transmission between every two reservoirs. The dynamics of VMP3-DESN are formulated as follows:(10)xlt=tanhWinut+W1resx1t−τ2,ifl=1,tanhWlextxl−1t−τ1+Wlresxlt−τ2,if1<l≤M.In VMP3-DESN, it should be satisfied thatτ1≠0 and τ2>1. It can be observed that the VMP3-DESN model is an extension of DESN, VMP1-DESN, and VMP2-DESN.Remark 3.
In order to study the influence of different memory patterns on network performance, VMP1-DESN, VMP2-DESN, and VMP3-DESN are developed, respectively, which correspond to different reservoir state update equation. VMP1-DESN, VMP2-DESN, and VMP3-DESN are three special forms of VMP-DESN (equation (2)) under different constraints. Therefore, the method to train the above three VMPs-DESN can refer to Algorithm 1. Furthermore, the effect of three various VMPs-DESN on network performance will be discussed and compared in later simulation.


### 2.3. Echo State Property Analysis of VMP-DESN

A valid ESN should satisfy the echo state property (ESP), which plays an important role in ensuring the asymptotical stability [31]. ESP denotes that the reservoir states only depend on the input signal with time and independent of its initial values. Similarly, the ESP of the VMP-DESN system is mainly discussed in this section, to guarantee the asymptotically stable operation. To facilitate the theoretical analysis of ESP, the dynamics of VMP-DESN in equation (2) are rewritten as follows:(11)xlt=tanhWlinIlt+Wlresxlt−τ2,where the input signal Ilt is defined as(12)Ilt=ut,ifl=1,xl−1t−τ1,if1<l≤M,and the input connection matrices Wlin is denoted as(13)Wlin=Win,ifl=1,Wlext,if1<l≤M.Then, the Euclidean distance betweenxlt and xl′t randomly initialized from xl0 and xl′0 is computed by(14)xlt−xl′t=tanhWlinIlt+Wlresxlt−τ2−tanhWlinIlt+Wlresxl′t−τ2.According to Lagrange’s mean value theorem and the facttanh′∈0,1, one can obtain that(15)xlt−xl′t=tanhWlresxlt−τ2−Wlresxl′t−τ2≤Wlresxlt−τ2−xl′t−τ2≤……≤Wlrest−τ2xl0−xl′0,xl0 and xl′0 are randomly initialized state values, i.e., ‖xl0−xl′0‖ is bounded. According to equation (15), it can be obviously obtained that ‖xlt−xl′t‖⟶0, when ‖Wlres‖<1 and t⟶∞. Therefore, ρWlres<1 can ensure the ESP of reservoir l, as a result of ‖Wlres‖≥ρWlres (ρ⋅ denotes the spectral radius of Wlres). Furthermore, the necessary condition to guarantee the ESP of VMP-DESN model is deduced as follows:(16)maxl=1,2,…,MρWlres<1.


## 2.1. VMP-DESN Model

As shown in Figure1, VMP-DESN is composed of an input layer, a hidden layer with multiple subreservoirs in series and time delay links, and an output layer. On the one hand, the input signals can be encoded into a richer state space as a result of the hierarchical topology in VMP-DESN. On the other hand, it is more flexible to deal with various input signals as different types of memory patterns are included in VMP-DESN. To remember more input history features from the states, the time delay links are added in both each reservoir and between every two reservoirs. Therefore, VMP-DESN has selectively variable memory modes, which will be helpful for the reservoirs to map more characteristics of input signals with long-term dependency.Figure 1
Structure of VMP-DESN.Assume VMP-DESN hasM subreservoirs with the same neurons N for simplification, K and L are the number of input and output neurons, respectively. Denote the input and output signals at time step t as ut=u1t,u2t,…,uKtT and yt=y1t,y2t,…,yLtT. The global reservoir states are given as Xt=x1t,x2t,…,xMtT, where xlt=xl1,xl2,…xlN represents the states of reservoir layer l, l=1,2,3,…M. The input signal and reservoir states satisfy the compact sets through this paper. Win∈ℝN×K is the input connection weights matrix, Wlres∈ℝN×N denotes the internal connection weights matrix for reservoir l, and Wext/ll=2M∈ℝN×N is the external connection weights matrix between every two adjacent subreservoirs. τ1 denotes the delayed time between every two adjacent reservoirs, while τ2 is the delayed time in each reservoir.The reservoir states of VMP-DESN are updated according to(1)x1t=tanhWinut+W1resx1t−τ2,x2t=tanhW2extx1t−τ1+W2resx2t−τ2,x3t=tanhW3extx2t−τ1+W3resx3t−τ2,⋮xMt=tanhWMextxM−1t−τ1+WMresxMt−τ2.It can be noticed that the input signal of layerll>1 is the delayed state value of reservoir layer l−1. The above equations can be generalized as follows:(2)xlt=tanhWinut+W1resx1t−τ2,ifl=1,tanhWlextxl−1t−τ1+Wlresxlt−τ2if1<l≤M.The network outputs of VMP-DESN are expressed as follows:(3)yt=WoutXt,where Wout∈ℝL×MN is the output connection weights matrix.The method of training VMP-DESN is similar to classical ESN. That is to say, only the output connection matrixWout needs to be learned in VMP-DESN, while other connection matrices remain fixed after proper initialization. The ridge regression training mechanism [30] is adopted to compute Wout by minimizing the cost function as follows:(4)Wout=argminWMW−D22+γW22.The solution of equation (4) is given by(5)WoutT=MTM+γE−1MTD,where M is a Ltr-by-MN reservoir state matrix, D is a Ltr-by-L teacher signal vector matrix, and Ltr is the training length. E denotes a MN-by-MN identity matrix, and γ is the regulation parameter. During the training stage, M and D are collected as follows:(6)M=x11x21…xM1x12x22…xM2⋮⋮⋮⋮x1Ltrx2Ltr…xMLtrLtr×MN,(7)D=d11d21…dL1d12d22…dL2⋮⋮⋮⋮d1Ltrd2Ltr…dLLtrLtr×L,where the t-th column of M includes the state signal of all reservoir layers at time t and the t-th column of D denotes the corresponding expected output signal.The training procedure of VMP-DESN is given in Algorithm1. According to Algorithm 1, the model parameters are firstly initialized by trial and error. Then, the states are updated according to equation (1), when the system is driven by input signal. At the same time, the reservoir states matrix and corresponding expected signal are collected according to equations (6)-(7), respectively. The output connection weights are finally computed as equation (5).Algorithm 1:Training procedures of VMP-DESN.
(1)
Input: u;(2)
Output: Wout;(3)
Initialization:Win,Wlext,Wlres,K,N,L,M,γ,τ1,τ2;(4)
fort←1toLtr(5)
forl←1toM(6)
ifl=1then(7)
xlt←tanhWinut+W1resx1t−τ2;(8)
else(9)
xlt←tanhWlextxl−1t−τ1+Wlyesxlt−τ2;(10)
end(11)
end(12)
yt←ydesiredt; Xt←x1t,x2t,…,xMt;(13)
end(14)
M←X1,X2,…,XLtr;(15)
D←y1,y2,…,yLtr;(16)
WoutT←MTM+γE−1MTD;Remark 1.
When the delayed time are set toτ1=0 and τ2=1, the DESN can be obtained. On this basis, if the number of reservoir is further set to 1 (i.e., M = 1), the DESN model will degrade to the standard ESN. In addition, the variable memory pattern ESN (VMP-ESN) can be obtained when τ1=0, τ2>1, and M=1. DESN, ESN, and VMP-ESN are special cases of VMP-DESN, which will be taken as benchmarks to verify the effectiveness of VMP-DESN in later comparative simulation.Remark 2.
The proposed VMP-DESN includes multiple reservoirs in series array and different types of time delay links. On the one hand, the input signals can be encoded into a richer state space representation due to the hierarchical structure of VMP-DESN. On the other hand, the reservoir state update equation of VMP-DESN is variable and designable, which can deal with different input signals more flexibly.


## 2.2. Pattern Classification of VMP-DESN

From the dynamics of VMP-DESN in equation (2), it can be noticed that the VMP-DESN mainly includes two types of delayed links, i.e., the delay of internal state in each reservoir and the delay of state information transmission between every two subreservoirs. In order to explore the impact of different memory patterns on the network performance, three various VMPs-DESN are discussed in this current study, as shown in Table 1.Table 1
Different pattern cases of VMP-DESN.
PatternModelParameter settingIVMP1-DESNτ1=0, τ2≥1IIVMP2-DESNτ1≠0, τ2=1IIIVMP3-DESNτ1≠0, τ2>1


### 2.2.1. Pattern I: VMP1-DESN

To test the delay of internal state in each reservoir on network performance, the VMP1-DESN is developed withτ1=0 and τ2≥1. The update rules of VMP1-DESN are rewritten as follows:(8)xlt=tanhWinut+W1resx1t−τ2,ifl=1,tanhWlextxl−1t+Wlresxlt−τ2,if1<l≤M.It can be seen that only one kind of memory mode is retained in VMP1-DESN. VMP1-DESN is a special case of VMP-DESN (equation (2)) which can also be regarded as an extension of DESN. The ridge regression training mechanism in equation (4) can also be used to regulate Wout of VMP1-DESN.


### 2.2.2. Pattern II: VMP2-DESN

In order to verify the delay of state information transmission between every two subreservoirs on network performance, the VMP2-DESN is established withτ1≠0 and τ2=1. The update equations of VMP2-DESN are expressed as follows:(9)xlt=tanhWinut+W1resx1t−1,ifl=1,tanhWlextxl−1t−τ1+Wlresxlt−1,if1<l≤M.Obviously, the VMP2-DESN is also an extension of DESN. Compared with traditional ESN and DESN, the VMP2-DESN can obtain richer asynchronous state signals, which can express more input history features.


### 2.2.3. Pattern III: VMP3-DESN

VMP3-DESN includes two kinds of delayed links, i.e., the delay of internal state in each reservoir and the delay of state information transmission between every two reservoirs. The dynamics of VMP3-DESN are formulated as follows:(10)xlt=tanhWinut+W1resx1t−τ2,ifl=1,tanhWlextxl−1t−τ1+Wlresxlt−τ2,if1<l≤M.In VMP3-DESN, it should be satisfied thatτ1≠0 and τ2>1. It can be observed that the VMP3-DESN model is an extension of DESN, VMP1-DESN, and VMP2-DESN.Remark 3.
In order to study the influence of different memory patterns on network performance, VMP1-DESN, VMP2-DESN, and VMP3-DESN are developed, respectively, which correspond to different reservoir state update equation. VMP1-DESN, VMP2-DESN, and VMP3-DESN are three special forms of VMP-DESN (equation (2)) under different constraints. Therefore, the method to train the above three VMPs-DESN can refer to Algorithm 1. Furthermore, the effect of three various VMPs-DESN on network performance will be discussed and compared in later simulation.


## 2.2.1. Pattern I: VMP1-DESN

To test the delay of internal state in each reservoir on network performance, the VMP1-DESN is developed withτ1=0 and τ2≥1. The update rules of VMP1-DESN are rewritten as follows:(8)xlt=tanhWinut+W1resx1t−τ2,ifl=1,tanhWlextxl−1t+Wlresxlt−τ2,if1<l≤M.It can be seen that only one kind of memory mode is retained in VMP1-DESN. VMP1-DESN is a special case of VMP-DESN (equation (2)) which can also be regarded as an extension of DESN. The ridge regression training mechanism in equation (4) can also be used to regulate Wout of VMP1-DESN.


## 2.2.2. Pattern II: VMP2-DESN

In order to verify the delay of state information transmission between every two subreservoirs on network performance, the VMP2-DESN is established withτ1≠0 and τ2=1. The update equations of VMP2-DESN are expressed as follows:(9)xlt=tanhWinut+W1resx1t−1,ifl=1,tanhWlextxl−1t−τ1+Wlresxlt−1,if1<l≤M.Obviously, the VMP2-DESN is also an extension of DESN. Compared with traditional ESN and DESN, the VMP2-DESN can obtain richer asynchronous state signals, which can express more input history features.


## 2.2.3. Pattern III: VMP3-DESN

VMP3-DESN includes two kinds of delayed links, i.e., the delay of internal state in each reservoir and the delay of state information transmission between every two reservoirs. The dynamics of VMP3-DESN are formulated as follows:(10)xlt=tanhWinut+W1resx1t−τ2,ifl=1,tanhWlextxl−1t−τ1+Wlresxlt−τ2,if1<l≤M.In VMP3-DESN, it should be satisfied thatτ1≠0 and τ2>1. It can be observed that the VMP3-DESN model is an extension of DESN, VMP1-DESN, and VMP2-DESN.Remark 3.
In order to study the influence of different memory patterns on network performance, VMP1-DESN, VMP2-DESN, and VMP3-DESN are developed, respectively, which correspond to different reservoir state update equation. VMP1-DESN, VMP2-DESN, and VMP3-DESN are three special forms of VMP-DESN (equation (2)) under different constraints. Therefore, the method to train the above three VMPs-DESN can refer to Algorithm 1. Furthermore, the effect of three various VMPs-DESN on network performance will be discussed and compared in later simulation.


## 2.3. Echo State Property Analysis of VMP-DESN

A valid ESN should satisfy the echo state property (ESP), which plays an important role in ensuring the asymptotical stability [31]. ESP denotes that the reservoir states only depend on the input signal with time and independent of its initial values. Similarly, the ESP of the VMP-DESN system is mainly discussed in this section, to guarantee the asymptotically stable operation. To facilitate the theoretical analysis of ESP, the dynamics of VMP-DESN in equation (2) are rewritten as follows:(11)xlt=tanhWlinIlt+Wlresxlt−τ2,where the input signal Ilt is defined as(12)Ilt=ut,ifl=1,xl−1t−τ1,if1<l≤M,and the input connection matrices Wlin is denoted as(13)Wlin=Win,ifl=1,Wlext,if1<l≤M.Then, the Euclidean distance betweenxlt and xl′t randomly initialized from xl0 and xl′0 is computed by(14)xlt−xl′t=tanhWlinIlt+Wlresxlt−τ2−tanhWlinIlt+Wlresxl′t−τ2.According to Lagrange’s mean value theorem and the facttanh′∈0,1, one can obtain that(15)xlt−xl′t=tanhWlresxlt−τ2−Wlresxl′t−τ2≤Wlresxlt−τ2−xl′t−τ2≤……≤Wlrest−τ2xl0−xl′0,xl0 and xl′0 are randomly initialized state values, i.e., ‖xl0−xl′0‖ is bounded. According to equation (15), it can be obviously obtained that ‖xlt−xl′t‖⟶0, when ‖Wlres‖<1 and t⟶∞. Therefore, ρWlres<1 can ensure the ESP of reservoir l, as a result of ‖Wlres‖≥ρWlres (ρ⋅ denotes the spectral radius of Wlres). Furthermore, the necessary condition to guarantee the ESP of VMP-DESN model is deduced as follows:(16)maxl=1,2,…,MρWlres<1.


## 3. Experimental Design and Results

In this section, the effectiveness of the proposed VMP-DESN is substantiated by solar irradiance forecasting. The solar irradiance datasets in the whole year 2017 with hourly time interval are used for simulation, which are obtained from California Irrigation Management Information System (CIMIS) [32]. CIMIS provides public access to data on the solar irradiance and other details about the stations. For each dataset, the training period and testing period are shown in Table 2.Table 2
Different evaluation periods for prediction models.
Evaluation periodTraining periodTesting periodIJan. 1∼Feb. 28Mar. 1∼Mar. 31IIApr. 1∼May. 31Jun. 1∼Jun. 30IIIJul. 1∼Aug. 31Sep. 1∼Sep. 30IVOct. 1∼Nov. 30Dec. 1∼Dec. 31


### 3.1. Performance Evaluation Metrics

In this current study, the performance of the proposed methodology is comprehensively evaluated through quantitative and qualitative analysis. Four different external metrics in [1] are used to quantitatively evaluate the accuracy of VMPs-DESN, i.e., root mean square error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), and normalized root mean square error (nRMSE), as shown in equations (17)–(20).(17)RMSE=1lte∑t=1lteyforecasted,t−yactual,t2,(18)MAE=1lte∑t=1lteyforecasted,t−yactual,t,(19)MAPE=1lte∑t=1lteyforecasted,t−yactual,tyactual,t,(20)nRMSE=1y¯1lte∑t=1lteyforecasted,t−yactual,t2,where yforecasted,t is the predicted value, yactual,t is the target value, y¯ is the mean of actual value, and lte denotes the length of testing step. In addition, the antidisturbance ability and memory capacity of VMP-DESN will be further qualitatively analyzed in later simulation. The influence of disturbance on one network with strong antidisturbance ability will disappear quickly, which plays an important role in practical applications. Memory capacity refers to the information length that one network can remember in a short time.


### 3.2. Modeling Accuracy of VMP-DESN

The prediction performance of VMP-DESN is verified by one-hour-ahead solar irradiance prediction, and only the historical solar irradiance data are adopted to model in this study. In order to provide appropriate input and output signals for VMP-DESN, the time series analysis [23] is used to characterize the historical solar irradiance data. The network parameters are determined by trial and error, and they are set as follows: the total reservoir size is set as 200, the number of reservoir layer is 4, the input number is 2, and the output layer is 1; connection weights in matrices Win and Wlextl=2M are sampled from a uniform distribution over [−0.1,0.1], while the spectral radius of matrices Wlresl=1M is set as 0.85 to meet the ESP condition. The regularization parameter γ is 10−3.The influence of different delayed time on the four-layer VMP-DESN is firstly studied. Evaluation period II in Seeley station is adopted to test the prediction accuracy versus delayed time. The iteration process is stopped when the testing error is greater than the initial error. The statistical results are shown in Table3, in terms of RMSE, nRMSE, MAE, and MAPE, respectively. In addition, the effect of different memory patterns on VMP-DESN performance is also compared in Table 3. Note that VMP-ESN with τ1=0,τ2=1,andM=1 corresponds to standard ESN and VMP1-ESN with τ1=0,τ1=1,andM>1 corresponds to traditional DESN. From the results in Table 3, it can be obtained that (1) the prediction accuracy of DESN is higher than that of ESN with the same reservoir units; (2) the proposed VMP-DESN and VMP-ESN perform better than DESN and ESN; (3) VMP1-DESN, VMP2-DESN, and VMP3-DESN have equivalent prediction accuracy, indicating that the performance of DESN can be effectively improved by adding these three different memory patterns. Additionally, the prediction error of VMP-ESN, VMP1-DESN, and VMP3-DESN varies rapidly with the delayed time, and it starts to increase when the delayed time is greater than 3. However, the prediction error of VMP2-ESN varies slowly with the delayed time, and it starts to increase when τ1>7. The reason behind this phenomenon may be that the delayed link in reservoir can speed up the network response, while the delayed link between every two subreservoirs can stabilize the network performance. The performance of classical ESN, DESN, VMP-ESN, and VMPs-DESN with the best prediction accuracy is compared in Figure 2(a), and the corresponding relative errors are shown in Figure 2(b). Obviously, experimental results clearly show that the proposed VMP-DESN could match the actual value better than ESN, DESN, and VMP-ESN, especially for some inflection points.Table 3
RMSE, nRMSE, MAE, and MAPE versus delayed time for evaluation period II at Seeley station.
SeeleyModelParameterRMSEnRMSEMAEMAPEVMP-ESNτ1=0,M=1τ2=1109.7560.069680.6390.0662τ2=277.4740.049154.9720.0442τ2=372.4030.045948.9030.0389τ2=477.1880.048955.2610.0444τ2=5129.5480.082196.5600.0807VMP1-DESNτ1=0,M=4τ2=184.4990.053661.6420.0495τ2=273.3130.046553.6470.0421τ2=367.3900.042746.6390.0360τ2=475.0110.047654.4900.0425τ2=5125.3460.0795103.0690.0813VMP2-DESNτ2=1,M=4τ1=178.2600.049657.7780.0458τ1=274.6740.047353.7970.0425τ1=373.3750.046551.7990.0409τ1=470.2430.044549.4990.0385τ1=567.9090.043146.9050.0364τ1=667.4250.042746.5080.0357τ1=765.6080.041644.6450.0342τ1=865.6860.041643.8290.0333τ1=969.9910.044446.5000.0349τ1=1078.8800.050056.2070.0437VMP3-DESNτ1=τ2,M=4τ1=178.2600.049657.7780.0458τ1=269.9170.044349.4950.0386τ1=365.9520.041844.7360.0342τ1=474.1590.047053.2020.0416τ1=5125.8690.0798101.5360.0812Figure 2
(a) The actual values versus predicted solar irradiance by different models. (b) The corresponding relative error comparison.Taking Seeley and Blythe NE stations as two examples, the performance of VMPs-DESN is further verified. The statistical results of different prediction models are recalculated and reported in Table4, in terms of RMSE, nRMSE, MAE, and MAPE. According to the results obtained, it can be seen that VMPs-DESN can achieve higher accuracy than other models for the most testing evaluation periods. Comparative results between ESN and DESN indicate the advantage of hierarchical topology, while the comparative results between ESN and VMP-ESN verify the effectiveness of variable memory. In addition, the equivalent prediction accuracy of VMP1-DESN, VMP2-DESN, and VMP3-DESN further demonstrates the validity of three different types of memory patterns in improving the accuracy of DESN.Table 4
Statistical results comparison of different prediction models for Seeley and Blythe NE stations.
SeeleyBlythe NEEvaluation periodModelRMSEnRMSEMAEMAPERMSEnRMSEMAEMAPEIESN167.5950.1384123.1260.1495154.0640.1460110.3960.1904DESN158.3390.1307116.4980.1373136.3540.129297.5280.1631VMP-ESN149.7460.123697.8040.1229127.2950.120687.5270.1506VMP1-DESN145.0110.119799.7920.1209123.4270.117087.3510.1479VMP2-DESN148.2650.122498.7900.1195117.9980.111882.1290.1409VMP3-DESN142.7450.117996.2830.1189120.9410.114684.4560.1454IIESN109.7560.069680.6390.0662128.5310.085690.4490.0869DESN84.4990.053661.6420.049590.7230.060459.6590.0582VMP-ESN72.4030.045948.9030.038965.0090.043343.9610.0401VMP1-DESN67.3900.042746.6390.036060.5660.040341.6450.0363VMP2-DESN65.6080.041644.6450.034255.1070.036733.1620.0296VMP3-DESN65.9520.041844.7360.034258.4380.038940.0990.0346IIIESN185.9860.1517126.3470.1913194.4150.1713127.7200.4790DESN168.2410.1372104.3680.1707172.3620.151999.4620.4144VMP-ESN160.9270.1313107.7360.1597163.2450.143896.2280.3936VMP1-DESN161.7220.1319104.9340.1549157.2740.138690.0410.3935VMP2-DESN158.6410.129499.4640.1531154.3560.136087.8080.3664VMP3-DESN160.2260.130799.3020.1582155.1230.136788.0440.3878IVESN233.2590.3791177.5880.6571119.3220.187189.3700.7635DESN178.2970.2897132.6550.435099.6330.156269.8620.4839VMP-ESN165.1940.2685114.8090.3371108.3990.169971.8080.3168VMP1-DESN154.9030.2517109.3310.3229100.3910.157468.0390.3304VMP2-DESN152.5370.2479108.1420.328595.8380.150264.5190.2994VMP3-DESN154.3110.2508109.0100.319399.6790.156367.2640.3258


### 3.3. Antidisturbance Ability of VMP-DESN

The antidisturbance ability of VMP-DESN will be qualitatively evaluated in this subsection. The effect of two kinds of disturbances on the networks is mainly analyzed, i.e., the disturbances caused by differences in state initialization and noise in the input signal. For a certain model, the first kind of disturbance can be constructed by initializing two different reservoir states, while other model parameters remain the same. Similarly, the second type of disturbance can be built by adding small disturbance to the input signal, while the model parameters remain consistent. The effect caused by disturbance is expressed by the difference between reservoir states. In this current study, Euclidean distance is adopted to characterize the state differences, as shown in the following equation:(21)DX1−X2=∑i=1MNx1i−x2i2,where X1 and X2 are the reservoir states generated under different conditions. If the distance is close to zero as the iteration progresses, it can indicate that the difference between the two states has disappeared. That is to say, the corresponding model has the antidisturbance capability. The faster the difference disappears, the stronger the antidisturbance ability of a model, and vice versa.The impact of state initialization differences on above prediction models is firstly studied. The only variable is the initialized reservoir states, while other model parameters remain the same. The antidisturbance ability of ESN, DESN, VMP-ESN, and VMP-DESN is compared in Figure3, with the initialized state values as X1=0 and X2=0.01. From Figure 3, it can be observed that (1) the state Euclidean distance of six prediction models gradually decreases to zero as the training progresses; (2) the disturbance curves of VMP-ESN decay are significantly slower than the other five models, and the decay speed of the disturbance curves corresponding to ESN, DESN, and VMP2-DESN is faster than the other three models; and (3) the state difference of ESN, DESN, and VMP2-DESN has decayed to zero, when the training reaches the time step 30. Therefore, it can be concluded that each of the six models has a certain antidisturbance ability, and the antidisturbance ability of ESN, DESN, and VMP2-DESN is stronger than the other three models. The reason is that ESN, DESN, and VMP2-DESN contain their own delayed states, which affect the ability to resist such kind of disturbance caused by initial states.Figure 3
Euclidean distance curves comparison of six prediction models under different initial states.Then, the ability of six models to resist the second type of disturbance is further explored. The single variable is input signal, while the other model parameters remain unchanged. Assume that the input signal at200th time step receives a small disturbance and changes from the original value u200 to u200+0.005. For better presentation, the semilogarithmic Euclidean distance curves of six models under input disturbance are plotted and compared in Figure 4. As can be seen from Figure 4, there is an immediate difference between states at time step 200, when the disturbance is added to the input signal. Simultaneously, the difference between states reaches the maximum at time step 200. As the iteration goes on, the impact of input disturbance on six models gradually decreases. In addition, the disturbance curves corresponding to DESN, VMP-DESN2, and ESN decay faster than the other three models, further indicating the stronger ability to resist input disturbance.Figure 4
Semilogarithmic Euclidean distance comparison of six prediction models under different input signals.The quantitative results in Subsection3.2 have illustrated that VMP1-DESN, VMP2-DESN, and VMP3-DESN corresponding to the three different types of memory patterns can achieve equivalent performance in improving the prediction accuracy. However, the qualitative analysis results in this subsection show that VMP2-DESN performs better than VMP1-DESN and VMP3-DESN in resisting the input and initial states disturbances.


### 3.4. Memory Capacity of VMP-DESN

The memory capacity (MC) of several models is investigated in this section. MC refers to the information length that one network can remember in a short time, i.e., the ability to recall the input signal. According to [33], MC is denoted and computed as follows:(22)MC=∑d=0∞MCd,MCd=R2ut−d,ydt,where ydt is the actual network output and ut−d is the input signal with a delay d. Rut−d,ydt denotes the correlation coefficient between ut−d and ydt. The goal is to train the readout of a network to recall the input signal with a delay d. That is to say, ut−d is the target output with the input signal ut. The closer MCd is to 1, the higher the accuracy of a network is to recall the delayed input signal.In this experiment, MC of traditional ESN, DESN, VMP-ESN, and the proposed VMPs-DESN is calculated and compared. The input signalut is sampled from a uniform distribution −1,1, and the upper limit of d is set as 200 for practical consideration. The number of reservoir layer is 4, and the total reservoir size is set as 200, while other model parameters remain unchanged.Taking a four-layer VMP2-DESN as an example, the influence of delayed timeτ1 on memory capacity is discussed. Figure 5 shows the changes of MCd with d under different τ1 in VMP2-DESN. As illustrated in Figure 5, MCd is almost monotonously decreasing over delay d, when τ1 is relatively small, such as τ1=3 and τ1=7. However, MCd decreases to zero through a series of discrete time spans, when τ1 is large, such as τ1=12 and τ1=15. The reason behind this phenomenon may be the special memory mode in VMP2-DESN. When τ1 is relatively small, there exist duplicate memories in different reservoir layers, and only a small amount of past input information can be recalled accurately by VMP2-DESN. Duplicate memories denote that some input features preserved in former reservoir layer are also contained in the latter reservoir layer. When τ1 is large, the duplicate memories among reservoirs in VMP2-DESN can be effectively decreased to a certain extent. The input features far away from the current time can also be recalled accurately under this circumstance.Figure 5
MCd versus delay d under different delayed time τ1 in VMP2-DESN.Table5 and Figure 6 further compare the MC of several models. From the results in Table 5, it can be seen that (1) the MC of DESN is larger than that of ESN due to multiple reservoir layers; (2) VMP-DESN has higher MC than ESN because of the adjustable memory intervals. In addition, VMP2-DESN has the highest MC value among three different types of VMP-DESN, which further illustrates the advantages of adding delay links between every two subreservoirs. Figure 6 shows MCd versus delay d in ESN, DESN, and VMP2-DESN. Obviously, VMP2-DESN has a higher MC than ESN and DESN. For ESN and DESN, the memory capacity is limited under constant reservoir size. However, the memory of this proposed VMP-DESN can be designable in advance by adjusting the delayed time. Therefore, VMP-DESN can obtain higher memory capacity under proper memory pattern and delayed time.Table 5
MC comparison of six different models.
ModelMCESN12.377DESN18.754VMP-ESNτ2=311.791VMP-ESNτ2=711.612VMP1-DESNτ2=317.629VMP1-DESNτ2=715.933VMP3-DESNτ1=319.682VMP3-DESNτ1=718.476VMP2-DESNτ1=327.156VMP2-DESNτ1=741.617VMP2-DESNτ1=1254.289VMP2-DESNτ1=1556.645VMP2-DESNτ1=1756.225Figure 6
MCd versus delay d in ESN, DESN, and VMP2-DESN.


## 3.1. Performance Evaluation Metrics

In this current study, the performance of the proposed methodology is comprehensively evaluated through quantitative and qualitative analysis. Four different external metrics in [1] are used to quantitatively evaluate the accuracy of VMPs-DESN, i.e., root mean square error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), and normalized root mean square error (nRMSE), as shown in equations (17)–(20).(17)RMSE=1lte∑t=1lteyforecasted,t−yactual,t2,(18)MAE=1lte∑t=1lteyforecasted,t−yactual,t,(19)MAPE=1lte∑t=1lteyforecasted,t−yactual,tyactual,t,(20)nRMSE=1y¯1lte∑t=1lteyforecasted,t−yactual,t2,where yforecasted,t is the predicted value, yactual,t is the target value, y¯ is the mean of actual value, and lte denotes the length of testing step. In addition, the antidisturbance ability and memory capacity of VMP-DESN will be further qualitatively analyzed in later simulation. The influence of disturbance on one network with strong antidisturbance ability will disappear quickly, which plays an important role in practical applications. Memory capacity refers to the information length that one network can remember in a short time.


## 3.2. Modeling Accuracy of VMP-DESN

The prediction performance of VMP-DESN is verified by one-hour-ahead solar irradiance prediction, and only the historical solar irradiance data are adopted to model in this study. In order to provide appropriate input and output signals for VMP-DESN, the time series analysis [23] is used to characterize the historical solar irradiance data. The network parameters are determined by trial and error, and they are set as follows: the total reservoir size is set as 200, the number of reservoir layer is 4, the input number is 2, and the output layer is 1; connection weights in matrices Win and Wlextl=2M are sampled from a uniform distribution over [−0.1,0.1], while the spectral radius of matrices Wlresl=1M is set as 0.85 to meet the ESP condition. The regularization parameter γ is 10−3.The influence of different delayed time on the four-layer VMP-DESN is firstly studied. Evaluation period II in Seeley station is adopted to test the prediction accuracy versus delayed time. The iteration process is stopped when the testing error is greater than the initial error. The statistical results are shown in Table3, in terms of RMSE, nRMSE, MAE, and MAPE, respectively. In addition, the effect of different memory patterns on VMP-DESN performance is also compared in Table 3. Note that VMP-ESN with τ1=0,τ2=1,andM=1 corresponds to standard ESN and VMP1-ESN with τ1=0,τ1=1,andM>1 corresponds to traditional DESN. From the results in Table 3, it can be obtained that (1) the prediction accuracy of DESN is higher than that of ESN with the same reservoir units; (2) the proposed VMP-DESN and VMP-ESN perform better than DESN and ESN; (3) VMP1-DESN, VMP2-DESN, and VMP3-DESN have equivalent prediction accuracy, indicating that the performance of DESN can be effectively improved by adding these three different memory patterns. Additionally, the prediction error of VMP-ESN, VMP1-DESN, and VMP3-DESN varies rapidly with the delayed time, and it starts to increase when the delayed time is greater than 3. However, the prediction error of VMP2-ESN varies slowly with the delayed time, and it starts to increase when τ1>7. The reason behind this phenomenon may be that the delayed link in reservoir can speed up the network response, while the delayed link between every two subreservoirs can stabilize the network performance. The performance of classical ESN, DESN, VMP-ESN, and VMPs-DESN with the best prediction accuracy is compared in Figure 2(a), and the corresponding relative errors are shown in Figure 2(b). Obviously, experimental results clearly show that the proposed VMP-DESN could match the actual value better than ESN, DESN, and VMP-ESN, especially for some inflection points.Table 3
RMSE, nRMSE, MAE, and MAPE versus delayed time for evaluation period II at Seeley station.
SeeleyModelParameterRMSEnRMSEMAEMAPEVMP-ESNτ1=0,M=1τ2=1109.7560.069680.6390.0662τ2=277.4740.049154.9720.0442τ2=372.4030.045948.9030.0389τ2=477.1880.048955.2610.0444τ2=5129.5480.082196.5600.0807VMP1-DESNτ1=0,M=4τ2=184.4990.053661.6420.0495τ2=273.3130.046553.6470.0421τ2=367.3900.042746.6390.0360τ2=475.0110.047654.4900.0425τ2=5125.3460.0795103.0690.0813VMP2-DESNτ2=1,M=4τ1=178.2600.049657.7780.0458τ1=274.6740.047353.7970.0425τ1=373.3750.046551.7990.0409τ1=470.2430.044549.4990.0385τ1=567.9090.043146.9050.0364τ1=667.4250.042746.5080.0357τ1=765.6080.041644.6450.0342τ1=865.6860.041643.8290.0333τ1=969.9910.044446.5000.0349τ1=1078.8800.050056.2070.0437VMP3-DESNτ1=τ2,M=4τ1=178.2600.049657.7780.0458τ1=269.9170.044349.4950.0386τ1=365.9520.041844.7360.0342τ1=474.1590.047053.2020.0416τ1=5125.8690.0798101.5360.0812Figure 2
(a) The actual values versus predicted solar irradiance by different models. (b) The corresponding relative error comparison.Taking Seeley and Blythe NE stations as two examples, the performance of VMPs-DESN is further verified. The statistical results of different prediction models are recalculated and reported in Table4, in terms of RMSE, nRMSE, MAE, and MAPE. According to the results obtained, it can be seen that VMPs-DESN can achieve higher accuracy than other models for the most testing evaluation periods. Comparative results between ESN and DESN indicate the advantage of hierarchical topology, while the comparative results between ESN and VMP-ESN verify the effectiveness of variable memory. In addition, the equivalent prediction accuracy of VMP1-DESN, VMP2-DESN, and VMP3-DESN further demonstrates the validity of three different types of memory patterns in improving the accuracy of DESN.Table 4
Statistical results comparison of different prediction models for Seeley and Blythe NE stations.
SeeleyBlythe NEEvaluation periodModelRMSEnRMSEMAEMAPERMSEnRMSEMAEMAPEIESN167.5950.1384123.1260.1495154.0640.1460110.3960.1904DESN158.3390.1307116.4980.1373136.3540.129297.5280.1631VMP-ESN149.7460.123697.8040.1229127.2950.120687.5270.1506VMP1-DESN145.0110.119799.7920.1209123.4270.117087.3510.1479VMP2-DESN148.2650.122498.7900.1195117.9980.111882.1290.1409VMP3-DESN142.7450.117996.2830.1189120.9410.114684.4560.1454IIESN109.7560.069680.6390.0662128.5310.085690.4490.0869DESN84.4990.053661.6420.049590.7230.060459.6590.0582VMP-ESN72.4030.045948.9030.038965.0090.043343.9610.0401VMP1-DESN67.3900.042746.6390.036060.5660.040341.6450.0363VMP2-DESN65.6080.041644.6450.034255.1070.036733.1620.0296VMP3-DESN65.9520.041844.7360.034258.4380.038940.0990.0346IIIESN185.9860.1517126.3470.1913194.4150.1713127.7200.4790DESN168.2410.1372104.3680.1707172.3620.151999.4620.4144VMP-ESN160.9270.1313107.7360.1597163.2450.143896.2280.3936VMP1-DESN161.7220.1319104.9340.1549157.2740.138690.0410.3935VMP2-DESN158.6410.129499.4640.1531154.3560.136087.8080.3664VMP3-DESN160.2260.130799.3020.1582155.1230.136788.0440.3878IVESN233.2590.3791177.5880.6571119.3220.187189.3700.7635DESN178.2970.2897132.6550.435099.6330.156269.8620.4839VMP-ESN165.1940.2685114.8090.3371108.3990.169971.8080.3168VMP1-DESN154.9030.2517109.3310.3229100.3910.157468.0390.3304VMP2-DESN152.5370.2479108.1420.328595.8380.150264.5190.2994VMP3-DESN154.3110.2508109.0100.319399.6790.156367.2640.3258


## 3.3. Antidisturbance Ability of VMP-DESN

The antidisturbance ability of VMP-DESN will be qualitatively evaluated in this subsection. The effect of two kinds of disturbances on the networks is mainly analyzed, i.e., the disturbances caused by differences in state initialization and noise in the input signal. For a certain model, the first kind of disturbance can be constructed by initializing two different reservoir states, while other model parameters remain the same. Similarly, the second type of disturbance can be built by adding small disturbance to the input signal, while the model parameters remain consistent. The effect caused by disturbance is expressed by the difference between reservoir states. In this current study, Euclidean distance is adopted to characterize the state differences, as shown in the following equation:(21)DX1−X2=∑i=1MNx1i−x2i2,where X1 and X2 are the reservoir states generated under different conditions. If the distance is close to zero as the iteration progresses, it can indicate that the difference between the two states has disappeared. That is to say, the corresponding model has the antidisturbance capability. The faster the difference disappears, the stronger the antidisturbance ability of a model, and vice versa.The impact of state initialization differences on above prediction models is firstly studied. The only variable is the initialized reservoir states, while other model parameters remain the same. The antidisturbance ability of ESN, DESN, VMP-ESN, and VMP-DESN is compared in Figure3, with the initialized state values as X1=0 and X2=0.01. From Figure 3, it can be observed that (1) the state Euclidean distance of six prediction models gradually decreases to zero as the training progresses; (2) the disturbance curves of VMP-ESN decay are significantly slower than the other five models, and the decay speed of the disturbance curves corresponding to ESN, DESN, and VMP2-DESN is faster than the other three models; and (3) the state difference of ESN, DESN, and VMP2-DESN has decayed to zero, when the training reaches the time step 30. Therefore, it can be concluded that each of the six models has a certain antidisturbance ability, and the antidisturbance ability of ESN, DESN, and VMP2-DESN is stronger than the other three models. The reason is that ESN, DESN, and VMP2-DESN contain their own delayed states, which affect the ability to resist such kind of disturbance caused by initial states.Figure 3
Euclidean distance curves comparison of six prediction models under different initial states.Then, the ability of six models to resist the second type of disturbance is further explored. The single variable is input signal, while the other model parameters remain unchanged. Assume that the input signal at200th time step receives a small disturbance and changes from the original value u200 to u200+0.005. For better presentation, the semilogarithmic Euclidean distance curves of six models under input disturbance are plotted and compared in Figure 4. As can be seen from Figure 4, there is an immediate difference between states at time step 200, when the disturbance is added to the input signal. Simultaneously, the difference between states reaches the maximum at time step 200. As the iteration goes on, the impact of input disturbance on six models gradually decreases. In addition, the disturbance curves corresponding to DESN, VMP-DESN2, and ESN decay faster than the other three models, further indicating the stronger ability to resist input disturbance.Figure 4
Semilogarithmic Euclidean distance comparison of six prediction models under different input signals.The quantitative results in Subsection3.2 have illustrated that VMP1-DESN, VMP2-DESN, and VMP3-DESN corresponding to the three different types of memory patterns can achieve equivalent performance in improving the prediction accuracy. However, the qualitative analysis results in this subsection show that VMP2-DESN performs better than VMP1-DESN and VMP3-DESN in resisting the input and initial states disturbances.


## 3.4. Memory Capacity of VMP-DESN

The memory capacity (MC) of several models is investigated in this section. MC refers to the information length that one network can remember in a short time, i.e., the ability to recall the input signal. According to [33], MC is denoted and computed as follows:(22)MC=∑d=0∞MCd,MCd=R2ut−d,ydt,where ydt is the actual network output and ut−d is the input signal with a delay d. Rut−d,ydt denotes the correlation coefficient between ut−d and ydt. The goal is to train the readout of a network to recall the input signal with a delay d. That is to say, ut−d is the target output with the input signal ut. The closer MCd is to 1, the higher the accuracy of a network is to recall the delayed input signal.In this experiment, MC of traditional ESN, DESN, VMP-ESN, and the proposed VMPs-DESN is calculated and compared. The input signalut is sampled from a uniform distribution −1,1, and the upper limit of d is set as 200 for practical consideration. The number of reservoir layer is 4, and the total reservoir size is set as 200, while other model parameters remain unchanged.Taking a four-layer VMP2-DESN as an example, the influence of delayed timeτ1 on memory capacity is discussed. Figure 5 shows the changes of MCd with d under different τ1 in VMP2-DESN. As illustrated in Figure 5, MCd is almost monotonously decreasing over delay d, when τ1 is relatively small, such as τ1=3 and τ1=7. However, MCd decreases to zero through a series of discrete time spans, when τ1 is large, such as τ1=12 and τ1=15. The reason behind this phenomenon may be the special memory mode in VMP2-DESN. When τ1 is relatively small, there exist duplicate memories in different reservoir layers, and only a small amount of past input information can be recalled accurately by VMP2-DESN. Duplicate memories denote that some input features preserved in former reservoir layer are also contained in the latter reservoir layer. When τ1 is large, the duplicate memories among reservoirs in VMP2-DESN can be effectively decreased to a certain extent. The input features far away from the current time can also be recalled accurately under this circumstance.Figure 5
MCd versus delay d under different delayed time τ1 in VMP2-DESN.Table5 and Figure 6 further compare the MC of several models. From the results in Table 5, it can be seen that (1) the MC of DESN is larger than that of ESN due to multiple reservoir layers; (2) VMP-DESN has higher MC than ESN because of the adjustable memory intervals. In addition, VMP2-DESN has the highest MC value among three different types of VMP-DESN, which further illustrates the advantages of adding delay links between every two subreservoirs. Figure 6 shows MCd versus delay d in ESN, DESN, and VMP2-DESN. Obviously, VMP2-DESN has a higher MC than ESN and DESN. For ESN and DESN, the memory capacity is limited under constant reservoir size. However, the memory of this proposed VMP-DESN can be designable in advance by adjusting the delayed time. Therefore, VMP-DESN can obtain higher memory capacity under proper memory pattern and delayed time.Table 5
MC comparison of six different models.
ModelMCESN12.377DESN18.754VMP-ESNτ2=311.791VMP-ESNτ2=711.612VMP1-DESNτ2=317.629VMP1-DESNτ2=715.933VMP3-DESNτ1=319.682VMP3-DESNτ1=718.476VMP2-DESNτ1=327.156VMP2-DESNτ1=741.617VMP2-DESNτ1=1254.289VMP2-DESNτ1=1556.645VMP2-DESNτ1=1756.225Figure 6
MCd versus delay d in ESN, DESN, and VMP2-DESN.


## 4. Conclusion

A novel deep echo state network is proposed in this paper, i.e., VMP-DESN. This model introduces different types of memory patterns, which is helpful to extract more input features of solar irradiance with long-range dependency. Compared with traditional ESN and DESN, the memory length of VMP-DESN is variable and designable. Therefore, it is more flexible for VMP-DESN to deal with various input signals. The effect of three memory patterns on VMP-DESN is explored by quantitative and qualitative analysis in detail. The quantitative simulation results illustrate that the VMPs-DESN under different memory modes can achieve equivalent performance in improving the prediction accuracy. From the qualitative analysis results, VMP-DESN under the second memory pattern has a stronger antidisturbance ability and a higher memory capacity than the other two modes.In this current study, the delayed time between every two subreservoirs are set to be the same with the delayed time in each reservoir for simplification. For more complex modeling in practice, the parameters can be different. However, it should be noted that different delayed time will increase the difficulty of parameter optimization.



---

*Source: 8506312-2022-10-26.xml*