# Memristive Chebyshev Neural Network and Its Applications in Function Approximation

Authors: Lidan Wang; Meitao Duan; Shukai Duan
Journal: Mathematical Problems in Engineering
 (2013)

---

## Abstract

A novel Chebyshev neural network combined with memristors is proposed to perform the function approximation. The relationship between memristive conductance and weight update is derived, and the model of a single-input memristive Chebyshev neural network is established. Corresponding BP algorithm and deriving algorithm are introduced to the memristive Chebyshev neural networks. Their advantages include less model complexity, easy convergence of the algorithm, and easy circuit implementation. Through the MATLAB simulation results, we verify the feasibility and effectiveness of the memristive Chebyshev neural networks.


---

## Body

## 1. Introduction

Many researchers have further studied the approximation capacity of neural networks. Research indicates that there are some drawbacks in Back Propagation (BP) neural networks, such as slow convergence, getting easily into a local minimum, and difficulties in determining the numbers of hidden neurons. The Radical Basis Function (RBF) neural networks are superior to BP neural networks in approximation capacity because their learning rate is fast, but the lack of theoretical guidance in determining the central position of basis function makes us set the central position based on experience. We cannot ensure whether the performance of networks is the best or not. Especially, there is a dilemma between approximation accuracy and networks complexity when we use the aforementioned two neural networks to approximate the nonlinear function. However, the Chebyshev neural networks can solve the bottleneck; their hidden layer neuron activation functions are a group of Chebyshev orthogonal polynomials. Nowadays, Chebyshev neural networks are widely used in complex nonlinear systems [1], chaos systems [2], and discrete-time nonlinear systems [3]. Their learning rate and approximation accuracy are better than traditional neural networks, and they can quickly determine the numbers of hidden neurons. Meanwhile, Chebyshev neural networks are widely used in aerospace [4, 5] and chemistry areas [6]. The nanoscale memristor has the potential of information storage because of the nonvolatility with respect to long periods of power-down, so it can be used as synapse in the neural networks. Many researchers in the field of memristors have suggested that this device has high potential for implementing artificial synapses. Afifi et al. studied the realization of STDP learning rules based on the pulsing neuromorphic networks with memristor cross array [7]. Hu et al. built a novel chaotic neural work with memristor to implement associative memory [8]. Wang et al. proposed a PID controller based on memristive CMAC network [9]. Pershin and di Ventra realized the associative memory based on memristive neural networks [10]. Sharifi and Banadaki applied the memristors as memory units to the nonvolatile RAMs and as synapse to the artificial neural networks [11]. Chabi and Klein put forward a neural network with high fault tolerance based on a structure of crossbar switches [12]. Cantley et al. built neural networks with synapses made up from amorphous silicon thin film transistor and memristor, which realized the Hebb learning rules [13]. Gao et al. designed cellular neural network, which is applied to image denoising and edge detection [14]. Kim et al. changed the synapse weight of artificial neural networks with a pulse-based programmable memristor [15].The memristive Chebyshev neural networks have the following advantages: (a) single-input memristive Chebyshev neural networks can realize any nonlinear function approximation by only three layers; (b) the number of hidden neurons in memristive Chebyshev neural networks is significantly smaller than that in traditional BP neural networks; (c) we only need to adjust the weight from the hidden layer to the output layer, which can greatly help in quickening the convergence of the algorithm; (d) as memristive synapse is small and passive, the hardware circuits of the memristive Chebyshev neural networks can be implemented by the VLSI circuits easily.In this paper, we establish the correspondence between memristor conductance and synapse weight through theoretical analysis and MATLAB simulations. The nanoscale memristor is the synapse in Chebyshev neural networks to realize function approximation by memristive BP algorithm and memristive deriving algorithm, respectively.


## 2. Memristive Synapse

The physical model of the memristor [16] is shown in Figure 1. W(t) and D represent the thickness of the doped layer and total oxide films, respectively. RON is the value of the memristor whenW(t) is equal to D, and ROFF is the value of the memristor when W(t) is equal to zero. M(0) is the initial value of the memristor.Figure 1
The physical model of the memristor of HP [16].According to the mathematical model of HP memristor, and the flux-controlled model of memristor, we can obtain the following equation [17]:
(1)M(t)={ROFF,φ(t)<c1,2kφ(t)+M2(0),c1≤φ(t)<c2,RON,φ(t)≥c2,
where k=(RON-ROFF)μνRON/D^2,  c1=(ROFF^2-M^2(0))/2k,c2=(RON^2-M^2(0))/2k.  μν is the average drift rate of oxygen vacancies, so we can obtain the following equation from (1):
(2)G(t)=1M(t)={1ROFF,φ(t)<c1,12kφ(t)+M2(0),c1≤φ(t)<c2,1RON,φ(t)≥c2.The derivate of equation (2) is
(3)dG(t)dt={[2kφ(t)+M2(0)](-3/2)×(-k)×dφ(t)dt,c1≤φ(t)≤c2,0,otherwise.Using calculus,dG(t) is approximately equal to ΔG when  Δtapproaches to zero, and then we can obtain the following equation:
(4)ΔG(t)={[2kφ(t)+M2(0)](-3/2)×(-k)×dφ(t)dt×Δt,c1≤φ(t)≤c2,0,otherwise,
where dφ/dt=v(t); the relationship curve between memristive conductance change and voltage is shown in Figure 2. It can be seen that memristive conductance change  ΔGincreases along with the increasing of applied voltage v. If the learning error e of neural networks is regarded as voltage v, the memristive conductance change can reasonably be described as synapse weight update whose correspondence can be built.Figure 2
The relation curve of the change of memristive conductance and voltage.


## 3. Modeling of the Memristive Chebyshev Neural Networks

Thenth power orthogonal polynomials Tn(x) which are related to the weight function ρ(x)=1/1-x2 are called Chebyshev polynomials of the first kind [18]. Based on definition, if the polynomial system {Tn(x)} (where  n=0,1,2,…) and the weight function satisfy the following relationship:
(5)∫-11ρ(x)Tl(x)Tk(x)dx={0,l≠k,∫-11ρ(x)Tk2(x)>0,l=k,
the orthogonal polynomial system {Tn(x)} can be structured in the interval of [-1,1].In this paper, a model (see Figure3) of the single-input memristive Chebyshev neural networks is proposed, which consists of the input layer, hidden layer, and output layer. The weights from input layer to hidden layer are set to 1, and the memristor conductance stores the weights of hidden layer to output layer. The thresholds of all neurons are set to 0. The hidden layer has m neurons, and their transfer functions are a group of Chebyshev orthogonal polynomial functions Tn(x). Namely, the hidden layer of the nth neuron transfer function is Tn-1(x), where
(6)T0(x)=1,T1(x)=x,Tn+1(x)=2xTn(x)-Tn-1(x),n=1,2,…Figure 3
Model of single-input memristive Chebyshev neural networks.


## 4. Realization of Function Approximation Algorithm

Set memristive Chebyshev neural networks model as follows:input layer:o=x;input of hidden layer neuron:neti=o;output of hidden layer neuron:oi = Ti (neti);output layer:(7)y=∑i=0m-1WiOi=∑i=0m-1WiTi(x).Samples (xt, f(xt)), t=1,2,…,s (s is the number of the samples), are assumed as inputs, and the errors of network output y and the target value f(x) are et = f(xt)-yt.Network training index is(8)E=12∑t=1set2.


### 4.1. Memristive BP Algorithm

In order to establish the relationship between the memristor and the synapse, we propose the learning rule for memristive synaptic weight update. According to (4), we assume the memristive BP algorithm as follows:
(9)ΔWi=[2kΓ(t)+M2(0)](-3/2)×(-k)×et(t)×Δt×Ti(xt),Wi(K+1)=Wi(K)+ΔWi(K),
where i=0,1,2,…,m-1,Δt=η is the learning rate, K is the number of learning, and Γ is the integral of the error e.The learning algorithm is described as follows.Step 1.
Take any number of hidden neuronsn≥3, choose the initial weights Wi(0) as initial memristor conductance, let the learning rate 0<η<1, give the small positive number ε, and set the training sample set (xt, f(xt)) as E=0, t=1, K=0.Step 2.
Calculate(10)yt(K)=∑i=0m-1WiTi(xt),et=f(xt)-yt(K),E⟵E+0.5et2.Step 3.
Adjust weight,(11)Wi(K+1)=Wi(K)+ΔWi(K).Step 4.
t
←
t
+
1; if t<s, then jump back to Step 2, otherwise Step 5.Step 5.
IfE≤ε, then stop the learning; otherwise, E=0,  t=1, K←K+1, and jump back to Step 2.


### 4.2. Memristive Deriving Algorithm

The basic principles of the memristive deriving algorithm are as follows. Take any lessn hidden neurons as the initial cells. After training K times, the errore isno longer changed; that is, E(K) = E(K-1)ε (ε is a pretraining given precision index). The network derives automatically by adding a hidden neuron (n←n+1). The previous process is repeated until E(K)≤ε when the hidden neurons stop deriving and learning. Until reaching target accuracy, the neural network topology is automatically generated.The specific algorithm can be described as follows.Step 1.
Take number of hidden neuronsn=2, choose the initial weights Wij(0) and Cj(0) as initial memristor conductance,let the learning rate 0<η<1, give the small positive number ε>0, and set the training set {(xt,dt), t=1,2,…,s} and the number of learning K=1.Step 2.
Input of hidden layer neuron is(12)netj=∑i=1mWijxi,j=0,1,2,…,n-1.
Calculate(13)yt(K)=∑j=0nCj(K-1)Tj(netj),et=dt-yt.Step 3.
Weight update rule is(14)ΔCj(K)=[2kΓ(t)+M2(0)](-3/2)×(-k)×et(t)×Δt×Ti(netj),Cj(K)=Cj(K-1)+ΔCj(K),ΔWij(K)=[2kΓ(t)+M2(0)](-3/2)×(-k)×et(t)×CjΔt×Tj′(netj)×xi,Wij(K+1)=Wij(K-1)+ΔWij(K).Step 4.
Calculate(15)E(K)=12∑t=1set2.
IfE(K)≤ε, stop derivative and learning which illustrate that neural network topology is automatically generated. Otherwise, go to Step 5.Step 5.
IfE(K)=E(K-1), the training error of the network has no longer reduced. We should increase the number of hidden neurons in order to improve network performance. Namely,  n←n+1,  K←K+1, and jump to Step 2 to continue learning.


## 4.1. Memristive BP Algorithm

In order to establish the relationship between the memristor and the synapse, we propose the learning rule for memristive synaptic weight update. According to (4), we assume the memristive BP algorithm as follows:
(9)ΔWi=[2kΓ(t)+M2(0)](-3/2)×(-k)×et(t)×Δt×Ti(xt),Wi(K+1)=Wi(K)+ΔWi(K),
where i=0,1,2,…,m-1,Δt=η is the learning rate, K is the number of learning, and Γ is the integral of the error e.The learning algorithm is described as follows.Step 1.
Take any number of hidden neuronsn≥3, choose the initial weights Wi(0) as initial memristor conductance, let the learning rate 0<η<1, give the small positive number ε, and set the training sample set (xt, f(xt)) as E=0, t=1, K=0.Step 2.
Calculate(10)yt(K)=∑i=0m-1WiTi(xt),et=f(xt)-yt(K),E⟵E+0.5et2.Step 3.
Adjust weight,(11)Wi(K+1)=Wi(K)+ΔWi(K).Step 4.
t
←
t
+
1; if t<s, then jump back to Step 2, otherwise Step 5.Step 5.
IfE≤ε, then stop the learning; otherwise, E=0,  t=1, K←K+1, and jump back to Step 2.


## 4.2. Memristive Deriving Algorithm

The basic principles of the memristive deriving algorithm are as follows. Take any lessn hidden neurons as the initial cells. After training K times, the errore isno longer changed; that is, E(K) = E(K-1)ε (ε is a pretraining given precision index). The network derives automatically by adding a hidden neuron (n←n+1). The previous process is repeated until E(K)≤ε when the hidden neurons stop deriving and learning. Until reaching target accuracy, the neural network topology is automatically generated.The specific algorithm can be described as follows.Step 1.
Take number of hidden neuronsn=2, choose the initial weights Wij(0) and Cj(0) as initial memristor conductance,let the learning rate 0<η<1, give the small positive number ε>0, and set the training set {(xt,dt), t=1,2,…,s} and the number of learning K=1.Step 2.
Input of hidden layer neuron is(12)netj=∑i=1mWijxi,j=0,1,2,…,n-1.
Calculate(13)yt(K)=∑j=0nCj(K-1)Tj(netj),et=dt-yt.Step 3.
Weight update rule is(14)ΔCj(K)=[2kΓ(t)+M2(0)](-3/2)×(-k)×et(t)×Δt×Ti(netj),Cj(K)=Cj(K-1)+ΔCj(K),ΔWij(K)=[2kΓ(t)+M2(0)](-3/2)×(-k)×et(t)×CjΔt×Tj′(netj)×xi,Wij(K+1)=Wij(K-1)+ΔWij(K).Step 4.
Calculate(15)E(K)=12∑t=1set2.
IfE(K)≤ε, stop derivative and learning which illustrate that neural network topology is automatically generated. Otherwise, go to Step 5.Step 5.
IfE(K)=E(K-1), the training error of the network has no longer reduced. We should increase the number of hidden neurons in order to improve network performance. Namely,  n←n+1,  K←K+1, and jump to Step 2 to continue learning.


## 5. Experimental Results

We use the memristive Chebyshev neural networks to realize nonlinear functiony=x3sinx+sin2x+cos(3x) approximation through memristive BP algorithm and deriving algorithm, respectively. We suppose that memristor model parameters are RON=100Ω, ROFF=10000Ω, M(0)=1000Ω, D=10nm, μv=10-14m2s-1V-1, G(0)=1/M(0).


### 5.1. Memristive BP Algorithm Realizes Function Approximation

The 1 × 8 × 1 network structure is shown in Figure3. The learning rate η=0.02. The number of samples s=200. The number of learning is 1000. The target mean square error E=1.0×10-5. The experimental result is shown in Figure 4. The relationship between training error and the number of learning is shown in Figure 4(a). The number of final learning is 735 when the minimum training error meets the requirements of function approximation. The memristive synaptic weights are constantly updated in the learning process as shown in Figure 4(b). Figures 4(c) and 4(d) describe the final memristive weights and the approximation curve of actual output signal relative to the teacher signal, respectively.Simulation results for memristive BP algorithm.
(a)
Relationship between training error and the number of learning
(b)
Relationship between memristive weights and the number of learning
(c)
The final memristive weights in the hidden layer
(d)
The approximation curve


### 5.2. Memristive Deriving Algorithm Realizes Function Approximation

We adopt the 1 × 2 × 1 initial neural network, withε=1.0×10-5 and learning rate 0.02, and the experimental result is shown in Figure 5. The final network structure is 1 × 25 × 1, and the time of learning is 420. Figure 5(a) shows the final memristive synaptic weights in hidden layer. Figure 5(b) shows the approximation curve of the actual output signal relative to the teacher signal.Simulation result for memristive deriving algorithm.
(a)
The final memristive weights in the hidden layer
(b)
The approximation curveCompared with the memristive BP algorithm, the memristive deriving algorithm has less learning times (735 − 420 = 315) for its neural network is automatically generated.


## 5.1. Memristive BP Algorithm Realizes Function Approximation

The 1 × 8 × 1 network structure is shown in Figure3. The learning rate η=0.02. The number of samples s=200. The number of learning is 1000. The target mean square error E=1.0×10-5. The experimental result is shown in Figure 4. The relationship between training error and the number of learning is shown in Figure 4(a). The number of final learning is 735 when the minimum training error meets the requirements of function approximation. The memristive synaptic weights are constantly updated in the learning process as shown in Figure 4(b). Figures 4(c) and 4(d) describe the final memristive weights and the approximation curve of actual output signal relative to the teacher signal, respectively.Simulation results for memristive BP algorithm.
(a)
Relationship between training error and the number of learning
(b)
Relationship between memristive weights and the number of learning
(c)
The final memristive weights in the hidden layer
(d)
The approximation curve


## 5.2. Memristive Deriving Algorithm Realizes Function Approximation

We adopt the 1 × 2 × 1 initial neural network, withε=1.0×10-5 and learning rate 0.02, and the experimental result is shown in Figure 5. The final network structure is 1 × 25 × 1, and the time of learning is 420. Figure 5(a) shows the final memristive synaptic weights in hidden layer. Figure 5(b) shows the approximation curve of the actual output signal relative to the teacher signal.Simulation result for memristive deriving algorithm.
(a)
The final memristive weights in the hidden layer
(b)
The approximation curveCompared with the memristive BP algorithm, the memristive deriving algorithm has less learning times (735 − 420 = 315) for its neural network is automatically generated.


## 6. Conclusions

The HP memristor model is analyzed, and the correspondence between the memristive conductance change and the synapse weight update in Chebyshev neural networks is established. Two learning algorithms of memristive synapse weight update are proposed. The nonlinear function approximation is implemented by memristive BP algorithm and memristive deriving algorithm, respectively. On the one hand, the memristive BP networks have fixed structure which is easily implemented. On the other hand, the memristive derivative networks have dynamic structure reaching the optimum solution. Memristive Chebyshev neural networks are hopeful to be used in aspects of pattern recognition and data compression.



---

*Source: 429402-2013-06-02.xml*