# Stochastic Search Algorithms for Identification, Optimization, and Training of Artificial Neural Networks

Authors: Kostantin P. Nikolic
Journal: Advances in Artificial Neural Systems
 (2015)

---

## Abstract

This paper presents certain stochastic search algorithms (SSA) suitable for effective identification, optimization, and training of artificial neural networks (ANN). The modified algorithm of nonlinear stochastic search (MN-SDS) has been introduced by the author. Its basic objectives are to improve convergence property of the source defined nonlinear stochastic search (N-SDS) method as per Professor  Rastrigin. Having in mind vast range of possible algorithms and procedures a so-called method of stochastic direct search (SDS) has been practiced (in the literature is called stochastic local search-SLS). The MN-SDS convergence property is rather advancing over N-SDS; namely it has even better convergence over range of gradient procedures of optimization. The SDS, that is, SLS, has not  been practiced enough in the process of identification, optimization, and training of ANN. Their efficiency in some cases of pure nonlinear systems makes them suitable for optimization and training of ANN. The presented examples illustrate only partially operatively end efficiency of SDS, that is, MN-SDS. For comparative method backpropagation error (BPE)
method was used.


---

## Body

## 1. Introduction

The main target of this paper is a presentation of a specific option of direct SS and its application in identification and optimisation of linear and nonlinear objects or processes. The method of stochastic search was introduced by Ashby [1] related to gomeostat. Till 60th of last century the said gomeostat of Ashby’s was adopted mostly as philosophic concept in cybernetics trying to explain the question of stability of rather complex systems having impacts of stochastic nature [2].The stochastic direct search (SDS) had not been noticed as advanced concurrent option for quite a long time. The researches and developments works of Professor Rastrigin and his associates promoted the SS to be competing method for solving various problems of identification and optimization of complex systems [3].It has been shown that SDS algorithms besides being competing are even advancing over well-known methods. Parameter for comparing is a property of convergence during solving the set task. For comparing purposes gradient methods were used in reference [4]. The SDS method showed remarkable advance. For systems with noise certain numerical options offer the method of stochastic approximation (MSA) [5]. In some cases procedures of SDS are more efficient than MSA [6].During the last 20 years, vast interests have been shown for advanced SDS, especially on the case where classical deterministic techniques do not apply. Direct SS algorithms are one part of the SSA family. The important subjects of random search were being made: theorems of global optimization, convergence theorems, and applications on complex control systems [7–9].The author has been using SDS algorithms (in several of his published papers) regarding identification of complex control systems [10], as well as synthesis and training of artificial neural networks [11–13].Through experience in application of certain SDS basic definition the author was motivated to introduce the so-called modified nonlinear SDS (MN-SDS) applicable as numerical method for identification and optimization of substantial nonlinear systems. The main reason is rather slow convergence of N-SDS of basic definition and this deficiency has been overcome.The application of SDS is efficient for both determined and stochastic description of systems.The SDS algorithm is characterized by introduction of random variables. An applicable option is generator of random numbers [14, 15].The previously said is enhanced by well-developed modern computer hardware and software providing suitable ambient conditions for creation and implementation of SDS methods and procedures.


## 2. Method and Materials

### 2.1. Definition of SDS Method

The solving of theoretical and/or practical problems usually requests firstly an identification task followed by final stage, that is, a system optimization. The analysis and synthesis of systems always consider the previously said [16, 17].Methods of SDS are ones of competing options for numerical procedures providing solution for identification and optimization of complex control systems [18], but so ANN. Let us start with an internal system description in general form [19]:(1)dxdt=fx,u,s,a,t;t0≥0,xt0=x0,(2)y=gx,n¯,b,t,(3)h1x,n¯,c=0,(4)h2x,n¯,d≥0,where f(·) and g(·) are nonlinear vector functions; h1(·) and h2(·) are vector functions of constrains (variables and parameters); x is system state vector, u is control vector, and s is vector of disturbance; a,b,c,d are parameters describing the system structure such as constants, matrices, and vectors; t is real time; n¯ is noise usually added in (2).A parameters identification of the above system anticipates certain measurements of the system variables observing the criteria function:(5)Q=Qx,u,y.The criteria functionQ in (5) is without involvement of constrains h1(·) and h2(·); in case that constrains are not possible to avoid, Qh is introduced [20]:(6)Qh=Q+∑i=1i=mλih1,i·+∑j=1j=lλjh2,j·i=1,2,…,m;j=1,2,3,…,l,where λi and λj are Langrage multiplicators and λi>0, a λj{0forh2,j≥0;Λjforh2,j<0,Λj>0} [20].Whenλi and λj are rather large and tend to ∞, then both Qh and Q tend to the same value for variables x,u,s, that is, corresponding optimal parameters.Further for the purpose of simplicity of this presentation a functionQ from (5) is to be used and to be the function of one vector variable x, so Q=Q(x).Methods of optimization start with iterative form where from current statexi system transfers into xi+1 by the following rule:(7)xi+1=xi+αFQ.So,αF(Q) is a function stepping into new state where α>0 is a step and F(Q) is vector function of guiding search. For iterative optimization by using gradient method [21]:(8)FQ=-gradQ,gradQ=∇Q,xi+1=xi-αgradQ,0<α<1.In case of SDS the relation (7) gets the next form:(9)xi+1=xi+αFξ,Q,where the direction of search is function of Q and random vector ξ (Figure 1).Figure 1
A random variableξ; (a) on sphere radius r=1, (b) on cube edge a=1, and (c) on cube edge a=2; dimension n=3.
(a)
(b)
(c)If it is introduced terms(10)Δxi+1=xi+1-xi,ΔQi=Qi-Qi-1the general expression (9) gives some of basic algorithm of SDS.


#### 2.1.1. Nonlinear SDS

Consider(11)Δxi+1≜-Δxi,ΔQi≥0;ΔQi=Qi-Qi-1αζi,ΔQi<0;α>0,ζi=ortξi.


#### 2.1.2. Linear SDS

Consider(12)Δxi+1≜Δxi,ΔQi<0αζi,ΔQi≥0;α>0,ζi=ortξi.Some of the more complex forms of SDS are as follows.


#### 2.1.3. SDS Stochastic Gradient

Consider(13)xi+1≜xi-αζi,α≥0,ζi≜dir∑P=1PξisignΔQP;ζi=1,ΔQP≜QPxi+σζP-QPxi-σζPp=1,2,3,…,P,where p is number of tests before effective step forward.


#### 2.1.4. SDS Fastest Descent

Iterative procedure for a SDS fastest descent is(14)Δxi+1≜Δxi,Qi-1<0,-Δxi-1+αζi-1;ΔQ≥0-αζi+αζi+1;Δxi≠Δxi-1,ΔQ≥0.The graphic presentation of SDS behavior in the space of gradientQ is given in Figure 2.Figure 2
SDS algorithms in gradient field.Gradient is starting from pointP1, linear SDS starts from point P2, nonlinear SDS starts from point P3, and SDS-statistic gradient starts from point P5 (vector pairs marked dash opposes failure tests). The gradient-fastest descend is presented from starting point P1. SDS fastest descend is not presented in Figure 2 (it is similar to linear SDS from P2).The random vectorξn is created on n-dimensional sphere rn;|rn|=1, or on n-dimensional cube; n is system dimension in Euclide space E. A presentation is shown in Figures 1(a), 1(b), and 1(c) for n=3.The SSA properties used for ranking with other competitive methods (gradient, fastest descend, Gaus-Zeidel, and scanning and others) are(i)
local: describing algorithm in i-step,(ii)
nonlocal: describing algorithm from start to the final stage of optimization.The main property is dissipation, that is, losses in one step of search and displacement on hipper-surface of the functionQ. The said properties describe the algorithm convergence, that is, the method itself.The convergenceKn is defined by expression:(15)Kn=M(Ni)M(ΔQ¯i),that is, ratio of math expectation of test number Ni in i-iteration over relative change of ΔQi¯ (criteria function Q) [22].Reciprocal value of relation (15) gives an average value q of Q in i-step of searching.The local SDS property includes probability of failing step of searching:(16)p=PQx<Qx+Δx,where x is vector of initial state and Δx is working step of state change in x+Δx. The probability of failing step is rather important regarding choosing of algorithm for real process optimisation.So, the choice of optimal algorithm from local property point of view guides to compromise of choice of three:(17)(K,q,p).Besides local properties the relevant ones are properties describing the search in whole [23] which are(i)
total number of steps, that is, number of effective stepsN,(ii)
accuracy, that is, allowed error the procedure is endingε or relative error δ in %.The SDS option choice is guided by the above indicated local characteristics properties(K,q,p) as well as nonlocal ones (N,ε).It is necessary also to observe that the dispersionD(ΔQ¯) of ΔQ¯ could be useful in some statistical estimation of searching procedure properties. Also, it is necessary to notice that dispersion depends on how the vector ξ is generated (hipper sphere or cube) [23].Finally, it is worthwhile to mention that the choice of algorithm is subject to request to have procedures (identification and optimization) effective, having in minds that system model or criteria function is nonlinear. Substantially nonlinear process models so, described by (1) not having linear approximation which simulates original behavior (1) within narrow range of variables and parameters change against defined error limits. SDS methods are effective in this case so [22, 23].


### 2.2. Modified Nonlinear SDS: The Definition

Good properties of SDS as per (11) are simplicity, reliability, and applicability over nonlinear systems but slow convergence during the optimization procedure shows weak point. A rough explanation of nonlinear SDS of basic definition within gradient field Q is shown in Figure 2, with starting point P3. Comparing with regular gradient method this SDS becomes competitive when the system dimension is over n>12 [24].By increasing the system dimension the probability for acceptable tests decrease indicating that the target is reached after rather numerous tests and effective stepsN. The stochastic gradient is SDS having competitive performance during optimisation of nonlinear systems; however, the algorithm itself accumulates information and becomes fed up in numerical proceeding.The idea to have failed tests converted onto useful accumulated information guides toward the so-called modified nonlinear SDS. The previously said is shown in Figure2. For the 3 failed tests from starting point P4 it effects vector ξi,R to turn over toward the target under angle γ. So, if the accumulation of failed tests is J, between i and i+1 successful steps on hyper area of Q then:(18)∑j=1Jξj(i)=ξR(i,J);ortξR(i,J)=ζR(i,J).Now it is possible to form the next form of SSA search:(19)Δxi+1=αζi,ΔQi<0,-αζR(i,J)+αζi+1,ΔQi+1<0,where ζi+1=ortξi+1 is successful test after J failed, +αζR(i,J), ΔQj=Qj-Qj-1≥0; j=1,2,3,…,J, generate of accumulation information in MN-SDS algorithm and possibly being used so:(20)-αξji,max⁡+αξi+1,where ζi(j,max⁡)corresponds to max Qi(j).A modification defined by (18) and (19) is naturally linked on basically defined nonlinear SDS and further on will be referred to as MN-SDS. At certain extent MN-SSA possesses property of nonlinear algorithm, linear SDS with accumulation and extrapolation and stochastic gradient.Having in mind that MN-SDS explore accumulated information some sort of self-learning features are obtained (specifically when all data are memorized) leading to the conclusion that stochastic probability of testing depends on accumulated information:(21)P′=p′xw,where, memory vector, could be defined as(22)w=∫p′xwdx.This brings that searching is guided within the vicinity of the best probe or test [23]; a memory vector indicates adaptation features and can be calculated for l step like(23)wi(l)=kwi-1(l-1)+χζiΔQi(l-1),where(24)ΔQi(l-1)=Qi+1(l-1)-Qi(l-1),where k is coefficient of erasing and χ is coefficient of learning intensity or self-learning.Now the vector guidance toward the target is(25)ζl(sl)=ortw+ξ=w+ξw+ξ,where ζi(sl) is corrected direction by self-learning (sl) on l-step; l=1,2,3,…,L, L≤J, “steps of accumulation” (J failed test after i step).In practice the optimisation starts without self-learning option. In that sense MN-SDS regardless of the range of accumulated information of failed tests not memorized enables sampling of those most useful ones.


## 2.1. Definition of SDS Method

The solving of theoretical and/or practical problems usually requests firstly an identification task followed by final stage, that is, a system optimization. The analysis and synthesis of systems always consider the previously said [16, 17].Methods of SDS are ones of competing options for numerical procedures providing solution for identification and optimization of complex control systems [18], but so ANN. Let us start with an internal system description in general form [19]:(1)dxdt=fx,u,s,a,t;t0≥0,xt0=x0,(2)y=gx,n¯,b,t,(3)h1x,n¯,c=0,(4)h2x,n¯,d≥0,where f(·) and g(·) are nonlinear vector functions; h1(·) and h2(·) are vector functions of constrains (variables and parameters); x is system state vector, u is control vector, and s is vector of disturbance; a,b,c,d are parameters describing the system structure such as constants, matrices, and vectors; t is real time; n¯ is noise usually added in (2).A parameters identification of the above system anticipates certain measurements of the system variables observing the criteria function:(5)Q=Qx,u,y.The criteria functionQ in (5) is without involvement of constrains h1(·) and h2(·); in case that constrains are not possible to avoid, Qh is introduced [20]:(6)Qh=Q+∑i=1i=mλih1,i·+∑j=1j=lλjh2,j·i=1,2,…,m;j=1,2,3,…,l,where λi and λj are Langrage multiplicators and λi>0, a λj{0forh2,j≥0;Λjforh2,j<0,Λj>0} [20].Whenλi and λj are rather large and tend to ∞, then both Qh and Q tend to the same value for variables x,u,s, that is, corresponding optimal parameters.Further for the purpose of simplicity of this presentation a functionQ from (5) is to be used and to be the function of one vector variable x, so Q=Q(x).Methods of optimization start with iterative form where from current statexi system transfers into xi+1 by the following rule:(7)xi+1=xi+αFQ.So,αF(Q) is a function stepping into new state where α>0 is a step and F(Q) is vector function of guiding search. For iterative optimization by using gradient method [21]:(8)FQ=-gradQ,gradQ=∇Q,xi+1=xi-αgradQ,0<α<1.In case of SDS the relation (7) gets the next form:(9)xi+1=xi+αFξ,Q,where the direction of search is function of Q and random vector ξ (Figure 1).Figure 1
A random variableξ; (a) on sphere radius r=1, (b) on cube edge a=1, and (c) on cube edge a=2; dimension n=3.
(a)
(b)
(c)If it is introduced terms(10)Δxi+1=xi+1-xi,ΔQi=Qi-Qi-1the general expression (9) gives some of basic algorithm of SDS.


### 2.1.1. Nonlinear SDS

Consider(11)Δxi+1≜-Δxi,ΔQi≥0;ΔQi=Qi-Qi-1αζi,ΔQi<0;α>0,ζi=ortξi.


### 2.1.2. Linear SDS

Consider(12)Δxi+1≜Δxi,ΔQi<0αζi,ΔQi≥0;α>0,ζi=ortξi.Some of the more complex forms of SDS are as follows.


### 2.1.3. SDS Stochastic Gradient

Consider(13)xi+1≜xi-αζi,α≥0,ζi≜dir∑P=1PξisignΔQP;ζi=1,ΔQP≜QPxi+σζP-QPxi-σζPp=1,2,3,…,P,where p is number of tests before effective step forward.


### 2.1.4. SDS Fastest Descent

Iterative procedure for a SDS fastest descent is(14)Δxi+1≜Δxi,Qi-1<0,-Δxi-1+αζi-1;ΔQ≥0-αζi+αζi+1;Δxi≠Δxi-1,ΔQ≥0.The graphic presentation of SDS behavior in the space of gradientQ is given in Figure 2.Figure 2
SDS algorithms in gradient field.Gradient is starting from pointP1, linear SDS starts from point P2, nonlinear SDS starts from point P3, and SDS-statistic gradient starts from point P5 (vector pairs marked dash opposes failure tests). The gradient-fastest descend is presented from starting point P1. SDS fastest descend is not presented in Figure 2 (it is similar to linear SDS from P2).The random vectorξn is created on n-dimensional sphere rn;|rn|=1, or on n-dimensional cube; n is system dimension in Euclide space E. A presentation is shown in Figures 1(a), 1(b), and 1(c) for n=3.The SSA properties used for ranking with other competitive methods (gradient, fastest descend, Gaus-Zeidel, and scanning and others) are(i)
local: describing algorithm in i-step,(ii)
nonlocal: describing algorithm from start to the final stage of optimization.The main property is dissipation, that is, losses in one step of search and displacement on hipper-surface of the functionQ. The said properties describe the algorithm convergence, that is, the method itself.The convergenceKn is defined by expression:(15)Kn=M(Ni)M(ΔQ¯i),that is, ratio of math expectation of test number Ni in i-iteration over relative change of ΔQi¯ (criteria function Q) [22].Reciprocal value of relation (15) gives an average value q of Q in i-step of searching.The local SDS property includes probability of failing step of searching:(16)p=PQx<Qx+Δx,where x is vector of initial state and Δx is working step of state change in x+Δx. The probability of failing step is rather important regarding choosing of algorithm for real process optimisation.So, the choice of optimal algorithm from local property point of view guides to compromise of choice of three:(17)(K,q,p).Besides local properties the relevant ones are properties describing the search in whole [23] which are(i)
total number of steps, that is, number of effective stepsN,(ii)
accuracy, that is, allowed error the procedure is endingε or relative error δ in %.The SDS option choice is guided by the above indicated local characteristics properties(K,q,p) as well as nonlocal ones (N,ε).It is necessary also to observe that the dispersionD(ΔQ¯) of ΔQ¯ could be useful in some statistical estimation of searching procedure properties. Also, it is necessary to notice that dispersion depends on how the vector ξ is generated (hipper sphere or cube) [23].Finally, it is worthwhile to mention that the choice of algorithm is subject to request to have procedures (identification and optimization) effective, having in minds that system model or criteria function is nonlinear. Substantially nonlinear process models so, described by (1) not having linear approximation which simulates original behavior (1) within narrow range of variables and parameters change against defined error limits. SDS methods are effective in this case so [22, 23].


## 2.1.1. Nonlinear SDS

Consider(11)Δxi+1≜-Δxi,ΔQi≥0;ΔQi=Qi-Qi-1αζi,ΔQi<0;α>0,ζi=ortξi.


## 2.1.2. Linear SDS

Consider(12)Δxi+1≜Δxi,ΔQi<0αζi,ΔQi≥0;α>0,ζi=ortξi.Some of the more complex forms of SDS are as follows.


## 2.1.3. SDS Stochastic Gradient

Consider(13)xi+1≜xi-αζi,α≥0,ζi≜dir∑P=1PξisignΔQP;ζi=1,ΔQP≜QPxi+σζP-QPxi-σζPp=1,2,3,…,P,where p is number of tests before effective step forward.


## 2.1.4. SDS Fastest Descent

Iterative procedure for a SDS fastest descent is(14)Δxi+1≜Δxi,Qi-1<0,-Δxi-1+αζi-1;ΔQ≥0-αζi+αζi+1;Δxi≠Δxi-1,ΔQ≥0.The graphic presentation of SDS behavior in the space of gradientQ is given in Figure 2.Figure 2
SDS algorithms in gradient field.Gradient is starting from pointP1, linear SDS starts from point P2, nonlinear SDS starts from point P3, and SDS-statistic gradient starts from point P5 (vector pairs marked dash opposes failure tests). The gradient-fastest descend is presented from starting point P1. SDS fastest descend is not presented in Figure 2 (it is similar to linear SDS from P2).The random vectorξn is created on n-dimensional sphere rn;|rn|=1, or on n-dimensional cube; n is system dimension in Euclide space E. A presentation is shown in Figures 1(a), 1(b), and 1(c) for n=3.The SSA properties used for ranking with other competitive methods (gradient, fastest descend, Gaus-Zeidel, and scanning and others) are(i)
local: describing algorithm in i-step,(ii)
nonlocal: describing algorithm from start to the final stage of optimization.The main property is dissipation, that is, losses in one step of search and displacement on hipper-surface of the functionQ. The said properties describe the algorithm convergence, that is, the method itself.The convergenceKn is defined by expression:(15)Kn=M(Ni)M(ΔQ¯i),that is, ratio of math expectation of test number Ni in i-iteration over relative change of ΔQi¯ (criteria function Q) [22].Reciprocal value of relation (15) gives an average value q of Q in i-step of searching.The local SDS property includes probability of failing step of searching:(16)p=PQx<Qx+Δx,where x is vector of initial state and Δx is working step of state change in x+Δx. The probability of failing step is rather important regarding choosing of algorithm for real process optimisation.So, the choice of optimal algorithm from local property point of view guides to compromise of choice of three:(17)(K,q,p).Besides local properties the relevant ones are properties describing the search in whole [23] which are(i)
total number of steps, that is, number of effective stepsN,(ii)
accuracy, that is, allowed error the procedure is endingε or relative error δ in %.The SDS option choice is guided by the above indicated local characteristics properties(K,q,p) as well as nonlocal ones (N,ε).It is necessary also to observe that the dispersionD(ΔQ¯) of ΔQ¯ could be useful in some statistical estimation of searching procedure properties. Also, it is necessary to notice that dispersion depends on how the vector ξ is generated (hipper sphere or cube) [23].Finally, it is worthwhile to mention that the choice of algorithm is subject to request to have procedures (identification and optimization) effective, having in minds that system model or criteria function is nonlinear. Substantially nonlinear process models so, described by (1) not having linear approximation which simulates original behavior (1) within narrow range of variables and parameters change against defined error limits. SDS methods are effective in this case so [22, 23].


## 2.2. Modified Nonlinear SDS: The Definition

Good properties of SDS as per (11) are simplicity, reliability, and applicability over nonlinear systems but slow convergence during the optimization procedure shows weak point. A rough explanation of nonlinear SDS of basic definition within gradient field Q is shown in Figure 2, with starting point P3. Comparing with regular gradient method this SDS becomes competitive when the system dimension is over n>12 [24].By increasing the system dimension the probability for acceptable tests decrease indicating that the target is reached after rather numerous tests and effective stepsN. The stochastic gradient is SDS having competitive performance during optimisation of nonlinear systems; however, the algorithm itself accumulates information and becomes fed up in numerical proceeding.The idea to have failed tests converted onto useful accumulated information guides toward the so-called modified nonlinear SDS. The previously said is shown in Figure2. For the 3 failed tests from starting point P4 it effects vector ξi,R to turn over toward the target under angle γ. So, if the accumulation of failed tests is J, between i and i+1 successful steps on hyper area of Q then:(18)∑j=1Jξj(i)=ξR(i,J);ortξR(i,J)=ζR(i,J).Now it is possible to form the next form of SSA search:(19)Δxi+1=αζi,ΔQi<0,-αζR(i,J)+αζi+1,ΔQi+1<0,where ζi+1=ortξi+1 is successful test after J failed, +αζR(i,J), ΔQj=Qj-Qj-1≥0; j=1,2,3,…,J, generate of accumulation information in MN-SDS algorithm and possibly being used so:(20)-αξji,max⁡+αξi+1,where ζi(j,max⁡)corresponds to max Qi(j).A modification defined by (18) and (19) is naturally linked on basically defined nonlinear SDS and further on will be referred to as MN-SDS. At certain extent MN-SSA possesses property of nonlinear algorithm, linear SDS with accumulation and extrapolation and stochastic gradient.Having in mind that MN-SDS explore accumulated information some sort of self-learning features are obtained (specifically when all data are memorized) leading to the conclusion that stochastic probability of testing depends on accumulated information:(21)P′=p′xw,where, memory vector, could be defined as(22)w=∫p′xwdx.This brings that searching is guided within the vicinity of the best probe or test [23]; a memory vector indicates adaptation features and can be calculated for l step like(23)wi(l)=kwi-1(l-1)+χζiΔQi(l-1),where(24)ΔQi(l-1)=Qi+1(l-1)-Qi(l-1),where k is coefficient of erasing and χ is coefficient of learning intensity or self-learning.Now the vector guidance toward the target is(25)ζl(sl)=ortw+ξ=w+ξw+ξ,where ζi(sl) is corrected direction by self-learning (sl) on l-step; l=1,2,3,…,L, L≤J, “steps of accumulation” (J failed test after i step).In practice the optimisation starts without self-learning option. In that sense MN-SDS regardless of the range of accumulated information of failed tests not memorized enables sampling of those most useful ones.


## 3. Theoretical Importance for Application of MN-SDS on ANN

### 3.1. Basic Theoretical Notes of MN-SDS

The main result achieved by MN-SDS is improved convergence compared to that of nonlinear SSA of basic definition (Section2.1, relation (11)).For the SSA purposes uniform distribution of random numbersξi within chosen interval [a,b] of real axes is used. Most often it is interval [-1,+1], Figure 1. More complex distribution produces difficulties in numerical calculation [25].By increasing of system dimension the probabilityP(ξi) of each testing is decreased; ξi∈ξ,ξ=ξ1,ξ2,…,ξi,…,ξnT. The consequence of the previously said is less and less successful testings producing more steps toward the target.The idea to use failed SDS tests for increase of efficiency of nonlinear SDS initiated creation of the so-called modified nonlinear SDS (MN-SDS). In fact failed tests give accumulated information for increasing of N-SDS convergence. Rearranged iterative procedure for minimization of criteria functionQ mentioned in Section 2.2 brought rather effective procedure, that is, MN-SDS.The MN-SDS procedure provides acceptable choice referring to the set of three characteristics(K,q,p), number of steps N, and set-up error ε during aiming the target.The convergenceKn for N-SDS algorithm with prediction [23, 26] for system dimension n is given by the following relations:(26)Kn=π(n-1)Γ(n-1/2)Γ(n/2).This is shown like curve ABC in Figure3. The curve ABC is the boundary case for search with MN-SDS algorithm when after i-step exist only one failed test [24]. In the same Figure 3 it is given Kn for gradient method with single testing (AD) and in case pair (2n) testing (AE). Sum feathers in Figure 3 has been taken over reference [24].Figure 3
Comparability of convergence MN SDS and N-SDS with gradient.Nonlinear SSA (N-SDS) of basic definition (the curveAND in Figure 3) has better local convergence than gradient method when system dimension is n>12. MN-SDS confirms that SS algorithms overcome gradient method in efficiency whenever system dimension rapidly increases; n>3. SDS numerical procedures are simple, specifically when P(ξ) is uniform distribution by system dimension increase. In case very large number of dimensions the concerns in term of efficiency between SDS methods and the gradient method changed in the favor of the gradient [24]. MN-SDS its advantages over gradient procedures retains much longer than most SDS algorithms.It is worthwhile to recognise when to use SDS or some other optimisation method. In that sense in Figure4 is shown that gradient procedure is more effective within the vicinity of the optimum (Figure 4 has been taken over on [26]). The MN-SDS in certain situation incorporate features of nonlinear SSA and stochastic gradient (SG-SDS). The stochastic gradient is based on accumulation of numerous information and as such it targets optimum with the same efficiency as regular gradient method. Regular-determined gradient method could be considered as specific case of stochastic gradient [26].Figure 4
The applicability areas of gradient and SDS methods.It is necessary to mention that random numbers generator should pass more strict tests whenever the system dimension is large [25]. Figure 5 shows diagram of MN-SDS numerical procedure. The random vector generator ξ is shown as outside device just to point necessity of such generator. The said generator is SW-software solution within the computer system in form an application package [15].Figure 5
Diagram of procedure for computer processing of MN-SSA.


### 3.2. SDS Implementation for ANN Training

Hereinafter an implementation of MN-SDS over multilayer ANN with feedforward information flow through network will be considered (FANN).The FANN properties (well adopted by researchers and designers) enable wide range of ANN to be transformed into FANN.The SDS, that is, MN-SDS, can be applied on both FANN model forms, oriented graph and matrix form.For MN-SDS the first mentioned form is more practical having in mind available heuristic options offering a more efficient MN-SDS.In this part of the paper and onward a multilayer FANN will be observed as shown in Figure6.Figure 6
The FANN a general type (a) and used model of neuron (perceptron) (b) whereu0=1.
(a)
(b)After adjustment of symbols (in expression (19)) for an ANN, the following form is obtained for MN-SDS: (27)Δωi+1=αζi,ΔQi=Qi-Qi-1<0,αζj(i),ΔQj=Qj-Qj-1≥0,-αζRi,J+αζi+1,ΔQi+1=Qi+1-Qi<0,where Δωi+1areincrement vector ωi in optimization process iterationand ζi are random vectors of uniform distribution. The cost function will stay Q observing that now Q=Q(t,y(L)).The vector of parameterω is changed in pace with iterative optimization procedure as follows:(28)ωi+1=ωi+Δωi.The vectorω dimension is determined by the ANN level complexity and also complexity of an optimization procedure, that is, training:(29)dim⁡ω=dim⁡ξ=Nω;ortξ=ζ,ζ=1,where ξ vector has coordinates as random vector i(30)ξ=ξ1,ξ2,…,ξNωT,where Nω is set of all parameters in parametric space, while T means transposition into column matrix. Gradient method and so the backpropagation error (BEP) method use this iterative relation:(31)ωi+1=ωi-α·∇Qωi.Stochastic methods (also including MN-SDS) instead of∇Qωi use random vector ξ which is heuristically chosen enabling iterative parametric optimization with certain efficiency measured by convergence range. Previously the rank of MN-SDS compared to SSA of SDS type as well as other gradient procedures has been set up.An application of MN-SDS on the FANN (Figure6) is used known linear interaction function netj(l) and corresponding output yj(l):(32)netj(l)=∑iωij(l)yj(l-1),yj(l)=gj∑iωijlyjl-1,where is:(i)
ωijl—components of weights of vector ω,(ii)
i=1,2,3,…,Nνl-1—neurons in layer l-1,(iii)
j=1,2,3,…,Nνl—neurons in layer l,(iv)
l=0,1,2,…,L—all layers in network,(v)
Nνl-1 and Nνl—number of neurons for adjacent layers.Application MN-SDS algorithm of the training FAAN involve the introduction of random vectorξ of the same size as vector ω:(33)dim⁡ω=dim⁡ξ=∑l∑νNνl-1Nνl+∑lNνl.The correspondents between components ωijl and ξijl must be make before of feedward numerical procedure:(34)ωijl⟶ξijl;Δωijl⟶αortξijl.For the each training pair pk=(uk,tk) is made training; that is, minimization of criteria function Qk. In the set of training there is K pairs; pk∈P, k=1,2,3,…,K. If the training of network performed for entire set P, then it is achieved an epoch of training.If in the FANN there are more outputs than only one, previously it must be from an error for one output:(35)εsk(L)=tsk-yskL.After this it can to form criteria function Qk:(36)Qk=12∑sεskL2=12∑tsk-yskL2,where it is: s=1,2,3,…,m and k=1,2,3,…,K.The increment ofΔωijl, by weights of the layer l, for one step iteration can be presented as follows:(37a)Δωijl=α·ortξijl;inMN-SDSmethod,(37b)Δωijl=-α∂Qk∂ωijl=-α∂Qk∂netj(l)∂netj(l)∂ωijl,inBPEmethod(37c)δjl=∂Qk∂netj(l),the indexes denoted the same in the previous expressions.δ
j
l is local gradient, an important characteristic in BPE method [27, 44].δ
j
l can be calculated via SDS procedures (with application the relations (37a) and (37b)) but only through MN-SDS or SDS gradient which gives the BPE primal version [27].


### 3.3. SDS Algorithms and FANN Synthesis

Synthesis of ANN is engineering design. An ANN design starts with set-up of an initial architecture based on experience and intuition of a designer. In this Section3.3 it was presented the formally recommendations which are relatively good orientation in design of FANN.An ANN architecture is determined by the number of inputs and outputs, neurons, and interlinks between them and biases a perceptron is a neuron with adjustable inputs and activation function not necessary to be of step type-threshold.Experience shows, that is quite clear that for solution of complex problem it is necessary to create a complex ANN architecture.Working with SDS various experiences have confirmed the previously noticed. The SDS methods are more efficient comparing to numerous known ones specifically when complex optimization problems are in question specifically when complexity is measured by system dimensions. Having in mind the significance of multilayer of an FANN hereinafter the structure shown in Figure6 will be considered.It is worthwhile to mention that successful optimization process of an FANN does not mean that it would have necessary features: first of all required capacity and properties for generalization [29].Capacity features (C) is one of ANN properties to memorize certain information during the process of training, that is, learning.An FANN property of generalization is basic paradigm of its performance. In fact it is an essential property approving that “network has learnt” and should provide valid answers to inputs not within training pairs. In other words testing data should have the same distribution as a set of training pairs.The synthesis issue is an open problem. In [29, 30] some theoretical results have been shown mostly for three layer networks processing binary data. Some researches were working to attempt and expand implementation on three-layer (and multilayer) processing analogue information what have brought the so-called universal approximator [31–33].The name universal approximator is linked to a three-layer network having perceptrons in hidden layer with nonlinear activation function (type sigmoid) and perceptrons at outputs with linear activation function (Figure7).Figure 7
Basic FANN architecture of universal approximation.By introducing the following designations, for hidden neuronsNνh, for other neurons Nν, for interlinks-synapses Nω, for threshold Nθ, then by simplifying the theoretical results of [29, 31–33] certain indicators are obtained as relatively good orientation for creation of a FANN architecture.When a starting network structure has been set up, then its dimension is(38)NNd=Nω+Nθ.The range of training pairs samples forG level of generalization above 90% (expressions (39), (40) and (41) represent compress (in simple form) of ideas in references [29, 31–33]) is (39)Np=NNd×10=Nω+Nθ10.The FANN ability to memorize and capacityC are determined by relation(40)C=Nωmif the following condition is fulfilled(41)n+Nνh≫m;n and m are number of network inputs and outputs respectively (Figure 7).In case of collision between ANN dimension and required training samplesNp changing of NNd is required. It is point out that Nν+Nνh is in collision with generalization G. For training under the same conditions better generalization is got for networks with less neurons number. The aforesaid indicates that ANN capacity is reduced [34].Previous consideration with fix configuration during training procedures is characterized as static.There are some methods approaching to optimization dynamically; during training procedures network structures are changed such as cascade correlation algorithm [35], tiling algorithm [36], dynamic learning by simulated annealing [37], and others. Most complex one is dynamic training by simulated annealing [37, 38]. The aforesaid method resolves numerous FANN synthesis problems.Training with MN-SDS is processed through forward phase. Numerical procedure is simple and gives enough information SDS shows on dynamically approach in training, optimization and syntheses of artificial neural networks.


### 3.4. Examples

Example 1 (searching by matrix type).
This example is concerned with the theory and system control [39]. Here is presented to show when the SS model system works in matrix form as well as differences in the efficiency of algorithms N_SDS and MN-SDS.
The linearized multivariable system described in internal presentation (see relation (1) and (2) (in Section 2.1)) is as follows:(42)dxdt=Ax+Bu=0,y=Cx+Du,where A,B,C,D are matrix of parameters of system; x,y,u are vectors with corresponding dimensions.
If the static of the real system is described by matrix form,
In the expression (42)  dx/dt=0, then the reduced equations describing the steady-static behavior of the system is:(43)y=C′v+D′u,where C′ and D′ are the matrix of parameters corresponding to the static of the observed system.
The relation (43) in the developed form can represent energy network with passive consumers:(44)y1y2y3y4=c11c12c13c14c15c21c22c23c24c25c31c32c33c34c35c42c43c44c45c41v1v2v3v4+d11d12d21d22d31d32d41d42u1u2,where [y], [v], and [u] are sizes that can be traced through the check in checkpoint; [v] includes certain set of all measurable sizes in the system (42).
Linear forms type (44) are more complex in numerical processes of form type:(45)y=Cv;matrixCistypem×m.Checks consistency and livelihoods solutions [40]:(46)det⁡Cm×m≠0is inapplicable to form (44).
Having in mind the previously indicated, numerical experiment for coefficients{cij} and {dik}; i=1,2,3,4; j=1,2,3,4,5; k=1,2 identification has been created. In fact the procedures with SSA algorithm have been used. The data of {yi} were collected by changing variables {vj} and {uk} that are in Table 1.
For identification of{cij} and {dik} matrix searching is used with random matrix [ξ,], generated on hipper sphere in E-space with n=28 [25].
The current values of the sizesy, v, and u monitored through the checkpoint. At certain intervals performs registration thereof. With a certain number of the same is formed by a collection of required data (Table 1). A series of 28 randomly selected numbers filled matrix C′ and D′; in each iteration of optimization process. The use of any of the algorithms N-SDS or MN-SDS requires the formation of error ε: ευ,i=yi,υξ-yi,υM, i=1,2,3,4; υ=1,2,3,…,N; N is number of last iteration υ, and then coresponding function criteria Qυ is:(47)Qυ=12∑iευ,i2=12∑iyi,υξ-yi,υM2,where yi,υξ are components [y] for the random selection parameters and yi,υM required measurement values.
For step iteration usedα=0.1; 0,001; and 0,0001 respectively. The initial random parameters of this procedure are(48)c0,ij4×5=0,1000,2010,3020,4030,5000,5000,5030,5010,5000,5041,0101,0051,0091,0102,0001,0121,0111,0062,0182,001;d0,ik4×2=0,1001,0121,0130,1000,1011,0080,0010,000.
The final results after random procedure with N-SSA are(49)cN,ij4×5=0,2130,0510,5240,9111,0040,3010,4010,5120,7010,8160,9311,6310,6221,7431,3201,5200,7210,8310,9970,802;dN,ik4×2=0,3010,8340,8240,5010,3130,5420,1160,311.
The accuracy ofδ=1% calls to have α=0,0001 and number of steps lN≥8000. There is no noise in system.
Implementation of MN-SDS is possible after transforming of equation system (44) into matrix form:(50)y1y2y3y4=v1c11c21c31c41+v2c12c22c32c42+v3c13c23c33c43+v4c14c24c34c44+v5c15c25c35c45+u1d11d21d31d41+u2d12d22d32d42.
By application of MN-SDS method someN= 2,156 steps are needed indicating that some 4 times less steps are required for the same accuracy of δ=1%.Table 1
Collection data of variables{yi}, {vj}, {uk}.
ν
1
2
3
4
5
6
7
{
y
i
}
1,3
2,1
2,9
2,0
2,3
−0,8
−0,9
1,1
3,9
6,0
2,8
2,1
−0,6
−1,0
7,0
7,0
2,7
2,6
7,0
+0,8
−3,0
7,0
1,9
1,1
3,0
3,0
+2,0
−4,0
{
v
j
}
1,0
−1,0
2,0
1,0
−1,0
1,0
0,0
1,0
−1,0
2,0
1,0
1,0
0,0
0,0
0,0
−1,0
2,0
1,0
2,0
−1,0
0,0
0,0
1,0
2,0
1,0
1,0
0,0
−1,0
−1,0
0,0
2,0
1,0
0,0
1,0
−1,0
{
u
k
}
−1,0
+1,0
−1,0
+1,0
0,0
−1,0
0,0
+1,0
0,0
+1,0
0,0
+1,0
−1,0
0,0Example 2 (training of multilayer perceptron).
This example is linked to treatment of training of perceptron related to “XOR” logic circuit by using of SDS procedures. The said example had an important role in R&D works in the field of artificial neural network. In fact Minsky and Pappert (1969) confirmed that perceptron cannot “learn” to simulate XOR logic circuit and not to expect much of ANN [41].
If from XOR true table training pairs are formed(000,011,101,110) then it is possible to obtain similar conclusion as Minsky and Pappert. If we observe the definition of neuron (perceptron) as per McCulloch and Pitts [42] for neuron with two inputs,(1)
o·ω1+o·ω2<θ; or 0<θ(t-ois0),(2)
o·ω1+1·ω2>θ; or ω2>θ(t-ois1),(3)
1·ω1+o·ω2>θ; or ω1>θ(t-ois1),(4)
1·ω1+1·ω2<0; or ω1+ω2<0(t-ois0),where t-o are training outputs: 0,1,1,0.
It is obvious that relations (2), (3), and (4) are excluding each other. A perceptron cannot be trained to simulate XOR logic circuit.
The aforesaid is understood as obstacle “in principum”.
The previously indicated somehow interrupted further development of ANN more than 20 years up to works of Hopfield [43] and others.
The problem has been overcome by studying of multilayer and recurrent ANN properties as well as creation of advance numerical procedures for training of the same [27].
This example incorporates the results of training having at least one hidden layer of neurons which could be trained to simulate XOR logic circuit.
Figure8(a) shows an FANN configuration having one hidden layer of two neurons and one neuron at the ANN output. That ANN can learn to simulate XOR logic circuit.
For training pairs two options can be used (Figures8(b1) and 8(b2)):(51)Pkuk,tk⟶P10,0;0,P20,1;1,P31,0;1,P41,1;0,Pk′uk′,tk′⟶P1′-1,-1,-1,P2′-1,1;1,P3′(1,-1;1),P4′(1,1;-1).
Further on results realized with training pairsPk(uk,tk) shown in Figure 8(b1) shall be used. It is necessary to observe that for some of variables should have fixed values since do not contribute in solving the problem:(52)U01=U02=0,ω01=ω02=0,U03=U04=U05=-1.
At the time of training it was shown thatω12, ω13, ω23 and ω24 have very little variations near number 1 and it is possible to used(53)ω12=ω13=ω23=ω24=+1.
Values of all otherω0j(l) and ωij(l) are changeable parameters where l is indication of neuron layer. By that the dimension of the random vector ξ is decreased. For the first training pair and activation function of logistic type(54)g1(x)=1[1+exp⁡(-x)]training will be performed through BPE method [27, 28] and MN-SDS; x presents a linear function interaction of neuron inputs.
The criteria function for both cases is(55)Qk=εk=12∑s=1myskNL-tsk2.
The SSA iterative procedures were used to minimizeQk; the results are presented in Figure 9(a). The cost functions Qk are formed for the same training pairs and initial values, and ponderation is done against the highest value of one of Qk.
In Figure9(a) diagram has been singed with number 4, is criteria function Q of training by NN-SDS method.
The results of training are shown so for a step activation function (threshold):(56)g2(x)=1,x≥00,x<0.
The method BPE is disabled sinceg2(x) is not differentiable. For application BPE in this case it must be to approximate g2(x) with logistic function S(x)=1/(1+exp⁡(cx)); c≥100. The process optimization is finished after more than 600 iterations (Figure 9(b)). The final results for MN-SDS of the ponderation Q as shown in Figure 9(b); training was done with set. SS procedure with MN-SDS has been initiated with random parameters (weights): ω0,03=1.01; ω0,04=0.55; ω0,05=0.74; ω0,35=0.19; ω0,45=0.57.
Finally results afterN>300 iterations are ω03,N=1.50, ω04,N=0.99, ω05,N=0.51, ω35,N=-1.98, ω45,N=0.98. with relative error of 2%.
The random vector of forward propagation is with dimensionn=5 is:(57)ξ≡ξ1,ξ2,ξ3,ξ4,ξ5T.
Let us refer to an example when an activation functiong2(x) is given by relation (56), with training pairs P′(uk′,tk′). Then it shows that a training procedure depends on choice of training pairs as well. The minimization process of such case is done rather faster (Figure 9(b)).
BPE methods implementation was made with known procedures [27, 28]. BPE is used as comparative method.
In this paper MN-SDS and BPE use ECR (procedures error correction rules) which are more convenient for “online” ANN training [45].
Whenever the training coefficient (α) is small then ECR and GDR procedures provide the same results for parameters estimation within set-up range of an acceptable error [44, 45]. GDR (gradient rules) is called “batch” optimization procedure.Figure 8
Optimization, for example, training of multilayer ANN; XOR problem.
(a)
(b)Figure 9
Cost function; (a) forg1(x)=1/(1+exp⁡(-x)), (b) for g2(x) step function; BPI optimization is finished after 600 iterations.
(a)
(b)Example 3 (synthesis FANN to approximate model FSR).
In this example some of theoretical R&D results of [29–33] are applied.
The real problem is approximated model of an ANN training program related on technical term for process in reactor in which the concentrate heated to powder so that will behave like a fluid [46, 47].
When the said FSR is to be tempered either the first time or after service attempts, there is a program of tempering to temperatureT as per diagram (see Figure 10). After reaching working temperature it is maintained by control system. The tempering to T=600±25°C is to be completed within 130 min (range AB). In Figure 10 the whole cycle is within 240 min. The FSR was under operation 40 minutes (range BC). Due to some maintenance problem the FSR is often shut down (point C). Before next campaign the FSR should be cooled down (range CD) for maintenance purposes and afterwards the tempering operation is repeated.
The mapping in real conditions have not enough data to make FANN (Figure11, [46]). There is not enough, to model approximations over ANN that learns. Overall number of pairs (tk,Tk) collected from the said diagram (Figure 10) is 80 pairs; sampling period is 3 min. More frequent sampling is not realistic. The task is to determine a FANN, which is to give the best approximate model of the FSR within required accuracy. The relative error of correct response should be more than 3% of obtained measuring data.
On Figure12 is given a starting architecture of FANN NN1=(1-10-10-1): with one input and one output, with 10 neurons in the first and in second layers. Only neurons in the second layer has non-linear activation function. All neurons has a linear interaction function.
Application of the relations ((39)–(41)) of Section 3.2 on NN1 gives:NNd=120+21=141 interlinks between unknown parameters and biases.Np=141×10=1410 required training pairs for generalization G>90%!,C=120 memorized information under condition that n+Nuh≫m, 11≫1; condition is satisfied.
Based on the aforesaid there is overnumbered required training pairs(1410≫80).
Basic moments in conjunction approximation model FSR through the use of MN-SDS in the training of FANNNN1=(1-10-10-1) are that(i)
assignment trainingNN1  does of 80 pairs of training is achieved through the diagrams on Figure 10 that define Table 2,(ii)
random vectorξ replaces ωij and ω0j at expression (32), so dim⁡ξ=141, unknown parameters,(iii)
feedforward phase in layers for each training pair achieved numerical value fornetj(l) and yj(l),l=1,2,3, to the expression (32),(iv)
cost function for sequence for each training parQk, k=1,2,3,…,80, was calculated with the expression (36). Figure 13 presents the trend of Qk for NN1,(v)
the procedure optimization, that is, training for FANNNN1 takes more than 1000 iterations.
Due to the big volume of numerical data in the paper, we will not list all the details.
Out of the two ANN under training with same training samples more strong generalization has network with less number of neurons.Table 2
Data defining program warming FSR (t, T).
t [min]
T [°C]
0
20
10
20
20
50
30
50
40
100
50
100
70
150
80
250
90
250
100
350
105
350
110
450
115
450
120
550
125
600
130
600
140
600
160
600
170
600
175
425
185
325
205
150
215
110
225
75
235
60
240
50Figure 10
Plan introducing FSR in operative mode (AB), work (BC), possibly down time (CD).Figure 11
The real mapping data of FSR at operative environment.Figure 12
The initial structure of FANN for approximation model’s tempering of FSR.Figure 13
Trend cost functions multilayers FANN’sNN1 and NN5.That is the reason to refer to less complex network structures:N
N
2
=
(
1
-
5
-
5
-
1
), NN3=(1-3-3-1) and NN4=(1-2-2-1).StructureNN4 eventually could be trained and have certain generalization possibility since for NN4NNd=13 interlink, that is, unknown parameters,Np=130 required training pairs for G>90%; 130>80 !,C=Nω/m=8, condition n+Nνh≫m; 3>1, barely accepted.More acceptable solution isNN5=(1-3-1), although it presents a bared architecture FANN, since for NN5NNd=10, dimension of NN5,Np=100; there are not 100 but only 80 training pairs,C=6, condition n+Nνh≫m; 4≫1, acceptable.It has been possible to continue to a minimized architectureNN6=(1-2-1), but choice of NN5=(1-3-1) provides better performance of a hidden layer.The closest solution is the FANNNN5=(1-3-1), Figure 14. In hidden layer this network has 3 perceptrons with sigmoid type for activation function.Figure 14
Oriented graph ofNN5=(1-3-1).Having in mind that determining of an ANN, that is, FANN architecture is always open problem then estimation of an adopted structure is suitable after the final numerical result including validity test of a network training.Theoretical results for universal approximator are derived for nonlinear activation, function of hidden neurons of typeS(x)=1/(1+exp⁡(-x)).Since2S(x)-1=tanhx/2 bring an option to use tanh function for a FANN model approximation.Application the relation ofnetj(l) and yj(l) in the expression (32), for structure on Figure 14, NN5=(1-3-1) in general form:(58)Tt=ω25tanhω12t+ω02+ω35tanhω13t+ω03+ω45tanh(ω14+ω04)+ω05;represents an approximation model of the temperature regime of FSR.Here will be presented numerical data and the results of network training forNN5 by MN-SDS only.On the beginning of the numerical procedure for practical reasonst and T should be reduced 100 times.The symbolic presentation of the vector unknow parametersω, in the network NN5, at the beginning (ω0) and end of training procedure (ωN) is given by:(59)ω0=ω12(0),ω13(0),ω14(0),ω25(0),ω35(0),ω45(0),ω02(0),ω03(0),ω04(0),ω05(0)TωN=ω12(N),ω13(N),ω14(N),ω25(N),ω35(N),ω45(N),ω02(N),ω03(N),ω04(N),ω05(N)T.The initial random value of the parameters ω0 is:(60)ω0=0.1576,0.9572,0.4218,0.8003,0.7922,0.9706,kkk0.9706,0.4854,0.1419,0.9157T.Random vectorξ in this case is(61)ξ=ξ1,ξ2,ξ3,…,ξ10T.Behind the training networkNN5 by algorithm MN-SDS after 700 iterations (N>700;Qk,N<ε,ε=10-4) vector ωN of unknown parameters is given by(62)ωN=1.705,3.847,11.61,3.159,-3.597,0.731,klk-1.670,-7.021,-13.23,02939T.Previous data characterize the approximation process model tempering temperature of FSR (58), overtraining FANN NN5 by the MN-SDS algorithm, with 5% of the relative error. Trend of Qk for NN5 is given on Figure 13.Some responses to the test inputs for checking the validity of the obtained model deviate a large error. Counting these to interrelate (sharing) the total number received relatively rough estimate of generalization capabilities appropriate network. Based Figure10 test values have special graphic symbols (O, X for MN-SDS and Δ and ∇ for BPE). For a training set of 80 pairs generalization G ability of the network NN5 is about 70%. For the network NN1 it is about 20%. Previous values obtained training using MN-SDS.Application BPE method gave the following values of generalization: about 70% for networkNN5 and below 20% for the network NN1.The previous presented FSR model could be used in more sophisticated option as an intelligent process monitoring.


## 3.1. Basic Theoretical Notes of MN-SDS

The main result achieved by MN-SDS is improved convergence compared to that of nonlinear SSA of basic definition (Section2.1, relation (11)).For the SSA purposes uniform distribution of random numbersξi within chosen interval [a,b] of real axes is used. Most often it is interval [-1,+1], Figure 1. More complex distribution produces difficulties in numerical calculation [25].By increasing of system dimension the probabilityP(ξi) of each testing is decreased; ξi∈ξ,ξ=ξ1,ξ2,…,ξi,…,ξnT. The consequence of the previously said is less and less successful testings producing more steps toward the target.The idea to use failed SDS tests for increase of efficiency of nonlinear SDS initiated creation of the so-called modified nonlinear SDS (MN-SDS). In fact failed tests give accumulated information for increasing of N-SDS convergence. Rearranged iterative procedure for minimization of criteria functionQ mentioned in Section 2.2 brought rather effective procedure, that is, MN-SDS.The MN-SDS procedure provides acceptable choice referring to the set of three characteristics(K,q,p), number of steps N, and set-up error ε during aiming the target.The convergenceKn for N-SDS algorithm with prediction [23, 26] for system dimension n is given by the following relations:(26)Kn=π(n-1)Γ(n-1/2)Γ(n/2).This is shown like curve ABC in Figure3. The curve ABC is the boundary case for search with MN-SDS algorithm when after i-step exist only one failed test [24]. In the same Figure 3 it is given Kn for gradient method with single testing (AD) and in case pair (2n) testing (AE). Sum feathers in Figure 3 has been taken over reference [24].Figure 3
Comparability of convergence MN SDS and N-SDS with gradient.Nonlinear SSA (N-SDS) of basic definition (the curveAND in Figure 3) has better local convergence than gradient method when system dimension is n>12. MN-SDS confirms that SS algorithms overcome gradient method in efficiency whenever system dimension rapidly increases; n>3. SDS numerical procedures are simple, specifically when P(ξ) is uniform distribution by system dimension increase. In case very large number of dimensions the concerns in term of efficiency between SDS methods and the gradient method changed in the favor of the gradient [24]. MN-SDS its advantages over gradient procedures retains much longer than most SDS algorithms.It is worthwhile to recognise when to use SDS or some other optimisation method. In that sense in Figure4 is shown that gradient procedure is more effective within the vicinity of the optimum (Figure 4 has been taken over on [26]). The MN-SDS in certain situation incorporate features of nonlinear SSA and stochastic gradient (SG-SDS). The stochastic gradient is based on accumulation of numerous information and as such it targets optimum with the same efficiency as regular gradient method. Regular-determined gradient method could be considered as specific case of stochastic gradient [26].Figure 4
The applicability areas of gradient and SDS methods.It is necessary to mention that random numbers generator should pass more strict tests whenever the system dimension is large [25]. Figure 5 shows diagram of MN-SDS numerical procedure. The random vector generator ξ is shown as outside device just to point necessity of such generator. The said generator is SW-software solution within the computer system in form an application package [15].Figure 5
Diagram of procedure for computer processing of MN-SSA.


## 3.2. SDS Implementation for ANN Training

Hereinafter an implementation of MN-SDS over multilayer ANN with feedforward information flow through network will be considered (FANN).The FANN properties (well adopted by researchers and designers) enable wide range of ANN to be transformed into FANN.The SDS, that is, MN-SDS, can be applied on both FANN model forms, oriented graph and matrix form.For MN-SDS the first mentioned form is more practical having in mind available heuristic options offering a more efficient MN-SDS.In this part of the paper and onward a multilayer FANN will be observed as shown in Figure6.Figure 6
The FANN a general type (a) and used model of neuron (perceptron) (b) whereu0=1.
(a)
(b)After adjustment of symbols (in expression (19)) for an ANN, the following form is obtained for MN-SDS: (27)Δωi+1=αζi,ΔQi=Qi-Qi-1<0,αζj(i),ΔQj=Qj-Qj-1≥0,-αζRi,J+αζi+1,ΔQi+1=Qi+1-Qi<0,where Δωi+1areincrement vector ωi in optimization process iterationand ζi are random vectors of uniform distribution. The cost function will stay Q observing that now Q=Q(t,y(L)).The vector of parameterω is changed in pace with iterative optimization procedure as follows:(28)ωi+1=ωi+Δωi.The vectorω dimension is determined by the ANN level complexity and also complexity of an optimization procedure, that is, training:(29)dim⁡ω=dim⁡ξ=Nω;ortξ=ζ,ζ=1,where ξ vector has coordinates as random vector i(30)ξ=ξ1,ξ2,…,ξNωT,where Nω is set of all parameters in parametric space, while T means transposition into column matrix. Gradient method and so the backpropagation error (BEP) method use this iterative relation:(31)ωi+1=ωi-α·∇Qωi.Stochastic methods (also including MN-SDS) instead of∇Qωi use random vector ξ which is heuristically chosen enabling iterative parametric optimization with certain efficiency measured by convergence range. Previously the rank of MN-SDS compared to SSA of SDS type as well as other gradient procedures has been set up.An application of MN-SDS on the FANN (Figure6) is used known linear interaction function netj(l) and corresponding output yj(l):(32)netj(l)=∑iωij(l)yj(l-1),yj(l)=gj∑iωijlyjl-1,where is:(i)
ωijl—components of weights of vector ω,(ii)
i=1,2,3,…,Nνl-1—neurons in layer l-1,(iii)
j=1,2,3,…,Nνl—neurons in layer l,(iv)
l=0,1,2,…,L—all layers in network,(v)
Nνl-1 and Nνl—number of neurons for adjacent layers.Application MN-SDS algorithm of the training FAAN involve the introduction of random vectorξ of the same size as vector ω:(33)dim⁡ω=dim⁡ξ=∑l∑νNνl-1Nνl+∑lNνl.The correspondents between components ωijl and ξijl must be make before of feedward numerical procedure:(34)ωijl⟶ξijl;Δωijl⟶αortξijl.For the each training pair pk=(uk,tk) is made training; that is, minimization of criteria function Qk. In the set of training there is K pairs; pk∈P, k=1,2,3,…,K. If the training of network performed for entire set P, then it is achieved an epoch of training.If in the FANN there are more outputs than only one, previously it must be from an error for one output:(35)εsk(L)=tsk-yskL.After this it can to form criteria function Qk:(36)Qk=12∑sεskL2=12∑tsk-yskL2,where it is: s=1,2,3,…,m and k=1,2,3,…,K.The increment ofΔωijl, by weights of the layer l, for one step iteration can be presented as follows:(37a)Δωijl=α·ortξijl;inMN-SDSmethod,(37b)Δωijl=-α∂Qk∂ωijl=-α∂Qk∂netj(l)∂netj(l)∂ωijl,inBPEmethod(37c)δjl=∂Qk∂netj(l),the indexes denoted the same in the previous expressions.δ
j
l is local gradient, an important characteristic in BPE method [27, 44].δ
j
l can be calculated via SDS procedures (with application the relations (37a) and (37b)) but only through MN-SDS or SDS gradient which gives the BPE primal version [27].


## 3.3. SDS Algorithms and FANN Synthesis

Synthesis of ANN is engineering design. An ANN design starts with set-up of an initial architecture based on experience and intuition of a designer. In this Section3.3 it was presented the formally recommendations which are relatively good orientation in design of FANN.An ANN architecture is determined by the number of inputs and outputs, neurons, and interlinks between them and biases a perceptron is a neuron with adjustable inputs and activation function not necessary to be of step type-threshold.Experience shows, that is quite clear that for solution of complex problem it is necessary to create a complex ANN architecture.Working with SDS various experiences have confirmed the previously noticed. The SDS methods are more efficient comparing to numerous known ones specifically when complex optimization problems are in question specifically when complexity is measured by system dimensions. Having in mind the significance of multilayer of an FANN hereinafter the structure shown in Figure6 will be considered.It is worthwhile to mention that successful optimization process of an FANN does not mean that it would have necessary features: first of all required capacity and properties for generalization [29].Capacity features (C) is one of ANN properties to memorize certain information during the process of training, that is, learning.An FANN property of generalization is basic paradigm of its performance. In fact it is an essential property approving that “network has learnt” and should provide valid answers to inputs not within training pairs. In other words testing data should have the same distribution as a set of training pairs.The synthesis issue is an open problem. In [29, 30] some theoretical results have been shown mostly for three layer networks processing binary data. Some researches were working to attempt and expand implementation on three-layer (and multilayer) processing analogue information what have brought the so-called universal approximator [31–33].The name universal approximator is linked to a three-layer network having perceptrons in hidden layer with nonlinear activation function (type sigmoid) and perceptrons at outputs with linear activation function (Figure7).Figure 7
Basic FANN architecture of universal approximation.By introducing the following designations, for hidden neuronsNνh, for other neurons Nν, for interlinks-synapses Nω, for threshold Nθ, then by simplifying the theoretical results of [29, 31–33] certain indicators are obtained as relatively good orientation for creation of a FANN architecture.When a starting network structure has been set up, then its dimension is(38)NNd=Nω+Nθ.The range of training pairs samples forG level of generalization above 90% (expressions (39), (40) and (41) represent compress (in simple form) of ideas in references [29, 31–33]) is (39)Np=NNd×10=Nω+Nθ10.The FANN ability to memorize and capacityC are determined by relation(40)C=Nωmif the following condition is fulfilled(41)n+Nνh≫m;n and m are number of network inputs and outputs respectively (Figure 7).In case of collision between ANN dimension and required training samplesNp changing of NNd is required. It is point out that Nν+Nνh is in collision with generalization G. For training under the same conditions better generalization is got for networks with less neurons number. The aforesaid indicates that ANN capacity is reduced [34].Previous consideration with fix configuration during training procedures is characterized as static.There are some methods approaching to optimization dynamically; during training procedures network structures are changed such as cascade correlation algorithm [35], tiling algorithm [36], dynamic learning by simulated annealing [37], and others. Most complex one is dynamic training by simulated annealing [37, 38]. The aforesaid method resolves numerous FANN synthesis problems.Training with MN-SDS is processed through forward phase. Numerical procedure is simple and gives enough information SDS shows on dynamically approach in training, optimization and syntheses of artificial neural networks.


## 3.4. Examples

Example 1 (searching by matrix type).
This example is concerned with the theory and system control [39]. Here is presented to show when the SS model system works in matrix form as well as differences in the efficiency of algorithms N_SDS and MN-SDS.
The linearized multivariable system described in internal presentation (see relation (1) and (2) (in Section 2.1)) is as follows:(42)dxdt=Ax+Bu=0,y=Cx+Du,where A,B,C,D are matrix of parameters of system; x,y,u are vectors with corresponding dimensions.
If the static of the real system is described by matrix form,
In the expression (42)  dx/dt=0, then the reduced equations describing the steady-static behavior of the system is:(43)y=C′v+D′u,where C′ and D′ are the matrix of parameters corresponding to the static of the observed system.
The relation (43) in the developed form can represent energy network with passive consumers:(44)y1y2y3y4=c11c12c13c14c15c21c22c23c24c25c31c32c33c34c35c42c43c44c45c41v1v2v3v4+d11d12d21d22d31d32d41d42u1u2,where [y], [v], and [u] are sizes that can be traced through the check in checkpoint; [v] includes certain set of all measurable sizes in the system (42).
Linear forms type (44) are more complex in numerical processes of form type:(45)y=Cv;matrixCistypem×m.Checks consistency and livelihoods solutions [40]:(46)det⁡Cm×m≠0is inapplicable to form (44).
Having in mind the previously indicated, numerical experiment for coefficients{cij} and {dik}; i=1,2,3,4; j=1,2,3,4,5; k=1,2 identification has been created. In fact the procedures with SSA algorithm have been used. The data of {yi} were collected by changing variables {vj} and {uk} that are in Table 1.
For identification of{cij} and {dik} matrix searching is used with random matrix [ξ,], generated on hipper sphere in E-space with n=28 [25].
The current values of the sizesy, v, and u monitored through the checkpoint. At certain intervals performs registration thereof. With a certain number of the same is formed by a collection of required data (Table 1). A series of 28 randomly selected numbers filled matrix C′ and D′; in each iteration of optimization process. The use of any of the algorithms N-SDS or MN-SDS requires the formation of error ε: ευ,i=yi,υξ-yi,υM, i=1,2,3,4; υ=1,2,3,…,N; N is number of last iteration υ, and then coresponding function criteria Qυ is:(47)Qυ=12∑iευ,i2=12∑iyi,υξ-yi,υM2,where yi,υξ are components [y] for the random selection parameters and yi,υM required measurement values.
For step iteration usedα=0.1; 0,001; and 0,0001 respectively. The initial random parameters of this procedure are(48)c0,ij4×5=0,1000,2010,3020,4030,5000,5000,5030,5010,5000,5041,0101,0051,0091,0102,0001,0121,0111,0062,0182,001;d0,ik4×2=0,1001,0121,0130,1000,1011,0080,0010,000.
The final results after random procedure with N-SSA are(49)cN,ij4×5=0,2130,0510,5240,9111,0040,3010,4010,5120,7010,8160,9311,6310,6221,7431,3201,5200,7210,8310,9970,802;dN,ik4×2=0,3010,8340,8240,5010,3130,5420,1160,311.
The accuracy ofδ=1% calls to have α=0,0001 and number of steps lN≥8000. There is no noise in system.
Implementation of MN-SDS is possible after transforming of equation system (44) into matrix form:(50)y1y2y3y4=v1c11c21c31c41+v2c12c22c32c42+v3c13c23c33c43+v4c14c24c34c44+v5c15c25c35c45+u1d11d21d31d41+u2d12d22d32d42.
By application of MN-SDS method someN= 2,156 steps are needed indicating that some 4 times less steps are required for the same accuracy of δ=1%.Table 1
Collection data of variables{yi}, {vj}, {uk}.
ν
1
2
3
4
5
6
7
{
y
i
}
1,3
2,1
2,9
2,0
2,3
−0,8
−0,9
1,1
3,9
6,0
2,8
2,1
−0,6
−1,0
7,0
7,0
2,7
2,6
7,0
+0,8
−3,0
7,0
1,9
1,1
3,0
3,0
+2,0
−4,0
{
v
j
}
1,0
−1,0
2,0
1,0
−1,0
1,0
0,0
1,0
−1,0
2,0
1,0
1,0
0,0
0,0
0,0
−1,0
2,0
1,0
2,0
−1,0
0,0
0,0
1,0
2,0
1,0
1,0
0,0
−1,0
−1,0
0,0
2,0
1,0
0,0
1,0
−1,0
{
u
k
}
−1,0
+1,0
−1,0
+1,0
0,0
−1,0
0,0
+1,0
0,0
+1,0
0,0
+1,0
−1,0
0,0Example 2 (training of multilayer perceptron).
This example is linked to treatment of training of perceptron related to “XOR” logic circuit by using of SDS procedures. The said example had an important role in R&D works in the field of artificial neural network. In fact Minsky and Pappert (1969) confirmed that perceptron cannot “learn” to simulate XOR logic circuit and not to expect much of ANN [41].
If from XOR true table training pairs are formed(000,011,101,110) then it is possible to obtain similar conclusion as Minsky and Pappert. If we observe the definition of neuron (perceptron) as per McCulloch and Pitts [42] for neuron with two inputs,(1)
o·ω1+o·ω2<θ; or 0<θ(t-ois0),(2)
o·ω1+1·ω2>θ; or ω2>θ(t-ois1),(3)
1·ω1+o·ω2>θ; or ω1>θ(t-ois1),(4)
1·ω1+1·ω2<0; or ω1+ω2<0(t-ois0),where t-o are training outputs: 0,1,1,0.
It is obvious that relations (2), (3), and (4) are excluding each other. A perceptron cannot be trained to simulate XOR logic circuit.
The aforesaid is understood as obstacle “in principum”.
The previously indicated somehow interrupted further development of ANN more than 20 years up to works of Hopfield [43] and others.
The problem has been overcome by studying of multilayer and recurrent ANN properties as well as creation of advance numerical procedures for training of the same [27].
This example incorporates the results of training having at least one hidden layer of neurons which could be trained to simulate XOR logic circuit.
Figure8(a) shows an FANN configuration having one hidden layer of two neurons and one neuron at the ANN output. That ANN can learn to simulate XOR logic circuit.
For training pairs two options can be used (Figures8(b1) and 8(b2)):(51)Pkuk,tk⟶P10,0;0,P20,1;1,P31,0;1,P41,1;0,Pk′uk′,tk′⟶P1′-1,-1,-1,P2′-1,1;1,P3′(1,-1;1),P4′(1,1;-1).
Further on results realized with training pairsPk(uk,tk) shown in Figure 8(b1) shall be used. It is necessary to observe that for some of variables should have fixed values since do not contribute in solving the problem:(52)U01=U02=0,ω01=ω02=0,U03=U04=U05=-1.
At the time of training it was shown thatω12, ω13, ω23 and ω24 have very little variations near number 1 and it is possible to used(53)ω12=ω13=ω23=ω24=+1.
Values of all otherω0j(l) and ωij(l) are changeable parameters where l is indication of neuron layer. By that the dimension of the random vector ξ is decreased. For the first training pair and activation function of logistic type(54)g1(x)=1[1+exp⁡(-x)]training will be performed through BPE method [27, 28] and MN-SDS; x presents a linear function interaction of neuron inputs.
The criteria function for both cases is(55)Qk=εk=12∑s=1myskNL-tsk2.
The SSA iterative procedures were used to minimizeQk; the results are presented in Figure 9(a). The cost functions Qk are formed for the same training pairs and initial values, and ponderation is done against the highest value of one of Qk.
In Figure9(a) diagram has been singed with number 4, is criteria function Q of training by NN-SDS method.
The results of training are shown so for a step activation function (threshold):(56)g2(x)=1,x≥00,x<0.
The method BPE is disabled sinceg2(x) is not differentiable. For application BPE in this case it must be to approximate g2(x) with logistic function S(x)=1/(1+exp⁡(cx)); c≥100. The process optimization is finished after more than 600 iterations (Figure 9(b)). The final results for MN-SDS of the ponderation Q as shown in Figure 9(b); training was done with set. SS procedure with MN-SDS has been initiated with random parameters (weights): ω0,03=1.01; ω0,04=0.55; ω0,05=0.74; ω0,35=0.19; ω0,45=0.57.
Finally results afterN>300 iterations are ω03,N=1.50, ω04,N=0.99, ω05,N=0.51, ω35,N=-1.98, ω45,N=0.98. with relative error of 2%.
The random vector of forward propagation is with dimensionn=5 is:(57)ξ≡ξ1,ξ2,ξ3,ξ4,ξ5T.
Let us refer to an example when an activation functiong2(x) is given by relation (56), with training pairs P′(uk′,tk′). Then it shows that a training procedure depends on choice of training pairs as well. The minimization process of such case is done rather faster (Figure 9(b)).
BPE methods implementation was made with known procedures [27, 28]. BPE is used as comparative method.
In this paper MN-SDS and BPE use ECR (procedures error correction rules) which are more convenient for “online” ANN training [45].
Whenever the training coefficient (α) is small then ECR and GDR procedures provide the same results for parameters estimation within set-up range of an acceptable error [44, 45]. GDR (gradient rules) is called “batch” optimization procedure.Figure 8
Optimization, for example, training of multilayer ANN; XOR problem.
(a)
(b)Figure 9
Cost function; (a) forg1(x)=1/(1+exp⁡(-x)), (b) for g2(x) step function; BPI optimization is finished after 600 iterations.
(a)
(b)Example 3 (synthesis FANN to approximate model FSR).
In this example some of theoretical R&D results of [29–33] are applied.
The real problem is approximated model of an ANN training program related on technical term for process in reactor in which the concentrate heated to powder so that will behave like a fluid [46, 47].
When the said FSR is to be tempered either the first time or after service attempts, there is a program of tempering to temperatureT as per diagram (see Figure 10). After reaching working temperature it is maintained by control system. The tempering to T=600±25°C is to be completed within 130 min (range AB). In Figure 10 the whole cycle is within 240 min. The FSR was under operation 40 minutes (range BC). Due to some maintenance problem the FSR is often shut down (point C). Before next campaign the FSR should be cooled down (range CD) for maintenance purposes and afterwards the tempering operation is repeated.
The mapping in real conditions have not enough data to make FANN (Figure11, [46]). There is not enough, to model approximations over ANN that learns. Overall number of pairs (tk,Tk) collected from the said diagram (Figure 10) is 80 pairs; sampling period is 3 min. More frequent sampling is not realistic. The task is to determine a FANN, which is to give the best approximate model of the FSR within required accuracy. The relative error of correct response should be more than 3% of obtained measuring data.
On Figure12 is given a starting architecture of FANN NN1=(1-10-10-1): with one input and one output, with 10 neurons in the first and in second layers. Only neurons in the second layer has non-linear activation function. All neurons has a linear interaction function.
Application of the relations ((39)–(41)) of Section 3.2 on NN1 gives:NNd=120+21=141 interlinks between unknown parameters and biases.Np=141×10=1410 required training pairs for generalization G>90%!,C=120 memorized information under condition that n+Nuh≫m, 11≫1; condition is satisfied.
Based on the aforesaid there is overnumbered required training pairs(1410≫80).
Basic moments in conjunction approximation model FSR through the use of MN-SDS in the training of FANNNN1=(1-10-10-1) are that(i)
assignment trainingNN1  does of 80 pairs of training is achieved through the diagrams on Figure 10 that define Table 2,(ii)
random vectorξ replaces ωij and ω0j at expression (32), so dim⁡ξ=141, unknown parameters,(iii)
feedforward phase in layers for each training pair achieved numerical value fornetj(l) and yj(l),l=1,2,3, to the expression (32),(iv)
cost function for sequence for each training parQk, k=1,2,3,…,80, was calculated with the expression (36). Figure 13 presents the trend of Qk for NN1,(v)
the procedure optimization, that is, training for FANNNN1 takes more than 1000 iterations.
Due to the big volume of numerical data in the paper, we will not list all the details.
Out of the two ANN under training with same training samples more strong generalization has network with less number of neurons.Table 2
Data defining program warming FSR (t, T).
t [min]
T [°C]
0
20
10
20
20
50
30
50
40
100
50
100
70
150
80
250
90
250
100
350
105
350
110
450
115
450
120
550
125
600
130
600
140
600
160
600
170
600
175
425
185
325
205
150
215
110
225
75
235
60
240
50Figure 10
Plan introducing FSR in operative mode (AB), work (BC), possibly down time (CD).Figure 11
The real mapping data of FSR at operative environment.Figure 12
The initial structure of FANN for approximation model’s tempering of FSR.Figure 13
Trend cost functions multilayers FANN’sNN1 and NN5.That is the reason to refer to less complex network structures:N
N
2
=
(
1
-
5
-
5
-
1
), NN3=(1-3-3-1) and NN4=(1-2-2-1).StructureNN4 eventually could be trained and have certain generalization possibility since for NN4NNd=13 interlink, that is, unknown parameters,Np=130 required training pairs for G>90%; 130>80 !,C=Nω/m=8, condition n+Nνh≫m; 3>1, barely accepted.More acceptable solution isNN5=(1-3-1), although it presents a bared architecture FANN, since for NN5NNd=10, dimension of NN5,Np=100; there are not 100 but only 80 training pairs,C=6, condition n+Nνh≫m; 4≫1, acceptable.It has been possible to continue to a minimized architectureNN6=(1-2-1), but choice of NN5=(1-3-1) provides better performance of a hidden layer.The closest solution is the FANNNN5=(1-3-1), Figure 14. In hidden layer this network has 3 perceptrons with sigmoid type for activation function.Figure 14
Oriented graph ofNN5=(1-3-1).Having in mind that determining of an ANN, that is, FANN architecture is always open problem then estimation of an adopted structure is suitable after the final numerical result including validity test of a network training.Theoretical results for universal approximator are derived for nonlinear activation, function of hidden neurons of typeS(x)=1/(1+exp⁡(-x)).Since2S(x)-1=tanhx/2 bring an option to use tanh function for a FANN model approximation.Application the relation ofnetj(l) and yj(l) in the expression (32), for structure on Figure 14, NN5=(1-3-1) in general form:(58)Tt=ω25tanhω12t+ω02+ω35tanhω13t+ω03+ω45tanh(ω14+ω04)+ω05;represents an approximation model of the temperature regime of FSR.Here will be presented numerical data and the results of network training forNN5 by MN-SDS only.On the beginning of the numerical procedure for practical reasonst and T should be reduced 100 times.The symbolic presentation of the vector unknow parametersω, in the network NN5, at the beginning (ω0) and end of training procedure (ωN) is given by:(59)ω0=ω12(0),ω13(0),ω14(0),ω25(0),ω35(0),ω45(0),ω02(0),ω03(0),ω04(0),ω05(0)TωN=ω12(N),ω13(N),ω14(N),ω25(N),ω35(N),ω45(N),ω02(N),ω03(N),ω04(N),ω05(N)T.The initial random value of the parameters ω0 is:(60)ω0=0.1576,0.9572,0.4218,0.8003,0.7922,0.9706,kkk0.9706,0.4854,0.1419,0.9157T.Random vectorξ in this case is(61)ξ=ξ1,ξ2,ξ3,…,ξ10T.Behind the training networkNN5 by algorithm MN-SDS after 700 iterations (N>700;Qk,N<ε,ε=10-4) vector ωN of unknown parameters is given by(62)ωN=1.705,3.847,11.61,3.159,-3.597,0.731,klk-1.670,-7.021,-13.23,02939T.Previous data characterize the approximation process model tempering temperature of FSR (58), overtraining FANN NN5 by the MN-SDS algorithm, with 5% of the relative error. Trend of Qk for NN5 is given on Figure 13.Some responses to the test inputs for checking the validity of the obtained model deviate a large error. Counting these to interrelate (sharing) the total number received relatively rough estimate of generalization capabilities appropriate network. Based Figure10 test values have special graphic symbols (O, X for MN-SDS and Δ and ∇ for BPE). For a training set of 80 pairs generalization G ability of the network NN5 is about 70%. For the network NN1 it is about 20%. Previous values obtained training using MN-SDS.Application BPE method gave the following values of generalization: about 70% for networkNN5 and below 20% for the network NN1.The previous presented FSR model could be used in more sophisticated option as an intelligent process monitoring.


## 4. Discussion

Why someone should go to the application of stochastic search methods (SSMs) to solve problems that arise in the optimization and training of ANN? Our answer to this question is based on the demonstration that the SSMs, including SDS (stochastic direct search), have proved to be very productive in solving the problems of complex systems of different nature.In particular, previous experience with ANN and relatively simple architecture suggest that they can exhibit quite a complex behavior, which can be traced to (i) a large number of neuron-perceptron elements in ANN system, (ii) the complexity of the nonlinearity in the activation function of neurons within ANN, (iii) the complexity of the neuron activation function model (i.e., higher level models), (iv) complexity in the optimization procedure due to the large volume of data in the training set, and (v) the complexity of the specification of internal architecture of particular types of ANN.The features listed above require competitive methods which can deal efficiently with such complexity. The SDS represent a combinatorial approach offering great potential via certain heuristics and algorithms they provide for numerical procedures.For example, various methods based on the notion of gradient and which are considered competitive when applied to complex systems cannot avoid the linear scaling of convergence in numerical implementations. In SDS, the trend is roughly speaking proportional ton where n represents the dimension of the vector of parameters in the parameter space of a complex system. This indicates that, with increasing complexity of the system, the relative advantage of SDS method increases when compared to the gradient scheme. That is the key conclusion of why code optimization and training of ANN should employ SDS.The author has previously used some algorithms belonging to the SDS methodology, such as nonlinear SDS (N-SDS) and statistical gradient (SG-SDS). Both of these methods have exhibited poor convergence. By starting from N-SDS, we have designed MN-SDS which already forn>3 is superior to gradient descent (with original N-SDS this is achieved only for n>12).In this paper, Section2 overviewed the concept of SDS and SSA and then introduced MN-SDS. Section 3 examined the possibilities of MN-SDS algorithm and its application on FANN as the target architecture. Section 3 also presents steps in synthesis of FANN in order to emphasize that performed optimization of FANN does not guarantee that the network will achieve the required level of generalization (i.e., ability to learn). Generally speaking the problem of syntheses ANN remains open.The present synthesis elements are simplified theoretical results of the recent studies. This is illustrated by Example3, which is directly connected to practical applications. Example 2 should give an insight about the relationship between N-SDS and MN- SDS, as well as connection between MN-SDS and BPE methods, where the latter was used as a reference. Example 1 confirms efficiency of MN-SDS methods for problems outside of ANN.Let us finally mention that there is an increasing interest in using SSM, both from academia and industry. This is due to the fact that SSM, and in a particular SDS, can find increasing applications in economics, bioinformatics, and artificial intelligence, where the last area is intrinsically linked to ANN.


## 5. Conclusion

The central goal of this study is the presentation of stochastic search approach applied to identification, optimization, and training of artificial neural networks. Based on the author’s extensive experience in using SDS approach to the problems of identification and optimization of complex automatic control system, a new algorithm based on nonlinear SDS (N-SDS), which is termed MN-SDS, is proposed here. The MN-SDS offers significant improvement in convergence properties compared to nonlinear N-SDS and some other SSA.MN-SDS maintains all the other good features of the existing SDS: a relatively easy adaptation to problem solving; simple mathematical construction of algorithmic steps; low sensitivity to noise.The convergence properties of MN-SDS make it superior to majority of standard algorithms based on gradient scheme. Note that convergence is the most suitable characteristics for comparing the efficiency of algorithms for systems with the same number of optimization parameters. For example, already for more than three parameters the MN-SDS exhibits better convergence properties than most other algorithms, including those based on the gradient. This means that, in certain optimization procedures and training, MN-SDS is superior to widely used BPE method for ANN and in the development of artificial intelligence.The MN-SDS in optimization and training of ANN employs only feedforward phase flow of information in FANN. The parameters that are used in optimization within MN-SDS are changed using random number generator. The efficiency of MN-SDS in numerical experiments suggests that it can be applied to very complex ANN. This study has shown its application to feedforward ANN (FANN). The obtained results were compared with results obtained with BPE method, of course, when applied to the same problems.Numerical experiments performed here can be implemented even on simple multicore PC using MATLAB package.



---

*Source: 931379-2015-02-28.xml*