# A Brief Review of Neural Networks Based Learning and Control and Their Applications for Robots

Authors: Yiming Jiang; Chenguang Yang; Jing Na; Guang Li; Yanan Li; Junpei Zhong
Journal: Complexity
 (2017)

---

## Abstract

As an imitation of the biological nervous systems, neural networks (NNs), which have been characterized as powerful learning tools, are employed in a wide range of applications, such as control of complex nonlinear systems, optimization, system identification, and patterns recognition. This article aims to bring a brief review of the state-of-the-art NNs for the complex nonlinear systems by summarizing recent progress of NNs in both theory and practical applications. Specifically, this survey also reviews a number of NN based robot control algorithms, including NN based manipulator control, NN based human-robot interaction, and NN based cognitive control.


---

## Body

## 1. Introduction

In recent years, the research of neural network (NN) has attracted great attention. It is well known that, mammals’ brain, which consists of billions of interconnected neurons, has the ability to deal with complex and computationally demanding tasks, such as face recognition, body motion planning, and muscles activities control. Figure1 shows a cellular structure of a mammalian neuron. Inspired by the neuron structure, artificial NN (ANN) was developed to emulate the learning ability of the biological neurons system [1–3]. The concept of artificial NNs was initially investigated by McCulloch and Pitts in the 1940s [3], where the network is established with a parallel structure. The basically mathematical model of NN consists of three layers, that is, input layer, hidden layer, and output layer, which are of simple parallel computational structure but with appealing learning ability and computational power to predict nonlinear dynamic patterns.Figure 1
An example of mammalian neuron (modified from [32]).In past decades, the NN technique has been studied extensively in areas such as control engineering, aerospace, medicine, automotive, psychology, economics, energy science, and many other fields [4–7]. It has been reported that NN can approximate any unknown continuous nonlinear function by overlapping the outputs of each neuron. Moreover, the approximation errors could be made arbitrarily small by choosing sufficient neurons. This enables us to deal with control problems for complex nonlinear systems [8–13]. In addition to system modeling and control, NN has also been successfully applied in various fields such as learning [14–17], pattern recognition [18], and signal processing [19]. And NN has been extensively used for functions approximation, such as to compensate for the effect of unknown dynamics in nonlinear systems [20–31]. The NN control has been proved to be effective for controlling uncertain nonlinear systems and demonstrated superiority in many aspects.Recently, the researchers have focused on the study of robotics for its increasing importance in both industrial applications and daily life [33–38]. Many advanced robots such as YuMi made by ABB, Baxter made by Rethink, and Rolins’ Justin developed by German Aerospace Agency (DLR) have also been widely allocated. The robots manipulator system is characterized with high-nonlinearity, strong coupling, and time-varying dynamics, thus controlling a robot with not only positioning accuracy, but also enough flexibility to complete a complex task became an interesting yet challenge work. To achieve a high performance control, dynamics of the robot should be known in advance. However, in practice, the robot dynamic model is often rarely known due to the complex robot mechanism, let alone various uncertainties such as parametric uncertainties or modeling errors existing in the robot dynamics. Therefore, advance control algorithm is imperative for next-generation robots. Thanks to the universal approximation and learning ability, the NN has been widely applied in robot control with various applications. The combination of NN and robot controller can provide possible solutions for complex manipulation tasks, for example, robot control with unknown dynamics and robot control with unstructured environment. In this paper, we present a brief review of robot control by means of neural network. The rest of the paper is organized as follows.After the introduction, in Section2, we present preliminaries of several popular neural network structures, such as RBFNN and CMAC NN. Section 3 introduces a number of theoretical developments of NN in the fields of adaptive control, optimization, and evolutionary computing. In addition, Section 4 revisits the robot neural network control with the applications in manipulation, human-robot interaction, and robot cognitive control. Section 5 gives a brief discussion about the neural network control and its future research.


## 2. Preliminaries of Neural Networks

In this section, we will introduce several types of NN structure, which are popularly employed in the control engineering.


### 2.1. Radial Basis Function Neural Network (RBFNN) [14, 15]

A basic architecture of RBFNN network is shown in Figure2, which consists of three layers, namely, input layer, hidden layer, and output layer. In the input layer, the NN inputs are applied. In hidden layer, the data is transformed from input space to hidden space, which is always with a higher dimension. The RBFNN can be used to approximate any continuous vector function, for example, F(Z): (1)F^Z=WTSZ,where F^(Z) is the estimation of F(Z) and Z is NN inputs vector. W^=[W^1,W^2,…,W^n]∈Rn×l is the estimation of NN optimal weight, S(Z)=[s1(Z),s2(Z),…,sl(Z)]T is the regressor, and l denotes the number of NN nodes. Generally, the regressor could be chosen as a Gaussian radical basis function as follows:(2)siZ-ui=exp⁡-Z-uiTZ-uiσi2,where ui(i=1,…,l) are distinct points in state space and σi is the width of Gaussian membership function. It has been well recognized that, using the powerful approximate ability of the RBFNN, we can approximate any continuous nonlinear function over a compact set as (3)FZ=W∗TSZ+ε,where W∗ is the optimal weight vector and ε is the approximate error.Figure 2
Structure of the RBFNN.


### 2.2. Cerebellar Model Articulation Controller (CMAC) NN [39]

There has been a predominant tendency to study the learning and control techniques of robots by exploring the principles of biological systems. This is because the biological creatures, mechanisms, and underlying principles are likely to bring novel ideas to improve control performance of the robot in a complex environment. In 1972, Albus proposed a learning mechanism that imitates the structure and function of the cerebellum, called cerebellar model articulation controller (CMAC), which is designed based on a cerebellum neurophysiological model [40]. In comparison to the backpropagation neural network, the CMAC NN was adopted widely in modeling and control of robots system for its rapid learning speed, simple structure, insensitivity of data sequence, and easy implementation [39, 41].Figure3 shows the basic structure of the CMAC neural network. The CMAC could be used to approximate the unknown continuous function, G(Z)=[f1(Z),f2(Z),…,fn(Z)], where Z∈Rm denotes the m dimensional inputs space. As shown in Figure 3, two components are involved in the CMAC neural network to determine the value of the approximated nonlinear function G(Z):(4)R:Z⟶CP:C⟶F,where Z  ism-dimensional input spaceF isn-dimensional output spaceC is Nc-dimensional association spaceFigure 3
Structure of a CMAC neural network.andR(·) denotes the mapping from the input vector to the association space; that is, α=R(Z). The outputs are computed through P(α), by using a projection of the association vector α onto a weights vector, such that (5)f=Pα=WTα,It should be noted thatR(Z) can be represented by a multidimensional receptive filed function such that each point in input Z is assigned with an activation value. The receptive-field basis functions of the association vector could be chosen as Gaussian functions as follows:(6)hikzi-uik=exp⁡-zi-uik2ϑik2,k=1,2,…,l,wherel is number of blocks of the associate space, hik denotes thekth block associated with the input zi, uik denotes the receptive field’s center, and ϑik is the variance of Gaussian function. Then, the multidimensional receptive-field function can be described as(7)SZ=s1,s2,…,snT,where sk(Z,uk,ϑk)=∏i=1mhik(zi), uk=u1k,u2k,…,umkT, and ϑk=[ϑ1k,ϑ2k,…,ϑmk]T. The following property shows the approximation ability provided by the CMAC neural network.Lemma 1.
For a continuous nonlinear functionF(Z), there exists an ideal weight value W∗, such that F(Z) could be uniformly approximated by a CMAC with the multiplication of the optimal weights W∗ and the associate vector S(Z) as(8)FZ=WTSZ+ε,where ε is the NN construction errors and satisfied ε≤εN and εN is a small bounded positive value.


## 2.1. Radial Basis Function Neural Network (RBFNN) [14, 15]

A basic architecture of RBFNN network is shown in Figure2, which consists of three layers, namely, input layer, hidden layer, and output layer. In the input layer, the NN inputs are applied. In hidden layer, the data is transformed from input space to hidden space, which is always with a higher dimension. The RBFNN can be used to approximate any continuous vector function, for example, F(Z): (1)F^Z=WTSZ,where F^(Z) is the estimation of F(Z) and Z is NN inputs vector. W^=[W^1,W^2,…,W^n]∈Rn×l is the estimation of NN optimal weight, S(Z)=[s1(Z),s2(Z),…,sl(Z)]T is the regressor, and l denotes the number of NN nodes. Generally, the regressor could be chosen as a Gaussian radical basis function as follows:(2)siZ-ui=exp⁡-Z-uiTZ-uiσi2,where ui(i=1,…,l) are distinct points in state space and σi is the width of Gaussian membership function. It has been well recognized that, using the powerful approximate ability of the RBFNN, we can approximate any continuous nonlinear function over a compact set as (3)FZ=W∗TSZ+ε,where W∗ is the optimal weight vector and ε is the approximate error.Figure 2
Structure of the RBFNN.


## 2.2. Cerebellar Model Articulation Controller (CMAC) NN [39]

There has been a predominant tendency to study the learning and control techniques of robots by exploring the principles of biological systems. This is because the biological creatures, mechanisms, and underlying principles are likely to bring novel ideas to improve control performance of the robot in a complex environment. In 1972, Albus proposed a learning mechanism that imitates the structure and function of the cerebellum, called cerebellar model articulation controller (CMAC), which is designed based on a cerebellum neurophysiological model [40]. In comparison to the backpropagation neural network, the CMAC NN was adopted widely in modeling and control of robots system for its rapid learning speed, simple structure, insensitivity of data sequence, and easy implementation [39, 41].Figure3 shows the basic structure of the CMAC neural network. The CMAC could be used to approximate the unknown continuous function, G(Z)=[f1(Z),f2(Z),…,fn(Z)], where Z∈Rm denotes the m dimensional inputs space. As shown in Figure 3, two components are involved in the CMAC neural network to determine the value of the approximated nonlinear function G(Z):(4)R:Z⟶CP:C⟶F,where Z  ism-dimensional input spaceF isn-dimensional output spaceC is Nc-dimensional association spaceFigure 3
Structure of a CMAC neural network.andR(·) denotes the mapping from the input vector to the association space; that is, α=R(Z). The outputs are computed through P(α), by using a projection of the association vector α onto a weights vector, such that (5)f=Pα=WTα,It should be noted thatR(Z) can be represented by a multidimensional receptive filed function such that each point in input Z is assigned with an activation value. The receptive-field basis functions of the association vector could be chosen as Gaussian functions as follows:(6)hikzi-uik=exp⁡-zi-uik2ϑik2,k=1,2,…,l,wherel is number of blocks of the associate space, hik denotes thekth block associated with the input zi, uik denotes the receptive field’s center, and ϑik is the variance of Gaussian function. Then, the multidimensional receptive-field function can be described as(7)SZ=s1,s2,…,snT,where sk(Z,uk,ϑk)=∏i=1mhik(zi), uk=u1k,u2k,…,umkT, and ϑk=[ϑ1k,ϑ2k,…,ϑmk]T. The following property shows the approximation ability provided by the CMAC neural network.Lemma 1.
For a continuous nonlinear functionF(Z), there exists an ideal weight value W∗, such that F(Z) could be uniformly approximated by a CMAC with the multiplication of the optimal weights W∗ and the associate vector S(Z) as(8)FZ=WTSZ+ε,where ε is the NN construction errors and satisfied ε≤εN and εN is a small bounded positive value.


## 3. Theoretical Developments

### 3.1. Adaptive Neural Control

During the past two decades, various neural networks have been incorporated into adaptive control for nonlinear systems with unknown dynamics. In [42], a multiplayer discrete-time neural network controller was constructed for a class of multi-input multioutput (MIMO) dynamical systems, where NN weights were trained using an improved online tuning algorithm. An adaptive NN output feedback control was proposed to control two classes of discrete-time systems in the presence of unknown control directions [4]. A robust adaptive neural controller was developed for a class of strict-feedback systems in [43], where a Nussbaum gain technique was employed to deal with unknown virtual control coefficients. A dynamic recurrent NN was employed for construction of an adaptive observer with online turned weights parameters in [44] and to deal with the time-delay of a class of nonlinear dynamical systems in [45]. The time-delay of strict-feedback nonlinear systems was also addressed by using NN control with proper designed Lyapunov-Krasovskii functions in [46]. For a class of unknown nonlinear affine time-delay systems, an adaptive control scheme was proposed by constructing two high-order NNs for identifying system uncertainties [47]. This idea has been further extended to affine nonlinear systems with input time-delay in [48].It should be noticed that, piecewise continuous functions such as frictions, backlash, and dead-zone are widely existed in industrial plants. Other than continuous nonlinear function, the approximation of these piecewise functions is more challenging since the NN’s universal approximation only holds for continues functions. To approximate these piecewise continuous functions, a novel NN structure was designed by involving a standard activation function and a jump approximation basis function [49]. In [47], a CMAC NN was employed for the closed-loop control of nonlinear dynamical systems with rigorous stability analysis, and in [50] a robust adaptive neural network control scheme was developed for cooperative tracking control of higher-order nonlinear systems.The adaptive NN control scheme was also proposed for pure-feedback systems. In [51], a high-order sliding mode observer was proposed to estimate the unknown system states while two NNs were constructed to deal with approximation errors and the unknown nonlinearities, respectively. In comparison to the conventional control design for pure-feedback systems, the state-feedback control was achieved without using the backstepping technique. In [52], a neural control framework was proposed for nonlinear servo mechanism to guarantee both the steady-state and transient tracking performance. In this work, a prescribed performance function was employed in an output error transformation, such that the tracking performance can be guaranteed by the regulation control of the outputs. In [53], an adaptive neural control was also designed for a class of nonlinear systems in the presence of time-delays and input dead-zone, and high-order neural networks were employed to deal the unknown uncertainties. In this work, a salient feature lies in the fact that only the norm of the NNs’ weights (a scalar) needs to be online updated, such that the computational efficiency in the online implementation could be significantly improved. In [54], the authors developed a neural network based feedforward control to compensate for the nonlinearities and uncertainties of a dynamically substructured system consisting of both numerical and physical substructures, where an adaptive law with a new leakage term of NN weights error information was developed to achieve improved convergence. An experiment on a quasi-motorcycle testing rig validated the efficacy of this control strategy. In [55], a neural dynamic control was incorporated into the strict-feedback control of a class of unknown nonlinear systems by using the dynamic surface control technique. For a class of uncertain nonlinear systems with unknown hysteresis, NN was used for compensation of the nonlinearities [56]. In [57], to deal with unknown nonsymmetrical input saturations of unknown nonaffine systems, NNs were used in the state/output feedback control based on the mean value theorem and the implicit function. To avoid using the backstepping synthesis, a dynamic surface control scheme was designed by combining the NN with a nonlinear disturbance observer [58].


### 3.2. NN Based Adaptive Dynamic Programming

In addition to adaptive control, neural networks have also been adopted to solve the optimization problem for nonlinear systems. In convention optimal control, the dynamic programming method was widely used. It aims to minimize a predefined cost function, such that a sequence of optimal control inputs could be derived. However, the cost function is usually difficult to online calculate due to the computation complexity in obtaining the solution of the Hamilton-Jacobi-Bellman (HJB) equation. Therefore, an adaptive/approximate dynamic programming (ADP) technique was developed in [59], where a NN was trained to estimate the cost function and then to derive solutions for the ADP. Generally, the ADP has several different synonyms, including approximate dynamic programming, heuristic dynamic programming (HDP), critic network, and reinforcement learning (RL) [60–62]. Figure 4 shows the basic framework of the HDP with a critic-actor structure. In [63], a discrete-time HJB equation was solved using an NN based HDP algorithm to derive the optimal control of nonlinear discrete-time systems. In [64], three neural networks were constructed for an iterative ADP, such that optimal feedback control of a discrete-time affine nonlinear system could be realized. In [65], a globalized dual heuristic programming was presented to address the optimal control of discrete-time systems. In each iteration, three neural networks were used to learn the cost function and the unknown nonlinear systems. In [66], a reference network combining with an action network and a critic network was introduced in the ADP architecture to derive an internal goal representation, such that the learning and optimization process could be facilitated. The reference network has also been introduced in the online action-dependent heuristic dynamic programming by employing a dual critic network framework. A policy iteration algorithm was introduced for infinite horizon optimal control of nonlinear systems using ADP in [67]. In [68], a reinforcement learning method was introduced for the stabilizing control of uncertain nonlinear systems in the presence of input constraints. By using this RL-based controller, a constrained optimal control problem was solved with construction of only one critic neural network. In [69], an ADP technique for online control and learning of a generalized multiple-input-multiple-output (MIMO) system was investigated. In [70], an adaptive NN based ADP control scheme was presented for a class of nonlinear systems with unknown dynamics. The optimal control law was calculated by using a dual neural network scheme with a critic NN and an identifier NN. Particularly, parameters estimation error was used to online identify the learning weights to achieve the finite-time convergence. Optimal tracking control for a class of nonlinear systems was investigated in [71], where a new “identifier-critic” based ADP framework was proposed.Figure 4
An overview of the HDP structure.


### 3.3. Evolutionary Computing

In addition to the capacity of approximation and optimization of the NN, there has been also a great interest in using the evolutionary approaches to train the neural networks. With the evolvement of NN architectures, learning rules, connection weights, and input features, an evolutionary artificial neural network (EANN) was designed to provide superior performance in comparison to conventional training approaches [72]. A literature review of the EANN was given in [73], where the evolution strategies such as feedforward artificial NN and genetic algorithms (GA) have been introduced for the EANNs. In [72], several EANN frameworks were introduced by embedding the evolution algorithms (EA) to evolve the NN structure. In [74], an EPNet evolution system was proposed for evolving the feedforward NN based on Fogel’s evolutionary programming (EP) method, which could improve the NN’s connection weights and architectures at the same time as well as decrease noise in the fitness evaluation. Good generalization ability of the evolved NN has been constructed and verified in the experiments. In [75], a GA based technique has been employed to train the NNs in direct neural control systems such that the NN architectures could be optimized. A deficiency of the EANN is that the optimization process would often result in a low training speed. To overcome this problem and facilitate adaptation processes, a hybrid multiobjective evolutionary method was developed in [76], where the singular-value-decomposition (SVD) technique was employed to choose the necessary neurons number in the training of a feedforward NN. The evolutionary approach was applied to identify a grey-box model with a multiobjective optimization between the clearly known practical systems and approximated nonlinear systems [77]. Applications of evolutionary algorithms for robotic navigation have been introduced and investigated in [78]. A survey of machine learning technique was reported in [79], where several methods to improve the evolutionary computation were reviewed.The evolution algorithms have been employed in many aspects for evolvements of NNs, such as to train the NN connection weights or to obtain near-optimal NN architectures, as well as adapting learning rules of NNs to their environment. In a word, the evolution algorithms provide NNs with the ability of learning to learn and also to build the relationship between evolution and learning, such that the EANN could perform favorable ability to adapt to changes of the dynamic environment.


## 3.1. Adaptive Neural Control

During the past two decades, various neural networks have been incorporated into adaptive control for nonlinear systems with unknown dynamics. In [42], a multiplayer discrete-time neural network controller was constructed for a class of multi-input multioutput (MIMO) dynamical systems, where NN weights were trained using an improved online tuning algorithm. An adaptive NN output feedback control was proposed to control two classes of discrete-time systems in the presence of unknown control directions [4]. A robust adaptive neural controller was developed for a class of strict-feedback systems in [43], where a Nussbaum gain technique was employed to deal with unknown virtual control coefficients. A dynamic recurrent NN was employed for construction of an adaptive observer with online turned weights parameters in [44] and to deal with the time-delay of a class of nonlinear dynamical systems in [45]. The time-delay of strict-feedback nonlinear systems was also addressed by using NN control with proper designed Lyapunov-Krasovskii functions in [46]. For a class of unknown nonlinear affine time-delay systems, an adaptive control scheme was proposed by constructing two high-order NNs for identifying system uncertainties [47]. This idea has been further extended to affine nonlinear systems with input time-delay in [48].It should be noticed that, piecewise continuous functions such as frictions, backlash, and dead-zone are widely existed in industrial plants. Other than continuous nonlinear function, the approximation of these piecewise functions is more challenging since the NN’s universal approximation only holds for continues functions. To approximate these piecewise continuous functions, a novel NN structure was designed by involving a standard activation function and a jump approximation basis function [49]. In [47], a CMAC NN was employed for the closed-loop control of nonlinear dynamical systems with rigorous stability analysis, and in [50] a robust adaptive neural network control scheme was developed for cooperative tracking control of higher-order nonlinear systems.The adaptive NN control scheme was also proposed for pure-feedback systems. In [51], a high-order sliding mode observer was proposed to estimate the unknown system states while two NNs were constructed to deal with approximation errors and the unknown nonlinearities, respectively. In comparison to the conventional control design for pure-feedback systems, the state-feedback control was achieved without using the backstepping technique. In [52], a neural control framework was proposed for nonlinear servo mechanism to guarantee both the steady-state and transient tracking performance. In this work, a prescribed performance function was employed in an output error transformation, such that the tracking performance can be guaranteed by the regulation control of the outputs. In [53], an adaptive neural control was also designed for a class of nonlinear systems in the presence of time-delays and input dead-zone, and high-order neural networks were employed to deal the unknown uncertainties. In this work, a salient feature lies in the fact that only the norm of the NNs’ weights (a scalar) needs to be online updated, such that the computational efficiency in the online implementation could be significantly improved. In [54], the authors developed a neural network based feedforward control to compensate for the nonlinearities and uncertainties of a dynamically substructured system consisting of both numerical and physical substructures, where an adaptive law with a new leakage term of NN weights error information was developed to achieve improved convergence. An experiment on a quasi-motorcycle testing rig validated the efficacy of this control strategy. In [55], a neural dynamic control was incorporated into the strict-feedback control of a class of unknown nonlinear systems by using the dynamic surface control technique. For a class of uncertain nonlinear systems with unknown hysteresis, NN was used for compensation of the nonlinearities [56]. In [57], to deal with unknown nonsymmetrical input saturations of unknown nonaffine systems, NNs were used in the state/output feedback control based on the mean value theorem and the implicit function. To avoid using the backstepping synthesis, a dynamic surface control scheme was designed by combining the NN with a nonlinear disturbance observer [58].


## 3.2. NN Based Adaptive Dynamic Programming

In addition to adaptive control, neural networks have also been adopted to solve the optimization problem for nonlinear systems. In convention optimal control, the dynamic programming method was widely used. It aims to minimize a predefined cost function, such that a sequence of optimal control inputs could be derived. However, the cost function is usually difficult to online calculate due to the computation complexity in obtaining the solution of the Hamilton-Jacobi-Bellman (HJB) equation. Therefore, an adaptive/approximate dynamic programming (ADP) technique was developed in [59], where a NN was trained to estimate the cost function and then to derive solutions for the ADP. Generally, the ADP has several different synonyms, including approximate dynamic programming, heuristic dynamic programming (HDP), critic network, and reinforcement learning (RL) [60–62]. Figure 4 shows the basic framework of the HDP with a critic-actor structure. In [63], a discrete-time HJB equation was solved using an NN based HDP algorithm to derive the optimal control of nonlinear discrete-time systems. In [64], three neural networks were constructed for an iterative ADP, such that optimal feedback control of a discrete-time affine nonlinear system could be realized. In [65], a globalized dual heuristic programming was presented to address the optimal control of discrete-time systems. In each iteration, three neural networks were used to learn the cost function and the unknown nonlinear systems. In [66], a reference network combining with an action network and a critic network was introduced in the ADP architecture to derive an internal goal representation, such that the learning and optimization process could be facilitated. The reference network has also been introduced in the online action-dependent heuristic dynamic programming by employing a dual critic network framework. A policy iteration algorithm was introduced for infinite horizon optimal control of nonlinear systems using ADP in [67]. In [68], a reinforcement learning method was introduced for the stabilizing control of uncertain nonlinear systems in the presence of input constraints. By using this RL-based controller, a constrained optimal control problem was solved with construction of only one critic neural network. In [69], an ADP technique for online control and learning of a generalized multiple-input-multiple-output (MIMO) system was investigated. In [70], an adaptive NN based ADP control scheme was presented for a class of nonlinear systems with unknown dynamics. The optimal control law was calculated by using a dual neural network scheme with a critic NN and an identifier NN. Particularly, parameters estimation error was used to online identify the learning weights to achieve the finite-time convergence. Optimal tracking control for a class of nonlinear systems was investigated in [71], where a new “identifier-critic” based ADP framework was proposed.Figure 4
An overview of the HDP structure.


## 3.3. Evolutionary Computing

In addition to the capacity of approximation and optimization of the NN, there has been also a great interest in using the evolutionary approaches to train the neural networks. With the evolvement of NN architectures, learning rules, connection weights, and input features, an evolutionary artificial neural network (EANN) was designed to provide superior performance in comparison to conventional training approaches [72]. A literature review of the EANN was given in [73], where the evolution strategies such as feedforward artificial NN and genetic algorithms (GA) have been introduced for the EANNs. In [72], several EANN frameworks were introduced by embedding the evolution algorithms (EA) to evolve the NN structure. In [74], an EPNet evolution system was proposed for evolving the feedforward NN based on Fogel’s evolutionary programming (EP) method, which could improve the NN’s connection weights and architectures at the same time as well as decrease noise in the fitness evaluation. Good generalization ability of the evolved NN has been constructed and verified in the experiments. In [75], a GA based technique has been employed to train the NNs in direct neural control systems such that the NN architectures could be optimized. A deficiency of the EANN is that the optimization process would often result in a low training speed. To overcome this problem and facilitate adaptation processes, a hybrid multiobjective evolutionary method was developed in [76], where the singular-value-decomposition (SVD) technique was employed to choose the necessary neurons number in the training of a feedforward NN. The evolutionary approach was applied to identify a grey-box model with a multiobjective optimization between the clearly known practical systems and approximated nonlinear systems [77]. Applications of evolutionary algorithms for robotic navigation have been introduced and investigated in [78]. A survey of machine learning technique was reported in [79], where several methods to improve the evolutionary computation were reviewed.The evolution algorithms have been employed in many aspects for evolvements of NNs, such as to train the NN connection weights or to obtain near-optimal NN architectures, as well as adapting learning rules of NNs to their environment. In a word, the evolution algorithms provide NNs with the ability of learning to learn and also to build the relationship between evolution and learning, such that the EANN could perform favorable ability to adapt to changes of the dynamic environment.


## 4. Applications in Robots

### 4.1. NN Based Robotic Manipulator Control

Generally speaking, the control methods for robot manipulators can be roughly divided into two groups, model-free control and model based control. For the model-free control approaches like proportional-integral-derivative (PID) control, satisfactory control performance may not be guaranteed. In contrast, the model based control approaches exhibit better control behavior but heavily depend on the validity of the robot model. In practice, however, a perfect robotic dynamic model is always not available due to the complex mechanisms and uncertainties. Additionally, the payload may be varied according to different tasks, which makes the accurate dynamics model hard to be obtained in advance. To solve such problems, the NN approximation-based control methods have been used extensively in applications of robot manipulator control. A basic structure of the adaptive neural network control for robot manipulator is shown in Figure5. Consider a dynamic model of a robot manipulator given as follows [80]:(9)Mqq¨+Cq˙,qq˙+Gq=τ,where M(q), C(q˙,q), and G(q) are the inertial matrix, Coriolis matrix, and gravity vector, respectively. Then NN control design could be given as follows:(10)τd=-K1e1-K2e2+W^SZ,where e1 is the tracking error, e2 is the velocity tracking error, W^S(Z) is the NN controller with W^ being the weights matrix and S(Z) being the NN regressor vector, and K1 and K2 are control gains specified by the designer.Figure 5
A basic structure of the adaptive NN robot control.From (10), we can see that the robot controller consists of a PD-like controller and a NN controller. In traditional model based controllers, the dynamic model of the robot could be regarded as a feedforward to address the effect caused by the robot motion. In practice, however, M(q), C(q,q), and G(q) may not be known. Therefore, the NNs are used to approximate the unknown dynamics f=M(q)q¨+C(q˙,q)q˙+G(q) and to improve the performance of the system via the online estimation. To adapt the NN weights, adaptive laws are designed as follows:(11)W^˙=ΓSe-σW^,where Γ and σ are specified positive parameters. The last term of right-hand side of (11) is the sigma modification, which is used to enhance the convergence and robustness of the parameters adaptation.In [80], a NN based share control method was developed to control a teleoperated robot with environmental uncertainties. In this work, the RBFNN was constructed to compensate for the unknown dynamics of the teleoperated robot. Particularly, a shared control strategy was developed into the controller to achieve the automatic obstacle avoidance combining with the information of visual camera and the robot body, such that the obstacle could be successfully avoided and the operator could focus more on the operated task rather than the environment to guarantee the stability and manipulation. In addition, error transformations were integrated into the adaptive NN control to guarantee the transient control performance. It was shown that, by using the NN technique, the control performance in both kinematic level and dynamic level of the teleoperated robot was enhanced. In [81], an extreme learning machine (ELM) based control strategy was proposed for uncertain robot manipulators to identify both the elasticity and geometry of an object. This ELM was applied to deal with the unknown nonlinearity of the robot manipulator to enhance the control performance. Particularly, by utilizing ELM, the proposed controller could guarantee that the robot dynamics follow a reference model, such that the desired set point and the feedforward force could be updated to estimate the geometry and stiffness of the object. As a result, the reference model could be exactly matched with a limited number of iterations.In [82], the NN controller was also employed to control a wheel inverted pendulum, which has been decomposed into two subsystems, a fully actuated second-order planar moving subsystem and a passive first-order pendulum subsystem. Then the RBFNN was employed to compensate for the uncertain dynamics of the two subsystems by using its powerful learning ability, such that the enhanced control performance could be realized by using the NN learning. In [83], a global adaptive neural control was proposed for a class of robot manipulators with finite-time convergence learning performance. This control scheme employed a smooth switching mechanism combining with a nominal neural network controller and a robust controller to ensure global uniform ultimately bounded stability. The optimal weights were obtained by the finite-time estimation algorithm such that, after the learning process, the learning weights could be reused next time for repeated tasks. The global NN control mechanism has been further extended to the control of dual arm robot manipulator in [84], where knowledge of both robot manipulator and the grasping object is unavailable in advance. By integrating prescribed functions into the design of controller, the transient performance of the dual arm robot control was regularly guaranteed. The NN was also employed to deal with synchronization problem of multiple robot manipulators in [85], where the reference trajectories are only available for part of the team members. By using the NN approximation controller, the robot has shown better control performance with enhanced transient performance and enhanced robustness. A RBFNN was constructed to compensate for the nonlinear terms of a five-bar manipulator based on an error transformation function [86]. The NN control was also applied in the robot teleoperation control [87, 88]. Moreover, a NN approximation technique was employed to deal with the unknown dynamics, kinematics, and actuator properties in the manipulator tracking control [89].


### 4.2. NN Based Robot Control with Input Nonlinearities

Another challenge of the robot manipulator is that the input nonlinearities such as friction, dead-zone, and actuator saturation may inevitably exist in the robot systems. These input nonlinearities may lead to larger tracking errors and degeneration of the control performance. Therefore, a number of works have been proposed to handle the nonlinearities by utilizing the neural network design. A neural adaptive controller was designed to deal with the effect of input saturation of the robot manipulator in [90] as follows: (12)τ=-z1+W^DSDZDα1+W^CSCZCα1+W^GSGZG+Kpz2+ξ+Krsgn⁡z2,where z1 is the robot position tracking error, z2 is the velocity tracking error, and α1 is an auxiliary controller. W^D, W^C, and W^G are the NN weights, SD(ZD), SC(ZC), and SG(ZG) are the NN regressor vectors, and Kp and Kr are control gains specified by the designer. ξ is an auxiliary system designed to reduce the effect of the saturation with ξ˙ defined as follows. (13)ξ˙=-Kξξ-z2TΔτ+1/2ΔτTΔτξ2ξ+Δτξ≥μ0ξ<μ,where Δτ is the torque error caused by saturation, and Kξ is a small positive value. To update the NN weights, adaptive laws are designed as follows:(14)W^˙D=ΓDkSDkα1e2k-σDkW^DkW^˙C=ΓCkSCkα1e2k-σCkW^CkW^˙G=ΓGkSGkα1e2k-σGkW^Gk,where ΓDk, ΓCk, and ΓGk are specified positive parameters and σDk, σCk, and σGk are positive parameters.In [91], an adaptive neural network controller was constructed to approximate the input dead-zone and the uncertain dynamics of the robotic manipulator, while the output constraint was also considered in the feedback control. In [92], the NN was applied for the estimation of the unknown model parameters of a marine surface vessel and in [93] the full-state constraint of ann-link robotic manipulator was achieved by using the NN control. The NN controller was also constructed for flexible robotic manipulators to deal with the vibration suppression based on a lumped spring-mass model [94] while in [95], two RBFNNs were constructed for flexible robot manipulators to compensate for the unknown dynamics and the dead-zone effect, respectively.The NN has also been used in many important industrial fields, such as autonomous underwater vehicles (AUVs) and hypersonic flight vehicle (HFV). In [96], the NN has been constructed to deal with the attitude of AUVs in the presence of input dead-zone and uncertain model parameters. In [97], the adaptive neural control was employed to deal with underwater vehicle control in discrete-time domain encountered with the unknown input nonlinearities, external disturbance, and model uncertainties. Then the reinforcement learning was applied to address these uncertainties by using a critic NN and an action NN. The hypersonic flight vehicle control was investigated in [98] where the aerodynamic uncertainties and unknown disturbances were addressed by a disturbance observer based NN. In [99], a neural learning control was embedded in the HFV controller to achieve the global stability via a switching mechanism and a robust controller.


### 4.3. NN Based Human-Robot Interaction Control

Recently, there is a predominant tendency to employ the robots in the human-surrounded environment, such as household services or industrial applications, where humans and robots may interact with each other directly. Therefore, interaction control has become a promising research field and has been widely studied. In [100], a learning method was developed such that the dynamics of a robot arm could follow a target impedance model with only knowledge of the robotic structure (see Figure 6).Figure 6
Block of the learning impedence control.The NN was further employed in robot control in interaction with an environment [101], where impedance control was achieved with the completely unknown robotic dynamics. In [102], a learning method was developed such that the robot was able to adjust the impedance parameters when it interacted with unknown environments. In order to learn optimal impedance parameters in the robot manipulator control, an adaptive dynamic programming (ADP) method was employed when the robot interacted with unknown time-varying environments, where NNs were used for both critic and actor networks [103]. The ADP was also employed for coordination of multirobots [104], in which possible disagreement between different manipulators was handled and dynamics of both robots and the manipulated object were not required to be known.In this work, the controller consists of two parts, a critic network which was used to approximate the cost function, and an actual NN which was designed to control the robot. The critic NN is designed as follows [104]: (15)Υ^t=W^CSCZC,where ZC=ξ, ξ=[x˙oT,zT,xoT]T with xo being the position of the object and z being the tracking error, W^C is the NN weight, and SC is the regressor vector.The critic NN is used to approximate a cost functionc(t)=ξTQξ+uTRu, where u denotes the control input, and Q and R are positive definite matrix. Since the control objective is to minimize the control effort, the adaptation law is designed as(16)W^˙c=-σc∂Ec∂W^c=σcc-W^cTS˙CS˙C,where σc is the learning rate and Ec=(1/2)(c-W^cTS˙C)2.On the other hand, the actual NN control is designed to control the robot as(17)u=W^aTSaZa-e-K2ev,where W^aTSa(Za) could learn the dynamics of the robot, with W^a being the NN weight and Sa being the regressor vector. e and ev are the position and velocity tracking errors, respectively, and K2 is the control gain.Since the control objective is to guarantee the estimation of both robot dynamics and the cost functionΥ^(t), the adaptive law is selected as follows:(18)W^˙a,iT=-σaW^a,iTSa+krΥ^Sa,where σa and kr are positive constants.On the other hand, as a fundamental element of the next-generation robots, the human-robot collaboration (HRC) has been widely studied by roboticists and NN is employed in HRC with its powerful learning ability. In [105], the NNs were employed to estimate the human partner’s motion intention in human-robot collaboration, such that the robot was able to actively follow its human partner. To adjust the robot’s role to lead or to follow according to the human’s intention, game theory was employed for fundamental analysis of human-robot interaction and an adaptation law was developed in [106]. Policy iteration combining with NN was adopted to provide a rigorous solution to the problem of the system equilibrium in human-robot interaction [107].


### 4.4. NN Based Robot Cognitive Control

According to the predictive processing theory [108], the human brain is always actively anticipating the incoming sensorimotor information. This process exists because the living beings exhibit latencies due to neural processing delays and a limited bandwidth in their sensorimotor processing. To compensate for such a delay, in human brain, neural feedback signals (including lateral and top-down connections) modulate the neural activities via inhibitory or excitatory connections by influencing the neuronal population coding of the bottom-up sensory-driven signals in the perception-action system. Similarly, in robotic systems, it is claimed that such a delay and a limited bandwidth also can be compensated by the predictive functions learnt by recurrent neural models. Such a learning process can be done via only visual processing [109] or in the loop of perception and action [110].Based on the hierarchical sensorimotor integration theory, which advocates that action and perception are intertwined by sharing the same representational basis [111], the representation on different levels of sensory perception does not explicitly represent actions; instead, there is an encoding of the possible future percept which is learnt from prior sensorimotor knowledge.In the Bayesian, once this perception and action links have been established after learning, these perception-action associations in this architecture allow the following operations.First, these associations allow predicting the perceptual outcome of given actions by means of the forward models (e.g., Bayesian model). It can be written as(19)PE∣A,I∝PA∣EPE∣I,whereE estimates the upcoming perception evidence given an executed actionA and other prior information you have already known in I. The term PE∣A,I suggests that a prelearnt model representing the possibility of a motor actionA will be executed given that a (possible) resulting sensory evidence E is perceived (backward computation).Second, these associations allow selecting an appropriate movement given an intended perceptual representation. From the backward computations introduced in the following equation, a predictive sensorimotor integration occurs:(20)PA∣E,G∝PE∣APA∣G,whereA indicates a particular action selected given the (intended) sensory information E and a goalG. Here we assume that one’s action is only determined by the current sensory input and the goal.In terms of its hierarchical organization, it also allows this operation: with bidirectional information pathways, a low level perception representation can be expressed on a higher level, with a more complex receptive field, and vice versaelow⇔ehigh. This can be realized by the bidirectional deep architectures such as [112]. Conceptually, these operations can be achieved by extracting statistical regularity shown in Figure 7.Figure 7
The process of cognitive control.Since both perception and action processes can be seen as temporal sequences, from the mathematical perspective, the recurrent networks are Turing-Complete and have a learning capacity to learn time sequences with arbitrary length [113], if properly trained. Furthermore, such recurrent connections can be placed in a hierarchical way in which the prediction functions on different layers attempt to predict the nonlinear time-series in different time-scales [114]. From this point, the recurrent neural network with parametric bias units (RNNPB) [115] and multiple time-scale recurrent neural networks (MTRNN) [116] were applied to predict sequences by understanding them in various temporal levels.The difference of the temporal levels controls the properties of the different levels of the presentation in the deep recurrent network. For instance, in the MTRNN network [112], the learning of each neuron follows the updating rule of classical firing rate models, in which the activity of a neuron is determined by the average firing rate of all the connected neurons. Additionally, the neuronal activity is also decaying over time following an updating rule of leaky integrator model. Assuming thei-th MTRNN neuron has the number ofN connections, the current membrane potential status of a neuron can be defined by both the previous activation and the current synaptic inputs:(21)ui,t+1=1-1τiui,t+1τi∑wi,jxj,t,where wi,j represents the synaptic weight from thej-th neuron to thei-th neuron, xj,t is the activity ofj-th neuron att-th time-step, and τ is the time-scale parameter which determines the decay rate of this neuron: a larger τ means their activities change slowly over time compared with those with a smaller time-scale parameter τ.In [117], the concepts of predictive coding were discussed in detail, where the learning, generation, and recognition of actions can be conducted by means of the principle of prediction error minimization. By using the predictive coding, the RNNPB and MTRNN are capable for both generating own actions and recognizing the same actions performed by others. Recently, the study on neurorobotics experiments has shown that the dynamic predictive coding scheme can be used to address fluctuations in temporal patterns when training a recurrent neural network (RNN) model [118]. This predictive coding scheme enables organisms to predict perceptual outcomes based on current intentions of actions to the external environment and to forecast perceptual sequences corresponding to given intention states [118].Based on this architecture, two-layer RNN models were utilized to extract visual information [119] and to understand intentions [120] or emotion status [121] in social robotics; three-layer RNN models were used to integrate and understand multimodal information for a humanoid iCub robot [112, 122]. Moreover, the predictive coding framework has been extended to variational Bayes predictive coding MTRNN, which can arbitrate between deterministic model and probabilistic model by setting a metaparameter [123]. Such extension could provide significant improvement in dealing with noisy fluctuated sensory inputs which robots are expected to experience in more real world setting. In [124], a MTRNN was employed to control a humanoid robot and experimental results have shown that, by using only partial training data, the control model can achieve generalization by learning in a lower feature perception level.The hierarchical structure of RNN exhibits a great learning capacity to store multimodal information which is beneficial for the robotic systems to understand and to predict in a complex environment. As the future models and applications, the state-of-the-art deep learning techniques or the motor actions of robotic systems can be further integrated into this predictive architecture.


## 4.1. NN Based Robotic Manipulator Control

Generally speaking, the control methods for robot manipulators can be roughly divided into two groups, model-free control and model based control. For the model-free control approaches like proportional-integral-derivative (PID) control, satisfactory control performance may not be guaranteed. In contrast, the model based control approaches exhibit better control behavior but heavily depend on the validity of the robot model. In practice, however, a perfect robotic dynamic model is always not available due to the complex mechanisms and uncertainties. Additionally, the payload may be varied according to different tasks, which makes the accurate dynamics model hard to be obtained in advance. To solve such problems, the NN approximation-based control methods have been used extensively in applications of robot manipulator control. A basic structure of the adaptive neural network control for robot manipulator is shown in Figure5. Consider a dynamic model of a robot manipulator given as follows [80]:(9)Mqq¨+Cq˙,qq˙+Gq=τ,where M(q), C(q˙,q), and G(q) are the inertial matrix, Coriolis matrix, and gravity vector, respectively. Then NN control design could be given as follows:(10)τd=-K1e1-K2e2+W^SZ,where e1 is the tracking error, e2 is the velocity tracking error, W^S(Z) is the NN controller with W^ being the weights matrix and S(Z) being the NN regressor vector, and K1 and K2 are control gains specified by the designer.Figure 5
A basic structure of the adaptive NN robot control.From (10), we can see that the robot controller consists of a PD-like controller and a NN controller. In traditional model based controllers, the dynamic model of the robot could be regarded as a feedforward to address the effect caused by the robot motion. In practice, however, M(q), C(q,q), and G(q) may not be known. Therefore, the NNs are used to approximate the unknown dynamics f=M(q)q¨+C(q˙,q)q˙+G(q) and to improve the performance of the system via the online estimation. To adapt the NN weights, adaptive laws are designed as follows:(11)W^˙=ΓSe-σW^,where Γ and σ are specified positive parameters. The last term of right-hand side of (11) is the sigma modification, which is used to enhance the convergence and robustness of the parameters adaptation.In [80], a NN based share control method was developed to control a teleoperated robot with environmental uncertainties. In this work, the RBFNN was constructed to compensate for the unknown dynamics of the teleoperated robot. Particularly, a shared control strategy was developed into the controller to achieve the automatic obstacle avoidance combining with the information of visual camera and the robot body, such that the obstacle could be successfully avoided and the operator could focus more on the operated task rather than the environment to guarantee the stability and manipulation. In addition, error transformations were integrated into the adaptive NN control to guarantee the transient control performance. It was shown that, by using the NN technique, the control performance in both kinematic level and dynamic level of the teleoperated robot was enhanced. In [81], an extreme learning machine (ELM) based control strategy was proposed for uncertain robot manipulators to identify both the elasticity and geometry of an object. This ELM was applied to deal with the unknown nonlinearity of the robot manipulator to enhance the control performance. Particularly, by utilizing ELM, the proposed controller could guarantee that the robot dynamics follow a reference model, such that the desired set point and the feedforward force could be updated to estimate the geometry and stiffness of the object. As a result, the reference model could be exactly matched with a limited number of iterations.In [82], the NN controller was also employed to control a wheel inverted pendulum, which has been decomposed into two subsystems, a fully actuated second-order planar moving subsystem and a passive first-order pendulum subsystem. Then the RBFNN was employed to compensate for the uncertain dynamics of the two subsystems by using its powerful learning ability, such that the enhanced control performance could be realized by using the NN learning. In [83], a global adaptive neural control was proposed for a class of robot manipulators with finite-time convergence learning performance. This control scheme employed a smooth switching mechanism combining with a nominal neural network controller and a robust controller to ensure global uniform ultimately bounded stability. The optimal weights were obtained by the finite-time estimation algorithm such that, after the learning process, the learning weights could be reused next time for repeated tasks. The global NN control mechanism has been further extended to the control of dual arm robot manipulator in [84], where knowledge of both robot manipulator and the grasping object is unavailable in advance. By integrating prescribed functions into the design of controller, the transient performance of the dual arm robot control was regularly guaranteed. The NN was also employed to deal with synchronization problem of multiple robot manipulators in [85], where the reference trajectories are only available for part of the team members. By using the NN approximation controller, the robot has shown better control performance with enhanced transient performance and enhanced robustness. A RBFNN was constructed to compensate for the nonlinear terms of a five-bar manipulator based on an error transformation function [86]. The NN control was also applied in the robot teleoperation control [87, 88]. Moreover, a NN approximation technique was employed to deal with the unknown dynamics, kinematics, and actuator properties in the manipulator tracking control [89].


## 4.2. NN Based Robot Control with Input Nonlinearities

Another challenge of the robot manipulator is that the input nonlinearities such as friction, dead-zone, and actuator saturation may inevitably exist in the robot systems. These input nonlinearities may lead to larger tracking errors and degeneration of the control performance. Therefore, a number of works have been proposed to handle the nonlinearities by utilizing the neural network design. A neural adaptive controller was designed to deal with the effect of input saturation of the robot manipulator in [90] as follows: (12)τ=-z1+W^DSDZDα1+W^CSCZCα1+W^GSGZG+Kpz2+ξ+Krsgn⁡z2,where z1 is the robot position tracking error, z2 is the velocity tracking error, and α1 is an auxiliary controller. W^D, W^C, and W^G are the NN weights, SD(ZD), SC(ZC), and SG(ZG) are the NN regressor vectors, and Kp and Kr are control gains specified by the designer. ξ is an auxiliary system designed to reduce the effect of the saturation with ξ˙ defined as follows. (13)ξ˙=-Kξξ-z2TΔτ+1/2ΔτTΔτξ2ξ+Δτξ≥μ0ξ<μ,where Δτ is the torque error caused by saturation, and Kξ is a small positive value. To update the NN weights, adaptive laws are designed as follows:(14)W^˙D=ΓDkSDkα1e2k-σDkW^DkW^˙C=ΓCkSCkα1e2k-σCkW^CkW^˙G=ΓGkSGkα1e2k-σGkW^Gk,where ΓDk, ΓCk, and ΓGk are specified positive parameters and σDk, σCk, and σGk are positive parameters.In [91], an adaptive neural network controller was constructed to approximate the input dead-zone and the uncertain dynamics of the robotic manipulator, while the output constraint was also considered in the feedback control. In [92], the NN was applied for the estimation of the unknown model parameters of a marine surface vessel and in [93] the full-state constraint of ann-link robotic manipulator was achieved by using the NN control. The NN controller was also constructed for flexible robotic manipulators to deal with the vibration suppression based on a lumped spring-mass model [94] while in [95], two RBFNNs were constructed for flexible robot manipulators to compensate for the unknown dynamics and the dead-zone effect, respectively.The NN has also been used in many important industrial fields, such as autonomous underwater vehicles (AUVs) and hypersonic flight vehicle (HFV). In [96], the NN has been constructed to deal with the attitude of AUVs in the presence of input dead-zone and uncertain model parameters. In [97], the adaptive neural control was employed to deal with underwater vehicle control in discrete-time domain encountered with the unknown input nonlinearities, external disturbance, and model uncertainties. Then the reinforcement learning was applied to address these uncertainties by using a critic NN and an action NN. The hypersonic flight vehicle control was investigated in [98] where the aerodynamic uncertainties and unknown disturbances were addressed by a disturbance observer based NN. In [99], a neural learning control was embedded in the HFV controller to achieve the global stability via a switching mechanism and a robust controller.


## 4.3. NN Based Human-Robot Interaction Control

Recently, there is a predominant tendency to employ the robots in the human-surrounded environment, such as household services or industrial applications, where humans and robots may interact with each other directly. Therefore, interaction control has become a promising research field and has been widely studied. In [100], a learning method was developed such that the dynamics of a robot arm could follow a target impedance model with only knowledge of the robotic structure (see Figure 6).Figure 6
Block of the learning impedence control.The NN was further employed in robot control in interaction with an environment [101], where impedance control was achieved with the completely unknown robotic dynamics. In [102], a learning method was developed such that the robot was able to adjust the impedance parameters when it interacted with unknown environments. In order to learn optimal impedance parameters in the robot manipulator control, an adaptive dynamic programming (ADP) method was employed when the robot interacted with unknown time-varying environments, where NNs were used for both critic and actor networks [103]. The ADP was also employed for coordination of multirobots [104], in which possible disagreement between different manipulators was handled and dynamics of both robots and the manipulated object were not required to be known.In this work, the controller consists of two parts, a critic network which was used to approximate the cost function, and an actual NN which was designed to control the robot. The critic NN is designed as follows [104]: (15)Υ^t=W^CSCZC,where ZC=ξ, ξ=[x˙oT,zT,xoT]T with xo being the position of the object and z being the tracking error, W^C is the NN weight, and SC is the regressor vector.The critic NN is used to approximate a cost functionc(t)=ξTQξ+uTRu, where u denotes the control input, and Q and R are positive definite matrix. Since the control objective is to minimize the control effort, the adaptation law is designed as(16)W^˙c=-σc∂Ec∂W^c=σcc-W^cTS˙CS˙C,where σc is the learning rate and Ec=(1/2)(c-W^cTS˙C)2.On the other hand, the actual NN control is designed to control the robot as(17)u=W^aTSaZa-e-K2ev,where W^aTSa(Za) could learn the dynamics of the robot, with W^a being the NN weight and Sa being the regressor vector. e and ev are the position and velocity tracking errors, respectively, and K2 is the control gain.Since the control objective is to guarantee the estimation of both robot dynamics and the cost functionΥ^(t), the adaptive law is selected as follows:(18)W^˙a,iT=-σaW^a,iTSa+krΥ^Sa,where σa and kr are positive constants.On the other hand, as a fundamental element of the next-generation robots, the human-robot collaboration (HRC) has been widely studied by roboticists and NN is employed in HRC with its powerful learning ability. In [105], the NNs were employed to estimate the human partner’s motion intention in human-robot collaboration, such that the robot was able to actively follow its human partner. To adjust the robot’s role to lead or to follow according to the human’s intention, game theory was employed for fundamental analysis of human-robot interaction and an adaptation law was developed in [106]. Policy iteration combining with NN was adopted to provide a rigorous solution to the problem of the system equilibrium in human-robot interaction [107].


## 4.4. NN Based Robot Cognitive Control

According to the predictive processing theory [108], the human brain is always actively anticipating the incoming sensorimotor information. This process exists because the living beings exhibit latencies due to neural processing delays and a limited bandwidth in their sensorimotor processing. To compensate for such a delay, in human brain, neural feedback signals (including lateral and top-down connections) modulate the neural activities via inhibitory or excitatory connections by influencing the neuronal population coding of the bottom-up sensory-driven signals in the perception-action system. Similarly, in robotic systems, it is claimed that such a delay and a limited bandwidth also can be compensated by the predictive functions learnt by recurrent neural models. Such a learning process can be done via only visual processing [109] or in the loop of perception and action [110].Based on the hierarchical sensorimotor integration theory, which advocates that action and perception are intertwined by sharing the same representational basis [111], the representation on different levels of sensory perception does not explicitly represent actions; instead, there is an encoding of the possible future percept which is learnt from prior sensorimotor knowledge.In the Bayesian, once this perception and action links have been established after learning, these perception-action associations in this architecture allow the following operations.First, these associations allow predicting the perceptual outcome of given actions by means of the forward models (e.g., Bayesian model). It can be written as(19)PE∣A,I∝PA∣EPE∣I,whereE estimates the upcoming perception evidence given an executed actionA and other prior information you have already known in I. The term PE∣A,I suggests that a prelearnt model representing the possibility of a motor actionA will be executed given that a (possible) resulting sensory evidence E is perceived (backward computation).Second, these associations allow selecting an appropriate movement given an intended perceptual representation. From the backward computations introduced in the following equation, a predictive sensorimotor integration occurs:(20)PA∣E,G∝PE∣APA∣G,whereA indicates a particular action selected given the (intended) sensory information E and a goalG. Here we assume that one’s action is only determined by the current sensory input and the goal.In terms of its hierarchical organization, it also allows this operation: with bidirectional information pathways, a low level perception representation can be expressed on a higher level, with a more complex receptive field, and vice versaelow⇔ehigh. This can be realized by the bidirectional deep architectures such as [112]. Conceptually, these operations can be achieved by extracting statistical regularity shown in Figure 7.Figure 7
The process of cognitive control.Since both perception and action processes can be seen as temporal sequences, from the mathematical perspective, the recurrent networks are Turing-Complete and have a learning capacity to learn time sequences with arbitrary length [113], if properly trained. Furthermore, such recurrent connections can be placed in a hierarchical way in which the prediction functions on different layers attempt to predict the nonlinear time-series in different time-scales [114]. From this point, the recurrent neural network with parametric bias units (RNNPB) [115] and multiple time-scale recurrent neural networks (MTRNN) [116] were applied to predict sequences by understanding them in various temporal levels.The difference of the temporal levels controls the properties of the different levels of the presentation in the deep recurrent network. For instance, in the MTRNN network [112], the learning of each neuron follows the updating rule of classical firing rate models, in which the activity of a neuron is determined by the average firing rate of all the connected neurons. Additionally, the neuronal activity is also decaying over time following an updating rule of leaky integrator model. Assuming thei-th MTRNN neuron has the number ofN connections, the current membrane potential status of a neuron can be defined by both the previous activation and the current synaptic inputs:(21)ui,t+1=1-1τiui,t+1τi∑wi,jxj,t,where wi,j represents the synaptic weight from thej-th neuron to thei-th neuron, xj,t is the activity ofj-th neuron att-th time-step, and τ is the time-scale parameter which determines the decay rate of this neuron: a larger τ means their activities change slowly over time compared with those with a smaller time-scale parameter τ.In [117], the concepts of predictive coding were discussed in detail, where the learning, generation, and recognition of actions can be conducted by means of the principle of prediction error minimization. By using the predictive coding, the RNNPB and MTRNN are capable for both generating own actions and recognizing the same actions performed by others. Recently, the study on neurorobotics experiments has shown that the dynamic predictive coding scheme can be used to address fluctuations in temporal patterns when training a recurrent neural network (RNN) model [118]. This predictive coding scheme enables organisms to predict perceptual outcomes based on current intentions of actions to the external environment and to forecast perceptual sequences corresponding to given intention states [118].Based on this architecture, two-layer RNN models were utilized to extract visual information [119] and to understand intentions [120] or emotion status [121] in social robotics; three-layer RNN models were used to integrate and understand multimodal information for a humanoid iCub robot [112, 122]. Moreover, the predictive coding framework has been extended to variational Bayes predictive coding MTRNN, which can arbitrate between deterministic model and probabilistic model by setting a metaparameter [123]. Such extension could provide significant improvement in dealing with noisy fluctuated sensory inputs which robots are expected to experience in more real world setting. In [124], a MTRNN was employed to control a humanoid robot and experimental results have shown that, by using only partial training data, the control model can achieve generalization by learning in a lower feature perception level.The hierarchical structure of RNN exhibits a great learning capacity to store multimodal information which is beneficial for the robotic systems to understand and to predict in a complex environment. As the future models and applications, the state-of-the-art deep learning techniques or the motor actions of robotic systems can be further integrated into this predictive architecture.


## 5. Conclusion

In summary, great achievements for control design of nonlinear system by means of neural networks have been gained in the last two decades. Despite the impossibility in identifying or listing all the related contributions in this short review, efforts have been made to summarize the recent progress in the area of NN control and its particular applications in the robot learning control, the robot interaction control, and the robot recognition control. In this paper, we have shown that significant progress of NN has been made in control of the nonlinear systems, in solving the optimization problem, in approximating the system dynamics, in dealing with the input nonlinearities, in human-robot interaction, and in the pattern recognition. All these developments accompany not only the development of techniques in control and advanced manufactures, but also theatrical progress in constructing and developing the neural networks. Although huge efforts have been made to embed the NN in practical control systems, there is still a large gap between the theory and practice. To improve the feasibility and usability, the evolutionary computing theory has been proposed to train the NNs. It can automatically find a near-optimal NN architecture and allow a NN to adapt its learning rule to its environment. However, the complex and long training process of the evolutionary algorithms deters their practical applications. More efforts need to be made to evolve the NN architecture and NN learning technique in the control design. On the other hand, in human brain, the neural activities are modulated via inhibitory or excitatory connections by influencing the neuronal population coding of the bottom-up sensory-driven signals in the perception-action system. In this sense, how to integrate the sensor-motor information into the network to make NNs more feasible to adapt to the environment and to resemble the capacity of the human brain deserves further investigations.In conclusion, a brief review on neural networks for the complex nonlinear systems is provided with adaptive neural control, NN based dynamic programming, evolution computing, and their practical applications in the robotic fields. We believe this area may promote increasing investigations in both theories and applications. And emerging topics, like deep learning [125–128], big data [129–131], and cloud computing, may be incorporated into the neural network control for complex systems; for example, deep neural networks could be used to process massive amounts of unsupervised data in complex scenarios, neural networks can be helpful in reducing the data dimensionality, and the optimization of NN training may be employed to enhance the learning and adaptation performance of robots.



---

*Source: 1895897-2017-10-31.xml*