brief paper adaptive optimal control for continuous time linear systems based on policy iteration in this paper we propose a new scheme based on adaptive critics for finding online the state feedback infinite horizon optimal control solution of linear continuous time systems using only partial knowledge regarding the system dynamics in other words the algorithm solves online an algebraic riccati equation without knowing the internal dynamics model of the system being based on a policy iteration technique the algorithm alternates between the policy evaluation and policy update steps until an update of the control policy will no longer improve the system performance the result is a direct adaptive control algorithm which converges to the optimal control solution without using an explicit a priori obtained model of the system internal dynamics the effectiveness of the algorithm is shown while finding the optimal load frequency controller for a power system

nearly optimal control laws for nonlinear systems with saturating actuators using a neural network hjb approach the hamilton jacobi bellman hjb equation corresponding to constrained control is formulated using a suitable nonquadratic functional it is shown that the constrained optimal control law has the largest region of asymptotic stability ras the value function of this hjb equation is solved for by solving for a sequence of cost functions satisfying a sequence of lyapunov equations le a neural network is used to approximate the cost function associated with each le using the method of least squares on a well defined region of attraction of an initial stabilizing controller as the order of the neural network is increased the least squares solution of the hjb equation converges uniformly to the exact solution of the inherently nonlinear hjb equation associated with the saturating control inputs the result is a nearly optimal constrained state feedback controller that has been tuned a priori off line

analysis and modification of newton s method for algebraic riccati equations when newton s method is applied to find the maximal symmetric solution of an algebraic riccati equation convergence can be guaranteed under moderate conditions in particular the initial guess need not be close to the solution the convergence is quadratic if the frechet derivative is invertible at the solution in this paper we examine the behaviour of the newton iteration when the derivative is not invertible at the solution we find that a simple modification can improve the performance of the newton iteration dramatically

multi step heuristic dynamic programming for optimal control of nonlinear discrete time systems abstract policy iteration and value iteration are two main iterative adaptive dynamic programming frameworks for solving optimal control problems policy iteration converges fast while requiring an initial stabilizing control policy which is a strict constraint in practice value iteration avoids the requirement of initial admissible control policy while converging much slowly this paper tries to utilize the advantages of policy iteration and value iteration and avoids their drawbacks at the same time therefore a multi step heuristic dynamic programming mshdp method is developed for solving the optimal control problem of nonlinear discrete time systems mshdp speeds up value iteration and avoids the requirement of initial admissible control policy in policy iteration at the same time the convergence theory of mshdp is established by proving that it converges to the solution of the bellman equation for implementation purpose the actor critic neural network nn structure is developed the critic nn is employed to estimate the value function and its nn weight vector is computed with a least square scheme the actor nn is used to estimate the control policy and a gradient descent method is proposed for updating its nn weight vector according to the comparative simulation studies on two examples the effectiveness and advantages of mshdp are verified

robust adaptive dynamic programming of two player zero sum games for continuous time linear systems in this brief an online robust adaptive dynamic programming algorithm is proposed for two player zero sum games of continuous time unknown linear systems with matched uncertainties which are functions of system outputs and states of a completely unknown exosystem the online algorithm is developed using the policy iteration pi scheme with only one iteration loop a new analytical method is proposed for convergence proof of the pi scheme the sufficient conditions are given to guarantee globally asymptotic stability and suboptimal property of the closed loop system simulation studies are conducted to illustrate the effectiveness of the proposed method

online adaptive optimal control for continuous time nonlinear systems with completely unknown dynamics an online adaptive optimal control is proposed for continuous time nonlinear systems with completely unknown dynamics which is achieved by developing a novel identifier critic based approximate dynamic programming algorithm with a dual neural network nn approximation structure first an adaptive nn identifier is designed to obviate the requirement of complete knowledge of system dynamics and a critic nn is employed to approximate the optimal value function then the optimal control law is computed based on the information from the identifier nn and the critic nn so that the actor nn is not needed in particular a novel adaptive law design method with the parameter estimation error is proposed to online update the weights of both identifier nn and critic nn simultaneously which converge to small neighbourhoods around their ideal values the closed loop system stability and the convergence to small vicinity around the optimal solution are all proved by means of the lyapunov theory the proposed ada

integral q learning and explorized policy iteration for adaptive optimal control of continuous time linear systems this paper proposes an integral q learning for continuous time ct linear time invariant lti systems which solves a linear quadratic regulation lqr problem in real time for a given system and a value function without knowledge about the system dynamics a and b here q learning is referred to as a family of reinforcement learning methods which find the optimal policy by interaction with an uncertain environment in the evolution of the algorithm we first develop an explorized policy iteration pi method which is able to deal with known exploration signals then the integral q learning algorithm for ct lti systems is derived based on this pi and the variants of q functions derived from the singular perturbation of the control input the proposed q learning scheme evaluates the current value function and the improved control policy at the same time and are proven stable and convergent to the lq optimal solution provided that the initial policy is stabilizing for the proposed algorithms practical online implementation methods are investigated in terms of persistency of excitation pe and explorations finally simulation results are provided for the better comparison and verification of the performance

scatter balance an angle based supervised dimensionality reduction subspace selection is widely applied in data classification clustering and visualization the samples projected into subspace can be processed efficiently in this paper we research the linear discriminant analysis lda and maximum margin criterion mmc algorithms intensively and analyze the effects of scatters to subspace selection meanwhile we point out the boundaries of scatters in lda and mmc algorithms to illustrate the differences and similarities of subspace selection in different circumstances besides the effects of outlier classes on subspace selection are also analyzed according to the above analysis we propose a new subspace selection method called angle linear discriminant embedding alde on the basis of angle measurement alde utilizes the cosine of the angle to get new within class and between class scatter matrices and avoids the small sample size problem simultaneously to deal with high dimensional data we extend alde to a two stage alde ts alde the synthetic data experiments indicate that alde can balance the within class and between class scatters and be robust to outlier classes the experimental results based on uci machine learning repository and image databases show that ts alde has a lower time complexity than alde while processing high dimensional data

data driven zero sum neuro optimal control for a class of continuous time unknown nonlinear systems with disturbance using adp this paper is concerned with a new data driven zero sum neuro optimal control problem for continuous time unknown nonlinear systems with disturbance according to the input output data of the nonlinear system an effective recurrent neural network is introduced to reconstruct the dynamics of the nonlinear system considering the system disturbance as a control input a two player zero sum optimal control problem is established adaptive dynamic programming adp is developed to obtain the optimal control under the worst case of the disturbance three single layer neural networks including one critic and two action networks are employed to approximate the performance index function the optimal control law and the disturbance respectively for facilitating the implementation of the adp method convergence properties of the adp method are developed to show that the system state will converge to a finite neighborhood of the equilibrium the weight matrices of the critic and the two action networks are also convergent to finite neighborhoods of their optimal ones finally the simulation results will show the effectiveness of the developed data driven adp methods

locality pursuit embedding dimensionality reduction techniques are widespread in pattern recognition research principal component analysis as one of the most popular methods used is optimal when the data points reside on a linear subspace nevertheless it may fail to preserve the local structure if the data reside on some nonlinear manifold which is indisputably important in many real applications especially when nearest neighbor search is involved in this paper we propose locality pursuit embedding a linear algorithm that arises by solving a variational problem it produces a linear embedding that respects the local geometrical structure described by the euclidean distances some illustrative examples are presented along with applications to real data sets

an efficient discriminant based solution for small sample size problem classification of high dimensional statistical data is usually not amenable to standard pattern recognition techniques because of an underlying small sample size problem to address the problem of high dimensional data classification in the face of a limited number of samples a novel principal component analysis pca based feature extraction classification scheme is proposed the proposed method yields a piecewise linear feature subspace and is particularly well suited to difficult recognition problems where achievable classification rates are intrinsically low such problems are often encountered in cases where classes are highly overlapped or in cases where a prominent curvature in data renders a projection onto a single linear subspace inadequate the proposed feature extraction classification method uses class dependent pca in conjunction with linear discriminant feature extraction and performs well on a variety of real world datasets ranging from digit recognition to classification of high dimensional bioinformatics and brain imaging data

reinforcement learning design based adaptive tracking control with less learning parameters for nonlinear discrete time mimo systems based on the neural network nn approximator an online reinforcement learning algorithm is proposed for a class of affine multiple input and multiple output mimo nonlinear discrete time systems with unknown functions and disturbances in the design procedure two networks are provided where one is an action network to generate an optimal control signal and the other is a critic network to approximate the cost function an optimal control signal and adaptation laws can be generated based on two nns in the previous approaches the weights of critic and action networks are updated based on the gradient descent rule and the estimations of optimal weight vectors are directly adjusted in the design consequently compared with the existing results the main contributions of this paper are 1 only two parameters are needed to be adjusted and thus the number of the adaptation laws is smaller than the previous results and 2 the updating parameters do not depend on the number of the subsystems for mimo systems and the tuning rules are replaced by adjusting the norms on optimal weight vectors in both action and critic networks it is proven that the tracking errors the adaptation laws and the control inputs are uniformly bounded using lyapunov analysis method the simulation examples are employed to illustrate the effectiveness of the proposed algorithm

neural network based online simultaneous policy update algorithm for solving the hji equation in nonlinear h_ infty control it is well known that the nonlinear h state feedback control problem relies on the solution of the hamilton jacobi isaacs hji equation which is a nonlinear partial differential equation that has proven to be impossible to solve analytically in this paper a neural network nn based online simultaneous policy update algorithm spua is developed to solve the hji equation in which knowledge of internal system dynamics is not required first we propose an online spua which can be viewed as a reinforcement learning technique for two players to learn their optimal actions in an unknown environment the proposed online spua updates control and disturbance policies simultaneously thus only one iterative loop is needed second the convergence of the online spua is established by proving that it is mathematically equivalent to newton s method for finding a fixed point in a banach space third we develop an actor critic structure for the implementation of the online spua in which only one critic nn is needed for approximating the cost function and a least square method is given for estimating the nn weight parameters finally simulation studies are provided to demonstrate the effectiveness of the proposed algorithm

h _ infty tracking control of completely unknown continuous time systems via off policy reinforcement learning this paper deals with the design of an h infty tracking controller for nonlinear continuous time systems with completely unknown dynamics a general bounded l 2 gain tracking problem with a discounted performance function is introduced for the h infty tracking a tracking hamilton jacobi isaac hji equation is then developed that gives a nash equilibrium solution to the associated min max optimization problem a rigorous analysis of bounded l 2 gain and stability of the control solution obtained by solving the tracking hji equation is provided an upper bound is found for the discount factor to assure local asymptotic stability of the tracking error dynamics an off policy reinforcement learning algorithm is used to learn the solution to the tracking hji equation online without requiring any knowledge of the system dynamics convergence of the proposed algorithm to the solution to the tracking hji equation is shown simulation examples are provided to verify the effectiveness of the proposed method

output feedback adaptive optimal control of interconnected systems based on robust adaptive dynamic programming this paper studies the adaptive and optimal output feedback problem for continuous time uncertain systems with nonlinear dynamic uncertainties data driven output feedback control policies are developed by approximate adaptive dynamic programming adp based on both policy iteration and value iteration methods the obtained adaptive and optimal output feedback controllers differ from the existing literature on the adp in that they are derived from sampled data systems theory and are guaranteed to be robust to dynamic uncertainties a small gain condition is given under which the overall system is globally asymptotically stable at the origin an application to power systems is given to test the effectiveness of the proposed approaches

finite horizon control constrained nonlinear optimal control using single network adaptive critics to synthesize fixed final time control constrained optimal controllers for discrete time nonlinear control affine systems a single neural network nn based controller called the finite horizon single network adaptive critic is developed in this paper inputs to the nn are the current system states and the time to go and the network outputs are the costates that are used to compute optimal feedback control control constraints are handled through a nonquadratic cost function convergence proofs of 1 the reinforcement learning based training method to the optimal solution 2 the training error and 3 the network weights are provided the resulting controller is shown to solve the associated time varying hamilton jacobi bellman equation and provide the fixed final time optimal solution performance of the new synthesis technique is demonstrated through different examples including an attitude control problem wherein a rigid spacecraft performs a finite time attitude maneuver subject to control bounds the new formulation has great potential for implementation since it consists of only one nn with single set of weights and it provides comprehensive feedback solutions online though it is trained offline

lie point symmetries conservation laws and solutions of a space dependent reaction diffusion equation we find the lie point symmetries of a nonlinear population model i e a second order reaction diffusion equation with a variable coefficient b x and classify the model into three kinds then with the help of the lie point symmetries and self adjointness of each kind using a general theorem on conservation law ibragimov 2007 we establish the conservation laws corresponding to every lie point symmetry obtained in addition some exact solutions are constructed

multi agent discrete time graphical games and reinforcement learning solutions this paper introduces a new class of multi agent discrete time dynamic games known in the literature as dynamic graphical games for that reason a local performance index is defined for each agent that depends only on the local information available to each agent nash equilibrium policies and best response policies are given in terms of the solutions to the discrete time coupled hamilton jacobi equations since in these games the interactions between the agents are prescribed by a communication graph structure we have to introduce a new notion of nash equilibrium it is proved that this notion holds if all agents are in nash equilibrium and the graph is strongly connected a novel reinforcement learning value iteration algorithm is given to solve the dynamic graphical games in an online manner along with its proof of convergence the policies of the agents form a nash equilibrium when all the agents in the neighborhood update their policies and a best response outcome when the agents in the neighborhood are kept constant the paper brings together discrete hamiltonian mechanics distributed multi agent control optimal control theory and game theory to formulate and solve these multi agent dynamic graphical games a simulation example shows the effectiveness of the proposed approach in a leader synchronization case along with optimality guarantees

policy iterations on the hamilton jacobi isaacs equation for h_ infty state feedback control with input saturation an hinfin suboptimal state feedback controller for constrained input systems is derived using the hamilton jacobi isaacs hji equation of a corresponding zero sum game that uses a special quasi norm to encode the constraints on the input the unique saddle point in feedback strategy form is derived using policy iterations on both players the hji equation is broken into a sequence of differential equations linear in the cost for which closed form solutions are easier to obtain policy iterations on the disturbance are shown to converge to the available storage function of the associated l2 gain dissipative dynamics the resulting constrained optimal control feedback strategy has the largest domain of validity within which l2 performance for a given gamma is guaranteed

value iteration and adaptive dynamic programming for data driven adaptive optimal control design this paper presents a novel non model based data driven adaptive optimal controller design for linear continuous time systems with completely unknown dynamics inspired by the stochastic approximation theory a continuous time version of the traditional value iteration vi algorithm is presented with rigorous convergence analysis this vi method is crucial for developing new adaptive dynamic programming methods to solve the adaptive optimal control problem and the stochastic robust optimal control problem for linear continuous time systems fundamentally different from existing results the a priori knowledge of an initial admissible control policy is no longer required the efficacy of the proposed methodology is illustrated by two examples and a brief comparative study between vi and earlier policy iteration methods

a numerical algorithm for optimal feedback gains in high dimensional linear quadratic regulator problems a hybrid method for computing the feedback gains in linear quadratic regulator problems is proposed the method which combines use of a chandrasekhar type system with an iteration of the newton kleinman form with variable acceleration parameter smith schemes is formulated to efficiently compute directly the feedback gains rather than solutions of an associated riccati equation the hybrid method is particularly appropirate when used with large dimensional systems such as those arising in approximating infinite dimensional distributed parameter control systems e g those governed by delay differential and partial differential equations computational advantages of the proposed algorithm over the standard eigenvector potter laub schur based techniques are discussed and numerical evidence of the efficacy of these ideas is presented

solving the algebraic riccati equation with the matrix sign function abstract this paper presents some improvements to the matrix sign function algorithm for the algebraic riccati equation a simple reorganization changes nonsymmetric matrix inversions into symmetric matrix inversions scaling accelerates convergence of the basic iteration and yields a new quadratic formula for certain 2 by 2 algebraic riccati equations numerical experience suggests the algorithm be supplemented with a refinement strategy similar to iterative refinement for systems of linear equations refinement also produces an error estimate the resulting procedure is numerically stable it compares favorably with current schur vector based algorithms