optimal tracking control of nonlinear partially unknown constrained input systems using integral reinforcement learning in this paper a new formulation for the optimal tracking control problem otcp of continuous time nonlinear systems is presented this formulation extends the integral reinforcement learning irl technique a method for solving optimal regulation problems to learn the solution to the otcp unlike existing solutions to the otcp the proposed method does not need to have or to identify knowledge of the system drift dynamics and it also takes into account the input constraints a priori an augmented system composed of the error system dynamics and the command generator dynamics is used to introduce a new nonquadratic discounted performance function for the otcp this encodes the input constrains into the optimization problem a tracking hamilton jacobi bellman hjb equation associated with this nonquadratic performance function is derived which gives the optimal control solution an online irl algorithm is presented to learn the solution to the tracking hjb equation without knowing the system drift dynamics convergence to a near optimal control solution and stability of the whole system are shown under a persistence of excitation condition simulation examples are provided to show the effectiveness of the proposed method

the petrov galerkin and iterated petrov galerkin methods for second kind integral equations we develop in this paper a theoretical framework for the analysis of convergence for the petrov galerkin method and superconvergence for the iterated petrov galerkin method for fredholm integral equations of the second kind as important approaches to the analysis we introduce notions of the generalized best approximation and the regular pair of trial space sequence and test space sequence in hilbert spaces we characterize the regular pair in terms of the angle of two space sequences or the generalized best approximation projections several specific constructions of the petrov galerkin elements for equations of both one dimension and two dimensions are presented and the convergence of the petrov galerkin method and the iterated petrov galerkin method using these elements is proved

adaptive dynamic programming for a class of complex valued nonlinear systems in this brief an optimal control scheme based on adaptive dynamic programming adp is developed to solve infinite horizon optimal control problems of continuous time complex valued nonlinear systems a new performance index function is established on the basis of complex valued state and control using system transformations the complex valued system is transformed into a real valued one which overcomes cauchy riemann conditions effectively with the transformed system and the performance index function a new adp method is developed to obtain the optimal control law by using neural networks a compensation controller is developed to compensate the approximation errors of neural networks stability properties of the nonlinear system are analyzed and convergence properties of the weights for neural networks are presented finally simulation results demonstrate the performance of the developed optimal control scheme for complex valued nonlinear systems

online solution of two player zero sum games for continuous time nonlinear systems with completely unknown dynamics regarding two player zero sum games of continuous time nonlinear systems with completely unknown dynamics this paper presents an online adaptive algorithm for learning the nash equilibrium solution i e the optimal policy pair first for known systems the simultaneous policy updating algorithm spua is reviewed a new analytical method to prove the convergence is presented then based on the spua without using a priori knowledge of any system dynamics an online algorithm is proposed to simultaneously learn in real time either the minimal nonnegative solution of the hamilton jacobi isaacs hji equation or the generalized algebraic riccati equation for linear systems as a special case along with the optimal policy pair the approximate solution to the hji equation and the admissible policy pair is reexpressed by the approximation theorem the unknown constants or weights of each are identified simultaneously by resorting to the recursive least square method the convergence of the online algorithm to the optimal solutions is provided a practical online algorithm is also developed simulation results illustrate the effectiveness of the proposed method

brief paper adaptive optimal control for continuous time linear systems based on policy iteration in this paper we propose a new scheme based on adaptive critics for finding online the state feedback infinite horizon optimal control solution of linear continuous time systems using only partial knowledge regarding the system dynamics in other words the algorithm solves online an algebraic riccati equation without knowing the internal dynamics model of the system being based on a policy iteration technique the algorithm alternates between the policy evaluation and policy update steps until an update of the control policy will no longer improve the system performance the result is a direct adaptive control algorithm which converges to the optimal control solution without using an explicit a priori obtained model of the system internal dynamics the effectiveness of the algorithm is shown while finding the optimal load frequency controller for a power system

fast fourier galerkin methods for solving singular boundary integral equations numerical integration and precondition we develop a fast fully discrete fourier galerkin method for solving a class of singular boundary integral equations we prove that the number of multiplications used in generating the compressed matrix is o nlog 3n and the solution of the proposed method preserves the optimal convergence order o n t where n is the order of the fourier basis functions used in the method and t denotes the degree of regularity of the exact solution moreover we propose a preconditioning which ensures the numerical stability when solving the preconditioned linear system numerical examples are presented to confirm the theoretical estimates and to demonstrate the approximation accuracy and computational efficiency of the proposed algorithm

an iterative adaptive dynamic programming method for solving a class of nonlinear zero sum differential games in this paper a new iterative adaptive dynamic programming adp method is proposed to solve a class of continuous time nonlinear two person zero sum differential games the idea is to use the adp technique to obtain the optimal control pair iteratively which makes the performance index function reach the saddle point of the zero sum differential games if the saddle point does not exist the mixed optimal control pair is obtained to make the performance index function reach the mixed optimum stability analysis of the nonlinear systems is presented and the convergence property of the performance index function is also proved two simulation examples are given to illustrate the performance of the proposed method

nearly optimal control laws for nonlinear systems with saturating actuators using a neural network hjb approach the hamilton jacobi bellman hjb equation corresponding to constrained control is formulated using a suitable nonquadratic functional it is shown that the constrained optimal control law has the largest region of asymptotic stability ras the value function of this hjb equation is solved for by solving for a sequence of cost functions satisfying a sequence of lyapunov equations le a neural network is used to approximate the cost function associated with each le using the method of least squares on a well defined region of attraction of an initial stabilizing controller as the order of the neural network is increased the least squares solution of the hjb equation converges uniformly to the exact solution of the inherently nonlinear hjb equation associated with the saturating control inputs the result is a nearly optimal constrained state feedback controller that has been tuned a priori off line

computational adaptive optimal control for continuous time linear systems with completely unknown dynamics this paper presents a novel policy iteration approach for finding online adaptive optimal controllers for continuous time linear systems with completely unknown system dynamics the proposed approach employs the approximate adaptive dynamic programming technique to iteratively solve the algebraic riccati equation using the online information of state and input without requiring the a priori knowledge of the system matrices in addition all iterations can be conducted by using repeatedly the same state and input information on some fixed time intervals a practical online algorithm is developed in this paper and is applied to the controller design for a turbocharged diesel engine with exhaust gas recirculation finally several aspects of future work are discussed

von neumann indices and classes of positive definite functions with view to applications we establish a correspondence between two problems i the problem of finding continuous positive definite extensions of functions f which are defined on open bounded domains in r on the one hand and ii spectral theory for elliptic differential operators acting on constant coefficients a novelty in our approach is the use of a reproducing kernel hilbert space hf computed directly from f as well as algorithms for computing relevant orthonormal bases in hf

multi agent differential graphical games online adaptive learning solution for synchronization with optimality multi agent systems arise in several domains of engineering and can be used to solve problems which are difficult for an individual agent to solve strategies for team decision problems including optimal control n player games h infinity control and non zero sum and so on are normally solved for off line by solving associated matrix equations such as the coupled riccati equations or coupled hamilton jacobi equations however using that approach players cannot change their objectives online in real time without calling for a completely new off line solution for the new strategies therefore in this paper we bring together cooperative control reinforcement learning and game theory to present a multi agent formulation for the online solution of team games the notion of graphical games is developed for dynamical systems where the dynamics and performance indices for each node depend only on local neighbor information it is shown that standard definitions for nash equilibrium are not sufficient for graphical games and a new definition of interactive nash equilibrium is given we give a cooperative policy iteration algorithm for graphical games that converges to the best response when the neighbors of each agent do not update their policies and to the cooperative nash equilibrium when all agents update their policies simultaneously this is used to develop methods for online adaptive learning solutions of graphical games in real time along with proofs of stability and convergence

reinforcement learning in continuous time and space this article presents a reinforcement learning framework for continuous time dynamical systems without a priori discretization of time state and action based on the hamilton jacobi bellman hjb equation for infinite horizon discounted reward problems we derive algorithms for estimating value functions and improving policies with the use of function approximators the processof value function estimation is formulated asthe minimization of a continuous time form of the temporal difference td error update methods based on backward euler approximation and exponential eligibility traces are derived and their correspondences with the conventional residual gradient td 0 and td lambda algorithms are shown for policy improvement two methods a continuous actor critic method and a value gradient based greedy policy are formulated as a special case of the latter a nonlinear feedback control law using the value gradient and the model of the input gain is derived the advantage updating a model free algorithm derived previously is also formulated in the hjbbased framework the performance of the proposed algorithms is first tested in a nonlinear control task of swinging a pendulum up with limited torque it is shown in the simulations that 1 the task is accomplished by the continuous actor critic method in a number of trials several times fewer than by the conventional discrete actor critic method 2 among the continuous policy update methods the value gradient based policy with a known or learned dynamic model performs several times better than the actor critic method and 3 a value function update using exponential eligibility traces is more efficient and stable than that based on euler approximation the algorithms are then tested in a higher dimensional task cartpole swing up this task is accomplished in several hundred trials using the value gradient based policy with a learned dynamic model

adaptive optimal control of highly dissipative nonlinear spatially distributed processes with neuro dynamic programming highly dissipative nonlinear partial differential equations pdes are widely employed to describe the system dynamics of industrial spatially distributed processes sdps in this paper we consider the optimal control problem of the general highly dissipative sdps and propose an adaptive optimal control approach based on neuro dynamic programming ndp initially karhunen loeve decomposition is employed to compute empirical eigenfunctions eefs of the sdp based on the method of snapshots these eefs together with singular perturbation technique are then used to obtain a finite dimensional slow subsystem of ordinary differential equations that accurately describes the dominant dynamics of the pde system subsequently the optimal control problem is reformulated on the basis of the slow subsystem which is further converted to solve a hamilton jacobi bellman hjb equation hjb equation is a nonlinear pde that has proven to be impossible to solve analytically thus an adaptive optimal control method is developed via ndp that solves the hjb equation online using neural network nn for approximating the value function and an online nn weight tuning law is proposed without requiring an initial stabilizing control policy moreover by involving the nn estimation error we prove that the original closed loop pde system with the adaptive optimal control policy is semiglobally uniformly ultimately bounded finally the developed method is tested on a nonlinear diffusion convection reaction process and applied to a temperature cooling fin of high speed aerospace vehicle and the achieved results show its effectiveness

online adaptive optimal control for continuous time nonlinear systems with completely unknown dynamics an online adaptive optimal control is proposed for continuous time nonlinear systems with completely unknown dynamics which is achieved by developing a novel identifier critic based approximate dynamic programming algorithm with a dual neural network nn approximation structure first an adaptive nn identifier is designed to obviate the requirement of complete knowledge of system dynamics and a critic nn is employed to approximate the optimal value function then the optimal control law is computed based on the information from the identifier nn and the critic nn so that the actor nn is not needed in particular a novel adaptive law design method with the parameter estimation error is proposed to online update the weights of both identifier nn and critic nn simultaneously which converge to small neighbourhoods around their ideal values the closed loop system stability and the convergence to small vicinity around the optimal solution are all proved by means of the lyapunov theory the proposed ada

integral q learning and explorized policy iteration for adaptive optimal control of continuous time linear systems this paper proposes an integral q learning for continuous time ct linear time invariant lti systems which solves a linear quadratic regulation lqr problem in real time for a given system and a value function without knowledge about the system dynamics a and b here q learning is referred to as a family of reinforcement learning methods which find the optimal policy by interaction with an uncertain environment in the evolution of the algorithm we first develop an explorized policy iteration pi method which is able to deal with known exploration signals then the integral q learning algorithm for ct lti systems is derived based on this pi and the variants of q functions derived from the singular perturbation of the control input the proposed q learning scheme evaluates the current value function and the improved control policy at the same time and are proven stable and convergent to the lq optimal solution provided that the initial policy is stabilizing for the proposed algorithms practical online implementation methods are investigated in terms of persistency of excitation pe and explorations finally simulation results are provided for the better comparison and verification of the performance

neural network observer based optimal control for unknown nonlinear systems using adaptive dynamic programming in this paper an observer based optimal control scheme is developed for unknown nonlinear systems using adaptive dynamic programming adp algorithm first a neural network nn observer is designed to estimate system states then based on the observed states a neuro controller is constructed via adp method to obtain the optimal control in this design two nn structures are used a three layer nn is used to construct the observer which can be applied to systems with higher degrees of nonlinearity and without a priori knowledge of system dynamics and a critic nn is employed to approximate the value function the optimal control law is computed using the critic nn and the observer nn uniform ultimate boundedness of the closed loop system is guaranteed the actor critic and observer structures are all implemented in real time continuously and simultaneously finally simulation results are presented to demonstrate the effectiveness of the proposed control scheme

model based reinforcement learning for infinite horizon approximate optimal tracking this brief paper provides an approximate online adaptive solution to the infinite horizon optimal tracking problem for control affine continuous time nonlinear systems with unknown drift dynamics to relax the persistence of excitation condition model based reinforcement learning is implemented using a concurrent learning based system identifier to simulate experience by evaluating the bellman error over unexplored areas of the state space tracking of the desired trajectory and convergence of the developed policy to a neighborhood of the optimal policy are established via lyapunov based stability analysis simulation results demonstrate the effectiveness of the developed technique

l sub 2 gain analysis of nonlinear systems and nonlinear state feedback h sub infinity control previously obtained results on l2 gain analysis of smooth nonlinear systems are unified and extended using an approach based on hamilton jacobi equations and inequalities and their relation to invariant manifolds of an associated hamiltonian vector field on the basis of these results a nonlinear analog is obtained of the simplest part of a state space approach to linear h sub infinity control namely the state feedback h sub infinity optimal control problem furthermore the relation with h sub infinity control of the linearized system is dealt with

a sparse finite element method with high accuracy part i in this paper we develop and analyze a new finite element method called the sparse finite element method for second order elliptic problems this method involves much fewer degrees of freedom than the standard finite element method we show nevertheless that such a sparse finite element method still possesses the superconvergence and other high accuracy properties same as those of the standard finite element method the main technique in our analysis is the use of some integral identities

a fast collocation method for solving stochastic integral equations based on sparse grid multiscale piecewise polynomial bases we develop a fast collocation method for solving fredholm integral equations of the second kind with stochastic loading terms it is proved that the proposed method preserves the optimal rate of convergence and has linear up to a logarithmic factor computational complexity the reduction in computational costs come from three aspects the adoption of the collocation principle the use of the sparse grid for the solution space and the employment of a truncation strategy for the coefficient matrix of the resulting linear system numerical experiments confirm the theoretical results on approximation accuracy and computational efficiency of the proposed method

error analysis of the combination technique the combination technique is a method to reduce the computational time in the numerical approximation of partial differential equations in this paper we present a new technique to analyze the convergence rate of the combination technique this technique is applied to general second order elliptic differential equations in two dimensions furthermore it is proved that the combination technique for poisson s equation convergences in arbitrary dimensions

hamiltonian driven adaptive dynamic programming for continuous nonlinear dynamical systems this paper presents a hamiltonian driven framework of adaptive dynamic programming adp for continuous time nonlinear systems which consists of evaluation of an admissible control comparison between two different admissible policies with respect to the corresponding the performance function and the performance improvement of an admissible control it is showed that the hamiltonian can serve as the temporal difference for continuous time systems in the hamiltonian driven adp the critic network is trained to output the value gradient then the inner product between the critic and the system dynamics produces the value derivative under some conditions the minimization of the hamiltonian functional is equivalent to the value function approximation an iterative algorithm starting from an arbitrary admissible control is presented for the optimal control approximation with its convergence proof the implementation is accomplished by a neural network approximation two simulation studies demonstrate the effectiveness of hamiltonian driven adp

data driven zero sum neuro optimal control for a class of continuous time unknown nonlinear systems with disturbance using adp this paper is concerned with a new data driven zero sum neuro optimal control problem for continuous time unknown nonlinear systems with disturbance according to the input output data of the nonlinear system an effective recurrent neural network is introduced to reconstruct the dynamics of the nonlinear system considering the system disturbance as a control input a two player zero sum optimal control problem is established adaptive dynamic programming adp is developed to obtain the optimal control under the worst case of the disturbance three single layer neural networks including one critic and two action networks are employed to approximate the performance index function the optimal control law and the disturbance respectively for facilitating the implementation of the adp method convergence properties of the adp method are developed to show that the system state will converge to a finite neighborhood of the equilibrium the weight matrices of the critic and the two action networks are also convergent to finite neighborhoods of their optimal ones finally the simulation results will show the effectiveness of the developed data driven adp methods

sparse grids for boundary integral equations the potential of sparse grid discretizations for solving boundary integral equations is studied for the screen problem on a square in mathbb r 3 theoretical and numerical results on approximation rates preconditioning adaptivity and compression for piecewise constant and linear sparse grid spaces are obtained

reinforcement learning for adaptive optimal control of unknown continuous time nonlinear systems with input constraints in this paper an adaptive reinforcement learning based solution is developed for the infinite horizon optimal control problem of constrained input continuous time nonlinear systems in the presence of nonlinearities with unknown structures two different types of neural networks nns are employed to approximate the hamilton jacobi bellman equation that is an recurrent nn is constructed to identify the unknown dynamical system and two feedforward nns are used as the actor and the critic to approximate the optimal control and the optimal cost respectively based on this framework the action nn and the critic nn are tuned simultaneously without the requirement for the knowledge of system drift dynamics moreover by using lyapunov s direct method the weights of the action nn and the critic nn are guaranteed to be uniformly ultimately bounded while keeping the closed loop system stable to demonstrate the effectiveness of the present approach simulation results are illustrated

nonlinear inversion based output tracking an inversion procedure is introduced for nonlinear systems which constructs a bounded input trajectory in the preimage of a desired output trajectory in the case of minimum phase systems the trajectory produced agrees with that generated by hirschorn s inverse dynamic system however the preimage trajectory is noncausal rather than unstable in the nonminimum phase case in addition the analysis leads to a simple geometric connection between the unstable manifold of the system zero dynamics and noncausality in the nonminimum phase case with the addition of stabilizing feedback to the preimage trajectory asymptotically exact output tracking is achieved tracking is demonstrated with a numerical example and compared to the well known byrnes isidori regulator rather than solving a partial differential equation to construct a regulator the inverse is calculated using a picard like interaction when preactuation is not possible noncausal inverse trajectories can be truncated resulting in the tracking error transients found in other control schemes

h _ infty tracking control of completely unknown continuous time systems via off policy reinforcement learning this paper deals with the design of an h infty tracking controller for nonlinear continuous time systems with completely unknown dynamics a general bounded l 2 gain tracking problem with a discounted performance function is introduced for the h infty tracking a tracking hamilton jacobi isaac hji equation is then developed that gives a nash equilibrium solution to the associated min max optimization problem a rigorous analysis of bounded l 2 gain and stability of the control solution obtained by solving the tracking hji equation is provided an upper bound is found for the discount factor to assure local asymptotic stability of the tracking error dynamics an off policy reinforcement learning algorithm is used to learn the solution to the tracking hji equation online without requiring any knowledge of the system dynamics convergence of the proposed algorithm to the solution to the tracking hji equation is shown simulation examples are provided to verify the effectiveness of the proposed method

degrees of freedom for the mimo interference channel in this correspondence we show that the exact number of spatial degrees of freedom dof for a two user nondegenerate full rank channel matrices multiple input multiple output mimo gaussian interference channel with m1 m2 antennas at transmitters 1 2 and n1 n2 antennas at the corresponding receivers and perfect channel knowledge at all transmitters and receivers is min m1 m2 n1 m2 max m1 n2 max m2 n1 a constructive achievability proof shows that zero forcing is sufficient to achieve all the available dof on the two user mimo interference channel we also show through an example of a share and transmit scheme how the gains of transmitter cooperation may be entirely offset by the cost of enabling that cooperation so that the available dof are not increased

neural network based online simultaneous policy update algorithm for solving the hji equation in nonlinear h_ infty control it is well known that the nonlinear h state feedback control problem relies on the solution of the hamilton jacobi isaacs hji equation which is a nonlinear partial differential equation that has proven to be impossible to solve analytically in this paper a neural network nn based online simultaneous policy update algorithm spua is developed to solve the hji equation in which knowledge of internal system dynamics is not required first we propose an online spua which can be viewed as a reinforcement learning technique for two players to learn their optimal actions in an unknown environment the proposed online spua updates control and disturbance policies simultaneously thus only one iterative loop is needed second the convergence of the online spua is established by proving that it is mathematically equivalent to newton s method for finding a fixed point in a banach space third we develop an actor critic structure for the implementation of the online spua in which only one critic nn is needed for approximating the cost function and a least square method is given for estimating the nn weight parameters finally simulation studies are provided to demonstrate the effectiveness of the proposed algorithm

sampling uniformly from the set of positive definite matrices with trace constraint we derive a parameterization of positive definite matrices using the cholesky decomposition in combination with hyperspherical coordinates based on the parameterization we develop a simple and efficient method to randomly generate positive definite matrices with constant or bounded trace according to a uniform distribution further we present an efficient implementation using the inversion method and either rejection sampling or transforming a beta distribution the matrix parameterization might be of independent interest whereas the random sampling algorithm finds applications in monte carlo simulations testing of algorithms and performance studies with the help of an abstract example we describe how the sampling method can be used to approximate the optimum in a difficult e g nonconvex optimization problem for which no solution or efficient global optimization algorithm is known in this paper we consider real as well as complex matrices

reinforcement q learning for optimal tracking control of linear discrete time systems with unknown dynamics abstract in this paper a novel approach based on the q learning algorithm is proposed to solve the infinite horizon linear quadratic tracker lqt for unknown discrete time systems in a causal manner it is assumed that the reference trajectory is generated by a linear command generator system an augmented system composed of the original system and the command generator is constructed and it is shown that the value function for the lqt is quadratic in terms of the state of the augmented system using the quadratic structure of the value function a bellman equation and an augmented algebraic riccati equation are for solving the lqt are derived in contrast to the standard solution of the lqt which requires the solution of an are and a noncausal difference equation simultaneously in the proposed method the optimal control input is obtained by only solving an augmented are a q learning algorithm is developed to solve online the augmented are without any knowledge about the system dynamics or the command generator convergence to the optimal solution is shown a simulation example is used to verify the effectiveness of the proposed control scheme

error bounds of adaptive dynamic programming algorithms for solving undiscounted optimal control problems in this paper we establish error bounds of adaptive dynamic programming algorithms for solving undiscounted infinite horizon optimal control problems of discrete time deterministic nonlinear systems we consider approximation errors in the update equations of both value function and control policy we utilize a new assumption instead of the contraction assumption in discounted optimal control problems we establish the error bounds for approximate value iteration based on a new error condition furthermore we also establish the error bounds for approximate policy iteration and approximate optimistic policy iteration algorithms it is shown that the iterative approximate value function can converge to a finite neighborhood of the optimal value function under some conditions to implement the developed algorithms critic and action neural networks are used to approximate the value function and control policy respectively finally a simulation example is given to demonstrate the effectiveness of the developed algorithms

model based reinforcement learning for approximate optimal regulation reinforcement learning rl based online approximate optimal control methods applied to deterministic systems typically require a restrictive persistence of excitation pe condition for convergence this paper develops a concurrent learning cl based implementation of model based rl to solve approximate optimal regulation problems online under a pe like rank condition the development is based on the observation that given a model of the system rl can be implemented by evaluating the bellman error at any number of desired points in the state space in this result a parametric system model is considered and a cl based parameter identifier is developed to compensate for uncertainty in the parameters uniformly ultimately bounded regulation of the system states to a neighborhood of the origin and convergence of the developed policy to a neighborhood of the optimal policy are established using a lyapunov based analysis and simulation results are presented to demonstrate the performance of the developed controller

model free optimal tracking control via critic only q learning model free control is an important and promising topic in control fields which has attracted extensive attention in the past few years in this paper we aim to solve the model free optimal tracking control problem of nonaffine nonlinear discrete time systems a critic only q learning coql method is developed which learns the optimal tracking control from real system data and thus avoids solving the tracking hamilton jacobi bellman equation first the q learning algorithm is proposed based on the augmented system and its convergence is established using only one neural network for approximating the q function the coql method is developed to implement the q learning algorithm furthermore the convergence of the coql method is proved with the consideration of neural network approximation error with the convergent q function obtained from the coql method the adaptive optimal tracking control is designed based on the gradient descent scheme finally the effectiveness of the developed coql method is demonstrated through simulation studies the developed coql method learns with off policy data and implements with a critic only structure thus it is easy to realize and overcome the inadequate exploration problem

feedback and optimal sensitivity model reference transformations multiplicative seminorms and approximate inverses in this paper the problem of sensitivity reduction by feedback is formulated as an optimization problem and separated from the problem of stabilization stable feedback schemes obtainable from a given plant are parameterized salient properties of sensitivity reducing schemes are derived and it is shown that plant uncertainty reduces the ability of feedback to reduce sensitivity the theory is developed for input output systems in a general setting of banach algebras and then specialized to a class of multivariable time invariant systems characterized by n times n matrices of h infty frequency response functions either with or without zeros in the right half plane the approach is based on the use of a weighted seminorm on the algebra of operators to measure sensitivity and on the concept of an approximate inverse approximate invertibility of the plant is shown to be a necessary and sufficient condition for sensitivity reduction an indicator of approximate invertibility called a measure of singularity is introduced the measure of singularity of a linear time invariant plant is shown to be determined by the location of its right half plane zeros in the absence of plant uncertainty the sensitivity to output disturbances can be reduced to an optimal value approaching the singularity measure in particular if there are no right half plane zeros sensitivity can be made arbitrarily small the feedback schemes used in the optimization of sensitivity resemble the lead lag networks of classical control design some of their properties and methods of constructing them in special cases are presented

two scale boolean galerkin discretizations for fredholm integral equations of the second kind in this paper some two scale boolean galerkin discretizations are proposed and analyzed for a class of fredholm integral equations of the second kind in multidimensions it is shown by both theory and numerics that this type of multiscale discretization algorithm not only significantly reduces the number of degrees of freedom but also produces very accurate approximations

online actor critic algorithm to solve the continuous time infinite horizon optimal control problem in this paper we discuss an online algorithm based on policy iteration for learning the continuous time ct optimal control solution with infinite horizon cost for nonlinear systems with known dynamics that is the algorithm learns online in real time the solution to the optimal control design hj equation this method finds in real time suitable approximations of both the optimal cost and the optimal control policy while also guaranteeing closed loop stability we present an online adaptive algorithm implemented as an actor critic structure which involves simultaneous continuous time adaptation of both actor and critic neural networks we call this synchronous policy iteration a persistence of excitation condition is shown to guarantee convergence of the critic to the actual optimal value function novel tuning algorithms are given for both critic and actor networks with extra nonstandard terms in the actor tuning law being required to guarantee closed loop dynamical stability the convergence to the optimal controller is proven and the stability of the system is also guaranteed simulation examples show the effectiveness of the new algorithm

finite horizon control constrained nonlinear optimal control using single network adaptive critics to synthesize fixed final time control constrained optimal controllers for discrete time nonlinear control affine systems a single neural network nn based controller called the finite horizon single network adaptive critic is developed in this paper inputs to the nn are the current system states and the time to go and the network outputs are the costates that are used to compute optimal feedback control control constraints are handled through a nonquadratic cost function convergence proofs of 1 the reinforcement learning based training method to the optimal solution 2 the training error and 3 the network weights are provided the resulting controller is shown to solve the associated time varying hamilton jacobi bellman equation and provide the fixed final time optimal solution performance of the new synthesis technique is demonstrated through different examples including an attitude control problem wherein a rigid spacecraft performs a finite time attitude maneuver subject to control bounds the new formulation has great potential for implementation since it consists of only one nn with single set of weights and it provides comprehensive feedback solutions online though it is trained offline

multi player non zero sum games online adaptive learning solution of coupled hamilton jacobi equations in this paper we present an online adaptive control algorithm based on policy iteration reinforcement learning techniques to solve the continuous time ct multi player non zero sum nzs game with infinite horizon for linear and nonlinear systems nzs games allow for players to have a cooperative team component and an individual selfish component of strategy the adaptive algorithm learns online the solution of coupled riccati equations and coupled hamilton jacobi equations for linear and nonlinear systems respectively this adaptive control method finds in real time approximations of the optimal value and the nzs nash equilibrium while also guaranteeing closed loop stability the optimal adaptive algorithm is implemented as a separate actor critic parametric network approximator structure for every player and involves simultaneous continuous time adaptation of the actor critic networks a persistence of excitation condition is shown to guarantee convergence of every critic to the actual optimal value function for that player a detailed mathematical analysis is done for 2 player nzs games novel tuning algorithms are given for the actor critic networks the convergence to the nash equilibrium is proven and stability of the system is also guaranteed this provides optimal adaptive control solutions for both non zero sum games and their special case the zero sum games simulation examples show the effectiveness of the new algorithm

a novel actor critic identifier architecture for approximate optimal control of uncertain nonlinear systems an online adaptive reinforcement learning based solution is developed for the infinite horizon optimal control problem for continuous time uncertain nonlinear systems a novel actor critic identifier aci is proposed to approximate the hamilton jacobi bellman equation using three neural network nn structures actor and critic nns approximate the optimal control and the optimal value function respectively and a robust dynamic neural network identifier asymptotically approximates the uncertain system dynamics an advantage of using the aci architecture is that learning by the actor critic and identifier is continuous and simultaneous without requiring knowledge of system drift dynamics convergence of the algorithm is analyzed using lyapunov based adaptive control methods a persistence of excitation condition is required to guarantee exponential convergence to a bounded region in the neighborhood of the optimal control and uniformly ultimately bounded uub stability of the closed loop system simulation results demonstrate the performance of the actor critic identifier method for approximate optimal control

multi agent discrete time graphical games and reinforcement learning solutions this paper introduces a new class of multi agent discrete time dynamic games known in the literature as dynamic graphical games for that reason a local performance index is defined for each agent that depends only on the local information available to each agent nash equilibrium policies and best response policies are given in terms of the solutions to the discrete time coupled hamilton jacobi equations since in these games the interactions between the agents are prescribed by a communication graph structure we have to introduce a new notion of nash equilibrium it is proved that this notion holds if all agents are in nash equilibrium and the graph is strongly connected a novel reinforcement learning value iteration algorithm is given to solve the dynamic graphical games in an online manner along with its proof of convergence the policies of the agents form a nash equilibrium when all the agents in the neighborhood update their policies and a best response outcome when the agents in the neighborhood are kept constant the paper brings together discrete hamiltonian mechanics distributed multi agent control optimal control theory and game theory to formulate and solve these multi agent dynamic graphical games a simulation example shows the effectiveness of the proposed approach in a leader synchronization case along with optimality guarantees

infinite horizon self learning optimal control of nonaffine discrete time nonlinear systems in this paper a novel iterative adaptive dynamic programming adp based infinite horizon self learning optimal control algorithm called generalized policy iteration algorithm is developed for nonaffine discrete time dt nonlinear systems generalized policy iteration algorithm is a general idea of interacting policy and value iteration algorithms of adp the developed generalized policy iteration algorithm permits an arbitrary positive semidefinite function to initialize the algorithm where two iteration indices are used for policy improvement and policy evaluation respectively it is the first time that the convergence admissibility and optimality properties of the generalized policy iteration algorithm for dt nonlinear systems are analyzed neural networks are used to implement the developed algorithm finally numerical examples are presented to illustrate the performance of the developed algorithm

on the numerical solution of nonlinear volterra fredholm integral equations by collocation methods particular cases of nonlinear mixed volterra fredholm integral equations of the second kind arise in the mathematical modeling of the spatio temporal development of an epidemic this paper is concerned with the numerical solution of general integral equations of this type by continuous time and discrete time spline collocation methods its focus is on the derivation and analysis of methods of high order of convergence

global l 2 gain design for a class of nonlinear systems abstract it is known that the so called h control problem of a nonlinear system is locally solvable if the corresponding problem for the linearized system can be solved by linear feedback in this paper we prove that this condition suffices to solve also a global h control problem for a fairly large class of nonlinear systems if one is free to choose a state dependent weight of the control input using a two way backward and forward recursive induction argument we simultaneously construct starting from a solution of the riccati algebraic equation a global solution of the hamilton jacobi isaacs partial differential equation arising in the nonlinear h control as well as a state feedback control law that achieves global disturbance attenuation with internal stability for the nonlinear systems

iterative adaptive dynamic programming for solving unknown nonlinear zero sum game based on online data h infty control is a powerful method to solve the disturbance attenuation problems that occur in some control systems the design of such controllers relies on solving the zero sum game zsg but in practical applications the exact dynamics is mostly unknown identification of dynamics also produces errors that are detrimental to the control performance to overcome this problem an iterative adaptive dynamic programming algorithm is proposed in this paper to solve the continuous time unknown nonlinear zsg with only online data a model free approach to the hamilton jacobi isaacs equation is developed based on the policy iteration method control and disturbance policies and value are approximated by neural networks nns under the critic actor disturber structure the nn weights are solved by the least squares method according to the theoretical analysis our algorithm is equivalent to a gauss newton method solving an optimization problem and it converges uniformly to the optimal solution the online data can also be used repeatedly which is highly efficient simulation results demonstrate its feasibility to solve the unknown nonlinear zsg when compared with other algorithms it saves a significant amount of online measurement time

observer based distributed consensus for general nonlinear multi agent systems with interval control inputs in this paper observer based distributed consensus for general nonlinear multi agent systems with interval control inputs under strongly connected balanced topology is encountered when the relative states of agents are unavailable or undesirable theoretical analysis method is further extended to the case of general nonlinear multi agent systems under switching setting moreover tracking problem on the leader follower scenario is also explicitly investigated under a mutual assumption that the communication graph which represents the interaction among agents contains a directed spanning tree with the leader as its root it is shown that the consensus for underlying considered multi agent systems can be desirable as long as the data missing rate does not exceed a certain threshold finally simulation examples are presented to effectively corroborate the analytical findings

galerkin approximations of the generalized hamilton jacobi bellman equation in this paper we study the convergence of the galerkin approximation method applied to the generalized hamilton jacobi bellman ghjb equation over a compact set containing the origin the ghjb equation gives the cost of an arbitrary control law and can be used to improve the performance of this control the ghjb equation can also be used to successively approximate the hamilton jacobi bellman equation we state sufficient conditions that guarantee that the galerkin approximation converges to the solution of the ghjb equation and that the resulting approximate control is stabilizing on the same region as the initial control the method is demonstrated on a simple nonlinear system and is compared to a result obtained by using exact feedback linearization in conjunction with the lqr design method

convergence of the combination technique for second order elliptic differential equations the combination technique is an algorithm for the approximate solution of partial differential equations on sparse grids that has to be combined with a suitable standard discretization the advantage of the combination technique compared to the standard discretization is that the same accuracy is achieved with many fewer grid points in this paper the combination technique is used with a bilinear finite element discretization depending on the smoothness of the solution and the coefficients it is proved for general second order elliptic differential equations on the unit square that the combined solution converges with order o h or o h log h 1 in the energy norm and with order o h2 log h 1 or o h3 2 in the l2 norm respectively this holds even if the bilinear form corresponding to the elliptic equation is not symmetric positive definite the proof does not use an asymptotic error expansion but sobolev space techniques

finite horizon h_ infty tracking control for unknown nonlinear systems with saturating actuators in this paper a neural network nn based online model free integral reinforcement learning algorithm is developed to solve the finite horizon h infty optimal tracking control problem for completely unknown nonlinear continuous time systems with disturbance and saturating actuators constrained control input an augmented system is constructed with the tracking error system and the command generator system a time varying hamilton jacobi isaacs hji equation is formulated for the augmented problem which is extremely difficult or impossible to solve due to its time dependent property and nonlinearity then an actor critic disturbance nn structure based scheme is proposed to learn the time varying solution to the hji equation in real time without using the knowledge of system dynamics since the solution to the hji equation is time dependent the form of nns representation with constant weights and time dependent activation functions is considered furthermore an extra error is incorporated in order to satisfy the terminal constraints in the weight update law convergence and stability proofs are given based on the lyapunov theory for nonautonomous systems two simulation examples are provided to demonstrate the effectiveness of the designed algorithm

neurodynamic programming and zero sum games for constrained control systems in this paper neural networks are used along with two player policy iterations to solve for the feedback strategies of a continuous time zero sum game that appears in l2 gain optimal control suboptimal hinfin control of nonlinear systems affine in input with the control policy having saturation constraints the result is a closed form representation on a prescribed compact set chosen a priori of the feedback strategies and the value function that solves the associated hamilton jacobi isaacs hji equation the closed loop stability l2 gain disturbance attenuation of the neural network saturated control feedback strategy and uniform convergence results are proven finally this approach is applied to the rotational translational actuator rtac nonlinear benchmark problem under actuator saturation offering guaranteed stability and disturbance attenuation

extrapolation combination and sparse grid techniques for elliptic boundary value problems several variants of extrapolation can be used for the solution of partial differential equations there are richardson extrapolation truncation error extrapolation and extrapolation of the functional in multi dimensional problems multivariate error expansions can be exploited by multivariate extrapolation where the asymptotic expansions in different mesh parameters are exploited particularly interesting cases are the combination technique that uses all the grids that have a constant product of the meshspacings in the different coordinate directions another related technique is the sparse grid finite element technique that can be interpreted as a combination extrapolation of the functional

state space solutions to standard h sub 2 and h sub infinity control problems simple state space formulas are derived for all controllers solving the following standard h sub infinity problem for a given number gamma 0 find all controllers such that the h sub infinity norm of the closed loop transfer function is strictly less than gamma it is known that a controller exists if and only if the unique stabilizing solutions to two algebraic riccati equations are positive definite and the spectral radius of their product is less than gamma sup 2 under these conditions a parameterization of all controllers solving the problem is given as a linear fractional transformation lft on a contractive stable free parameter the state dimension of the coefficient matrix for the lft constructed using the two riccati solutions equals that of the plant and has a separation structure reminiscent of classical lqg i e h sub 2 theory this paper is intended to be of tutorial value so a standard h sub 2 solution is developed in parallel

linear quadratic tracking control of partially unknown continuous time systems using reinforcement learning in this technical note an online learning algorithm is developed to solve the linear quadratic tracking lqt problem for partially unknown continuous time systems it is shown that the value function is quadratic in terms of the state of the system and the command generator based on this quadratic form an lqt bellman equation and an lqt algebraic riccati equation are are derived to solve the lqt problem the integral reinforcement learning technique is used to find the solution to the lqt are online and without requiring the knowledge of the system drift dynamics or the command generator dynamics the convergence of the proposed online algorithm to the optimal control solution is verified to show the efficiency of the proposed approach a simulation example is provided

approximate optimal trajectory tracking for continuous time nonlinear systems adaptive dynamic programming has been investigated and used as a method to approximately solve optimal regulation problems however the extension of this technique to optimal tracking problems for continuous time nonlinear systems has remained a non trivial open problem the control development in this paper guarantees ultimately bounded tracking of a desired trajectory while also ensuring that the enacted controller approximates the optimal controller

finite horizon approximate optimal guaranteed cost control of uncertain nonlinear systems with application to mars entry guidance this paper studies the finite horizon optimal guaranteed cost control gcc problem for a class of time varying uncertain nonlinear systems the aim of this problem is to find a robust state feedback controller such that the closed loop system has not only a bounded response in a finite duration of time for all admissible uncertainties but also a minimal guaranteed cost a neural network nn based approximate optimal gcc design is developed initially by modifying the cost function to account for the nonlinear perturbation of system the optimal gcc problem is transformed into a finite horizon optimal control problem of the nominal system subsequently with the help of the modified cost function together with a parametrized bounding function for all admissible uncertainties the solution to the optimal gcc problem is given in terms of a parametrized hamilton jacobi bellman phjb equation then a nn method is developed to solve offline the phjb equation approximately and thus obtain the nearly optimal gcc policy furthermore the convergence of approximate phjb equation and the robust admissibility of nearly optimal gcc policy are also analyzed finally by applying the proposed design method to the entry guidance problem of the mars lander the achieved simulation results show the effectiveness of the proposed controller

fast algorithms for periodic spline wavelets on sparse grids we consider boolean sums of univariate interpolation operators which define multivariate jth order blending interpolation operators on sparse grids sample spaces are defined as range of the blending operators sample and wavelet spaces have significantly lower dimension and good approximation order for certain function spaces fast decomposition and reconstruction algorithms for bivariate spline wavelets based on algorithms for univariate functions are described operation counts for the algorithms are given and it is shown that the complexity depends linearly on the dimension of sample spaces

linear transmit processing in mimo communications systems we examine and compare the different types of linear transmit processing for multiple input multiple output systems where we assume that the receive filter is independent of the transmit filter contrary to the joint optimization of transmit and receive filters we can identify three filter types similar to receive processing the transmit matched filter the transmit zero forcing filter and the transmit wiener filter we show that the transmit filters are based on similar optimizations as the respective receive filters with an additional constraint for the transmit power moreover the transmit wiener filter has similar convergence properties as the receive wiener filter i e it converges to the matched filter and the zero forcing filter for low and high signal to noise ratio respectively we give closed form solutions for all transmit filters and present the fundamental result that their mean square errors are equal to the errors of the respective receive filters if the information symbols and the additive noise are uncorrelated however our simulations reveal that the bit error ratio results of the transmit filters differ from the results for the respective receive filters

neural network based robust optimal control design for a class of uncertain nonlinear systems via adaptive dynamic programming in this paper the neural network based robust optimal control design for a class of uncertain nonlinear systems via adaptive dynamic programming approach is investigated first the robust controller of the original uncertain system is derived by adding a feedback gain to the optimal controller of the nominal system it is also shown that this robust controller can achieve optimality under a specified cost function which serves as the basic idea of the robust optimal control design then a critic network is constructed to solve the hamilton jacobi bellman equation corresponding to the nominal system where an additional stabilizing term is introduced to verify the stability the uniform ultimate boundedness of the closed loop system is also proved by using the lyapunov approach moreover the obtained results are extended to solve decentralized optimal control problem of continuous time nonlinear interconnected large scale systems finally two simulation examples are presented to illustrate the effectiveness of the established control scheme

distributed cooperative h optimal tracking control of mimo nonlinear multi agent systems in strict feedback form via adaptive dynamic programming the design of distributed cooperative h optimal controllers for multi agent systems is a major challenge when the agents models are uncertain multi input and multi output nonlinear systems in str