least squares temporal difference methods an analysis under general conditions we consider approximate policy evaluation for finite state and action markov de cision processes mdp with the least squares temporal difference lstd algorithm lstd in an exploration enhanced learning context where policy costs are computed from observations of a markov chain different from the one corresponding to the policy under evaluation we establish for the discounted cost criterion that lstd converges almost surely under mild minimal conditions we also analyze other properties of the iterates involved in the algorithm including convergence in mean and boundedness our analysis draws on theories of both finite space markov chains and weak feller markov chains on a topological space our results can be applied to other temporal difference algorithms and mdp models as examples we give a convergence analysis of a td algorithm and extensions to mdp with compact state and action spaces as well as a convergence proof of a new lstd algorithm with state dependent parameters

the monotonicity of the period function for planar hamiltonian vector fields if the hamiltonian system with hamiltonian h x y 12y2 v x has a center at the origin it is shown that the period function for the family of periodic trajectories surrounding the center is monotone when the function v v 2 is convex this theorem is applied to determine existence and uniqueness of solutions for the neumann boundary value problem x a x 2 bx c x 0 x 1 0

on actor critic algorithms in this article we propose and analyze a class of actor critic algorithms these are two time scale algorithms in which the critic uses temporal difference learning with a linearly parameterized approximation architecture and the actor is updated in an approximate gradient direction based on information provided by the critic we show that the features for the critic should ideally span a subspace prescribed by the choice of parameterization of the actor we study actor critic algorithms for markov decision processes with polish state and action spaces we state and prove two results regarding their convergence

rigidity of time changes for horocycle flows soit t t un flot preservant la mesure sur un espace de probabilite x on etudie la rigidite des changements de t pour des lots horocycles

period function for a class of hamiltonian systems abstract this paper studies the period function of the class of hamiltonian systems x h y y h x where h x y has the special form h x y f x g y and the origin is a non degenerate center more concretely if t h denotes the period of the periodic orbit contained in h x y h we solve the inverse problem of characterizing all systems with a given function t h we also characterize the limiting behaviour of t at infinity when the origin is a global center and apply this result to prove among other results that there are no nonlinear polynomial isochronous centers in this family

proto value functions a laplacian framework for learning representation and control in markov decision processes this paper introduces a novel spectral framework for solving markov decision processes mdps by jointly learning representations and optimal policies the major components of the framework described in this paper include i a general scheme for constructing representations or basis functions by diagonalizing symmetric diffusion operators ii a specific instantiation of this approach where global basis functions called proto value functions pvfs are formed using the eigenvectors of the graph laplacian on an undirected graph formed from state transitions induced by the mdp iii a three phased procedure called representation policy iteration comprising of a sample collection phase a representation learning phase that constructs basis functions from samples and a final parameter estimation phase that determines an approximately optimal policy within the linear subspace spanned by the current basis functions iv a specific instantiation of the rpi framework using least squares policy iteration lspi as the parameter estimation method v several strategies for scaling the proposed approach to large discrete and continuous state spaces including the nystrom extension for out of sample interpolation of eigenfunctions and the use of kronecker sum factorization to construct compact eigenfunctions in product spaces such as factored mdps vi finally a series of illustrative discrete and continuous control tasks which both illustrate the concepts and provide a benchmark for evaluating the proposed approach many challenges remain to be addressed in scaling the proposed framework to large mdps and several elaboration of the proposed framework are briefly summarized at the end

effective equidistribution for closed orbits of semisimple groups on homogeneous spaces we prove effective equidistribution with polynomial rate for large closed orbits of semisimple groups on homogeneous spaces under certain technical restrictions notably the acting group should have finite centralizer in the ambient group the proofs make extensive use of spectral gaps and also of a closing lemma for such actions

policy approximation in policy iteration approximate dynamic programming for discrete time nonlinear systems policy iteration approximate dynamic programming dp is an important algorithm for solving optimal decision and control problems in this paper we focus on the problem associated with policy approximation in policy iteration approximate dp for discrete time nonlinear systems using infinite horizon undiscounted value functions taking policy approximation error into account we demonstrate asymptotic stability of the control policy under our problem setting show boundedness of the value function during each policy iteration step and introduce a new sufficient condition for the value function to converge to a bounded neighborhood of the optimal value function aiming for practical implementation of an approximate policy we consider using volterra series which has been extensively covered in controls literature for its good theoretical properties and for its success in practical applications we illustrate the effectiveness of the main ideas developed in this paper using several examples including a practical problem of excitation control of a hydrogenerator

projected equation methods for approximate solution of large linear systems we consider linear systems of equations and solution approximations derived by projection on a low dimensional subspace we propose stochastic iterative algorithms based on simulation which converge to the approximate solution and are suitable for very large dimensional problems the algorithms are extensions of recent approximate dynamic programming methods known as temporal difference methods which solve a projected form of bellman s equation by using simulation based approximations to this equation or by using a projected value iteration method

algorithmic survey of parametric value function approximation reinforcement learning rl is a machine learning answer to the optimal control problem it consists of learning an optimal control policy through interactions with the system to be controlled the quality of this policy being quantified by the so called value function a recurrent subtopic of rl concerns computing an approximation of this value function when the system is too large for an exact representation this survey reviews state of the art methods for parametric value function approximation by grouping them into three main categories bootstrapping residual and projected fixed point approaches related algorithms are derived by considering one of the associated cost functions and a specific minimization method generally a stochastic gradient descent or a recursive least squares approach

a convex analytic approach to markov decision processes this paper develops a new framework for the study of markov decision processes in which the control problem is viewed as an optimization problem on the set of canonically induced measures on the trajectory space of the joint state and control process this set is shown to be compact convex one then associates with each of the usual cost criteria infinite horizon discounted cost finite horizon control up to an exit time a naturally defined occupation measure such that the cost is an integral of some function with respect to this measure these measures are shown to form a compact convex set whose extreme points are characterized classical results about existence of optimal strategies are recovered from this and several applications to multicriteria and constrained optimization problems are briefly indicated

chaos in periodically perturbed planar hamiltonian systems using linked twist maps abstract we present a simple topological approach for the search of fixed points and the detection of chaotic dynamics for two dimensional maps satisfying a twist condition on linked annuli applications to planar hamiltonian systems are given

the period function of the generalized lotka volterra centers abstract the present paper deals with the period function of the quadratic centers in the literature different terminologies are used to classify these centers but essentially there are four families hamiltonian reversible q 3 r codimension four q 4 and generalized lotka volterra systems q 3 lv chicone c chicone review in mathscinet ref 94h 58072 conjectured that the reversible centers have at most two critical periods and that the centers of the three other families have a monotonic period function with regard to the second part of this conjecture only the monotonicity of the hamiltonian and q 4 families w a coppel l gavrilov the period function of a hamiltonian quadratic system differential integral equations 6 1993 1357 1365 y zhao the monotonicity of period function for codimension four quadratic system q 4 j differential equations 185 2002 370 387 has been proved concerning the q 3 lv family no substantial progress has been made since the middle 80s when several authors showed independently the monotonicity of the classical lotka volterra centers f rothe the periods of the volterra lokta system j reine angew math 355 1985 129 138 r schaaf global behaviour of solution branches for some neumann problems depending on one or several parameters j reine angew math 346 1984 1 31 j waldvogel the period in the lotka volterra system is monotonic j math anal appl 114 1986 178 184 by means of the first period constant one can easily conclude that the period function of the centers in the q 3 lv family is monotone increasing near the inner boundary of its period annulus i e the center itself thus according to chicone s conjecture it should be also monotone increasing near the outer boundary which in the poincare disc is a polycycle in this paper we show that this is true in addition we prove that except for a zero measure subset of the parameter plane there is no bifurcation of critical periods from the outer boundary finally we show that the period function is globally i e in the whole period annulus monotone increasing in two other cases different from the classical one

kernel based least squares policy iteration for reinforcement learning in this paper we present a kernel based least squares policy iteration klspi algorithm for reinforcement learning rl in large or continuous state spaces which can be used to realize adaptive feedback control of uncertain dynamic systems by using klspi near optimal control policies can be obtained without much a priori knowledge on dynamic models of control plants in klspi mercer kernels are used in the policy evaluation of a policy iteration process where a new kernel based least squares temporal difference algorithm called klstd q is proposed for efficient policy evaluation to keep the sparsity and improve the generalization ability of klstd q solutions a kernel sparsification procedure based on approximate linear dependency ald is performed compared to the previous works on approximate rl methods klspi makes two progresses to eliminate the main difficulties of existing results one is the better convergence and near optimality guarantee by using the klstd q algorithm for policy evaluation with high precision the other is the automatic feature selection using the ald based kernel sparsification therefore the klspi algorithm provides a general rl method with generalization performance and convergence guarantee for large scale markov decision problems mdps experimental results on a typical rl task for a stochastic chain problem demonstrate that klspi can consistently achieve better learning efficiency and policy quality than the previous least squares policy iteration lspi algorithm furthermore the klspi method was also evaluated on two nonlinear feedback control problems including a ship heading control problem and the swing up control of a double link underactuated pendulum called acrobot simulation results illustrate that the proposed method can optimize controller performance using little a priori information of uncertain dynamic systems it is also demonstrated that klspi can be applied to online learning control by incorporating an initial controller to ensure online performance

an example of chaotic dynamics for the motion of a charged particle in a magnetic field abstract in this paper we discuss the presence of chaotic dynamics of charged particles inside an idealized magnetic field suggested by a tokamak type configuration our model is based on a periodically perturbed hamiltonian system in a half plane r 0 we propose a simple mechanism producing complex dynamics based on the theory of linked twist maps jointly with the method of stretching along the paths a key step in our argument relies on the monotonicity of the period map associated with the unperturbed planar system

an analysis of temporal difference learning with function approximation we discuss the temporal difference learning algorithm as applied to approximating the cost to go function of an infinite horizon discounted markov chain the algorithm we analyze updates parameters of a linear function approximator online during a single endless trajectory of an irreducible aperiodic markov chain with a finite or infinite state space we present a proof of convergence with probability one a characterization of the limit of convergence and a bound on the resulting approximation error furthermore our analysis is based on a new line of reasoning that provides new intuition about the dynamics of temporal difference learning in addition to proving new and stronger positive results than those previously available we identify the significance of online updating and potential hazards associated with the use of nonlinear function approximators first we prove that divergence may occur when updates are not based on trajectories of the markov chain this fact reconciles positive and negative results that have been discussed in the literature regarding the soundness of temporal difference learning second we present an example illustrating the possibility of divergence when temporal difference learning is used in the presence of a nonlinear function approximator

new developments in state estimation for nonlinear systems based on an interpolation formula accurate state estimators for nonlinear systems can be derived the estimators do not require derivative information which makes them simple to implement

first derivative of the period function with applications abstract given a centre of a planar differential system we extend the use of the lie bracket to the determination of the monotonicity character of the period function as far as we know there are no general methods to study this function and the use of commutators and lie bracket was restricted to prove isochronicity we give several examples and a special method which simplifies the computations when a first integral is known

performance bounds in l_p norm for approximate value iteration approximate value iteration avi is a method for solving large markov decision problems by approximating the optimal value function with a sequence of value function representations v n processed according to the iterations v n 1 mathcal at v n where mathcal t is the so called bellman operator and mathcal a an approximation operator which may be implemented by a supervised learning sl algorithm usual bounds on the asymptotic performance of avi are established in terms of the l infty norm approximation errors induced by the sl algorithm however most widely used sl algorithms such as least squares regression return a function the best fit that minimizes an empirical approximation error in l p norm p geq 1 in this paper we extend the performance bounds of avi to weighted l p norms which enables us to directly relate the performance of avi to the approximation power of the sl algorithm hence assuring the tightness and practical relevance of these bounds the main result is a performance bound of the resulting policies expressed in terms of the l p norm errors introduced by the successive approximations the new bound takes into account a concentration coefficient that estimates how much the discounted future state distributions starting from a probability measure used to assess the performance of avi can possibly differ from the distribution used in the regression operation we illustrate the tightness of the bounds on an optimal replacement problem

the period in the lotka volterra system is monotonic abstract periodic solutions in a class of hamiltonian systems with one degree of freedom containing the lotka volterra system are considered their period is represented as an integral appropriate for numerical evaluation as well as for theoretical investigations the monotonicity of the period in the lotka volterra system follows as a simple consequence

least squares policy iteration we propose a new approach to reinforcement learning for control problems which combines value function approximation with linear architectures and approximate policy iteration this new approach is motivated by the least squares temporal difference learning algorithm lstd for prediction problems which is known for its efficient use of sample experiences compared to pure temporal difference algorithms heretofore lstd has not had a straightforward application to control problems mainly because lstd learns the state value function of a fixed policy which cannot be used for action selection and control without a model of the underlying process our new algorithm least squares policy iteration lspi learns the state action value function which allows for action selection without a model and for incremental policy improvement within a policy iteration framework lspi is a model free off policy method which can use efficiently and reuse in each iteration sample experiences collected in any manner by separating the sample collection method the choice of the linear approximation architecture and the solution method lspi allows for focused attention on the distinct elements that contribute to practical reinforcement learning lspi is tested on the simple task of balancing an inverted pendulum and the harder task of balancing and riding a bicycle to a target location in both cases lspi learns to control the pendulum or the bicycle by merely observing a relatively small number of trials where actions are selected randomly lspi is also compared against q learning both with and without experience replay using the same value function architecture while lspi achieves good performance fairly consistently on the difficult bicycle task q learning variants were rarely able to balance for more than a small fraction of the time needed to reach the target location

an invariance principle for nonlinear switched systems abstract in this paper we address the problem of extending lasalle invariance principle to switched system we prove an extension of the invariance principle relative to dwell time switched solutions and a second one relative to constrained switched systems

the linear programming approach to approximate dynamic programming the curse of dimensionality gives rise to prohibitive computational requirements that render infeasible the exact solution of large scale stochastic control problems we study an efficient method based on linear programming for approximating solutions to such problems the approach fits a linear combination of pre selected basis functions to the dynamic programming cost to go function we develop error bounds that offer performance guarantees and also guide the selection of both basis functions and state relevance weights that influence quality of the approximation experimental results in the domain of queueing network control provide empirical support for the methodology

point vortex dynamics a classical mathematics playground the idealization of a two dimensional ideal flow as a collection of point vortices embedded in otherwise irrotational flow yields a surprisingly large number of mathematical insights and connects to a large number of areas of classical mathematics several examples are given including the integrability of the three vortex problem the interplay of relative equilibria of identical vortices and the roots of certain polynomials addition formulas for the cotangent and the weierstras function projective geometry and other topics the hope and intent of the article is to garner further participation in the exploration of this intriguing dynamical system from the mathematical physics community

global behaviour of the period function of the sum of two quasi homogeneous vector fields abstract we study the global behaviour of the period function on the period annulus of degenerate centres for two families of planar polynomial vector fields these families are the quasi homogeneous vector fields and the vector fields given by the sum of two quasi homogeneous hamiltonian ones in the first case we prove that the period function is globally decreasing extending previous results that deal either with the hamiltonian quasi homogeneous case or with the general homogeneous situation in the second family and after adding some more additional hypotheses we show that the period function of the origin is either decreasing or has at most one critical period and that both possibilities may happen this result also extends some previous results that deal with the situation where both vector fields are homogeneous and the origin is a non degenerate centre

sparse equidistribution problems period bounds and subconvexity we introduce a geometric method to bound periods of automorphic forms the key features of this method are the use of equidistribution results in place of mean value theorems and the systematic use of mixing and the spectral gap applications are given to equidistribution of sparse subsets of horocycles and to equidistribution of cm points to subconvexity of the triple product period in the level aspect over number fields which implies subconvexity for certain standard and rankin selberg l functions and to bounding fourier coefficients of automorphic forms

policy iteration adaptive dynamic programming algorithm for discrete time nonlinear systems this paper is concerned with a new discrete time policy iteration adaptive dynamic programming adp method for solving the infinite horizon optimal control problem of nonlinear systems the idea is to use an iterative adp technique to obtain the iterative control law which optimizes the iterative performance index function the main contribution of this paper is to analyze the convergence and stability properties of policy iteration method for discrete time nonlinear systems for the first time it shows that the iterative performance index function is nonincreasingly convergent to the optimal solution of the hamilton jacobi bellman equation it is also proven that any of the iterative control laws can stabilize the nonlinear systems neural networks are used to approximate the performance index function and compute the optimal control law respectively for facilitating the implementation of the iterative adp algorithm where the convergence of the weight matrices is analyzed finally the numerical results and analysis are presented to illustrate the performance of the developed method

adaptive aggregation methods for infinite horizon dynamic programming a class of iterative aggregation algorithms for solving infinite horizon dynamic programming problems is proposed the idea is to interject aggregation iterations in the course of the usual successive approximation method an important feature that sets this method apart from earlier ones is that the aggregate groups of states change adaptively from one aggregation iteration to the next depending on the progress of the computation this allows acceleration of convergence in difficult problems involving multiple ergodic classes for which methods using fixed groups of aggregate states are ineffective no knowledge of special problem structure is utilized by the algorithms