metrics for ergodicity and design of ergodic dynamics for multi agent systems abstract in this paper we propose a metric that quantifies how far trajectories are from being ergodic with respect to a given probability measure this metric is based on comparing the fraction of time spent by the trajectories in spherical sets to the measure of the spherical sets this metric is shown to be equivalent to a metric obtained as a distance between a certain delta like distribution on the trajectories and the desired probability distribution using this metric we formulate centralized feedback control laws for multi agent systems so that agents trajectories sample a given probability distribution as uniformly as possible the feedback controls we derive are essentially model predictive controls in the limit as the receding horizon goes to zero and the agents move with constant speed or constant forcing in the case of second order dynamics we numerically analyze the closed loop dynamics of the multi agents systems in various scenarios the algorithm presented in this paper for the design of ergodic dynamics will be referred to as spectral multiscale coverage smc

least squares temporal difference methods an analysis under general conditions we consider approximate policy evaluation for finite state and action markov de cision processes mdp with the least squares temporal difference lstd algorithm lstd in an exploration enhanced learning context where policy costs are computed from observations of a markov chain different from the one corresponding to the policy under evaluation we establish for the discounted cost criterion that lstd converges almost surely under mild minimal conditions we also analyze other properties of the iterates involved in the algorithm including convergence in mean and boundedness our analysis draws on theories of both finite space markov chains and weak feller markov chains on a topological space our results can be applied to other temporal difference algorithms and mdp models as examples we give a convergence analysis of a td algorithm and extensions to mdp with compact state and action spaces as well as a convergence proof of a new lstd algorithm with state dependent parameters

comparison of systems with complex behavior abstract we present a formalism for comparing the asymptotic dynamics of dynamical systems with physical systems that they model based on the spectral properties of the koopman operator we first compare invariant measures and discuss this in terms of a statistical takens theorem proved here we also identify the need to go beyond comparing only invariant ergodic measures of systems and introduce an ergodic theoretic treatment of a class of spectral functionals that allow for this the formalism is extended for a class of stochastic systems discrete random dynamical systems the ideas introduced in this paper can be used for parameter identification and model validation of driven nonlinear models with complicated behavior as an illustration we provide an example in which we compare the asymptotic behavior of a combustion system measured experimentally with the asymptotic behavior of a class of models that have the form of a random dynamical system

look ahead control of conveyor serviced production station by using potential based online policy iteration we consider the look ahead control of a conveyor serviced production station csps in the context of semi markov decision process smdp model and our goal is to find an optimal control policy under either average or discounted cost criteria policy iteration pi combined with the concept of performance potential can be applied to provide a unified optimisation framework for both criteria however a major difficulty arises in the exact solution scheme that is it requires not only the full knowledge of model parameters but also a considerable amount of work to obtain and process the necessary system and performance matrices to overcome this difficulty we propose a potential based online pi algorithm in this article during implementation by analysing and utilising the historic information of all the past operation of a practical csps system the potentials and state action values are learned on line through an effective exploration scheme we finally illustrate the successful application of th

the boundedness conditions for model free hdp lambda this paper provides the stability analysis for a model free action dependent heuristic dynamic programing hdp approach with an eligibility trace long term prediction parameter lambda hdp lambda learns from more than one future reward eligibility traces have long been popular in q learning this paper proves and demonstrates that they are worthwhile to use with hdp in this paper we prove its uniformly ultimately bounded uub property under certain conditions previous works present a uub proof for traditional hdp hdp lambda 0 but we extend the proof with the lambda parameter by using lyapunov stability we demonstrate the boundedness of the estimated error for the critic and actor neural networks as well as learning rate parameters three case studies demonstrate the effectiveness of hdp lambda the trajectories of the internal reinforcement signal nonlinear system are considered as the first case we compare the results with the performance of hdp and traditional temporal difference td lambda with different lambda values the second case study is a single link inverted pendulum we investigate the performance of the inverted pendulum by comparing hdp lambda with regular hdp with different levels of noise the third case study is a 3 d maze navigation benchmark which is compared with state action reward state action q lambda hdp and hdp lambda all these simulation results illustrate that hdp lambda has a competitive performance thus this contribution is not only uub but also useful in comparison with traditional hdp

almost invariant sets and invariant manifolds connecting probabilistic and geometric descriptions of coherent structures in flows abstract we study the transport and mixing properties of flows in a variety of settings connecting the classical geometrical approach via invariant manifolds with a probabilistic approach via transfer operators for non divergent fluid like flows we demonstrate that eigenvectors of numerical transfer operators efficiently decompose the domain into invariant regions for dissipative chaotic flows such a decomposition into invariant regions does not exist instead the transfer operator approach detects almost invariant sets we demonstrate numerically that the boundaries of these almost invariant regions are predominantly comprised of segments of co dimension 1 invariant manifolds for a mixing periodically driven fluid like flow we show that while sets bounded by stable and unstable manifolds are almost invariant the transfer operator approach can identify almost invariant sets with smaller mass leakage thus the transport mechanism of lobe dynamics need not correspond to minimal transport the transfer operator approach is purely probabilistic it directly determines those regions that minimally mix with their surroundings the almost invariant regions are identified via eigenvectors of a transfer operator and are ranked by the corresponding eigenvalues in the order of the sets invariance or leakiness while we demonstrate that the almost invariant sets are often bounded by segments of invariant manifolds without such a ranking it is not at all clear which intersections of invariant manifolds form the major barriers to mixing furthermore in some cases invariant manifolds do not bound sets of minimal leakage our transfer operator constructions are very simple and fast to implement they require a sample of short trajectories followed by eigenvector calculations of a sparse matrix

characterizations iterative method sign pattern and perturbation analysis for the dmp inverse with its applications abstract this paper is a study on main properties and characterizations of the dmp inverse also corresponding representations and computational procedures are derived particularly an upgrade of the bordering method to the case of the dmp inverse is considered as well as applications of the dmp inverse in solving singular linear systems also the dmp inverse of an upper block triangular matrix and its sign pattern are investigated furthermore several computational procedures aimed to approximating the dmp inverse are developed implemented and tested the main of them are the limit representation a revised successive matrix squaring iterations and the gradient neural network gnn dynamical system finally some perturbation bounds and continuity of the dmp inverse are studied

a variational theory of hyperbolic lagrangian coherent structures abstract we develop a mathematical theory that clarifies the relationship between observable lagrangian coherent structures lcss and invariants of the cauchy green strain tensor field motivated by physical observations of trajectory patterns we define hyperbolic lcss as material surfaces i e codimension one invariant manifolds in the extended phase space that extremize an appropriate finite time normal repulsion or attraction measure over all nearby material surfaces we also define weak lcss wlcss as stationary solutions of the above variational problem solving these variational problems we obtain computable sufficient and necessary criteria for wlcss and lcss that link them rigorously to the cauchy green strain tensor field we also prove a condition for the robustness of an lcs under perturbations such as numerical errors or data imperfection on several examples we show how these results resolve earlier inconsistencies in the theory of lcs finally we introduce the notion of a constrained lcs clcs that extremizes normal repulsion or attraction under constraints this construct allows for the extraction of a unique observed lcs from linear systems and for the identification of the most influential weak unstable manifold of an unstable node

finite sample analysis of least squares policy iteration in this paper we report a performance bound for the widely used least squares policy iteration lspi algorithm we first consider the problem of policy evaluation in reinforcement learning that is learning the value function of a fixed policy using the least squares temporal difference lstd learning method and report finite sample analysis for this algorithm to do so we first derive a bound on the performance of the lstd solution evaluated at the states generated by the markov chain and used by the algorithm to learn an estimate of the value function this result is general in the sense that no assumption is made on the existence of a stationary distribution for the markov chain we then derive generalization bounds in the case when the markov chain possesses a stationary distribution and is b mixing finally we analyze how the error at each policy evaluation step is propagated through the iterations of a policy iteration method and derive a performance bound for the lspi algorithm

lagrangian coherent structures and mixing in two dimensional turbulence abstract we introduce a lagrangian definition for the boundaries of coherent structures in two dimensional turbulence the boundaries are defined as material lines that are linearly stable or unstable for longer times than any of their neighbors such material lines are responsible for stretching and folding in the mixing of passive tracers we derive an analytic criterion that can be used to extract coherent structures with high precision from numerical or experimental data sets the criterion provides a rigorous link between the lagrangian concept of hyperbolicity the okubo weiss criterion and vortex boundaries we apply the results to simulations of two dimensional barotropic turbulence

distinguished material surfaces and coherent structures in three dimensional fluid flows we prove analytic criteria for the existence of finite time attracting and repelling material surfaces and lines in three dimensional unsteady flows the longest lived such structures define coherent structures in a lagrangian sense our existence criteria involve the invariants of the velocity gradient tensor along fluid trajectories an alternative approach to coherent structures is shown to lead to their characterization as local maximizers of the largest finite time lyapunov exponent field computed directly from particle paths both approaches provide effective tools for extracting distinguished lagrangian structures from three dimensional velocity data we illustrate the results on steady and unsteady abc type flows

stabilization of stochastic iterative methods for singular and nearly singular linear systems we consider linear systems of equations ax b with an emphasis on the case where a is singular under certain conditions necessary as well as sufficient linear deterministic iterative methods generate sequences xk that converge to a solution as long as there exists at least one solution this convergence property can be impaired when these methods are implemented with stochastic simulation as is often done in important classes of large scale problems we introduce additional conditions and novel algorithmic stabilization schemes under which xk converges to a solution when a is singular and may also be used with substantial benefit when a is nearly singular

index splitting for the drazin inverse and the singular linear system abstract an index splitting of a singular matrix a for the drazin inverse and its relative iterations for the minimal p norm solution of the singular linear system ax b b r a k k ind a are presented some convergence criteria are also given and a method is developed for constructing the index splitting of matrix a

on some properties of the determinants of tensors abstract we use the hilbert s nullstellensatz hilbert s zero point theorem to give a direct proof of the formula for the determinants of the products of tensors by using this determinant formula and using tensor product to represent the transformations of the slices of tensors we prove some basic properties of the determinants of tensors which are the generalizations of the corresponding properties of the determinants for matrices we also study the determinants of tensors after two types of transposes we use the permutational similarity of tensors to discuss the relation between weakly reducible tensors and the triangular block tensors and give a canonical form of the weakly reducible tensors

hyperbolic and elliptic transport barriers in three dimensional unsteady flows abstract we develop a general theory of transport barriers for three dimensional unsteady flows with arbitrary time dependence the barriers are obtained as two dimensional lagrangian coherent structures lcss that create locally maximal deformation along hyperbolic lcss this deformation is induced by locally maximal normal repulsion or attraction along shear lcss the deformation is created by locally maximal tangential shear hyperbolic lcss therefore play the role of generalized stable and unstable manifolds while closed shear lcss elliptic lcss act as generalized kam tori or kam type cylinders all these barriers can be computed from our theory as explicitly parametrized surfaces we illustrate our results by visualizing two dimensional hyperbolic and elliptic barriers in steady and unsteady versions of the abc flow

importance sampling for stochastic simulations importance sampling is one of the classical variance reduction techniques for increasing the efficiency of monte carlo algorithms for estimating integrals the basic idea is to replace the original random mechanism in the simulation by a new one and at the same time modify the function being integrated in this paper the idea is extended to problems arising in the simulation of stochastic systems discrete time markov chains continuous time markov chains and generalized semi markov processes are covered applications are given to a gi g 1 queueing problem and response surface estimation computation of the theoretical moments arising in importance sampling is discussed and some numerical examples given

proto value functions a laplacian framework for learning representation and control in markov decision processes this paper introduces a novel spectral framework for solving markov decision processes mdps by jointly learning representations and optimal policies the major components of the framework described in this paper include i a general scheme for constructing representations or basis functions by diagonalizing symmetric diffusion operators ii a specific instantiation of this approach where global basis functions called proto value functions pvfs are formed using the eigenvectors of the graph laplacian on an undirected graph formed from state transitions induced by the mdp iii a three phased procedure called representation policy iteration comprising of a sample collection phase a representation learning phase that constructs basis functions from samples and a final parameter estimation phase that determines an approximately optimal policy within the linear subspace spanned by the current basis functions iv a specific instantiation of the rpi framework using least squares policy iteration lspi as the parameter estimation method v several strategies for scaling the proposed approach to large discrete and continuous state spaces including the nystrom extension for out of sample interpolation of eigenfunctions and the use of kronecker sum factorization to construct compact eigenfunctions in product spaces such as factored mdps vi finally a series of illustrative discrete and continuous control tasks which both illustrate the concepts and provide a benchmark for evaluating the proposed approach many challenges remain to be addressed in scaling the proposed framework to large mdps and several elaboration of the proposed framework are briefly summarized at the end

existence of invariant tori in three dimensional maps with degeneracy abstract we prove a kam type result for the persistence of two dimensional invariant tori in perturbations of integrable action angle angle maps with degeneracy satisfying the intersection property such degenerate action angle angle maps arise upon generic perturbation of three dimensional volume preserving vector fields which are invariant under volume preserving action of s 1 when there is no motion in the group action direction for the unperturbed map this situation is analogous to degeneracy in hamiltonian systems the degenerate nature of the map and the unequal number of action and angle variables make the persistence proof non standard the persistence of the invariant tori as predicted by our result has implications for the existence of barriers to transport in three dimensional incompressible fluid flows simulation results indicating existence of two dimensional tori in a perturbation of swirling hill s spherical vortex flow are presented

the drazin inverse of an even order tensor and its application to singular tensor equations abstract the notion of the moore penrose inverses of matrices was recently extended from matrix space to even order tensor space with einstein product in the literature in this paper we further study the properties of even order tensors with einstein product we define the index and characterize the invertibility of an even order square tensor we also extend the notion of the drazin inverse of a square matrix to an even order square tensor an expression for the drazin inverse through the core nilpotent decomposition for a tensor of even order is obtained as an application the drazin inverse solution of the singular linear tensor equation a x b will also be included

spectra of uniform hypergraphs abstract we present a spectral theory of uniform hypergraphs that closely parallels spectral graph theory a number of recent developments building upon classical work has led to a rich understanding of symmetric hyperdeterminants of hypermatrices a k a multidimensional arrays symmetric hyperdeterminants share many properties with determinants but the context of multilinear algebra is substantially more complicated than the linear algebra required to address spectral graph theory i e ordinary matrices nonetheless it is possible to define eigenvalues of a hypermatrix via its characteristic polynomial as well as variationally we apply this notion to the adjacency hypermatrix of a uniform hypergraph and prove a number of natural analogs of basic results in spectral graph theory open problems abound and we present a number of directions for further study

an eulerian method for computing the coherent ergodic partition of continuous dynamical systems we develop an efficient eulerian numerical approach to extract invariant sets in a continuous dynamical system in the extended phase space the x t space we extend the idea of ergodic partition and propose a concept called coherent ergodic partition for visualizing ergodic components in a continuous flow numerically we first apply the level set method 33 and extend the backward phase flow method 25 to determine the long time flow map to compute all required long time averages of observables along particle trajectories we propose an eulerian approach by simply incorporating flow maps to iteratively interpolate those short time averages numerical experiments will demonstrate the effectiveness of the approach as an application of the method we apply the approach to the field of geometrical optics for high frequency wave propagation and propose to use the result from the coherent ergodic partition as a criteria for adaptivity in typical lagrangian ray tracing methods

nonautonomous control of stable and unstable manifolds in two dimensional flows abstract we outline a method for controlling the location of stable and unstable manifolds in the following sense from a known location of the stable and unstable manifolds in a steady two dimensional flow the primary segments of the manifolds are to be moved to a user specified time varying location which is near the steady location we determine the nonautonomous perturbation to the vector field required to achieve this control and give a theoretical bound for the error in the manifolds resulting from applying this control the efficacy of the control strategy is illustrated via a numerical example

cover tree bayesian reinforcement learning this paper proposes an online tree based bayesian approach for reinforcement learning for inference we employ a generalised context tree model this defines a distribution on multivariate gaussian piecewise linear models which can be updated in closed form the tree structure itself is constructed using the cover tree method which remains efficient in high dimensional spaces we combine the model with thompson sampling and approximate dynamic programming to obtain effective exploration policies in unknown environments the flexibility and computational simplicity of the model render it suitable for many reinforcement learning problems in continuous state spaces we demonstrate this in an experimental comparison with a gaussian process model a linear model and simple least squares policy iteration

a general product of tensors with applications abstract we study a general product of two n dimensional tensors a and b with orders m 2 and k 1 this product satisfies the associative law and is a generalization of the usual matrix product using this product many concepts and known results of tensors can be simply expressed and or proved and a number of applications of it will be given using the associative law of this tensor product and some properties on the resultant of a system of homogeneous equations on n variables we define the similarity and congruence of tensors which are also the generalizations of the corresponding relations for matrices and prove that similar tensors have the same characteristic polynomials thus the same spectra we study two special kinds of similarity permutational similarity and diagonal similarity and their applications in the study of the spectra of hypergraphs and nonnegative irreducible tensors we also define the direct product of tensors in matrix case it is also called the kronecker product and give its applications in the study of the spectra of two kinds of the products of hypergraphs we also give applications of this general product in the study of nonnegative tensors including a characterization of primitive tensors the upper bounds of primitive degrees and the cyclic indices of some nonnegative irreducible tensors

the generalized inverses of tensors and an application to linear models abstract in this paper we recall and extend some tensor operations then the generalized inverse of tensors is established by using tensor equations moreover we investigate the least squares solutions of tensor equations an algorithm to compute the moore penrose inverse of an arbitrary tensor is constructed finally we apply the obtained results to higher order gauss markov theorem

further results on moore penrose inverses of tensors with application to tensor nearness problems abstract the notion of the moore penrose inverse of a matrix has been extended to tensor case with einstein product recently in this paper we continue this work and propose the full rank decomposition of tensors by introducing a new kind of tensor rank which gives a novel representation of the moore penrose inverse moreover we also obtain several formulae related to the moore penrose inverse of a tensor and its i j k inverses in addition we consider the tensor nearness problem associated with a tensor equation a generalization of matrix nearness problems that arise in many areas of applied matrix computations it is shown that the unique solution to the tensor nearness problem can be represented by using the moore penrose inverses of the given tensors

an approach to fuzzy control of nonlinear systems stability and design issues presents a design methodology for stabilization of a class of nonlinear systems first the authors represent a nonlinear plant with a takagi sugeno fuzzy model then a model based fuzzy controller design utilizing the concept of the so called parallel distributed compensation is employed the main idea of the controller design is to derive each control rule so as to compensate each rule of a fuzzy system the design procedure is conceptually simple and natural moreover the stability analysis and control design problems can be reduced to linear matrix inequality lmi problems therefore they can be solved efficiently in practice by convex programming techniques for lmis the design methodology is illustrated by application to the problem of balancing and swing up of an inverted pendulum on a cart

absolute flux optimising curves of flows on a surface abstract given a flow on a surface we consider the problem of connecting two distinct trajectories by a curve of extremal absolute instantaneous flux we develop a complete classification of flux optimal curves accounting for the possibility of the flux having spatially and temporally varying weight this weight enables modelling the flux of non equilibrium distributions of tracer particles pollution concentrations or active scalar fields such as vorticity our results are applicable to all smooth autonomous flows area preserving or not

approximate dynamic programming via direct search in the space of value function approximations this paper deals with approximate value iteration avi algorithms applied to discounted dynamic programming dp problems for a fixed control policy the span semi norm of the so called bellman residual is shown to be convex in the banach space of candidate solutions to the dp problem this fact motivates the introduction of an avi algorithm with local search that seeks to minimize the span semi norm of the bellman residual in a convex value function approximation space the novelty here is that the optimality of a point in the approximation architecture is characterized by means of convex optimization concepts and necessary and sufficient conditions to local optimality are derived the procedure employs the classical avi algorithm direction bellman residual combined with a set of independent search directions to improve the convergence rate it has guaranteed convergence and satisfies at least the necessary optimality conditions over a prescribed set of directions to illustrate the method examples are presented that deal with a class of problems from the literature and a large state space queueing problem setting

learning the variance of the reward to go in markov decision processes mdps the variance of the reward to go is a natural measure of uncertainty about the long term performance of a policy and is important in domains such as finance resource allocation and process control currently however there is no tractable procedure for calculating it in large scale mdps this is in contrast to the case of the expected reward to go also known as the value function for which effective simulation based algorithms are known and have been used successfully in various domains in this paper we extend temporal difference td learning algorithms to estimating the variance of the reward to go for a fixed policy we propose variants of both td 0 and lstd with linear function approximation prove their convergence and demonstrate their utility in an option pricing problem our results show a dramatic improvement in terms of sample efficiency over standard monte carlo methods which are currently the state of the art

projected equation methods for approximate solution of large linear systems we consider linear systems of equations and solution approximations derived by projection on a low dimensional subspace we propose stochastic iterative algorithms based on simulation which converge to the approximate solution and are suitable for very large dimensional problems the algorithms are extensions of recent approximate dynamic programming methods known as temporal difference methods which solve a projected form of bellman s equation by using simulation based approximations to this equation or by using a projected value iteration method

an emphatic approach to the problem of off policy temporal difference learning in this paper we introduce the idea of improving the performance of parametric temporal difference td learning algorithms by selectively emphasizing or de emphasizing their updates on different time steps in particular we show that varying the emphasis of linear td s updates in a particular way causes its expected update to become stable under off policy training the only prior model free td methods to achieve this with per step computation linear in the number of function approximation parameters are the gradient td family of methods including tdc gtd and gq compared to these methods our emphatic td is simpler and easier to use it has only one learned parameter vector and one step size parameter our treatment includes general state dependent discounting and bootstrapping functions and a way of specifying varying degrees of interest in accurately valuing different states

geometry of the ergodic quotient reveals coherent structures in flows abstract dynamical systems that exhibit diverse behaviors can rarely be completely understood using a single approach however by identifying coherent structures in their state spaces i e regions of uniform and simpler behavior we could hope to study each of the structures separately and then form the understanding of the system as a whole the method we present in this paper uses trajectory averages of scalar functions on the state space to a identify invariant sets in the state space and b to form coherent structures by aggregating invariant sets that are similar across multiple spatial scales first we construct the ergodic quotient the object obtained by mapping trajectories to the space of the trajectory averages of a function basis on the state space second we endow the ergodic quotient with a metric structure that successfully captures how similar the invariant sets are in the state space finally we parametrize the ergodic quotient using intrinsic diffusion modes on it by segmenting the ergodic quotient based on the diffusion modes we extract coherent features in the state space of the dynamical system the algorithm is validated by analyzing the arnold beltrami childress flow which was the test bed for alternative approaches the ulam s approximation of the transfer operator and the computation of lagrangian coherent structures furthermore we explain how the method extends the poincare map analysis for periodic flows as a demonstration we apply the method to a periodically driven three dimensional hill s vortex flow discovering unknown coherent structures in its state space finally we discuss differences between the ergodic quotient and alternatives propose a generalization to analysis of quasi periodic structures and lay out future research directions

width of stochastic layers in near integrable two dimensional symplectic maps abstract width of stochastic layers around separatrix eight is studied for two dimensional symplectic maps close to integrable we prove that under rather general assumptions the relation w d 1 log holds where w is the width of the stochastic layer d is the width of a lobe domain and is a multiplier at the corresponding hyperbolic point

tensor decompositions and applications this survey provides an overview of higher order tensor decompositions their applications and available software a tensor is a multidimensional or n way array decompositions of higher order tensors i e n way arrays with n geq 3 have applications in psycho metrics chemometrics signal processing numerical linear algebra computer vision numerical analysis data mining neuroscience graph analysis and elsewhere two particular tensor decompositions can be considered to be higher order extensions of the matrix singular value decomposition candecomp parafac cp decomposes a tensor as a sum of rank one tensors and the tucker decomposition is a higher order form of principal component analysis there are many other tensor decompositions including indscal parafac2 candelinc dedicom and paratuck2 as well as nonnegative variants of all of the above the n way toolbox tensor toolbox and multilinear engine are examples of software packages for working with tensors

gradient methods for computing the drazin inverse solution the problem of computing the vector a db in the cases when it provides a solution of a singular linear system is reduced to the problem of finding a least squares solution of an appropriate linear system later the presented ideas are extended to arbitrary linear systems as a consequence we present iterative methods for computing the vector a db in general the functionality of the exposed algorithms is based on a specific representation of the drazin inverse solution as well as the properties that we introduce convergence results of the proposed methods are also considered illustrative numerical examples are presented

ray pattern of drazin inverses of some matrices abstract let q a be the set of complex matrices with the same ray pattern as a for any a q a if the drazin inverses of a and a have the same ray pattern then a is said to have ray drazin inverse in this paper we give some properties on matrices whose drazin inverses or m p inverses are ray unique and use these properties to obtain necessary and sufficient conditions for some block matrices to have ray drazin inverses

algorithmic survey of parametric value function approximation reinforcement learning rl is a machine learning answer to the optimal control problem it consists of learning an optimal control policy through interactions with the system to be controlled the quality of this policy being quantified by the so called value function a recurrent subtopic of rl concerns computing an approximation of this value function when the system is too large for an exact representation this survey reviews state of the art methods for parametric value function approximation by grouping them into three main categories bootstrapping residual and projected fixed point approaches related algorithms are derived by considering one of the associated cost functions and a specific minimization method generally a stochastic gradient descent or a recursive least squares approach

min max generalization for deterministic batch mode reinforcement learning relaxation schemes we study the min max optimization problem introduced in fonteneau et al towards min max reinforcement learning icaart 2010 springer heidelberg 2011 pp 61 77 for computing policies for batch mode reinforcement learning in a deterministic setting with fixed finite time horizon first we show that the min part of this problem is np hard we then provide two relaxation schemes the first relaxation scheme works by dropping some constraints in order to obtain a problem that is solvable in polynomial time the second relaxation scheme based on a lagrangian relaxation where all constraints are dualized can also be solved in polynomial time we also theoretically prove and empirically illustrate that both relaxation schemes provide better results than those given in fonteneau et al 2011 as cited above

kernel based least squares temporal difference with gradient correction a least squares temporal difference with gradient correction ls tdc algorithm and its kernel based version kernel based ls tdc kls tdc are proposed as policy evaluation algorithms for reinforcement learning rl ls tdc is derived from the tdc algorithm attributed to tdc derived by minimizing the mean square projected bellman error ls tdc has better convergence performance the least squares technique is used to omit the size step tuning of the original tdc and enhance robustness for kls tdc since the kernel method is used feature vectors can be selected automatically the approximate linear dependence analysis is performed to realize kernel sparsification in addition a policy iteration strategy motivated by kls tdc is constructed to solve control learning problems the convergence and parameter sensitivities of both ls tdc and kls tdc are tested through on policy learning off policy learning and control learning problems experimental results as compared with a series of corresponding rl algorithms demonstrate that both ls tdc and kls tdc have better approximation and convergence performance higher efficiency for sample usage smaller burden of parameter tuning and less sensitivity to parameters

estimating long term behavior of flows without trajectory integration the infinitesimal generator approach the long term distributions of trajectories of a flow are described by invariant densities i e fixed points of an associated transfer operator in addition global slowly mixing structures such as almost invariant sets which partition phase space into regions that are almost dynamically disconnected can also be identified by certain eigenfunctions of this operator indeed these structures are often hard to obtain by brute force trajectory based analyses in a wide variety of applications transfer operators have proven to be very efficient tools for an analysis of the global behavior of a dynamical system the computationally most expensive step in the construction of an approximate transfer operator is the numerical integration of many short term trajectories in this paper we propose to directly work with the infinitesimal generator instead of the operator completely avoiding trajectory integration we propose two different discretization schemes a cell based discretization and a spectral coll

manifold parametrizations by eigenfunctions of the laplacian and heat kernels we use heat kernels or eigenfunctions of the laplacian to construct local coordinates on large classes of euclidean domains and riemannian manifolds not necessarily smooth e g with alpha metric these coordinates are bi lipschitz on large neighborhoods of the domain or manifold with constants controlling the distortion and the size of the neighborhoods that depend only on natural geometric properties of the domain or manifold the proof of these results relies on novel estimates from above and below for the heat kernel and its gradient as well as for the eigenfunctions of the laplacian and their gradient that hold in the non smooth category and are stable with respect to perturbations within this category finally these coordinate systems are intrinsic and efficiently computable and are of value in applications

policy evaluation with temporal differences a survey and comparison policy evaluation is an essential step in most reinforcement learning approaches it yields a value function the quality assessment of states for a given policy which can be used in a policy improvement step since the late 1980s this research area has been dominated by temporal difference td methods due to their data efficiency however core issues such as stability guarantees in the off policy scenario improved sample efficiency and probabilistic treatment of the uncertainty in the estimates have only been tackled recently which has led to a large number of new approaches this paper aims at making these new developments accessible in a concise overview with foci on underlying cost functions the off policy scenario as well as on regularization in high dimensional feature spaces by presenting the first extensive systematic comparative evaluations comparing td lstd lspe fpkf the residual gradient algorithm bellman residual minimization gtd gtd2 and tdc we shed light on the strengths and weaknesses of the methods moreover we present alternative versions of lstd and lspe with drastically improved off policy performance

the inverse rank and product of tensors abstract in this paper we give some basic properties for the left right inverse rank and product of tensors the existence of order 2 left right inverses of tensors is characterized we obtain some equalities and inequalities on the tensor rank we also show that the rank of a uniform hypergraph is independent of the ordering of its vertices and the laplacian tensor and the signless laplacian tensor have the same rank for odd bipartite even uniform hypergraphs

linearization in the large of nonlinear systems and koopman operator spectrum abstract according to the hartman grobman theorem a nonlinear system can be linearized in a neighborhood of a hyperbolic stationary point here we extend this linearization around stable unstable equilibria or periodic orbits to the whole basin of attraction for both discrete diffeomorphisms and flows we discuss the connection of the linearizing transformation to the spectrum of koopman operator

natural gradient works efficiently in learning when a parameter space has a certain underlying structure the ordinary gradient of a function does not represent its steepest direction but the natural gradient does information geometry is used for calculating the natural gradients in the parameter space of perceptrons the space of matrices for blind source separation and the space of linear dynamical systems for blind source deconvolution the dynamical behavior of natural gradient online learning is analyzed and is proved to be fisher efficient implying that it has asymptotically the same performance as the optimal batch estimation of parameters this suggests that the plateau phenomenon which appears in the backpropagation learning algorithm of multilayer perceptrons might disappear or might not be so serious when the natural gradient is used an adaptive method of updating the learning rate is proposed and analyzed

regularized policy iteration with nonparametric function spaces we study two regularization based approximate policy iteration algorithms namely reg lspi and reg brm to solve reinforcement learning and planning problems in discounted markov decision processes with large state and finite action spaces the core of these algorithms are the regularized extensions of the least squares temporal difference lstd learning and bellman residual minimization brm which are used in the algorithms policy evaluation steps regularization provides a convenient way to control the complexity of the function space to which the estimated value function belongs and as a result enables us to work with rich nonparametric function spaces we derive efficient implementations of our methods when the function space is a reproducing kernel hilbert space we analyze the statistical properties of reg lspi and provide an upper bound on the policy evaluation error and the performance loss of the policy returned by this method our bound shows the dependence of the loss on the number of samples the capacity of the function space and some intrinsic properties of the underlying markov decision process the dependence of the policy evaluation bound on the number of samples is minimax optimal this is the first work that provides such a strong guarantee for a nonparametric approximate policy iteration algorithm

natural actor critic algorithms we present four new reinforcement learning algorithms based on actor critic natural gradient and function approximation ideas and we provide their convergence proofs actor critic reinforcement learning methods are online approximations to policy iteration in which the value function parameters are estimated using temporal difference learning and the policy parameters are updated by stochastic gradient descent methods based on policy gradients in this way are of special interest because of their compatibility with function approximation methods which are needed to handle large or infinite state spaces the use of temporal difference learning in this way is of special interest because in many applications it dramatically reduces the variance of the gradient estimates the use of the natural gradient is of interest because it can produce better conditioned parameterizations and has been shown to further reduce variance in some cases our results extend prior two timescale convergence results for actor critic methods by konda and tsitsiklis by using temporal difference learning in the actor and by incorporating natural gradients our results extend prior empirical studies of natural actor critic methods by peters vijayakumar and schaal by providing the first convergence proofs and the first fully incremental algorithms

kernel based least squares policy iteration for reinforcement learning in this paper we present a kernel based least squares policy iteration klspi algorithm for reinforcement learning rl in large or continuous state spaces which can be used to realize adaptive feedback control of uncertain dynamic systems by using klspi near optimal control policies can be obtained without much a priori knowledge on dynamic models of control plants in klspi mercer kernels are used in the policy evaluation of a policy iteration process where a new kernel based least squares temporal difference algorithm called klstd q is proposed for efficient policy evaluation to keep the sparsity and improve the generalization ability of klstd q solutions a kernel sparsification procedure based on approximate linear dependency ald is performed compared to the previous works on approximate rl methods klspi makes two progresses to eliminate the main difficulties of existing results one is the better convergence and near optimality guarantee by using the klstd q algorithm for policy evaluation with high precision the other is the automatic feature selection using the ald based kernel sparsification therefore the klspi algorithm provides a general rl method with generalization performance and convergence guarantee for large scale markov decision problems mdps experimental results on a typical rl task for a stochastic chain problem demonstrate that klspi can consistently achieve better learning efficiency and policy quality than the previous least squares policy iteration lspi algorithm furthermore the klspi method was also evaluated on two nonlinear feedback control problems including a ship heading control problem and the swing up control of a double link underactuated pendulum called acrobot simulation results illustrate that the proposed method can optimize controller performance using little a priori information of uncertain dynamic systems it is also demonstrated that klspi can be applied to online learning control by incorporating an initial controller to ensure online performance

constructing dynamic treatment regimes over indefinite time horizons summaryexisting methods for estimating optimal dynamic treatment regimes are limited to cases where a utility function is optimized over a fixed time period we develop an estimation procedure for the optimal dynamic treatment regime over an indefinite time period and derive associated large sample results the proposed method can be used to estimate the optimal dynamic treatment regime in chronic disease settings we illustrate this by simulating a dataset corresponding to a cohort of patients with diabetes that mimics the third wave of the national health and nutrition examination survey and examining the performance of the proposed method in controlling the level of haemoglobin a1c

convergence properties of krylov subspace methods for singular linear systems with arbitrary index abstract krylov subspace methods have been recently considered to solve singular linear systems ax b in this paper we derive the necessary and sufficient conditions guaranteeing that a krylov subspace method converges to a vector adb px0 where ad is the drazin inverse of a and p is the projection p i a d a let k be the index of a we further show that a d b px 0 x 0 r a k 1 n a is a generalized least squares solution of ax b in r a k n a finally we present the convergence bounds for the quasi minimal residual algorithm qmr and transpose free quasi minimal residual algorithm tfqmr the index k of a in this paper can be arbitrary which extends to the main results of freund and hochbruck numer linear algebra appl 1 1994 403 420 that only considers the case k 1

analysis of classification based policy iteration algorithms we introduce a variant of the classification based approach to policy iteration which uses a cost sensitive loss function weighting each classification mistake by its actual regret that is the difference between the action value of the greedy action and of the action chosen by the classifier for this algorithm we provide a full finite sample analysis our results state a performance bound in terms of the number of policy improvement steps the number of rollouts used in each iteration the capacity of the considered policy space classifier and a capacity measure which indicates how well the policy space can approximate policies that are greedy with respect to any of its members the analysis reveals a tradeoff between the estimation and approximation errors in this classification based policy iteration setting furthermore it confirms the intuition that classification based policy iteration algorithms could be favorably compared to value based approaches when the policies can be approximated more easily than their corresponding value functions we also study the consistency of the algorithm when there exists a sequence of policy spaces with increasing capacity

mreklm a fast multiple empirical kernel learning machine abstract multiple empirical kernel learning mekl explicitly maps samples into different empirical feature spaces in which the kernel features of the mapped samples can be directly provided thus mekl is much easier than the conventional multiple kernel learning mkl in terms of processing and analyzing the structure of mapped feature spaces however the computational complexity of mekl with m empirical feature spaces is o mn 3 where n is the number of training samples the dimensions of the generated empirical feature spaces are approximate to n when dealing with large scale problems mekl cannot handle them properly due to the severe computation and memory burden moreover most existing mekl utilizes the gradient decent optimization to learn classifiers but it is time consuming for training therefore this paper proposes a multiple random empirical kernel learning machine mreklm to overcome these problems the proposed mreklm adopts the random projection idea to map samples into multiple low dimensional empirical feature spaces with lower computational complexity o mp 3 where p n is the number of the randomly selected samples after that mreklm adopts an analytical optimization approach to directly deal with multi class problems the computational complexity of mreklm is o m 3 p 3 experimental results also validate both efficiency and effectiveness of the proposed mreklm the contributions of this work are 1 proposing a fast mekl algorithm named mreklm 2 introducing an efficient random empirical kernel mapping approach and 3 extending the capability of mekl to handle large scale problems

normal bandits of unknown means and variances consider the problem of sampling sequentially from a finite number of n 2 populations specified by random variables xki i 1 n and k 1 2 where xki denotes the outcome from population i the kth time it is sampled it is assumed that for each fixed i xki k 1 is a sequence of i i d normal random variables with unknown mean i and unknown variance i2 the objective is to have a policy for deciding from which of the n populations to sample from at any time t 1 2 so as to maximize the expected sum of outcomes of n total samples or equivalently to minimize the regret due to lack on information of the parameters i and i2 in this paper we present a simple inflated sample mean ism index policy that is asymptotically optimal in the sense of theorem 4 below this resolves a standing open problem from burnetas and katehakis 1996b additionally finite horizon regret bounds are given

weak convergence properties of constrained emphatic temporal difference learning with constant and slowly diminishing stepsize we consider the emphatic temporal difference td algorithm etd for learning the value functions of stationary policies in a discounted finite state and action markov decision process the etd algorithm was recently proposed by sutton mahmood and white 2016 to solve a long standing divergence problem of the standard td algorithm when it is applied to off policy training where data from an exploratory policy are used to evaluate other policies of interest the almost sure convergence of etd has been proved in our recent work under general off policy training conditions but for a narrow range of diminishing stepsize in this paper we present convergence results for constrained versions of etd with constant stepsize and with diminishing stepsize from a broad range our results characterize the asymptotic behavior of the trajectory of iterates produced by those algorithms and are derived by combining key properties of etd with powerful convergence theorems from the weak convergence methods in stochastic approximation theory for the case of constant stepsize in addition to analyzing the behavior of the algorithms in the limit as the stepsize parameter approaches zero we also analyze their behavior for a fixed stepsize and bound the deviations of their averaged iterates from the desired solution these results are obtained by exploiting the weak feller property of the markov chains associated with the algorithms and by using ergodic theorems for weak feller markov chains in conjunction with the convergence results we get from the weak convergence methods besides etd our analysis also applies to the off policy td algorithm when the divergence issue is avoided by setting sufficiently large it yields for that case new results on the asymptotic convergence properties of constrained off policy td with constant or slowly diminishing stepsize

an analysis of temporal difference learning with function approximation we discuss the temporal difference learning algorithm as applied to approximating the cost to go function of an infinite horizon discounted markov chain the algorithm we analyze updates parameters of a linear function approximator online during a single endless trajectory of an irreducible aperiodic markov chain with a finite or infinite state space we present a proof of convergence with probability one a characterization of the limit of convergence and a bound on the resulting approximation error furthermore our analysis is based on a new line of reasoning that provides new intuition about the dynamics of temporal difference learning in addition to proving new and stronger positive results than those previously available we identify the significance of online updating and potential hazards associated with the use of nonlinear function approximators first we prove that divergence may occur when updates are not based on trajectories of the markov chain this fact reconciles positive and negative results that have been discussed in the literature regarding the soundness of temporal difference learning second we present an example illustrating the possibility of divergence when temporal difference learning is used in the presence of a nonlinear function approximator

generalized polynomial approximations in markovian decision processes abstract fitting the value function in a markovian decision process by a linear superposition of m basis functions reduces the problem dimensionality from the number of states down to m with good accuracy retained if the value function is a smooth function of its argument the state vector this paper provides for both the discounted and undiscounted cases three algorithms for computing the coefficients in the linear superposition linear programming policy iteration and least squares

robust approximate bilinear programming for value function approximation value function approximation methods have been successfully used in many applications but the prevailing techniques often lack useful a priori error bounds we propose a new approximate bilinear programming formulation of value function approximation which employs global optimization the formulation provides strong a priori guarantees on both robust and expected policy loss by minimizing specific norms of the bellman residual solving a bilinear program optimally is np hard but this worst case complexity is unavoidable because the bellman residual minimization itself is np hard we describe and analyze the formulation as well as a simple approximate algorithm for solving bilinear programs the analysis shows that this algorithm offers a convergent generalization of approximate policy iteration we also briefly analyze the behavior of bilinear programming algorithms under incomplete samples finally we demonstrate that the proposed approach can consistently minimize the bellman residual on simple benchmark problems

construction of approximation spaces for reinforcement learning linear reinforcement learning rl algorithms like least squares temporal difference learning lstd require basis functions that span approximation spaces of potential value functions this article investigates methods to construct these bases from samples we hypothesize that an ideal approximation spaces should encode diffusion distances and that slow feature analysis sfa constructs such spaces to validate our hypothesis we provide theoretical statements about the lstd value approximation error and induced metric of approximation spaces constructed by sfa and the state of the art methods krylov bases and proto value functions pvf in particular we prove that sfa minimizes the average over all tasks in the same environment bound on the above approximation error compared to other methods sfa is very sensitive to sampling and can sometimes fail to encode the whole state space we derive a novel importance sampling modification to compensate for this effect finally the lstd and least squares policy iteration lspi performance of approximation spaces constructed by krylov bases pvf sfa and pca is compared in benchmark tasks and a visual robot navigation experiment both in a realistic simulation and with a robot the results support our hypothesis and suggest that i sfa provides subspace invariant features for mdps with self adjoint transition operators which allows strong guarantees on the approximation error ii the modified sfa algorithm is best suited for lspi in both discrete and continuous state spaces and iii approximation spaces encoding diffusion distances facilitate lspi performance

online model free n step hdp with stability analysis because of a powerful temporal difference td with lambda td lambda learning method this paper presents a novel n step adaptive dynamic programming adp architecture that combines td lambda with regular td learning for solving optimal control problems with reduced iterations in contrast with a backward view learning of td lambda that is required an extra parameter named eligibility traces to update at the end of each episode offline training the new design in this paper has forward view learning which is updated at each time step online training without needing the eligibility trace parameter in various applications without mathematical models therefore the new design is called the online model free n step action dependent ad heuristic dynamic programming nshdp lambda nshdp lambda has three neural networks the critic network cn with regular one step td td 0 the cn with n step td learning or td lambda and the actor network an because the forward view learning does not require any extra eligibility traces associated with each state the nshdp lambda architecture has low computational costs and is memory efficient furthermore the stability is proven for nshdp lambda under certain conditions by using lyapunov analysis to obtain the uniformly ultimately bounded uub property we compare the results with the performance of hdp and traditional action dependent hdp lambda adhdp lambda with different lambda values moreover a complex nonlinear system and 2 d maze problem are two simulation benchmarks in this paper and the third one is an inverted pendulum simulation benchmark which is presented in the supplemental material part of this paper nshdp lambda performance is examined and compared with other adp methods

diffusion maps and coarse graining a unified framework for dimensionality reduction graph partitioning and data set parameterization we provide evidence that nonlinear dimensionality reduction clustering and data set parameterization can be solved within one and the same framework the main idea is to define a system of coordinates with an explicit metric that reflects the connectivity of a given data set and that is robust to noise our construction which is based on a markov random walk on the data offers a general scheme of simultaneously reorganizing and subsampling graphs and arbitrarily shaped data sets in high dimensions using intrinsic geometry we show that clustering in embedding spaces is equivalent to compressing operators the objective of data partitioning and clustering is to coarse grain the random walk on the data while at the same time preserving a diffusion operator for the intrinsic geometry or connectivity of the data set up to some accuracy we show that the quantization distortion in diffusion space bounds the error of compression of the operator thus giving a rigorous justification for k means clustering in diffusion space and a precise measure of the performance of general clustering algorithms

approximate dynamic programming with a fuzzy parameterization dynamic programming dp is a powerful paradigm for general nonlinear optimal control computing exact dp solutions is in general only possible when the process states and the control actions take values in a small discrete set in practice it is necessary to approximate the solutions therefore we propose an algorithm for approximate dp that relies on a fuzzy partition of the state space and on a discretization of the action space this fuzzy q iteration algorithm works for deterministic processes under the discounted return criterion we prove that fuzzy q iteration asymptotically converges to a solution that lies within a bound of the optimal solution a bound on the suboptimality of the solution obtained in a finite number of iterations is also derived under continuity assumptions on the dynamics and on the reward function we show that fuzzy q iteration is consistent i e that it asymptotically obtains the optimal solution as the approximation accuracy increases these properties hold both when the parameters of the approximator are updated in a synchronous fashion and when they are updated asynchronously the asynchronous algorithm is proven to converge at least as fast as the synchronous one the performance of fuzzy q iteration is illustrated in a two link manipulator control problem

e cospectral hypergraphs and some hypergraphs determined by their spectra two k uniform hypergraphs are said to be cospectral e cospectral if their adjacency tensors have the same characteristic polynomial e characteristic polynomial a k uniform hypergraph h is said to be determined by its spectrum if there is no non isomorphic k uniform hypergraph cospectral with h in this note we give a method for constructing e cospectral hypergraphs which is similar with godsil mckay switching some hypergraphs are shown to be determined by their spectra

detecting and locating near optimal almost invariant sets and cycles the behaviors of trajectories of nonlinear dynamical systems are notoriously hard to characterize and predict rather than characterizing dynamical behavior at the level of trajectories we consider following the evolution of sets there are often collections of sets that behave in a very predictable way in spite of the fact that individual trajectories are entirely unpredictable such special collections of sets are invisible to studies of long trajectories we describe a global set oriented method to detect and locate these large dynamical structures our approach is a marriage of new ideas in modern dynamical systems theory and the novel application of graph dissection algorithms

definition and properties of lagrangian coherent structures from finite time lyapunov exponents in two dimensional aperiodic flows this paper develops the theory and computation of lagrangian coherent structures lcs which are defined as ridges of finite time lyapunov exponent ftle fields these ridges can be seen as finite time mixing templates such a framework is common in dynamical systems theory for autonomous and time periodic systems in which examples of lcs are stable and unstable manifolds of fixed points and periodic orbits the concepts defined in this paper remain applicable to flows with arbitrary time dependence and in particular to flows that are only defined computed or measured over a finite interval of time previous work has demonstrated the usefulness of ftle fields and the associated lcss for revealing the lagrangian behavior of systems with general time dependence however ridges of the ftle field need not be exactly advected with the flow the main result of this paper is an estimate for the flux across an lcs which shows that the flux is small and in most cases negligible for well defined lcss or those that rotate at a speed comparable to the local eulerian velocity field and are computed from ftle fields with a sufficiently long integration time under these hypotheses the structures represent nearly invariant manifolds even in systems with arbitrary time dependence the results are illustrated on three examples the first is a simplified dynamical model of a double gyre flow the second is surface current data collected by high frequency radar stations along the coast of florida and the third is unsteady separation over an airfoil in all cases the existence of lcss governs the transport and it is verified numerically that the flux of particles through these distinguished lines is indeed negligible

least squares policy iteration we propose a new approach to reinforcement learning for control problems which combines value function approximation with linear architectures and approximate policy iteration this new approach is motivated by the least squares temporal difference learning algorithm lstd for prediction problems which is known for its efficient use of sample experiences compared to pure temporal difference algorithms heretofore lstd has not had a straightforward application to control problems mainly because lstd learns the state value function of a fixed policy which cannot be used for action selection and control without a model of the underlying process our new algorithm least squares policy iteration lspi learns the state action value function which allows for action selection without a model and for incremental policy improvement within a policy iteration framework lspi is a model free off policy method which can use efficiently and reuse in each iteration sample experiences collected in any manner by separating the sample collection method the choice of the linear approximation architecture and the solution method lspi allows for focused attention on the distinct elements that contribute to practical reinforcement learning lspi is tested on the simple task of balancing an inverted pendulum and the harder task of balancing and riding a bicycle to a target location in both cases lspi learns to control the pendulum or the bicycle by merely observing a relatively small number of trials where actions are selected randomly lspi is also compared against q learning both with and without experience replay using the same value function architecture while lspi achieves good performance fairly consistently on the difficult bicycle task q learning variants were rarely able to balance for more than a small fraction of the time needed to reach the target location

spectral connectivity analysis spectral kernel methods are techniques for mapping data into a coordinate system that efficiently reveals the geometric structure in particular the connectivity of the data these methods depend on tuning parameters we analyze the dependence of the method on these tuning parameters we focus on one particular technique diffusion maps but our analysis can be used for other spectral methods as well we identify the key population quantities we define an appropriate risk function for analyzing the estimators and we explain how these methods relate to classical kernel smoothing we also show that in some cases fast rates of convergence are possible even in high dimensions the appendix of the article is available online as supplementary materials

an analytic framework for identifying finite time coherent sets in time dependent dynamical systems abstract the study of transport and mixing processes in dynamical systems is particularly important for the analysis of mathematical models of physical systems barriers to transport which mitigate mixing are currently the subject of intense study in the autonomous setting the use of transfer operators perron frobenius operators to identify invariant and almost invariant sets has been particularly successful in the nonautonomous time dependent setting coherent sets a time parameterised family of minimally dispersive sets are a natural extension of almost invariant sets the present work introduces a new analytic transfer operator construction that enables the calculation of finite time coherent sets sets are that minimally dispersive over a finite time interval this new construction also elucidates the role of diffusion in the calculation and we show how properties such as the spectral gap and the regularity of singular vectors scale with noise amplitude the construction can also be applied to general markov processes on continuous state space

reconstruction of normal forms by learning informed observation geometries from data abstract the discovery of physical laws consistent with empirical observations is at the heart of applied science and engineering these laws typically take the form of nonlinear differential equations depending on parameters dynamical systems theory provides through the appropriate normal forms an intrinsic prototypical characterization of the types of dynamical regimes accessible to a given model using an implementation of data informed geometry learning we directly reconstruct the relevant normal forms a quantitative mapping from empirical observations to prototypical realizations of the underlying dynamics interestingly the state variables and the parameters of these realizations are inferred from the empirical observations without prior knowledge or understanding they parametrize the dynamics intrinsically without explicit reference to fundamental physical quantities

kam tori are very sticky rigorous lower bounds on the time to move away from an invariant lagrangian torus with linear flow abstract hamiltonian perturbation theory is used to derive rigorous nekhoroshev type lower bounds on the time it takes to move away from an invariant lagrangian torus when the flow on the torus is conjugate to a highly non resonant linear flow in particular these results give bounds on the time to move away from a kam torus if r is the distance from the torus then the lower bound on the time to double the distance from the torus is c r e k r where c and k are constants 1 n 2 n is the number of degrees of freedom and is a measure of the irrationality of the flow we thus prove that nearby trajectories take an extremely long time to move away from such an invariant torus and that the more irrational the linear flow the stickier the torus will be these estimates do not require the steepness or quasi convexity assumptions of nekherosev s theorem

extraction and prediction of coherent patterns in incompressible flows through space time koopman analysis abstract we develop methods for detecting and predicting the evolution of coherent spatiotemporal patterns in incompressible time dependent fluid flows driven by ergodic dynamical systems our approach is based on representations of the generators of the koopman and perron frobenius groups of operators governing the evolution of observables and probability measures on lagrangian tracers respectively in a smooth orthonormal basis learned from velocity field snapshots through the diffusion maps algorithm these operators are defined on the product space between the state space of the fluid flow and the spatial domain in which the flow takes place and as a result their eigenfunctions correspond to global space time coherent patterns under a skew product dynamical system moreover using this data driven representation of the generators in conjunction with leja interpolation for matrix exponentiation we construct model free prediction schemes for the evolution of observables and probability densities defined on the tracers we present applications to periodic gaussian vortex flows and aperiodic flows generated by lorenz 96 systems