nonconvergence to unstable points in urn models and stochastic approximations a particle in mathbf r d moves in discrete time the size of the n th step is of order 1 n and when the particle is at a position mathbf v the expectation of the next step is in the direction mathbf f mathbf v for some fixed vector function mathbf f of class c 2 it is well known that the only possible points mathbf p where mathbf v n may converge are those satisfying mathbf f mathbf p mathbf 0 this paper proves that convergence to some of these points is in fact impossible as long as the noise the difference between each step and its expectation is sufficiently omnidirectional the points where convergence is impossible are the unstable critical points for the autonomous flow d dt mathbf v t mathbf f v t this generalizes several known results that say convergence is impossible at a repelling node of the flow

lie semialgebras are real phenomena in recent year there has been considerable interest in local semigroups in lie groups partly from the view point of geometric control theory partly from the view point of differential geometry for the former aspect see br hi js l for the latter lo o11 and 2 pa ro vi lie theory relates local subgroups of a lie group with subalgebras of its lie algebra in just the same spirit it relates certain local subsemigroups with lie semialgebras in its lie algebra here a lie semialgebra in a finite dimensional real lie algebra is a wedge i e a topologically closed subset which is stable under addition and multiplication by non negative scalars for which there is a neighborhood b of 0 in l such that for all x y b the campbellhausdorff product x y x y 89 y l is well defined and in addition stays in w for x ye w if w is a lie semialgebra in l then w w is a lie subalgebra hl 2 l if w is any wedge in l and if w contains the lie algebra w w generated by all commutators w w with w w w then w is a lie semialgebra lie semialgebras arising in this fashion we will call trivial we shall prove the following

learning in games with continuous action sets and unknown payoff functions this paper examines the convergence of no regret learning in games with continuous action sets for concreteness we focus on learning via dual averaging a widely used class of no regret learning schemes where players take small steps along their individual payoff gradients and then mirror the output back to their action sets in terms of feedback we assume that players can only estimate their payoff gradients up to a zero mean error with bounded variance to study the convergence of the induced sequence of play we introduce the notion of variational stability and we show that stable equilibria are locally attracting with high probability whereas globally stable equilibria are globally attracting with probability 1 we also discuss some applications to mixed strategy learning in finite games and we provide explicit estimates of the method s convergence speed

reinforcement learning with restrictions on the action set consider a two player normal form game repeated over time we introduce an adaptive learning procedure where the players only observe their own realized payoff at each stage we assume that agents do not know their own payoff function and have no information on the other player furthermore we assume that they have restrictions on their own actions such that at each stage their choice is limited to a subset of their action set we prove that the empirical distributions of play converge to the set of nash equilibria for zero sum and potential games and games where one player has two actions

an adjusted payoff based procedure for normal form games we study a simple adaptive model in the framework of an n player normal form game the model consists of a repeated game where the players only know their own action space and their own payoff scored at each stage not those of the other agents each player in order to update her mixed action computes the average vector payoff she has obtained by using the number of times she has played each pure action the resulting stochastic process is analyzed via the ode method from stochastic approximation theory we are interested in the convergence of the process to rest points of the related continuous dynamics results concerning almost sure convergence and convergence with positive probability are obtained and applied to a traffic game we also provide some examples where convergence occurs with probability zero

on sophus lie s fundamental theorem sophus lie s third theorem asserts that for every finite dimensional lie algebra there is a local lie group with the given algebra as tangent algebra at the identity in particular each subalgebra of the lie algebra of a local lie group determines a local subgroup whose lie algebra is the given one recent interest in subsemigroups and local subsemigroups of lie groups motivate the investigation to which extent local semigroups in lie groups are determined by infinitesimal generators it is known that the set l s of tangent vectors at the identity of a local semigroup s with identity in a local lie group g is an additively closed convex subset of the lie algebra l g which is topologically closed and contains 0 its largest vector subspace h is a subalgebra and all automorphisms of l g of the form eadx with x h leave l s invariant as a whole with adx y x y we call an additively and topologically closed convex subset w containing 0 in a lie algebra l a lie wedge if it satisfies eadxw w for all x w t w in this paper we show that lie wedges are precisely the infinitesimal generators of local semigroups in lie groups more precisely we show that in a banach lie algebra l for every lie wedge w whose edge w w has a closed vector space complement in l there is an open ball b around 0 in l such that the cambell hausdorff series x y x y 12 x y converges absolutely for x y b and there is a subset s b with i 0 s ii s s b s and l s w

a simple adaptive procedure leading to correlated equilibrium we propose a new and simple adaptive procedure for playing a game regret matching in this procedure players may depart from their current play with probabilities that are proportional to measures of regret for not having used other strategies in the past it is shown that our adaptive procedure guarantees that with probability one the empirical distributions of play converge to the set of correlated equilibria of the game

invariant convex cones and causality in semisimple lie algebras and groups the convex cones in a simple lie algebra g invariant under the adjoint group g of g are studied using a earlier abstract classification of such cones we find explicit algebraic presentations of such cones in all the classical hermitian symmetric lie algebras nontrivial such cones exist only in these cases the g orbits in such cones are listed the notion of a temporal action of a lie group with an invariant causal orientation upon a causally oriented manifold is defined the canonical actions of such classical groups g as above on the shilov boundaries of the associated tube type hermitian symmetric spaces are shown to be temporal actions corollaries are 1 the existence of nontrivial lie semigroups s in the infinite sheeted coverings g of g which are invariant under conjugation by g and satisfy s s 1 e and 2 the global causality i e no closed time like curves of such covering groups g

dynamics of boltzmann q learning in two player two action games we consider the dynamics of q learning in two player two action games with a boltzmann exploration mechanism for any nonzero exploration rate the dynamics is dissipative which guarantees that agent strategies converge to rest points that are generally different from the game s nash equlibria nes we provide a comprehensive characterization of the rest point structure for different games and examine the sensitivity of this structure with respect to the noise due to exploration our results indicate that for a class of games with multiple nes the asymptotic behavior of learning dynamics can undergo drastic changes at critical exploration rates furthermore we demonstrate that for certain games with a single ne it is possible to have additional rest points not corresponding to any ne that persist for a finite range of the exploration rates and disappear when the exploration rates of both players tend to zero

penalty regulated dynamics and robust learning procedures in games starting from a heuristic learning scheme for n person games we derive a new class of continuous time learning dynamics consisting of a replicator like drift adjusted by a penalty term that renders the boundary of the game s strategy space repelling these penalty regulated dynamics are equivalent to players keeping an exponentially discounted aggregate of their ongoing payoffs and then using a smooth best response to pick an action based on these performance scores owing to this inherent duality the proposed dynamics satisfy a variant of the folk theorem of evolutionary game theory and they converge to arbitrarily precise approximations of nash equilibria in potential games motivated by applications to traffic engineering we exploit this duality further to design a discrete time payoff based learning algorithm which retains these convergence properties and only requires players to observe their in game payoffs moreover the algorithm remains robust in the presence of stochastic perturbations and observation errors and it does not require any synchronization between players

individual q learning in normal form games the single agent multi armed bandit problem can be solved by an agent that learns the values of each action using reinforcement learning however the multi agent version of the problem the iterated normal form game presents a more complex challenge since the rewards available to each agent depend on the strategies of the others we consider the behavior of value based learning agents in this situation and show that such agents cannot generally play at a nash equilibrium although if smooth best responses are used a nash distribution can be reached we introduce a particular value based learning algorithm which we call individual q learning and use stochastic approximation to study the asymptotic behavior showing that strategies will converge to nash distribution almost surely in 2 player zero sum games and 2 player partnership games player dependent learning rates are then considered and it is shown that this extension converges in some games for which many algorithms including the basic algorithm initially considered fail to converge

convergent learning algorithms for unknown reward games in this paper we address the problem of convergence to nash equilibria in games with rewards that are initially unknown and must be estimated over time from noisy observations these games arise in many real world applications whenever rewards for actions cannot be prespecified and must be learned online but standard results in game theory do not consider such settings for this problem we derive a multiagent version of mathcal q learning to estimate the reward functions using novel forms of the epsilon greedy learning policy using these mathcal q learning schemes to estimate reward functions we then provide conditions guaranteeing the convergence of adaptive play and the better reply processes to nash equilibria in potential games and games with more general forms of acyclicity and of regret matching to the set of correlated equilibria in generic games a secondary result is that we prove the strong ergoditicity of stochastic adaptive play and stochastic better reply processes in the ca

repetitive control system a new type servo system for periodic exogenous signals a control scheme called repetitive control is proposed in which the controlled variables follow periodic reference commands a high accuracy asymptotic tracking property is achieved by implementing a model that generates the periodic signals of period l into the closed loop system sufficient conditions for the stability of repetitive control systems and modified repetitive control systems are derived by applying the small gain theorem and the stability theorem for time lag systems synthesis algorithms are presented by both the state space approach and the factorization approach in the former approach the technique of the kalman filter and perfect regulation is utilized while coprime factorization over the matrix ring of proper stable rational functions and the solution of the hankel norm approximation are used in the latter one

a new periodic adaptive control approach for time varying parameters with known periodicity a new adaptive control approach is proposed to deal with nonlinear systems with a class of time varying parametric uncertainties which are in an unknown compact set periodic nonvanishing rapid time varying and the only prior knowledge is the periodicity unlike most continuous time adaptation laws which are of differential types in this work a periodic type adaptation law is introduced for continuous time systems the new adaptive controller updates the parameters and the control signal periodically in a pointwise manner over one entire period in the sequel achieves the asymptotic tracking convergence

adaptive neural network control for a class of low triangular structured nonlinear systems in this paper a class of unknown perturbed nonlinear systems is theoretically stabilized by using adaptive neural network control the systems with disturbances and nonaffine unknown functions have low triangular structure which generalizes both strict feedback uncertain systems and pure feedback ones there do not exist any effective methods to stabilize this kind of systems with some new conclusions for nussbaum gain functions ngf and the idea of backstepping semiglobal uniformal and ultimate boundedness of all the signals in the closed loop is proved at equilibrium point the two problems control directions and control singularity are well dealt with the effectiveness of proposed scheme is shown by simulation on a proper nonlinear system

adaptive learning control of nonlinear systems by output error feedback this paper addresses the problem of designing an output error feedback control for single input single output nonlinear systems with uncertain smooth output dependent nonlinearities whose local lipschitz constants are known the considered systems are required to be observable minimum phase with known relative degree and known high frequency gain sign linear systems are included the reference output signal is assumed to be smooth and periodic with known period by developing in fourier series expansion a suitable periodic input reference signal an output error feedback adaptive learning control is designed which ldquolearnsrdquo the input reference signal by identifying its fourier coefficients bounded closed loop signals and exponential tracking of both input and output reference signals are obtained when the fourier series expansion is finite while arbitrary small tracking errors are exponentially achieved otherwise the resulting control is not model based is independent of the system order and depends only on the relative degree the reference signal period and the high frequency gain sign

learning in games via reinforcement and regularization we investigate a class of reinforcement learning dynamics where players adjust their strategies based on their actions cumulative payoffs over time specifically by playing mixed strategies that maximize their expected cumulative payoff minus a regularization term a widely studied example is exponential reinforcement learning a process induced by an entropic regularization term which leads mixed strategies to evolve according to the replicator dynamics however in contrast to the class of regularization functions used to define smooth best responses in models of stochastic fictitious play the functions used in this paper need not be infinitely steep at the boundary of the simplex in fact dropping this requirement gives rise to an important dichotomy between steep and nonsteep cases in this general framework we extend several properties of exponential learning including the elimination of dominated strategies the asymptotic stability of strict nash equilibria and the convergence of time averaged trajectories in zero sum games with an interior nash equilibrium

brief robust learning control for a class of nonlinear systems with periodic and aperiodic uncertainties this paper addresses the robust learning control problem for a class of nonlinear systems with structured periodic and unstructured aperiodic uncertainties a recursive technique is proposed which extends the backstepping idea to the robust repetitive learning control systems a learning evaluation function instead of a lyapunov function is formulated as a guideline for derivation of the control strategy which guarantees the asymptotic stability of the tracking system a design example is given

adaptive tracking for periodically time varying and nonlinearly parameterized systems using multilayer neural networks this brief addresses the problem of designing adaptive neural network tracking control for a class of strict feedback systems with unknown time varying disturbances of known periods which nonlinearly appear in unknown functions multilayer neural network mnn and fourier series expansion fse are combined into a novel approximator to model each uncertainty in systems dynamic surface control dsc approach and integral type lyapunov function ilf technique are combined to design the control algorithm the ultimate uniform boundedness of all closed loop signals is guaranteed the tracking error is proved to converge to a small residual set around the origin two simulation examples are provided to illustrate the feasibility of control scheme proposed in this brief

semigroups in lie groups semialgebras in lie algebras consider a subsemigroup of a lie group containing the identity and being ruled by one parameter semigroups near the identity we associate with it the set w of its tangent vectors at the identity and obtain a subset of the lie algebra l of the group the set w has the following properties i w w w ii r w c w iii w w and the crucial property iv for all sufficiently small elements x and v in w one has x y x y 2 x vi campbell haus dorff e w we call a subset w of a finite dimensional real lie algebra l a lie semialgebra if it satisfies these conditions and develop a theory of lie semialgebras in particular we show that a subset w satisfying i iii is a lie semialgebra if and only if for each point x of w and the appropriately defined tangent space t to w in x one has x tj c t the lie semialgebra w of a subgroup is always a vector space and for vector spaces w we have t w for all x in w and thus the condition reduces to the old property that w is a lie algebra in the introduction we fully discuss all lie semialgebras of dimension not exceeding three our methods include a full duality theory for closed convex wedges basic lie group theory and certain aspects of ordinary differential equations