refined large deviations asymptotics for markov modulated infinite server systems many networking related settings can be modeled by markov modulated infinite server systems in such models the customers arrival rates and service rates are modulated by a markovian background process additionally there are infinitely many servers and consequently the resulting model is often used as a proxy for the corresponding many server model the markov modulated infinite server model hardly allows any explicit analysis apart from results in terms of systems of ordinary or partial differential equations for the underlying probability generating functions and recursions to obtain all moments as a consequence recent research efforts have pursued an asymptotic analysis in various limiting regimes notably the central limit regime describing fluctuations around the average behavior and the large deviations regime focusing on rare events many of these results use the property that the number of customers in the system obeys a poisson distribution with a random parameter the objective of this paper is to develop techniques to accurately approximate tail probabilities in the large deviations regime we consider the scaling in which the arrival rates are inflated by a factor n and we are interested in the probability that the number of customers exceeds a given level na where earlier contributions focused on so called logarithmic asymptotics of this exceedance probability which are inherently imprecise the present paper improves upon those results in that exact asymptotics are established these are found in two steps first the distribution of the random parameter of the poisson distribution is characterized and then this knowledge is used to identify the exact asymptotics the paper is concluded by a set of numerical experiments in which the accuracy of the asymptotic results is assessed

large deviations for two scaled diffusions we formulate large deviations principle ldp for diffusion pair x x t t where first component has a small diffusion parameter while the second is ergodic markovian process with fast time more exactly the ldp is established for x with dt dz being an occupation type measure corresponding to t in some sense we obtain a combination of freidlin wentzell s and donsker varadhan s results our approach relies on the concept of the exponential tightness and puhalskii s theorem

the continuum armed bandit problem in this paper we consider the multiarmed bandit problem where the arms are chosen from a subset of the real line and the mean rewards are assumed to be a continuous function of the arms the problem with an infinite number of arms is much more difficult than the usual one with a finite number of arms because the built in learning task is now infinite dimensional we devise a kernel estimator based learning scheme for the mean reward as a function of the arms using this learning scheme we construct a class of certainty equivalence control with forcing schemes and derive asymptotic upper bounds on their learning loss to the best of our knowledge these bounds are the strongest rates yet available moreover they are stronger than the o n required for optimality with respect to the average cost per unit time criterion

two scale large deviations for chemical reaction kinetics through second quantization path integral motivated by the study of rare events for a typical genetic switching model in systems biology in this paper we aim to establish the general two scale large deviations for chemical reaction systems we build a formal approach to explicitly obtain the large deviation rate functionals for the considered two scale processes based upon the second quantization path integral technique we get three important types of large deviation results when the underlying two timescales are in three different regimes this is realized by singular perturbation analysis to the rate functionals obtained by the path integral we find that the three regimes possess the same deterministic mean field limit but completely different chemical langevin approximations the obtained results are natural extensions of the classical large volume limit for chemical reactions we also discuss its implication on the single molecule michaelis menten kinetics our framework and results can be applied to understand general multi scale systems including diffusion processes

kernel estimation and model combination in a bandit problem with covariates multi armed bandit problem is an important optimization game that requires an exploration exploitation tradeoff to achieve optimal total reward motivated from industrial applications such as online advertising and clinical research we consider a setting where the rewards of bandit machines are associated with covariates and the accurate estimation of the corresponding mean reward functions plays an important role in the performance of allocation rules under a exible problem setup we establish asymptotic strong consistency and perform a finite time regret analysis for a sequential randomized allocation strategy based on kernel estimation in addition since many nonparametric and parametric methods in supervised learning may be applied to estimating the mean reward functions but guidance on how to choose among them is generally unavailable we propose a model combining allocation strategy for adaptive performance simulations and a real data evaluation are conducted to illustrate the performance of the proposed allocation strategy

combining expert advice in reactive environments experts algorithms constitute a methodology for choosing actions repeatedly when the rewards depend both on the choice of action and on the unknown current state of the environment an experts algorithm has access to a set of strategies experts each of which may recommend which action to choose the algorithm learns how to combine the recommendations of individual experts so that in the long run for any fixed sequence of states of the environment it does as well as the best expert would have done relative to the same sequence this methodology may not be suitable for situations where the evolution of states of the environment depends on past chosen actions as is usually the case for example in a repeated non zero sum game a general exploration exploitation experts method is presented along with a proper definition of value the definition is shown to be adequate in that it both captures the impact of an expert s actions on the environment and is learnable the new experts method is quite different from previously proposed experts algorithms it represents a shift from the paradigms of regret minimization and myopic optimization to consideration of the long term effect of a player s actions on the environment the importance of this shift is demonstrated by the fact that this algorithm is capable of inducing cooperation in the repeated prisoner s dilemma game whereas previous experts algorithms converge to the suboptimal non cooperative play the method is shown to asymptotically perform as well as the best available expert several variants are analyzed from the viewpoint of the exploration exploitation tradeoff including explore then exploit polynomially vanishing exploration constant frequency exploration and constant size exploration phases complexity and performance bounds are proven

stochastic processes in random graphs we study the asymptotics of large moderate and normal deviations for the connected components of the sparse random graph by the method of stochastic processes we obtain the logarithmic asymptotics of large deviations of the joint distribution of the number of connected components of the sizes of the giant components and of the numbers of the excess edges of the giant components for the supercritical case we obtain the asymptotics of normal deviations and the logarithmic asymptotics of large and moderate deviations of the joint distribution of the number of components of the size of the largest component and of the number of the excess edges of the largest component for the critical case we obtain the logarithmic asymptotics of moderate deviations of the joint distribution of the sizes of connected components and of the numbers of the excess edges some related asymptotics are also established the proofs of the large and moderate deviation asymptotics employ methods of idempotent probability theory as a byproduct of the results we provide some additional insight into the nature of phase transitions in sparse random graphs

the performance of universal encoding universal coding theory is surveyed from the viewpoint of the interplay between delay and redundancy the price for universality turns out to be acceptably small

proto value functions a laplacian framework for learning representation and control in markov decision processes this paper introduces a novel spectral framework for solving markov decision processes mdps by jointly learning representations and optimal policies the major components of the framework described in this paper include i a general scheme for constructing representations or basis functions by diagonalizing symmetric diffusion operators ii a specific instantiation of this approach where global basis functions called proto value functions pvfs are formed using the eigenvectors of the graph laplacian on an undirected graph formed from state transitions induced by the mdp iii a three phased procedure called representation policy iteration comprising of a sample collection phase a representation learning phase that constructs basis functions from samples and a final parameter estimation phase that determines an approximately optimal policy within the linear subspace spanned by the current basis functions iv a specific instantiation of the rpi framework using least squares policy iteration lspi as the parameter estimation method v several strategies for scaling the proposed approach to large discrete and continuous state spaces including the nystrom extension for out of sample interpolation of eigenfunctions and the use of kronecker sum factorization to construct compact eigenfunctions in product spaces such as factored mdps vi finally a series of illustrative discrete and continuous control tasks which both illustrate the concepts and provide a benchmark for evaluating the proposed approach many challenges remain to be addressed in scaling the proposed framework to large mdps and several elaboration of the proposed framework are briefly summarized at the end

optional polya tree and bayesian inference we introduce an extension of the polya tree approach for constructing distributions on the space of probability measures by using optional stopping and optional choice of splitting variables the construction gives rise to random measures that are absolutely continuous with piecewise smooth densities on partitions that can adapt to fit the data the resulting optional polya tree distribution has large support in total variation topology and yields posterior distributions that are also optional polya trees with computable parameter values

learning to optimize via posterior sampling this paper considers the use of a simple posterior sampling algorithm to balance between exploration and exploitation when learning to optimize actions such as in multiarmed bandit problems the algorithm also known as thompson sampling and as probability matching offers significant advantages over the popular upper confidence bound ucb approach and can be applied to problems with finite or infinite action spaces and complicated relationships among action rewards we make two theoretical contributions the first establishes a connection between posterior sampling and ucb algorithms this result lets us convert regret bounds developed for ucb algorithms into bayesian regret bounds for posterior sampling our second theoretical contribution is a bayesian regret bound for posterior sampling that applies broadly and can be specialized to many model classes this bound depends on a new notion we refer to as the eluder dimension which measures the degree of dependence among action rewards compared to ucb

cover tree bayesian reinforcement learning this paper proposes an online tree based bayesian approach for reinforcement learning for inference we employ a generalised context tree model this defines a distribution on multivariate gaussian piecewise linear models which can be updated in closed form the tree structure itself is constructed using the cover tree method which remains efficient in high dimensional spaces we combine the model with thompson sampling and approximate dynamic programming to obtain effective exploration policies in unknown environments the flexibility and computational simplicity of the model render it suitable for many reinforcement learning problems in continuous state spaces we demonstrate this in an experimental comparison with a gaussian process model a linear model and simple least squares policy iteration

on explicit conditions for the asymptotic stability of linear higher order difference equations we derive some explicit sufficient conditions for the asymptotic stability of the zero solution in a general linear higher order difference equation and compare our estimations with other related results in the literature our main result also applies to some nonlinear perturbations satisfying a kind of sublinearity condition

universal prediction this paper consists of an overview on universal prediction from an information theoretic perspective special attention is given to the notion of probability assignment under the self information loss function which is directly related to the theory of universal data compression both the probabilistic setting and the deterministic setting of the universal prediction problem are described with emphasis on the analogy and the differences between results in the two settings

an approach to fuzzy control of nonlinear systems stability and design issues presents a design methodology for stabilization of a class of nonlinear systems first the authors represent a nonlinear plant with a takagi sugeno fuzzy model then a model based fuzzy controller design utilizing the concept of the so called parallel distributed compensation is employed the main idea of the controller design is to derive each control rule so as to compensate each rule of a fuzzy system the design procedure is conceptually simple and natural moreover the stability analysis and control design problems can be reduced to linear matrix inequality lmi problems therefore they can be solved efficiently in practice by convex programming techniques for lmis the design methodology is illustrated by application to the problem of balancing and swing up of an inverted pendulum on a cart

learning the variance of the reward to go in markov decision processes mdps the variance of the reward to go is a natural measure of uncertainty about the long term performance of a policy and is important in domains such as finance resource allocation and process control currently however there is no tractable procedure for calculating it in large scale mdps this is in contrast to the case of the expected reward to go also known as the value function for which effective simulation based algorithms are known and have been used successfully in various domains in this paper we extend temporal difference td learning algorithms to estimating the variance of the reward to go for a fixed policy we propose variants of both td 0 and lstd with linear function approximation prove their convergence and demonstrate their utility in an option pricing problem our results show a dramatic improvement in terms of sample efficiency over standard monte carlo methods which are currently the state of the art

the nonstochastic multiarmed bandit problem in the multiarmed bandit problem a gambler must decide which arm of k nonidentical slot machines to play in a sequence of trials so as to maximize his reward this classical problem has received much attention because of the simple model it provides of the trade off between exploration trying out each arm to find the best one and exploitation playing the arm believed to give the best payoff past solutions for the bandit problem have almost always relied on assumptions about the statistics of the slot machines in this work we make no statistical assumptions whatsoever about the nature of the process generating the payoffs of the slot machines we give a solution to the bandit problem in which an adversary rather than a well behaved stochastic process has complete control over the payoffs in a sequence of t plays we prove that the per round payoff of our algorithm approaches that of the best arm at the rate o t 1 2 we show by a matching lower bound that this is the best possible we also prove that our algorithm approaches the per round payoff of any set of strategies at a similar rate if the best strategy is chosen from a pool of n strategies then our algorithm approaches the per round payoff of the strategy at the rate o log n1 2 t 1 2 finally we apply our results to the problem of playing an unknown repeated matrix game we show that our algorithm approaches the minimax payoff of the unknown game at the rate o t 1 2

averaging principle of sde with small diffusion moderate deviations consider the following stochastic differential equation in rd dxetxe0 b xet t e dt e a xet t e dwt x0 where the random environment t is an exponentially ergodic markov process independent of the wiener process wt with invariant probability measure and e is some small parameter in this paper we prove the moderate deviations for the averaging principle of xe that is deviations of xet around its limit averaging system x t given by d dx t b x t dt and x 0 x0 where b x e b x more precisely we obtain the large deviation estimation about et xet x te h e t 0 1 in the space of continuous trajectories as e decreases to 0 where h e is some deviation scale satisfying 1 h e e 1 2 our strategy will be first to show the exponential tightness and then the local moderate deviation principle which relies on some new method involving a conditional schilder s theorem and a moderate deviation principle for inhomogeneous integral functionals of markov processes previously established by the author

min max generalization for deterministic batch mode reinforcement learning relaxation schemes we study the min max optimization problem introduced in fonteneau et al towards min max reinforcement learning icaart 2010 springer heidelberg 2011 pp 61 77 for computing policies for batch mode reinforcement learning in a deterministic setting with fixed finite time horizon first we show that the min part of this problem is np hard we then provide two relaxation schemes the first relaxation scheme works by dropping some constraints in order to obtain a problem that is solvable in polynomial time the second relaxation scheme based on a lagrangian relaxation where all constraints are dualized can also be solved in polynomial time we also theoretically prove and empirically illustrate that both relaxation schemes provide better results than those given in fonteneau et al 2011 as cited above

universal piecewise linear prediction via context trees this paper considers the problem of piecewise linear prediction from a competitive algorithm approach in prior work prediction algorithms have been developed that are universal with respect to the class of all linear predictors such that they perform nearly as well in terms of total squared prediction error as the best linear predictor that is able to observe the entire sequence in advance in this paper we introduce the use of a context tree to compete against a doubly exponential number of piecewise linear affine models we use the context tree to achieve the total squared prediction error performance of the best piecewise linear model that can choose both its partitioning of the regressor space and its real valued prediction parameters within each region of the partition based on observing the entire sequence in advance uniformly for every bounded individual sequence this performance is achieved with a prediction algorithm whose complexity is only linear in the depth of the context tree per prediction upper bounds on the regret with respect to the best piecewise linear predictor are given for both the scalar and higher order case and lower bounds on the regret are given for the scalar case an explicit algorithmic description and examples demonstrating the performance of the algorithm are given

the context tree weighting method basic properties describes a sequential universal data compression procedure for binary tree sources that performs the double mixture using a context tree this method weights in an efficient recursive way the coding distributions corresponding to all bounded memory tree sources and achieves a desirable coding distribution for tree sources with an unknown model and unknown parameters computational and storage complexity of the proposed procedure are both linear in the source sequence length the authors derive a natural upper bound on the cumulative redundancy of the method for individual sequences the three terms in this bound can be identified as coding parameter and model redundancy the bound holds for all source sequence lengths not only for asymptotically large lengths the analysis that leads to this bound is based on standard techniques and turns out to be extremely simple the upper bound on the redundancy shows that the proposed context tree weighting procedure is optimal in the sense that it achieves the rissanen 1984 lower bound

universal reinforcement learning we consider an agent interacting with an unmodeled environment at each time the agent makes an observation takes an action and incurs a cost its actions can influence future observations and costs the goal is to minimize the long term average cost we propose a novel algorithm known as the active lz algorithm for optimal control based on ideas from the lempel ziv scheme for universal data compression and prediction we establish that under the active lz algorithm if there exists an integer k such that the future is conditionally independent of the past given a window of k consecutive actions and observations then the average cost converges to the optimum experimental results involving the game of rock paper scissors illustrate merits of the algorithm

large deviations in fast slow systems the incidence of rare events in fast slow systems is investigated via analysis of the large deviation principle ldp that characterizes the likelihood and pathway of large fluctuations of the slow variables away from their mean behavior such fluctuations are rare on short time scales but become ubiquitous eventually classical results prove that this ldp involves an hamilton jacobi equation whose hamiltonian is related to the leading eigenvalue of the generator of the fast process and is typically non quadratic in the momenta in other words the ldp for the slow variables in fast slow systems is different in general from that of any stochastic differential equation sde one would write for the slow variables alone it is shown here that the eigenvalue problem for the hamiltonian can be reduced to a simpler algebraic equation for this hamiltonian for a specific class of systems in which the fast variables satisfy a linear equation whose coefficients depend nonlinearly on the slow variables and the fast variables enter quadratically the equation for the slow variables these results are illustrated via examples inspired by kinetic theories of turbulent flows and plasma in which the quasipotential characterizing the long time behavior of the system is calculated and shown again to be different from that of an sde

kernel based least squares policy iteration for reinforcement learning in this paper we present a kernel based least squares policy iteration klspi algorithm for reinforcement learning rl in large or continuous state spaces which can be used to realize adaptive feedback control of uncertain dynamic systems by using klspi near optimal control policies can be obtained without much a priori knowledge on dynamic models of control plants in klspi mercer kernels are used in the policy evaluation of a policy iteration process where a new kernel based least squares temporal difference algorithm called klstd q is proposed for efficient policy evaluation to keep the sparsity and improve the generalization ability of klstd q solutions a kernel sparsification procedure based on approximate linear dependency ald is performed compared to the previous works on approximate rl methods klspi makes two progresses to eliminate the main difficulties of existing results one is the better convergence and near optimality guarantee by using the klstd q algorithm for policy evaluation with high precision the other is the automatic feature selection using the ald based kernel sparsification therefore the klspi algorithm provides a general rl method with generalization performance and convergence guarantee for large scale markov decision problems mdps experimental results on a typical rl task for a stochastic chain problem demonstrate that klspi can consistently achieve better learning efficiency and policy quality than the previous least squares policy iteration lspi algorithm furthermore the klspi method was also evaluated on two nonlinear feedback control problems including a ship heading control problem and the swing up control of a double link underactuated pendulum called acrobot simulation results illustrate that the proposed method can optimize controller performance using little a priori information of uncertain dynamic systems it is also demonstrated that klspi can be applied to online learning control by incorporating an initial controller to ensure online performance

constructing dynamic treatment regimes over indefinite time horizons summaryexisting methods for estimating optimal dynamic treatment regimes are limited to cases where a utility function is optimized over a fixed time period we develop an estimation procedure for the optimal dynamic treatment regime over an indefinite time period and derive associated large sample results the proposed method can be used to estimate the optimal dynamic treatment regime in chronic disease settings we illustrate this by simulating a dataset corresponding to a cohort of patients with diabetes that mimics the third wave of the national health and nutrition examination survey and examining the performance of the proposed method in controlling the level of haemoglobin a1c

qualitative properties of some higher order difference equations we present sufficient conditions which guarantee that all positive solutions of some higher order rational difference equations are global asymptotically stable the boundedness of the solutions and the existence of prime period two solutions of such equations are also investigated

regret and convergence bounds for a class of continuum armed bandit problems we consider a class of multi armed bandit problems where the set of available actions can be mapped to a convex compact region of ropfd sometimes denoted as the ldquocontinuum armed banditrdquo problem the paper establishes bounds on the efficiency of any arm selection procedure under certain conditions on the class of possible underlying reward functions both finite time lower bounds on the growth rate of the regret as well as asymptotic upper bounds on the rates of convergence of the selected control values to the optimum are derived we explicitly characterize the dependence of these convergence rates on the minimal rate of variation of the mean reward function in a neighborhood of the optimal control the bounds can be used to demonstrate the asymptotic optimality of the kiefer wolfowitz method of stochastic approximation with regard to a large class of possible mean reward functions

new explicit global asymptotic stability criteria for higher order difference equations new explicit sufficient conditions for the asymptotic stability of the zero solution of higher order difference equations are obtained these criteria can be applied to autonomous and nonautonomous equations the celebrated clark asymptotic stability criterion is improved also applications to models from mathematical biology and macroeconomics are given

x armed bandits we consider a generalization of stochastic bandits where the set of arms x is allowed to be a generic measurable space and the mean payoff function is locally lipschitz with respect to a dissimilarity function that is known to the decision maker under this condition we construct an arm selection policy called hoo hierarchical optimistic optimization with improved regret bounds compared to previous results for a large class of problems in particular our results imply that if x is the unit hypercube in a euclidean space and the mean payoff function has a finite number of global maxima around which the behavior of the function is locally continuous with a known smoothness degree then the expected regret of hoo is bounded up to a logarithmic factor by n that is the rate of growth of the regret is independent of the dimension of the space we also prove the minimax optimality of our algorithm when the dissimilarity is a metric our basic strategy has quadratic computational complexity as a function of the number of time steps and does not rely on the doubling trick we also introduce a modified strategy which relies on the doubling trick but runs in linearithmic time both results are improvements with respect to previous approaches

regret minimization for reserve prices in second price auctions we show a regret minimization algorithm for setting the reserve price in a sequence of second price auctions under the assumption that all bids are independently drawn from the same unknown and arbitrary distribution our algorithm is computationally efficient and achieves a regret of o t in a sequence of t auctions this holds even when the number of bidders is stochastic with a known distribution

normal bandits of unknown means and variances consider the problem of sampling sequentially from a finite number of n 2 populations specified by random variables xki i 1 n and k 1 2 where xki denotes the outcome from population i the kth time it is sampled it is assumed that for each fixed i xki k 1 is a sequence of i i d normal random variables with unknown mean i and unknown variance i2 the objective is to have a policy for deciding from which of the n populations to sample from at any time t 1 2 so as to maximize the expected sum of outcomes of n total samples or equivalently to minimize the regret due to lack on information of the parameters i and i2 in this paper we present a simple inflated sample mean ism index policy that is asymptotically optimal in the sense of theorem 4 below this resolves a standing open problem from burnetas and katehakis 1996b additionally finite horizon regret bounds are given

generalized polynomial approximations in markovian decision processes abstract fitting the value function in a markovian decision process by a linear superposition of m basis functions reduces the problem dimensionality from the number of states down to m with good accuracy retained if the value function is a smooth function of its argument the state vector this paper provides for both the discounted and undiscounted cases three algorithms for computing the coefficients in the linear superposition linear programming policy iteration and least squares

pure exploration in finitely armed and continuous armed bandits we consider the framework of stochastic multi armed bandit problems and study the possibilities and limitations of forecasters that perform an on line exploration of the arms these forecasters are assessed in terms of their simple regret a regret notion that captures the fact that exploration is only constrained by the number of available rounds not necessarily known in advance in contrast to the case when the cumulative regret is considered and when exploitation needs to be performed at the same time we believe that this performance criterion is suited to situations when the cost of pulling an arm is expressed in terms of resources rather than rewards we discuss the links between the simple and the cumulative regret one of the main results in the case of a finite number of arms is a general lower bound on the simple regret of a forecaster in terms of its cumulative regret the smaller the latter the larger the former keeping this result in mind we then exhibit upper bounds on the simple regret of some forecasters the paper ends with a study devoted to continuous armed bandit problems we show that the simple regret can be minimized with respect to a family of probability distributions if and only if the cumulative regret can be minimized for it based on this equivalence we are able to prove that the separable metric spaces are exactly the metric spaces on which these regrets can be minimized with respect to the family of all probability distributions with continuous mean payoff functions

robust approximate bilinear programming for value function approximation value function approximation methods have been successfully used in many applications but the prevailing techniques often lack useful a priori error bounds we propose a new approximate bilinear programming formulation of value function approximation which employs global optimization the formulation provides strong a priori guarantees on both robust and expected policy loss by minimizing specific norms of the bellman residual solving a bilinear program optimally is np hard but this worst case complexity is unavoidable because the bellman residual minimization itself is np hard we describe and analyze the formulation as well as a simple approximate algorithm for solving bilinear programs the analysis shows that this algorithm offers a convergent generalization of approximate policy iteration we also briefly analyze the behavior of bilinear programming algorithms under incomplete samples finally we demonstrate that the proposed approach can consistently minimize the bellman residual on simple benchmark problems

construction of approximation spaces for reinforcement learning linear reinforcement learning rl algorithms like least squares temporal difference learning lstd require basis functions that span approximation spaces of potential value functions this article investigates methods to construct these bases from samples we hypothesize that an ideal approximation spaces should encode diffusion distances and that slow feature analysis sfa constructs such spaces to validate our hypothesis we provide theoretical statements about the lstd value approximation error and induced metric of approximation spaces constructed by sfa and the state of the art methods krylov bases and proto value functions pvf in particular we prove that sfa minimizes the average over all tasks in the same environment bound on the above approximation error compared to other methods sfa is very sensitive to sampling and can sometimes fail to encode the whole state space we derive a novel importance sampling modification to compensate for this effect finally the lstd and least squares policy iteration lspi performance of approximation spaces constructed by krylov bases pvf sfa and pca is compared in benchmark tasks and a visual robot navigation experiment both in a realistic simulation and with a robot the results support our hypothesis and suggest that i sfa provides subspace invariant features for mdps with self adjoint transition operators which allows strong guarantees on the approximation error ii the modified sfa algorithm is best suited for lspi in both discrete and continuous state spaces and iii approximation spaces encoding diffusion distances facilitate lspi performance

approximate dynamic programming with a fuzzy parameterization dynamic programming dp is a powerful paradigm for general nonlinear optimal control computing exact dp solutions is in general only possible when the process states and the control actions take values in a small discrete set in practice it is necessary to approximate the solutions therefore we propose an algorithm for approximate dp that relies on a fuzzy partition of the state space and on a discretization of the action space this fuzzy q iteration algorithm works for deterministic processes under the discounted return criterion we prove that fuzzy q iteration asymptotically converges to a solution that lies within a bound of the optimal solution a bound on the suboptimality of the solution obtained in a finite number of iterations is also derived under continuity assumptions on the dynamics and on the reward function we show that fuzzy q iteration is consistent i e that it asymptotically obtains the optimal solution as the approximation accuracy increases these properties hold both when the parameters of the approximator are updated in a synchronous fashion and when they are updated asynchronously the asynchronous algorithm is proven to converge at least as fast as the synchronous one the performance of fuzzy q iteration is illustrated in a two link manipulator control problem

unbounded positive solutions of higher order difference equations with singular nonlinear term in this paper several necessary and or sufficient conditions are given for the existence of unbounded positive solutions of nonlinear difference equations of the form mxn 1 f n xn 0 nen0 where m 2 1 or 1 and f n0 r r is a real valued continuous function such that f n x 0 for all n x n0 0 where n0 0 1

on the structure of optimal real time encoders and decoders in noisy communication the output of a discrete time markov source must be encoded into a sequence of discrete variables the encoded sequence is transmitted through a noisy channel to a receiver that must attempt to reproduce reliably the source sequence encoding and decoding must be done in real time and the distortion measure does not tolerate delays the structure of real time encoding and decoding strategies that jointly minimize an average distortion measure over a finite horizon is determined the results are extended to the real time broadcast problem and a real time variation of the wyner ziv problem

ranked bandits in metric spaces learning diverse rankings over large document collections most learning to rank research has assumed that the utility of different documents is independent which results in learned ranking functions that return redundant results the few approaches that avoid this have rather unsatisfyingly lacked theoretical foundations or do not scale we present a learning to rank formulation that optimizes the fraction of satisfied users with several scalable algorithms that explicitly takes document similarity and ranking context into account our formulation is a non trivial common generalization of two multi armed bandit models from the literature ranked bandits radlinski et al 2008 and lipschitz bandits kleinberg et al 2008b we present theoretical justifications for this approach as well as a near optimal algorithm our evaluation adds optimizations that improve empirical performance and shows that our algorithms learn orders of magnitude more quickly than previous approaches

least squares policy iteration we propose a new approach to reinforcement learning for control problems which combines value function approximation with linear architectures and approximate policy iteration this new approach is motivated by the least squares temporal difference learning algorithm lstd for prediction problems which is known for its efficient use of sample experiences compared to pure temporal difference algorithms heretofore lstd has not had a straightforward application to control problems mainly because lstd learns the state value function of a fixed policy which cannot be used for action selection and control without a model of the underlying process our new algorithm least squares policy iteration lspi learns the state action value function which allows for action selection without a model and for incremental policy improvement within a policy iteration framework lspi is a model free off policy method which can use efficiently and reuse in each iteration sample experiences collected in any manner by separating the sample collection method the choice of the linear approximation architecture and the solution method lspi allows for focused attention on the distinct elements that contribute to practical reinforcement learning lspi is tested on the simple task of balancing an inverted pendulum and the harder task of balancing and riding a bicycle to a target location in both cases lspi learns to control the pendulum or the bicycle by merely observing a relatively small number of trials where actions are selected randomly lspi is also compared against q learning both with and without experience replay using the same value function architecture while lspi achieves good performance fairly consistently on the difficult bicycle task q learning variants were rarely able to balance for more than a small fraction of the time needed to reach the target location

on the dynamics of the recursive sequence xn 1 xn 1β γxn 22xn 4 γxn 2xn 42 in this paper we investigate the global behavior of the difference equation x n 1 x n email protected email protected n 2 2x n email protected n 2x n 4 2 n 0 1 with positive parameters and non negative initial conditions

universal prediction of individual sequences the problem of predicting the next outcome of an individual binary sequence using finite memory is considered the finite state predictability of an infinite sequence is defined as the minimum fraction of prediction errors that can be made by any finite state fs predictor it is proven that this fs predictability can be achieved by universal sequential prediction schemes an efficient prediction procedure based on the incremental parsing procedure of the lempel ziv data compression algorithm is shown to achieve asymptotically the fs predictability some relations between compressibility and predictability are discussed and the predictability is proposed as an additional measure of the complexity of a sequence