queue aware distributive resource control for delay sensitive two hop mimo cooperative systems in this paper we consider a queue aware distributive resource control algorithm for two hop mimo cooperative systems we shall illustrate that relay buffering is an effective way to reduce the intrinsic half duplex penalty in cooperative systems the complex interactions of the queues at the source node and the relays are modeled as an average cost infinite horizon markov decision process mdp the traditional approach solving this mdp problem involves centralized control with huge complexity to obtain a distributive and low complexity solution we introduce a linear structure which approximates the value function of the associated bellman equation by the sum of per node value functions we derive a distributive two stage two winner auction based control policy which is a function of the local csi and local qsi only furthermore to estimate the best fit approximation parameter we propose a distributive online stochastic learning algorithm using stochastic approximation theory finally we establish technical conditions for almost sure convergence and show that under heavy traffic the proposed low complexity distributive control is global optimal

transcendence and drinfeld modules it has been ten years since v g drinfeld introduced in 3 the concept of elliptic a module which we now call drinfeld a module these modules are analogues of elliptic curves for global function field k and we have witnessed a beautiful algebraic theory gradually taking shape in 13 we have asked a few natural t ranscendence questions conjectures concerning these modules having in mind the analogue of the classical work of siegel schneider about elliptic integrals and based on evidences provided by the old papers of carlitz and wade cf 1 11 here in this article we will prove those conjectures we will show that indeed there is an interesting transcendence theory for each algebraic curve over a finite field fol lowing drinfeld we let m be a lattice in i e a finitely generated discrete a module in k and form

the multi armed bandit problem with covariates we consider a multi armed bandit problem in a setting where each arm produces a noisy reward realization which depends on an observable random covariate as opposed to the traditional static multi armed bandit problem this setting allows for dynamically changing rewards that better describe applications where side information is available we adopt a nonparametric model where the expected rewards are smooth functions of the covariate and where the hardness of the problem is captured by a margin parameter to maximize the expected cumulative reward we introduce a policy called adaptively binned successive elimination abse that adaptively decomposes the global problem into suitably localized static bandit problems this policy constructs an adaptive partition using a variant of the successive elimination se policy our results include sharper regret bounds for the se policy in a static bandit problem and minimax optimal regret bounds for the abse policy in the dynamic problem

the sample complexity of exploration in the multi armed bandit problem we consider the multi armed bandit problem under the pac probably approximately correct model it was shown by even dar et al 2002 that given n arms a total of o n e2 log 1 trials suffices in order to find an e optimal arm with probability at least 1 we establish a matching lower bound on the expected number of trials under any sampling policy we furthermore generalize the lower bound and show an explicit dependence on the unknown statistics of the arms we also provide a similar bound within a bayesian setting the case where the statistics of the arms are known but the identities of the arms are not is also discussed for this case we provide a lower bound of 1 e2 n log 1 on the expected number of trials as well as a sampling policy with a matching upper bound if instead of the expected number of trials we consider the maximum over all sample paths number of trials we establish a matching upper and lower bound of the form n e2 log 1 finally we derive lower bounds on the expected regret in the spirit of lai and robbins

learning algorithms for markov decision processes with average cost this paper gives the first rigorous convergence analysis of analogues of watkins s q learning algorithm applied to average cost control of finite state markov chains we discuss two algorithms which may be viewed as stochastic approximation counterparts of two existing algorithms for recursively computing the value function of the average cost problem the traditional relative value iteration rvi algorithm and a recent algorithm of bertsekas based on the stochastic shortest path ssp formulation of the problem both synchronous and asynchronous implementations are considered and analyzed using the ode method this involves establishing asymptotic stability of associated ode limits the ssp algorithm also uses ideas from two time scale stochastic approximation

actor critic type learning algorithms for markov decision processes algorithms for learning the optimal policy of a markov decision process mdp based on simulated transitions are formulated and analyzed these are variants of the well known actor critic or adaptive critic algorithm in the artificial intelligence literature distributed asynchronous implementations are considered the analysis involves two time scale stochastic approximations

asynchronous stochastic approximations the asymptotic behavior of a distributed asynchronous stochastic approximation scheme is analyzed in terms of a limiting nonautonomous differential equation the relation between the latter and the relative values of suitably rescaled relative frequencies of updates of different components is underscored

stochastic approximation for nonexpansive maps application to q learning algorithms we discuss synchronous and asynchronous iterations of the form xk 1 x k gamma k h x k w k where h is a suitable map and wk is a deterministic or stochastic sequence satisfying suitable conditions in particular in the stochastic case these are stochastic approximation iterations that can be analyzed using the ode approach based either on kushner and clark s lemma for the synchronous case or on borkar s theorem for the asynchronous case however the analysis requires that the iterates xk be bounded a fact which is usually hard to prove we develop a novel framework for proving boundedness in the deterministic framework which is also applicable to the stochastic case when the deterministic hypotheses can be verified in the almost sure sense this is based on scaling ideas and on the properties of lyapunov functions we then combine the boundedness property with borkar s stability analysis of odes involving nonexpansive mappings to prove convergence with probability 1 in the stochastic case we also apply our convergence analysis to q learning algorithms for stochastic shortest path problems and are able to relax some of the assumptions of the currently available results

performance guarantees for empirical markov decision processes with applications to multiperiod inventory models we consider markov decision processes with unknown transition probabilities and unknown single period expected cost functions and we study a method for estimating these quantities from historical or simulated data the method requires knowledge of the system equations that govern state transitions as well as the single period cost functions but not the single period expected cost functions the estimation procedure is based upon taking expectations with respect to the empirical distribution functions of such data once the estimates are in place the method computes a policy by solving the obtained empirical markov decision process as if the estimates were correct for mdps that satisfy some conditions we provide explicit easily computed expressions for the probability that the procedure will produce a policy whose true expected cost is within any specified absolute distance of the actual optimal expected cost of the true markov decision process we also provide expressions for the number of historical or simulated data values that is sufficient for the procedure to produce a policy whose true expected cost is with a prescribed probability within a prescribed absolute distance of the actual optimal expected cost of the true markov decision process we apply our results to multiperiod inventory models in addition we provide a specialized analysis of such inventory models that also yields relative rather than absolute accuracy guarantees we make comparisons with related results that have recently appeared and we provide numerical examples

de finetti s coherence and statistical inference on donne la definition d une inference coherente en conformite avec une theorie des probabilites conditionnelles derivee du principe de coherence de de finetti et on effectue une comparaison critique d une telle definition avec celle proposee par heath lane et sudderth

the nonstochastic multiarmed bandit problem in the multiarmed bandit problem a gambler must decide which arm of k nonidentical slot machines to play in a sequence of trials so as to maximize his reward this classical problem has received much attention because of the simple model it provides of the trade off between exploration trying out each arm to find the best one and exploitation playing the arm believed to give the best payoff past solutions for the bandit problem have almost always relied on assumptions about the statistics of the slot machines in this work we make no statistical assumptions whatsoever about the nature of the process generating the payoffs of the slot machines we give a solution to the bandit problem in which an adversary rather than a well behaved stochastic process has complete control over the payoffs in a sequence of t plays we prove that the per round payoff of our algorithm approaches that of the best arm at the rate o t 1 2 we show by a matching lower bound that this is the best possible we also prove that our algorithm approaches the per round payoff of any set of strategies at a similar rate if the best strategy is chosen from a pool of n strategies then our algorithm approaches the per round payoff of the strategy at the rate o log n1 2 t 1 2 finally we apply our results to the problem of playing an unknown repeated matrix game we show that our algorithm approaches the minimax payoff of the unknown game at the rate o t 1 2

a mixed value and policy iteration method for stochastic control with universally measurable policies we consider stochastic optimal control models with borel spaces and universally measurable policies for such models the standard policy iteration is known to have difficult measurability issues and cannot be carried out in general we present a mixed value and policy iteration method that circumvents this difficulty the method allows the use of stationary policies in computing the optimal cost function in a manner that resembles policy iteration it can also be used to address similar difficulties of policy iteration in the context of upper and lower semicontinuous models we analyze the convergence of the method in infinite horizon total cost problems for the discounted case where the one stage costs are bounded and for the undiscounted case where the one stage costs are nonpositive or nonnegative for undiscounted total cost problems with nonnegative one stage costs we also give a new convergence theorem for value iteration that shows that value iteration converges whenever it is initialized with a f

empirical dynamic programming we propose empirical dynamic programming algorithms for markov decision processes in these algorithms the exact expectation in the bellman operator in classical value iteration is replaced by an empirical estimate to get empirical value iteration evi policy evaluation and policy improvement in classical policy iteration are also replaced by simulation to get empirical policy iteration epi thus these empirical dynamic programming algorithms involve iteration of a random operator the empirical bellman operator we introduce notions of probabilistic fixed points for such random monotone operators we develop a stochastic dominance framework for convergence analysis of such operators we then use this to give sample complexity bounds for both evi and epi we then provide various variations and extensions to asynchronous empirical dynamic programming the minimax empirical dynamic program and show how this can also be used to solve the dynamic newsvendor problem preliminary experimental results

a learning algorithm for risk sensitive cost a linear function approximation based reinforcement learning algorithm is proposed for markov decision processes with infinite horizon risk sensitive cost its convergence is proved using the o d e method for stochastic approximation the scheme is also extended to continuous state space processes

action elimination and stopping conditions for the multi armed bandit and reinforcement learning problems we incorporate statistical confidence intervals in both the multi armed bandit and the reinforcement learning problems in the bandit problem we show that given n arms it suffices to pull the arms a total of o n e2 log 1 times to find an e optimal arm with probability of at least 1 this bound matches the lower bound of mannor and tsitsiklis 2004 up to constants we also devise action elimination procedures in reinforcement learning algorithms we describe a framework that is based on learning the confidence interval around the value function or the q function and eliminating actions that are not optimal with high probability we provide a model based and a model free variants of the elimination method we further derive stopping conditions guaranteeing that the learned policy is approximately optimal with high probability simulations demonstrate a considerable speedup and added robustness over e greedy q learning

the o d e method for convergence of stochastic approximation and reinforcement learning it is shown here that stability of the stochastic approximation algorithm is implied by the asymptotic stability of the origin for an associated ode this in turn implies convergence of the algorithm several specific classes of algorithms are considered as applications it is found that the results provide i a simpler derivation of known results for reinforcement learning algorithms ii a proof for the first time that a class of asynchronous stochastic approximation algorithms are convergent without using any a priori assumption of stability iii a proof for the first time that asynchronous adaptive critic and q learning algorithms are convergent for the average cost optimal control problem

some remarks on the use of improper priors for the analysis of exponential regression models summary we consider bayesian inference on the exponential regression model it is known that standard improper priors on the parameters of this model lead to marginal posterior distributions on one of the parameters involved which are also improper on the interval 0 1 this has sometimes been interpreted as meaning that the observations are irrelevant we re analyse this problem using a finitely additive approach and show that the above conclusion is not generally correct this result emphasizes once again the dangers of a routine approach to bayesian inference using improper priors

noninformative priors for inferences in exponential regression models summary in the exponential regression model inference concerning the regression parameter is notoriously difficult even when using the bayesian noninformative prior approach the reference prior approach bernardo 1979 berger bernardo 1989 is considered and argued to yield very satisfactory inferences estimation and credible sets are considered in a specific example