asymptotically efficient adaptive allocation schemes for controlled markov chains finite parameter space the authors consider a controlled markov chain whose transition probabilities and initial distribution are parametrized by an unknown parameter theta belonging to some known parameter space theta there is a one step reward associated with each pair of control and the following state of the process the objective is to maximize the expected value of the sum of one step rewards over an infinite horizon the loss associated with a control scheme at a parameter value is the function of time giving the difference between the maximum reward that could have been achieved if the parameter were known and the reward achieved by the scheme since it is impossible to minimize the loss uniformly for all parameter values the authors define uniformly good adaptive control schemes and restrict attention to these schemes they develop a lower bound on the loss associated with any uniformly good control scheme they construct an adaptive control scheme whose loss equals the lower bound for every parameter value and is therefore asymptotically efficient

the numerical analysis of ordinary differential equations runge kutta and general linear methods mathematical and computational introduction the euler method and its generalizations analysis of runge kutta methods general linear methods bibliography

restless bandit marginal productivity indices diminishing returns and optimal control of make to order make to stock m g 1 queues this paper presents a framework grounded on convex optimization and economics ideas to solve by index policies problems of optimal dynamic allocation of effort to a discrete state finite or countable binary action work rest semi markov restless bandit project elucidating issues raised by previous work its contributions include i the concept of a restless bandits marginal productivity index mpi characterizing optimal policies relative to general cost and work measures ii the characterization of indexable restless bandits as those satisfying diminishing marginal returns to work consistently with a nested family of threshold policies iii sufficient indexability conditions via partial conservation laws pcls iv the characterization of the mpi as an optimal marginal productivity rate relative to feasible active state sets v application to semi markov bandits under several criteria including a new mixed average bias criterion and vi pcl indexability analyses and mpis for optimal service control of make to order make to stock queues with convex holding costs under discounted and average bias criteria

asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays part ii markovian rewards at each instant of time we are required to sample a fixed number m geq 1 out of n i i d processes whose distributions belong to a family suitably parameterized by a real number theta the objective is to maximize the long run total expected value of the samples following lai and robbins the learning loss of a sampling scheme corresponding to a configuration of parameters c theta 1 theta n is quantified by the regret r n c this is the difference between the maximum expected reward at time n that could be achieved if c were known and the expected reward actually obtained by the sampling scheme we provide a lower bound for the regret associated with any uniformly good scheme and construct a scheme which attains the lower bound for every configuration c the lower bound is given explicitly in terms of the kullback liebler number between pairs of distributions part ii of this paper considers the same problem when the reward processes are markovian

the knowledge gradient algorithm for a general class of online learning problems we derive a one period look ahead policy for finite and infinite horizon online optimal learning problems with gaussian rewards our approach is able to handle the case where our prior beliefs about the rewards are correlated which is not handled by traditional multiarmed bandit methods experiments show that our kg policy performs competitively against the best known approximation to the optimal policy in the classic bandit problem and it outperforms many learning policies in the correlated case

submodular stochastic probing on matroids in a stochastic probing problem we are given a universe e and a probability that each element e in e is active we determine if an element is active by probing it and whenever a probed element is active we must permanently include it in our solution moreover throughout the process we need to obey inner constraints on the set of elements taken into the solution and outer constraints on the set of all probed elements all previous algorithmic results in this framework have considered only the problem of maximizing a linear function of the active elements here we consider submodular objectives we provide new constant factor approximations for maximizing a monotone submodular function subject to multiple matroid constraints on both the elements that may be taken and the elements that may be probed we also obtain an improved approximation for linear objective functions and show how our approach may be generalized to handle k matchoid constraints

machine learning and nonparametric bandit theory in its most basic form bandit theory is concerned with the design problem of sequentially choosing members from a given collection of random variables so that the regret i e r sub n spl sigma sub j spl mu spl mu sub j et sub n j grows as slowly as possible with increasing n here spl mu sub j is the expected value of the bandit arm i e random variable indexed by j t sub n j is the number of times arm j has been selected in the first n decision stages and spl mu sup sup sub j spl mu sub j the present paper contributes to the theory by considering the situation in which observations are dependent to begin with the dependency is presumed to depend only on past observations of the same arm but later we allow that it may be with respect to the entire past and that the set of arms is infinite this brings queues and more generally controlled markov processes into our purview thus our black box methodology is suitable for the case when the only observables are cost values and in particular the probability structure and loss function are unknown to the designer the conclusion of the analysis is that under lenient conditions using algorithms prescribed herein risk growth is commensurate with that in the simplest i i d cases our methods represent an alternative to stochastic approximation perturbation analysis ideas for tuning queues

adaptivity in the stochastic blackjack knapsack problem we consider a stochastic variant of the np hard 0 1 knapsack problem in which item values are deterministic and item sizes are independent random variables with known arbitrary distributions items are placed in the knapsack sequentially and the act of placing an item in the knapsack instantiates its size in every stage of insertion if the subset of the items inserted thus far is feasible then it has a total value that equals to the sum of values of all items of this subset otherwise if the subset violates the constraint then its value equals to zero the goal is to compute a policy for insertion of the items that maximizes the expected total value of items placed in the knapsack we consider both non adaptive policies that designate a priori a fixed subset of items to insert and adaptive policies that can make dynamic decisions based on the instantiated sizes of items placed in the knapsack thus far our work characterizes the benefit of adaptivity for this purpose we use a measure called the adaptivity gap the supremum over instances of the ratio between the expected value obtained by an optimal adaptive policy and the expected value obtained by an optimal non adaptive policy first we show a tight bound of 32 on the adaptivity gap for the case of inputs consisting of only two items then we present a non adaptive policy with expected value that is at least 2 1 2 2 1 11 66 times the expected value of the optimal adaptive policy thus the adaptivity gap in this model is at most 11 66 additionally this non adaptive policy is computed in polynomial time finally we consider a special case of the model where all sizes are distributed according to bernoulli distribution with different parameters for this special case we improve our result and bound the adaptivity gap by 8

resource constrained management of heterogeneous assets with stochastic deterioration we consider a collection of heterogeneous assets which exhibit independent stochastic behavior but are economically interdependent due to resource constraints on management decisions we represent the collection of assets as a network of nonhomogeneous markov decision processes linked by side constraints to facilitate a procedure for obtaining nonrandomized decision policies we express the resource limitations as integrated chance constraints and relax them into the objective function within a lagrangian penalty term we utilize subgradient optimization to establish upper bounds and a greedy randomized lagrangian repair heuristic to obtain feasible solutions we empirically validate the tightness of these a posteriori upper and lower bounds with computational experiments on a pair of applications for pavement maintenance we examine the effect that the reward structure of each constituent mdp has on the maintenance policy in the presence of budget constraints for equipment replacement we consider constraints on two resource types

positive and conservative schemes for mass action kinetics abstract some implicit schemes for the discretization of mass action kinetics are presented and discussed although all the schemes are classical the solution procedure takes great advantage of the peculiar structure of the ode all the schemes maintain linear first integrals such as atomic mass conservation the main features of the first order scheme is positivity preservation for arbitrarily large time steps while for the higher order schemes there is numerical evidence that this property is maintained with fairly large time steps for the first order scheme the existence of a nonnegative solution of the algebraic system arising in the advancing step is proved for arbitrarily large time steps while uniqueness is proved for limited time steps finally an efficient procedure is presented for solving the nonlinear systems involved in the time step these systems are solved without the use of jacobian matrices or their approximations but by the repeated inversion of m matrices a procedure which is both easy and fast considerably simplifying and accelerating the computer implementation of the schemes

regret bounds for restless markov bandits we consider the restless markov bandit problem in which the state of each arm evolves according to a markov process independently of the learner s actions we suggest an algorithm that first represents the setting as an mdp which exhibits some special structural properties in order to grasp this information we introduce the notion of e structured mdps which are a generalization of concepts like approximate state aggregation and mdp homomorphisms we propose a general algorithm for learning e structured mdps and show regret bounds that demonstrate that additional structural information enhances learning applied to the restless bandit setting this algorithm achieves after any t steps regret of order o t with respect to the best policy that knows the distributions of all arms we make no assumptions on the markov chains underlying each arm except that they are irreducible in addition we show that index based policies are necessarily suboptimal for the considered problem

r extreme signalling for congestion control abstractin many smart city applications congestion arises in part due to the nature of signals received by individuals from a central authority in the model of marecek et al each agent uses one out of multiple resources at each time instant the per use cost of a resource depends on the number of concurrent users a central authority has up to date knowledge of the congestion across all resources and uses randomisation to provide a scalar or an interval for each resource at each time in this paper the interval to broadcast per resource is obtained by taking the minima and maxima of costs observed within a time window of length r rather than by randomisation we show that the resulting distribution of agents across resources also converges in distribution under plausible assumptions about the evolution of the population over time

the irrevocable multiarmed bandit problem this paper considers the multiarmed bandit problem with multiple simultaneous arm pulls and the additional restriction that we do not allow recourse to arms that were pulled at some point in the past but then discarded this additional restriction is highly desirable from an operational perspective and we refer to this problem as the irrevocable multiarmed bandit problem we observe that natural modifications to well known heuristics for multiarmed bandit problems that satisfy this irrevocability constraint have unsatisfactory performance and thus motivated introduce a new heuristic the packing heuristic we establish through numerical experiments that the packing heuristic offers excellent performance even relative to heuristics that are not constrained to be irrevocable we also provide a theoretical analysis that studies the price of irrevocability i e the performance loss incurred in imposing the constraint we propose on the multiarmed bandit model we show that this performance loss is uniformly bounded for a general class of multiarmed bandit problems and indicate its dependence on various problem parameters finally we obtain a computationally fast algorithm to implement the packing heuristic the algorithm renders the packing heuristic computationally cheaper than methods that rely on the computation of gittins indices

multiclass queueing systems polymatroidal structure and optimal scheduling control in many multiclass queueing systems certain performance measures of interest satisfy strong conservation laws that is the total performance over all job types is invariant under any nonidling service control rule and the total performance over any subset say a of job types is minimized or maximized by offering absolute priority to the types in a over all other types we develop a formal definition of strong conservation laws and show that as a necessary consequence of these strong conservation laws the state space of the performance vector is a base of a polymatroid from known results in polymatroidal theory the vertices of this polyhedron are easily identified and these vertices correspond to absolute priority rules a wide variety of multiclass queueing systems are shown to have this polymatroidal structure which greatly facilitates the study of the optimal scheduling control of such systems when the defining set function of the performance space belongs to the class of generalized symmet

adaptive treatment allocation and the multi armed bandit problem a class of simple adaptive allocation rules is proposed for the problem often called the multi armed bandit problem of sampling x 1 cdots x n sequentially from k populations with densities belonging to an exponential family in order to maximize the expected value of the sum s n x 1 cdots x n these allocation rules are based on certain upper confidence bounds which are developed from boundary crossing theory for the k population parameters the rules are shown to be asymptotically optimal as n rightarrow infty from both bayesian and frequentist points of view monte carlo studies show that they also perform very well for moderate values of the horizon n

asymptotically efficient strategies for a stochastic scheduling problem with order constraints motivated by an application in computerized adaptive tests we consider the following sequential design problem there are j jobs to be processed according to a predetermined order a single machine is available to process these j jobs each job under processing evolves stochastically as a markov chain and earns rewards as it is processed not otherwise the markov chain has transition probabilities parameterized by an unknown parameter the objective is to determine how long each job should be processed so that the total expected rewards over an extended time interval is maximized we define the regret associated with a strategy as the shortfall from the maximum expected reward under complete information on therefore the problem is equivalent to minimizing the regret the asymptotic lower bound for the regret associated with any uniformly good strategy is characterized by a deterministic constraint minimization problem in ignorance of the parameter value we construct a class of efficient strategies which achieve the lower bound based on the theory of sequential testing

minimizing the bias and variance of the gradient estimate in rsm simulation studies abstract to minimize the variance of the response estimator in response surface methodology rsm based simulations most users extend the design points to the limits of the subregion under investigation our analysis however shows that when common pseudorandom numbers are used this practice likely increases the variance of the estimated gradient under the criterion of the variance of the gradient and the mahalanobis distance function we show that smaller subregions reduce both the bias and the variance of the estimated gradient furthermore we discuss how users may take advantage of the magnitude of the induced correlations to draw inferences about the search process and to manipulate the subregion size our analysis offers a new direction for future research

decomposable markov decision processes a fluid optimization approach decomposable markov decision processes mdps are problems where the stochastic system can be decomposed into multiple individual components although such mdps arise naturally in many practical applications they are often difficult to solve exactly due to the enormous size of the state space of the complete system which grows exponentially with the number of components in this paper we propose an approximate solution approach to decomposable mdps that is based on re solving a fluid linear optimization formulation of the problem at each decision epoch this formulation tractably approximates the problem by modeling transition behavior at the level of the individual components rather than the complete system we prove that our fluid formulation provides a tighter bound on the optimal value function than three state of the art formulations the approximate linear optimization formulation the classical lagrangian relaxation formulation and a novel alternate lagrangian relaxation that is based on relax

the complexity of optimal queuing network control we show that several well known optimization problems related to the optimal control of queues are provably intractable independently of any unproven conjecture such as p np in particular we show that several versions of the problem of optimally controlling a simple network of queues with simple arrival and service distributions and multiple customer classes is complete for exponential time this is perhaps the first such intractability result for a well known optimization problem we also show that the restless bandit problem the generalization of the multi armed bandit problem to the case in which the unselected processes are not quiescent is complete for polynomial space

batched bandit problems motivated by practical applications chiefly clinical trials we study the regret achievable for stochastic bandits under the constraint that the employed policy must split trials into a small number of batches we propose a simple policy and show that a very small number of batches gives close to minimax optimal regret bounds as a byproduct we derive optimal policies with low switching cost for stochastic bandits

approximation algorithms for optimal decision trees and adaptive tsp problems we consider the problem of constructing optimal decision trees given a collection of tests that can disambiguate between a set of m possible diseases each test having a cost and the a priori likelihood of any particular disease what is a good adaptive strategy to perform these tests to minimize the expected cost to identify the disease this problem has been studied in several works with o log m approximations known in the special cases when either costs or probabilities are uniform in this paper we settle the approximability of the general problem by giving a tight o log m approximation algorithm we also consider a substantial generalization the adaptive traveling salesman problem given an underlying metric space a random subset s of vertices is drawn from a known distribution but s is initially unknown we get information about whether any vertex is in s only when it is visited what is a good adaptive strategy to visit all vertices in the random subset s while minimizing the expected dista

approximation algorithms for stochastic inventory control models we consider two classical stochastic inventory control models the periodic review stochastic inventory control problem and the stochastic lot sizing problem the goal is to coordinate a sequence of orders of a single commodity aiming to supply stochastic demands over a discrete finite horizon with minimum expected overall ordering holding and backlogging costs in this paper we address the important problem of finding computationally efficient and provably good inventory control policies for these models in the presence of correlated nonstationary time dependent and evolving stochastic demands this problem arises in many domains and has many practical applications in supply chain management our approach is based on a new marginal cost accounting scheme for stochastic inventory control models combined with novel cost balancing techniques specifically in each period we balance the expected cost of overordering i e costs incurred by excess inventory against the expected cost of underordering i e costs incurred by not satisfying demand on time this leads to what we believe to be the first computationally efficient policies with constant worst case performance guarantees for a general class of important stochastic inventory control models that is there exists a constant c such that for any instance of the problem the expected cost of the policy is at most c times the expected cost of an optimal policy in particular we provide a worst case guarantee of two for the periodic review stochastic inventory control problem and a worst case guarantee of three for the stochastic lot sizing problem our results are valid for all of the currently known approaches in the literature to model correlation and nonstationarity of demands over time

characterization and optimization of achievable performance in general queueing systems this paper considers general single facility queueing systems with exponential service times dealing with a finite number j of distinct customer classes performance of the system as measured by the vector of steady state expected sojourn times of the customer classes the performance vector may be controlled by adopting an appropriate preemptive priority discipline we show that the performance space the set of performance vectors which are achievable under some preemptive work conserving rule is a polyhedron described by 2j 1 inequalities the special structure of this polyhedron nevertheless allows for efficient procedures to minimize any separable convex function of the performance vector linear objectives are shown to be minimized by absolute priority rules thus generalizing a well known result for m m 1 systems we also show that each point in the performance space may be achieved by a specific randomization of at most j 1 absolute priority rules

relaxations of weakly coupled stochastic dynamic programs we consider a broad class of stochastic dynamic programming problems that are amenable to relaxation via decomposition these problems comprise multiple subproblems that are independent of each other except for a collection of coupling constraints on the action space we fit an additively separable value function approximation using two techniques namely lagrangian relaxation and the linear programming lp approach to approximate dynamic programming we prove various results comparing the relaxations to each other and to the optimal problem value we also provide a column generation algorithm for solving the lp based relaxation to any desired optimality tolerance and we report on numerical experiments on bandit like problems our results provide insight into the complexity versus quality trade off when choosing which of these relaxations to implement

optimal strategies for a class of sequential control problems with precedence relations consider the following multi phase project management problem each project is divided into several phases all projects enter the next phase at the same point chosen by the decision maker based on observations up to that point within each phase one can pursue the projects in any order when pursuing the project with one unit of resource the project state changes according to a markov chain the probability distribution of the markov chain is known up to an unknown parameter when pursued the project generates a random reward depending on the phase and the state of the project and the unknown parameter the decision maker faces two problems a how to allocate resources to projects within each phase and b when to enter the next phase so that the total expected reward is as large as possible in this paper we formulate the preceding problem as a stochastic scheduling problem and propose asymptotic optimal strategies which minimize the shortfall from perfect information payoff concrete examples are given to illustrate our method

on the computation of the reachable observable canonical form this note is concerned with questions related to the definition of canonical form for linear systems some misunderstandings in a previous note by boley in this journal siam j control optim 18 1980 pp 624 626 are clarified and some new research problems are suggested

simulation designs for quadratic response surface models in the presence of model misspecification this article considers the selection of experimental designs for the estimation of second order response surface metamodels in a simulation environment rather than construct designs based on the premise that the postulated model exactly represents the simulated response as is the case in optimal design theory we assume that the estimation process may be biased by the presence of unfitted third order terms we therefore seek to specify experimental plans that address the bias due to possible model misspecification as well as traditional variance considerations the performance measure used for this fit protect scenario is box and draper s design criterion of average mean squared error of predicted response four important classes of response surface experimental plans are examined 1 central composite designs 2 box behnken plans 3 three level factorial experiments and 4 small central composite designs each design class is studied in conjunction with three pseudorandom number assignment str

approximating the stochastic knapsack problem the benefit of adaptivity we consider a stochastic variant of the np hard 0 1 knapsack problem in which item values are deterministic and item sizes are independent random variables with known arbitrary distributions items are placed in the knapsack sequentially and the act of placing an item in the knapsack instantiates its size our goal is to compute a solution policy that maximizes the expected value of items successfully placed in the knapsack where the final overflowing item contributes no value we consider both nonadaptive policies that designate a priori a fixed sequence of items to insert and adaptive policies that can make dynamic choices based on the instantiated sizes of items placed in the knapsack thus far an important facet of our work lies in characterizing the benefit of adaptivity for this purpose we advocate the use of a measure called the adaptivity gap the ratio of the expected value obtained by an optimal adaptive policy to that obtained by an optimal nonadaptive policy we bound the adaptivity

regression metamodels for simulation with common random numbers comparison of validation tests and confidence intervals linear regression analysis is important in many fields in the analysis of simulation results a regression metamodel can be applied even when common pseudorandom numbers are used to test the validity of the specified regression model rao 1959 generalized the f statistic for lack of fit whereas kleijnen 1983 proposed a cross validation procedure using a student s t statistic combined with bonferroni s inequality this paper reports on an extensive monte carlo experiment designed to compare these two methods under the normality assumption cross validation is conservative whereas rao s test realizes its nominal type i error and has high power robustness is investigated through lognormal and uniform distributions when simulation responses are distributed lognormally then cross validation using ordinary least squares is the only technique that has acceptable type i error uniform distributions give results similar to the normal case once the regression model is validated confidence intervals for the individual regression parameters are computed the monte carlo experiment compares several confidence interval procedures under normality rao s procedure is preferred since it has good coverage probability and acceptable half length under lognormality ordinary least squares achieves nominal coverage probability uniform distributions again give results similar to the normal case

stochastic scheduling with release dates and due dates we consider stochastic scheduling problems in which the processing times of jobs are independent exponentially distributed random variables the release dates are random variables with an arbitrary joint distribution and the due dates are random variables with a joint distribution that satisfies certain conditions our development establishes simple policies that minimize such criteria as the expected weighted sum of completion times and the expected weighted number of late jobs these results contrast markedly with the deterministic counterparts of these models for which no polynomial time algorithms are known

cones of matrices and set functions and 0 1 optimization it has been recognized recently that to represent a polyhedron as the projection of a higher dimensional but simpler polyhedron is a powerful tool in polyhedral combinatorics a general method is developed to construct higher dimensional polyhedra or in some cases convex sets whose projection approximates the convex hull of 0 1 valued solutions of a system of linear inequalities an important feature of these approximations is that one can optimize any linear objective function over them in polynomial time in the special case of the vertex packing polytope a sequence of systems of inequalities is obtained such that the first system already includes clique odd hole odd antihole wheel and orthogonality constraints in particular for perfect and many other graphs this first system gives the vertex packing polytope for various classes of graphs including t perfect graphs it follows that the stable set polytope is the projection of a polytope with a polynomial number of facets an extension o

analyzing simulation experiments with common random numbers to analyze simulation runs which use the same random numbers the blocking concept of experimental design is not needed instead this paper applies a linear regression model with a nondiagonal covariance matrix this covariance matrix does not need to have a specific pattern such as constant covariances a simple example yields surprising results the paper proposes a new framework for the error analysis this framework consists of three factors namely common random numbers replication model validity each with three levels

optimal selection of customers for a last minute offer we analyze a short term revenue optimization problem involving the targeting of customers for a promotion in which a finite number of perishable items are sold on a last minute offer the goal is to select the subset of customers to whom the offer will be made available in order to maximize the expected return each client replies with a certain probability and reports a specific value that might depend on the customer type so that the selected subset has to balance the risk of not selling all items with the risk of assigning an item to a low value customer we show that threshold strategies which select all those clients with values above a certain optimal threshold might fail to achieve the maximal revenue however using a linear programming relaxation we prove that they attain a constant factor of the optimal value specifically the achieved factor is 1 2 when a single item is to be sold and approaches 1 as the number of available items grows to infinity also for the single item case we propose an upper bound based on a sharper linear relaxation that allows us to obtain a threshold strategy achieving at least 2 3 of the optimal revenue moreover although the complexity status of the problem is open we provide a polynomial time approximation scheme for the single item case

stochastic set packing problem abstract in this paper a stochastic version of the set packing problem spp is studied via scenario analysis we consider a one stage recourse approach to deal with the uncertainty in the coefficients it consists of maximizing in the stochastic spp a composite function of the expected value minus the weighted risk of obtaining a scenario whose objective function value is worse than a given threshold the splitting variable representation is decomposed by dualizing the nonanticipativity constraints that link the deterministic spp with a 0 1 knapsack problem for each scenario under consideration as a result a structured larger pure 0 1 model is created we present several procedures for obtaining good feasible solutions as well as a preprocessing approach for fixing variables the lagrange multipliers updating is performed by using the volume algorithm computational experience is reported for a broad variety of instances which shows that the new approach usually outperforms a state of the art optimization engine producing a comparable optimality gap with smaller several orders of magnitude computing time

a dynamic traveling salesman problem with stochastic arc costs we propose a dynamic traveling salesman problem tsp with stochastic arc costs motivated by applications such as dynamic vehicle routing in which the cost of a decision is known only probabilistically beforehand but is revealed dynamically before the decision is executed we formulate this as a dynamic program dp and compare it to static counterparts to demonstrate the advantage of the dynamic paradigm over an a priori approach we then apply approximate linear programming alp to overcome the dp s curse of dimensionality obtain a semi infinite linear programming lower bound and discuss its tractability we also analyze a rollout version of the price directed policy implied by our alp and derive worst case guarantees for its performance our computational study demonstrates the quality of a heuristically modified rollout policy using a computationally effective a posteriori bound

asymptotically efficient adaptive strategies in repeated games part i certainty equivalence strategies this paper addresses the problem of dynamic decision making in an uncertain and competitive environment a decision maker player 1 faces a system about which he has some parametric uncertainty and which is affected also by the actions of other agents we focus on a worst case analysts from the viewpoint of player 1 using the simplified model of a repeated matrix game with lack of information on one side where single stage rewards are random but announced and perfect observations are assumed certain ideas from the field of stochastic adaptive control are used to formulate performance criteria in a non bayesian setting and to devise appropriate control strategies the basic performance measure is the total reward accumulated by player 1 over all stages played the purpose of player 1 is to guarantee that his expected total reward will be close to what he could guarantee under complete information the present paper considers adaptive decision strategies of the certainty equivalence type based on a modified maximum likelihood estimator and studies their asymptotic long term performance a sequel paper will be devoted to asymptotically optimal strategies

maximizing stochastic monotone submodular functions we study the problem of maximizing a stochastic monotone submodular function with respect to a matroid constraint because of the presence of diminishing marginal values in real world problems our model can capture the effect of stochasticity in a wide range of applications we show that the adaptivity gap the ratio between the values of optimal adaptive and optimal nonadaptive policies is bounded and is equal to e e 1 we propose a polynomial time nonadaptive policy that achieves this bound we also present an adaptive myopic policy that obtains at least half of the optimal value furthermore when the matroid is uniform the myopic policy achieves the optimal approximation ratio of 1 1 e this paper was accepted by dimitris bertsimas and yinyu ye optimization

on the adaptivity gap of stochastic orienteering the input to the stochastic orienteering problem gupta et al in soda pp 1522 1538 2012 consists of a budget b and metric v d where each vertex v in v v v has a job with a deterministic reward and a random processing time drawn from a known distribution the processing times are independent across vertices the goal is to obtain a non anticipatory policy originating from a given root vertex to run jobs at different vertices that maximizes expected reward subject to the total distance traveled plus processing times being at most b an adaptive policy can choose the next vertex to visit based on observed random instantiations whereas a non adaptive policy is just given by a fixed ordering of vertices the adaptivity gap is the worst case ratio of the optimal adaptive and non adaptive rewards we prove an varomega left log log b 1 2 right loglogb 1 2 lower bound on the adaptivity gap of stochastic orienteering this provides a negative answer to the o 1 adaptivity gap conjectured by gupta et al 2012 and comes close to the o log log b o loglogb upper bound this result holds even on a line metric we also show an o log log b o loglogb upper bound on the adaptivity gap for the correlated stochastic orienteering problem where the reward of each job is random and possibly correlated to its processing time using this we obtain an improved quasi polynomial time min log n log b cdot tilde o log 2 log b min logn logb o log2logb approximation algorithm for correlated stochastic orienteering

partially observable markov decision processes with reward information basic ideas and models in a partially observable markov decision process pomdp if the reward can be observed at each step then the observed reward history contains information on the unknown state this information in addition to the information contained in the observation history can be used to update the state probability distribution the policy thus obtained is called a reward information policy ri policy an optimal ri policy performs no worse than any normal optimal policy depending only on the observation history the above observation leads to four different problem formulations for pomdps depending on whether the reward function is known and whether the reward at each step is observable this exploratory work may attract attention to these interesting problems

a learning approach for interactive marketing to a customer segment when a marketer in an interactive environment decides which messages to send to her customers she may send messages currently thought to be most promising exploitation or use poorly understood messages for the purpose of information gathering exploration we assume that customers are already clustered into homogeneous segments and we consider the adaptive learning of message effectiveness within a customer segment we present a bayesian formulation of the problem in which decisions are made for batches of customers simultaneously although decisions may vary within a batch this extends the classical multiarmed bandit problem for sampling one by one from a set of reward populations our solution methods include a lagrangian decomposition based approximate dynamic programming approach and a heuristic based on a known asymptotic approximation to the multiarmed bandit solution computational results show that our methods clearly outperform approaches that ignore the effects of information gain

the learning component of dynamic allocation indices this article is free to read on the publisher s website for a multiarmed bandit problem with exponential discounting the optimal allocation rule is defined by a dynamic allocation index defined for each arm on its space the index for an arm is equal to the expected immediate reward from the arm with an upward adjustment reflecting any uncertainty about the prospects of obtaining rewards from the arm and the possibilities of resolving those uncertainties by selecting that arm thus the learning component of the index is defined to be the difference between the index and the expected immediate reward for two arms with the same expected immediate reward the learning component should be larger for the arm for which the reward rate is more uncertain this is shown to be true for arms based on independent samples from a fixed distribution with an unknown parameter in the cases of bernoulli and normal distributions and similar results are obtained in other cases

restless bandits linear programming relaxations and a primal dual index heuristic we develop a mathematical programming approach for the classicalpspace hard restless bandit problem in stochastic optimization we introduce a hierarchy ofn wheren is the number of bandits increasingly stronger linear programming relaxations the last of which is exact and corresponds to the exponential size formulation of the problem as a markov decision chain while the other relaxations provide bounds and are efficiently computed we also propose a priority index heuristic scheduling policy from the solution to the firstorder relaxation where the indices are defined in terms of optimal dual variables in this way we propose a policy and a suboptimality guarantee we report results of computational experiments that suggest that the proposed heuristic policy is nearly optimal moreover the second order relaxation is found to provide strong bounds on the optimal value

combinatorial multi armed bandit and its extension to probabilistically triggered arms we define a general framework for a large class of combinatorial multi armed bandit cmab problems where subsets of base arms with unknown distributions form super arms in each round a super arm is played and the base arms contained in the super arm are played and their outcomes are observed we further consider the extension in which more base arms could be probabilistically triggered based on the outcomes of already triggered arms the reward of the super arm depends on the outcomes of all played arms and it only needs to satisfy two mild assumptions which allow a large class of nonlinear reward instances we assume the availability of an offline approximation oracle that takes the means of the outcome distributions of arms and outputs a super arm that with probability generates an fraction of the optimal expected reward the objective of an online learning algorithm for cmab is to minimize approximation regret which is the difference in total expected reward between the fraction of expected reward when always playing the optimal super arm and the expected reward of playing super arms according to the algorithm we provide cucb algorithm that achieves o log n distribution dependent regret where n is the number of rounds played and we further provide distribution independent bounds for a large class of reward functions our regret analysis is tight in that it matches the bound of ucb1 algorithm up to a constant factor for the classical mab problem and it significantly improves the regret bound in an earlier paper on combinatorial bandits with linear rewards we apply our cmab framework to two new applications probabilistic maximum coverage pmc for online advertising and social in uence maximization for viral marketing both having nonlinear reward structures in particular application to social in uence maximization requires our extension on probabilistically triggered arms

asymptotically efficient adaptive allocation rules for the multiarmed bandit problem with switching cost the authors consider multiarmed bandit problems with switching cost define uniformly good allocation rules and restrict attention to such rules they present a lower bound on the asymptotic performance of uniformly good allocation rules and construct an allocation scheme that achieves the bound it is found that despite the inclusion of a switching cost the proposed allocation scheme achieves the same asymptotic performance as the optimal rule for the bandit problem without switching cost this is made possible by grouping together samples into blocks of increasing sizes thereby reducing the number of switches to o log n finally an optimal allocation scheme for a large class of distributions which includes members of the exponential family is illustrated

dynamic allocation indices for restless projects and queueing admission control a polyhedral approach this paper develops a polyhedral approach to the design analysis and computation of dynamic allocation indices for scheduling binary action engage rest markovian stochastic projects which can change state when rested restless bandits rbs based on partial conservation laws pcls this extends previous work by the author j nino mora 2001 restless bandits partial conservation laws and indexability adv appl probab 33 76 98 where pcls were shown to imply the optimality of index policies with a postulated structure in stochastic scheduling problems under admissible linear objectives and they were deployed to obtain simple sufficient conditions for the existence of whittle s 1988 rb index indexability along with an adaptive greedy index algorithm the new contributions include i we develop the polyhedral foundation of the pcl framework based on the structural and algorithmic properties of a new polytope associated with an accessible set system extended polymatroid ii we present new dynamic allocation indices for rbs motivated by an admission control model which extend whittle s and have a significantly increased scope iii we deploy pcls to obtain both sufficient conditions for the existence of the new indices pcl indexability and a new adaptive greedy index algorithm iv we interpret pcl indexability as a form of the classic economics law of diminishing marginal returns and characterize the index as an optimal marginal cost rate we further solve a related optimal constrained control problem v we carry out a pcl indexability analysis of the motivating admission control model under time discounted and long run average criteria this gives under mild conditions a new index characterization of optimal threshold policies and vi we apply the latter to present new heuristic index policies for two hard queueing control problems admission control and routing to parallel queues and scheduling a multiclass make to stock queue with lost sales both under state dependent holding cost rates and birth death dynamics

approximations to stochastic dynamic programs via information relaxation duality in the analysis of complex stochastic dynamic programs we often seek strong theoretical guarantees on the suboptimality of heuristic policies a common technique for obtaining performance bounds i

a note on performance limitations in bandit problems with side information we consider a sequential adaptive allocation problem which is formulated as a traditional two armed bandit problem but with one important modification at each time step t before selecting which arm to pull the decision maker has access to a random variable xt which provides information on the reward in each arm performance is measured as the fraction of time an inferior arm generating lower mean reward is pulled we derive a minimax lower bound that proves that in the absence of sufficient statistical diversity in the distribution of the covariate x a property that we shall refer to as lack of persistent excitation no policy can improve on the best achievable performance in the traditional bandit problem without side information

technical note the adaptive knapsack problem with stochastic rewards given a set of items with associated deterministic weights and random rewards the adaptive stochastic knapsack problem adaptive skp maximizes the probability of reaching a predetermined target reward level when items are inserted sequentially into a capacitated knapsack before the reward of each item is realized this model arises in resource allocation problems that permit or require sequential allocation decisions in a probabilistic setting one particular application is in obsolescence inventory management in this paper the adaptive skp is formulated as a dynamic programming dp problem for discrete random rewards the paper also presents a heuristic that mixes adaptive and static policies to overcome the curse of dimensionality in the dp the proposed heuristic is extended to problems with normally distributed random rewards the heuristic can solve large problems quickly and its solution always outperforms a static policy the numerical study indicates that a near optimal solution can be obtained by using an algorithm with limited look ahead capabilities

complete algebraic characterization of a stable runge kutta methods important stability concepts for runge kutta methods are i a and b stability for these properties there exist very similar algebraic characterizations the characterization of b stability is known for s irreducible methods in this paper an algebraic characterization of i stability and a stability related to the coefficients of the method is deduced without any assumption on the runge kutta methods the corresponding linear dynamic system and its transfer function is considered the positive real lemma characterizes the passivity of the system or equivalently the positive realness of the transfer function by the lyapunov equation dropping the assumption of controllability and observability a generalization is possible using the kalman canonical decomposition interpreting the modified stability function of a runge kutta method as the transfer function the positive real lemma yields a complete algebraic characterization of a stable runge kutta methods

bandit problems with side observations an extension of the traditional two armed bandit problem is considered in which the decision maker has access to some side information before deciding which arm to pull at each time t before making a selection the decision maker is able to observe a random variable x sub t that provides some information on the rewards to be obtained the focus is on finding uniformly good rules that minimize the growth rate of the inferior sampling time and on quantifying how much the additional information helps various settings are considered and for each setting lower bounds on the achievable inferior sampling time are developed and asymptotically optimal adaptive schemes achieving these lower bounds are constructed

randomized algorithms for online knapsack problems in this paper we study online knapsack problems the input is a sequence of items e 1 e 2 e n each of which has a size and a value given the ith item e i we either put e i into the knapsack or reject it in the removable setting when e i is put into the knapsack some items in the knapsack are removed with no cost if the sum of the size of e i and the total size in the current knapsack exceeds the capacity of the knapsack our goal is to maximize the profit i e the sum of the values of items in the last knapsack we present a simple randomized 2 competitive algorithm for the unweighted non removable case and show that it is the best possible where knapsack problem is called unweighted if the value of each item is equal to its size for the removable case we propose a randomized 2 competitive algorithm despite there is no constant competitive deterministic algorithm we also provide a lower bound 1 1 e 1 368 for the competitive ratio for the unweighted removable case we propose a 10 7 competitive algorithm and provide a lower bound 1 25 for the competitive ratio

on the value of learning for bernoulli bandits with unknown parameters investigates the multiarmed bandit problem where each arm generates an infinite sequence of bernoulli distributed rewards the parameters of these bernoulli distributions are unknown and initially assumed to be beta distributed every time a bandit is selected its beta distribution is updated to new information in a bayesian way the objective is to maximize the long term discounted rewards we study the relationship between the necessity of acquiring additional information and the reward this is done by considering two extreme situations which occur when a bandit has been played n times the situation where the decision maker stops learning and the situation where the decision maker acquires full information about that bandit we show that the difference in reward between this lower and upper bound goes to zero as n grows large

weighted multivariate mean square error for processes optimization a case study on flux cored arc welding for stainless steel claddings a mathematical programming technique developed recently that optimizes multiple correlated characteristics is the multivariate mean square error mmse the mmse approach has obtained noteworthy results by avoiding the production of inappropriate optimal points that can occur when a method fails to take into account a correlation structure where the mmse approach is deficient however is in cases where the multiple correlated characteristics need to be optimized with varying degrees of importance the mmse approach in treating all responses as having the same importance is unable to attribute the desired weights this paper thus introduces a strategy that weights the responses in the mmse approach the method called the weighted multivariate mean square error wmmse utilizes a weighting procedure that integrates principal component analysis pca and response surface methodology rsm in doing so wmmse obtains uncorrelated weighted objective functions from the original responses after being mathematically programmed these functions are optimized by employing optimization algorithms we applied wmmse to optimize a stainless steel cladding application executed via the flux cored arc welding fcaw process four input parameters and eight response variables were considered stainless steel cladding which carries potential benefits for a variety of industries takes low cost materials and deposits over their surfaces materials having anti corrosive properties optimal results were confirmed which ensured the deposition of claddings with defect free beads exhibiting the desired geometry and demonstrating good productivity indexes

a fully polynomial time approximation scheme for single item stochastic inventory control with discrete demand the single item stochastic inventory control problem is to find an inventory replenishment policy in the presence of independent discrete stochastic demands under periodic review and finite time horizon in this paper we prove that this problem is intractable and design for it a fully polynomial time approximation scheme

indexability of restless bandit problems and optimality of whittle index for dynamic multichannel access in this paper we consider a class of restless multiarmed bandit processes rmabs that arises in dynamic multichannel access user server scheduling and optimal activation in multiagent systems for this class of rmabs we establish the indexability and obtain whittle index in closed form for both discounted and average reward criteria these results lead to a direct implementation of whittle index policy with remarkably low complexity when arms are stochastically identical we show that whittle index policy is optimal under certain conditions furthermore it has a semiuniversal structure that obviates the need to know the markov transition probabilities the optimality and the semiuniversal structure result from the equivalence between whittle index policy and the myopic policy established in this work for nonidentical arms we develop efficient algorithms for computing a performance upper bound given by lagrangian relaxation the tightness of the upper bound and the near optimal performance of whittle index policy are illustrated with simulation examples

risk criteria in a stochastic knapsack problem we consider an investor who wants to allocate funds among several projects each project is expected to yield a certain reward and the objective is that a total reward will achieve a certain given amount called the target this problem is relatively easy to solve when rewards are deterministic but may be hard in a more realistic setting when the rewards are stochastic and the investor wants to maximize the probability of attaining the target we show that by combining dynamic programming with a search procedure the stochastic version of the problem can be solved relatively fast when rewards are normally distributed the procedure is also useful for other risk criteria which involve both the mean and the variance of the total reward

response surface methodology s steepest ascent and step size revisited abstract response surface methodology rsm searches for the input combination maximizing the output of a real system or its simulation rsm is a heuristic that locally fits first order polynomials and estimates the corresponding steepest ascent sa paths however sa is scale dependent and its step size is selected intuitively to tackle these two problems this paper derives novel techniques combining mathematical statistics and mathematical programming technique 1 called adapted sa asa accounts for the covariances between the components of the estimated local gradient asa is scale independent the step size problem is solved tentatively technique 2 does follow the sa direction but with a step size inspired by asa mathematical properties of the two techniques are derived and interpreted numerical examples illustrate these properties the search directions of the two techniques are explored in monte carlo experiments these experiments show that in general asa gives a better search direction than sa

simulation designs for the estimation of quadratic response surface gradients in the presence of model misspecification this article considers the construction of simulation designs for the ordinary least squares estimation of second order metamodels two premises underlie the development of these experimental strategies first it is assumed that the postulated metamodel may be misspecified due to the true model structure being of third order it is therefore important that the locations of the simulation experiments be specified to provide protection against bias as well as variance in the estimation of metamodel parameters the second premise is based on the observation that in many applications of metamodels functions of the fitted model coefficients such as the slope gradients are of greater interest than the response function the integrated mean squared error of slopes design criterion that is implemented here addresses both premises this criterion finds application in various optimum seeking methods and sensitivity analysis procedures combinations of four important classes of response surface designs and three pseudorandom number assignment strategies constitute the basis structure of the simulation designs studied the performance of these simulation designs is evaluated and subsequently compared to a similar set of experimental plans that have as their focus the estimation of the response function

approximation algorithms for restless bandit problems the restless bandit problem is one of the most well studied generalizations of the celebrated stochastic multi armed bandit mab problem in decision theory in its ultimate generality the restless bandit problem is known to be pspace hard to approximate to any nontrivial factor and little progress has been made on this problem despite its significance in modeling activity allocation under uncertainty in this article we consider the feedback mab problem where the reward obtained by playing each of n independent arms varies according to an underlying on off markov process whose exact state is only revealed when the arm is played the goal is to design a policy for playing the arms in order to maximize the infinite horizon time average expected reward this problem is also an instance of a partially observable markov decision process pomdp and is widely studied in wireless scheduling and unmanned aerial vehicle uav routing unlike the stochastic mab problem the feedback mab problem does not admit to greedy index based optimal policies we develop a novel duality based algorithmic technique that yields a surprisingly simple and intuitive 2 approximate greedy policy to this problem we show that both in terms of approximation factor and computational efficiency our policy is closely related to the whittle index which is widely used for its simplicity and efficiency of computation subsequently we define a multi state generalization that we term monotone bandits which remains subclass of the restless bandit problem we show that our policy remains a 2 approximation in this setting and further our technique is robust enough to incorporate various side constraints such as blocking plays switching costs and even models where determining the state of an arm is a separate operation from playing it our technique is also of independent interest for other restless bandit problems and we provide an example in nonpreemptive machine replenishment interestingly in this case our policy provides a constant factor guarantee whereas the whittle index is provably polynomially worse by presenting the first o 1 approximations for nontrivial instances of restless bandits as well as of pomdps our work initiates the study of approximation algorithms in both these contexts

relaxation analysis for the dynamic knapsack problem with stochastic item sizes we consider a version of the knapsack problem in which an item size is random and revealed only when the decision maker attempts to insert it after every successful insertion the decision maker ca

a generalization of the positive real lemma the positive real lemma also called the kalman yacubovich popov lemma characterizes the positive realness of the transfer function matrix of a linear dynamic system by algebraic conditions in the case of a pole zero cancellation appearing in the transfer function matrix or in other words missing the assumptions of controllability and observability there exist generalized versions which are discussed and proven applying the kalman canonical decomposition

on optimality of myopic policy for restless multi armed bandit problem an axiomatic approach due to its application in numerous engineering problems the restless multi armed bandit rmab problem is of fundamental importance in stochastic decision theory however solving the rmab problem is well known to be pspace hard with the optimal policy usually intractable due to the exponential computation complexity a natural alternative approach is to seek simple myopic policies which are easy to implement this paper presents a generic study on the optimality of the myopic policy for the rmab problem more specifically we develop three axioms characterizing a family of generic and practically important functions termed as regular functions by performing a mathematical analysis based on the developed axioms we establish the closed form conditions under which the myopic policy is guaranteed to be optimal the axiomatic analysis also illuminates important engineering implications of the myopic policy including the intrinsic tradeoff between exploration and exploitation a case study is then presented to illustrate the application of the derived results in analyzing a class of rmab problems arising from multi channel opportunistic access

running errands in time approximation algorithms for stochastic orienteering in the stochastic orienteering problem we are given a finite metric space where each node contains a job with some deterministic reward and a random processing time the processing time distributions are known and independent across nodes however the actual processing time of a job is not known until it is completely processed the objective is to compute a nonanticipatory policy to visit nodes and run the corresponding jobs so as to maximize the total expected reward subject to the total distance traveled plus the total processing time being at most a given budget of b this problem combines aspects of the stochastic knapsack problem with uncertain item sizes as well as the deterministic orienteering problem in this paper we consider both nonadaptive and adaptive policies for stochastic orienteering we present a constant factor approximation algorithm for the nonadaptive version and an olog log b approximation algorithm for the adaptive version we extend both these results to directed metrics and a more general sequence orienteering problem finally we address the stochastic orienteering problem when the node rewards are also random and possibly correlated with the processing time and obtain an olog n log b approximation algorithm here n is the number of nodes in the metric all our results for adaptive policies also bound the corresponding adaptivity gaps

method of lines approximations of delay differential equations we consider a special type of numerical methods for delay differential equations ddes by introducing a new independent variable an initial value problem for ddes is converted into an initial boundary value problem for the convection equation thus it is also possible to get an approximate solution to the dde problem by solving the initial boundary value problem with a suitable numerical method instead of solving the original problem in this paper we study a family of method of lines mol approximations to the problem which is obtained by applying runge kutta rk methods for space discretization and prove their convergence under the assumption that the rk methods satisfy a condition known as an algebraic characterization of a stability the result is also confirmed by numerical experiments moreover we show that the condition derives several stability properties of the mol approximations

m g c queueing systems with multiple customer classes characterization and control of achievable performance under nonpreemptive priority rules this paper considers an m g c queueing system serving a finite number j of distinct customer classes performance of the system as measured by the vector of steady state expected waiting times of the customer classes the performance vector may be controlled by adopting an appropriate priority discipline we show that the performance space the set of performance vectors which are achievable under some nonpreemptive work conserving priority rule is a polyhedron described by 2j 1 inequalities the special polymatroidal structure of this polyhedron nevertheless allows for efficient oj2 log j procedures to minimize any convex separable function of the performance vector linear objectives are shown to be minimized by absolute priority rules thus generalizing a well known result for m g 1 systems we also show that each point in the performance space may be achieved by a unique generalized dynamic priority rule specified by j 1 parameters which may be determined by the recursive solution of j 1 single variable quadratic equations this class of rules contains the absolute priority rules and the pure dynamic rules as special cases our results are accurate up to one extremely accurate approximation and completely exact for m g 1 and m m c systems as well as in heavy traffic

simulation designs and correlation induction for reducing second order bias in first order response surfaces construction of simulation designs for the estimation of response surface metamodels is often based on optimal design theory underlying such designs is the assumption that the postulated model provides the correct representation of the simulated response as a result the location of design points and the assignment of pseudorandom number streams to these experiments are determined through the minimization of some function of the covariance matrix of the model coefficient estimators in contrast we assume that the postulated model may be incorrect attention is therefore directed to the development of simulation designs that offer protection against the bias due to possible model misspecification as well as error variance the particular situation examined is the estimation of first order response surface models in the presence of polynomials of order two traditional two level factorial plans combined with one of three pseudorandom number assignment strategies define the simulation designs specificati

finite time lower bounds for the two armed bandit problem we obtain minimax lower bounds on the regret for the classical two armed bandit problem we provide a finite sample minimax version of the well known log n asymptotic lower bound of lai and robbins 1985 the finite time lower bound allows us to derive conditions for the amount of time necessary to make any significant gain over a random guessing strategy these bounds depend on the class of possible distributions of the rewards associated with the arms for example in contrast to the log n asymptotic results on the regret we show that the minimax regret is achieved by mere random guessing under fairly mild conditions on the set of allowable configurations of the two arms that is we show that for every allocation rule and for every n there is a configuration such that the regret at time n is at least 1 spl epsiv times the regret of random guessing where spl epsiv is any small positive constant

beyond competitive analysis the competitive analysis of online algorithms has been criticized as being too crude and unrealistic we propose refinements of competitive analysis in two directions the first restricts the power of the adversary by allowing only certain input distributions while the other allows for comparisons between information regimes for online decision making we illustrate the first with an application to the paging problem as a byproduct we characterize completely the work functions of this important special case of the k server problem we use the second refinement to explore the power of lookahead in server and task systems

the dynamic and stochastic knapsack problem with deadlines in this paper a dynamic and stochastic model of the well known knapsack problem is developed and analyzed the problem is motivated by a wide variety of real world applications objects of random weight and reward arrive according to a stochastic process in time the weights and rewards associated with the objects are distributed according to a known probability distribution each object can either be accepted to be loaded into the knapsack of known weight capacity or be rejected the objective is to determine the optimal policy for loading the knapsack within a fixed time horizon so as to maximize the expected accumulated reward the optimal decision rules are derived and are shown to exhibit surprising behavior in some cases it is also shown that if the distribution of the weights is concave then the decision rules behave according to intuition

improvements and generalizations of stochastic knapsack and markovian bandits approximation algorithms we study the multi armed bandit problem with arms which are markov chains with rewards in the finite horizon setting the celebrated gittins indices do not apply and the exact solution is intractable we provide approximation algorithms for the general model of markov decision processes with nonunit transition times when preemption is allowed we provide a 1 2 e approximation along with an example showing this is tight when preemption isn t allowed we provide a 1 12 approximation which improves to a 4 27 approximation when transition times are unity our model captures the markovian bandits model of gupta et al the stochastic knapsack model of dean et al and the budgeted learning model of guha and munagala our algorithms improve existing results in all three areas in our analysis we encounter and overcome to our knowledge a new obstacle an algorithm that provably exists via analytical arguments but cannot be found in polynomial time