Efficient Sequential Decision Making

This thesis studies three problems in sequential decision making/nacross two different frameworks. The first framework we consider is/nonline learning: for each round of a $T$ round repeated game, the/nlearner makes a prediction, the adversary observes this prediction/nand reveals the true outcome, and the learner suffers some loss/nbased on the accuracy of the prediction. The learner's aim is to/nminimize the regret, which is defined to be the difference between/nthe learner's cumulative loss and the cumulative loss of the best/nprediction strategy in some class. We study the minimax strategy,/nwhich guarantees the lowest regret against all possible adversary/nstrategies. In general, computing the minimax strategy is/nexponential in $T$; we focus on two setting where efficient/nalgorithms are possible./nThe first is prediction under squared Euclidean loss. The learner/npredicts a point in $\Reals^d$ and the adversary is constrained to/nrespond with a point in some compact set. The regret is with respect/nto the single best prediction in the set. We compute the minimax/nstrategy and the value of the game for any compact set and show that/nthe value is the product of a horizon-dependent constant and the/nsquared radius of the smallest enclosing ball of the set. We also/npresent the optimal strategy of the adversary for two important/nsets: ellipsoids and polytopes that intersect their smallest/nenclosing ball at all vertices. The minimax strategy can be cast as/na simple shrinkage of the past data towards the center of this/nminimum enclosing ball, where the shrinkage factor can be/nefficiently computed before the start of the game. Noting that the/nvalue does not have any explicit dimension dependence, we then/nextend these results to Hilbert space, finding, once again, that the/nvalue is proportional to the squared radius of the smallest/nenclosing ball./nThe second setting where we derive efficient minimax strategies is/nonline linear regression. At the start of each round, the adversary/nchooses and reveals a vector of covariates. The/nregret is defined with respect to the best linear function of the/ncovariates. We show that the minimax strategy is an easily computed/nlinear predictor, provided that the adversary adheres to some/nnatural constraints that prevent him from misrepresenting/nthe scale of the problem. This strategy is horizon-independent:/nregardless of the length of the game, this strategy incurs no more/nregret than any strategy that has knowledge of the number of/nrounds. We also provide an interpretation of the minimax algorithm/nas a follow-the-regularized-leader strategy with a data-dependent/nregularizer and obtain an explicit expression for the minimax/nregret./nWe then turn to the second framework, reinforcement learning. More/nspecifically, we consider the problem of controlling a Markov/ndecision process (MDP) with a large state-space. Since it is/nintractable to compete with the optimal policy for large scale/nproblems, we pursue the more modest goal of competing with a/nlow-dimensional family of policies. Specifically, we restrict the/nvariables of the dual linear program to lie in some low-dimensional/nsubspace, and show that we can find a policy that performs almost as/nwell as the best policy in this class. We derive separate results/nfor the average cost and discounted cost cases. Most importantly,/nthe complexity of our method depends on the size of the comparison/nclass but not the size of the state-space. Preliminary experiments/nshow the effectiveness of the proposed algorithms in a queuing/napplication.

Advisor: Bartlett, Peter L