the knowledge gradient algorithm for a general class of online learning problems we derive a one period look ahead policy for finite and infinite horizon online optimal learning problems with gaussian rewards our approach is able to handle the case where our prior beliefs about the rewards are correlated which is not handled by traditional multiarmed bandit methods experiments show that our kg policy performs competitively against the best known approximation to the optimal policy in the classic bandit problem and it outperforms many learning policies in the correlated case

kernel estimation and model combination in a bandit problem with covariates multi armed bandit problem is an important optimization game that requires an exploration exploitation tradeoff to achieve optimal total reward motivated from industrial applications such as online advertising and clinical research we consider a setting where the rewards of bandit machines are associated with covariates and the accurate estimation of the corresponding mean reward functions plays an important role in the performance of allocation rules under a exible problem setup we establish asymptotic strong consistency and perform a finite time regret analysis for a sequential randomized allocation strategy based on kernel estimation in addition since many nonparametric and parametric methods in supervised learning may be applied to estimating the mean reward functions but guidance on how to choose among them is generally unavailable we propose a model combining allocation strategy for adaptive performance simulations and a real data evaluation are conducted to illustrate the performance of the proposed allocation strategy

adaptive treatment allocation and the multi armed bandit problem a class of simple adaptive allocation rules is proposed for the problem often called the multi armed bandit problem of sampling x 1 cdots x n sequentially from k populations with densities belonging to an exponential family in order to maximize the expected value of the sum s n x 1 cdots x n these allocation rules are based on certain upper confidence bounds which are developed from boundary crossing theory for the k population parameters the rules are shown to be asymptotically optimal as n rightarrow infty from both bayesian and frequentist points of view monte carlo studies show that they also perform very well for moderate values of the horizon n

spectral regularization algorithms for learning large incomplete matrices we use convex relaxation techniques to provide a sequence of regularized low rank solutions for large scale matrix completion problems using the nuclear norm as a regularizer we provide a simple and very efficient convex algorithm for minimizing the reconstruction error subject to a bound on the nuclear norm our algorithm soft impute iteratively replaces the missing elements with those obtained from a soft thresholded svd with warm starts this allows us to efficiently compute an entire regularization path of solutions on a grid of values of the regularization parameter the computationally intensive part of our algorithm is in computing a low rank svd of a dense matrix exploiting the problem structure we show that the task can be performed with a complexity of order linear in the matrix dimensions our semidefinite programming algorithm is readily scalable to large matrices for example soft impute takes a few hours to compute low rank approximations of a 106 x 106 incomplete matrix with 107 observed entries and fits a rank 95 approximation to the full netflix training set in 3 3 hours our methods achieve good training and test errors and exhibit superior timings when compared to other competitive state of the art techniques

new analysis and results for the frank wolfe method we present new results for the frank wolfe method also known as the conditional gradient method we derive computational guarantees for arbitrary step size sequences which are then applied to various step size rules including simple averaging and constant step sizes we also develop step size rules and computational guarantees that depend naturally on the warm start quality of the initial and subsequent iterates our results include computational guarantees for both duality bound gaps and the so called fw gaps lastly we present complexity bounds in the presence of approximate computation of gradients and or linear optimization subproblem solutions

on the hausdorff dimension of some graphs consider the functions wb x b cn 1 bnx on 1 0n n oo where b 1 0 0 such that if b is large enough then the hausdorff dimension of the graph of wb is bounded below by 2a c ln b we also show that if a function f is convex lipschitz of order a then the graph of f has a finite measure with respect to hausdorff s measure in dimension 2 a the convex lipschitz functions of order a include zygmund s class a our analysis shows that the graph of the classical van der waerdentagaki nowhere differentiable function has a finite measure with respect to h t t in 1 t we consider the hausdorff dimension of the graphs of various continuous functions we introduce a new geometric property of a function convex lipschitz of some order and obtain an upper bound on the dimension of a graph with this property in particular our analysis includes functions of the form

response surface bandits in this paper we define a response surface bandit as the sequential design problem that maximizes an expected bandit utility but where the outcomes y n are continuous and can be related through a response surface to a set of controllable variables x n x 1n x 2n x kn we link this problem to other traditional optimization problems from industrial engineering and to the traditional bandit problem we consider two approaches to the problem the first is based on a myopic sequential design the second approach uses the best design out of a family of designs related to upper bounds for the predicted surface the family includes myopic and sequential versions of d optimal designs these approaches can be generalized to more broadly defined sequential problems

fast low rank modifications of the thin singular value decomposition this paper develops an identity for additive modivations of a singular value decomposition svd to reflect updates downdates shifts and edits of the data matrix this sets the stage for fast and ememory efficient sequential algorithms for tracking singular values and subspaces in conjunction with a fast solution for the pseudo inverse of a submatrix of an orthogonal matrix we develop a scheme for computing a thin svd of streaming data in a single pass with linear time complexity a rank r think svd of a p x q matrix can be computed in o pqr time for r less than or equal sqroot min p q linear algebra and its applications

self normalized processes exponential inequalities moment bounds and iterated logarithm laws self normalized processes arise naturally in statistical applications being unit free they are not affected by scale changes moreover self normalization often eliminates or weakens moment assumptions in this paper we present several exponential and moment inequalities particularly those related to laws of the iterated logarithm for self normalized random variables including martingales tail probability bounds are also derived for random variables bt 0 and at let yt exp at 2bt2 2 we develop inequalities for the moments of at bt or supt 0at bt log logbt 1 2 and variants thereof when eyt 1 or when yt is a supermartingale for all belonging to some interval our results are valid for a wide class of random processes including continuous martingales with at mt and b t sqrt langle m rangle t and sums of conditionally symmetric variables di with at i 1tdi and b t sqrt sum i 1 t d i 2 a sharp maximal inequality for conditionally symmetric random variables and for continuous local martingales with values in rm m 1 is also established another development in this paper is a bounded law of the iterated logarithm for general adapted sequences that are centered at certain truncated conditional expectations and self normalized by the square root of the sum of squares the key ingredient in this development is a new exponential supermartingale involving i 1tdi and i 1tdi2 a compact law of the iterated logarithm for self normalized martingales is also derived in this connection

linearly convergent away step conditional gradient for non strongly convex functions we consider the problem of minimizing the sum of a linear function and a composition of a strongly convex function with a linear transformation over a compact polyhedral set jaggi and lacoste julien an affine invariant linear convergence analysis for frank wolfe algorithms nips 2013 workshop on greedy algorithms frank wolfe and friends 2014 show that the conditional gradient method with away steps employed on the aforementioned problem without the additional linear term has a linear rate of convergence depending on the so called pyramidal width of the feasible set we revisit this result and provide a variant of the algorithm and an analysis based on simple linear programming duality arguments as well as corresponding error bounds this new analysis a enables the incorporation of the additional linear term and b depends on a new constant that is explicitly expressed in terms of the problem s parameters and the geometry of the feasible set this constant replaces the pyramidal width which is difficult to evaluate

asymptotically efficient adaptive allocation schemes for controlled i i d processes finite parameter space the authors consider a controlled i i d independently identically distributed process whose distribution is parametrized by an unknown parameter theta belonging to some known parameter space theta and a one step reward associated with each pair of control and the following state of the process the objective is to maximize the expected value of the sum of one step rewards over an infinite horizon by introducing the loss associated with a control scheme it is shown that the problem is equivalent to minimizing this loss uniformly good adaptive control schemes are defined and emphasized a lower bound on the loss associated with any uniformly good control scheme is developed finally an adaptive control scheme is constructed whose loss equals the lower bound and is therefore asymptotically efficient

conditional gradient algorithms for norm regularized smooth convex optimization motivated by some applications in signal processing and machine learning we consider two convex optimization problems where given a cone k k a norm vert cdot vert and a smooth convex function f f we want either 1 to minimize the norm over the intersection of the cone and a level set of f f or 2 to minimize over the cone the sum of f f and a multiple of the norm we focus on the case where a the dimension of the problem is too large to allow for interior point algorithms b vert cdot vert is too complicated to allow for computationally cheap bregman projections required in the first order proximal gradient algorithms on the other hand we assume that it is relatively easy to minimize linear forms over the intersection of k k and the unit vert cdot vert ball motivating examples are given by the nuclear norm with k k being the entire space of matrices or the positive semidefinite cone in the space of symmetric matrices and the total variation norm on the space of 2d images we discuss versions of the conditional gradient algorithm capable to handle our problems of interest provide the related theoretical efficiency estimates and outline some applications

scalable robust matrix recovery frank wolfe meets proximal methods recovering matrices from compressive and grossly corrupted observations is a fundamental problem in robust statistics with rich applications in computer vision and machine learning in theory under certain conditions this problem can be solved in polynomial time via a natural convex relaxation known as compressive principal component pursuit cpcp however many existing provably convergent algorithms for cpcp suffer from superlinear per iteration cost which severely limits their applicability to large scale problems in this paper we propose provably convergent scalable and efficient methods to solve cpcp with essentially linear per iteration cost our method combines classical ideas from frank wolfe and proximal methods in each iteration we mainly exploit frank wolfe to update the low rank component with rank one singular value decomposition and exploit the proximal step for the sparse term convergence results and implementation details are discussed we demonstrate the practicability a

global bandits multiarmed bandits mabs model sequential decision making problems in which a learner sequentially chooses arms with unknown reward distributions in order to maximize its cumulative reward most of the prior works on mab assume that the reward distributions of each arm are independent but in a wide variety of decision problems from drug dosage to dynamic pricing the expected rewards of different arms are correlated so that selecting one arm provides information about the expected rewards of other arms as well we propose and analyze a class of models of such decision problems which we call global bandits gb in the case in which rewards of all arms are deterministic functions of a single unknown parameter we construct a greedy policy that achieves bounded regret with a bound that depends on the single true parameter of the problem hence this policy selects suboptimal arms only finitely many times with probability one for this case we also obtain a bound on regret that is independent of the true parameter this bound is sublinear with an exponent that depends on the informativeness of the arms we also propose a variant of the greedy policy that achieves tilde mathcal o sqrt t worst case and mathcal o 1 parameter dependent regret finally we perform experiments on dynamic pricing and show that the proposed algorithms achieve significant gains with respect to the well known benchmarks

bandit problems with side observations an extension of the traditional two armed bandit problem is considered in which the decision maker has access to some side information before deciding which arm to pull at each time t before making a selection the decision maker is able to observe a random variable x sub t that provides some information on the rewards to be obtained the focus is on finding uniformly good rules that minimize the growth rate of the inferior sampling time and on quantifying how much the additional information helps various settings are considered and for each setting lower bounds on the achievable inferior sampling time are developed and asymptotically optimal adaptive schemes achieving these lower bounds are constructed

a new positive definite geometric mean of two positive definite matrices abstract we introduce and study a new positive definite in certain singular cases positive semidefinite geometric mean of two positive definite under certain conditions positive semidefinite matrices

low rank optimization with trace norm penalty the paper addresses the problem of low rank trace norm minimization we propose an algorithm that alternates between fixed rank optimization and rank one updates the fixed rank optimization is characterized by an efficient factorization that makes the trace norm differentiable in the search space and the computation of duality gap numerically tractable the search space is nonlinear but is equipped with a riemannian structure that leads to efficient computations we present a second order trust region algorithm with a guaranteed quadratic rate of convergence overall the proposed optimization scheme converges superlinearly to the global solution while maintaining complexity that is linear in the number of rows and columns of the matrix to compute a set of solutions efficiently for a grid of regularization parameters we propose a predictor corrector approach that outperforms the naive warm restart approach on the fixed rank quotient manifold the performance of the proposed algorithm is illustrated on p

general bounds and finite time improvement for the kiefer wolfowitz stochastic approximation algorithm we consider the kiefer wolfowitz kw stochastic approximation algorithm and derive general upper bounds on its mean squared error the bounds are established using an elementary induction argument and phrased directly in the terms of tuning sequences of the algorithm from this we deduce the nonnecessity of one of the main assumptions imposed on the tuning sequences by kiefer and wolfowitz kiefer j j wolfowitz 1952 stochastic estimation of the maximum of a regression function ann math statist 23 3 462 466 and essentially all subsequent literature the optimal choice of sequences is derived for various cases of interest and an adaptive version of the kw algorithm scaled and shifted kw or sskw is proposed with the aim of improving its finite time behavior the key idea is to dynamically scale and shift the tuning sequences to better match them with characteristics of the unknown function and noise level and thus improve algorithm performance numerical results are provided that illustrate that the proposed algorithm retains the convergence properties of the original kw algorithm while dramatically improving its performance in some cases

the power of convex relaxation near optimal matrix completion this paper is concerned with the problem of recovering an unknown matrix from a small fraction of its entries this is known as the matrix completion problem and comes up in a great number of applications including the famous netflix prize and other similar questions in collaborative filtering in general accurate recovery of a matrix from a small number of entries is impossible but the knowledge that the unknown matrix has low rank radically changes this premise making the search for solutions meaningful this paper presents optimality results quantifying the minimum number of entries needed to recover a matrix of rank r exactly by any method whatsoever the information theoretic limit more importantly the paper shows that under certain incoherence assumptions on the singular vectors of the matrix recovery is possible by solving a convenient convex program as soon as the number of entries is on the order of the information theoretic limit up to logarithmic factors this convex program simply finds among all matrices consistent with the observed entries that with minimum nuclear norm as an example we show that on the order of nr log n samples are needed to recover a random n x n matrix of rank r by any method and to be sure nuclear norm minimization succeeds as soon as the number of entries is of the form nr polylog n

an extended frank wolfe method with in face directions and its application to low rank matrix completion motivated principally by the low rank matrix completion problem we present an extension of the frank wolfe method that is designed to induce near optimal solutions on low dimensional faces of the feasible region this is accomplished by a new approach to generating in face directions at each iteration as well as through new choice rules for selecting between in face and regular frank wolfe steps our framework for generating in face directions generalizes the notion of away steps introduced by wolfe in particular the in face directions always keep the next iterate within the minimal face containing the current iterate we present computational guarantees for the new method that trade off efficiency in computing near optimal solutions with upper bounds on the dimension of minimal faces of iterates we apply the new method to the matrix completion problem where low dimensional faces correspond to low rank matrices we present computational results that demonstrate the effectiveness of our methodo

maximizing a class of utility functions over the vertices of a polytope given a polytope x a monotone concave univariate function g and two vectors c and d we study the discrete optimization problem of finding a vertex of x that maximizes the utility function c x g d x this problem has numerous applications in combinatorial optimization with a probabilistic objective including estimation of project duration with stochastic times in reliability models in multinomial logit models and in robust optimization we show that the problem is x1d4a9 x1d4ab hard for any strictly concave function g even for simple polytopes such as the uniform matroid assignment and path polytopes and propose a 1 2 approximation algorithm for it we discuss improvements for special cases where g is the square root log utility negative exponential utility and multinomial logit probability function in particular for the square root function the approximation ratio is 4 5 we also propose a 1 25 approximation algorithm for a class of minimization problems in which the maximization of the utility func

a remark on decomposition of continuous functions abstract in this note we show that given a real 1 2 any real valued continuous function on the unit interval can be decomposed as a sum of two real valued continuous functions whose graphs have hausdorff dimension which answers affirmatively a question raised by bayart and heurteaux 2012 2

graphs of continuous functions and packing dimension abstract we prove that for a typical continuous function f c x over an uncountable compact metric space x the packing dimension of its graph g f x f x x x is dim p x 1 we also consider decompositions of functions in c 0 1 in terms of upper box dimension as well as packing dimension which are quite different from the case of hausdorff dimension

linearly parameterized bandits we consider bandit problems involving a large possibly infinite collection of arms in which the expected reward of each arm is a linear function of an r dimensional random vector z rr where r 2 the objective is to minimize the cumulative regret and bayes risk when the set of arms corresponds to the unit sphere we prove that the regret and bayes risk is of order r t by establishing a lower bound for an arbitrary policy and showing that a matching upper bound is obtained through a policy that alternates between exploration and exploitation phases the phase based policy is also shown to be effective if the set of arms satisfies a strong convexity condition for the case of a general set of arms we describe a near optimal policy whose regret and bayes risk admit upper bounds of the form o r t log3 2t

using confidence bounds for exploitation exploration trade offs we show how a standard tool from statistics namely confidence bounds can be used to elegantly deal with situations which exhibit an exploitation exploration trade off our technique for designing and analyzing algorithms for such situations is general and can be applied when an algorithm has to make exploitation versus exploration decisions based on uncertain information provided by a random process we apply our technique to two models with such an exploitation exploration trade off for the adversarial bandit problem with shifting our new algorithm suffers only o st 1 2 regret with high probability over t trials with s shifts such a regret bound was previously known only in expectation the second model we consider is associative reinforcement learning with linear value functions for this model our technique improves the regret from o t3 4 to o t1 2