Artificial Constraints and Hints for Unbounded Online Learning

We provide algorithms that guarantees regret RT (u) ≤ Õ(G‖u‖ +G(‖u‖+ 1) √ T ) or RT (u) ≤ Õ(G‖u‖T 1/3 + GT 1/3 + G‖u‖ √ T ) for online convex optimization with G-Lipschitz losses for any comparison point u without prior knowledge of either G or ‖u‖. Previous algorithms dispense with the O(‖u‖) term at the expense of knowledge of one or both of these parameters, while a lower bound shows that some additional penalty term over G‖u‖ √ T is necessary. Previous penalties were exponential while our bounds are polynomial in all quantities. Further, given a known bound ‖u‖ ≤ D, our same techniques allow us to design algorithms that adapt optimally to the unknown value of ‖u‖ without requiring knowledge of G. 1. Unconstrained Online Convex Optimization Online convex optimization (OCO) is a popular theoretical framework for designing algorithms that operate on streams of input data (Shalev-Shwartz, 2011; Zinkevich, 2003). Such problems abound in today’s world of extremely large datasets, and so many empirically successful algorithms are analyzed in the OCO framework (e.g. (Duchi et al., 2010; Ross et al., 2013; McMahan et al., 2013)). In detail, OCO is a game consisting of T rounds. In each round, the learning algorithm first outputs a point wt in some Hilbert space W 1, and then the environment outputs a convex loss function `t : W → R. The learner’s goal is to minimize the regret, which is the total loss suffered by the learner in comparison to the loss suffered at some benchmark point ẘ ∈W : RT (ẘ) = T ∑ t=1 `t(wt)− `t(ẘ) Online learning algorithms can be naturally applied to stochastic optimization problems, in which each `t is an i.i. d. random variable with E[`t] = L for some fixed loss functionL. In this case, the online-to-batch conversion argument (Cesa-Bianchi et al., 2004) tells us that E[L(w)−L(ẘ)] ≤ E[RT (ẘ)] T , where w = ∑T t=1 wt t . Thus, we refer to RT (ẘ) T as the convergence rate. We wish to achieve a convergence rate such that limT→∞ RT (ẘ) T = 0, known as sublinear regret. We can reduce OCO to online linear optimization (OLO) in which each loss `t must be linear through the use of subgradients: if gt ∈ ∂`t(wt), we have RT (ẘ) ≤ T ∑ t=1 〈gt, wt − ẘ〉 (1) 1. Our results also apply in Banach spaces, but we focus our exposition on Hilbert spaces for simplicity c © 2019 A. Cutkosky. ARTIFICIAL CONSTRAINTS AND HINTS FOR UNBOUNDED ONLINE LEARNING Thus by supplying the linear lossesw 7→ 〈gt, w〉 to an OLO algorithm, we obtain an OCO algorithm. Because it is often easy to compute gradients, many of the popular OCO algorithms are in fact OLO algorithms. We will also take this road, and consider exclusively the case that losses are linear functions specified by the subgradients gt so that (1) is an equality. Our goal is to design algorithms which take no data-dependent parameters as input and yet nevertheless guarantee regret matching the minimax optimal regret for online linear optimization algorithms, sometimes called “parameter-free” algorithms (Abernethy et al., 2008; McMahan and Streeter, 2012; Orabona, 2013; Foster et al., 2015, 2017). Specifically, we want to obtain: RT (ẘ) = O ‖ẘ‖ √√√√ T ∑ t=1 ‖gt‖? log (GT + 1)  = O ( ‖ẘ‖G √ T log (GT + 1) ) (2) for all ẘ simultaneously, where G = max ‖gt‖? , ‖ · ‖ is some norm and ‖ · ‖? is the dual norm. Classical gradient-descent algorithms require learning rates that are tuned to the values of ‖ẘ‖ and ‖gt‖? , while parameter-free algorithms automatically adapt to these unknown parameters, and so can largely dispense with tuning. 1.1. Previous Progress and Lower Bounds Previous approaches to designing parameter-free algorithms typically relax the problem by assuming either a bound D on ‖ẘ‖, or a bound G on G = maxt ‖gt‖? In the case of known value for D, classical approaches based on Follow-the-Regularized-Leader or Mirror Descent with stronglyconvex potentials can obtain regret O ( D √∑T t=1 ‖gt‖ ) (Duchi et al., 2010; Hazan et al., 2008; Orabona and Pál, 2018). In the alternative when a bound G ≤ G is known, the problem seems somewhat harder. Most prior work in this setting instead obtains the slightly-less-good regret bound RT (ẘ) = Õ ( ‖ẘ‖ √ G ∑T t=1 ‖gt‖? ) (Orabona, 2014; Orabona and Tommasi, 2017; Cutkosky and Boahen, 2017), and the better rate of Õ ( ‖ẘ‖ √ G2 + ∑T t=1 ‖gt‖? ) has only recently been developed (Cutkosky and Orabona, 2018). When neither ‖ẘ‖ nor G is known, prior lower bounds show that it is actually impossible to obtain the desired result (Cutkosky and Boahen, 2016, 2017). Instead, there is a frontier of lowerbounds trading off between the “ideal” Õ(‖ẘ‖G √ T ) term and an exponential penalty of up to O(exp(T )/T ), depending on how fast the gradients grow2. We will improve upon this background in two ways. First, observe that there is a curious asymmetry between the results for known D and known G. On the one hand, in the known D case the algorithms do not adapt very well to the true value of ‖ẘ‖, instead obtaining a regret bound proportional to DG √ T . On the other hand, in the known G case, so long as G is not greater than G √ T , the resulting bound depends on ‖ẘ‖G √ T . So in a sense, the known G algorithms are more robust than than the knownD ones. We close this gap by providing an algorithm that takes an upper bound 2. It is possible to sacrifice the near-linear dependence on ‖ẘ‖ to obtain a bound of ‖ẘ‖G √ T without any knowledge (Orabona and Pál, 2018).