# Introduction to Online Convex Optimization - Second Edition

# List of Symbols {#list-of-symbols .unnumbered}

### General {#general .unnumbered}

  ----------------------------------- --------------------------------------------------
  $\stackrel{\text{\tiny def}}{=}$    definition
  $\mathop{\mathrm{\arg\min}}\{ \}$   the argument minimizing the expression in braces
  $[n]$                               the set of integers $\{1,2,\ldots,n\}$
  ----------------------------------- --------------------------------------------------

### Geometry and Calculus {#geometry-and-calculus .unnumbered}

  ---------------------------- ----------------------------------------------------------------------------------------------------
  ${\mathbb R}^d$              $d$ dimensional Euclidean space
  $\Delta_d$                   $d$ dimensional simplex, $\{ \sum_i \ensuremath{\mathbf x}_i=1, \ensuremath{\mathbf x}_i \geq 0\}$
  $\ensuremath{\mathbb {S}}$   $d$ dimensional sphere, $\{ \|\ensuremath{\mathbf x}\| =1\}$
  $\mathbb{B}$                 $d$ dimensional ball, $\{ \|\ensuremath{\mathbf x}\| \leq 1 \}$
  ${\mathbb R}$                real numbers
  $\mathbb{C}$                 complex numbers
  $|A|$                        determinant of matrix $A$
  ---------------------------- ----------------------------------------------------------------------------------------------------

### Learning Theory {#learning-theory .unnumbered}

  -------------------------------- ---------------------------------------------------------
  ${\mathcal X},{\mathcal Y}$      feature/label sets
  ${\mathcal D}$                   distribution over examples $(\ensuremath{\mathbf x},y)$
  ${\mathcal H}$                   hypothesis class in ${\mathcal X}\mapsto Y$
  $h$                              single hypothesis $h \in {\mathcal H}$
  $m$                              training set size
  $\mathop{\mbox{\rm error}}(h)$   generalization error of hypothesis $h \in {\mathcal H}$
  -------------------------------- ---------------------------------------------------------

### Optimization {#optimization .unnumbered}

  -------------------------------- ---------------------------------------------------------------------------------------------------------------
  $\ensuremath{\mathbf x}$         vectors in the decision set
  $\ensuremath{\mathcal K}$        decision set
  $\nabla^k f$                     the $k$'th differential of $f$; note $\nabla^k f \in {\mathbb R}^{d^k}$
  $\nabla^{-2} f$                  the inverse Hessian of $f$
  $\nabla f$                       the gradient of $f$
  $\nabla_t$                       the gradient of $f$ at point $\ensuremath{\mathbf x}_t$
  $\ensuremath{\mathbf x}^\star$   the global or local optima of objective $f$
  $h_t$                            objective value distance to optimality, $h_t = f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^\star)$
  $d_t$                            Euclidean distance to optimality $d_t =  \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^\star\|$
  $G$                              upper bound on norm of subgradients
  $D$                              upper bound on Euclidean diameter
  $D_p,G_p$                        upper bound on the $p$-norm of the subgradients/diameter
  -------------------------------- ---------------------------------------------------------------------------------------------------------------

### Regularization {#regularization .unnumbered}

  -------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  $R$                                                      strongly convex and smooth regularization function
  $B_R(\ensuremath{\mathbf x}|| \ensuremath{\mathbf y})$   $R$-Bregman-divergence $R(\ensuremath{\mathbf x}) - R(\ensuremath{\mathbf y}) - \nabla R(\ensuremath{\mathbf y})^\top (\ensuremath{\mathbf x}-\ensuremath{\mathbf y})$
  $G_R$                                                    upper bound on norm of (sub)gradients
  $D_R^2$                                                  squared $R$ diameter $\max_{\ensuremath{\mathbf x},\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}} \{ R(\ensuremath{\mathbf x}) - R(\ensuremath{\mathbf y}) \}$
  $\| \ensuremath{\mathbf x}\|_A^2$                        squared matrix norm $\ensuremath{\mathbf x}^\top A \ensuremath{\mathbf x}$
  $\| \ensuremath{\mathbf x}\|_\ensuremath{\mathbf y}^2$   local norm according to local regularization $\ensuremath{\mathbf x}^\top \nabla^2 R(\ensuremath{\mathbf y}) \ensuremath{\mathbf x}$
  $\| \ensuremath{\mathbf x}\|^*$                          dual norm to $\| \ensuremath{\mathbf x}\|$
  -------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------

# Introduction {#chap:intro}

This book considers *optimization as a process*. In many practical
applications, the environment is so complex that it is not feasible to
lay out a comprehensive theoretical model and use classical algorithmic
theory and mathematical optimization. It is necessary, as well as
beneficial, to take a robust approach, by applying an optimization
method that learns as more aspects of the problem are observed. This
view of optimization as a process has become prominent in various
fields, which has led to spectacular successes in modeling and systems
that are now part of our daily lives.

The growing body of literature of machine learning, statistics, decision
science, and mathematical optimization blurs the classical distinctions
between deterministic modeling, stochastic modeling, and optimization
methodology. We continue this trend in this book, studying a prominent
optimization framework whose precise location in the mathematical
sciences is unclear: the framework of *online convex optimization*
(OCO), which was first defined in the machine learning literature (see
section [1.4](#sec:bib-of-sec-1){reference-type="ref"
reference="sec:bib-of-sec-1"}, later in this chapter). The metric of
success is borrowed from game theory, and the framework is closely tied
to statistical learning theory and convex optimization.

We embrace these fruitful connections and, on purpose, do not try to use
any particular jargon in the discussion. Rather, this book will start
with actual problems that can be modeled and solved via OCO. We will
proceed to present rigorous definitions, backgrounds, and algorithms.
Throughout, we provide connections to the literature in other fields. It
is our hope that you, the reader, will contribute to our understanding
of these connections from your domain of expertise, and expand the
growing amount of literature on this fascinating subject.

## The Online Convex Optimization Setting {#section:formaldef}

In OCO, an online player iteratively makes decisions. At the time of
each decision, the outcome or outcomes associated with it are unknown to
the player.

After committing to a decision, the decision maker suffers a loss: every
possible decision incurs a (possibly different) loss. These losses are
unknown to the decision maker beforehand. The losses can be
adversarially chosen, and even depend on the action taken by the
decision maker.

Already at this point, several restrictions are necessary in order for
this framework to make any sense at all:

-   The losses determined by an adversary should not be allowed to be
    unbounded . Otherwise, the adversary could keep decreasing the scale
    of the loss at each step, and never allow the algorithm to recover
    from the loss of the first step. Thus, we assume that the losses lie
    in some bounded region.

-   The decision set must be somehow bounded and/or structured, though
    not necessarily finite.

    To see why this is necessary, consider decision making with an
    infinite set of possible decisions. An adversary can assign high
    loss to all the strategies chosen by the player indefinitely, while
    setting apart some strategies with zero loss. This precludes any
    meaningful performance metric.

Surprisingly, interesting statements and algorithms can be derived with
not much more than these two restrictions. The online convex
optimization (OCO) framework models the decision set as a convex set in
Euclidean space denoted as $\ensuremath{\mathcal K}
\subseteq {\mathbb R}^n$. The costs are modeled as bounded convex
functions over $\ensuremath{\mathcal K}$.

The OCO framework can be seen as a structured repeated game. The
protocol of this learning framework is as follows.

At iteration $t$, the online player chooses
$\ensuremath{\mathbf x}_t \in \ensuremath{\mathcal K}$ . After the
player has committed to this choice, a convex cost function
$f_t \in {\mathcal F}: \ensuremath{\mathcal K}\mapsto {\mathbb R}$ is
revealed. Here, ${\mathcal F}$ is the bounded family of cost functions
available to the adversary. The cost incurred by the online player is
$f_t(\ensuremath{\mathbf x}_t)$, the value of the cost function for the
choice $\ensuremath{\mathbf x}_t$. Let $T$ denote the total number of
game iterations.

What would make an algorithm a good OCO algorithm? As the framework is
game-theoretic and adversarial in nature, the appropriate performance
metric also comes from game theory: define the *regret* of the decision
maker to be the difference between the total cost she has incurred and
that of the best fixed decision in hindsight. In OCO, we are usually
interested in an upper bound on the worst-case regret of an algorithm.

Let ${\mathcal A}$ be an algorithm for OCO, which maps a certain game
history to a decision in the decision set:
$$\ensuremath{\mathbf x}_t^{\mathcal A}= {\mathcal A}(f_1,...,f_{t-1}) \in \ensuremath{\mathcal K}.$$
We formally define the regret of ${\mathcal A}$ after $T$ iterations as:
$$\label{eqn:regret-defn}
\ensuremath{\mathrm{{Regret}}}_T({\mathcal A}) = \sup_{\{f_1,...,f_T\} \subseteq {\mathcal F}}  \left\{   \sum_{t=1}^T f_t(\ensuremath{\mathbf x}_t^{\mathcal A}) -\min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \sum_{t=1}^T f_t(\ensuremath{\mathbf x}) \right\} .$$

If the algorithm is clear from the context, we henceforth omit the
superscript and denote the algorithm's decision at time $t$ simply as
$\ensuremath{\mathbf x}_t$. Intuitively, an algorithm performs well if
its regret is sublinear as a function of $T$ (i.e.
$\ensuremath{\mathrm{{Regret}}}_T({\mathcal A}) = o(T)$), since this
implies that on average, the algorithm performs as well as the best
fixed strategy in hindsight.

The running time of an algorithm for OCO is defined to be the worst-case
expected time to produce $\ensuremath{\mathbf x}_t$, for an iteration
$t \in [T]$ in a $T$-iteration repeated game. Typically, the running
time will depend on $n$ (the dimensionality of the decision set
$\mathcal{K}$), $T$ (the total number of game iterations), and the
parameters of the cost functions and underlying convex set.

## Examples of Problems That Can Be Modeled via Online Convex Optimization  {#subsec:OCOexamples}

Perhaps the main reason that OCO has become a leading online learning
framework in recent years is its powerful modeling capability: problems
from diverse domains such as online routing, ad selection for search
engines, and spam filtering can all be modeled as special cases. In this
section, we briefly survey a few special cases and how they fit into the
OCO framework.

### Prediction from expert advice

Perhaps the most well known problem in prediction theory is the *experts
problem*. The decision maker has to choose among the advice of $n$ given
experts. After making her choice, a loss between zero and $1$ is
incurred. This scenario is repeated iteratively, and at each iteration,
the costs of the various experts are arbitrary (and possibly even
adversarial, trying to mislead the decision maker). The goal of the
decision maker is to do as well as the best expert in hindsight.

The OCO setting captures this as a special case: the set of decisions is
the set of all distributions over $n$ elements (experts); that is, the
$n$-dimensional simplex
$\ensuremath{\mathcal K}= \Delta_n = \{ \ensuremath{\mathbf x}\in {\mathbb R}^n \  , \ \sum_i \ensuremath{\mathbf x}_i = 1 \  ,  \  \ensuremath{\mathbf x}_i \geq 0\}$.
Let the cost of the $i$th expert at iteration $t$ be
$\ensuremath{\mathbf g_{t}}(i)$, and let $\ensuremath{\mathbf g_{t}}$ be
the cost vector of all $n$ experts. Then the cost function is the
expected cost of choosing an expert according to distribution
$\ensuremath{\mathbf x}$, and it is given by the linear function
$f_t(\ensuremath{\mathbf x}) = \ensuremath{\mathbf g_{t}}^\top \ensuremath{\mathbf x}$.

Thus, prediction from expert advice is a special case of OCO, in which
the decision set is the simplex and the cost functions are linear and
bounded, in the $\ell_\infty$ norm, to be at most $1$. The bound on the
cost functions is derived from the bound on the elements of the cost
vector $\ensuremath{\mathbf g_{t}}$.

The fundamental importance of the experts problem in machine learning
warrants special attention, and we shall return to it and analyze it in
detail at the end of this chapter.

### Online spam filtering

Consider an online spam-filtering system. Repeatedly, emails arrive in
the system and are classified as spam or valid. Obviously, such a system
has to cope with adversarially generated data and dynamically change
with the varying input---a hallmark of the OCO model.

The linear variant of this model is captured by representing the emails
as vectors according to the "bag-of-words" representation. Each email is
represented as a vector $\mathbf{a}\in {\mathbb R}^d$, where $d$ is the
number of words in the dictionary. The entries of this vector are all
zero, except for those coordinates that correspond to words appearing in
the email, which are assigned the value one.

To predict whether an email is spam, we learn a filter, for example a
vector $\ensuremath{\mathbf x}\in {\mathbb R}^d$. Usually a bound on the
Euclidean norm of this vector is decided upon a priori, and is a
parameter of great importance in practice.

Classification of an email $\mathbf{a}\in {\mathbb R}^d$ by a filter
$\ensuremath{\mathbf x}\in {\mathbb R}^d$ is given by the sign of the
inner product between these two vectors, i.e.,
$\hat{b} = \mathop{\mbox{\rm sign}}(  \ensuremath{\mathbf x}^\top \mathbf{a})$
(with, for example, $+1$ meaning valid and $-1$ meaning spam).

In the OCO model of online spam filtering, the decision set is taken to
be the set of all such norm-bounded linear filters, i.e., the Euclidean
ball of a certain radius. The cost functions are determined according to
a stream of incoming emails arriving into the system, and their labels
(which may be known by the system, partially known, or not known at
all). Let $(\mathbf{a},b)$ be an email/label pair. Then the
corresponding cost function over filters is given by
$f(\ensuremath{\mathbf x}) = \ell( \hat{b},b)$. Here $\hat{b}$ is the
classification given by the filter $\ensuremath{\mathbf x}$, $b$ is the
true label, and $\ell$ is a convex loss function, for example, the
scaled square loss $\ell (\hat{b},b) = \frac{1}{4}(\hat{b} - b)^2$.

At this point the reader may wonder - why use a square loss rather than
any other function? The most natural choice being perhaps a loss of one
if $b = \hat{b}$ and zero otherwise.

To answer this, notice first that if both $b$ and $\hat{b}$ are binary
and in $\{-1,1\}$, then the square loss is indeed one or zero. However,
moving to a continuous function allows us much more flexibility in the
decision making process. We can allow, for example, the algorithm to
return a number in the interval $[-1,1]$ depending on its confidence.

Another reason has to do with the algorithmic efficiency of finding a a
good solution. This will be the subject of future chapters.

### Online shortest paths

In the online shortest path problem, the decision maker is given a
directed graph $G=(V,E)$ and a source-sink pair $u,v \in V$. At each
iteration $t \in [T]$, the decision maker chooses a path
$p_t \in {\mathcal P}_{u,v}$, where
${\mathcal P}_{u,v} \subseteq E^{|V|}$ is the set of all $u$-$v$-paths
in the graph. The adversary independently chooses weights (lengths) on
the edges of the graph, given by a function from the edges to the real
numbers $\mathbf{w}_t: E \mapsto {\mathbb R}$, which can be represented
as a vector $\mathbf{w}_t \in {\mathbb R}^m$, where $m=|E|$. The
decision maker suffers and observes a loss, which is the weighted length
of the chosen path $\sum_{e \in p_t} \mathbf{w}_t(e)$.

The discrete description of this problem as an experts problem, where we
have an expert for each path, presents an efficiency challenge. There
are potentially exponentially many paths in terms of the graph
representation size.

Alternatively, the online shortest path problem can be cast in the
online convex optimization framework as follows. Recall the standard
description of the set of all distributions over paths (flows) in a
graph as a convex set in ${\mathbb R}^{m}$, with $O(m+|V|)$ constraints
(figure [\[flow polytope\]](#flow polytope){reference-type="ref"
reference="flow polytope"}). Denote this flow polytope by
$\ensuremath{\mathcal K}$. The expected cost of a given flow
$\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}$ (distribution over
paths) is then a linear function, given by
$f_t(\ensuremath{\mathbf x}) = \mathbf{w}_t^\top \ensuremath{\mathbf x}$,
where, as defined above, $\mathbf{w}_t(e)$ is the length of the edge
$e \in E$. This inherently succinct formulation leads to computationally
efficient algorithms.

$$\begin{aligned}
&  \sum_{ e = (u,w) , w\in V} \ensuremath{\mathbf x}_{e} = 1 = \sum_{ e = (w,v), w \in V } \ensuremath{\mathbf x}_{e}  & \mbox{ flow value is one} \\
& \forall w \in V \setminus \{u,v\}  \ \   \sum_{e = (w,x) \in E } \ensuremath{\mathbf x}_{e} = \sum_{e = (x,w) \in E } \ensuremath{\mathbf x}_{e}  & \mbox{ flow conservation} \\
& \forall e \in E   \ \  0 \leq \ensuremath{\mathbf x}_{e} \leq 1  &  \mbox{ capacity constraints}
\end{aligned}$$

### Portfolio selection {#section:portfolios}

In this section we consider a portfolio selection model that does not
make any statistical assumptions about the stock market (as opposed to
the standard geometric Brownian motion model for stock prices), and is
called the "universal portfolio selection" model.

At each iteration $t\in[T]$, the decision maker chooses a distribution
of her wealth over $n$ assets $\ensuremath{\mathbf x_{t}}\in \Delta_n$.
The adversary independently chooses market returns for the assets, i.e.,
a vector $\ensuremath{\mathbf r_{t}}\in {\mathbb R}^n$ with strictly
positive entries such that each coordinate
$\ensuremath{\mathbf r_{t}}(i)$ is the price ratio for the $i$'th asset
between the iterations $t$ and $t+1$. The ratio between the wealth of
the investor at iterations $t+1$ and $t$ is
$\ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x_{t}}$, and hence
the gain in this setting is defined to be the logarithm of this change
ratio in wealth
$\log (\ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x_{t}})$.
Notice that since $\ensuremath{\mathbf x_{t}}$ is the distribution of
the investor's wealth, even if
$\ensuremath{\mathbf x_{t+1}} =\ensuremath{\mathbf x_{t}}$, the investor
may still need to trade to adjust for price changes.

The goal of regret minimization, which in this case corresponds to
minimizing the difference
$\max_{\ensuremath{\mathbf x}^\star \in \Delta_n} {\textstyle \sum}_{t=1}^T \log(\ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x}^\star) -  {\textstyle \sum}_{t=1}^T \log(\ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x}_t)$,
has an intuitive interpretation. The first term is the logarithm of the
wealth accumulated by the best possible in-hindsight distribution
$\ensuremath{\mathbf x}^\star$. Since this distribution is fixed, it
corresponds to a strategy of rebalancing the position after every
trading period, and hence, is called a *constant rebalanced portfolio*.
The second term is the logarithm of the wealth accumulated by the online
decision maker. Hence regret minimization corresponds to maximizing the
ratio of the investor's wealth to the wealth of the best benchmark from
a pool of investing strategies.

A *universal* portfolio selection algorithm is defined to be one that,
in this setting, attains regret converging to zero. Such an algorithm,
albeit requiring exponential time, was first described by Cover (see
bibliographic notes at the end of this chapter). The online convex
optimization framework has given rise to much more efficient algorithms
based on Newton's method. We shall return to study these in detail in
chapter [4](#chap:second order-methods){reference-type="ref"
reference="chap:second order-methods"}.

### Matrix completion and recommendation systems

The prevalence of large-scale media delivery systems such as the Netflix
online video library, Spotify music service and many others, give rise
to very large scale recommendation systems. One of the most popular and
successful models for automated recommendation is the matrix completion
model.

In this mathematical model, recommendations are thought of as composing
a matrix. The customers are represented by the rows, the different media
are the columns, and at the entry corresponding to a particular
user/media pair we have a value scoring the preference of the user for
that particular media.

For example, for the case of binary recommendations for music, we have a
matrix $X \in \{0,1\}^{n \times m}$ where $n$ is the number of persons
considered, $m$ is the number of songs in our library, and $0/1$
signifies dislike/like respectively: $$X_{ij} = {
\left\{
\begin{array}{ll}
    {0}, & {\mbox{person $i$ dislikes song $j$}} \\\\
    {1}, & {\mbox{person $i$ likes song $j$}}
\end{array}
\right. } .$$

In the online setting, for each iteration the decision maker outputs a
preference matrix $X_t \in \ensuremath{\mathcal K}$, where
$\ensuremath{\mathcal K}\subseteq \{0,1\}^{n \times m}$ is a subset of
all possible zero/one matrices. An adversary then chooses a user/song
pair $(i_t,j_t)$ along with a "real" preference for this pair
$y_t \in \{0,1\}$. Thus, the loss experienced by the decision maker can
be described by the convex loss function,
$$f_t(X) = ( X_{i_t,j_t} - y_t)^2 .$$

The natural comparator in this scenario is a low-rank matrix, which
corresponds to the intuitive assumption that preference is determined by
few unknown factors. Regret with respect to this comparator means
performing, on the average, as few preference-prediction errors as the
best low-rank matrix.

We return to this problem and explore efficient algorithms for it in
chapter [7](#chap:FW){reference-type="ref" reference="chap:FW"}.

## A Gentle Start: Learning from Expert Advice {#sec:experts}

Consider the following fundamental iterative decision making problem:

At each time step $t=1,2,\ldots,T$, the decision maker faces a choice
between two actions $A$ or $B$ (i.e., buy or sell a certain stock). The
decision maker has assistance in the form of $N$ "experts" that offer
their advice. After a choice between the two actions has been made, the
decision maker receives feedback in the form of a loss associated with
each decision. For simplicity one of the actions receives a loss of zero
(i.e., the "correct" decision) and the other a loss of one.

We make the following elementary observations:

1.  A decision maker that chooses an action uniformly at random each
    iteration, trivially attains a loss of $\frac{T}{2}$ and is
    "correct" $50\%$ of the time.

2.  In terms of the number of mistakes, no algorithm can do better in
    the worst case! In a later exercise, we will devise a randomized
    setting in which the expected number of mistakes of any algorithm is
    at least $\frac{T}{2}$.

We are thus motivated to consider a *relative performance metric*: can
the decision maker make as few mistakes as the best expert in hindsight?
The next theorem shows that the answer in the worst case is negative for
a deterministic decision maker.

::: theorem
**Theorem 1.1**. *Let $L \leq \frac{T} {2}$ denote the number of
mistakes made by the best expert in hindsight. Then there does not exist
a deterministic algorithm that can guarantee less than $2L$ mistakes.*
:::

::: proof
*Proof.* Assume that there are only two experts and one always chooses
option $A$ while the other always chooses option $B$. Consider the
setting in which an adversary always chooses the opposite of our
prediction (she can do so, since our algorithm is deterministic). Then,
the total number of mistakes the algorithm makes is $T$. However, the
best expert makes no more than $\frac{T}{2}$ mistakes (at every
iteration exactly one of the two experts is mistaken). Therefore, there
is no algorithm that can always guarantee less than $2L$ mistakes. ◻
:::

This observation motivates the design of random decision making
algorithms, and indeed, the OCO framework gracefully models decisions on
a continuous probability space. Henceforth we prove Lemmas
[1.3](#lem:wm){reference-type="ref" reference="lem:wm"} and
[1.4](#lem:rwm){reference-type="ref" reference="lem:rwm"} that show the
following:

::: theorem
**Theorem 1.2**. *Let $\varepsilon\in (0,\frac{1}{2} )$. Suppose the
best expert makes $L$ mistakes. Then:*

1.  *There is an efficient deterministic algorithm that can guarantee
    less than $2(1+\varepsilon)L + \frac{2\log N}{\varepsilon}$
    mistakes;*

2.  *There is an efficient randomized algorithm for which the expected
    number of mistakes is at most
    $(1+\varepsilon)L  + \frac{\log N}{\varepsilon}$.*
:::

### The weighted majority algorithm

The weighted majority (WM) algorithm is intuitive to describe: each
expert $i$ is assigned a weight $W_t(i)$ at every iteration $t$.
Initially, we set $W_1(i) = 1$ for all experts $i \in [N]$. For all
$t \in [T]$ let $S_t(A),S_t(B) \subseteq [N]$ be the set of experts that
choose $A$ (and respectively $B$) at time $t$. Define,
$$W_t(A) = \smashoperator[r]{\sum_{i \in S_t(A)}} W_t(i) \qquad  \qquad W_t(B) = \smashoperator[r]{\sum_{i \in S_t(B)}} W_t(i)$$
and predict according to $$a_t =
\begin{cases}
A & \text{if $W_t(A) \ge W_t(B)$}\\
B & \text{otherwise.}
\end{cases}$$ Next, update the weights $W_t(i)$ as follows:
$$W_{t+1}(i) =
\begin{cases}
W_t(i) & \text{if expert $i$ was correct}\\
W_t(i)  (1-\varepsilon) & \text{if expert $i$ was wrong}
\end{cases}
,$$ where $\varepsilon$ is a parameter of the algorithm that will affect
its performance. This concludes the description of the WM algorithm. We
proceed to bound the number of mistakes it makes.

::: {#lem:wm .lemma}
**Lemma 1.3**. *Denote by $M_t$ the number of mistakes the algorithm
makes until time $t$, and by $M_t(i)$ the number of mistakes made by
expert $i$ until time $t$. Then, for any expert $i \in [N]$ we have
$$M_T \le 2(1+\varepsilon)M_T(i) + \frac{2\log N}{\varepsilon} .$$*
:::

We can optimize $\varepsilon$ to minimize the above bound. The
expression on the right hand side is of the form $f(x)=ax+b/x$, that
reaches its minimum at $x=\sqrt{b/a}$. Therefore the bound is minimized
at $\varepsilon^\star = \sqrt{\log N/M_T(i)}$. Using this optimal value
of $\varepsilon$, we get that for the best expert $i^\star$
$$M_T \le 2M_T(i^\star) + O\left(\sqrt {M_T(i^\star)\log N}\right).$$ Of
course, this value of $\varepsilon^\star$ cannot be used in advance
since we do not know which expert is the best one ahead of time (and
therefore we do not know the value of $M_T(i^\star)$). However, we shall
see later on that the same asymptotic bound can be obtained even without
this prior knowledge.

Let us now prove Lemma [1.3](#lem:wm){reference-type="ref"
reference="lem:wm"}.

::: proof
*Proof.* Let $\Phi_t= \sum_{i=1}^N W_t(i)$ for all $t \in [T]$, and note
that $\Phi_1=N$.

Notice that $\Phi_{t+1} \le \Phi_t$. However, on iterations in which the
WM algorithm erred, we have
$$\Phi_{t+1} \le \Phi_t(1-\frac{\varepsilon}{2}) ,$$ the reason being
that experts with at least half of total weight were wrong (else WM
would not have erred), and therefore
$$\Phi_{t+1} \le  \frac{1}{2} \Phi_t(1-\varepsilon) + \frac {1} {2} \Phi_t =\Phi_t(1-\frac {\varepsilon}{2}) .$$
From both observations,
$$\Phi_{t} \le \Phi_1 (1-\frac{\varepsilon}{2})^{M_t} = N (1-\frac{\varepsilon}{2})^{M_t} .$$
On the other hand, by definition we have for any expert $i$ that
$$W_T(i) = (1-\varepsilon)^{M_T(i)} .$$ Since the value of $W_T(i)$ is
always less than the sum of all weights $\Phi_T$, we conclude that
$$(1-\varepsilon)^{M_T(i)} = W_T(i) \le \Phi_T \le N(1-\frac{\varepsilon}{2})^{M_T}.$$
Taking the logarithm of both sides we get
$$M_T(i)\log(1-\varepsilon) \le \log{N} + M_T\log{(1-\frac{\varepsilon}{2})}  .$$
Next, we use the approximations
$$-x-x^2 \le \log{(1-x)} \le -x  \qquad  \quad 0 < x < \frac{1}{2},$$
which follow from the Taylor series of the logarithm function, to obtain
that
$$-M_T(i)(\varepsilon+\varepsilon^2) \le \log{N} - M_T\frac {\varepsilon}{2} ,$$
and the lemma follows. ◻
:::

### Randomized weighted majority

In the randomized version of the WM algorithm, denoted RWM, we choose
expert $i$ w.p. $p_t(i) = W_t(i) /  \sum_{j=1}^N W_t(j)$ at time $t$.

::: {#lem:rwm .lemma}
**Lemma 1.4**. *Let $M_t$ denote the number of mistakes made by RWM
until iteration $t$. Then, for any expert $i \in [N]$ we have
$$\mathop{\mbox{\bf E}}[ M_T]  \le (1+\varepsilon)M_T(i) + \frac{\log N}{\varepsilon} .$$*
:::

The proof of this lemma is very similar to the previous one, where the
factor of two is saved by the use of randomness:

::: proof
*Proof.* As before, let $\Phi_t= \sum_{i=1}^N W_t(i)$ for all
$t \in [T]$, and note that $\Phi_1=N$. Let $\tilde{m}_t = M_t - M_{t-1}$
be the indicator variable that equals one if the RWM algorithm makes a
mistake on iteration $t$. Let $m_t(i)$ equal one if the $i$'th expert
makes a mistake on iteration $t$ and zero otherwise. Inspecting the sum
of the weights: $$\begin{aligned}
\Phi_{t+1} & = \sum_i W_t(i) (1 - \varepsilon m_t(i)) \\
& = \Phi_t (1 - \varepsilon\sum_i p_t(i) m_t(i)) & \mbox{ $p_t(i) = \frac{W_t(i)}{\sum_j W_t(j) }$} \\
& = \Phi_t ( 1 - \varepsilon\mathop{\mbox{\bf E}}[\tilde{m}_t ]) \\
& \leq \Phi_t e^{-\varepsilon\mathop{\mbox{\bf E}}[\tilde{m}_t] }. & \mbox{ $1  + x \leq e^x $}
\end{aligned}$$ On the other hand, by definition we have for any expert
$i$ that $$W_T(i) = (1-\varepsilon)^{M_T(i)}$$ Since the value of
$W_T(i)$ is always less than the sum of all weights $\Phi_T$, we
conclude that
$$(1-\varepsilon)^{M_T(i)} = W_T(i) \le \Phi_T \le N e^{-\varepsilon\mathop{\mbox{\bf E}}[M_T]}.$$
Taking the logarithm of both sides we get
$$M_T(i)\log(1-\varepsilon) \le \log{N} - \varepsilon\mathop{\mbox{\bf E}}[ M_T]$$
Next, we use the approximation
$$-x-x^2 \le \log{(1-x)} \le -x \qquad , \quad 0 < x <  \frac{1}{2}$$ to
obtain
$$-M_T(i)(\varepsilon+\varepsilon^2) \le \log{N} - \varepsilon\mathop{\mbox{\bf E}}[M_T] ,$$
and the lemma follows. ◻
:::

### Hedge

The RWM algorithm is in fact more general: instead of considering a
discrete number of mistakes, we can consider measuring the performance
of an expert by a non-negative real number $\ell_t(i)$, which we refer
to as the *loss* of the expert $i$ at iteration $t$. The randomized
weighted majority algorithm guarantees that a decision maker following
its advice will incur an average expected loss approaching that of the
best expert in hindsight.

Historically, this was observed by a different and closely related
algorithm called Hedge, whose total loss bound will be of interest to us
later on in the book.

::: algorithm
::: algorithmic
Initialize: $\forall i\in [N], \ W_1(i) = 1$ Pick $i_t \sim_R W_t$,
i.e., $i_t = i$ with probability
$\ensuremath{\mathbf x}_t(i) = \frac{W_t(i) } {\sum_j W_t(j) }$ Incur
loss $\ell_t(i_t)$. Update weights
$W_{t+1}(i) = W_{t}(i) e^{-\varepsilon\ell_t(i)}$
:::
:::

Henceforth, denote in vector notation the expected loss of the algorithm
by
$$\mathop{\mbox{\bf E}}[ \ell_t(i_t) ] = \sum_{i=1}^N \ensuremath{\mathbf x}_t(i) \ell_t(i) = \ensuremath{\mathbf x}_t^\top \ell_t$$

::: {#lem:hedge .theorem}
**Theorem 1.5**. *Let $\ell_t^2$ denote the $N$-dimensional vector of
square losses, i.e., $\ell_t^2(i) = \ell_t(i)^2$, let $\varepsilon> 0$,
and assume all losses to be non-negative. The Hedge algorithm satisfies
for any expert $i^\star \in [N]$:
$$\sum_{t=1}^T  \ensuremath{\mathbf x}_t^\top \ell_t   \le \sum_{t=1}^T \ell_t(i^\star) + \varepsilon\sum_{t=1}^T  \ensuremath{\mathbf x}_t^\top \ell_t^2   + \frac{\log N}{\varepsilon}$$*
:::

::: proof
*Proof.* As before, let $\Phi_t= \sum_{i=1}^N W_t(i)$ for all
$t \in [T]$, and note that $\Phi_1=N$.

Inspecting the sum of weights: $$\begin{aligned}
	\Phi_{t+1} & = \sum_i W_t(i) e^{- \varepsilon\ell_t(i)}  \\
	& = \Phi_t  \sum_i \ensuremath{\mathbf x}_t(i) e^{- \varepsilon\ell_t(i)}  & \mbox{ $\ensuremath{\mathbf x}_t(i) = \frac{W_t(i)}{\sum_j W_t(j) }$} \\
	& \leq \Phi_t \sum_i \ensuremath{\mathbf x}_t(i) ( 1 - \varepsilon\ell_t(i) + \varepsilon^2 \ell_t(i)^2 ) )  & \mbox{ for $x \geq 0$, }    \\
	&   & \mbox{  $e^{-x} \leq 1 - x + x^2 $}   \\
	& = \Phi_t (  1 - \varepsilon\ensuremath{\mathbf x}_t^\top \ell_t  + \varepsilon^2 \ensuremath{\mathbf x}_t^\top \ell_t^2   )  \\
		& \leq \Phi_t e^{-\varepsilon\ensuremath{\mathbf x}_t^\top \ell_t + \varepsilon^2 \ensuremath{\mathbf x}_t^\top \ell_t^2  }. & \mbox{ $1  + x \leq e^x $}
\end{aligned}$$ On the other hand, by definition, for expert $i^\star$
we have that
$$W_{T+1}(i^\star) = e^{ -\varepsilon\sum_{t=1}^{T} \ell_t(i^\star) }$$
Since the value of $W_T(i^\star)$ is always less than the sum of all
weights $\Phi_t$, we conclude that
$$W_{T+1}(i^\star) \le \Phi_{T+1} \le N e^{-\varepsilon\sum_t  \ensuremath{\mathbf x}_t^\top \ell_{t} + \varepsilon^2 \sum_{t} \ensuremath{\mathbf x}_t^\top \ell_t^2 }.$$
Taking the logarithm of both sides we get
$$-\varepsilon\sum_{t=1}^T \ell_t(i^\star)  \le \log{N} - \varepsilon\sum_{t=1}^T  \ensuremath{\mathbf x}_t^\top \ell_t + \varepsilon^2 \sum_{t=1}^T \ensuremath{\mathbf x}_t^\top \ell_t^2$$
and the theorem follows by simplifying. ◻
:::

## Bibliographic Remarks {#sec:bib-of-sec-1}

The OCO model was first defined by @Zinkevich03 and has since become
widely influential in the learning community and significantly extended
since (see thesis and surveys
[@HazanThesis; @HazanSurvey; @shalev2011online]).

The problem of prediction from expert advice and the Weighted Majority
algorithm were devised in [@WarmuthLittlestone89; @LitWar94]. This
seminal work was one of the first uses of the multiplicative updates
method---a ubiquitous meta-algorithm in computation and learning, see
the survey [@AHK-MW] for more details. The Hedge algorithm was
introduced by @FreundSch1997.

The Universal Portfolios model was put forth in [@cover], and is one of
the first examples of a worst-case online learning model. Cover gave an
optimal-regret algorithm for universal portfolio selection that runs in
exponential time. A polynomial time algorithm was given in
[@KalaiVempalaPortfolios], which was further sped up in
[@AgarwalHKS06; @HAK07]. Numerous extensions to the model also appeared
in the literature, including addition of transaction costs
[@BlumKalaiPortfolios] and relation to the Geometric Brownian Motion
model for stock prices [@HazanKNips09].

In their influential paper, @AweKle08 put forth the application of
online convex optimization to online routing. A great deal of work has
been devoted since then to improve the initial bounds, and generalize it
into a complete framework for decision making with limited feedback.
This framework is an extension of OCO, called Bandit Convex Optimization
(BCO). We defer further bibliographic remarks to chapter
[6](#chap:bandits){reference-type="ref" reference="chap:bandits"} which
is devoted to the BCO framework.

## Exercises

# Basic Concepts in Convex Optimization {#chap:opt}

In this chapter we give a gentle introduction to convex optimization and
present some basic algorithms for solving convex mathematical programs.
Although offline convex optimization is not our main topic, it is useful
to recall the basic definitions and results before we move on to OCO.
This will help in assessing the advantages and limitations of OCO.
Furthermore, we describe some tools that will be our bread-and-butter
later on.

The material in this chapter is far from being new. A broad and
significantly more detailed literature exists, and the reader is
deferred to the bibliography at the end of this chapter for references.
We give here only the most elementary analysis, and focus on the
techniques that will be of use to us later on.

## Basic Definitions and Setup {#sec:optdefs}

The goal in this chapter is to minimize a continuous and convex function
over a convex subset of Euclidean space. Henceforth, let
$\ensuremath{\mathcal K}\subseteq {\mathbb R}^d$ be a bounded convex and
closed set in Euclidean space. We denote by $D$ an upper bound on the
diameter of $\ensuremath{\mathcal K}$:
$$\forall \ensuremath{\mathbf x},\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}, \ \|\ensuremath{\mathbf x}-\ensuremath{\mathbf y}\| \leq D.$$
A set $\ensuremath{\mathcal K}$ is convex if for any
$\ensuremath{\mathbf x},\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}$,
all the points on the line segment connecting $\ensuremath{\mathbf x}$
and $\ensuremath{\mathbf y}$ also belong to $\ensuremath{\mathcal K}$,
i.e.,
$$\forall \alpha \in [0,1]  , \ \alpha \ensuremath{\mathbf x}+ (1-\alpha)\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}.$$
A function $f: \ensuremath{\mathcal K}\mapsto {\mathbb R}$ is convex if
for any
$\ensuremath{\mathbf x},\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}$
$$\forall \alpha \in [0,1] , \  f( (1 - \alpha) \ensuremath{\mathbf x}+ \alpha \ensuremath{\mathbf y}) \leq (1- \alpha) f(\ensuremath{\mathbf x}) + \alpha f(\ensuremath{\mathbf y}).$$
This inequality, and generalizations thereof, is also known as Jensen's
inequality. Equivalently, if $f$ is differentiable, that is, its
gradient $\nabla f(\ensuremath{\mathbf x})$ exists for all
$\ensuremath{\mathbf x}\in\ensuremath{\mathcal K}$, then it is convex if
and only if
$\forall \ensuremath{\mathbf x},\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}$
$$f(\ensuremath{\mathbf y}) \geq f(\ensuremath{\mathbf x}) + \nabla f(\ensuremath{\mathbf x})^\top (\ensuremath{\mathbf y}-\ensuremath{\mathbf x}).$$
For convex and non-differentiable functions $f$, the subgradient at
$\ensuremath{\mathbf x}$ is *defined* to be any member of the set of
vectors $\{ \nabla f(\ensuremath{\mathbf x}) \}$ that satisfies the
above for all $\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}$.

We denote by $G > 0$ an upper bound on the norm of the subgradients of
$f$ over $\ensuremath{\mathcal K}$, i.e.,
$\|\nabla f(\ensuremath{\mathbf x})\| \leq G$ for all
$\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}$. Such an upper bound
implies that the function $f$ is Lipschitz continuous with parameter
$G$, that is, for all
$\ensuremath{\mathbf x},\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}$
$$|f(\ensuremath{\mathbf x}) - f(\ensuremath{\mathbf y})| \leq G \|\ensuremath{\mathbf x}-\ensuremath{\mathbf y}\|.$$

The optimization and machine learning literature studies special types
of convex functions that admit useful properties, which in turn allow
for more efficient optimization. Notably, we say that a function is
$\alpha$-strongly convex if
$$f(  \ensuremath{\mathbf y}) \geq  f(\ensuremath{\mathbf x}) + \nabla f(\ensuremath{\mathbf x})^\top (\ensuremath{\mathbf y}-\ensuremath{\mathbf x})  + \frac{\alpha}{2} \|\ensuremath{\mathbf y}-\ensuremath{\mathbf x}\|^2.$$
A function is $\beta$-smooth if
$$f(  \ensuremath{\mathbf y}) \leq  f(\ensuremath{\mathbf x}) + \nabla f(\ensuremath{\mathbf x})^\top (\ensuremath{\mathbf y}-\ensuremath{\mathbf x})  + \frac{\beta}{2} \|\ensuremath{\mathbf y}-\ensuremath{\mathbf x}\|^2.$$
The latter condition is equivalent to a Lipschitz condition over the
gradients, i.e.,
$$\| \nabla f(\ensuremath{\mathbf x}) - \nabla f(\ensuremath{\mathbf y}) \| \leq {\beta} \|\ensuremath{\mathbf x}-\ensuremath{\mathbf y}\|.$$

If the function is twice differentiable and admits a second derivative,
known as a Hessian for a function of several variables, the above
conditions are equivalent to the following condition on the Hessian,
denoted $\nabla^2 f(\ensuremath{\mathbf x})$:
$$\alpha I \preccurlyeq  \nabla^2 f(\ensuremath{\mathbf x}) \preccurlyeq \beta I,$$
where $A\preccurlyeq B$ if the matrix $B-A$ is positive semidefinite.

When the function $f$ is both $\alpha$-strongly convex and
$\beta$-smooth, we say that it is $\gamma$-well-conditioned where
$\gamma$ is the ratio between strong convexity and smoothness, also
called the *condition number* of $f$
$$\gamma = \frac{\alpha}{\beta} \leq 1$$

### Projections onto convex sets {#sec:projections}

In the following algorithms we shall make use of a projection operation
onto a convex set, which is defined as the closest point in terms of
Euclidean distance inside the convex set to a given point. Formally,
$$\mathop{\Pi}_\ensuremath{\mathcal K}(\ensuremath{\mathbf y}) \stackrel{\text{\tiny def}}{=}\mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \| \ensuremath{\mathbf x}- \ensuremath{\mathbf y}\|.$$
When clear from the context, we shall remove the
$\ensuremath{\mathcal K}$ subscript. It is left as an exercise to the
reader to prove that the projection of a given point over a closed,
bounded and convex set exists and is unique.

The computational complexity of projections is a subtle issue that
depends much on the characterization of $\ensuremath{\mathcal K}$
itself. Most generally, $\ensuremath{\mathcal K}$ can be represented by
a membership oracle---an efficient procedure that is capable of deciding
whether a given $\ensuremath{\mathbf x}$ belongs to
$\ensuremath{\mathcal K}$ or not. In this case, projections can be
computed in polynomial time. In certain special cases, projections can
be computed very efficiently in near-linear time. The computational cost
of projections, as well as optimization algorithms that avoid them
altogether, is discussed in chapter [7](#chap:FW){reference-type="ref"
reference="chap:FW"}.

A crucial property of projections that we shall make extensive use of is
the Pythagorean theorem, which we state here for completeness:

::: center
![Pythagorean theorem](images/fig_pyth.png){width="3.5in"}
:::

::: {#thm:pythagoras .theorem}
**Theorem 2.1** (Pythagoras, circa 500 BC). *Let
$\ensuremath{\mathcal K}\subseteq {\mathbb R}^d$ be a convex set,
$\ensuremath{\mathbf y}\in {\mathbb R}^d$ and
$\ensuremath{\mathbf x}= \mathop{\Pi}_\ensuremath{\mathcal K}(\ensuremath{\mathbf y})$.
Then for any $\ensuremath{\mathbf z}\in \ensuremath{\mathcal K}$ we have
$$\| \ensuremath{\mathbf y}- \ensuremath{\mathbf z}\| \geq \| \ensuremath{\mathbf x}- \ensuremath{\mathbf z}\|.$$*
:::

We note that there exists a more general version of the Pythagorean
theorem. The above theorem and the definition of projections are true
and valid not only for Euclidean norms, but for projections according to
other distances that are not norms. In particular, an analogue of the
Pythagorean theorem remains valid with respect to Bregman divergences
(see chapter [5](#chap:regularization){reference-type="ref"
reference="chap:regularization"}).

### Introduction to optimality conditions {#subsec:optimality-conditions}

The standard curriculum of high school mathematics contains the basic
facts concerning when a function (usually in one dimension) attains a
local optimum or saddle point. The generalization of these conditions to
more than one dimension is called the KKT (Karush-Kuhn-Tucker)
conditions, and the reader is referred to the bibliographic material at
the end of this chapter for an in-depth rigorous discussion of
optimality conditions in general mathematical programming.

For our purposes, we describe only briefly and intuitively the main
facts that we will require henceforth. Naturally, we restrict ourselves
to convex programming, and thus a local minimum of a convex function is
also a global minimum (see exercises at the end of this chapter). In
general there can be many points in which a function is minimized, and
thus we refer to the *set* of minima of a given objective function,
denoted as
$\mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in {\mathbb R}^n} \{ f(\ensuremath{\mathbf x})\}$
.

The generalization of the fact that a minimum of a convex differentiable
function on ${\mathbb R}$ is a point in which its derivative is equal to
zero, is given by the multi-dimensional analogue that its gradient is
zero:
$$\nabla f(\ensuremath{\mathbf x}) =  0  \ \ \Longleftrightarrow  \ \ \ensuremath{\mathbf x}\in \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in {\mathbb R}^n} \{ f(\ensuremath{\mathbf x}) \}.$$

We will require a slightly more general, but equally intuitive, fact for
constrained optimization: at a minimum point of a constrained convex
function, the inner product between the negative gradient and direction
towards the interior of $\ensuremath{\mathcal K}$ is non-positive. This
is depicted in figure [2.1](#fig:optimality){reference-type="ref"
reference="fig:optimality"}, which shows that
$-\nabla f(\ensuremath{\mathbf x}^\star)$ defines a supporting
hyperplane to $\ensuremath{\mathcal K}$. The intuition is that if the
inner product were positive, one could improve the objective by moving
in the direction of the projected negative gradient. This fact is stated
formally in the following theorem.

::: {#thm:optim-conditions .theorem}
**Theorem 2.2** (Karush-Kuhn-Tucker). *Let
$\ensuremath{\mathcal K}\subseteq {\mathbb R}^d$ be a convex set,
$\ensuremath{\mathbf x}^\star \in \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in  \ensuremath{\mathcal K}} f(\ensuremath{\mathbf x})$.
Then for any $\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}$ we have
$$\nabla f(\ensuremath{\mathbf x}^\star) ^\top ( \ensuremath{\mathbf y}- \ensuremath{\mathbf x}^\star ) \geq 0.$$*
:::

::: center
![Optimality conditions: negative subgradient pointing outwards
](images/fig_kt.jpg){#fig:optimality width="4in"}
:::

## Gradient Descent

Gradient descent (GD) is the simplest and oldest of optimization
methods. It is an *iterative method*---the optimization procedure
proceeds in iterations, each improving the objective value. The basic
method amounts to iteratively moving the current point in the direction
of the gradient, which is a linear time operation if the gradient is
given explicitly (indeed, for many functions computing the gradient at a
certain point is a simple linear-time operation).

The basic template algorithm, for unconstrained optimization, is given
in [\[alg:basic\]](#alg:basic){reference-type="ref"
reference="alg:basic"}, and a depiction of the iterates it produced in
figure [2.2](#fig:gradient_descent){reference-type="ref"
reference="fig:gradient_descent"}.

::: center
![Iterates of the GD algorithm ](images/gd.png){#fig:gradient_descent
width="2.3in"}
:::

::: algorithm
::: algorithmic
Input: time horizon $T$, initial point $x_0$, step sizes $\{\eta_t\}$
$\ensuremath{\mathbf x}_{t+1}  =   \ensuremath{\mathbf x}_t - \eta_t \nabla_t$
$\bar{\mathbf{x}}= \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}_t} \{ f(\ensuremath{\mathbf x}_t) \}$
:::
:::

For a convex function there always exists a choice of step sizes that
will cause GD to converge to the optimal solution. The rates of
convergence, however, differ greatly and depend on the smoothness and
strong convexity properties of the objective function. The following
table summarises the convergence rates of GD variants for convex
functions with different convexity parameters. The rates described omit
the (usually small) constants in the bounds---we focus on asymptotic
rates.

::: center
::: {#table:offline}
  ------------------ ---------------------- ---------------------- ------------------------ ---------------------------
                            general           $\alpha$-strongly         $\beta$-smooth             $\gamma$-well
                                                    convex                                          conditioned
   Gradient descent   $\frac{1}{\sqrt{T}}$   $\frac{1}{\alpha T}$     $\frac{\beta}{ T}$         $e^{- \gamma T }$
    Accelerated GD            ---                    ---            $\frac{{\beta}}{ T^2}$   $e^{- \sqrt{\gamma} \ T}$
  ------------------ ---------------------- ---------------------- ------------------------ ---------------------------

  : Rates of convergence of first order (gradient-based) methods as a
  function of the number of iterations and the smoothness and
  strong-convexity of the objective. Dependence on other parameters and
  constants, namely the Lipchitz constant, diameter of constraint set
  and initial distance to the objective is omitted. Acceleration for
  non-smooth functions is not possible in general.
:::
:::

[]{#table:offline label="table:offline"}

In this section we address only the first row of Table
[\[table:GD\]](#table:GD){reference-type="ref" reference="table:GD"}.
For accelerated methods and their analysis see references at the
bibliographic section.

### The Polyak stepsize

Luckily, there exists a simple choice of step sizes that yields the
optimal convergence rate, called the Polyak stepsize. It has a huge
advantage of not depending on the strong convexity and/or smoothness
parameters of the objective function.

However, it does depend on the distance in function value to optimality
and gradient norm. While the latter can be efficiently estimated, the
distance to optimality is not always available if
$f(\ensuremath{\mathbf x}^*)$ is not known ahead of time. This can be
remedied, as referred to in the bibliography.

We henceforth denote:

1.  Distance to optimality in value:
    $h_t = h(\ensuremath{\mathbf x}_t) = f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^*)$

2.  Euclidean distance to optimality:
    $d_t = \| \ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^*\|$

3.  Current gradient norm
    $\| \nabla_t\| = \|\nabla f(\ensuremath{\mathbf x}_t)\|$

With these notations we can describe the algorithm precisely in
Algorithm [\[alg:basicpolyak\]](#alg:basicpolyak){reference-type="ref"
reference="alg:basicpolyak"}:

::: algorithm
::: algorithmic
Input: time horizon $T$, $x_0$ Set
$\eta_t =  \frac{h_t}{\|\nabla_t\|^2}$
$\ensuremath{\mathbf x}_{t+1}  =   \ensuremath{\mathbf x}_t - \eta_t \nabla_t$
Return
$\bar{\mathbf{x}}= \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}_t} \{ f(\ensuremath{\mathbf x}_t) \}$
:::
:::

To prove precise convergence bounds, assume $\|\nabla_t\| \leq G$, and
define: $$\begin{aligned}
  B_T    &=&  \min\left\{
  \frac{G d_0}{\sqrt{ T}},
  \frac {2 \beta d_0^2}{ T },
  \frac{3 G^2}{  \alpha  T } ,
   \beta d_0^2\left(1-\frac{\gamma}{4}\right)^T
 \right\}
\end{aligned}$$

We can now state the main guarantee of GD with the Polyak stepsize:

::: {#thm:simple .theorem}
**Theorem 2.3**. *(GD with the Polyak Step Size) Algorithm
[\[alg:basicpolyak\]](#alg:basicpolyak){reference-type="ref"
reference="alg:basicpolyak"} guarantees the following after $T$ steps:
$$\begin{aligned}
f(\bar{\mathbf{x}})- f(\ensuremath{\mathbf x}^\star) \leq \min_{ 0 \leq t \leq  T} \{ h_t \} \leq B_T % \\
\end{aligned}$$*
:::

### Measuring distance to optimality

When analyzing convergence of gradient methods, it is useful to use
potential functions in lieu of function distance to optimality, such as
gradient norm and/or Euclidean distance. The following relationships
hold between these quantities.

::: {#lem:elementary_properties .lemma}
**Lemma 2.4**. *The following properties hold for
$\alpha$-strongly-convex functions and/or $\beta$-smooth functions over
Euclidean space ${\mathbb R}^d$.*

1.  *$\frac{\alpha}{2} d_t^2 \leq h_t$*

2.  *$h_t \leq \frac{\beta}{2} d_t^2$*

3.  *$\frac{1}{2 \beta} \|\nabla_t\|^2 \leq h_t$*

4.  *$h_t \leq \frac{1}{2 \alpha} \|\nabla_t\|^2$*
:::

::: proof
*Proof.*

1.  $h_t \geq \frac{\alpha}{2} d_t^2$:

    By strong convexity, we have $$\begin{aligned}
    h_t & =  f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^{\star}) \\
    & \geq  \nabla f(\ensuremath{\mathbf x}^{\star})^\top (\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^{\star}) + \frac{\alpha}{2} \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^{\star}\|^2  \\
     & = \frac{\alpha}{2} \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^{\star}\|^2
    \end{aligned}$$ where the last inequality follows since the gradient
    at the global optimum is zero.

2.  $h_t \leq \frac{\beta}{2} d_t^2$:

    By smoothness, $$\begin{aligned}
    h_t & =  f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^{\star}) \\
    & \leq  \nabla f(\ensuremath{\mathbf x}^{\star})^\top (\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^{\star}) + \frac{\beta}{2} \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^{\star}\|^2  \\
     & = \frac{\beta}{2} \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^{\star}\|^2
    \end{aligned}$$ where the last inequality follows since the gradient
    at the global optimum is zero.

3.  $h_t \geq \frac{1}{2\beta} \|\nabla_t\|^2$: Using smoothness, and
    let
    $\ensuremath{\mathbf x}_{t+1} = \ensuremath{\mathbf x}_t - \eta \nabla_t$
    for $\eta = \frac{1}{\beta}$, $$\begin{aligned}
    h_t =  & f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^{\star}) \\
    & \geq  f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}_{t+1})   \\
     & \geq   \nabla f(\ensuremath{\mathbf x}_t)^\top (\ensuremath{\mathbf x}_{t} - \ensuremath{\mathbf x}_{t+1}) - \frac{\beta}{2} \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}_{t+1} \|^2   \\
     & = \eta \|\nabla_t\|^2  - \frac{\beta}{2} \eta^2 \|\nabla_t\|^2 \\
     & = \frac{1}{2\beta} \|\nabla_t\|^2  .
    \end{aligned}$$

4.  $h_t \leq \frac{1}{2\alpha} \|\nabla_t\|^2$:

    We have for any pair
    $\ensuremath{\mathbf x},\ensuremath{\mathbf y}\in {\mathbb R}^d$:
    $$\begin{aligned}
    f(\ensuremath{\mathbf y})  & \ge  f(\ensuremath{\mathbf x}) +   \nabla f(\ensuremath{\mathbf x})^\top  (\ensuremath{\mathbf y}- \ensuremath{\mathbf x}) + \frac{\alpha}{2}  \|\ensuremath{\mathbf x}- \ensuremath{\mathbf y}\|^2  \\
    &\ge  \min_{\ensuremath{\mathbf z}\in {\mathbb R}^d } \left\{ f(\ensuremath{\mathbf x}) +   \nabla f(\ensuremath{\mathbf x})^\top  (\ensuremath{\mathbf z}- \ensuremath{\mathbf x}) + \frac{\alpha}{2}  \|\ensuremath{\mathbf x}- \ensuremath{\mathbf z}\|^2 \right\} \\
    & =  f(\ensuremath{\mathbf x}) - \frac{1}{2  \alpha} \| \nabla f(\ensuremath{\mathbf x})\|^2. \\
    & \text{ by taking $\ensuremath{\mathbf z}= \ensuremath{\mathbf x}- \frac{1}{ \alpha} \nabla f(\ensuremath{\mathbf x}) $ }
    \end{aligned}$$ In particular, taking
    $\ensuremath{\mathbf x}= \ensuremath{\mathbf x}_t \ , \ \ensuremath{\mathbf y}= \ensuremath{\mathbf x}^\star$,
    we get $$\label{eqn:gradlowerbound}
     h_t =  f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^\star)  \leq \frac{1}{2 \alpha} \|\nabla_t\|^2  .$$

 ◻
:::

### Analysis of the Polyak stepsize

We are now ready to prove
Theorem [2.3](#thm:simple){reference-type="ref" reference="thm:simple"},
which directly follows from the following lemma.

::: {#lemma:shalom2 .lemma}
**Lemma 2.5**. *Suppose that a sequence $\ensuremath{\mathbf x}_0,
\ldots \ensuremath{\mathbf x}_t$ satisfies: $$\label{eqn:shalom3}
d_{t+1}^2 \leq d_t^2 -   \frac{ h_t^2}{\|\nabla_t\|^2}$$ then for
$\bar{\mathbf{x}}$ as defined in the algorithm, we have:
$$f(\bar{\mathbf{x}}) - f(\ensuremath{\mathbf x}^\star)  \leq \frac{1}{T} \sum_t h_t \leq   B_{T}\, .$$*
:::

::: proof
*Proof.* The proof analyzes different cases:

1.  For convex functions with gradient bounded by $G$, $$\begin{aligned}
    d_{t+1}^2 -  d_t^2 & \leq - \frac{ h_t^2}{\|\nabla_t\|^2} \leq -
                         \frac{ h_t^2}{G^2}
    \end{aligned}$$ Summing up over $T$ iterations, and using
    Cauchy-Schwartz on the $T$-dimensional vectors of
    $\frac{1}{T} \mathbf{1}$ and $(h_1,...,h_T)$, we have
    $$\begin{aligned}
    \frac{1}{T} \sum_t h_t
    & \leq&  \frac{1}{\sqrt{T}} \sqrt{\sum_t h_t^2} \\
    & \leq& \frac{ G}{\sqrt{ T}} \sqrt{\sum_t (d_{t}^2 - d_{t+1}^2)} \leq
    \frac{ G d_0 }{\sqrt{ T}} \, .
    \end{aligned}$$

2.  For smooth functions whose gradient is bounded by $G$,
    Lemma [2.4](#lem:elementary_properties){reference-type="ref"
    reference="lem:elementary_properties"} implies:
    $$d_{t+1}^2 - d_t^2 \leq - \frac{ h_t^2}{\|\nabla_t\|^2} \leq -
    \frac{ h_t}{2 \beta} \, .$$ This implies
    $$\frac{1}{T} \sum_t h_t  \leq \frac{2 \beta d_0^2}{ T}\, .$$

3.  For strongly convex functions,
    Lemma [2.4](#lem:elementary_properties){reference-type="ref"
    reference="lem:elementary_properties"} implies: $$d_{t+1}^2 - d_t^2
      \leq -  \frac{h_t^2}{\|\nabla_t\|^2}
      \leq -  \frac{h_t^2}{G^2}
      \leq  -   \frac{\alpha^2 d_t^4 }{4 G^2} \, .$$ In other words,
    $d_{t+1}^2  \leq  d_t^2 ( 1-  \frac{\alpha^2 d_t^2}{4 G^2} ) \, .$
    Defining $a_t := \frac{\alpha^2 d_t^2}{4 G^2}$, we have:
    $$a_{t+1}  \leq  a_t (1-a_t) \, .$$ This implies that
    $a_t \leq \frac{1}{t+1}$, which can be seen by induction. The proof
    is completed as follows : $$\begin{aligned}
    \frac{1}{ T/2 } \sum_{t= T/2 }^T h_t^2 &
    \leq& \frac{2G^2}{  T  }\sum_{t= T/2 }^T ( d_t^2 -
                                       d_{t+1}^2)  \\
      &=&\frac{2 G^2}{  T } ( d _{ T/2 }^2 - d_T^2)  \\
      &=&\frac{8 G^4}{  \alpha^2   T} ( a
        _{ T/2 } - a_T)  \\
       & \leq &\frac{9 G^4}{ \alpha^2 T ^2}
      \, .
    \end{aligned}$$ Thus, there exists a $t$ for which
    $h_t^2 \leq \frac{ 9 G^4}{ \alpha^2  T^2}$. Taking the square root
    completes the claim.

4.  For both strongly convex and smooth functions:
    $$d_{t+1}^2 - d_t^2 \leq -  \frac{h_t^2}{\|\nabla_t\|^2} \leq
     - \frac{ h_t}{2 \beta} \leq
      -  \frac{\alpha}{4\beta} d_t^2$$ Thus,
    $$h_{T} \leq \beta d_{T}^2 \leq \beta d_0^2
        \left(1-\frac{\alpha}{4 \beta}\right)^T = \beta d_0^2
        \left(1-\frac{\gamma}{4}\right)^T \, .$$

This completes the proof of all cases. ◻
:::

## Constrained Gradient/Subgradient Descent

The vast majority of the problems considered in this text include
constraints. Consider the examples given in section
[1.2](#subsec:OCOexamples){reference-type="ref"
reference="subsec:OCOexamples"}: a path is a point in the flow polytope,
a portfolio is a point in the simplex and so on. In the language of
optimization, we require $\ensuremath{\mathbf x}$ not only to minimize a
certain objective function, but also to belong to a convex set
$\ensuremath{\mathcal K}$.

In this section we describe and analyze constrained gradient descent.
Algorithmically, the change from the previous section is small: after
updating the current point in the direction of the gradient, one may
need to project back to the decision set. However, the analysis is
somewhat more involved, and instructive for the later parts of this
text.

### Basic gradient descent---linear convergence

Algorithmic box [\[alg:BasicGD\]](#alg:BasicGD){reference-type="ref"
reference="alg:BasicGD"} describes a template for gradient descent over
a constrained set. It is a template since the sequence of step sizes
$\{\eta_t\}$ is left as an input parameter, and the several variants of
the algorithm differ on its choice.

::: algorithm
::: algorithmic
Input: $f$, $T$, initial point
$\ensuremath{\mathbf x}_1 \in \ensuremath{\mathcal K}$, sequence of step
sizes $\{\eta_t\}$ Let
$\ensuremath{\mathbf y}_{t+1} = \ensuremath{\mathbf x}_{t}-\eta_t {\nabla f}(\ensuremath{\mathbf x}_t) , \  \ensuremath{\mathbf x}_{t+1}= \mathop{\Pi}_{\ensuremath{\mathcal K}} \left( \ensuremath{\mathbf y}_{t+1}  \right)$
${\ensuremath{\mathbf x}}_{T+1}$
:::
:::

As opposed to the unconstrained setting, here we require a precise
setting of the learning rate to obtain the optimal convergence rate.

::: {#thm:basicGD .theorem}
**Theorem 2.6**. *For constrained minimization of
$\gamma$-well-conditioned functions and $\eta_t = \frac{1}{\beta}$,
Algorithm [\[alg:BasicGD\]](#alg:BasicGD){reference-type="ref"
reference="alg:BasicGD"} converges as
$$h_{t+1} \leq  h_1 \cdot e^{-\frac{\gamma t}{ 4}}$$*
:::

::: proof
*Proof.* By strong convexity we have for every
$\ensuremath{\mathbf x},\ensuremath{\mathbf x}_t \in \ensuremath{\mathcal K}$
(where we denote $\nabla_t = \nabla f(\ensuremath{\mathbf x}_t)$ as
before): $$\label{eqn:tempGD}
   \nabla_t^\top (\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_t) \leq f(\ensuremath{\mathbf x}) - f(\ensuremath{\mathbf x}_t) - \frac{\alpha}{2} \|\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_t\|^2.$$
Next, appealing to the algorithm's definition and the choice
$\eta_t = \frac{1}{\beta}$, we have $$\begin{aligned}
\ensuremath{\mathbf x}_{t+1}
& = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}}  \left\{  \nabla_t^\top ( \ensuremath{\mathbf x}-\ensuremath{\mathbf x}_t) + \frac{\beta}{2 } \|\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_t\|^2 \right\} \label{eqn:alg_defn_GD} .
\end{aligned}$$ To see this, notice that $$\begin{aligned}
 %\label{eqn:quad-solution}
&  \mathop{\Pi}_\ensuremath{\mathcal K}( \ensuremath{\mathbf x}_t - \eta_t \nabla_t ) \\
& = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \left\{ \|\ensuremath{\mathbf x}- (\ensuremath{\mathbf x}_t - \eta_t \nabla_t) \|^2 \right\} & \mbox{ definition of projection}  \\
& =  \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}}  \left\{  \nabla_t^\top ( \ensuremath{\mathbf x}-\ensuremath{\mathbf x}_t) + \frac{1}{2 \eta_t} \|\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_t\|^2 \right\} . & \mbox{ see exercise 6}
\end{aligned}$$ Thus, we have $$\begin{aligned}
h_{t+1} - h_{t} & = f(\ensuremath{\mathbf x}_{t+1}) - f(\ensuremath{\mathbf x}_t)  \\
& \leq \nabla_t^\top ( \ensuremath{\mathbf x}_{t+1} - \ensuremath{\mathbf x}_t) + \frac{\beta}{2} \|\ensuremath{\mathbf x}_{t+1} - \ensuremath{\mathbf x}_{t} \|^2 & \mbox{ smoothness}  \\
& \leq \min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}}  \left\{  \nabla_t^\top ( \ensuremath{\mathbf x}-\ensuremath{\mathbf x}_t) + \frac{\beta}{2} \|\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_t\|^2 \right\} & \mbox{ by \eqref{eqn:alg_defn_GD} } \\
& \leq  \min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}}  \left\{ f(\ensuremath{\mathbf x}) - f(\ensuremath{\mathbf x}_t)  + \frac{\beta - \alpha}{2} \|\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_t\|^2 \right\}. & \mbox{by } \eqref{eqn:tempGD} \\
\end{aligned}$$ The minimum can only grow if we take it over a subset of
$\ensuremath{\mathcal K}$. Thus we can restrict our attention to all
points that are convex combination of $\ensuremath{\mathbf x}_t$ and
$\ensuremath{\mathbf x}^\star$, which we denote by the interval
$[\ensuremath{\mathbf x}_t,\ensuremath{\mathbf x}^\star] = \{ (1 - \mu )\ensuremath{\mathbf x}_t + \mu \ensuremath{\mathbf x}^\star , \mu \in [0,1]\}$,
and write $$\begin{aligned}
 \label{eqn:shalom22}
h_{t+1} - h_{t}  & \leq  \min_{\ensuremath{\mathbf x}\in [\ensuremath{\mathbf x}_t,\ensuremath{\mathbf x}^\star] }  \left\{ f(\ensuremath{\mathbf x}) - f(\ensuremath{\mathbf x}_t)  + \frac{\beta - \alpha}{2} \|\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_t\|^2 \right\} \notag \\
& =   f( (1-\mu) \ensuremath{\mathbf x}_t + \mu \ensuremath{\mathbf x}^\star ) - f(\ensuremath{\mathbf x}_t)  + \frac{\beta - \alpha}{2}\mu^2 \|\ensuremath{\mathbf x}^\star  - \ensuremath{\mathbf x}_t\|^2 \notag \\ %& \mbox{ $ \x = (1-\mu) \x_t + \mu \x^\star $ } \\
& \le   (1-\mu) f( \ensuremath{\mathbf x}_t)  + \mu f(\ensuremath{\mathbf x}^\star ) - f(\ensuremath{\mathbf x}_t)  + \frac{\beta - \alpha}{2}\mu^2 \|\ensuremath{\mathbf x}^\star  - \ensuremath{\mathbf x}_t\|^2  & \mbox{convexity}   \notag \\
& =  -  \mu h_t + \frac{\beta - \alpha}{2} \mu^2 \|\ensuremath{\mathbf x}^\star  - \ensuremath{\mathbf x}_t\|^2 .
\end{aligned}$$ Where the equality is by writing
$\ensuremath{\mathbf x}$ as
$\ensuremath{\mathbf x}= (1-\mu) \ensuremath{\mathbf x}_t + \mu \ensuremath{\mathbf x}^\star$.
Using strong convexity, we have for any $\ensuremath{\mathbf x}_t$ and
the minimizer $\ensuremath{\mathbf x}^\star$: $$\begin{aligned}
h_t & = f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^\star ) \\
 &\ge    \nabla f(\ensuremath{\mathbf x}^\star)^\top  (\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^\star ) + \frac{\alpha}{2}  \|\ensuremath{\mathbf x}^\star - \ensuremath{\mathbf x}_t\|^2 & \text{ $\alpha$-strong convexity} \\
& \ge   \frac{\alpha}{2} \| \ensuremath{\mathbf x}^\star - \ensuremath{\mathbf x}_t \|^2. & \text{  optimality Thm \ref{thm:optim-conditions} }
\end{aligned}$$ Thus, plugging this into equation
[\[eqn:shalom22\]](#eqn:shalom22){reference-type="eqref"
reference="eqn:shalom22"}, we get $$\begin{aligned}
h_{t+1} - h_t & \leq ( - \mu  + \frac{\beta - \alpha}{\alpha} \mu^2 ) h_t \\
& \leq  - \frac{\alpha}{4 (\beta- \alpha)} h_t. & \mbox{ optimal choice of $\mu$}
\end{aligned}$$ Thus,
$$h_{t+1} \leq h_t (1 - \frac{\alpha}{4(\beta - \alpha)}) \leq h_t( 1 - \frac{\alpha}{4 \beta} ) \leq h_t e^{ -\gamma/4}.$$
This gives the theorem statement by induction. ◻
:::

## Reductions to Non-smooth and Non-strongly Convex Functions {#sec:gd-reductions}

The previous section dealt with $\gamma$-well-conditioned functions,
which may seem like a significant restriction over vanilla convexity.
Indeed, many interesting convex functions are not strongly convex nor
smooth, and as we have seen, the convergence rate of gradient descent
greatly differs for these functions. We have completed the picture for
unconstrained optimization, and in this section we complete it for a
bounded set.

The literature on first order methods is abundant with specialized
analyses that explore the convergence rate of gradient descent for more
general functions. In this manuscript we take a different approach:
instead of analyzing variants of GD from scratch, we use reductions to
derive near-optimal convergence rates for smooth functions that are not
strongly convex, or strongly convex functions that are not smooth, or
general convex functions without any further restrictions.

While attaining sub-optimal convergence bounds (by logarithmic factors),
the advantage of this approach is two-fold: first, the reduction method
is very simple to state and analyze, and its analysis is significantly
shorter than analyzing GD from scratch. Second, the reduction method is
generic, and thus extends to the analysis of accelerated gradient
descent (or any other first order method) along the same lines. We turn
to these reductions next.

### Reduction to smooth, non strongly convex functions

Our first reduction applies the GD algorithm to functions that are
$\beta$-smooth but not strongly convex.

The idea is to add a controlled amount of strong convexity to the
function $f$, and then apply the algorithm
[\[alg:BasicGD\]](#alg:BasicGD){reference-type="ref"
reference="alg:BasicGD"} to optimize the new function. The solution is
distorted by the added strong convexity, but a tradeoff guarantees a
meaningful convergence rate.

::: algorithm
::: algorithmic
Input: $f$, $T$, $\ensuremath{\mathbf x}_1 \in \ensuremath{\mathcal K}$,
parameter $\tilde{\alpha}$. Let
$g(\ensuremath{\mathbf x}) = f(\ensuremath{\mathbf x}) + \frac{\tilde{\alpha}}{2} \|\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_1 \|^2$
Apply Algorithm [\[alg:BasicGD\]](#alg:BasicGD){reference-type="ref"
reference="alg:BasicGD"} with parameters
$g,T, \{\eta_t = \frac{1}{\beta}\},\ensuremath{\mathbf x}_1$, return
$\ensuremath{\mathbf x}_T$.
:::
:::

::: {#thm:smoothGDreduction .lemma}
**Lemma 2.7**. *For $\beta$-smooth convex functions, Algorithm
[\[alg:non-strongly convex-GD\]](#alg:non-strongly convex-GD){reference-type="ref"
reference="alg:non-strongly convex-GD"} with parameter
$\tilde{\alpha} = \frac{ \beta \log t }{D^2 t}$ converges as
$$h_{t+1} = O \left(  \frac{\beta \log t } {t} \right)$$*
:::

::: proof
*Proof.* The function $g$ is $\tilde{\alpha}$-strongly convex and
$(\beta+ \tilde{\alpha})$-smooth (see exercises). Thus, it is
$\gamma = \frac{\tilde{\alpha}}{\tilde{\alpha} + \beta}$-well-conditioned.
Notice that $$\begin{aligned}
h_t & = f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^\star) \\
& = g(\ensuremath{\mathbf x}_t) - g(\ensuremath{\mathbf x}^\star) + \frac{\tilde{\alpha}}{2} (\|\ensuremath{\mathbf x}^\star - \ensuremath{\mathbf x}_1\|^2 - \|\ensuremath{\mathbf x}_t -\ensuremath{\mathbf x}_1\|^2) \\
& \le  h_t^g + \tilde{\alpha}D^2. & \mbox{ def of $D$, \S \ref{sec:optdefs}}
\end{aligned}$$ Here, we denote
$h^g_t = g(\ensuremath{\mathbf x}_t) - g(\ensuremath{\mathbf x}^\star)$.
Since $g(\ensuremath{\mathbf x})$ is
$\frac{\tilde{\alpha}}{\tilde{\alpha} + \beta}$-well-conditioned,
$$\begin{aligned}
 h_{t+1} & \le h_{t+1}^g  + \tilde{\alpha} D^2 \\
& \leq   h_1^g e^{-\frac{\tilde{\alpha} t}{  4( \tilde{\alpha}+\beta)}} + \tilde{\alpha} D^2    & \mbox{ Theorem \ref{thm:basicGD}} \\
& = O ( \frac{ \beta \log t}{ t} ),   & \mbox { choosing $\tilde{\alpha} = \frac{ \beta \log t }{D^2 t}$}
\end{aligned}$$ where we ignore constants and terms depending on $D$ and
$h_1^g$. ◻
:::

Stronger convergence rates of $O(\frac{\beta}{t})$ can be obtained by
analyzing GD from scratch, and these are known to be tight. Thus, our
reduction is suboptimal by a factor of $O(\log T)$, which we tolerate
for the reasons stated at the beginning of this section.

### Reduction to strongly convex, non-smooth functions

Our reduction from non-smooth functions to $\gamma$-well-conditioned
functions is similar in spirit to the one of the previous subsection.
However, whereas for strong convexity the obtained rates were off by a
factor of $\log T$, in this section we will also be off by factor of
$d$, the dimension of the decision variable $\ensuremath{\mathbf x}$, as
compared to the standard analyses in convex optimization. For tight
bounds, the reader is referred to the excellent reference books and
surveys listed in the bibliography section
[2.6](#sec:bib_of_optimization){reference-type="ref"
reference="sec:bib_of_optimization"}.

::: algorithm
::: algorithmic
Input: $f,\mathbf{x}_1,T,\delta$ Let
$\hat{f}_\delta (\ensuremath{\mathbf x}) =  \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim \mathbb{B}} \left[ f ( \ensuremath{\mathbf x}+ \delta \ensuremath{\mathbf v}) \right]$
Apply Algorithm [\[alg:BasicGD\]](#alg:BasicGD){reference-type="ref"
reference="alg:BasicGD"} on
$\hat{f}_\delta,\ensuremath{\mathbf x}_1,T,\{\eta_t = {\delta}\}$,
return $\ensuremath{\mathbf x}_T$
:::
:::

We apply the GD algorithm to a smoothed variant of the objective
function. In contrast to the previous reduction, smoothing cannot be
obtained by simple addition of a smooth (or any other) function.
Instead, we need a smoothing operation. The one we describe is
particularly simple and amounts to taking a local integral of the
function. More sophisticated, but less general, smoothing operators
exist that are based on the Moreau-Yoshida regularization, see
bibliographic section for more details.

Let $f$ be $G$-Lipschitz continuous and $\alpha$-strongly convex. Define
for any $\delta > 0$,
$$S_\delta[f] : {\mathbb R}^d \mapsto {\mathbb R}\  \ , \ \  S_\delta[f](\ensuremath{\mathbf x}) =  \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim  \mathbb{B}} \left[ f ( \ensuremath{\mathbf x}+ \delta \ensuremath{\mathbf v}) \right] ,$$
where
$\mathbb{B}= \{ \ensuremath{\mathbf x}\in \mathbb{R} ^d : \|\ensuremath{\mathbf x}\|  \leq 1 \}$
is the Euclidean ball and $\ensuremath{\mathbf v}\sim \mathbb{B}$
denotes a random variable drawn from the uniform distribution over
$\mathbb{B}$. When the function $f$ is clear from the context, we
henceforth use the simpler notation $\hat{f}_\delta = S_\delta[f]$.

We will prove that the function
${\ensuremath{\hat{f}}}_\delta = S_\delta[f]$ is a smooth approximation
to $f: {\mathbb R}^d \mapsto {\mathbb R}$, i.e., it is both smooth and
close in value to $f$, as given in the following lemma.

::: {#lem:SmoothingLemma .lemma}
**Lemma 2.8**. *$\hat{f}_\delta$ has the following properties:*

1.  *If $f$ is $\alpha$-strongly convex, then so is
    ${\ensuremath{\hat{f}}}_\delta$*

2.  *$\hat{f}_\delta$ is $\frac{d G}{\delta}$-smooth*

3.  *$|\hat{f} _\delta (\ensuremath{\mathbf x}) - f(\ensuremath{\mathbf x}) | \le \delta G$
    for all $\ensuremath{\mathbf x}\in \mathcal{K}$ .*
:::

Before proving this lemma, let us first complete the reduction. Using
Lemma [2.8](#lem:SmoothingLemma){reference-type="ref"
reference="lem:SmoothingLemma"} and the convergence for
$\gamma$-well-conditioned functions the following approximation bound is
obtained.

::: lemma
**Lemma 2.9**. *For $\delta = \frac{dG}{\alpha} \frac{\log{t}}{t}$
Algorithm [\[alg:reduction2\]](#alg:reduction2){reference-type="ref"
reference="alg:reduction2"} converges as
$$h_t=O\left( \frac{G^2 d  \log{t}}{\alpha t}\right).$$*
:::

Before proving this lemma, notice that the gradient descent method is
applied with gradients of the smoothed function
${\ensuremath{\hat{f}}}_\delta$ rather than gradients of the original
objective $f$. In this section we ignore the computational cost of
computing such gradients given only access to gradients of $f$, which
may be significant. Techniques for estimating these gradients are
further explored in chapter [6](#chap:bandits){reference-type="ref"
reference="chap:bandits"}.

::: proof
*Proof.* Note that by Lemma
[2.8](#lem:SmoothingLemma){reference-type="ref"
reference="lem:SmoothingLemma"} the function
${\ensuremath{\hat{f}}}_\delta$ is $\gamma$-well-conditioned for
$\gamma = \frac{\alpha \delta}{d G}.$

$$\begin{aligned}
 h_{t+1} & = f(\ensuremath{\mathbf x}_{t+1})-f(\ensuremath{\mathbf x}^\star) \\
&\le \hat{f}_\delta(\ensuremath{\mathbf x}_{t+1})-\hat{f}_\delta(\ensuremath{\mathbf x}^\star) + 2\delta G &\mbox{Lemma \ref{lem:SmoothingLemma}} \\
&\le h_1 e^{-\frac{\gamma t}{4}}+2\delta G & \mbox{Theorem \ref{thm:basicGD}}\\
&= h_1 e^{-\frac{\alpha t \delta}{4 dG}}+2\delta G& \mbox{$\gamma = \frac{\alpha \delta}{d G}$ by Lemma \ref{lem:SmoothingLemma}}\\
&= O \left( \frac{dG^2 \log t }{\alpha t} \right). &\mbox{$\delta = \frac{dG}{\alpha} \frac{\log{t}}{t}$}\\
\end{aligned}$$ ◻
:::

We proceed to prove that ${\ensuremath{\hat{f}}}_\delta$ is indeed a
good approximation to the original function.

::: proof
*Proof of Lemma [2.8](#lem:SmoothingLemma){reference-type="ref"
reference="lem:SmoothingLemma"}.* First, since $\hat{f}_\delta$ is an
average of $\alpha$-strongly convex functions, it is also
$\alpha$-strongly convex. In order to prove smoothness, we will use
Stokes' theorem from calculus: For all
$\ensuremath{\mathbf x}\in {\mathbb R}^d$ and for a vector random
variable $\ensuremath{\mathbf v}$ which is uniformly distributed over
the Euclidean sphere
$\ensuremath{\mathbb {S}}= \{ \ensuremath{\mathbf y}\in \mathbb{R} ^d : \|\ensuremath{\mathbf y}\|  = 1 \}$,
$$\label{lem:stokes_application}
 \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim \ensuremath{\mathbb {S}}} [ f(\ensuremath{\mathbf x}+ \delta \ensuremath{\mathbf v}) \ensuremath{\mathbf v}] = \frac{\delta}{d} \nabla\hat{f}_\delta (\ensuremath{\mathbf x}).$$

Recall that a function $f$ is $\beta$-smooth if and only if for all
$\ensuremath{\mathbf x},\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}$,
it holds that
$\| \nabla f(\ensuremath{\mathbf x}) -\nabla f(\ensuremath{\mathbf y}) \| \le \beta \|\ensuremath{\mathbf x}-\ensuremath{\mathbf y}\|$.
Now, $$\begin{aligned}
& \| \nabla \hat{f}_\delta (\ensuremath{\mathbf x}) -\nabla \hat{f}_\delta (\ensuremath{\mathbf y}) \|  =  \\
& = \frac{d}{\delta} \| \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim \ensuremath{\mathbb {S}}} \left[ f(\ensuremath{\mathbf x}+ \delta \ensuremath{\mathbf v}) \ensuremath{\mathbf v}\right] -\mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim  \ensuremath{\mathbb {S}}} \left[ f(\ensuremath{\mathbf y}+ \delta \ensuremath{\mathbf v}) \ensuremath{\mathbf v}\right]\| & \mbox{by \eqref{lem:stokes_application}}  \\
& = \frac{d}{\delta} \| \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim \ensuremath{\mathbb {S}}} \left[ f(\ensuremath{\mathbf x}+ \delta \ensuremath{\mathbf v}) \ensuremath{\mathbf v}- f(\ensuremath{\mathbf y}+ \delta \ensuremath{\mathbf v}) \ensuremath{\mathbf v}\right] \|  &\mbox{linearity of expectation} \\
& \le \frac{d}{\delta} \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim \ensuremath{\mathbb {S}}}   \| f(\ensuremath{\mathbf x}+ \delta \ensuremath{\mathbf v}) \ensuremath{\mathbf v}- f(\ensuremath{\mathbf y}+ \delta \ensuremath{\mathbf v}) \ensuremath{\mathbf v}\|    & \mbox{Jensen's inequality}\\
& \le  \frac{dG}{\delta} \| \ensuremath{\mathbf x}- \ensuremath{\mathbf y}\|   \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim \ensuremath{\mathbb {S}}} \left[ \|\ensuremath{\mathbf v}\| \right] & \mbox{Lipschitz continuity}\\
&= \frac{dG}{\delta} \| \ensuremath{\mathbf x}- \ensuremath{\mathbf y}\|.   & \mbox{$ \ensuremath{\mathbf v}\in \ensuremath{\mathbb {S}}$}
\end{aligned}$$ This proves the second property of Lemma
[2.8](#lem:SmoothingLemma){reference-type="ref"
reference="lem:SmoothingLemma"}. We proceed to show the third property,
namely that ${\ensuremath{\hat{f}}}_\delta$ is a good approximation to
$f$.

$$\begin{aligned}
& |\hat{f}_\delta (\ensuremath{\mathbf x})-f(\ensuremath{\mathbf x})|
 = \left|\mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim  \mathbb{B}} \left[  f(\ensuremath{\mathbf x}+ \delta \ensuremath{\mathbf v})\right] - f(\ensuremath{\mathbf x}) \right| &\mbox{definition of $\hat{f}_\delta$}\\
& \leq \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim \mathbb{B}} \left[  |f(\ensuremath{\mathbf x}+ \delta \ensuremath{\mathbf v}) - f(\ensuremath{\mathbf x}) |\right]  &\mbox{ Jensen's inequality} \\
& \le \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim \mathbb{B}}\left[ G\| \delta \ensuremath{\mathbf v}\| \right] & \mbox{$f$ is $G$-Lipschitz}\\
& \leq G \delta. & \mbox{ $\ensuremath{\mathbf v}\in \mathbb{B}$}

\end{aligned}$$ ◻
:::

We note that GD variants for $\alpha$-strongly convex functions, even
without the smoothing approach used in our reduction, are known to
converge quickly and without dependence on the dimension. We state the
known algorithm and result here without proof (see bibliography for
references).

::: {#thm:strongly convex-GD-bubeck .theorem}
**Theorem 2.10**. *Let $f$ be $\alpha$-strongly convex, and let
$\ensuremath{\mathbf x}_1,...,\ensuremath{\mathbf x}_t$ be the iterates
of Algorithm [\[alg:BasicGD\]](#alg:BasicGD){reference-type="ref"
reference="alg:BasicGD"} applied to $f$ with
$\eta_t = \frac{2}{\alpha (t+1)}$. Then
$$f\left( \frac{1}{t} \sum_{s=1}^t \frac{2 s }{t+1} \ensuremath{\mathbf x}_s \right) - f(\ensuremath{\mathbf x}^\star) \leq \frac{2 G^2}{\alpha (t+1)}  .$$*
:::

### Reduction to general convex functions

One can apply both reductions simultaneously to obtain a rate of
$\tilde{O}(\frac{d}{\sqrt{t}})$. While near-optimal in terms of the
number of iterations, the weakness of this bound lies in its dependence
on the dimension. In the next chapter we shall show a rate of
$O(\frac{1}{\sqrt{t}})$ as a direct consequence of a more general online
convex optimization algorithm.

## Example: Support Vector Machine Training {#sec:svmexample}

To illustrate the usefulness of the gradient descent method, let us
describe an optimization problem that has gained much attention in
machine learning and can be solved efficiently using the methods we have
just analyzed.

A very basic and successful learning paradigm is the linear
classification model. In this model, the learner is presented with
positive and negative examples of a concept. Each example, denoted by
$\mathbf{a}_i$, is represented in Euclidean space by a $d$ dimensional
feature vector. For example, a common representation for emails in the
spam-classification problem are binary vectors in Euclidean space, where
the dimension of the space is the number of words in the language. The
$i$'th email is a vector $\mathbf{a}_i$ whose entries are given as ones
for coordinates corresponding to words that appear in the email, and
zero otherwise. In addition, each example has a label
$b_i \in \{-1,+1\}$, corresponding to whether the email has been labeled
spam/not spam. The goal is to find a hyperplane separating the two
classes of vectors: those with positive labels and those with negative
labels. If such a hyperplane, which completely separates the training
set according to the labels, does not exist, then the goal is to find a
hyperplane that achieves a separation of the training set with the
smallest number of mistakes.

Mathematically speaking, given a set of $n$ examples to train on, we
seek $\ensuremath{\mathbf x}\in {\mathbb R}^d$ that minimizes the number
of incorrectly classified examples, i.e.
$$\label{eqn:linear-classification}
\min_{\ensuremath{\mathbf x}\in {\mathbb R}^d} \sum_{i \in [n]}  \delta( \mathop{\mbox{\rm sign}}(\ensuremath{\mathbf x}^\top \mathbf{a}_i ) \neq b_i)$$
where $\mathop{\mbox{\rm sign}}(x) \in \{-1,+1\}$ is the sign function,
and $\delta(z) \in \{0,1\}$ is the indicator function that takes the
value $1$ if the condition $z$ is satisfied and zero otherwise.

This optimization problem, which is at the heart of the linear
classification formulation, is NP-hard, and in fact NP-hard to even
approximate non-trivially . However, in the special case that a linear
classifier (a hyperplane $\ensuremath{\mathbf x}$) that classifies all
of the examples correctly exists, the problem is solvable in polynomial
time via linear programming.

Various relaxations have been proposed to solve the more general case,
when no perfect linear classifier exists. One of the most successful in
practice is the Support Vector Machine (SVM) formulation.

The soft margin SVM relaxation replaces the $0/1$ loss in
[\[eqn:linear-classification\]](#eqn:linear-classification){reference-type="eqref"
reference="eqn:linear-classification"} with a convex loss function,
called the hinge-loss, given by
$$\ell_{\mathbf{a},b}(\ensuremath{\mathbf x}) = \text{hinge}(b \cdot \ensuremath{\mathbf x}^\top \mathbf{a})  = \max\{0, 1 - b \cdot \ensuremath{\mathbf x}^\top \mathbf{a}\}.$$
In figure [2.3](#fig:hinge){reference-type="ref" reference="fig:hinge"}
we depict how the hinge loss is a convex relaxation for the non-convex
$0/1$ loss.

::: center
![The hinge loss function versus the 0/1 loss function
](images/hinge.pdf){#fig:hinge width="2.3in"}
:::

Further, the SVM formulation adds to the loss minimization objective a
term that regularizes the size of the elements in
$\ensuremath{\mathbf x}$. The reason and meaning of this additional term
shall be addressed in later sections. For now, let us consider the SVM
convex program: $$\label{eqn:soft-margin}
\min_{\ensuremath{\mathbf x}\in {\mathbb R}^d} \left \{ \lambda \frac{1}{n} \sum_{i \in [n]}  \ell_{\mathbf{a}_i,b_i}(\ensuremath{\mathbf x})  + \frac{1}{2} \| \ensuremath{\mathbf x}\|^2  \right \}$$

::: algorithm
::: algorithmic
Input: training set of $n$ examples $\{(\mathbf{a}_i,b_i) \}$, $T$,
learning rates $\{\eta_t\}$, initial $\ensuremath{\mathbf x}_1 = 0$. Let
${\nabla_t} = \lambda \frac{1}{n} \sum_{i=1}^n \nabla \ell_{\mathbf{a}_i,b_i} (\ensuremath{\mathbf x}_t)  + \ensuremath{\mathbf x}_t$
where $$\nabla \ell_{\mathbf{a}_i,b_i}(\ensuremath{\mathbf x}) = {
\left\{
\begin{array}{ll}
    {0}, & { b_i \ensuremath{\mathbf x}^\top \mathbf{a}_i > 1 } \\\\
    { - b_i \mathbf{a}_i}, & { \text{otherwise}}
\end{array}
\right. }$$
${\ensuremath{\mathbf x}}_{t+1}  = \ensuremath{\mathbf x}_{t}-\eta_t {\nabla_t}$
for $\eta_t = \frac{2}{t+1}$
$\bar{\ensuremath{\mathbf x}}_T =  \frac{1}{T} \sum_{t=1}^T \frac{2 t }{T+1} \ensuremath{\mathbf x}_t$
:::
:::

This is an unconstrained non-smooth and strongly convex program. It
follows from Theorems [2.3](#thm:simple){reference-type="ref"
reference="thm:simple"} and
[2.10](#thm:strongly convex-GD-bubeck){reference-type="ref"
reference="thm:strongly convex-GD-bubeck"} that
${O}(\frac{1}{\varepsilon})$ iterations suffice to attain an
$\varepsilon$-approximate solution. We spell out the details of applying
the subgradient descent algorithm to this formulation in Algorithm
[\[alg:BasicGDSVM\]](#alg:BasicGDSVM){reference-type="ref"
reference="alg:BasicGDSVM"}.

Notice that the learning rates are left unspecified, even though they
can be explicitly set as in Theorem
[2.10](#thm:strongly convex-GD-bubeck){reference-type="ref"
reference="thm:strongly convex-GD-bubeck"}, or using the Polyak rate.
The Polyak rate requires knowing the function value at optimality,
although this can be relaxed (see bibliography).

A caveat of using gradient descent for SVM is the requirement to compute
the full gradient, which may require a full pass over the data for each
iteration. We will see a significantly more efficient algorithm in the
next chapter!

## Bibliographic Remarks {#sec:bib_of_optimization}

The reader is referred to dedicated books on convex optimization for
much more in-depth treatment of the topics surveyed in this background
chapter. For background in convex analysis see the texts
[@borwein2006convex; @rockafellar1997convex]. The classic textbook of
@boyd.convex gives a broad introduction to convex optimization with
numerous applications, see also [@BoydNotes]. For detailed rigorous
convergence proofs and in depth analysis of first order methods, see
lecture notes by @NesterovBook and books by @NY83
[@Nemirovski04lectures], as well as more recent lecture notes and texts
[@bubeckOPT; @hazan2019lecture]. Theorem
[2.10](#thm:strongly convex-GD-bubeck){reference-type="ref"
reference="thm:strongly convex-GD-bubeck"} is taken from [@bubeckOPT]
Theorem 3.9.

The logarithmic overhead in the reductions of section
[2.4](#sec:gd-reductions){reference-type="ref"
reference="sec:gd-reductions"} can be removed with a more careful
reduction and analysis, for details see [@ZeyuanH06]. A more
sophisticated smoothing operator is the Moreau-Yoshida regularization:
it avoids the dimension factor loss. However, it is sometimes less
computationally efficient to work with [@parikh2014proximal].

The Polyak learning rate is detailed in [@polyak]. A recent exposition
allows obtaining the same optimal rate without knowledge of the optimal
function value [@hazan2019revisiting].

Using linear separators and halfspaces to learn and separate data was
considered in the very early days of AI
[@rosenblatt1958perceptron; @minsky69perceptrons]. Notable the
Perceptron algorithm was one of the first learning algorithms, and
closely related to gradient descent. Support vector machines were
introduced in [@CortesV95; @Boser92], see also the book of @ScSm02.

Learning halfspaces with the zero-one loss is computationally hard, and
hard to even approximate non-trivially [@daniely2016complexity]. Proving
that a problem is hard to approximate is at the forefront of
computational complexity, and based on novel characterizations of the
complexity class NP [@AroraBarakbook].

## Exercises

# First-Order Algorithms for Online Convex Optimization {#chap:first order}

In this chapter we describe and analyze the most simple and basic
algorithms for online convex optimization (recall the definition of the
model as introduced in chapter [1](#chap:intro){reference-type="ref"
reference="chap:intro"}), which are also surprisingly useful and
applicable in practice. We use the same notation introduced in
§[2.1](#sec:optdefs){reference-type="ref" reference="sec:optdefs"}.
However, in contrast to the previous chapter, the goal of the algorithms
introduced in this chapter is to minimize *regret*, rather than the
optimization error (which is ill-defined in an online setting).

Recall the definition of regret in an OCO setting, as given in equation
[\[eqn:regret-defn\]](#eqn:regret-defn){reference-type="eqref"
reference="eqn:regret-defn"}, with subscripts, superscripts and the
supremum over the function class omitted when they are clear from the
context:
$$\ensuremath{\mathrm{{Regret}}}_T = \sum_{t=1}^{T} f_t(\mathbf{x}_t) -\min_{\mathbf{x}\in \ensuremath{\mathcal K}}\sum_{t=1}^{T} f_t(\mathbf{x}) .$$

Table [\[table:regret-rates\]](#table:regret-rates){reference-type="ref"
reference="table:regret-rates"} details known upper and lower bounds on
the regret for different types of convex functions as it depends on the
number of prediction iterations.

::: center
::: {#default}
                     $\alpha$-strongly convex       $\beta$-smooth          $\delta$-exp-concave
  ---------------- ---------------------------- ---------------------- ------------------------------
    Upper bound     $\frac{1}{ \alpha} \log T$        $\sqrt{T}$         $\frac{n}{ \delta} \log T$
    Lower bound     $\frac{1}{ \alpha} \log T$        $\sqrt{T}$        $\frac{n}{ \delta } \log T$
   Average regret   $\frac{\log T}{ \alpha T}$   $\frac{1}{\sqrt{T}}$   $\frac{n \log T}{ \delta T}$

  : Attainable asymptotic regret bounds for loss function classes.
:::
:::

[]{#default label="default"}

In order to compare regret to optimization error it is useful to
consider the average regret, or ${\ensuremath{\mathrm{{Regret}}}}/{T}$.
Let
$\bar{\ensuremath{\mathbf x}}_T = \frac{1}{T} \sum_{t=1}^T \ensuremath{\mathbf x}_t$
be the average decision. If the functions $f_t$ are all equal to a
single function $f : \ensuremath{\mathcal K}\mapsto {\mathbb R}$, then
Jensen's inequality implies that $f( \bar{\ensuremath{\mathbf x}}_T)$
converges to $f(\ensuremath{\mathbf x}^\star)$ at a rate at most the
average regret, since
$$f(\bar{\ensuremath{\mathbf x}}_T) - f(\ensuremath{\mathbf x}^\star ) \leq \frac{1}{T} \sum_{t=1} ^T [f(\ensuremath{\mathbf x}_t)  - f(\ensuremath{\mathbf x}^\star) ] = \frac{\ensuremath{\mathrm{{Regret}}}_T}{T} .$$

The reader may recall Table
[\[table:GD\]](#table:GD){reference-type="ref" reference="table:GD"}
describing offline convergence of first order methods: as opposed to
offline optimization, smoothness does not improve asymptotic regret
rates. However, exp-concavity, a weaker property than strong convexity,
comes into play and gives improved regret rates.

This chapter will present algorithms and lower bounds that realize the
above known results for OCO. The property of exp-concavity and its
applications, as well as logarithmic regret algorithms for exp-concave
functions are deferred to the next chapter.

## Online Gradient Descent {#section:ogd}

Perhaps the simplest algorithm that applies to the most general setting
of online convex optimization is online gradient descent. This
algorithm, which is based on standard gradient descent from offline
optimization, was introduced in its online form by Zinkevich (see
bibliography at the end of this section).

::: algorithm
::: algorithmic
Input: convex set $\ensuremath{\mathcal K}$, $T$,
$\ensuremath{\mathbf x}_1 \in \mathcal{K}$, step sizes $\{ \eta_t \}$
Play $\ensuremath{\mathbf x}_t$ and observe cost
$f_t(\ensuremath{\mathbf x}_t)$. Update and project: $$\begin{aligned}
			& \ensuremath{\mathbf y}_{t+1} = \mathbf{x}_{t}-\eta_{t} \nabla f_{t}(\mathbf{x}_{t}) \\
			& \mathbf{x}_{t+1} = \mathop{\Pi}_\ensuremath{\mathcal K}(\ensuremath{\mathbf y}_{t+1})

\end{aligned}$$
:::
:::

Pseudo-code for the algorithm is given in Algorithm
[\[alg:ogd\]](#alg:ogd){reference-type="ref" reference="alg:ogd"}, and a
conceptual illustration is given in figure
[3.1](#fig:ogd){reference-type="ref" reference="fig:ogd"}.

::: center
![OGD: the iterate $\ensuremath{\mathbf x}_{t+1}$ is derived by
advancing $\ensuremath{\mathbf x}_t$ in the direction of the current
gradient $\nabla_t$, and projecting back into $\ensuremath{\mathcal K}$
](images/fig_gd_poly3.jpg){#fig:ogd width="4in"}
:::

In each iteration, the algorithm takes a step from the previous point in
the direction of the gradient of the previous cost. This step may result
in a point outside of the underlying convex set. In such cases, the
algorithm projects the point back to the convex set, i.e. finds its
closest point in the convex set. Despite the fact that the next cost
function may be completely different than the costs observed thus far,
the regret attained by the algorithm is sublinear. This is formalized in
the following theorem (recall the definition of $G$ and $D$ from the
previous chapter).

::: {#thm:gradient .theorem}
**Theorem 3.1**. *Online gradient descent with step sizes
$\{\eta_t = \frac{D}{G \sqrt{t}} , \ t \in [T] \}$ guarantees the
following for all $T \geq 1$:
$$\ensuremath{\mathrm{{Regret}}}_T = \sum_{t=1}^{T} f_t(\mathbf{x}_t) -\min_{\mathbf{x}^\star \in \ensuremath{\mathcal K}}\sum_{t=1}^{T}
f_t(\mathbf{x}^\star)\ \leq  \frac{3}{2} {G D}\sqrt{T} .$$*
:::

::: proof
*Proof.* Let
$\mathbf{x}^\star \in \mathop{\mathrm{\arg\min}}_{\mathbf{x}\in \ensuremath{\mathcal K}} \sum_{t=1}^T f_t(\mathbf{x})$.
Define
$\nabla_t \stackrel{\text{\tiny def}}{=}\nabla f_{t}(\mathbf{x}_{t})$.
By convexity $$\begin{aligned}
  \label{eqn:gradient_inequality}
f_t(\mathbf{x}_t) - f_t(\mathbf{x}^\star) \leq   \nabla_t^\top (\mathbf{x}_t - \mathbf{x}^\star)
\end{aligned}$$ We first upper-bound
$\nabla_t^\top (\mathbf{x}_t-\mathbf{x}^\star)$ using the update rule
for $\mathbf{x}_{t+1}$ and Theorem
[2.1](#thm:pythagoras){reference-type="ref" reference="thm:pythagoras"}
(the Pythagorean theorem): $$\label{eqn:ogdtriangle}
\| \mathbf{x}_{t+1}-\mathbf{x}^\star \|^2\ =\  \left\|\mathop{\Pi}_\ensuremath{\mathcal K}(\mathbf{x}_t - \eta_t
\nabla_{t}) -\mathbf{x}^\star\right\|^2 \leq  \left\|\mathbf{x}_t - \eta_t \nabla_t-\mathbf{x}^\star\right\|^2 .$$

Hence, $$\begin{aligned}
 \label{eqn:ogd_eq2}
\|\mathbf{x}_{t+1}-\mathbf{x}^\star\|^2\ &\leq&\ \|\mathbf{x}_t- \mathbf{x}^\star\|^2 + \eta_t^2
\|\nabla_t\|^2 -2 \eta_t \nabla_t^\top (\mathbf{x}_t -\mathbf{x}^\star)\nonumber\\
2 \nabla_t^\top (\mathbf{x}_t-\mathbf{x}^\star)\ &\leq&\ \frac{ \|\mathbf{x}_t-
\mathbf{x}^\star\|^2-\|\mathbf{x}_{t+1}-\mathbf{x}^\star\|^2}{\eta_t} + \eta_t G^2 .
\end{aligned}$$ Summing
[\[eqn:gradient_inequality\]](#eqn:gradient_inequality){reference-type="eqref"
reference="eqn:gradient_inequality"} and
[\[eqn:ogd_eq2\]](#eqn:ogd_eq2){reference-type="eqref"
reference="eqn:ogd_eq2"} from $t= 1$ to $T$, and setting $\eta_t =
\frac{D}{G \sqrt{t}}$ (with
$\frac{1}{\eta_0} \stackrel{\text{\tiny def}}{=}0$): $$\begin{aligned}
& 2 \left( \sum_{t=1}^T f_t(\mathbf{x}_t)-f_t(\mathbf{x}^\star) \right ) \leq 2\sum_{t=1}^T \nabla_t^\top (\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x}^\star) \\
&\leq  \sum_{t=1}^T \frac{ \|\mathbf{x}_t-
	\mathbf{x}^\star\|^2-\|\mathbf{x}_{t+1}-\mathbf{x}^\star\|^2}{\eta_t} + G^2 \sum_{t=1}^T \eta_t    \\
&\leq  \sum_{t=1}^T \|\mathbf{x}_t - \mathbf{x}^\star\|^2 \left(
\frac{1}{\eta_{t}} -
\frac{1}{\eta_{t-1}} \right) + G^2 \sum_{t=1}^T \eta_t & \frac{1}{\eta_0} \stackrel{\text{\tiny def}}{=}0, \\
& &  \|\ensuremath{\mathbf x_{T+1}} - \ensuremath{\mathbf x_{}}^* \|^2 \geq 0 \\
&\leq D^2 \sum_{t=1}^T \left(
\frac{1}{\eta_{t}} -
\frac{1}{\eta_{t-1}} \right) + G^2 \sum_{t=1}^T \eta_t \\
& \leq  D^2  \frac{1}{\eta_{T}}  + G^2 \sum_{t=1}^T \eta_t  & \mbox{ telescoping series } \\
& \leq 3 DG \sqrt{T}.
\end{aligned}$$ The last inequality follows since
$\eta_t = \frac{D}{G \sqrt{t}}$ and
$\sum_{t=1}^T \frac{1}{\sqrt{t}} \leq 2 \sqrt{T}$. ◻
:::

The online gradient descent algorithm is straightforward to implement,
and updates take linear time given the gradient. However, there is a
projection step which may take significantly longer, as discussed in
§[2.1.1](#sec:projections){reference-type="ref"
reference="sec:projections"} and chapter
[7](#chap:FW){reference-type="ref" reference="chap:FW"}.

## Lower Bounds {#section:lowerbound}

The previous section introduces and analyzes a very simple and natural
approach to online convex optimization. Before continuing our venture,
it is worthwhile to consider whether the previous bound can be improved?
We measure performance of OCO algorithms both by regret and by
computational efficiency. Therefore, we ask ourselves whether even
simpler algorithms that attain tighter regret bounds exist.

The computational efficiency of online gradient descent seemingly leaves
little room for improvement, putting aside the projection step it runs
in linear time per iteration. What about obtaining better regret?

Perhaps surprisingly, the answer is negative: online gradient descent
attains, in the worst case, tight regret bounds up to small constant
factors! This is formally given in the following theorem.

::: {#thm:lowerbound .theorem}
**Theorem 3.2**. *Any algorithm for online convex optimization incurs
$\Omega(DG
\sqrt{T})$ regret in the worst case. This is true even if the cost
functions are generated from a fixed stationary distribution.*
:::

We give a sketch of the proof; filling in all details is left as an
exercise at the end of this chapter.

Consider an instance of OCO where the convex set
$\ensuremath{\mathcal K}$ is the $n$-dimensional hypercube, i.e.
$$\ensuremath{\mathcal K}= \{ \mathbf{x}\in {\mathbb R}^n \ , \  \|\mathbf{x}\|_\infty \leq 1 \}.$$
There are $2^n$ linear cost functions, one for each vertex
$\mathbf{v}\in \{
\pm 1\}^n$, defined as
$$\forall \mathbf{v}\in \{ \pm 1 \}^n \ , \ f_\mathbf{v}(\mathbf{x})  = \mathbf{v}^\top \mathbf{x}.$$
Notice that both the diameter of $\ensuremath{\mathcal K}$ and the bound
on the norm of the cost function gradients, denoted G, are bounded by
$$D \leq \sqrt{ \sum_{i=1}^n 2^2 } = 2 \sqrt{n} , \ G \leq \sqrt{ \sum_{i=1}^n (\pm1)^2 } = \sqrt{n}$$

The cost functions in each iteration are chosen at random, with uniform
probability, from the set
$\{ f_\mathbf{v}, \mathbf{v}\in \{\pm 1\}^n \}$. Denote by
$\mathbf{v}_t \in \{\pm 1\}^n$ the vertex chosen in iteration $t$, and
denote $f_t = f_{\mathbf{v}_t}$. By uniformity and independence, for any
$t$ and $\mathbf{x}_t$ chosen online,
$\mathop{\mbox{\bf E}}_{\mathbf{v}_t}[f_{t}(\mathbf{x}_t)]= \mathop{\mbox{\bf E}}_{\mathbf{v}_t}[ \mathbf{v}_t^\top
\mathbf{x}_t] = 0$. However, $$\begin{aligned}
\mathop{\mbox{\bf E}}_{\mathbf{v}_1,\ldots,\mathbf{v}_T}\left[\min_{\mathbf{x}\in \ensuremath{\mathcal K}} \sum_{t=1}^T f_t(\mathbf{x})\right] & =
\mathop{\mbox{\bf E}}\left[\min_{\mathbf{x}\in \ensuremath{\mathcal K}} \sum_{i \in [n]} \sum_{t=1}^T \mathbf{v}_t(i) \cdot \mathbf{x}_i \right] \\
& = n \mathop{\mbox{\bf E}}\left[-\left|\sum_{t=1}^T \mathbf{v}_t(1) \right|\right] & \mbox{i.i.d. coordinates}\\
& = -\Omega(n \sqrt{T}).
\end{aligned}$$ The last equality is left as an exercise.

The facts above nearly complete the proof of Theorem
[3.2](#thm:lowerbound){reference-type="ref" reference="thm:lowerbound"};
see the exercises at the end of this chapter.

## Logarithmic Regret

At this point, the reader may wonder: we have introduced a seemingly
sophisticated and obviously general framework for learning and
prediction, as well as a linear-time algorithm for the most general
case, complete with tight regret bounds, and done so with elementary
proofs! Is this all OCO has to offer?

The answer to this question is two-fold:

1.  Simple is good: the philosophy behind OCO treats simplicity as a
    merit. The main reason OCO has taken the stage in online learning in
    recent years is the simplicity of its algorithms and their analysis,
    which allow for numerous variations and tweaks in their host
    applications.

2.  A very wide class of settings, which will be the subject of the next
    sections, admit more efficient algorithms, in terms of both regret
    and computational complexity.

In §[2](#chap:opt){reference-type="ref" reference="chap:opt"} we
surveyed optimization algorithms with convergence rates that vary
greatly according to the convexity properties of the function to be
optimized. Do the regret bounds in online convex optimization vary as
much as the convergence bounds in offline convex optimization over
different classes of convex cost functions?

Indeed, next we show that for important classes of loss functions
significantly better regret bounds are possible.

### Online gradient descent for strongly convex functions {#section:ogdnew}

The first algorithm that achieves regret logarithmic in the number of
iterations is a twist on the online gradient descent algorithm, changing
only the step size. The following theorem establishes logarithmic bounds
on the regret if the cost functions are strongly convex.

::: {#thm:gradient2 .theorem}
**Theorem 3.3**. *For $\alpha$-strongly convex loss functions, online
gradient descent with step sizes $\eta_t = \frac{1}{\alpha {t}}$
achieves the following guarantee for all $T \geq 1$
$$\ensuremath{\mathrm{{Regret}}}_T\ \leq\ \frac{G^2}{2 \alpha}(1 + \log T).$$*
:::

::: proof
*Proof.* Let
$\mathbf{x}^\star \in \mathop{\mathrm{\arg\min}}_{\mathbf{x}\in \ensuremath{\mathcal K}} \sum_{t=1}^T f_t(\mathbf{x})$.
Recall the definition of regret
$$\ensuremath{\mathrm{{Regret}}}_T\ = \sum_{t=1}^{T} f_t(\mathbf{x}_t) - \sum_{t=1}^{T} f_t(\mathbf{x}^\star).$$

Define
$\nabla_t \stackrel{\text{\tiny def}}{=}\nabla f_t(\mathbf{x}_t)$.
Applying the definition of $\alpha$-strong convexity to the pair of
points $\{\ensuremath{\mathbf x}_t$,$\ensuremath{\mathbf x}^*\}$, we
have $$\begin{aligned}
2(f_t(\mathbf{x}_t)-f_t(\mathbf{x}^\star)) &\leq& 2\nabla_t^\top (\mathbf{x}_t-\mathbf{x}^\star)-\alpha
\|\mathbf{x}^\star-\mathbf{x}_t\|^2.\label{eqsz}
\end{aligned}$$ We proceed to upper-bound $\nabla_t^\top
(\mathbf{x}_t-\mathbf{x}^\star)$. Using the update rule for
$\mathbf{x}_{t+1}$ and the Pythagorean theorem
[2.1](#thm:pythagoras){reference-type="ref" reference="thm:pythagoras"},
we get
$$\| \mathbf{x}_{t+1}-\mathbf{x}^\star \|^2\ =\ \|\mathop{\Pi}_\ensuremath{\mathcal K}(\mathbf{x}_t - \eta_{t} \nabla_t)-\mathbf{x}^\star\|^2 \leq  \|\mathbf{x}_t - \eta_{t} \nabla_t-\mathbf{x}^\star\|^2.$$
Hence, $$\begin{aligned}
\|\mathbf{x}_{t+1}-\mathbf{x}^\star\|^2\ &\leq&\ \|\mathbf{x}_t- \mathbf{x}^\star\|^2 + \eta_{t}^2
\|\nabla_t\|^2 -2
\eta_{t} \nabla_t^\top (\mathbf{x}_t - \mathbf{x}^\star)\nonumber\\
\end{aligned}$$ and $$\begin{aligned}
2 \nabla_t^\top (\mathbf{x}_t-\mathbf{x}^\star)\ &\leq&\ \frac{ \|\mathbf{x}_t-
\mathbf{x}^\star\|^2-\|\mathbf{x}_{t+1}-\mathbf{x}^\star\|^2}{\eta_{t}} + \eta_{t} G^2.
\label{eqer}
\end{aligned}$$ Summing [\[eqer\]](#eqer){reference-type="eqref"
reference="eqer"} from $t= 1$ to $T$, setting
$\eta_{t} = \frac{1}{\alpha t}$ (define
$\frac{1}{\eta_0} \stackrel{\text{\tiny def}}{=}0$), and combining with
[\[eqsz\]](#eqsz){reference-type="eqref" reference="eqsz"}, we have:
$$\begin{aligned}
&  & 2 \sum_{t=1}^T (f_t(\mathbf{x}_t)-f_t(\mathbf{x}^\star) ) \\
&\leq &\
 \sum_{t=1}^T \|\mathbf{x}_t-\mathbf{x}^\star\|^2
\left(\frac{1}{\eta_{t}}-\frac{1}{\eta_{t-1}}-\alpha\right) +G^2
\sum_{t=1}^{T} \eta_{t}   \\
& & \mbox{ since }  \frac{1}{\eta_0} \stackrel{\text{\tiny def}}{=}0, \|\ensuremath{\mathbf x_{T+1}} - \ensuremath{\mathbf x_{}}^* \|^2 \geq 0 \\ \\
&=&\ 0 + G^2 \sum_{t=1}^{T} \frac{1}{\alpha t} \\
& \leq &  \frac{G^2}{\alpha}(1 + \log T )
\end{aligned}$$ ◻
:::

## Application: Stochastic Gradient Descent {#sec:sgd}

A special case of Online Convex Optimization is the well-studied setting
of stochastic optimization. In stochastic optimization, the optimizer
attempts to minimize a convex function over a convex domain as given by
the mathematical program: $$\begin{aligned}
\min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} f(\ensuremath{\mathbf x}).
\end{aligned}$$ However, unlike standard offline optimization, the
optimizer is given access to a noisy gradient oracle, defined by
$$\mathcal{O}(\ensuremath{\mathbf x}) \stackrel{\text{\tiny def}}{=}\tilde{\nabla }_\ensuremath{\mathbf x}\ \mbox{ s.t. } \  \mathop{\mbox{\bf E}}[\tilde{\nabla }_\ensuremath{\mathbf x}] = \nabla f(\ensuremath{\mathbf x}) \ , \   \mathop{\mbox{\bf E}}[ \|\tilde{\nabla }_\ensuremath{\mathbf x}\|^2 ] \leq G^2$$
That is, given a point in the decision set, a noisy gradient oracle
returns a random vector whose expectation is the gradient at the point
and whose variance is bounded by $G^2$.

We will show that regret bounds for OCO translate to convergence rates
for stochastic optimization. As a special case, consider the online
gradient descent algorithm whose regret is bounded by
$$\ensuremath{\mathrm{{Regret}}}_{T} = O(DG\sqrt{T})$$ Applying the OGD
algorithm over a sequence of linear functions that are defined by the
noisy gradient oracle at consecutive points, and finally returning the
average of all points along the way, we obtain the stochastic gradient
descent algorithm, presented in Algorithm
[\[alg:sgd\]](#alg:sgd){reference-type="ref" reference="alg:sgd"}.

::: algorithm
::: algorithmic
Input: ${\mathcal O}$,$\ensuremath{\mathcal K}$, $T$,
$\ensuremath{\mathbf x}_1 \in \mathcal{K}$, step sizes $\{ \eta_t \}$
[]{#alg:sgd-defnft label="alg:sgd-defnft"} Let
$\tilde{\nabla}_t = \mathcal{O}(\ensuremath{\mathbf x}_t)$ Update and
project:
$$\ensuremath{\mathbf y}_{t+1} = \mathbf{x}_{t}-\eta_t \tilde{\nabla}_t$$
$$\mathbf{x}_{t+1} = \mathop{\Pi}_\ensuremath{\mathcal K}(\ensuremath{\mathbf y}_{t+1})$$
$\bar{\ensuremath{\mathbf x}}_T \stackrel{\text{\tiny def}}{=}\frac{1}{T} \sum_{t=1}^T \ensuremath{\mathbf x}_t$
:::
:::

::: {#thm:sgd .theorem}
**Theorem 3.4**. *Algorithm [\[alg:sgd\]](#alg:sgd){reference-type="ref"
reference="alg:sgd"} with step sizes $\eta_t = \frac{D}{G \sqrt{t}}$
guarantees
$$\mathop{\mbox{\bf E}}[ f(\bar{\ensuremath{\mathbf x}}_T) ]  \leq \min_{\ensuremath{\mathbf x}^\star \in \ensuremath{\mathcal K}} f(\ensuremath{\mathbf x}^\star) + \frac{3 GD }{2\sqrt{T}} .$$*
:::

::: proof
*Proof.* For the analysis, we define the linear functions
$f_t(\ensuremath{\mathbf x}) \stackrel{\text{\tiny def}}{=}\tilde{\nabla}_t^\top  \ensuremath{\mathbf x}$.
Using the regret guarantee of OGD, we have $$\begin{aligned}
&  \mathop{\mbox{\bf E}}[ f(\bar{\ensuremath{\mathbf x}}_T) ] - f(\ensuremath{\mathbf x}^\star) \\
 & \leq \mathop{\mbox{\bf E}}[ \frac{1}{T} \sum_t  f(\ensuremath{\mathbf x}_t) ]  - f(\ensuremath{\mathbf x}^\star)  & \mbox{ convexity of $f$ (Jensen) }\\
&\leq \frac{1}{T}  \mathop{\mbox{\bf E}}[ \sum_t   \nabla f(\ensuremath{\mathbf x}_t)^\top(  \ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^\star)  ] & \mbox{ convexity again }\\
& = \frac{1}{T} \mathop{\mbox{\bf E}}[ \sum_t  \tilde{\nabla}_t^\top (  \ensuremath{\mathbf x}_t -\ensuremath{\mathbf x}^\star )  ] & \mbox{ noisy gradient estimator }\\
& = \frac{1}{T} \mathop{\mbox{\bf E}}[ \sum_t  f_t( \ensuremath{\mathbf x}_t)  -f_t(\ensuremath{\mathbf x}^\star) ] & \mbox{ Algorithm \ref{alg:sgd}, line \eqref{alg:sgd-defnft} }\\
& \leq  \frac{\ensuremath{\mathrm{{Regret}}}_T }{T} & \mbox{ definition }\\
& \leq  \frac{3GD }{2\sqrt{T}} & \mbox{ theorem \ref{thm:gradient}}
\end{aligned}$$ ◻
:::

It is important to note that in the proof above, we have used the fact
that the regret bounds of online gradient descent hold against an
adaptive adversary. This need arises since the cost functions $f_t$
defined in Algorithm [\[alg:sgd\]](#alg:sgd){reference-type="ref"
reference="alg:sgd"} depend on the choice of decision
$\ensuremath{\mathbf x}_t \in \ensuremath{\mathcal K}$.

In addition, the careful reader may notice that by plugging in different
step sizes (also called learning rates) and applying SGD to strongly
convex functions, one can attain $\tilde{O}({1}/{T})$ convergence rates,
where the $\tilde{O}$ notation hides logarithmic factors in $T$. Details
of this derivation are left as an exercise.

### Example: stochastic gradient descent for SVM training

Recall our example of Support Vector Machine training from
§[2.5](#sec:svmexample){reference-type="ref"
reference="sec:svmexample"}. The task of training an SVM over a given
data set amounts to solving the following convex program (equation
[\[eqn:soft-margin\]](#eqn:soft-margin){reference-type="eqref"
reference="eqn:soft-margin"}): $$\begin{aligned}
& f(\ensuremath{\mathbf x})  = \min_{\ensuremath{\mathbf x}\in {\mathbb R}^d} \left \{ \lambda \frac{1}{n} \sum_{i \in [n]}  \ell_{\mathbf{a}_i,b_i}(\ensuremath{\mathbf x})  + \frac{1}{2} \| \ensuremath{\mathbf x}\|^2  \right \} \\
& \ell_{\mathbf{a},b}(\ensuremath{\mathbf x}) = \max\{0, 1 - b \cdot \ensuremath{\mathbf x}^\top \mathbf{a}\} .
\end{aligned}$$

::: algorithm
::: algorithmic
Input: training set of $n$ examples $\{(\mathbf{a}_i,b_i) \}$, $T$. Set
$\ensuremath{\mathbf x}_1 = 0$ Pick an example uniformly at random
$t \in [n]$. Let
$\tilde{\nabla}_t = \lambda  \nabla \ell_{\mathbf{a}_t,b_t} (\ensuremath{\mathbf x}_t)  + \ensuremath{\mathbf x}_t$
where $$\nabla \ell_{{\mathbf{a}_t},b_t}(\ensuremath{\mathbf x}_t) = {
\left\{
\begin{array}{ll}
    {0}, & { b_t \ensuremath{\mathbf x}_t^\top \mathbf{a}_t > 1 } \\\\
    { - b_t \mathbf{a}_t}, & { \mbox{otherwise}}
\end{array}
\right. }$$
${\ensuremath{\mathbf x}}_{t+1}  = \ensuremath{\mathbf x}_{t}-\eta_t \tilde{\nabla}_t$
$\bar{\ensuremath{\mathbf x}}_T \stackrel{\text{\tiny def}}{=}\frac{1}{T} \sum_{t=1}^T \ensuremath{\mathbf x}_t$
:::
:::

Using the technique described in this chapter, namely the OGD and SGD
algorithms, we can devise a much faster algorithm than the one presented
in the previous chapter. The idea is to generate an unbiased estimator
for the gradient of the objective using a single example in the dataset,
and use it in lieu of the entire gradient. This is given formally in the
SGD algorithm for SVM training presented in Algorithm
[\[alg:sgd4svm\]](#alg:sgd4svm){reference-type="ref"
reference="alg:sgd4svm"}.

It follows from Theorem [3.4](#thm:sgd){reference-type="ref"
reference="thm:sgd"} that this algorithm, with appropriate parameters
$\eta_t$, returns an $\varepsilon$-approximate solution after
$T = O(\frac{1}{\varepsilon^2})$ iterations. Furthermore, with a little
more care and using Theorem [3.3](#thm:gradient2){reference-type="ref"
reference="thm:gradient2"}, a rate of $\tilde{O}(\frac{1}{\varepsilon})$
is obtained with parameters $\eta_t = O( \frac{1}{t})$.

This matches the convergence rate of standard offline gradient descent.
However, observe that each iteration is significantly cheaper---only one
example in the data set need be considered! That is the magic of SGD; we
have matched the nearly optimal convergence rate of first order methods
using extremely cheap iterations. This makes it the method of choice in
numerous applications.

## Bibliographic Remarks {#bibliographic-remarks}

The OCO framework was introduced by @Zinkevich03, where the OGD
algorithm was introduced and analyzed. Precursors to this algorithm,
albeit for less general settings, were introduced and analyzed in
[@KivWar97]. Logarithmic regret algorithms for Online Convex
Optimization were introduced and analyzed in [@HAK07].

The stochastic gradient descent (SGD) algorithm dates back to
@robbins1951, where it was called "stochastic approximation\". The
importance of SGD for machine learning was advocated for in
[@bottou1998online; @bottou2008tradeoffs]. The literature on SGD is vast
and the reader is referred to the text of @bubeckOPT and paper by
@lan2012optimal.

Application of SGD to soft-margin SVM training was explored in
[@Shalev-ShwartzSSC11]. Tight convergence rates of SGD for strongly
convex and non-smooth functions were only recently obtained in
[@hazan:beyond; @RSS; @SZ].

## Exercises

# Second-Order Methods {#chap:second order-methods}

The motivation for this chapter is the application of online portfolio
selection, considered in the first chapter of this book. We begin with a
detailed description of this application. We proceed to describe a new
class of convex functions that model this problem. This new class of
functions is more general than the class of strongly convex functions
discussed in the previous chapter. It allows for logarithmic regret
algorithms, which are based on second order methods from convex
optimization. In contrast to first order methods, which have been our
focus thus far and relied on (sub)gradients, second order methods
exploit information about the second derivative of the objective
function.

## Motivation: Universal Portfolio Selection

In this subsection we give the formal definition of the universal
portfolio selection problem that was informally described in
§[1.2](#subsec:OCOexamples){reference-type="ref"
reference="subsec:OCOexamples"}.

### Mainstream portfolio theory

Mainstream financial theory models stock prices as a stochastic process
known as Geometric Brownian Motion (GBM). This model assumes that the
fluctuations in the prices of the stocks behave essentially as a random
walk. It is perhaps easier to think about a price of an asset (stock) on
time segments, obtained from a discretization of time into equal
segments. Thus, the logarithm of the price at segment $t+1$, denoted
$l_{t+1}$, is given by the sum of the logarithm of the price at segment
$t$ and a Gaussian random variable with a particular mean and variance,
$$l_{t+1} \sim l_t + \mathcal{N}(\mu,\sigma).$$

This is only an informal way of thinking about GBM. The formal model is
a continuous time process, similar to the discrete time stochastic
process we have just described, obtained as the time intervals, means,
and variances approach zero.

The GBM model gives rise to particular algorithms for portfolio
selection (as well as more sophisticated applications such as options
pricing). Given the means and variances of the stock prices over time of
a set of assets, as well as their cross-correlations, a portfolio with
maximal expected gain (mean) for a specific risk (variance) threshold
can be formulated.

The fundamental question is, of course, how does one obtain the mean and
variance parameters, not to mention the cross-correlations, of a given
set of stocks? One accepted solution is to estimate these from
historical data, e.g., by taking the recent history of stock prices.

### Universal portfolio theory

The theory of universal portfolio selection is very different from the
GBM model. The main difference being the lack of statistical assumptions
about the stock market. The idea is to model investing as a repeated
decision making scenario, which fits nicely into our OCO framework, and
to measure regret as a performance metric.

Consider the following scenario: at each iteration $t \in [T]$, the
decision maker chooses $\ensuremath{\mathbf x}_t$, a distribution of her
wealth over $n$ assets, such that
$\ensuremath{\mathbf x_{t}}\in \Delta_n$. Here
$\Delta_n = \{ \ensuremath{\mathbf x}\in {\mathbb R}^n_+ , \sum_i \ensuremath{\mathbf x}_i = 1 \}$
is the $n$-dimensional simplex, i.e., the set of all distributions over
$n$ elements. An adversary independently chooses market returns for the
assets, i.e., a vector $\ensuremath{\mathbf r_{t}}\in {\mathbb R}_+^n$
such that each coordinate $\ensuremath{\mathbf r_{t}}(i)$ is the price
ratio for the $i$'th asset between the iterations $t$ and $t+1$. For
example, if the $i$'th coordinate is the Google ticker symbol GOOG
traded on the NASDAQ, then
$$\ensuremath{\mathbf r_{t}}(i) = \frac{\mbox{price of GOOG at time $t+1$}}{\mbox{price of GOOG at time $t$}}$$
How does the decision maker's wealth change? Let $W_t$ be her total
wealth at iteration $t$. Then, ignoring transaction costs, we have
$$W_{t+1} = W_t \cdot \ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x_{t}}$$
Over $T$ iterations, the total wealth of the investor is given by
$$W_{T} = W_1 \cdot \prod_{t=1}^{T-1} \ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x_{t}}$$
The goal of the decision maker, to maximize the overall wealth gain
${W_T}/{W_0}$, can be attained by maximizing the following more
convenient logarithm of this quantity, given by
$$\log \frac{W_T}{W_1} =  \sum_{t=1}^{T-1} \log \ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x_{t}}$$

The above formulation is already very similar to our OCO setting, albeit
phrased as a gain maximization rather than a loss minimization setting.
Let
$$f_t(\ensuremath{\mathbf x}) = \log (\ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x})$$
The convex set is the $n$-dimensional simplex
$\ensuremath{\mathcal K}= \Delta_n$, and define the regret to be
$$\ensuremath{\mathrm{{Regret}}}_T = \max_{\ensuremath{\mathbf x}^\star \in \ensuremath{\mathcal K}} \sum_{t=1}^T f_t(\ensuremath{\mathbf x}^\star) - \sum_{t=1}^T f_t(\ensuremath{\mathbf x}_t)$$
The functions $f_t$ are concave rather than convex, which is perfectly
fine as we are framing the problem as a maximization rather than a
minimization. Note also that the regret is the negation of the usual
regret notion
[\[eqn:regret-defn\]](#eqn:regret-defn){reference-type="eqref"
reference="eqn:regret-defn"} we have considered for minimization
problems.

Since this is an online convex optimization instance, we can use the
online gradient descent algorithm from the previous chapter to invest,
which ensures $O(\sqrt{T})$ regret (see exercises). What guarantee do we
attain in terms of investing? To answer this, in the next section we
reason about what $\ensuremath{\mathbf x}^\star$ in the above expression
may be.

### Constant rebalancing portfolios

As $\ensuremath{\mathbf x}^\star \in \ensuremath{\mathcal K}= \Delta_n$
is a point in the $n$-dimensional simplex, consider the special case of
$\ensuremath{\mathbf x}^\star = \ensuremath{\mathbf e_{1}}$, i.e., the
first standard basis vector (the vector that has zero in all coordinates
except the first, which is set to one). The term
$\sum_{t=1}^T f_t(\ensuremath{\mathbf e_{1}})$ becomes
$\sum_{t=1}^T \log \ensuremath{\mathbf r_{t}}(1)$, or
$$\log \prod_{t=1}^T  \ensuremath{\mathbf r_{t}}(1) = \log \left(  \frac{\mbox{price of  stock at time $T+1$}} {\mbox{initial price of  stock}} \right)$$
As $T$ becomes large, any sublinear regret guarantee (e.g., the
$O(\sqrt{T})$ regret guarantee achieved using online gradient descent)
achieves an average regret that approaches zero. In this context, this
implies that the log-wealth gain achieved (in average over $T$ rounds)
is as good as that of the first stock. Since
$\ensuremath{\mathbf x}^\star$ can be taken to be any vector, sublinear
regret guarantees average log-wealth growth as good as any stock!

However, $\ensuremath{\mathbf x}^\star$ can be significantly better, as
shown in the following example. Consider a market of two stocks that
fluctuate wildly. The first stock increases by $100\%$ every even day
and returns to its original price the following (odd) day. The second
stock does exactly the opposite: decreases by $50\%$ on even days and
rises back on odd days. Formally, we have
$$\ensuremath{\mathbf r_{t}}(1)  = (2 \ ,  \ \frac{1}{2} \  , \   2 \ , \ \frac{1}{2} , ... )$$
$$\ensuremath{\mathbf r_{t}}(2)  = (\frac{1}{2} \ , \  2 \ ,  \ \frac{1}{2} \  , \   2 \  , ... )$$
Clearly, any investment in either of the stocks will not gain in the
long run. However, the portfolio
$\ensuremath{\mathbf x}^\star = (0.5,0.5)$ increases wealth by a factor
of
$\ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x}^\star = (\frac{1}{2})^2 + 1 = 1.25$
daily! Such a mixed distribution is called a fixed rebalanced portfolio,
as it needs to rebalance the proportion of total capital invested in
each stock at each iteration to maintain this fixed distribution
strategy.

Thus, vanishing average regret guarantees long-run growth as the best
constant rebalanced portfolio in hindsight. Such a portfolio strategy is
called *universal*. We have seen that the online gradient descent
algorithm gives essentially a universal algorithm with regret
$O(\sqrt{T})$. Can we get better regret guarantees?

## Exp-Concave Functions

For convenience, we return to considering losses of convex functions,
rather than gains of concave functions as in the application for
portfolio selection. The two problems are equivalent: we simply replace
the maximization of the concave
$f(\ensuremath{\mathbf x})= \log(\ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x})$
with the minimization of the convex
$f(\ensuremath{\mathbf x})=-\log(\ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x})$.

In the previous chapter we have seen that the OGD algorithm with
carefully chosen step sizes can deliver logarithmic regret for strongly
convex functions. However, the loss function for the OCO setting of
portfolio selection,
$f_t(\ensuremath{\mathbf x}) = -\log (\ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x}),$
is not strongly convex. Instead, the Hessian of this function is given
by
$$\nabla^2 f_t(\ensuremath{\mathbf x}) =  \frac{\ensuremath{\mathbf r_{t}}\ensuremath{\mathbf r_{t}}^\top }{(\ensuremath{\mathbf r_{t}}^\top \ensuremath{\mathbf x})^2}$$
which is a rank one matrix. Recall that the Hessian of a
twice-differentiable strongly convex function is larger than a multiple
of identity matrix and is positive definite and in particular has full
rank. Thus, the loss function above is quite far from being strongly
convex.

However, an important observation is that this Hessian is large in the
direction of the gradient. This property is called exp-concavity. We
proceed to define this property rigorously and show that it suffices to
attain logarithmic regret.

::: definition
**Definition 4.1**. *A convex function
$f : {\mathbb R}^n \mapsto {\mathbb R}$ is defined to be
$\alpha$-exp-concave over
$\ensuremath{\mathcal K}\subseteq {\mathbb R}^n$ if the function $g$ is
concave, where $g: \ensuremath{\mathcal K}\mapsto {\mathbb R}$ is
defined as
$$g(\ensuremath{\mathbf x}) = e^{-\alpha f(\ensuremath{\mathbf x}) }$$*
:::

For the following discussion, recall the notation of
§[2.1](#sec:optdefs){reference-type="ref" reference="sec:optdefs"}, and
in particular our convention over matrices that $A \succcurlyeq B$ if
and only if $A - B$ is positive semidefinite. Exp-concavity implies
strong-convexity in the direction of the gradient. This reduces to the
following property:

::: {#lem:quadratic_approximation1 .lemma}
**Lemma 4.2**. *A twice-differentiable function
$f : {\mathbb R}^n \mapsto {\mathbb R}$ is $\alpha$-exp-concave at
$\ensuremath{\mathbf x}$ if and only if
$$\nabla^2 f(\ensuremath{\mathbf x})  \succcurlyeq {\alpha} \nabla f(\ensuremath{\mathbf x}) \nabla f(\ensuremath{\mathbf x})^\top.$$*
:::

The proof of this lemma is given as a guided exercise at the end of this
chapter. We prove a slightly stronger lemma below.

::: {#lem:quadratic_approximation2 .lemma}
**Lemma 4.3**. *Let $f :\ensuremath{\mathcal K}\rightarrow {\mathbb R}$
be an $\alpha$-exp-concave function, and $D,G$ denote the diameter of
$\ensuremath{\mathcal K}$ and a bound on the (sub)gradients of $f$
respectively. The following holds for all
$\gamma \leq \frac{1}{2}\min\{\frac{1}{GD},\alpha\}$ and all
$\ensuremath{\mathbf x},\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}$:
$$f(\mathbf{x}) \geq f(\mathbf{y}) + \nabla f(\mathbf{y})^\top (\mathbf{x}-\mathbf{y}) +
\frac{\gamma}{2} (\mathbf{x}- \mathbf{y})^\top \nabla f(\mathbf{y}) \nabla
f(\mathbf{y})^\top(\mathbf{x}-\mathbf{y}).$$*
:::

::: proof
*Proof.* The composition of a concave and non-decreasing function with
another concave function is concave . Therefore, since
$2\gamma \leq \alpha$, the composition of $g(x)= x^{2 \gamma/\alpha}$
with $f(\ensuremath{\mathbf x}) = \exp(-\alpha f(\mathbf{x}))$ is
concave. It follows that the function
$h(\mathbf{x}) \stackrel{\text{\tiny def}}{=}\exp(-2\gamma f(\mathbf{x}))$
is concave. Then by the concavity of $h(\mathbf{x})$,
$$h(\mathbf{x}) \leq h(\mathbf{y}) + \nabla h(\mathbf{y})^\top(\mathbf{x}- \mathbf{y})$$
Plugging in
$\nabla h(\mathbf{y}) = -2\gamma \exp(-2\gamma f(\mathbf{y})) \nabla
f(\mathbf{y})$ gives
$$\exp(-2\gamma f(\mathbf{x}))  \leq \exp(-2\gamma f(\mathbf{y})) [1 - 2\gamma \nabla f(\mathbf{y})^\top (\mathbf{x}- \mathbf{y})].$$
Simplifying gives
$$f(\mathbf{x}) \geq f(\mathbf{y}) - \frac{1}{2\gamma} \log \left( 1-2\gamma \nabla f(\mathbf{y})^\top
(\mathbf{x}-\mathbf{y}) \right).$$ Next, note that
$|2\gamma \nabla f(\mathbf{y})^\top
(\mathbf{x}-\mathbf{y})|\ \leq\ 2\gamma GD \leq 1$ and that using the
Taylor approximation, for $z \geq -1$, it holds that
$-\log(1-z) \geq z+\frac{1}{4}{z^2}$. Applying the inequality for
$z = 2\gamma \nabla f(\mathbf{y})^\top
(\mathbf{x}-\mathbf{y})$ implies the lemma. ◻
:::

## Exponentially Weighted Online Convex Optimization

Before diving into efficient second order methods, we first describe a
simple algorithm based on the multiplicative updates method which gives
logarithmic regret for exp-concave losses. Algorithm
[\[alg:ewoo\]](#alg:ewoo){reference-type="eqref" reference="alg:ewoo"}
below, called EWOO, is a close relative to the Hedge Algorithm
[\[alg:Hedge\]](#alg:Hedge){reference-type="eqref"
reference="alg:Hedge"}. Its regret guarantee is robust: it does not
include a Lipschitz constant or a diameter bound. In addition, it is
particularly simple to describe and analyze.

The downside of EWOO is its running time. A naive implementation would
run in exponential time of the dimension. It is possible to given a
randomized polynomial time implementation based on random sampling
techniques, where the polynomial depends both on the dimension as well
as the number of iterations, see bibliographic section for more details.

::: algorithm
::: algorithmic
Input: convex set $\ensuremath{\mathcal K}$, $T$, parameter
$\alpha > 0$. Let
$w_t(\mathbf{x}) = e^{-\alpha{\textstyle \sum}_{\tau=1}^{t-1} f_\tau(\mathbf{x})}$.
Play $\mathbf{x}_t$ given by
$$\ensuremath{\mathbf x}_t = \frac{\int_\ensuremath{\mathcal K}\mathbf{x}\ w_t(\mathbf{x}) d \mathbf{x}}{\int_\ensuremath{\mathcal K}w_t(\mathbf{x})d \mathbf{x}} .$$
:::
:::

In the analysis below, it can be observed that choosing $\mathbf{x}_t$
at random with density proportional to $w_t(\mathbf{x})$, instead of
computing the entire integral, also guarantees our regret bounds on the
expectation. This is the basis for the polynomial time implementation.
We proceed to give the logarithmic regret bounds.

::: {#thm:exp .theorem}
**Theorem 4.4**.
*$$\ensuremath{\mathrm{{Regret}}}_T(EWOO) \ \leq \ \frac{d}{\alpha} \log T + \frac{2}{\alpha} .$$*
:::

::: proof
*Proof.* Let $h_t(\mathbf{x}) = e^{- \alpha f_t(\mathbf{x})}$. Since
$f_t$ is $\alpha$-exp-concave, we have that $h_t$ is concave and thus
$$h_t(\mathbf{x}_t) \geq \frac{\int_\ensuremath{\mathcal K}h_t(\mathbf{x}) \prod_{\tau=1}^{t-1} h_\tau(\mathbf{x}) ~d \mathbf{x}}{\int_\ensuremath{\mathcal K}
 \prod_{\tau=1}^{t-1} h_\tau(\mathbf{x})  ~ d \mathbf{x}}.$$ Hence, we
have by telescoping product, $$\label{eqn:telescope}
\prod_{\tau=1}^t h_\tau(\mathbf{x}_\tau) \geq \frac{\int_\ensuremath{\mathcal K}
\prod_{\tau=1}^t h_\tau(\mathbf{x}) ~ d \mathbf{x}}{\int_\ensuremath{\mathcal K}1 ~ d \mathbf{x}} =
\frac{\int_\ensuremath{\mathcal K}\prod_{\tau=1}^t h_\tau(\mathbf{x}) ~ d \mathbf{x}}{\mbox{vol}(\ensuremath{\mathcal K})}$$

By definition of $\mathbf{x}^\star$ we have
$\mathbf{x}^\star \in \arg\max_{\mathbf{x}\in \ensuremath{\mathcal K}}
\prod_{t=1}^T h_t(\mathbf{x})$. Denote by
$S_\delta \subset \ensuremath{\mathcal K}$ the translated Minkowski set
given by
$$S_\delta  = (1-\delta) \ensuremath{\mathbf x}^\star +  \ensuremath{\mathcal K}_{1-\delta} =  \left\{  \mathbf{x}= (1-\delta)  \mathbf{x}^\star + \delta \mathbf{y}\ , \ \mathbf{y}\in
\ensuremath{\mathcal K}\right\}.$$ By concavity of $h_t$ and the fact
that $h_t$ is non-negative, we have that,
$$\forall \mathbf{x}\in S_\delta \ . \  \quad h_t(\mathbf{x}) \geq (1-\delta) h_t(\mathbf{x}^\star).$$
Hence,
$$\forall \mathbf{x}\in S_\delta \quad \prod_{\tau=1}^T h_\tau(\mathbf{x})
\geq \left( 1  - \delta\right)^T \prod_{\tau=1}^T h_\tau(\mathbf{x}^\star)$$
Finally, since
$S_\delta = (1-\delta)  \mathbf{x}^\star + \delta  \ensuremath{\mathcal K}$
is simply a rescaling of $\ensuremath{\mathcal K}$ followed by a
translation, and we are in $d$ dimensions,
$\mbox{vol}(S_\delta) = \mbox{vol}(\ensuremath{\mathcal K}) \times \delta^d$.
Putting this together with equation
[\[eqn:telescope\]](#eqn:telescope){reference-type="eqref"
reference="eqn:telescope"}, we have
$$\prod_{\tau=1}^T h_\tau(\mathbf{x}_\tau) \geq \frac{\mbox{vol}{(S_\delta)}}{\mbox{vol}(\ensuremath{\mathcal K})} (1-\delta)^T
\prod_{\tau=1}^T h_\tau(\mathbf{x}^\star) \geq
{\delta^d}(1-\delta)^T\prod_{\tau=1}^T h_\tau(\mathbf{x}^\star).$$ We
can now simplify by taking logarithms and changing sides,
$$\begin{aligned}
\ensuremath{\mathrm{{Regret}}}_T(EWOO) & = \sum_t f_t(\ensuremath{\mathbf x}_t) - f_t(\ensuremath{\mathbf x}^\star) \\
& = \frac{1}{\alpha} \log \frac { \prod_{\tau=1}^T h_\tau(\mathbf{x}^\star)} {\prod_{\tau=1}^T h_\tau(\mathbf{x}_\tau) } \\
& \leq \frac{1}{\alpha} \left( d \log \frac{1}{\delta} + T \log \frac{1}{1-\delta} \right) \leq \frac{d}{\alpha} \log T + \frac{2}{\alpha} ,
\end{aligned}$$ where the last step is by choosing
$\delta = \frac{1}{T}$. ◻
:::

## The Online Newton Step Algorithm {#section:ons}

Thus far we have only considered first order methods for regret
minimization. In this section we consider a quasi-Newton approach, i.e.,
an online convex optimization algorithm that approximates the second
derivative, or Hessian in more than one dimension. However, strictly
speaking, the algorithm we analyze is also first order, in the sense
that it only uses gradient information.

The algorithm we introduce and analyze, called online Newton step, is
detailed in Algorithm [\[alg:ons\]](#alg:ons){reference-type="ref"
reference="alg:ons"}. At each iteration, this algorithm chooses a vector
that is the projection of the sum of the vector chosen at the previous
iteration and an additional vector. Whereas for the online gradient
descent algorithm this added vector was the gradient of the previous
cost function, for online Newton step this vector is different: it is
reminiscent to the direction in which the Newton-Raphson method would
proceed if it were an offline optimization problem for the previous cost
function. The Newton-Raphson algorithm would move in the direction of
the vector which is the inverse Hessian times the gradient. In online
Newton step, this direction is $A_t^{-1} \nabla_t$, where the matrix
$A_t$ is related to the Hessian as will be shown in the analysis.

Since adding a multiple of the Newton vector $A_t^{-1} \nabla_t$ to the
current vector may result in a point outside the convex set, an
additional projection step is required to obtain
$\ensuremath{\mathbf x}_t$, the decision at time $t$. This projection is
different than the standard Euclidean projection used by online gradient
descent in Section [3.1](#section:ogd){reference-type="ref"
reference="section:ogd"}. It is the projection according to the norm
defined by the matrix $A_t$, rather than the Euclidean norm.

::: algorithm
::: algorithmic
Input: convex set $\ensuremath{\mathcal K}$, $T$,
$\ensuremath{\mathbf x}_1 \in \mathcal{K} \subseteq {\mathbb R}^n$,
parameters $\gamma,\varepsilon > 0$, $A_ 0 = \varepsilon \mathbf{I}_n$
Play $\ensuremath{\mathbf x}_t$ and observe cost
$f_t(\ensuremath{\mathbf x}_t)$. Rank-1 update:
${A}_t = {A}_{t-1} + \nabla_t \nabla_t^\top$ Newton step and generalized
projection:
$$\ensuremath{\mathbf y}_{t+1} =  \mathbf{x}_{t} - \frac{1}{\gamma}  {A}_{t}^{-1} \nabla_{t}$$
$$\mathbf{x}_{t+1} = \mathop{\Pi}_\ensuremath{\mathcal K}^{{A}_t} (\ensuremath{\mathbf y}_{t+1}) = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \left\{ \|\ensuremath{\mathbf y}_{t+1} - \ensuremath{\mathbf x}\|^2_{{A}_t} \right\}$$
:::
:::

The advantage of the online Newton step algorithm is its logarithmic
regret guarantee for exp-concave functions, as defined in the previous
section. The following theorem bounds the regret of online Newton step.

::: {#thm:onsregret .theorem}
**Theorem 4.5**. *Algorithm [\[alg:ons\]](#alg:ons){reference-type="ref"
reference="alg:ons"} with parameters
$\gamma = \frac{1}{2}\min\{\frac{1}{GD},\alpha\}$,
$\varepsilon = \frac{1}{\gamma^2 D^2}$ and $T \geq 4$ guarantees
$$\ensuremath{\mathrm{{Regret}}}_T \ \leq\  2 \left(\frac{1}{\alpha} + GD\right) n
\log T.$$*
:::

As a first step, we prove the following lemma.

::: {#lemma:onsbound .lemma}
**Lemma 4.6**. *The regret of online Newton step is bounded by
$$\ensuremath{\mathrm{{Regret}}}_T(\text{ONS})\ \leq\ \left(\frac{1}{\alpha}
+ GD\right) \left(\sum_{t=1}^T \nabla_t^\top {A}_t^{-1} \nabla_t + 1\right)$$*
:::

::: proof
*Proof.* Let
$\mathbf{x}^\star \in \arg\min_{\mathbf{x}\in \ensuremath{\mathcal K}} \sum_{t=1}^T f_t(\mathbf{x})$
be the best decision in hindsight. By Lemma
[4.3](#lem:quadratic_approximation2){reference-type="ref"
reference="lem:quadratic_approximation2"}, we have for
$\gamma = \frac{1}{2}\min\{\frac{1}{GD},\alpha\}$,
$$f_t(\mathbf{x}_t ) - f_t(\mathbf{x}^\star)\ \leq\ R_t  ,$$ where we
define $$R_t  \stackrel{\text{\tiny def}}{=}\ \nabla_t^\top
(\mathbf{x}_t - \mathbf{x}^\star) - \frac{\gamma}{2} (\mathbf{x}^\star - \mathbf{x}_t)^\top \nabla_t
\nabla_t^\top (\mathbf{x}^\star - \mathbf{x}_t) .$$ According to the
update rule of the algorithm
$\mathbf{x}_{t+1} = \mathop{\Pi}_{\ensuremath{\mathcal K}}^{{A}_t}(\mathbf{y}_{t+1})$.
Now, by the definition of $\mathbf{y}_{t+1}$: $$\label{eq:update-rule}
\mathbf{y}_{t+1} - \mathbf{x}^\star = \mathbf{x}_{t} - \mathbf{x}^\star - \frac{1}{\gamma} {A}_t^{-1}
\nabla_t, \text{ and}$$ $$\label{eq:A_t-multiply}
{A}_t (\mathbf{y}_{t+1} - \mathbf{x}^\star) = {A}_t(\mathbf{x}_t - \mathbf{x}^\star) - \frac{1}{\gamma}
\nabla_t.$$ Multiplying the transpose of
[\[eq:update-rule\]](#eq:update-rule){reference-type="eqref"
reference="eq:update-rule"} by
[\[eq:A_t-multiply\]](#eq:A_t-multiply){reference-type="eqref"
reference="eq:A_t-multiply"} we get $$\begin{gathered}
(\mathbf{y}_{t+1} - \mathbf{x}^\star)^\top {A}_t(\mathbf{y}_{t+1} - \mathbf{x}^\star) = \notag \\
(\mathbf{x}_t\! -\! \mathbf{x}^\star)^\top {A}_t(\mathbf{x}_t\! -\! \mathbf{x}^\star) -
\frac{2}{\gamma} \nabla_t^\top (\mathbf{x}_t\! -\! \mathbf{x}^\star) +
\frac{1}{\gamma^2} \nabla_t^\top {A}_t^{-1} \nabla_t.
\label{eq:multiplied}
\end{gathered}$$ Since $\mathbf{x}_{t+1}$ is the projection of
$\mathbf{y}_{t+1}$ in the norm induced by ${A}_t$, we have by the
Pythagorean theorem (see §[2.1.1](#sec:projections){reference-type="ref"
reference="sec:projections"}) $$\begin{aligned}
 (\mathbf{y}_{t+1} - \mathbf{x}^\star)^\top {A}_t(\mathbf{y}_{t+1} - \mathbf{x}^\star)  & = \| \mathbf{y}_{t+1} - \mathbf{x}^\star \|_{{A}_t}^2 \\
& \ge  \| \mathbf{x}_{t+1} - \mathbf{x}^\star \|_{{A}_t}^2   \\
&  = (\mathbf{x}_{t+1} - \mathbf{x}^\star)^\top {A}_t(\mathbf{x}_{t+1} - \mathbf{x}^\star ).
\end{aligned}$$ This inequality is the reason for using generalized
projections as opposed to standard projections, which were used in the
analysis of online gradient descent(see
§[3.1](#section:ogd){reference-type="ref" reference="section:ogd"}
Equation [\[eqn:ogdtriangle\]](#eqn:ogdtriangle){reference-type="eqref"
reference="eqn:ogdtriangle"}). This fact together with
[\[eq:multiplied\]](#eq:multiplied){reference-type="eqref"
reference="eq:multiplied"} gives $$\begin{aligned}
\nabla_t^\top (\mathbf{x}_t \! -\! \mathbf{x}^\star) &\leq \ \frac{1}{2\gamma}
\nabla_t^\top {A}_t^{-1} \nabla_t + \frac{\gamma}{2} (\mathbf{x}_t\! -\!
\mathbf{x}^\star)^\top {A}_t (\mathbf{x}_t\! -\! \mathbf{x}^\star) \\
&  - \frac{\gamma}{2}
(\mathbf{x}_{t+1} - \mathbf{x}^\star)^\top {A}_t(\mathbf{x}_{t+1} - \mathbf{x}^\star).
\end{aligned}$$ Now, summing up over $t=1$ to $T$ we get that
$$\begin{aligned}
&\sum_{t=1}^T \nabla_t^\top (\mathbf{x}_t - \mathbf{x}^\star)
 \leq \frac{1}{2\gamma} \sum_{t=1}^T \nabla_t^\top {A}_t^{-1} \nabla_t +
\frac{\gamma}{2}  (\mathbf{x}_{1} - \mathbf{x}^\star)^\top {A}_1 (\mathbf{x}_{1} - \mathbf{x}^\star) \\
&\quad + \frac{\gamma}{2} \sum_{t=2}^{T} (\mathbf{x}_t - \mathbf{x}^\star)^\top
({A}_t - {A}_{t-1}) (\mathbf{x}_t - \mathbf{x}^\star) \\
& \quad - \frac{\gamma}{2} (\mathbf{x}_{T+1} - \mathbf{x}^\star)^\top {A}_T (\mathbf{x}_{T+1} - \mathbf{x}^\star) \\
&\leq \frac{1}{2\gamma} \sum_{t=1}^T \nabla_t^\top {A}_t^{-1}
\nabla_t+ \frac{\gamma}{2} \sum_{t=1}^{T} (\mathbf{x}_t\! -\! \mathbf{x}^\star)^\top
\nabla_t \nabla_t^\top (\mathbf{x}_t\! -\! \mathbf{x}^\star) \\
& + \frac{\gamma}{2}
(\mathbf{x}_{1} - \mathbf{x}^\star)^\top ({A}_1 - \nabla_1\nabla_1^\top) (\mathbf{x}_{1} -
\mathbf{x}^\star).
\end{aligned}$$ In the last inequality we use the fact that
$A_t - A_{t-1} =
\nabla_t \nabla_t^\top$, and the fact that the matrix $A_T$ is PSD and
hence the last term before the inequality is negative. Thus,
$$\sum_{t=1}^T R_t\ \leq\ \frac{1}{2 \gamma } \sum_{t=1}^T
\nabla_t^\top {A}_t^{-1} \nabla_t + \frac{\gamma}{2} (\mathbf{x}_{1} -
\mathbf{x}^\star)^\top ({A}_1 - \nabla_1\nabla_1^\top) (\mathbf{x}_{1} - \mathbf{x}^\star).$$

Using the algorithm parameters
${A}_1 - \nabla_1 \nabla_1^\top = \varepsilon
\mathbf{I}_n$ , $\varepsilon = \frac{1}{\gamma^2 D^2}$ and our notation
for the diameter $\|\mathbf{x}_1 - \mathbf{x}^\star\|^2 \leq D^2$ we
have $$\begin{aligned}
\ensuremath{\mathrm{{Regret}}}_T(\text{\em ONS})\ & \leq\ &  \sum_{t=1}^T R_t\
\leq\ \frac{1}{2 \gamma } \sum_{t=1}^T \nabla_t^\top {A}_t^{-1}
\nabla_t + \frac{ \gamma }{2}  {D^2}{\varepsilon} \\
& \leq & \frac{1}{2 \gamma } \sum_{t=1}^T \nabla_t^\top {A}_t^{-1}
\nabla_t + \frac{1}{2 \gamma}.
\end{aligned}$$ Since $\gamma = \frac{1}{2}\min\{\frac{1}{GD},\alpha\}$,
we have $\frac{1}{\gamma} \leq 2( \frac{1}{\alpha} + GD)$. This gives
the lemma. ◻
:::

We can now prove Theorem [4.5](#thm:onsregret){reference-type="ref"
reference="thm:onsregret"}.

::: proof
*Proof of Theorem [4.5](#thm:onsregret){reference-type="ref"
reference="thm:onsregret"}.* First we show that the term
$\sum_{t=1}^T \nabla_t^\top {A}_t^{-1}
\nabla_t$ is upper bounded by a telescoping sum. Notice that
$$\nabla_t^\top {A}_t^{-1} \nabla_t = A_t^{-1} \bullet \nabla_t \nabla_t^\top = A_t^{-1} \bullet (A_{t} - A_{t-1})$$
where for matrices $A,B \in {\mathbb R}^{n \times n}$ we denote by $A
\bullet B = \sum_{i = 1}^n \sum_{j=1}^nA_{ij} B_{ij} = {\bf Tr}(AB^\top)$,
which is equivalent to the inner product of these matrices as vectors in
${\mathbb R}^{n^2}$.

For real numbers $a,b \in {\mathbb R}_+$, the first order Taylor
expansion of the logarithm of $b$ at $a$ implies
$a^{-1} (a-b) \leq \log \frac{a}{b}$. An analogous fact holds for
positive semidefinite matrices, i.e., $A^{-1} \bullet (A-B)
\leq \log \frac{|A|}{|B|}$, where $|A|$ denotes the determinant of the
matrix $A$ (this is proved in Lemma
[4.7](#lem:logdet){reference-type="ref" reference="lem:logdet"}). Using
this fact we have $$\begin{aligned}
\sum_{t=1}^T \nabla_t^\top {A}_t^{-1} \nabla_t & = & \sum_{t=1}^T
A_t^{-1} \bullet \nabla_t \nabla_t^\top \\
& = & \sum_{t=1}^T A_t^{-1} \bullet (A_{t} - A_{t-1}) \\
& \leq &  \sum_{t=1}^T \log \frac{ |A_t|} {|A_{t-1}|} = \log
\frac{|A_T|}{|A_0|}.
\end{aligned}$$

Since $A_T = \sum_{t=1}^T \nabla_t\nabla_t^\top + \varepsilon I_n$ and
$\|\nabla_t\| \leq G$, the largest eigenvalue of $A_T$ is at most
$T G^2 + \varepsilon$. Hence the determinant of $A_T$ can be bounded by
$|A_T | \leq (T G^2 + \varepsilon)^n$. Hence recalling that
$\varepsilon = \frac{1}{\gamma^2D^2}$ and $\gamma =
\frac{1}{2}\min\{\frac{1}{GD},\alpha\}$, for $T > 4$,

$$\begin{aligned}
\sum_{t=1}^T \nabla_t^\top {A}_t^{-1} \nabla_t\ & \leq\   \log
\left( \frac{T G^2 + \varepsilon}{\varepsilon }\right)^n \leq n
\log (TG^2 \gamma^2 D^2 + 1) \leq n \log T.
\end{aligned}$$ Plugging into Lemma
[4.6](#lemma:onsbound){reference-type="ref" reference="lemma:onsbound"}
we obtain
$$\ensuremath{\mathrm{{Regret}}}_T(\text{ONS})\ \leq\  \left(\frac{1}{\alpha} +  GD\right) (n \log T + 1),$$
which implies the theorem for $n > 1, \  T \geq 4$. ◻
:::

It remains to prove the technical lemma for positive semidefinite (PSD)
matrices used above.

::: {#lem:logdet .lemma}
**Lemma 4.7**. *Let $A \succcurlyeq B \succ 0$ be positive definite
matrices. Then $$A^{-1} \bullet (A - B) \ \leq\ \log \frac{|A|}{|B|}$$*
:::

::: proof
*Proof.* For any positive definite matrix $C$, denote by
$\lambda_1(C), \ldots, \lambda_n(C)$ its eigenvalues (which are
positive). $$\begin{aligned}
& A^{-1} \bullet (A - B) \   =\  {\bf Tr}(A^{-1} (A - B)) \\
& =  {\bf Tr}(A^{-1/2} (A - B) A^{-1/2}) & {\bf Tr}(XY) = {\bf Tr}(YX) \\
& =  {\bf Tr}(I - A^{-1/2} B  A^{-1/2}) \\
& =  \sum_{i=1}^n \left[ 1 - \lambda_i( A^{-1/2} B  A^{-1/2}) \right] &  {\bf Tr}(C) = \sum_{i=1}^n \lambda_i(C)  \\
& \leq  - \sum_{i=1}^n \log \left[  \lambda_i( A^{-1/2} B
A^{-1/2}) \right] &  1-x \leq -\log(x)  \\
& =  - \log \left[ \prod_{i=1}^n  \lambda_i( A^{-1/2} B
A^{-1/2}) \right] \\
& =  - \log  |  A^{-1/2} B A^{-1/2}|  = \log \frac{|A|}{|B|} &
 |C| = \prod_{i=1}^n \lambda_i(C)
\end{aligned}$$ In the last equality we use the facts $|AB| = |A||B|$
and $|A^{-1}| = \frac{1}{|A|}$ for positive definite matrices (see
exercises). ◻
:::

##### Implementation and running time. {#implementation-and-running-time. .unnumbered}

The online Newton step algorithm requires $O(n^2)$ space to store the
matrix $A_t$. Every iteration requires the computation of the matrix
$A_{t}^{-1}$, the current gradient, a matrix-vector product, and
possibly a projection onto the underlying convex set
$\ensuremath{\mathcal K}$.

A naı̈ve implementation would require computing the inverse of the matrix
$A_t$ on every iteration. However, in the case that $A_t$ is invertible,
the matrix inversion lemma (see bibliography) states that for invertible
matrix $A$ and vector $\mathbf{x}$,
$$(A + \mathbf{x}\mathbf{x}^\top)^{-1} = A^{-1} - \frac{A^{-1} \mathbf{x}\mathbf{x}^\top A^{-1}}{1 + \mathbf{x}^\top A^{-1} \mathbf{x}}.$$
Thus, given $A_{t-1}^{-1}$ and $\nabla_t$ one can compute $A_t^{-1}$ in
time $O(n^2)$ using only matrix-vector and vector-vector products.

The online Newton step algorithm also needs to make projections onto
$\ensuremath{\mathcal K}$, but of a slightly different nature than
online gradient descent and other online convex optimization algorithms.
The required projection, denoted by
$\mathop{\Pi}_\ensuremath{\mathcal K}^{A_t}$, is in the vector norm
induced by the matrix $A_t$, viz.
$\|\mathbf{x}\|_{A_t} = \sqrt{\mathbf{x}^\top A_t \mathbf{x}}$. It is
equivalent to finding the point $\mathbf{x}\in \ensuremath{\mathcal K}$
which minimizes
$(\mathbf{x}- \mathbf{y})^\top A_t(\mathbf{x}- \mathbf{y})$ where
$\ensuremath{\mathbf y}$ is the point we are projecting. This is a
convex program which can be solved up to any degree of accuracy in
polynomial time.

Modulo the computation of generalized projections, the online Newton
step algorithm can be implemented in time and space $O(n^2)$. In
addition, the only information required is the gradient at each step
(and the exp-concavity constant $\alpha$ of the loss functions).

## Bibliographic Remarks {#bibliographic-remarks-1}

The Geometric Brownian Motion model for stock prices was suggested and
studied as early as 1900 in the PhD thesis of Louis Bachelier
[@bachelier], see also [@osborne], and used in the Nobel Prize winning
work of Black and Scholes on options pricing [@black-scholes]. In a
strong deviation from standard financial theory, Thomas Cover [@cover]
put forth the universal portfolio model, whose algorithmic theory we
have historically sketched in chapter
[1](#chap:intro){reference-type="ref" reference="chap:intro"}. The EWOO
algorithm was essentially given in Cover's paper for the application of
portfolio selection and logarithmic loss functions, and extended to
exp-concave loss functions in [@HazanKKA06]. The randomized extension of
Cover's algorithm that runs in polynomial running time is due to
@KalaiVempalaPortfolios, and it naturally extends to EWOO.

Some bridges between classical portfolio theory and the universal model
appear in [@AbernethyStoc12]. Options pricing and its relation to regret
minimization was recently also explored in the work of [@DKM-options].

Exp-concave functions have been considered in the context of prediction
in [@kivinen-warmuth], see also [@CesaBianchiLugosi06book] (chapter 3.3
and bibliography). A more general condition than exp-concavity called
mixability was used by @vovk1990aggregating to give a general
multiplicative update algorithm, see also [@foster2018logistic]. For a
thorough discussion of various conditions that allow logarithmic regret
in online learning see [@van2015fast].

For the square-loss, [@Azoury] gave a specially tailored and
near-optimal prediction algorithm. Logarithmic regret algorithms for
online convex optimization and the Online Newton Step algorithm were
presented in [@HAK07].

Logarithmic regret algorithms were used to derive
$\tilde{O}(\frac{1}{\varepsilon})$-convergent algorithms for non-smooth
convex optimization in the context of training support vector machines
in [@Shalev-ShwartzSSC11]. Building upon these results, tight
convergence rates of SGD for strongly convex and non-smooth functions
were obtained in [@hazan:beyond].

The Sherman-Morrison formula, a.k.a. the matrix inversion lemma, gives
the form of the inverse of a matrix after a rank-1 update, see
[@pseudoinverse].

## Exercises

# Regularization {#chap:regularization}

In the previous chapters we have explored algorithms for OCO that are
motivated by convex optimization. However, unlike convex optimization,
the OCO framework optimizes the Regret performance metric. This
distinction motivates a family of algorithms, called "Regularized Follow
The Leader" (RFTL), which we introduce in this chapter.

In an OCO setting of regret minimization, the most straightforward
approach for the online player is to use at any time the optimal
decision (i.e., point in the convex set) in hindsight. Formally, let
$$\ensuremath{\mathbf x}_{t+1} = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \sum_{\tau=1}^{t} f_\tau(\ensuremath{\mathbf x}).$$
This flavor of strategy is known as "fictitious play" in economics, and
has been named "Follow the Leader" (FTL) in machine learning. It is not
hard to see that this simple strategy fails miserably in a worst-case
sense. That is, this strategy's regret can be linear in the number of
iterations, as the following example shows: Consider
$\ensuremath{\mathcal K}= [-1,1]$, let $f_1(x) = \frac{1}{2} x$, and let
$f_\tau$ for $\tau=2 , \ldots , T$ alternate between $- x$ or $x$. Thus,
$$\sum_{\tau=1}^t f_\tau(x)  = {
\left\{
\begin{array}{ll}
    { \frac{1}{2} x }, & {t \mbox{  is odd} } \\\\
    {-\frac{1}{2} x }, & {\text{otherwise}}
\end{array}
\right. }$$ The FTL strategy will keep shifting between $x_t = -1$ and
$x_t = 1$, always making the wrong choice.

The intuitive FTL strategy fails in the example above because it is
unstable. Can we modify the FTL strategy such that it won't change
decisions often, thereby causing it to attain low regret?

This question motivates the need for a general means of stabilizing the
FTL method. Such a means is referred to as "regularization".

## Regularization Functions

In this chapter we consider regularization functions, denoted
$R : \ensuremath{\mathcal K}\mapsto {\mathbb R}$, which are strongly
convex and smooth (recall definitions in
§[2.1](#sec:optdefs){reference-type="ref" reference="sec:optdefs"}).

Although it is not strictly necessary, we assume that the regularization
functions in this chapter are twice differentiable over
$\ensuremath{\mathcal K}$ and, for all points
$\ensuremath{\mathbf x}\in \text{int}(\ensuremath{\mathcal K})$ in the
interior of the decision set, have a Hessian
$\nabla^2 R(\ensuremath{\mathbf x})$ that is, by the strong convexity of
$R$, positive definite.

We denote the diameter of the set $\ensuremath{\mathcal K}$ relative to
the function $R$ as
$$D_R = \sqrt{ \max_{\ensuremath{\mathbf x},\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}} \{ R(\ensuremath{\mathbf x}) - R(\ensuremath{\mathbf y}) \}} .$$

Henceforth we make use of general norms and their dual. The dual norm to
a norm $\| \cdot \|$ is given by the following definition:
$$\| \ensuremath{\mathbf y}\|^* \stackrel{\text{\tiny def}}{=}\sup_{ \| \ensuremath{\mathbf x}\| \leq 1 }  \left\{ \ensuremath{\mathbf x}^\top \ensuremath{\mathbf y}\right\} .$$
A positive definite matrix $A$ gives rise to the matrix norm
$\|\ensuremath{\mathbf x}\|_A = \sqrt{\ensuremath{\mathbf x}^\top A \ensuremath{\mathbf x}}$.
The dual norm of a matrix norm is
$\|\ensuremath{\mathbf x}\|_A^*=\|\ensuremath{\mathbf x}\|_{A^{-1}}$.

The generalized Cauchy-Schwarz theorem asserts
$\ensuremath{\mathbf x}^\top  \ensuremath{\mathbf y}\leq \| \ensuremath{\mathbf x}\| \| \ensuremath{\mathbf y}\|^*$
and in particular for matrix norms,
$\ensuremath{\mathbf x}^\top \ensuremath{\mathbf y}\leq \|\ensuremath{\mathbf x}\|_A \| \ensuremath{\mathbf y}\|_A^*$
(see exercises).

In our derivations, we usually consider matrix norms with respect to
$\nabla^2R(\ensuremath{\mathbf x})$, the Hessian of the regularization
function $R(\ensuremath{\mathbf x})$, as well as the inverse Hessian
denoted $\nabla^{-2} R(\ensuremath{\mathbf x})$. In such cases, we use
the notation
$$\|\ensuremath{\mathbf x}\|_\ensuremath{\mathbf y}\stackrel{\text{\tiny def}}{=}\|\ensuremath{\mathbf x}\|_{\nabla^2 {R}(\ensuremath{\mathbf y})} ,$$
and similarly
$$\|\ensuremath{\mathbf x}\|_\ensuremath{\mathbf y}^* \stackrel{\text{\tiny def}}{=}\|\ensuremath{\mathbf x}\|_{\nabla^{-2} {R}(\ensuremath{\mathbf y})} .$$

A crucial quantity in the analysis of OCO algorithms that use
regularization is the remainder term of the Taylor approximation of the
regularization function, and especially the remainder term of the first
order Taylor approximation. The difference between the value of the
regularization function at $\ensuremath{\mathbf x}$ and the value of the
first order Taylor approximation is known as the Bregman divergence,
given by

::: definition
**Definition 5.1**. *Denote by
$B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y})$ the Bregman
divergence with respect to the function ${R}$, defined as
$$B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y}) = {R}(\ensuremath{\mathbf x}) - {R}(\ensuremath{\mathbf y}) - \nabla {R}(\ensuremath{\mathbf y})^\top  (\ensuremath{\mathbf x}-\ensuremath{\mathbf y}) .$$*
:::

For twice differentiable functions, Taylor expansion and the mean-value
theorem assert that the Bregman divergence is equal to the second
derivative at an intermediate point, i.e., (see exercises)
$$B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y}) = \frac{1}{2} \|\ensuremath{\mathbf x}- \ensuremath{\mathbf y}\|_\ensuremath{\mathbf z}^2,$$
for some point
$\ensuremath{\mathbf z}\in [\ensuremath{\mathbf x},\ensuremath{\mathbf y}]$,
meaning there exists some $\alpha \in [0,1]$ such that
$\ensuremath{\mathbf z}= \alpha \ensuremath{\mathbf x}+ (1-\alpha) \ensuremath{\mathbf y}$.
Therefore, the Bregman divergence defines a local norm, which has a dual
norm. We shall denote this dual norm by
$$\| \cdot \|_{\ensuremath{\mathbf x},\ensuremath{\mathbf y}}^*  \stackrel{\text{\tiny def}}{=}\| \cdot \|_\ensuremath{\mathbf z}^*.$$
With this notation we have
$$B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y}) = \frac{1}{2} \|\ensuremath{\mathbf x}- \ensuremath{\mathbf y}\|_{\ensuremath{\mathbf x},\ensuremath{\mathbf y}} ^2.$$
In online convex optimization, we commonly refer to the Bregman
divergence between two consecutive decision points
$\ensuremath{\mathbf x}_t$ and $\ensuremath{\mathbf x}_{t+1}$. In such
cases, we shorthand notation for the norm defined by the Bregman
divergence with respect to ${R}$ on the intermediate point in
$[\ensuremath{\mathbf x}_t,\ensuremath{\mathbf x}_{t+1}]$ as
$\| \cdot \|_t \stackrel{\text{\tiny def}}{=}\| \cdot \|_{\ensuremath{\mathbf x}_t,\ensuremath{\mathbf x}_{t+1}}$.
The latter norm is called the local norm at iteration $t$. With this
notation, we have
$B_{R}(\ensuremath{\mathbf x}_t||\ensuremath{\mathbf x}_{t+1}) = \frac{1}{2} \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}_{t+1}\|_t^2$.

Finally, we consider below generalized projections that use the Bregman
divergence as a distance instead of a norm. Formally, the projection of
a point $\ensuremath{\mathbf y}$ according to the Bregman divergence
with respect to function $R$ is given by
$$\mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y}) .$$

## The RFTL Algorithm and its Analysis

Recall the caveat with straightforward use of follow-the-leader: as in
the bad example we have considered, the predictions of FTL may vary
wildly from one iteration to the next. This motivates the modification
of the basic FTL strategy in order to stabilize the prediction. By
adding a regularization term, we obtain the RFTL (Regularized Follow the
Leader) algorithm.

We proceed to formally describe the RFTL algorithmic template and
analyze it. The analysis gives asymptotically optimal regret bounds.
However, we do not optimize the constants in the regret bounds in order
to improve clarity of presentation.

Throughout this chapter, recall the notation of $\nabla_t$ to denote the
gradient of the current cost function at the current point, i.e.,
$$\nabla_t \stackrel{\text{\tiny def}}{=}\nabla f_t(\ensuremath{\mathbf x}_t) .$$
In the OCO setting, the regret of convex cost functions can be bounded
by a linear function via the inequality
$f_t(\ensuremath{\mathbf x_{t}}) - f_t(\ensuremath{\mathbf x}^\star) \leq \nabla_t^\top (\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x}^\star)$.
Thus, the overall regret (recall definition
[\[eqn:regret-defn\]](#eqn:regret-defn){reference-type="eqref"
reference="eqn:regret-defn"}) of an OCO algorithm can be bounded by
$$\label{eqn:rftl-shalom}
\sum_t f_t(\ensuremath{\mathbf x}_t)  - f_t(\ensuremath{\mathbf x}^\star) \leq \sum_t \nabla_t^\top (\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^\star).$$

### Meta-algorithm definition

The generic RFTL meta-algorithm is defined in Algorithm
[\[alg:RFTLmain\]](#alg:RFTLmain){reference-type="ref"
reference="alg:RFTLmain"}. The regularization function ${R}$ is assumed
to be strongly convex, smooth, and twice differentiable.

::: algorithm
::: algorithmic
Input: $\eta > 0$, regularization function ${R}$, and a bounded, convex
and closed set $\ensuremath{\mathcal K}$. Let
$\ensuremath{\mathbf x_{1}}  = \arg\min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} {\left\{ {R}(\ensuremath{\mathbf x})\right\} }$.
Play $\ensuremath{\mathbf x}_t$ and observe cost
$f_t(\ensuremath{\mathbf x}_t)$. Update $$\begin{aligned}
			\ensuremath{\mathbf x_{t+1}} = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} {\left\{\eta\sum_{s=1}^t \nabla_s^\top \ensuremath{\mathbf x}+ {R}(\ensuremath{\mathbf x})\right\}}

\end{aligned}$$
:::
:::

### The regret bound {#sec:thm1.1}

::: {#thm:RFTLmain1 .theorem}
**Theorem 5.2**. *The RFTL Algorithm
[\[alg:RFTLmain\]](#alg:RFTLmain){reference-type="ref"
reference="alg:RFTLmain"} attains for every
$\ensuremath{\mathbf u}\in \ensuremath{\mathcal K}$ the following bound
on the regret:
$$\ensuremath{\mathrm{{Regret}}}_T \le    2   \eta  \sum_{t=1}^T \| \nabla_t \|_t^{* 2} + \frac{R(\ensuremath{\mathbf u}) - R(\ensuremath{\mathbf x}_1)}{\eta }  .$$*
:::

If an upper bound on the local norms is known, i.e.,
$\| \nabla_t\|_t^* \leq G_R$ for all times $t$, then we can further
optimize over the choice of $\eta$ to obtain
$$\ensuremath{\mathrm{{Regret}}}_T \leq  2  D_R G_R \sqrt{ 2T  } .$$

To prove Theorem [5.2](#thm:RFTLmain1){reference-type="ref"
reference="thm:RFTLmain1"}, we first relate the regret to the
"stability" in prediction. This is formally captured by the following
lemma.

::: {#lem:FTL-BTL .lemma}
**Lemma 5.3**. *Algorithm
[\[alg:RFTLmain\]](#alg:RFTLmain){reference-type="ref"
reference="alg:RFTLmain"} guarantees the following regret bound
$$\begin{aligned}
		\ensuremath{\mathrm{{Regret}}}_T  \leq  \sum_{t=1}^T \nabla_t^\top
		(\ensuremath{\mathbf x_{t}}-\ensuremath{\mathbf x_{t+1}}) + \frac{1}{\eta} D_R^2 % B_{R}(\uv||\xv[1])

\end{aligned}$$*
:::

::: proof
*Proof.* For convenience of the derivations, define the functions
$$g_0(\mathbf{x}) \stackrel{\text{\tiny def}}{=}\frac{1}{\eta}R(\mathbf{x})  \ , \ g_t(\mathbf{x}) \stackrel{\text{\tiny def}}{=}\nabla_t^\top \mathbf{x}.$$
By equation
[\[eqn:rftl-shalom\]](#eqn:rftl-shalom){reference-type="eqref"
reference="eqn:rftl-shalom"}, it suffices to bound
$\sum_{t=1}^T [ g_t(\ensuremath{\mathbf x_{t}}) - g_t (\ensuremath{\mathbf u})]$.
As a first step, we prove the following inequality:

::: {#prop:ftl-btl .lemma}
**Lemma 5.4**. *For every
$\ensuremath{\mathbf u}\in \ensuremath{\mathcal K}$,
$$\sum_{t=0}^T g_t(\ensuremath{\mathbf u})  \geq  \sum_{t=0}^T g_t(\ensuremath{\mathbf x_{t+1}}) .$$*
:::

::: proof
*Proof.* by induction on $T$:\
\
By definition, we have that
$\ensuremath{\mathbf x_{1}} = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \{R(\ensuremath{\mathbf x})\}$,
and thus
$g_0(\ensuremath{\mathbf u}) \ge g_0(\ensuremath{\mathbf x_{1}})$ for
all $\ensuremath{\mathbf u}$.\
**Induction step:**\
Assume that for $T$, we have $$\begin{aligned}
			  \sum_{t=0}^{T} g_t(\ensuremath{\mathbf u})  \geq \sum_{t=0}^{T} g_t(\ensuremath{\mathbf x_{t+1}} )

\end{aligned}$$ and let us prove the statement for $T+1$. Since
$\ensuremath{\mathbf x_{T+2}} = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \{ \sum_{t=0}^{T+1} g_t(\ensuremath{\mathbf x})\}$
we have: $$\begin{aligned}
			  \sum_{t=0}^{T+1} g_t (\ensuremath{\mathbf u}) & \geq  &   \sum_{t=0}^{T+1} g_t (\ensuremath{\mathbf x_{T+2}})  \\
			& = &    \sum_{t=0}^{T} g_t (\ensuremath{\mathbf x_{T+2}})  + g_{T+1}(\ensuremath{\mathbf x_{T+2}})  \\
			& \geq &    \sum_{t=0}^{T}   g_t (\ensuremath{\mathbf x_{t+1}})  + g_{T+1}(\ensuremath{\mathbf x_{T+2}})  \\
			& = &  \sum_{t=0}^{T+1}  g_t (\ensuremath{\mathbf x_{t+1}}).

\end{aligned}$$ Where in the third line we used the induction hypothesis
for $\ensuremath{\mathbf u}= \ensuremath{\mathbf x_{T+2}}$. ◻
:::

We conclude that $$\begin{aligned}
		\sum_{t=1}^T [ g_t(\ensuremath{\mathbf x_{t}}) - g_t (\ensuremath{\mathbf u})]  & \leq &  \sum_{t=1}^T [g_t(\ensuremath{\mathbf x_{t}}) - g_t (\ensuremath{\mathbf x_{t+1}})] + \left[  g_0(\ensuremath{\mathbf u}) - g_0(\ensuremath{\mathbf x_{1}}) \right] \\
		& = & \sum_{t=1}^T g_t(\ensuremath{\mathbf x_{t}}) - g_t (\ensuremath{\mathbf x_{t+1}}) + \frac{1}{\eta} \left[  R(\ensuremath{\mathbf u})  - R(\ensuremath{\mathbf x_{1}}) \right]   \\
		& \le & \sum_{t=1}^T g_t(\ensuremath{\mathbf x_{t}}) - g_t (\ensuremath{\mathbf x_{t+1}}) + \frac{1}{\eta} D_R^2 .

\end{aligned}$$ ◻
:::

::: proof
*Proof of Theorem [5.2](#thm:RFTLmain1){reference-type="ref"
reference="thm:RFTLmain1"}.* Recall that ${R}(\ensuremath{\mathbf x})$
is a convex function and $\ensuremath{\mathcal K}$ is a convex set.
Denote:
$$\Phi_t(\ensuremath{\mathbf x}) \stackrel{\text{\tiny def}}{=}\eta\sum_{s=1}^t \nabla_s^\top \ensuremath{\mathbf x}+ {R}(\ensuremath{\mathbf x}) .$$
By the Taylor expansion (with its explicit remainder term via the
mean-value theorem) at $\ensuremath{\mathbf x_{t+1}}$, and by the
definition of the Bregman divergence, $$\begin{aligned}
	\Phi_t(\ensuremath{\mathbf x_{t}}) & = & \Phi_t(\ensuremath{\mathbf x_{t+1}})
	+  (\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}})^\top \nabla \Phi_t(\ensuremath{\mathbf x_{t+1}})
	+ B_{\Phi_t}(\ensuremath{\mathbf x}_t||\ensuremath{\mathbf x}_{t+1} ) \\ % \frac{1}{2} \|\xv - \xv[t+1]\|^2_{\zv} \\
	& \geq &
	\Phi_t(\ensuremath{\mathbf x_{t+1}}) + B_{\Phi_t} (\ensuremath{\mathbf x}_t||\ensuremath{\mathbf x}_{t+1} ) \\ %\frac{1}{2}\|\xv - \xv[t+1]\|^2_{\zv}
	& = &
	\Phi_t(\ensuremath{\mathbf x_{t+1}}) + B_{{R}} (\ensuremath{\mathbf x}_t||\ensuremath{\mathbf x}_{t+1} ). %\frac{1}{2}\|\xv - \xv[t+1]\|^2_{\zv}
\end{aligned}$$ The inequality holds since
$\ensuremath{\mathbf x_{t+1}}$ is a minimum of $\Phi_t$ over
$\ensuremath{\mathcal K}$, as in Theorem
[2.2](#thm:optim-conditions){reference-type="ref"
reference="thm:optim-conditions"}. The last equality holds since the
component $\nabla_s^\top \ensuremath{\mathbf x}$ is linear and thus does
not affect the Bregman divergence. Thus, $$\begin{aligned}
 \label{eqn:chap5shalom}
		B_{R}(\ensuremath{\mathbf x}_t || \ensuremath{\mathbf x}_{t+1})  & \leq & \,\Phi_t(\ensuremath{\mathbf x_{t}}) - \,\Phi_t(\ensuremath{\mathbf x_{t+1}}) \\
		& = & \,\ (\Phi_{t-1}(\ensuremath{\mathbf x_{t}}) - \Phi_{t-1}(\ensuremath{\mathbf x_{t+1}})) +  \eta \nabla_t^\top (\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}}) \notag \\
		& \leq & \,\eta \,\nabla_t^\top (\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}}) \quad \mbox{($\ensuremath{\mathbf x}_t$ is the minimizer)} \notag

\end{aligned}$$ To proceed, recall the shorthand for the norm induced by
the Bregman divergence with respect to ${R}$ on point
$\ensuremath{\mathbf x}_t,\ensuremath{\mathbf x}_{t+1}$ as
$\| \cdot \|_t = \| \cdot \|_{\ensuremath{\mathbf x}_t,\ensuremath{\mathbf x}_{t+1}}$.
Similarly for the dual local norm
$\| \cdot \|^*_t = \| \cdot \|^*_{\ensuremath{\mathbf x}_t,\ensuremath{\mathbf x}_{t+1}}$.
With this notation, we have
$B_{R}(\ensuremath{\mathbf x}_t||\ensuremath{\mathbf x}_{t+1}) = \frac{1}{2} \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}_{t+1}\|_t^2$.
By the generalized Cauchy-Schwarz inequality, $$\begin{aligned}
		\nabla_t^\top (\ensuremath{\mathbf x_{t}}-\ensuremath{\mathbf x_{t+1}}) &\leq \|\nabla_t \|_{t}^* \cdot
		\|\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}} \|_{t} & \mbox{ Cauchy-Schwarz} \\
		& = \|\nabla_t \|_{t}^* \cdot
		\sqrt{2 B_{R}(\ensuremath{\mathbf x}_t||\ensuremath{\mathbf x}_{t+1}) } \\
		& \leq \|\nabla_t  \|_{t}^* \cdot \sqrt{2\, \eta\, \nabla_t^\top (\ensuremath{\mathbf x_{t}}-
			\ensuremath{\mathbf x_{t+1}})  }.  & \eqref{eqn:chap5shalom} \nonumber

\end{aligned}$$ After rearranging we get $$\begin{aligned}
		\nabla_t^\top (\ensuremath{\mathbf x_{t}}-\ensuremath{\mathbf x_{t+1}}) &\leq 2\, \eta \, \|\nabla_t  \|^{* 2}_{t}.

\end{aligned}$$ Combining this inequality with Lemma
[5.3](#lem:FTL-BTL){reference-type="ref" reference="lem:FTL-BTL"} we
obtain the theorem statement. ◻
:::

## Online Mirror Descent

In the convex optimization literature, "Mirror Descent" refers to a
general class of first order methods generalizing gradient descent.
Online Mirror descent (OMD) is the online counterpart of this class of
methods. This relationship is analogous to the relationship of online
gradient descent to traditional (offline) gradient descent.

OMD is an iterative algorithm that computes the current decision using a
simple gradient update rule and the previous decision, much like OGD.
The generality of the method stems from the update being carried out in
a "dual" space, where the duality notion is defined by the choice of
regularization: the gradient of the regularization function defines a
mapping from ${\mathbb R}^n$ onto itself, which is a vector field. The
gradient updates are then carried out in this vector field.

For the RFTL algorithm the intuition was straightforward---the
regularization was used to ensure stability of the decision. For OMD,
regularization has an additional purpose: regularization transforms the
space in which gradient updates are performed. This transformation
enables better bounds in terms of the geometry of the space.

The OMD algorithm comes in two flavors: an agile and a lazy version. The
lazy version keeps track of a point in Euclidean space and projects onto
the convex decision set $\ensuremath{\mathcal K}$ only at decision time.
In contrast, the agile version maintains a feasible point at all times,
much like OGD.

::: algorithm
::: algorithmic
Input: parameter $\eta > 0$, regularization function
${R}(\ensuremath{\mathbf x})$. Let $\ensuremath{\mathbf y_{1}}$ be such
that $\nabla {R}(\ensuremath{\mathbf y_{1}}) = \mathbf{0}$ and
$\ensuremath{\mathbf x_{1}} = \arg\min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y_{1}})$.
Play $\ensuremath{\mathbf x_{t}}$. Observe the loss function $f_t$ and
let $\nabla_t = \nabla f_t(\ensuremath{\mathbf x}_t)$. Update
$\ensuremath{\mathbf y}_t$ according to the rule: $$\begin{aligned}
			&\text{[Lazy version]}
			&\nabla {R}(\ensuremath{\mathbf y_{t+1}}) = \nabla {R}(\ensuremath{\mathbf y_{t}}) - \eta\, \nabla_{t}\\
			&\text{[Agile version]}
			&\nabla {R}(\ensuremath{\mathbf y_{t+1}}) = \nabla {R}(\ensuremath{\mathbf x_{t}}) - \eta\, \nabla_{t}

\end{aligned}$$ Project according to $B_{R}$:
$$\ensuremath{\mathbf x_{t+1}} = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y_{t+1}})$$
:::
:::

Both versions can be analyzed to give roughly the same regret bounds as
the RFTL algorithm. In light of what we will see next, this is not
surprising: for linear cost functions, the RFTL and lazy-OMD algorithms
are equivalent!

Thus, we get regret bounds for free for the lazy version. The agile
version can be shown to attain similar regret bounds, and is in fact
superior in certain settings that require adaptivity. This issue is
further explored in chapter [10](#chap:adaptive){reference-type="ref"
reference="chap:adaptive"}. The analysis of the agile version is of
independent interest and we give it below.

### Equivalence of lazy OMD and RFTL

The OMD (lazy version) and RFTL are identical for linear cost functions,
as we show next.

::: lemma
**Lemma 5.5**. *Let $f_1,...,f_T$ be linear cost functions. The lazy OMD
and RFTL algorithms produce identical predictions, i.e.,
$$\mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \left\{ B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y_{t}}) \right\} = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}}
	\left( \eta \sum_{s=1}^{t-1} \nabla_s^\top \ensuremath{\mathbf x}+ {R}(\ensuremath{\mathbf x}) \right) .$$*
:::

::: proof
*Proof.* First, observe that the unconstrained minimum
$$\ensuremath{\mathbf x_{t}}^\star \stackrel{\text{\tiny def}}{=}\mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in {\mathbb R}^n}
	\bigg\{\sum_{s=1}^{t-1} \nabla_s^\top \ensuremath{\mathbf x}+ \frac{1}{\eta} {R}(\ensuremath{\mathbf x}) \bigg\}$$
satisfies
$$\nabla {R}(\ensuremath{\mathbf x_{t}}^\star) = - \eta \sum_{s=1}^{t-1} \nabla_s.$$
By definition, $\ensuremath{\mathbf y_{t}}$ also satisfies the above
equation, but since ${R}(\ensuremath{\mathbf x})$ is strictly convex,
there is only one solution for the above equation and thus
$\ensuremath{\mathbf y_{t}}= \ensuremath{\mathbf x}^\star_t$. Hence,
$$\begin{aligned}
		B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y_{t}})\ &=\ {R}(\ensuremath{\mathbf x}) - {R}(\ensuremath{\mathbf y_{t}}) - (\nabla {R}(\ensuremath{\mathbf y_{t}}))^\top (\mathbf{x}-\ensuremath{\mathbf y_{t}})\\
		&=\ {R}(\ensuremath{\mathbf x}) - {R}(\ensuremath{\mathbf y_{t}}) + \eta\,  \sum_{s=1}^{t-1} \nabla_s^\top (\ensuremath{\mathbf x}-\ensuremath{\mathbf y_{t}})~.

\end{aligned}$$ Since ${R}(\ensuremath{\mathbf y_{t}})$ and
$\sum_{s=1}^{t-1} \nabla_s^\top \ensuremath{\mathbf y_{t}}$ are
independent of $\ensuremath{\mathbf x}$, it follows that
$B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y_{t}})$ is minimized
at the point $\ensuremath{\mathbf x}$ that minimizes
${R}(\ensuremath{\mathbf x}) + \eta\, \sum_{s=1}^{t-1} \nabla_s^\top \ensuremath{\mathbf x}$
over $\ensuremath{\mathcal K}$ which, in turn, implies that
$$\begin{aligned}
		\mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y_{t}})\ =\
		\mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \bigg\{ \sum_{s=1}^{t-1} \nabla_s^\top \ensuremath{\mathbf x}+
		\frac{1}{\eta} {R}(\ensuremath{\mathbf x}) \bigg\}~.

\end{aligned}$$ ◻
:::

### Regret bounds for Mirror Descent

In this subsection we prove regret bounds for the agile version of the
RFTL algorithm. The analysis is quite different than the one for the
lazy version, and of independent interest.

::: {#thm:mirrordescent .theorem}
**Theorem 5.6**. *The OMD Algorithm
[\[alg:flpl\]](#alg:flpl){reference-type="ref" reference="alg:flpl"}
attains for every $\ensuremath{\mathbf u}\in \ensuremath{\mathcal K}$
the following bound on the regret:
$$\ensuremath{\mathrm{{Regret}}}_T \le     \frac{\eta}{4}  \sum_{t=1}^T \| \nabla_t \|_t^{* 2} + \frac{R(\ensuremath{\mathbf u}) - R(\ensuremath{\mathbf x}_1)}{2\eta }  .$$*
:::

If an upper bound on the local norms is known, i.e.,
$\| \nabla_t\|_t^* \leq G_R$ for all times $t$, then we can further
optimize over the choice of $\eta$ to obtain
$$\ensuremath{\mathrm{{Regret}}}_T \leq    D_R G_R \sqrt{ T  } .$$

::: proof
*Proof.* Since the functions $\ensuremath{\mathbf f_{t}}$ are convex,
for any $\ensuremath{\mathbf x}^* \in K$,
$$\ensuremath{\mathbf f_{t}}(\ensuremath{\mathbf x}_t) - \ensuremath{\mathbf f_{t}}(\ensuremath{\mathbf x}^*) \leq \nabla \ensuremath{\mathbf f_{t}}(\ensuremath{\mathbf x}_t)^\top (\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^*)  .$$
The following property of Bregman divergences follows from the
definition: for any vectors
$\ensuremath{\mathbf x},\ensuremath{\mathbf y},\ensuremath{\mathbf z}$,
$$(\ensuremath{\mathbf x}- \ensuremath{\mathbf y})^\top (\nabla \ensuremath{\mathcal R}(\ensuremath{\mathbf z}) - \nabla \ensuremath{\mathcal R}(\ensuremath{\mathbf y})) = B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x},\ensuremath{\mathbf y})-B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x},\ensuremath{\mathbf z}) +
B_\ensuremath{\mathcal R}(\ensuremath{\mathbf y},\ensuremath{\mathbf z}).$$
Combining both observations, $$\begin{aligned}
\ensuremath{\mathbf f_{t}}(\ensuremath{\mathbf x}_t) - \ensuremath{\mathbf f_{t}}(\ensuremath{\mathbf x}^*) & \leq \nabla \ensuremath{\mathbf f_{t}}(\ensuremath{\mathbf x}_t)^\top (\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}^*)  \\
& =   \frac{1}{\eta}  (\nabla \ensuremath{\mathcal R}(\ensuremath{\mathbf y}_{t+1}) - \nabla \ensuremath{\mathcal R}(\ensuremath{\mathbf x}_{t}))^\top(\ensuremath{\mathbf x}^* - \ensuremath{\mathbf x}_t) \\
& =  \frac{1}{\eta} [B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}^*,\ensuremath{\mathbf x_{t}})-B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}^*,\ensuremath{\mathbf y}_{t+1}) + B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}_t,\ensuremath{\mathbf y}_{t+1})]   \\
& \leq  \frac{1}{\eta} [B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}^*,\ensuremath{\mathbf x_{t}})-B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}^*,\ensuremath{\mathbf x}_{t+1}) +
B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}_t,\ensuremath{\mathbf y}_{t+1})]
\end{aligned}$$ where the last inequality follows from the generalized
Pythagorean theorem, as $\ensuremath{\mathbf x}_{t+1}$ is the projection
w.r.t the Bregman divergence of $\ensuremath{\mathbf y}_{t+1}$ and
$\ensuremath{\mathbf x}^* \in K$ is in the convex set. Summing over all
iterations, $$\begin{aligned}
 \label{eq:general1}
\ensuremath{\mathrm{{Regret}}}& \leq & \frac{1}{\eta} [ B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}^*,\ensuremath{\mathbf x}_1) -  B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}^*,\ensuremath{\mathbf x}_T) ] + \sum_{t=1}^T \frac{1}{\eta} B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}_t,\ensuremath{\mathbf y}_{t+1}) \notag \\
& \leq & \frac{1}{\eta} D^2_R  + \sum_{t=1}^T \frac{1}{\eta} B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}_t,\ensuremath{\mathbf y}_{t+1})
\end{aligned}$$

We proceed to bound
$B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}_t,\ensuremath{\mathbf y}_{t+1})$.
By definition of Bregman divergence, and the generalized Cauchy-Schwartz
inequality, $$\begin{aligned}
 B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}_t,\ensuremath{\mathbf y}_{t+1}) + B_\ensuremath{\mathcal R}(\ensuremath{\mathbf y}_{t+1},\ensuremath{\mathbf x}_t) &= (\nabla \ensuremath{\mathcal R}(\ensuremath{\mathbf x}_t) - \nabla \ensuremath{\mathcal R}(\ensuremath{\mathbf y}_{t+1}))^\top (\ensuremath{\mathbf x}_t - \ensuremath{\mathbf y}_{t+1}) \\
 &=  \eta \nabla \ensuremath{\mathbf f_{t}}(\ensuremath{\mathbf x}_t)^\top(\ensuremath{\mathbf x}_t - \ensuremath{\mathbf y}_{t+1}) \\
 & \leq \eta \| \nabla \ensuremath{\mathbf f_{t}}(\ensuremath{\mathbf x}_t) \|^*_t \| \ensuremath{\mathbf x}_t - \ensuremath{\mathbf y}_{t+1} \|_t \\
 &\leq  \frac{1}{2} \eta^2 G_R^{ 2} + \frac{1}{2} \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf y}_{t+1}\|^2_t.
\end{aligned}$$ where in the last inequality follows from
$(a-b)^2 \geq 0$. Thus, we have
$$B_\ensuremath{\mathcal R}(\ensuremath{\mathbf x}_t,\ensuremath{\mathbf y}_{t+1}) \leq \frac{1}{2} \eta^2 G_R^2 + \frac{1}{2} \|\ensuremath{\mathbf x}_t -
\ensuremath{\mathbf y}_{t+1}\|^2_t - B_\ensuremath{\mathcal R}(\ensuremath{\mathbf y}_{t+1},\ensuremath{\mathbf x}_t) = \frac{1}{2} \eta^2 G^2_R.$$

Plugging back into Equation
[\[eq:general1\]](#eq:general1){reference-type="eqref"
reference="eq:general1"}, and by non-negativity of the Bregman
divergence, we get
$$\ensuremath{\mathrm{{Regret}}}\leq  \frac{1}{2} [\frac{1}{\eta} D^2_R + \frac{1}{2} \eta T G_{R}^{2} ] \leq  D_R G_R \sqrt{T} \ ,$$
by taking $\eta = \frac{ D_R}{\sqrt{T} G_R}$ ◻
:::

## Application and Special Cases

In this section we illustrate the generality of the regularization
technique: we show how to derive the two most important and famous
online algorithms---the online gradient descent algorithm and the online
exponentiated gradient (based on the multiplicative update
method)---from the RFTL meta-algorithm.

Other important special cases of the RFTL meta-algorithm are derived
with matrix-norm regularization---namely, the von Neumann entropy
function, and the log-determinant function, as well as self-concordant
barrier regularization---which we shall explore in detail in the next
chapter.

### Deriving online gradient descent

To derive the online gradient descent algorithm, we take
${R}(\ensuremath{\mathbf x}) = \frac{1}{2} \|\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_0\|_2^2$
for an arbitrary $\ensuremath{\mathbf x}_0 \in \ensuremath{\mathcal K}$.
Projection with respect to this divergence is the standard Euclidean
projection (see exercises), and in addition,
$\nabla {R}(\ensuremath{\mathbf x}) = \ensuremath{\mathbf x}- \ensuremath{\mathbf x}_0$.
Hence, the update rule for the OMD Algorithm
[\[alg:flpl\]](#alg:flpl){reference-type="ref" reference="alg:flpl"}
becomes: $$\begin{aligned}
	& \ensuremath{\mathbf x_{t}}= \mathop{\Pi}_\ensuremath{\mathcal K}(  \ensuremath{\mathbf y_{t}})  , \ \ensuremath{\mathbf y_{t}}=  \ensuremath{\mathbf y_{t-1}}  - \eta \nabla_{t-1}  & \mbox{lazy version}  \\
	& \ensuremath{\mathbf x_{t}}= \mathop{\Pi}_\ensuremath{\mathcal K}(  \ensuremath{\mathbf y_{t}})  , \ \ensuremath{\mathbf y_{t}}=  \ensuremath{\mathbf x_{t-1}}  - \eta \nabla_{t-1}  & \mbox{agile version}
\end{aligned}$$

The latter algorithm is exactly online gradient descent, as described in
Algorithm [\[alg:ogd\]](#alg:ogd){reference-type="ref"
reference="alg:ogd"} in chapter
[3](#chap:first order){reference-type="ref"
reference="chap:first order"}. However, both variants behave very
differently, as explored in chapter
[10](#chap:adaptive){reference-type="ref" reference="chap:adaptive"}
(see also exercises).

Theorem [5.2](#thm:RFTLmain1){reference-type="ref"
reference="thm:RFTLmain1"} gives us the following bound on the regret
(where $D_R, \| \cdot\|_t$ are the diameter and local norm defined with
respect to the regularizer $R$ as defined in the beginning of this
chapter, and $D$ is the Euclidean diameter as defined in chapter
[2](#chap:opt){reference-type="ref" reference="chap:opt"})
$$\ensuremath{\mathrm{{Regret}}}_T \le \frac{1}{\eta }  D_R ^2  + 2 \eta \sum_t \| \nabla_t \|_t^{* 2}    \leq \frac{1}{2 \eta} D^2 + 2 \eta  \sum_t \|\nabla_t \|^2 \leq 2GD \sqrt{  T  },$$
where the second inequality follows since for
${R}(\ensuremath{\mathbf x}) = \frac{1}{2} \|\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_0\|^2$,
the local norm $\|\cdot\|_t$ reduces to the Euclidean norm.

### Deriving multiplicative updates

Let
${R}(\ensuremath{\mathbf x_{}}) =  \ensuremath{\mathbf x_{}} \log \ensuremath{\mathbf x_{}} = \sum_i \ensuremath{\mathbf x}_i \log \ensuremath{\mathbf x}_i$
be the negative entropy function, where $\log \ensuremath{\mathbf x}$ is
to be interpreted element-wise. Then
$\nabla {R}(\ensuremath{\mathbf x}) = \mathbf{1}+ \log \ensuremath{\mathbf x}$,
and hence the update rules for the OMD algorithm become:
$$\begin{aligned}
	& \ensuremath{\mathbf x_{t}}= \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y_{t}})    , \ \log \ensuremath{\mathbf y_{t}}=  \log \ensuremath{\mathbf y_{t-1}}  - \eta \nabla_{t-1}  & \mbox{lazy version}  \\
	& \ensuremath{\mathbf x_{t}}= \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} B_{R}(\ensuremath{\mathbf x}||\ensuremath{\mathbf y_{t}})    , \ \log \ensuremath{\mathbf y_{t}}=  \log \ensuremath{\mathbf x_{t-1}}  - \eta \nabla_{t-1}  & \mbox{agile version}
\end{aligned}$$

With this choice of regularizer, a notable special case is the experts
problem we encountered in §[1.3](#sec:experts){reference-type="ref"
reference="sec:experts"}, for which the decision set
$\ensuremath{\mathcal K}$ is the $n$-dimensional simplex
$\Delta_n = \{ \ensuremath{\mathbf x}\in {\mathbb R}^n_+ \ | \ \sum_i \ensuremath{\mathbf x}_i =  1  \}$.
In this special case, the projection according to the negative entropy
becomes scaling by the $\ell_1$ norm (see exercises), which implies that
both update rules amount to the same algorithm:
$$\ensuremath{\mathbf x}_{t+1}(i) = \frac{\ensuremath{\mathbf x}_t(i) \cdot e^{-\eta \nabla_t(i)}}{\sum_{j=1}^n \ensuremath{\mathbf x}_t(j) \cdot e^{-\eta \nabla_t(j)} },$$
which is exactly the Hedge algorithm from the first chapter!

Theorem [5.6](#thm:mirrordescent){reference-type="ref"
reference="thm:mirrordescent"} gives us the following bound on the
regret:
$$\ensuremath{\mathrm{{Regret}}}_T \le 2 \sqrt{  2 D_R^2  \sum_t \| \nabla_t \|_t^{* 2}   }  .$$
If the costs per individual expert are in the range $[0,1]$, it can be
shown that $$\|\nabla_t\|_t^* \leq \| \nabla_t \|_\infty \leq 1 = G_R.$$
In addition, when $R$ is the negative entropy function, the diameter
over the simplex can be shown to be bounded by $D_R^2 \leq \log n$ (see
exercises), giving rise to the bound
$$\ensuremath{\mathrm{{Regret}}}_T \le 2  D_R G_R \sqrt{2 T   }  \leq 2\sqrt{2 T \log n}.$$

For an arbitrary range of costs, we obtain the exponentiated gradient
algorithm described in Algorithm
[\[alg:eg\]](#alg:eg){reference-type="ref" reference="alg:eg"}.

::: algorithm
::: algorithmic
Input: parameter $\eta > 0$. Let
$\ensuremath{\mathbf y_{1}} = \mathbf{1}\ , \ \ensuremath{\mathbf x_{1}} = \frac{\ensuremath{\mathbf y_{1}}}{\|\ensuremath{\mathbf y_{1}}\|_1}$.
Predict $\ensuremath{\mathbf x}_t$. Observe $f_t$, update
$\ensuremath{\mathbf y_{t+1}}(i) = \ensuremath{\mathbf y_{t}}(i) e^{- \eta\, \nabla_{t}(i)}$
for all $i \in [n]$. Project:
$\ensuremath{\mathbf x_{t+1}} = \frac{\ensuremath{\mathbf y_{t+1}}}{\| \ensuremath{\mathbf y_{t+1}} \|_1 }$
:::
:::

The regret achieved by the exponentiated gradient algorithm can be
bounded using the following corollary of Theorem
[5.2](#thm:RFTLmain1){reference-type="ref" reference="thm:RFTLmain1"}:

::: {#cor:eg .corollary}
**Corollary 5.7**. *The exponentiated gradient algorithm with gradients
bounded by $\|\nabla_t\|_\infty \leq G_\infty$ and parameter
$\eta = \sqrt{ \frac{\log n}{  2 T G_\infty^2 }}$ has regret bounded by
$$\ensuremath{\mathrm{{Regret}}}_T  \leq 2 G_\infty \sqrt{2  T \log n}.$$*
:::

## Randomized Regularization {#sec:randomized-regularization}

The connection between stability in decision making and low regret has
motivated our discussion of regularization thus far. However, this
stability need not be achieved only using strongly convex regularization
functions. An alternative method to achieve stability in decisions is by
introducing randomization into the algorithm. In fact, historically,
this method preceded methods based on strongly convex regularization
(see bibliography).

In this section we first describe a deterministic algorithm for online
convex optimization that is easily amenable to speedup via
randomization. We then give an efficient randomized algorithm for the
special case of OCO with linear losses.

##### Oblivious vs. adaptive adversaries. {#oblivious-vs.-adaptive-adversaries. .unnumbered}

For simplicity, we consider ourselves in this section with a slightly
restricted version of OCO. So far, we have not restricted the cost
functions in any way, and they could depend on the choice of decision by
the online learner. However, when dealing with randomized algorithms,
this issue becomes a bit more subtle: can the cost functions depend on
the randomness of the decision making algorithm itself? Furthermore,
when analyzing the regret, which is now a random variable, dependencies
across different iterations require probabilistic machinery which adds
little to the fundamental understanding of randomized OCO algorithms. To
avoid these complications, we make the following assumption throughout
this section: the cost functions $\{\ensuremath{\mathbf f_{t}}\}$ are
adversarially chosen ahead of time, and do not depend on the actual
decisions of the online learner. This version of OCO is called the
*oblivious* setting, to distinguish it from the *adaptive* setting.

### Perturbation for convex losses

The prediction in Algorithm [\[alg:FPL\]](#alg:FPL){reference-type="ref"
reference="alg:FPL"} is according to a version of the follow-the-leader
algorithm, augmented with an additional component of randomization. It
is a deterministic algorithm that computes the expected decision
according to a random variable. The random variable is the minimizer
over the decision set according to the sum of gradients of the cost
functions and an additional random vector.

In practice, the expectation need not be computed exactly. Estimation
(via random sampling) up to a precision that depends linearly on the
number of iterations would suffice.

The algorithm accepts as input a distribution, with the probability
density function (PDF) denoted ${\mathcal D}$, over vectors in
$n$-dimensional Euclidean space
$\ensuremath{\mathbf n}\in {\mathbb R}^n$. For
$\sigma, L \in {\mathbb R}$, we say that a distribution ${\mathcal D}$
is $(\sigma,L)=(\sigma_a,L_a)$ stable with respect to the norm
$\| \cdot \|_a$ if
$$\mathop{\mbox{\bf E}}_{\ensuremath{\mathbf n}\sim {\mathcal D}}  [ \|\ensuremath{\mathbf n}\|_a^* ] = \sigma_a  ,$$
and
$$\forall \ensuremath{\mathbf u},  \  \int_{\ensuremath{\mathbf n}} \left| {\mathcal D}(\ensuremath{\mathbf n}) - {\mathcal D}(\ensuremath{\mathbf n}- \ensuremath{\mathbf u}) \right| d \ensuremath{\mathbf n}\leq L_a \| \ensuremath{\mathbf u}\|_a^* .$$
Here $\ensuremath{\mathbf n}\sim {\mathcal D}$ denotes a vector
$\ensuremath{\mathbf n}\in {\mathbb R}^n$ sampled according to
distribution ${\mathcal D}$, and ${\mathcal D}(\ensuremath{\mathbf n})$
is the value of the probability density function ${\mathcal D}$ over
$\ensuremath{\mathbf n}$. The subscript $a$ is omitted if clear from the
context.

The first parameter, $\sigma$, is related to the variance of
${\mathcal D}$, while the second, $L$, is a measure of the sensitivity
of the distribution. For example, if ${\mathcal D}$ is the uniform
distribution over the hypercube $[0,1]^n$, then it holds that (see
exercises) for the Euclidean norm
$$\sigma_2 \leq \sqrt{n} \ ,\ L_2 \leq  1.$$ Reusing notation from
previous chapters, denote by $D= D_a$ the diameter of the set
$\ensuremath{\mathcal K}$ according to the norm $\| \cdot \|_a$, and by
$D^* = D_a^*$ the diameter according to its dual norm. Similarly, denote
by $G = G_a$ and $G^* = G_a^*$ an upper bound on the norm (and dual
norm) of the gradients.

::: algorithm
::: algorithmic
Input: $\eta > 0$, distribution ${\mathcal D}$ over ${\mathbb R}^n$,
decision set $\ensuremath{\mathcal K}\subseteq {\mathbb R}^n$. Let
$\ensuremath{\mathbf x}_1 = \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf n}\sim {\mathcal D}} \left[  \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}}
		\left\{  \ensuremath{\mathbf n}^\top \ensuremath{\mathbf x}\right\} \right]$.
Predict $\ensuremath{\mathbf x_{t}}$. Observe the loss function $f_t$,
suffer loss $f_t(\ensuremath{\mathbf x_{t}})$ and let
$\nabla_t = \nabla f_t(\ensuremath{\mathbf x}_t)$. Update
$$\begin{aligned}
 \label{eqn:fpl-oco}
		\ensuremath{\mathbf x_{t+1}} = \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf n}\sim {\mathcal D}} \left[  \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}}
		\left\{  \eta\sum_{s=1}^{t} \nabla_s^\top \ensuremath{\mathbf x}+
			\ensuremath{\mathbf n}^\top \ensuremath{\mathbf x}\right\} \right]

\end{aligned}$$
:::
:::

::: {#thm:fpl .theorem}
**Theorem 5.8**. *Let the distribution ${\mathcal D}$ be
$(\sigma,L)$-stable with respect to norm $\|\cdot \|_a$. The FPL
algorithm attains the following bound on the regret:
$$\ensuremath{\mathrm{{Regret}}}_T \le\eta D G^{* 2} L T+ \frac{1}{\eta} \sigma D .$$*
:::

We can further optimize over the choice of $\eta$ to obtain
$$\ensuremath{\mathrm{{Regret}}}_T \leq 2 L D G^* \sqrt{ \sigma  T }.$$

::: proof
*Proof.* Define the random variable
$\ensuremath{\mathbf x}_t^\ensuremath{\mathbf n}=  \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}}
\left\{  \eta\sum_{s=1}^{t} \nabla_s^\top \ensuremath{\mathbf x}+
\ensuremath{\mathbf n}^\top \ensuremath{\mathbf x}\right\}$, and the
random function $g_0^\ensuremath{\mathbf n}$ as
$$g_0^\ensuremath{\mathbf n}(\mathbf{x}) \stackrel{\text{\tiny def}}{=}\frac{1}{\eta} \ensuremath{\mathbf n}^\top \mathbf{x}.$$

It follows from Lemma [5.4](#prop:ftl-btl){reference-type="ref"
reference="prop:ftl-btl"} applied to the functions
$\{g_t(\ensuremath{\mathbf x}) = \nabla_t^\top \ensuremath{\mathbf x}\}$
that $$\begin{aligned}
\mathop{\mbox{\bf E}}\left[ \sum_{t=0}^T g_t (\ensuremath{\mathbf u}) \right] & \geq \mathop{\mbox{\bf E}}\left[ g_0^\ensuremath{\mathbf n}(\ensuremath{\mathbf x}_1^\ensuremath{\mathbf n}) + \sum_{t=1}^T g_t(\ensuremath{\mathbf x}_{t+1}^\ensuremath{\mathbf n}) \right] \\
& \geq  \mathop{\mbox{\bf E}}\left[ g_0^\ensuremath{\mathbf n}(\ensuremath{\mathbf x}_1^\ensuremath{\mathbf n}) \right] + \sum_{t=1}^T g_t(\mathop{\mbox{\bf E}}[ \ensuremath{\mathbf x}_{t+1}^\ensuremath{\mathbf n}]  ) & \mbox{convexity} \\
& =  \mathop{\mbox{\bf E}}\left[ g_0^\ensuremath{\mathbf n}(\ensuremath{\mathbf x}_1^\ensuremath{\mathbf n}) \right] + \sum_{t=1}^T g_t(\ensuremath{\mathbf x}_{t+1}  )
\end{aligned}$$ and thus, $$\begin{aligned}
 \label{eqn:ftl-shalom1}
& \sum_{t=1}^T \nabla_t(\ensuremath{\mathbf x_{t}}- \mathbf{x}^\star )  \\
& =  \sum_{t=1}^T g_t (\ensuremath{\mathbf x}_{t}) - \sum_{t=1}^T g_t(\ensuremath{\mathbf x}^\star) \\
& \leq \sum_{t=1}^T g_t (\ensuremath{\mathbf x}_{t}) - \sum_{t=1}^T g_t(\ensuremath{\mathbf x}_{t+1}) + \mathop{\mbox{\bf E}}[ g_0^\ensuremath{\mathbf n}(\ensuremath{\mathbf x}^\star) - g_0^\ensuremath{\mathbf n}(\ensuremath{\mathbf x}_1^\ensuremath{\mathbf n}) ] \\
& \leq \sum_{t=1}^T \nabla_t(\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}} ) + \frac{1}{\eta} \mathop{\mbox{\bf E}}[ \| \ensuremath{\mathbf n}\|^* \| \ensuremath{\mathbf x}^\star - \ensuremath{\mathbf x}_1^\ensuremath{\mathbf n}\| ] & \mbox { Cauchy-Schwarz } \\
& \leq \sum_{t=1}^T \nabla_t(\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}} ) + \frac{1}{\eta}\sigma D .
\end{aligned}$$ Hence, $$\begin{aligned}
 \label{eqn:ftl-shalom-main}
	&  \sum_{t=1}^T f_t(\ensuremath{\mathbf x}_t) - \sum_{t=1}^T f_t(\ensuremath{\mathbf x}^\star)  \notag \\
	 & \leq   \sum_{t=1}^T \nabla_t^\top (\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{}}^*)  \notag \\
	& \leq   \sum_{t=1}^T \nabla_t^\top (\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}}) + \frac{1}{\eta} \sigma D & \mbox{above}  \notag \\
	& \leq  G^* \sum_{t=1}^T \|\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}} \| + \frac{1}{\eta} \sigma D . & \mbox{ Cauchy-Schwarz}
\end{aligned}$$ We now argue that
$\|\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}}\|    = O(\eta)$.
Let
$$h_t(\ensuremath{\mathbf n}) =   \arg \min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \left\{ \eta \sum_{s=1}^{t-1} \nabla_s^\top \ensuremath{\mathbf x}+ \ensuremath{\mathbf n}^\top \ensuremath{\mathbf x}\right\}  ,$$
and hence
$\ensuremath{\mathbf x}_t  = \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf n}\sim {\mathcal D}} [h_t(\ensuremath{\mathbf n})]$.
Recalling that ${\mathcal D}(\ensuremath{\mathbf n})$ denotes the value
of the probability density function ${\mathcal D}$ over
$\ensuremath{\mathbf n}\in {\mathbb R}^n$, we can write:
$$\ensuremath{\mathbf x_{t}}=  \int\limits_{\ensuremath{\mathbf n}\in {\mathbb R}^n } h_t(\ensuremath{\mathbf n}) {\mathcal D}(\ensuremath{\mathbf n}) d \ensuremath{\mathbf n},$$
and:
$$\ensuremath{\mathbf x_{t+1}}  = \int\limits_{\ensuremath{\mathbf n}\in {\mathbb R}^n } h_t(\ensuremath{\mathbf n}+ \eta \nabla_t) {\mathcal D}(\ensuremath{\mathbf n}) d \ensuremath{\mathbf n}= \int\limits_{\ensuremath{\mathbf n}\in {\mathbb R}^n } h_t(\ensuremath{\mathbf n}) {\mathcal D}(\ensuremath{\mathbf n}- \eta \nabla_t ) d \ensuremath{\mathbf n}.$$
Notice that $\ensuremath{\mathbf x_{t}},\ensuremath{\mathbf x_{t+1}}$
may depend on each other. However, by linearity of expectation, we have
that $$\begin{aligned}
&  \| \ensuremath{\mathbf x_{t}}-  \ensuremath{\mathbf x_{t+1}}\|    \\
 & = \left\| \int\limits_{\ensuremath{\mathbf n}\in {\mathbb R}^n } ( h_t(\ensuremath{\mathbf n}) - h_t (\ensuremath{\mathbf n}+ \eta \nabla_t ) ) {\mathcal D}(\ensuremath{\mathbf n}) d \ensuremath{\mathbf n}\right\|  \\
& = \left\| \int\limits_{\ensuremath{\mathbf n}\in {\mathbb R}^n }  h_t(\ensuremath{\mathbf n}) ({\mathcal D}( \ensuremath{\mathbf n}) - {\mathcal D}( \ensuremath{\mathbf n}- \eta \nabla_t))  d \ensuremath{\mathbf n}\right\| \\
& = \left\| \int\limits_{\ensuremath{\mathbf n}\in {\mathbb R}^n }  (h_t(\ensuremath{\mathbf n}) - h_t(\mathbf{0}) )  ({\mathcal D}( \ensuremath{\mathbf n}) - {\mathcal D}( \ensuremath{\mathbf n}- \eta \nabla_t))  d \ensuremath{\mathbf n}\right\| \\
& \leq   \int\limits_{\ensuremath{\mathbf n}\in {\mathbb R}^n }  \|h_t(\ensuremath{\mathbf n}) - h_t(\mathbf{0}) \| |{\mathcal D}( \ensuremath{\mathbf n}) - {\mathcal D}( \ensuremath{\mathbf n}- \eta \nabla_t) |  d \ensuremath{\mathbf n}\\
& \leq  D  \int\limits_{\ensuremath{\mathbf n}\in {\mathbb R}^n }  \left|  {\mathcal D}(\ensuremath{\mathbf n}) - {\mathcal D}( \ensuremath{\mathbf n}- \eta \nabla_t) \right|  d \ensuremath{\mathbf n}\mbox{\ \ \ since } \|\ensuremath{\mathbf x}_t - h_t(\mathbf{0})\| \leq  D  \\
& \leq D L \cdot  \eta \|\nabla_t\|^*  \leq \eta D L G^*  .  \mbox{\ \ since ${\mathcal D}$ is $(\sigma,L)$-stable}.
\end{aligned}$$ Substituting this bound back into
[\[eqn:ftl-shalom-main\]](#eqn:ftl-shalom-main){reference-type="eqref"
reference="eqn:ftl-shalom-main"} we have $$\begin{aligned}
&  \sum\limits_{t=1}^T f_t(\ensuremath{\mathbf x}_t) - \sum\limits_{t=1}^T f_t(\ensuremath{\mathbf x}^\star)   \leq \eta  L D G^{* 2}  T   + \frac{1}{\eta}  \sigma D.
\end{aligned}$$ ◻
:::

For the choice of ${\mathcal D}$ as the uniform distribution over the
unit hypercube $[0,1]^n$, which has parameters $\sigma_2 \leq \sqrt{n}$
and $L_2 \leq 1$ for the Euclidean norm, the optimal choice of $\eta$
gives a regret bound of $DG n^{1/4}  \sqrt{ T}$. This is a factor
${n}^{1/4}$ worse than the online gradient descent regret bound of
Theorem [3.1](#thm:gradient){reference-type="ref"
reference="thm:gradient"}. For certain decision sets
$\ensuremath{\mathcal K}$ a better choice of distribution ${\mathcal D}$
results in near-optimal regret bounds.

### Perturbation for linear cost functions

The case of linear cost functions
$f_t(\ensuremath{\mathbf x}) = \ensuremath{\mathbf g_{t}}^\top \ensuremath{\mathbf x}$
is of particular interest in the context of randomized regularization.
Denote
$$w_t(\ensuremath{\mathbf n}) = \arg\min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \left\{  \eta\sum_{s=1}^{t} \ensuremath{\mathbf g_{s}}^\top \ensuremath{\mathbf x}+ \ensuremath{\mathbf n}^\top \ensuremath{\mathbf x}\right\} .$$

By linearity of expectation, we have that
$$f_t(\ensuremath{\mathbf x}_t) = f_t( \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf n}\sim {\mathcal D}} [w_t(\ensuremath{\mathbf n}) ] ) = \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf n}\sim {\mathcal D}} [ f_t(w_t(\ensuremath{\mathbf n})) ].$$
Thus, instead of computing $\ensuremath{\mathbf x}_t$ precisely, we can
sample a single vector $\ensuremath{\mathbf n}_0 \sim {\mathcal D}$, and
use it to compute $\hat{\mathbf{x}}_t = w_t(\ensuremath{\mathbf n}_0)$,
as illustrated in Algorithm
[\[alg:FPL-linear\]](#alg:FPL-linear){reference-type="ref"
reference="alg:FPL-linear"}.

::: algorithm
::: algorithmic
Input: $\eta > 0$, distribution ${\mathcal D}$ over ${\mathbb R}^n$,
decision set $\ensuremath{\mathcal K}\subseteq {\mathbb R}^n$. Sample
$\ensuremath{\mathbf n}_0 \sim {\mathcal D}$. Let
$\hat{\mathbf{x}}_1 \in \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}}  \{ -\ensuremath{\mathbf n}_0^\top \ensuremath{\mathbf x}\}$.
Predict $\hat{\mathbf{x}}_t$. Observe the linear loss function, suffer
loss $\ensuremath{\mathbf g_{t}}^\top\ensuremath{\mathbf x_{t}}$. Update
$$\begin{aligned}
		\hat{\mathbf{x}}_t =   \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}}
		\left\{  \eta\sum_{s=1}^{t-1} \ensuremath{\mathbf g_{s}}^\top \ensuremath{\mathbf x}+
			\ensuremath{\mathbf n}_0 ^\top \ensuremath{\mathbf x}\right\}

\end{aligned}$$
:::
:::

By the above arguments, we have that the expected regret for the random
variables $\hat{\mathbf{x}}_t$ is the same as that for
$\ensuremath{\mathbf x}_t$. We obtain the following Corollary:

::: {#cor:fpl-linear .corollary}
**Corollary 5.9**.
*$$\mathop{\mbox{\bf E}}_{\ensuremath{\mathbf n}_0 \sim {\mathcal D}}  \left[ \sum_{t=1}^T f_t(\hat{\mathbf{x}}_t) - \sum_{t=1}^T f_t(\ensuremath{\mathbf x}^\star) \right] \leq  \eta  L D G^{*  2}  T   + \frac{1}{\eta}  \sigma D  .$$*
:::

The main advantage of this algorithm is computational: with a single
linear optimization step over the decision set $\ensuremath{\mathcal K}$
(which does not even have to be convex!), we attain near optimal
expected regret bounds.

### Follow-the-perturbed-leader for expert advice

An interesting special case (and in fact the first use of perturbation
in decision making) is that of non-negative linear cost functions over
the unit $n$-dimensional simplex with costs bounded by one, or the
problem of prediction of expert advice we have considered in chapter
[1](#chap:intro){reference-type="ref" reference="chap:intro"}.

Algorithm [\[alg:FPL-linear\]](#alg:FPL-linear){reference-type="ref"
reference="alg:FPL-linear"} applied to the probability simplex and with
exponentially distributed noise is known as the
follow-the-perturbed-leader for prediction from expert advice method. We
spell it out in Algorithm
[\[alg:FPL\*\]](#alg:FPL*){reference-type="ref" reference="alg:FPL*"}.

::: algorithm
::: algorithmic
Input: $\eta > 0$ Draw $n$ exponentially distributed variables
$\ensuremath{\mathbf n}(i) \sim  e^{- \eta x}$. Let
$\ensuremath{\mathbf x_{1}} = \mathop{\mathrm{\arg\min}}_{\mathbf{e}_i \in \Delta_n}   \{ -\mathbf{e}_i^\top \ensuremath{\mathbf n}\}$.
Predict using expert $i_t$ such that
$\hat{\mathbf{x}}_t = \mathbf{e}_{i_t}$ Observe the loss vector and
suffer loss
$\ensuremath{\mathbf g_{t}}^\top \hat{\mathbf{x}}_t = \ensuremath{\mathbf g_{t}}(i_t)$
Update (w.l.o.g. choose $\hat{\mathbf{x}}_{t+1}$ to be a vertex)
$$\begin{aligned}
		\hat{\mathbf{x}}_{t+1} =   \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \Delta_n}
		\left\{  \sum_{s=1}^{t} \ensuremath{\mathbf g_{s}}^\top \ensuremath{\mathbf x}-
			\ensuremath{\mathbf n}^\top \ensuremath{\mathbf x}\right\}

\end{aligned}$$
:::
:::

Notice that we take the perturbation to be distributed according to the
one-sided negative exponential distribution, i.e.,
$\ensuremath{\mathbf n}(i) \sim 	e^{-\eta x}$, or more precisely
$$\Pr[  \ensuremath{\mathbf n}(i) \leq x ] = 1 -  e^{-\eta x} \quad \forall x  \geq 0 .$$

Corollary [5.9](#cor:fpl-linear){reference-type="ref"
reference="cor:fpl-linear"} gives regret bounds that are suboptimal for
this special case, thus we give here an alternative analysis that gives
tight bounds up to constants amounting to the following theorem.

::: {#thm:fpl-experts .theorem}
**Theorem 5.10**. *Algorithm
[\[alg:FPL\*\]](#alg:FPL*){reference-type="ref" reference="alg:FPL*"}
outputs a sequence of predictions
$\hat{\mathbf{x}}_1,...,\hat{\mathbf{x}}_T \in \Delta_n$ such that:
$$(1 - \eta) \mathop{\mbox{\bf E}}\left[ \sum_t  \ensuremath{\mathbf g_{t}}^\top \hat{\mathbf{x}}_t \right] \leq  \min_{\ensuremath{\mathbf x}^\star\in \Delta_n} \sum_t \ensuremath{\mathbf g_{t}}^\top \ensuremath{\mathbf x}^\star  + \frac{4 \log n}{\eta }  .$$*
:::

Notice that as a special case of the above theorem, choosing
$\eta = \sqrt{\frac{\log n}{T}}$ yields a regret bound of
$$\ensuremath{\mathrm{{Regret}}}_T = O ( \sqrt{  T \log n }),$$ which is
equivalent up to constant factors to the guarantee given for the Hedge
algorithm in Theorem [1.5](#lem:hedge){reference-type="ref"
reference="lem:hedge"}.

::: proof
*Proof.* We start with the same analysis technique throughout this
chapter: let $\ensuremath{\mathbf g_{0}} = -\ensuremath{\mathbf n}$. It
follows from Lemma [5.4](#prop:ftl-btl){reference-type="ref"
reference="prop:ftl-btl"} applied to the functions
$\{f_t(\ensuremath{\mathbf x}) = \ensuremath{\mathbf g_{t}}^\top \ensuremath{\mathbf x}\}$
that
$$\mathop{\mbox{\bf E}}\left[ \sum_{t=0}^T \ensuremath{\mathbf g_{t}}^\top \ensuremath{\mathbf u}\right] \geq \mathop{\mbox{\bf E}}\left[ \sum_{t=0}^T \ensuremath{\mathbf g_{t}}^\top \hat{\mathbf{x}}_{t+1} \right]  ,$$
and thus, $$\begin{aligned}
 \label{eqn:ftl-shalom3}
\mathop{\mbox{\bf E}}\left[ \sum_{t=1}^T \ensuremath{\mathbf g_{t}}^\top (\hat{\mathbf{x}}_t - \ensuremath{\mathbf x}^\star ) \right]
& \leq \mathop{\mbox{\bf E}}\left[ \sum_{t=1}^T \ensuremath{\mathbf g_{t}}^\top (\hat{\mathbf{x}}_{t}  - \hat{\mathbf{x}}_{t+1}) \right] + \mathop{\mbox{\bf E}}[ \ensuremath{\mathbf g_{0}}^\top (\ensuremath{\mathbf x}^\star - \ensuremath{\mathbf x}_1) ] \notag \\
& \leq \mathop{\mbox{\bf E}}\left[ \sum_{t=1}^T \ensuremath{\mathbf g_{t}}^\top (\hat{\mathbf{x}}_t - \hat{\mathbf{x}}_{t+1} ) \right] +  \mathop{\mbox{\bf E}}[ \| \ensuremath{\mathbf n}\|_\infty \| \ensuremath{\mathbf x}^\star - \ensuremath{\mathbf x}_1 \|_1 ]  \notag \\ % & \mbox { Cauchy-Schwarz} \notag \\
& \leq   \sum_{t=1}^T \mathop{\mbox{\bf E}}\left[ \ensuremath{\mathbf g_{t}}^\top (\hat{\mathbf{x}}_t - \hat{\mathbf{x}}_{t+1} ) \ | \ \hat{\mathbf{x}}_t \right] + \frac{4}{\eta} \log n ,
\end{aligned}$$ where the second inequality follows by the generalized
Cauchy-Schwarz inequality, and the last inequality follows since (see
exercises)
$$\mathop{\mbox{\bf E}}_{\ensuremath{\mathbf n}\sim {\mathcal D}}  [ \|\ensuremath{\mathbf n}\|_\infty ] \leq \frac{ 2 \log n}{\eta} .$$

We proceed to bound
$\mathop{\mbox{\bf E}}[ \ensuremath{\mathbf g_{t}}^\top (\hat{\mathbf{x}}_t - \hat{\mathbf{x}}_{t+1} ) | \hat{\mathbf{x}}_t ]$,
which is naturally bounded by the probability that
$\hat{\mathbf{x}}_{t}$ is not equal to $\hat{\mathbf{x}}_{t+1}$
multiplied by the maximum value that $\ensuremath{\mathbf g_{t}}$ can
attain (i.e., its $\ell_\infty$ norm):
$$\mathop{\mbox{\bf E}}[ \ensuremath{\mathbf g_{t}}^\top (\hat{\mathbf{x}}_t - \hat{\mathbf{x}}_{t+1} ) \ | \ \hat{\mathbf{x}}_t ] \leq \|\ensuremath{\mathbf g_{t}}\|_\infty \cdot \Pr[ \hat{\mathbf{x}}_t \neq \hat{\mathbf{x}}_{t+1} \ |\  \hat{\mathbf{x}}_t ]  \leq \Pr[ \hat{\mathbf{x}}_t \neq \hat{\mathbf{x}}_{t+1} \ |\  \hat{\mathbf{x}}_t ]  .$$
Above we have that $\|\ensuremath{\mathbf g_{t}}\|_\infty \leq 1$ by
assumption that the losses are bounded by one.

To bound the latter, notice that the probability
$\hat{\mathbf{x}}_t = \mathbf{e}_{i_t}$ is the leader at time $t$ is the
probability that $- \ensuremath{\mathbf n}({i_t}) > v$ for some value
$v$ that depends on the entire loss sequence till now. On the other
hand, given $\hat{\mathbf{x}}_t$, we have that
$\hat{\mathbf{x}}_{t+1} = \hat{\mathbf{x}}_t$ remains the leader if
$- \ensuremath{\mathbf n}(i_t) > v + \ensuremath{\mathbf g_{t}}(i_t)$,
since it was a leader by a margin of more than the cost it will suffer.
Thus, $$\begin{aligned}
\Pr[ \hat{\mathbf{x}}_t \neq \hat{\mathbf{x}}_{t+1} \ |\  \hat{\mathbf{x}}_t ] & = 1 - \Pr[- \ensuremath{\mathbf n}({i_t}) > v+ \ensuremath{\mathbf g_{t}}(i_t) \ |\  -\ensuremath{\mathbf n}({i_t}) > v ] \\
& = 1 - \frac{ \int_{v + \ensuremath{\mathbf g_{t}}(i_t) }^\infty \eta e^{-\eta x } } {\int _{v}^\infty \eta e^{-\eta x}} \\
& = 1 - e^{ - \eta \ensuremath{\mathbf g_{t}}(i_t) } \\
& \leq \eta \ensuremath{\mathbf g_{t}}(i_t)  = \eta \ensuremath{\mathbf g_{t}}^\top \hat{\mathbf{x}}_t .
\end{aligned}$$ Substituting this bound back into
[\[eqn:ftl-shalom3\]](#eqn:ftl-shalom3){reference-type="eqref"
reference="eqn:ftl-shalom3"} we have $$\begin{aligned}
&  \mathop{\mbox{\bf E}}[ \sum_{t=1}^T \ensuremath{\mathbf g_{t}}^\top (\hat{\mathbf{x}}_t - \ensuremath{\mathbf x}^\star ) ]  \leq \eta  \sum_t \mathop{\mbox{\bf E}}_t[ \ensuremath{\mathbf g_{t}}^\top \hat{\mathbf{x}}_t] + \frac{4 \log n}{\eta} ,
\end{aligned}$$ which simplifies to the Theorem. ◻
:::

## \* Adaptive Gradient Descent {#sec:adagrad}

Thus far we have introduced regularization as a general methodology for
deriving online convex optimization algorithms. The main theorem of this
chapter, Theorem [5.2](#thm:RFTLmain1){reference-type="ref"
reference="thm:RFTLmain1"}, bounds the regret of the RFTL algorithm for
any strongly convex regularizer as $$\label{eqn:general-regret-form}
 \ensuremath{\mathrm{{Regret}}}_T \leq  \max_{\ensuremath{\mathbf u}\in \ensuremath{\mathcal K}} \sqrt{ 2 \sum_t \|\nabla_t \|_t^{* 2}  B_{R}( \ensuremath{\mathbf u}||\ensuremath{\mathbf x}_1) }.$$
In addition, we have seen how to derive the online gradient descent and
the multiplicative weights algorithms as special cases of the RFTL
methodology. But are there other special cases of interest, besides
these two basic algorithms, that warrant such general and abstract
treatment?

There are surprisingly few cases of interest besides the Euclidean and
Entropic regularizations and their matrix analogues. However, in this
chapter we will give some justification of the abstract treatment of
regularization.

Our treatment is motivated by the following question: thus far we have
thought of $R$ as a strongly convex function. But which strongly convex
function should we choose to minimize regret? This is a deep and
difficult question which has been considered in the optimization
literature since its early developments. Naturally, the optimal
regularization should depend on both the convex underlying decision set,
as well as the actual cost functions (see exercises for a natural
candidate of a regularization function that depends on the convex
decision set).

We shall treat this question no differently than we treat other
optimization problems throughout this manuscript itself: we'll learn the
optimal regularization online! That is, a regularizer that adapts to the
sequence of cost functions and is in a sense the "optimal"
regularization to use in hindsight. This gives rise to the AdaGrad
(Adaptive subGradient method) algorithm
[\[alg:adagrad\]](#alg:adagrad){reference-type="ref"
reference="alg:adagrad"}, which explicitely optimizes over the
regularization choice in line
[\[eqn:adagrad1\]](#eqn:adagrad1){reference-type="eqref"
reference="eqn:adagrad1"} to minimize the gradient norms, which is the
dominant expression in
[\[eqn:general-regret-form\]](#eqn:general-regret-form){reference-type="eqref"
reference="eqn:general-regret-form"}.

::: algorithm
::: algorithmic
Input: parameters
$\eta, \ensuremath{\mathbf x}_1 \in \ensuremath{\mathcal K}$.
Initialize: $G_0 = \mathbf{0}$, Predict $\ensuremath{\mathbf x}_t$,
suffer loss $f_t(\ensuremath{\mathbf x}_t)$. []{#eqn:adagrad1
label="eqn:adagrad1"} Update $G_t = G_{t-1} + \nabla_t \nabla_t^\top$
and define $$\begin{aligned}
			&\text{[Diagonal version]}
			& H_t =  \mathop{\mathrm{\arg\min}}_{H \succeq 0,H={\bf diag}(H)} \left\{ G_t \bullet H^{-1} + {\bf Tr}(H) \right\} = {\bf diag}({G_t}^{1/2}) \\
			&\text{[Full matrix version]} 	& H_t =  \mathop{\mathrm{\arg\min}}_{H \succeq \mathbf{0}} \left\{ G_t \bullet H^{-1} + {\bf Tr}(H) \right\} =  {G_t}^{1/2}

\end{aligned}$$ Update
$$\ensuremath{\mathbf y_{t+1}} = \ensuremath{\mathbf x_{t}}- \eta H_t^{-1} \nabla_t$$
$$\ensuremath{\mathbf x_{t+1}} = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \| \ensuremath{\mathbf y_{t+1}}  - \ensuremath{\mathbf x}\|^{2}_{H_t}$$
:::
:::

AdaGrad comes in two versions: diagonal and full matrix, the first being
particularly efficient to implement with negligible computational
overhead over online gradient descent. In the algorithm definition and
throughout this chapter, the notation $A^{-1}$ refers to the
Moore-Penrose pseudoinverse of the matrix $A$.

The computation in line
[\[eqn:adagrad1\]](#eqn:adagrad1){reference-type="eqref"
reference="eqn:adagrad1"} finds the regularization matrix $H$ which
minimizes the norm of the gradients from within the positive
semi-definite cone, with or without a diagonal constraint. This is
closely related, as we shall see, to optimization w.r.t. two natural
sets of matrices:

1.  ${\mathcal H}_1 = \{ H = {\bf diag}(H) , H \succeq 0 \ , \ {\bf Tr}(H) \leq 1 \}$

2.  ${\mathcal H}_2 = \{ H \succeq 0 \ , \ {\bf Tr}(H) \leq 1 \}$.

This results in a regularization matrix that is provably optimal in the
following sense,

::: {#lem:regularzation-optimality-adagrad .lemma}
**Lemma 5.11**. *For
${\mathcal H}_i \in \{{\mathcal H}_1,{\mathcal H}_2\}$ with the
corresponding $H_T$, $$\begin{aligned}
\sqrt{  \min_{H \in {\mathcal H}_i} \sum_{t=1}^T \|\nabla_t \|_H^{* 2} }  & =  {\bf Tr}(H_T)  .
\end{aligned}$$*
:::

Using this lemma, we show the regret of AdaGrad is at most a constant
factor larger than the minimum regret of all RFTL algorithm with
regularization functions whose Hessian is fixed and belongs to the class
${\mathcal H}_i$. Furthermore, the regret of the diagonal version can be
a factor $\sqrt{d}$ smaller than that of online gradient descent for
certain gradient geometries. The regret bound on AdaGrad is formally
stated in the following theorem.

::: {#theorem:adagrad-main .theorem}
**Theorem 5.12**. *Let $\{\ensuremath{\mathbf x}_t\}$ be defined by
Algorithm [\[alg:adagrad\]](#alg:adagrad){reference-type="ref"
reference="alg:adagrad"} with parameters $\eta  = {D}$ (full matrix) or
$\eta = D_\infty$ (diagonal). Then for any
$\ensuremath{\mathbf x}^\star \in \ensuremath{\mathcal K}$,
$$\begin{aligned}
 \label{eqn:adagrad_regret}
& \ensuremath{\mathrm{{Regret}}}_{T}(\mbox{AdaGrad-diag}) \le \sqrt{2} D_\infty    \sqrt{  \min_{H \in {\mathcal H}_1} \sum_t \|\nabla_t \|_H^{* 2}  } , \\
& \ensuremath{\mathrm{{Regret}}}_{T}(\mbox{AdaGrad-full}) \le \sqrt{2} D    \sqrt{  \min_{H \in {\mathcal H}_2} \sum_t \|\nabla_t \|_H^{* 2}  } .
\end{aligned}$$*
:::

Before proceeding to the analysis, we consider when the regret bounds
for AdaGrad improve upon those of Online Gradient Descent. One such case
is when $\ensuremath{\mathcal K}$ is the unit cube in $d$-dimensional
Euclidean space. This convex set has $D_\infty =1$ and $D = \sqrt{d}$.
Lemma [5.11](#lem:regularzation-optimality-adagrad){reference-type="ref"
reference="lem:regularzation-optimality-adagrad"} and Theorems
[5.12](#theorem:adagrad-main){reference-type="ref"
reference="theorem:adagrad-main"},[5.2](#thm:RFTLmain1){reference-type="ref"
reference="thm:RFTLmain1"} imply that the regret of diagonal AdaGrad and
OGD are bounded by $$\begin{aligned}
& \ensuremath{\mathrm{{Regret}}}_{T}(\mbox{AdaGrad-diag}) \le \sqrt{2} {\bf Tr}({\bf diag}(G_T)^{1/2})  ,\\
& \ensuremath{\mathrm{{Regret}}}_{T}(\mbox{OGD}) \le \sqrt{2 d} \sqrt{ \sum_t \|\nabla_t\|^2} = \sqrt{2d {\bf Tr}({\bf diag}(G_T)) }  .
\end{aligned}$$ The relationship between the two terms depends on the
matrix ${\bf diag}(G_T)$. If this matrix is sparse, then AdaGrad has a
superior bound by at most $\sqrt{d}$ factor. For other convex bodies,
such as the Euclidean ball, and when the matrix $G_T$ is dense, the
regret of OGD can be a factor $\sqrt{d}$ lower.

### Analysis of adaptive regularization

We proceed with the proof of Theorem
[5.12](#theorem:adagrad-main){reference-type="ref"
reference="theorem:adagrad-main"}. The first component is the following
Lemma, which generalizes the RFTL analysis to changing regularization.

::: {#lem:adagradlem .lemma}
**Lemma 5.13**. *Let
$H_{0} = \mathop{\mathrm{\arg\min}}_{H \succeq 0} \left\{ {\bf Tr}(H) \right\} = 0$,
$$\ensuremath{\mathrm{{Regret}}}_T(\text{GenAdaReg}) \leq \frac{\eta}{2}  ( G_T \bullet H_T^{-1} + {\bf Tr}(H_T))
  +  \frac{1}{2 \eta} \sum_{t=0}^T  \| \mathbf{x}_t  -  \mathbf{x}^\star\|^2_{
H_t - H_{t-1}}  .$$*
:::

::: proof
*Proof.* By the definition of $\mathbf{y}_{t+1}$: $$\begin{aligned}
& \mathbf{y}_{t+1} - \mathbf{x}^\star = \mathbf{x}_{t} - \mathbf{x}^\star - \eta {H_t}^{-1}
\nabla_t \\
& H_t (\mathbf{y}_{t+1} - \mathbf{x}^\star) = H_t (\mathbf{x}_t - \mathbf{x}^\star) - \eta
\nabla_t.
\end{aligned}$$ Multiplying the transpose of the first equation by the
second we get $$\begin{gathered}
(\mathbf{y}_{t+1} - \mathbf{x}^\star)^\top H_t(\mathbf{y}_{t+1} - \mathbf{x}^\star) = \notag \\
(\mathbf{x}_t\! -\! \mathbf{x}^\star)^\top H_t(\mathbf{x}_t\! -\! \mathbf{x}^\star) -
2 \eta  \nabla_t^\top (\mathbf{x}_t\! -\! \mathbf{x}^\star) +
\eta^2 \nabla_t^\top H_t^{-1} \nabla_t.
\label{eq:multiplied-adagrad}
\end{gathered}$$ Since $\mathbf{x}_{t+1}$ is the projection of
$\mathbf{y}_{t+1}$ in the norm induced by $H_t$, we have (see
§[2.1.1](#sec:projections){reference-type="ref"
reference="sec:projections"}) $$\begin{aligned}
 (\mathbf{y}_{t+1} - \mathbf{x}^\star)^\top H_t(\mathbf{y}_{t+1} - \mathbf{x}^\star)  & = \| \mathbf{y}_{t+1} - \mathbf{x}^\star \|_{H_t}^2  \ge  \| \mathbf{x}_{t+1} - \mathbf{x}^\star \|_{H_t}^2   .
%&  = (\bx_{t+1} - \bx^\star)^\top G_t(\bx_{t+1} - \bx^\star ).
\end{aligned}$$ This inequality is the reason for using generalized
projections as opposed to standard projections, which were used in the
analysis of online gradient descent(see
§[3.1](#section:ogd){reference-type="ref" reference="section:ogd"}
Equation [\[eqn:ogdtriangle\]](#eqn:ogdtriangle){reference-type="eqref"
reference="eqn:ogdtriangle"}). This fact together with
[\[eq:multiplied-adagrad\]](#eq:multiplied-adagrad){reference-type="eqref"
reference="eq:multiplied-adagrad"} gives $$\begin{aligned}
\nabla_t^\top (\mathbf{x}_t \! -\! \mathbf{x}^\star) &\leq \ \frac{\eta}{2}
\nabla_t^\top H_t^{-1} \nabla_t  +  \frac{1}{2 \eta} \left( \| \mathbf{x}_{t} - \mathbf{x}^\star \|_{H_t}^2 - \| \mathbf{x}_{t+1} - \mathbf{x}^\star \|_{H_{t}}^2  \right) .
\end{aligned}$$ Now, summing up over $t=1$ to $T$ we get that
$$\begin{aligned}
  \label{eqn:adagrad-shalom}
&\sum_{t=1}^T \nabla_t^\top (\mathbf{x}_t - \mathbf{x}^\star)
 \leq \frac{\eta}{2} \sum_{t=1}^T \nabla_t^\top H_t^{-1} \nabla_t +
\frac{1}{2\eta}  \| \mathbf{x}_{1} - \mathbf{x}^\star \|_{H_{0}}^2  \\
& + \frac{1}{2 \eta} \sum_{t=1}^T  \left( \| \mathbf{x}_{t} - \mathbf{x}^\star \|_{H_t}^2 - \| \mathbf{x}_{t} - \mathbf{x}^\star \|_{H_{t-1}}^2 \right)   - \frac{1}{2 \eta} \| \mathbf{x}_{T+1}   - \mathbf{x}^\star \|_{H_{T}}^2  \notag \\
&\leq \frac{\eta}{2} \sum_{t=1}^T \nabla_t^\top H_t^{-1}
\nabla_t + \frac{1}{2\eta} \sum_{t=0}^{T} \| \mathbf{x}_t\! -\! \mathbf{x}^\star\|^2_{
H_t - H_{t-1}}  . \notag
\end{aligned}$$ In the last inequality we use the definition
$H_{0} = 0$. We proceed to bound the first term. To this end, define the
functions
$$\Psi_t(H) = \nabla_t \nabla_t^\top \bullet H^{-1} \ , \ \Psi_0(H) = {\bf Tr}(H) .$$
By definition, $H_t$ is the minimizer of $\sum_{i=0}^{t} \Psi_i$ over
${\mathcal H}$. Therefore, using the BTL Lemma
[5.4](#prop:ftl-btl){reference-type="ref" reference="prop:ftl-btl"}, we
have that $$\begin{aligned}
\sum_{t=1}^T \nabla_t^\top H_t^{-1} \nabla_t & = \sum_{t=1}^T \Psi_t(H_t) \\
& \leq \sum_{t=1}^T \Psi_t(H_T)  + \Psi_0(H_T) - \Psi_0(H_0) \\
& =  G_T \bullet H_T^{-1} + {\bf Tr}(H_T) . % = 2 \trace(H_T) ,
\end{aligned}$$ ◻
:::

We can now continue with the proof of Theorem
[5.12](#theorem:adagrad-main){reference-type="ref"
reference="theorem:adagrad-main"}.

::: proof
*Proof of Theorem [5.12](#theorem:adagrad-main){reference-type="ref"
reference="theorem:adagrad-main"}.* We bound both parts of Lemma
[5.13](#lem:adagradlem){reference-type="ref"
reference="lem:adagradlem"}, with the following two lemmas,

::: {#lemma:opt-distance-bound-adagrad .lemma}
**Lemma 5.14**. *For both the diagonal and full matrix versions of
AdaGrad, the following holds
$$G_T \bullet H_T^{-1} \leq {\bf Tr}(H_T) .$$*
:::

::: {#lemma:opt-reg-bound2-adagrad .lemma}
**Lemma 5.15**. *Let $D_\infty$ denote the $\ell_\infty$ diameter of
$\ensuremath{\mathcal K}$, and $D$ the Euclidean diameter. Then the
following bounds hold, $$\begin{aligned}
& \mbox{Diagonal AdaGrad: } & \sum_{t=1}^{T} \| \mathbf{x}_t  -  \mathbf{x}^\star\|_{H_t - H_{t-1}}^2  \leq D^2_\infty {\bf Tr}(H_T). \\
& \mbox{Full matrix AdaGrad: } &  \sum_{t=1}^{T} \| \mathbf{x}_t  -  \mathbf{x}^\star\|_{H_t - H_{t-1}}^2  \leq D^2 {\bf Tr}(H_T).
\end{aligned}$$*
:::

Now combining Lemma [5.13](#lem:adagradlem){reference-type="ref"
reference="lem:adagradlem"} with the above two lemmas, and using
$\eta = \frac{D}{\sqrt{2}}$ or $\eta = \frac{D_\infty}{\sqrt{2}}$
appropriately, we obtain the theorem. ◻
:::

We proceed to complete the proof of the two lemmas above.

::: proof
*Proof of Lemma
[5.14](#lemma:opt-distance-bound-adagrad){reference-type="ref"
reference="lemma:opt-distance-bound-adagrad"}.* The optimization problem
of choosing $H_t$ in line
[\[eqn:adagrad1\]](#eqn:adagrad1){reference-type="eqref"
reference="eqn:adagrad1"} of Algorithm
[\[alg:adagrad\]](#alg:adagrad){reference-type="ref"
reference="alg:adagrad"} has an explicit solution, given in the
following proposition (whose proof is left as an exercise).

::: {#proposition:solution-inv-trace .proposition}
**Proposition 5.16**. *Consider the following optimization problems, for
$A \succcurlyeq 0$: $$\begin{aligned}
\min_{X \succeq 0 , {\bf Tr}(X) \leq 1}  \left\{ X^{-1} \bullet A \right\}  \quad \quad \min_{X \succeq 0} \left\{ A \bullet X^{-1} + {\bf Tr}(X) \right\} .
\end{aligned}$$ Then the global optimizer to these problems is obtained
at $X = \frac{A^{1/2}} { {\bf Tr}(A^{1/2})}$ and $X = A^{1/2}$
respectively. Over the set of diagonal matrices, the global optimizer is
obtained at $X = \frac{{\bf diag}(A)^{1/2}} { {\bf Tr}(A^{1/2})}$ and
$X = {\bf diag}(A)^{1/2}$ respectively.*
:::

A direct corollary of this proposition gives Lemma
[5.11](#lem:regularzation-optimality-adagrad){reference-type="ref"
reference="lem:regularzation-optimality-adagrad"} as follows:

::: corollary
**Corollary 5.17**. *$$\begin{aligned}
\sqrt{  \min_{H \in {\mathcal H}} \sum_t \|\nabla_t \|_H^{* 2} }  & = \sqrt{ \min_{H \in {\mathcal H}} {\bf Tr}(  H^{-1} \sum_t \nabla_t \nabla_t^\top  ) }  \\
& =  {\bf Tr}{ \sqrt{ \sum_t \nabla_t \nabla_t^\top } } = {\bf Tr}(H_T)  % = \trace(\sqrt{S_T - \delta n}) \\
%& = \trace(G_T) - \delta n.
\end{aligned}$$*
:::

 ◻
:::

The remaining term from Lemma
[5.13](#lem:adagradlem){reference-type="ref" reference="lem:adagradlem"}
is the expression
$\sum_{t=0}^T  \| \mathbf{x}_t  -  \mathbf{x}^\star\|^2_{ H_t - H_{t-1}}$,
which we proceed to bound.

::: proof
*Proof of Lemma
[5.15](#lemma:opt-reg-bound2-adagrad){reference-type="ref"
reference="lemma:opt-reg-bound2-adagrad"}.* By definition
$G_t \succcurlyeq G_{t-1}$, and hence using proposition
[5.16](#proposition:solution-inv-trace){reference-type="ref"
reference="proposition:solution-inv-trace"} and the definition of $H_t$
in line [\[eqn:adagrad1\]](#eqn:adagrad1){reference-type="eqref"
reference="eqn:adagrad1"}, we have that
$H_t = {\bf diag}(G_t^{1/2} ) \succcurlyeq  {\bf diag}(G_{t-1}^{1/2} ) =  H_{t-1}$.
Since for a diagonal matrix $H$ it holds that
$\ensuremath{\mathbf x}^\top H \ensuremath{\mathbf x}\leq \|\ensuremath{\mathbf x}\|_\infty^2 {\bf Tr}(H)$,
we have $$\begin{aligned}
& \sum_{t=1}^{T} (\mathbf{x}_t\! -\! \mathbf{x}^\star)^\top (H_t - H_{t-1} ) (\mathbf{x}_t\! -\! \mathbf{x}^\star) \\
& \leq \sum_{t=1}^{T} D^2_\infty  {\bf Tr}( H_t - H_{t-1} ) & \mbox{diagonal structure, $H_t - H_{t-1} \succeq 0$}\\
%& \leq D^2_\infty \sum_{t=1}^{T}  \trace (H_t - H_{t-1})  & A \succcurlyeq 0 \ \Rightarrow \  \lambda_{\max}(A) \leq \trace(A) \\
& = D^2_\infty \sum_{t=1}^{T}  ({\bf Tr}(H_t ) - {\bf Tr}( H_{t-1}))  &  \mbox{ linearity of the trace} \\
& \leq D^2_\infty {\bf Tr}(H_T).
\end{aligned}$$

Next, we consider the full matrix case. By definition
$G_t \succcurlyeq G_{t-1}$, and hence $H_t \succcurlyeq H_{t-1}$. Thus,
$$\begin{aligned}
& \sum_{t=1}^{T} (\mathbf{x}_t\! -\! \mathbf{x}^\star)^\top (H_t - H_{t-1} ) (\mathbf{x}_t\! -\! \mathbf{x}^\star) \\
& \leq \sum_{t=1}^{T} D^2  \lambda_{\max}( H_t - H_{t-1} ) \\
& \leq D^2 \sum_{t=1}^{T}  {\bf Tr}(H_t - H_{t-1})  & A \succcurlyeq 0 \ \Rightarrow \  \lambda_{\max}(A) \leq {\bf Tr}(A) \\
& = D^2 \sum_{t=1}^{T}  ({\bf Tr}(H_t ) - {\bf Tr}( H_{t-1}))  &  \mbox{ linearity of the trace} \\
& \leq D^2 {\bf Tr}(H_T).
\end{aligned}$$ ◻
:::

## Bibliographic Remarks {#bibliographic-remarks-2}

Regularization in the context of online learning was first studied in
[@GroveLS01] and [@KivinenW01]. The influential paper of @KV-FTL coined
the term "follow-the-leader" and introduced many of the techniques that
followed in OCO. The latter paper studies random perturbation as a
regularization and analyzes the follow-the-perturbed-leader algorithm,
following an early development by @Hannan57 that was overlooked in
learning for many years.

In the context of OCO, the term follow-the-regularized-leader was coined
in [@ShwartzS07; @ShalevThesis], and at roughly the same time an
essentially identical algorithm was called "RFTL" in [@AbernethyHR08].
The equivalence of RFTL and Online Mirror Descent was observed by
[@DBLP:conf/colt/HazanK08]. The AdaGrad algorithm was introduced in
[@DuchiHS10; @duchi2011adaptive], its diagonal version was also
discovered in parallel in [@McMahanS10]. The analysis of AdaGrad
presented in this chapter is due to [@gupta2017unified].

Adaptive regularization has received significant attention due to its
success in training deep neural networks, and notably the development of
adaptive algorithms that incorporate momentum and other heuristics, most
popular of which are AdaGrad, RMSprop [@tieleman2012lecture] and Adam
[@kingma2014adam]. For a survey of optimization for deep learning, see
the comprehensive text of @Goodfellow-et-al-2016.

There is a strong connection between randomized perturbation and
deterministic regularization. For some special cases, adding
randomization can be thought of as a special case of deterministic
strongly convex regularization, see
[@abernethy2014online; @abernethy16perturbation].

## Exercises

# Bandit Convex Optimization {#chap:bandits}

In many real-world scenarios the feedback available to the decision
maker is noisy, partial or incomplete. Such is the case in online
routing in data networks, in which an online decision maker iteratively
chooses a path through a known network, and her loss is measured by the
length (in time) of the path chosen. In data networks, the decision
maker can measure the RTD (round trip delay) of a packet through the
network, but rarely has access to the congestion pattern of the entire
network.

Another useful example is that of online ad placement in web search. The
decision maker iteratively chooses an ordered set of ads from an
existing pool. Her reward is measured by the viewer's response---if the
user clicks a certain ad, a reward is generated according to the weight
assigned to the particular ad. In this scenario, the search engine can
inspect which ads were clicked through, but cannot know whether
different ads, had they been chosen to be displayed, would have been
clicked through or not.

The examples above can readily be modeled in the OCO framework, with the
underlying sets being the convex hull of decisions. The pitfall of the
general OCO model is the feedback; it is unrealistic to expect that the
decision maker has access to a gradient oracle at any point in the space
for every iteration of the game.

## The Bandit Convex Optimization Setting

The Bandit Convex Optimization (short: BCO) model is identical to the
general OCO model we have explored in previous chapters with the only
difference being the feedback available to the decision maker.

To be more precise, the BCO framework can be seen as a structured
repeated game. The protocol of this learning framework is as follows: At
iteration $t$, the online player chooses
$\ensuremath{\mathbf x}_t \in \ensuremath{\mathcal K}.$ After committing
to this choice, a convex cost function
$f_t \in {\mathcal F}: \ensuremath{\mathcal K}\mapsto {\mathbb R}$ is
revealed. Here ${\mathcal F}$ is the bounded family of cost functions
available to the adversary. The cost incurred to the online player is
the value of the cost function at the point she committed to
$f_t(\ensuremath{\mathbf x}_t)$. As opposed to the OCO model, in which
the decision maker has access to a gradient oracle for $f_t$ over
$\ensuremath{\mathcal K}$, in BCO **the loss
$f_t(\ensuremath{\mathbf x}_t)$ is the only feedback available to the
online player at iteration $t$.** In particular, the decision maker does
not know the loss had she chosen a different point
$\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}$ at iteration $t$.

As before, let $T$ denote the total number of game iterations (i.e.,
predictions and their incurred loss). Let ${\mathcal A}$ be an algorithm
for BCO, which maps a certain game history to a decision in the decision
set. We formally define the regret of ${\mathcal A}$ that predicted
$x_1,...,x_T$ to be

$$\ensuremath{\mathrm{{Regret}}}_T({\mathcal A}) = \sup_{\{f_1,...,f_T\} \subseteq {\mathcal F}}  \left\{   {\textstyle \sum}_{t=1}^T f_t(\ensuremath{\mathbf x}_t) -\min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} {\textstyle \sum}_{t=1}^T f_t(\ensuremath{\mathbf x}) \right\}.$$

## The Multiarmed Bandit (MAB) Problem

A classical model for decision making under uncertainty is the
multiarmed bandit (MAB) model. The term MAB nowadays refers to a
multitude of different variants and sub-scenarios that are too large to
survey. This section addresses perhaps the simplest variant---the
non-stochastic MAB problem---which is defined as follows:

Iteratively, a decision maker chooses between $n$ different actions
$i_t \in \{1,2,...,n\}$, while, at the same time, an adversary assigns
each action a loss in the range $[0,1]$. The decision maker receives the
loss for $i_t$ and observes this loss, and nothing else. The goal of the
decision maker is to minimize her regret.

The reader undoubtedly observes this setting is identical to the setting
of prediction from expert advice, the only difference being the feedback
available to the decision maker: whereas in the expert setting the
decision maker can observe the rewards or losses for all experts in
retrospect, in the MAB setting, only the losses of the decisions
actually chosen are known.

It is instructive to explicitly model this problem as a special case of
BCO. Take the decision set to be the set of all distributions over $n$
actions, i.e., $\ensuremath{\mathcal K}= \Delta_n$ is the
$n$-dimensional simplex. The loss function is taken to be the
linearization of the costs of the individual actions, that is:
$$f_t(\ensuremath{\mathbf x}) = \ell_t^\top \ensuremath{\mathbf x}=  \sum_{i=1}^n \ell_t(i) \ensuremath{\mathbf x}(i) \quad \forall \ensuremath{\mathbf x}\in \ensuremath{\mathcal K},$$
where $\ell_t(i)$ is the loss associated with the $i$'th action at the
$t$'th iteration. Thus, the cost functions are linear functions in the
BCO model.

The MAB problem exhibits an exploration-exploitation tradeoff: an
efficient (low regret) algorithm has to explore the value of the
different actions in order to make the best decision. On the other hand,
having gained sufficient information about the environment, a reasonable
algorithm needs to exploit this action by picking the best action.

The simplest way to attain a MAB algorithm would be to separate
exploration and exploitation. Such a method would proceed by

1.  With some probability, explore the action space (i.e., by choosing
    an action uniformly at random). Use the feedback to construct an
    estimate of the actions' losses.

2.  Otherwise, use the estimates to apply a full-information experts
    algorithm as if the estimates are the true historical costs.

This simple scheme already gives a sublinear regret algorithm, presented
in algorithm [\[alg:simpleMAB\]](#alg:simpleMAB){reference-type="ref"
reference="alg:simpleMAB"}.

::: algorithm
::: algorithmic
Input: OCO algorithm ${\mathcal A}$, parameter $\delta$. Let $b_t$ be a
Bernoulli random variable that equals 1 with probability $\delta$.
Choose $i_t \in \{1,2,...,n\}$ uniformly at random and play $i_t$.\
Let $$\hat{\ell}_t(i)= {
\left\{
\begin{array}{ll}
    { \frac{n}{\delta} \cdot \ell_t (i_t)}, & { i = i_t} \\\\
    {0 }, & {\text{otherwise}}
\end{array}
\right. } .$$ Let
${\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x})  = \hat{\ell}_t^\top \ensuremath{\mathbf x}$
and update
$\ensuremath{\mathbf x}_{t+1} = {\mathcal A}({\ensuremath{\hat{f}}}_1,...,{\ensuremath{\hat{f}}}_t)$.
Choose $i_t \sim \ensuremath{\mathbf x}_t$ and play $i_t$. Update
$\hat{f}_t = 0, \hat{\ell}_t = \mathbf{0}$,
$\ensuremath{\mathbf x}_{t+1} = \ensuremath{\mathbf x}_t$.
:::
:::

::: lemma
**Lemma 6.1**. *Algorithm
[\[alg:simpleMAB\]](#alg:simpleMAB){reference-type="ref"
reference="alg:simpleMAB"}, with $\mathcal{A}$ being the the online
gradient descent algorithm, guarantees the following regret bound:
$$\mathop{\mbox{\bf E}}\left[\sum_{t=1}^T {\ell_t(i_t)}-\min_i{\sum_{t=1}^T {\ell_t(i)}}\right] \leq O( T^{\frac{2}{3}} n^{\frac{2}{3}} ) \nonumber$$*
:::

::: proof
*Proof.* For the random functions $\{\hat{\ell}_t\}$ defined in
algorithm [\[alg:simpleMAB\]](#alg:simpleMAB){reference-type="ref"
reference="alg:simpleMAB"}, notice that

1.  $\mathop{\mbox{\bf E}}[ \hat{\ell}_t (i) ]  = \Pr[ b_t = 1] \cdot \Pr[ i_t = i | b_t=1] \cdot \frac{n}{\delta} \ell_t(i) = \ell_t(i)$.

2.  $\| \hat{\ell}_t \|_2 \leq \frac{n}{\delta} \cdot |\ell_t(i_t)| \leq \frac{n}{\delta}$.

Therefore the regret of the simple algorithm can be related to that of
$\mathcal{A}$ on the estimated functions.

On the other hand, the simple MAB algorithm does not always play
according to the distribution generated by $\mathcal{A}$: with
probability $\delta$ it plays uniformly at random, which may lead to a
regret of one on these exploration iterations. Let $S_t \subseteq [T]$
be those iterations in which $b_t=1$. This is captured by the following
lemma:

::: {#lem:shalom3 .lemma}
**Lemma 6.2**.
*$$\mathop{\mbox{\bf E}}[ \ell_t(i_t) ] \leq \mathop{\mbox{\bf E}}[ \hat{\ell}_t^\top x_t ] + \delta$$*
:::

::: proof
*Proof.* $$\begin{aligned}
&\mathop{\mbox{\bf E}}[\ell_t(i_t)]\\
&= \Pr[b_t=1] \cdot \mathop{\mbox{\bf E}}[ \ell_t(i_t)|b_t=1] \\ & + \Pr[b_t = 0] \cdot \mathop{\mbox{\bf E}}[\ell_t(i_t)|b_t=0] \\
& \le \delta + \Pr[b_t = 0] \cdot \mathop{\mbox{\bf E}}[\ell_t(i_t)|b_t=0] \\
& = \delta + (1-\delta)  \mathop{\mbox{\bf E}}[ \ell_t^\top \ensuremath{\mathbf x}_t | b_t = 0 ]  & \mbox{ $b_t = 0 \rightarrow i_t \sim \ensuremath{\mathbf x}_t$,  independent of $l_t$} \\
& \leq  \delta +   \mathop{\mbox{\bf E}}[ \ell_t^\top \ensuremath{\mathbf x}_t ]  & \mbox{non-negative random variables } \\
& = \delta +   \mathop{\mbox{\bf E}}[ \hat{\ell}_t^\top \ensuremath{\mathbf x}_t ]  & \mbox{$\hat{\ell}_t$ is independent of $\ensuremath{\mathbf x}_t$}
\end{aligned}$$ ◻
:::

We thus have, $$\begin{aligned}
& \mathop{\mbox{\bf E}}[ \ensuremath{\mathrm{{Regret}}}_T ]  \\
& =  \mathop{\mbox{\bf E}}[ \sum_{t=1}^T{\ell_t(i_t)}-{\sum_{t=1}^T{\ell_t(i^\star)}}] \\
& = \mathop{\mbox{\bf E}}[ \sum_{t  }{\ell_t(i_t)}-{\sum_{t } {\hat{\ell}_t(i^\star)}}   ] & \mbox{ $i^\star$ is indep. of $\hat{\ell}_t$} \\
& \leq \mathop{\mbox{\bf E}}[ \sum_{t  }{\hat{\ell}_t(\ensuremath{\mathbf x}_t)}-\min_i{\sum_{t } {\hat{\ell}_t(i)}} ]   + \delta T & \mbox{Lemma \ref{lem:shalom3} } \\
& =  \mathop{\mbox{\bf E}}[ \ensuremath{\mathrm{{Regret}}}_{S_T}(\mathcal{A}) ] + \delta \cdot T  \\
& \leq \frac{3}{2} GD  \sqrt{\delta T} + \delta \cdot T & \mbox{ Theorem \ref{thm:gradient}}, \mathop{\mbox{\bf E}}[ |S_T|] = \delta T  \\
& \leq 3 \frac{n}{ \sqrt{\delta}}  \sqrt{T  } + \delta \cdot T & \mbox{ For $\Delta_n$,  $D \leq 2$ , $\|\hat{\ell}_t\|\leq \frac{n}{\delta}  $}  \\
& =  O( T^{\frac{2}{3}} n^{\frac{2}{3}}) . & \delta= n^{\frac{2}{3}}  T^{-\frac{1}{3}}
\end{aligned}$$ ◻
:::

### EXP3: simultaneous exploration and exploitation

The simple algorithm of the previous section can be improved by
combining the exploration and exploitation steps. This gives a
near-optimal regret algorithm, called EXP3, presented below.

::: algorithm
::: algorithmic
Input: parameter $\varepsilon> 0$. Set
$\ensuremath{\mathbf x}_1 = ({1}/{n}) \mathbf{1}$. Choose
$i_t \sim \ensuremath{\mathbf x}_t$ and play $i_t$. Let
$$\hat{\ell}_t(i)= {
\left\{
\begin{array}{ll}
    { \frac{1}{\ensuremath{\mathbf x}_t(i_t)} \cdot \ell_t (i_t)}, & { i = i_t} \\\\
    {0 }, & {\text{otherwise}}
\end{array}
\right. }$$ Update
$\ensuremath{\mathbf y}_{t+1} (i)  = \ensuremath{\mathbf x}_t(i) e^{-\varepsilon\hat{\ell}_t(i)}  \ , \ \ensuremath{\mathbf x}_{t+1} = \frac{\ensuremath{\mathbf y}_{t+1} }{\|\ensuremath{\mathbf y}_{t+1}\|_1 }$
:::
:::

As opposed to the simple multiarmed bandit algorithm, the EXP3 algorithm
explores every iteration by always creating an unbiased estimator of the
entire loss vector. This results in a possibly large magnitude of the
vectors $\hat{\ell}$ and a large gradient bound for use with online
gradient descent. However, the large magnitude vectors are created with
low probability (proportional to their magnitude), which allows for a
finer analysis.

Ultimately, the EXP3 algorithm attains a worst case regret bound of
$O(\sqrt{T n \log n})$, which is nearly optimal (up to a logarithmic
term in the number of actions).

::: {#Lemma:exp3regret .lemma}
**Lemma 6.3**. *Algorithm [\[alg:EXP3\]](#alg:EXP3){reference-type="ref"
reference="alg:EXP3"} with non-negative losses and
$\varepsilon= \sqrt{\frac{\log n}{T n} }$ guarantees the following
regret bound:
$$\mathop{\mbox{\bf E}}[\sum{\ell_t(i_t)}-\min_i{\sum{\ell_t(i)}}] \leq 2 \sqrt{ T n \log n }  .\nonumber$$*
:::

::: proof
*Proof.* For the random losses $\{\hat{\ell}_t\}$ defined in algorithm
[\[alg:EXP3\]](#alg:EXP3){reference-type="ref" reference="alg:EXP3"},
notice that $$\begin{aligned}
		& \mathop{\mbox{\bf E}}[ \hat{\ell}_t (i) ]  = \Pr[ i_t = i] \cdot  \frac{ \ell_t(i)}{ \ensuremath{\mathbf x}_t(i) }  =  \ensuremath{\mathbf x}_t(i) \cdot  \frac{ \ell_t(i)}{ \ensuremath{\mathbf x}_t(i) }  = \ell_t(i) . \notag \\
		& \mathop{\mbox{\bf E}}[ \ensuremath{\mathbf x}_t^\top \hat{\ell}_t^2 ]  = \sum_i \Pr[ i_t = i] \cdot  \ensuremath{\mathbf x}_t(i)  \hat{\ell}_t(i)^2 \notag \\
		& =  \sum_i   \ensuremath{\mathbf x}_t(i)^2  \hat{\ell}_t(i)^2
		 = \sum_i \ell_t(i)^2  \leq  {n} . \label{eqn:shalom1234}

\end{aligned}$$ Therefore we have $E[\hat{f}_t]=f_t$, and the expected
regret with respect to the functions $\{\hat{f}_t\}$ is equal to that
with respect to the functions $\{f_t\}$. Thus, the regret with respect
to $\hat{\ell}_t$ can be related to that of $\ell_t$.

The EXP3 algorithm applies Hedge to the losses given by $\hat{\ell}_t$,
which are all non-negative and thus satisfy the conditions of Theorem
[1.5](#lem:hedge){reference-type="ref" reference="lem:hedge"}. Thus, the
expected regret with respect to $\hat{\ell}_t$, can be bounded by,

$$\begin{aligned}
		& \mathop{\mbox{\bf E}}[ \ensuremath{\mathrm{{Regret}}}_T ]   =   \mathop{\mbox{\bf E}}[ \sum_{t=1}^T{\ell_t(i_t)}-\min_i{\sum_{t=1}^T{\ell_t(i)}}] \\
		& =  \mathop{\mbox{\bf E}}[ \sum_{t=1}^T{\ell_t(i_t)}-{\sum_{t=1}^T{\ell_t(i^\star)}}] \\
		& \leq \mathop{\mbox{\bf E}}[ \sum_{t=1}^{T} {\hat{\ell}_t(\ensuremath{\mathbf x_{t}})}-{\sum_{t=1}^{T} {\hat{\ell}_t(i^\star)}}  ] & \mbox{ $i^\star$ is indep. of $\hat{\ell}_t$} \\
		& \leq \mathop{\mbox{\bf E}}[ \varepsilon\sum_{t=1}^T \sum_{i=1}^n \hat{\ell}_t(i)^2 \ensuremath{\mathbf x}_t(i) + \frac{\log n}{\varepsilon} ]  & \mbox{ Theorem \ref{lem:hedge} }  \\
		& \leq \varepsilon T n + \frac{\log n}{\varepsilon}  & \mbox{ equation \eqref{eqn:shalom1234}  }  \\
		& \leq  2 \sqrt{T n \log n }. & \mbox { by choice of $\varepsilon$ }

\end{aligned}$$ ◻
:::

We proceed to derive an algorithm for the more general setting of bandit
convex optimization that attains near-optimal regret.

## A Reduction from Limited Information to Full Information

In this section we derive a low regret algorithm for the general setting
of bandit convex optimization. In fact, we shall describe a general
technique for designing bandit algorithms, which is composed of two
parts:

1.  A general technique for taking an online convex optimization
    algorithm that uses only the gradients of the cost functions
    (formally defined below), and applying it to a family of vector
    random variables with carefully chosen properties.

2.  Designing the random variables that allow the template reduction to
    produce meaningful regret guarantees.

We proceed to describe the two parts of this reduction, and in the
remainder of this chapter we describe two examples of using this
reduction to design bandit convex optimization algorithms.

### Part 1: using unbiased estimators

The key idea behind many of the efficient algorithms for bandit convex
optimization is the following: although we cannot calculate
$\nabla f_t(\ensuremath{\mathbf x}_t)$ explicitly, it is possible to
find an *observable* random variable $\ensuremath{\mathbf g_{t}}$ that
satisfies
$\mathop{\mbox{\bf E}}[\ensuremath{\mathbf g_{t}}] \approx \nabla f_t (\ensuremath{\mathbf x}_t) = \nabla_t$.
Thus, $\ensuremath{\mathbf g_{t}}$ can be seen as an estimator of the
gradient. By substituting $\ensuremath{\mathbf g_{t}}$ for $\nabla_t$ in
an OCO algorithm, we will show that many times it retains its sublinear
regret bound.

Formally, the family of regret minimization algorithms for which this
reduction works is captured in the following definition.

::: definition
**Definition 6.4**. *(**first order OCO Algorithm**) Let ${\mathcal A}$
be an OCO (deterministic) algorithm receiving an arbitrary sequence of
differential loss functions $f_1,\ldots,f_T$, and producing decisions
$\ensuremath{\mathbf x}_1 \gets {\mathcal A}(\emptyset), \ensuremath{\mathbf x}_t \gets {\mathcal A}(f_1,\ldots,f_{t-1})$.
${\mathcal A}$ is called a *first order online algorithm* if the
following holds:*

-   *The family of loss functions $\mathcal{F}$ is closed under addition
    of linear functions: if $f\in \mathcal{F}$ and
    $\ensuremath{\mathbf u}\in {\mathbb R}^n$ then
    $f+ \ensuremath{\mathbf u}^\top \ensuremath{\mathbf x}\in \mathcal{F}$.*

-   *Let $\hat{f}_t$ be the linear function
    $\hat{f}_t(\ensuremath{\mathbf x}) = \nabla f_t(\ensuremath{\mathbf x}_t) ^\top \ensuremath{\mathbf x}$,
    then for every iteration $t\in[T]$:
    $${\mathcal A}(f_1,\ldots,f_{t-1}) = {\mathcal A}(\hat{f}_1,...,\hat{f}_{t-1})$$*
:::

We can now consider a formal reduction from any first order online
algorithm to a bandit convex optimization algorithm as follows.

::: algorithm
[]{#BCO2OCO label="BCO2OCO"}

::: algorithmic
Input: convex set $\ensuremath{\mathcal K}\subset {\mathbb R}^n$, first
order online algorithm ${\mathcal A}$. Let
$\ensuremath{\mathbf x}_1 = {\mathcal A}( \emptyset )$. Generate
distribution ${\mathcal D}_t$, sample
$\ensuremath{\mathbf y}_t \sim {\mathcal D}_t$ with
$\mathop{\mbox{\bf E}}[\ensuremath{\mathbf y}_t] = \ensuremath{\mathbf x}_t$.
Play $\ensuremath{\mathbf y}_t$. Observe
$f_t(\ensuremath{\mathbf y}_t)$, generate $\ensuremath{\mathbf g_{t}}$
with
$\mathop{\mbox{\bf E}}[\ensuremath{\mathbf g_{t}}] = \nabla f_t (\ensuremath{\mathbf x}_t)$.
Let
$\ensuremath{\mathbf x_{t+1}}   = {\mathcal A}(\ensuremath{\mathbf g_{1}},...,\ensuremath{\mathbf g_{t}})$.
:::
:::

Perhaps surprisingly, under very mild conditions the reduction above
guarantees the same regret bounds as the original first order algorithm
up to the magnitude of the estimated gradients. This is captured in the
following lemma.

::: {#Lemma:Flaxman_FirstOrderAlgos .lemma}
**Lemma 6.5**. *Let $\ensuremath{\mathbf u}$ be a *fixed* point in
$\ensuremath{\mathcal K}$. Let
$f_1,\ldots,f_T:\ensuremath{\mathcal K}\to {\mathbb R}$ be a sequence of
differentiable functions. Let ${\mathcal A}$ be a first order online
algorithm that ensures a regret bound of the form
$\ensuremath{\mathrm{{Regret}}}_T({{\mathcal A}}) \leq B_{{\mathcal A}}( \nabla f_1(\ensuremath{\mathbf x}_1),\ldots,\nabla f_T(\ensuremath{\mathbf x}_T))$
in the full information setting. Define the points
$\{ \ensuremath{\mathbf x}_t \}$ as:
$\ensuremath{\mathbf x}_1\gets{\mathcal A}(\emptyset)$,
$\ensuremath{\mathbf x}_t \gets {\mathcal A}(\ensuremath{\mathbf g_{1}},\ldots,\ensuremath{\mathbf g_{t-1}})$
where each $\ensuremath{\mathbf g_{t}}$ is a vector valued random
variable such that:
$$\mathop{\mbox{\bf E}}[\ensuremath{\mathbf g_{t}}\big \vert \ensuremath{\mathbf x}_1,f_1,\ldots, \ensuremath{\mathbf x}_t,f_t]=\nabla f_t(\ensuremath{\mathbf x}_t) .$$
Then the following holds for all
$\ensuremath{\mathbf u}\in \ensuremath{\mathcal K}$: $$\begin{aligned}
\mathop{\mbox{\bf E}}[\sum_{t=1}^T f_t(\ensuremath{\mathbf x}_t)] - \sum_{t=1}^T f_t(\ensuremath{\mathbf u}) \leq  \mathop{\mbox{\bf E}}[B_{{\mathcal A}}(\ensuremath{\mathbf g_{1}},\ldots,\ensuremath{\mathbf g_{T}})] .
\end{aligned}$$*
:::

::: proof
*Proof.* Define the functions
$h_t:\ensuremath{\mathcal K}\to{\mathbb R}$ as follows:
$$h_t(\ensuremath{\mathbf x}) = f_t(\ensuremath{\mathbf x}) + \boldsymbol\xi_t^\top  \ensuremath{\mathbf x}, \; \text{where } \boldsymbol\xi_t = \ensuremath{\mathbf g_{t}}-\nabla f_t(\ensuremath{\mathbf x}_t).$$
Note that
$$\nabla h_t(\ensuremath{\mathbf x}_t) =\nabla f_t(\ensuremath{\mathbf x}_t)+  \ensuremath{\mathbf g_{t}}-\nabla f_t(\ensuremath{\mathbf x}_t)=\ensuremath{\mathbf g_{t}}.$$
Therefore, deterministically applying a first order method
${\mathcal A}$ on the random functions $h_t$ is equivalent to applying
${\mathcal A}$ on a stochastic first order approximation of the
deterministic functions $f_t$. Thus by the full-information regret bound
of ${\mathcal A}$ we have: $$\begin{aligned}
\label{equation:regretBeforeExpectation}
 \sum_{t=1}^T h_t(\ensuremath{\mathbf x}_t) - \sum_{t=1}^T h_t(\ensuremath{\mathbf u}) \leq  B_{{\mathcal A}}(\ensuremath{\mathbf g_{1}},\ldots,\ensuremath{\mathbf g_{T}}).
\end{aligned}$$ Also note that: $$\begin{aligned}
\mathop{\mbox{\bf E}}[h_t(\ensuremath{\mathbf x}_t)]&=\mathop{\mbox{\bf E}}[f_t(\ensuremath{\mathbf x}_t)]+\mathop{\mbox{\bf E}}[\boldsymbol\xi_t^\top  \ensuremath{\mathbf x}_t] \\
& = \mathop{\mbox{\bf E}}[f_t(\ensuremath{\mathbf x}_t)]+\mathop{\mbox{\bf E}}[\mathop{\mbox{\bf E}}[\boldsymbol\xi_t^\top \ensuremath{\mathbf x}_t\big\vert \ensuremath{\mathbf x}_1,f_1,\ldots,\ensuremath{\mathbf x}_t,f_t] ] \\
&= \mathop{\mbox{\bf E}}[f_t(\ensuremath{\mathbf x}_t)]+\mathop{\mbox{\bf E}}[\mathop{\mbox{\bf E}}[\boldsymbol\xi_t \big\vert \ensuremath{\mathbf x}_1,f_1,\ldots,\ensuremath{\mathbf x}_t,f_t] ^\top \ensuremath{\mathbf x}_t] \\
& = \mathop{\mbox{\bf E}}[f_t(\ensuremath{\mathbf x}_t)].
\end{aligned}$$ where we used
$\mathop{\mbox{\bf E}}[\boldsymbol\xi_t\vert \ensuremath{\mathbf x}_1,f_1,\ldots,\ensuremath{\mathbf x}_t,f_t]=0$.
Similarly, since $\ensuremath{\mathbf u}\in\ensuremath{\mathcal K}$ is
fixed we have that
$\mathop{\mbox{\bf E}}[h_t(\ensuremath{\mathbf u})] = f_t(\ensuremath{\mathbf u})$.
The lemma follows from taking the expectation of Equation
[\[equation:regretBeforeExpectation\]](#equation:regretBeforeExpectation){reference-type="eqref"
reference="equation:regretBeforeExpectation"}. ◻
:::

### Part 2: point-wise gradient estimators

In the preceding part we have described how to convert a first order
algorithm for OCO to one that uses bandit information, using specially
tailored random variables. We now describe how to create these vector
random variables.

Although we cannot calculate $\nabla f_t(\ensuremath{\mathbf x}_t)$
explicitly, it is possible to find an *observable* random variable
$\ensuremath{\mathbf g_{t}}$ that satisfies
$\mathop{\mbox{\bf E}}[\ensuremath{\mathbf g_{t}}] \approx \nabla f_t$,
and serves as an estimator of the gradient.

The question is how to find an appropriate $\ensuremath{\mathbf g_{t}}$,
and in order to answer it we begin with an example in a 1-dimensional
case.

::: example
**Example 6.6**. *A 1-dimensional gradient estimate Recall the
definition of the derivative: $$\label{derivative}
f'(x)=\lim_{\delta \rightarrow 0}{\frac{f(x+\delta)-f(x-\delta)}{2 \delta}}. \nonumber$$
The above shows that for a 1-dimensional derivative, two evaluations of
$f$ are required. Since in our problem we can perform only one
evaluation, let us define $g(x)$ as follows: $$g(x) = {
\left\{
\begin{array}{ll}
    {\frac{f(x+\delta)}{\delta}}, & {\text{with probability }  \frac{1}{2}} \\\\
    { - \frac{f(x-\delta)}{\delta}}, & { \text{with probability }  \frac{1}{2}}
\end{array}
\right. }.
\label{gt}$$ It is clear that
$$\mathop{\mbox{\bf E}}[g(x)]={\frac{f(x+\delta)-f(x-\delta)}{2 \delta}}.$$
Thus, **in expectation**, for small $\delta$, $g(x)$ approximates
$f'(x)$.*
:::

#### The sphere sampling estimator

We will now show how the gradient estimator
[\[gt\]](#gt){reference-type="eqref" reference="gt"} can be extended to
the multidimensional case. Let $\ensuremath{\mathbf x}\in \mathbb{R}^n$,
and let $B_{\delta}$ and $S_{\delta}$ denote the $n$-dimensional ball
and sphere with radius $\delta:$
$$B_{\delta}=\left\{\ensuremath{\mathbf x}|\left\|\ensuremath{\mathbf x}\right\| \leq \delta \right\},$$
$$S_{\delta}=\left\{\ensuremath{\mathbf x}|\left\|\ensuremath{\mathbf x}\right\| = \delta \right\}.$$

We define
$\hat{f}(\ensuremath{\mathbf x})= \hat{f}_\delta(\ensuremath{\mathbf x})$
to be a $\delta$-smoothed version of $f(\ensuremath{\mathbf x})$:
$$\label{fhat}
\hat{f}_\delta \left(\ensuremath{\mathbf x}\right)=\mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\in \mathbb{B}}\left[f\left(\ensuremath{\mathbf x}+\delta \ensuremath{\mathbf v}\right)\right],$$
where $\ensuremath{\mathbf v}$ is drawn from a uniform distribution over
the unit ball. This construction is very similar to the one used in
Lemma [2.8](#lem:SmoothingLemma){reference-type="ref"
reference="lem:SmoothingLemma"} in context of convergence analysis for
convex optimization. However, our goal here is very different.

Note that when $f$ is linear, we have
$\hat{f}_\delta(\ensuremath{\mathbf x})=f(\ensuremath{\mathbf x})$. We
shall address the case in which $f$ is indeed linear as a special case,
and show how to estimate the gradient of
$\hat{f}(\ensuremath{\mathbf x})$, which, under the assumption, is also
the gradient of $f(\ensuremath{\mathbf x})$. The following lemma shows a
simple relation between the gradient $\nabla \hat{f}_\delta$ and a
uniformly drawn unit vector.

::: {#lem_stokes .lemma}
**Lemma 6.7**. *Fix $\delta>0$. Let
$\hat{f}_\delta(\ensuremath{\mathbf x})$ be as defined in
[\[fhat\]](#fhat){reference-type="eqref" reference="fhat"}, and let
$\ensuremath{\mathbf u}$ be a uniformly drawn unit vector
$\ensuremath{\mathbf u}\sim \ensuremath{\mathbb {S}}$. Then
$$\mathop{\mbox{\bf E}}_{\ensuremath{\mathbf u}\in \ensuremath{\mathbb {S}}}\left[f\left(\ensuremath{\mathbf x}+\delta \ensuremath{\mathbf u}\right) \ensuremath{\mathbf u}\right]=\frac{\delta}{n}\nabla\hat{f}_\delta \left( \ensuremath{\mathbf x}\right).$$*
:::

::: proof
*Proof.* Using Stokes' theorem from calculus, we have
$$\nabla\underset{B_{\delta}}{\int}f\left(\ensuremath{\mathbf x}+\ensuremath{\mathbf v}\right)d \ensuremath{\mathbf v}=\underset{S_{\delta}}{\int}f\left(\ensuremath{\mathbf x}+\ensuremath{\mathbf u}\right)\frac{\ensuremath{\mathbf u}}{\left\Vert \ensuremath{\mathbf u}\right\Vert }d \ensuremath{\mathbf u}.\label{stokes}$$
From [\[fhat\]](#fhat){reference-type="eqref" reference="fhat"}, and by
definition of expectation, we have
$$\hat{f}_\delta(\ensuremath{\mathbf x})=\frac{\underset{B_{\delta}}{\int}f\left(\ensuremath{\mathbf x}+ \ensuremath{\mathbf v}\right)d \ensuremath{\mathbf v}}{\mbox{vol}( B_{\delta})} . \label{vol1}$$
where $\mbox{vol}(B_{\delta})$ is the volume of an n-dimensional ball of
radius $\delta$. Similarly,
$$\mathop{\mbox{\bf E}}_{\ensuremath{\mathbf u}\in S}\left[f\left(\ensuremath{\mathbf x}+\delta \ensuremath{\mathbf u}\right)\ensuremath{\mathbf u}\right]=\frac{\underset{S_{\delta}}{\int}f\left(\ensuremath{\mathbf x}+ \ensuremath{\mathbf u}\right)\frac{\ensuremath{\mathbf u}}{\left\Vert \ensuremath{\mathbf u}\right\Vert }du}{\mbox{vol}(S_{\delta} ) }  . \label{vol2}$$
Combining [\[fhat\]](#fhat){reference-type="eqref" reference="fhat"},
[\[stokes\]](#stokes){reference-type="eqref" reference="stokes"},
[\[vol1\]](#vol1){reference-type="eqref" reference="vol1"}, and
[\[vol2\]](#vol2){reference-type="eqref" reference="vol2"}, and the fact
that the ratio of the volume of a ball in $n$ dimensions and the sphere
of dimension $n-1$ is
$\textrm{vol}_{n}B_{\delta}/\textrm{vol}_{n-1}S_{\delta}=\delta/n$ gives
the desired result. ◻
:::

Under the assumption that $f$ is linear, Lemma
[6.7](#lem_stokes){reference-type="ref" reference="lem_stokes"} suggests
a simple estimator for the gradient $\nabla f$. Draw a random unit
vector $\ensuremath{\mathbf u}$, and let
$g\left(\ensuremath{\mathbf x}\right)=\frac{n}{\delta}f\left(\ensuremath{\mathbf x}+\delta \ensuremath{\mathbf u}\right)\ensuremath{\mathbf u}$.

#### The ellipsoidal sampling estimator

The sphere estimator above is at times difficult to use: when the center
of the sphere is very close to the boundary of the decision set only a
very small sphere can fit completely inside. This results in a gradient
estimator with large variance.

In such cases, it is useful to consider ellipsoids rather than spheres.
Luckily, the generalisation to ellipsoidal sampling for gradient
estimation is a simple corollary of our derivation above:

::: {#Corollary:Gradient_Estimate_SinglePoint .corollary}
**Corollary 6.8**. *Consider a continuous function
$f:{\mathbb R}^n\to {\mathbb R}$, an invertible matrix
$A\in {\mathbb R}^{n \times n}$, and let
$\ensuremath{\mathbf v}\sim \mathbb{B}^n$ and
$\ensuremath{\mathbf u}\sim \ensuremath{\mathbb {S}}^n$. Define the
smoothed version of $f$ with respect to $A$: $$\begin{aligned}
\hat{f}(\ensuremath{\mathbf x}) = \mathop{\mbox{\bf E}}[ f(\ensuremath{\mathbf x}+A \ensuremath{\mathbf v})  ].
\end{aligned}$$ Then the following holds: $$\begin{aligned}
\nabla \hat{f}(\ensuremath{\mathbf x}) = n  \mathop{\mbox{\bf E}}[ f(\ensuremath{\mathbf x}+A \ensuremath{\mathbf u}) A^{-1} \ensuremath{\mathbf u}].
\end{aligned}$$*
:::

::: proof
*Proof.* Let $g(\ensuremath{\mathbf x}) = f(A \ensuremath{\mathbf x})$,
and
$\hat{g}(\ensuremath{\mathbf x}) = \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\in \mathbb{B}} [g(\ensuremath{\mathbf x}+ \ensuremath{\mathbf v})]$.
$$\begin{aligned}
n  \mathop{\mbox{\bf E}}[ f(\ensuremath{\mathbf x}+A \ensuremath{\mathbf u}) A^{-1} \ensuremath{\mathbf u}] & = n A^{-1}  \mathop{\mbox{\bf E}}[ f(\ensuremath{\mathbf x}+A \ensuremath{\mathbf u})  \ensuremath{\mathbf u}] \\
& = n A^{-1}  \mathop{\mbox{\bf E}}[ g ( A^{-1} \ensuremath{\mathbf x}+ \ensuremath{\mathbf u})  \ensuremath{\mathbf u}] \\
& = A^{-1}   \nabla \hat{g}(A^{-1} \ensuremath{\mathbf x})  & \mbox { Lemma \ref{lem_stokes} } \\
& = A^{-1}  A   \nabla \hat{f}( \ensuremath{\mathbf x})  = \nabla \hat{f}(\ensuremath{\mathbf x}).
\end{aligned}$$ ◻
:::

## Online Gradient Descent without a Gradient

The simplest and historically earliest application of the BCO-to-OCO
reduction outlined before is the application of the online gradient
descent algorithm to the bandit setting. The FKM algorithm (named after
its inventors, see bibliographic section) is outlined in algorithm
[\[FKM_alg\]](#FKM_alg){reference-type="ref" reference="FKM_alg"}.

For simplicity, we assume that the set $\ensuremath{\mathcal K}$
contains the unit ball centered at the zero vector, denoted
$\mathbf{0}$. Denote
$\ensuremath{\mathcal K}_\delta = \{ \ensuremath{\mathbf x}\ | \  \frac{1}{1-\delta} \ensuremath{\mathbf x}\in \ensuremath{\mathcal K}\}$.
It is left as an exercise to show that $\ensuremath{\mathcal K}_\delta$
is convex for any $0 <  \delta < 1$ and that all balls of radius
$\delta$ around points in $\ensuremath{\mathcal K}_\delta$ are contained
in $\ensuremath{\mathcal K}$.

We also assume for simplicity that the adversarially chosen cost
functions are bounded by one over $\ensuremath{\mathcal K}$, i.e., that
$| \ensuremath{\mathbf f_{t}}(\ensuremath{\mathbf x}) | \leq 1$ for all
$\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}$.

::: center
![The Minkowski set $\ensuremath{\mathcal K}_\delta$
](images/fig_mink.png){#fig:Minkowski width="3.5in"}
:::

::: algorithm
::: algorithmic
Input: decision set $\ensuremath{\mathcal K}$ containing $\mathbf{0}$,
set $\ensuremath{\mathbf x}_1 = \mathbf{0}$, parameters $\delta,\eta$.
Draw $\ensuremath{\mathbf u}_t \in \ensuremath{\mathbb {S}}_1$ uniformly
at random, set
$\ensuremath{\mathbf y}_t  = \ensuremath{\mathbf x}_t + \delta \ensuremath{\mathbf u}_t$.
Play $\ensuremath{\mathbf y}_t$, observe and incur loss
$f_t \left( \ensuremath{\mathbf y}_t \right)$. Let
$\ensuremath{\mathbf g_{t}}=  \frac{n}{\delta} f_{t}\left(\ensuremath{\mathbf y}_{t}\right)\ensuremath{\mathbf u}_{t}$.
Update
$\ensuremath{\mathbf x}_{t+1}=  \underset{\ensuremath{\mathcal K}_\delta}{\mathop{\Pi}}\left[\ensuremath{\mathbf x}_{t}- \eta \ensuremath{\mathbf g_{t}}\right]$.
:::
:::

The FKM algorithm is an instantiation of the generic reduction from
bandit convex optimization to online convex optimization with spherical
gradient estimators over the set $\ensuremath{\mathcal K}_\delta$. It
iteratively projects onto $\ensuremath{\mathcal K}_\delta$, in order to
have enough space for spherical gradient estimation. This degrades its
performance by a controlled quantity. Its regret is bounded as follows.

::: {#FKM_prop .theorem}
**Theorem 6.9**. *Algorithm [\[algFKM\]](#algFKM){reference-type="ref"
reference="algFKM"} with parameters
$\ \eta = \frac{D}{n T^{3/4} } , \delta = \frac{1}{T^{1/4}}$ guarantees
the following expected regret bound
$$\sum_{t=1}^T \mathop{\mbox{\bf E}}[ f_{t}(\ensuremath{\mathbf y}_{t}) ]-\min_{\ensuremath{\mathbf x}\in\mathcal{K}} \sum_{t=1}^T  f_{t} (\ensuremath{\mathbf x})  \leq 9 n D G T^{3/4}  = O(T^{3/4}) .$$*
:::

::: proof
*Proof.* Recall our notation of
$\ensuremath{\mathbf x}^\star = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \sum_{t=1}^T f_t(\ensuremath{\mathbf x})$.
Denote
$$\ensuremath{\mathbf x}_{\delta}^{\star}= \mathop{\Pi}_{\ensuremath{\mathcal K}_\delta} (\ensuremath{\mathbf x}^\star ) .$$
Then by properties of projections we have
$\|\ensuremath{\mathbf x}_\delta^\star - \ensuremath{\mathbf x}^\star\| \leq \delta D$,
where $D$ is the diameter of $\ensuremath{\mathcal K}$. Thus, assuming
that the cost functions $\{f_t\}$ are $G$-Lipschitz, we have
$$\label{Lip_step}
\sum_{t=1}^T \mathop{\mbox{\bf E}}[ f_{t}(\ensuremath{\mathbf y}_{t}) ]-  \sum_{t=1}^T  f_{t} (\ensuremath{\mathbf x}^\star)
 \leq  \sum_{t=1}^T \mathop{\mbox{\bf E}}[ f_{t}(\ensuremath{\mathbf y}_{t}) ]-  \sum_{t=1}^T  f_{t} (\ensuremath{\mathbf x}_\delta^\star)  +  \delta T G D .$$

Denote
$\hat{f}_t = \hat{f}_{\delta,t} = \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf u}\sim \mathbb{B}} [f(\ensuremath{\mathbf x}+ \delta \ensuremath{\mathbf u}) ]$
for shorthand. We can now bound the regret by $$\begin{aligned}
& \sum_{t=1}^T \mathop{\mbox{\bf E}}[ f_{t}(\ensuremath{\mathbf y}_{t}) ]-  \sum_{t=1}^T  f_{t} (\ensuremath{\mathbf x}^\star) \\
& \leq \sum_{t=1}^T \mathop{\mbox{\bf E}}[ f_{t}(\ensuremath{\mathbf x}_{t}) ]-  \sum_{t=1}^T  f_{t} (\ensuremath{\mathbf x}^\star) + \delta D GT & \mbox{$f_t$ is $G$-Lipschitz } \\
& \leq \sum_{t=1}^T \mathop{\mbox{\bf E}}[ {f}_{t}(\ensuremath{\mathbf x}_{t}) ]- \sum_{t=1}^T  {f}_{t} (\ensuremath{\mathbf x}^\star_\delta) +  2 \delta D G  T & \mbox{Inequality (\ref{Lip_step}})  \\
& \leq \sum_{t=1}^T \mathop{\mbox{\bf E}}[ \hat{f}_{t}(\ensuremath{\mathbf x}_{t}) ]- \sum_{t=1}^T  \hat{f}_{t} (\ensuremath{\mathbf x}^\star_\delta) + 4 \delta D G T  & \mbox{ Lemma \ref{lem:SmoothingLemma} } \\
& \leq \ensuremath{\mathrm{{Regret}}}_{OGD}( \ensuremath{\mathbf g_{1}} , ..., \ensuremath{\mathbf g_{T}} ) + 4 \delta D G T & \mbox{ Lemma \ref{Lemma:Flaxman_FirstOrderAlgos} } \\
& \leq  \eta \sum_{t=1}^T \| \ensuremath{\mathbf g_{t}}\|^2 + \frac{D^2}{\eta}  +  4 \delta D G T & \mbox{ OGD regret, Theorem \ref{thm:gradient} } \\
& \leq  \eta \frac{n^2}{\delta^2} T  + \frac{D^2}{\eta}  +  4 \delta D G T & |\ensuremath{\mathbf f_{t}}(\ensuremath{\mathbf x})| \leq 1  \\
& \leq  9 n D G T^{3/4}  . & \eta = \frac{D}{n T^{3/4} } , \delta = \frac{1}{T^{1/4}}
\end{aligned}$$ ◻
:::

## \* Optimal Regret Algorithms for Bandit Linear Optimization

A special case of BCO that is of considerable interest is BLO---Bandit
Linear Optimization. This setting has linear cost functions, and
captures the network routing and ad placement examples discussed in the
beginning of this chapter, as well as the non-stochastic MAB problem.

In this section we give near-optimal regret bounds for BLO using
techniques from interior point methods for convex optimization.

The generic OGD method of the previous section suffers from three
pitfalls:

1.  The gradient estimators are biased, and estimate the gradient of a
    smoothed version of the real cost function.

2.  The gradient estimators require enough "wiggle room" and are thus
    ill-defined on the boundary of the decision set.

3.  The gradient estimates have potentially large magnitude,
    proportional to the distance from the boundary.

Fortunately, the first issue is non-existent for linear functions - the
gradient estimators turn out to be unbiased for linear functions. In the
notation of the previous chapters, we have for linear functions:
$$\hat{f}_\delta(\ensuremath{\mathbf x})  = \mathop{\mbox{\bf E}}_{\ensuremath{\mathbf v}\sim \mathbb{B}} [f(\ensuremath{\mathbf x}+ \delta \ensuremath{\mathbf v}) ] = f(\ensuremath{\mathbf x}) .$$
Thus, Lemma [6.7](#lem_stokes){reference-type="ref"
reference="lem_stokes"} gives us a stronger guarantee:
$$\mathop{\mbox{\bf E}}_{\ensuremath{\mathbf u}\in \ensuremath{\mathbb {S}}}\left[f\left(\ensuremath{\mathbf x}+\delta \ensuremath{\mathbf u}\right) \ensuremath{\mathbf u}\right]=\frac{\delta}{n}\nabla\hat{f}_\delta \left( \ensuremath{\mathbf x}\right) = \frac{\delta}{n} \nabla f(\ensuremath{\mathbf x}) .$$

To resolve the second and third issues we use self-concordant barrier
functions, a rather advanced technique from interior point methods for
convex optimization.

### Self-concordant barriers

Self-concordant barrier functions were devised in the context of
interior point methods for optimization as a way of ensuring that the
Newton method converges in polynomial time over bounded convex sets. In
this brief introduction we survey some of their beautiful properties
that will allow us to derive an optimal regret algorithm for BLO.

::: definition
**Definition 6.10**. *Let $\ensuremath{\mathcal K}\in {\mathbb R}^n$ be
a convex set with a nonempty interior
$\text{int}(\ensuremath{\mathcal K})$. A function
$\ensuremath{\mathcal R}:\text{int}(\ensuremath{\mathcal K})\to {\mathbb R}$
is called $\nu$-self-concordant if:*

1.  *$\ensuremath{\mathcal R}$ is three times continuously
    differentiable and convex, and approaches infinity along any
    sequence of points approaching the boundary of
    $\ensuremath{\mathcal K}$.*

2.  *For every $\ensuremath{\mathbf h}\in {\mathbb R}^n$ and
    $\ensuremath{\mathbf x}\in \text{int}(\ensuremath{\mathcal K})$ the
    following holds: $$\begin{aligned}
    &|\nabla^3\ensuremath{\mathcal R}(\ensuremath{\mathbf x})[\ensuremath{\mathbf h},\ensuremath{\mathbf h},\ensuremath{\mathbf h}] |\leq 2( \nabla^2\ensuremath{\mathcal R}(\ensuremath{\mathbf x})[\ensuremath{\mathbf h},\ensuremath{\mathbf h}])^{3/2} ,\\
    &|\nabla\ensuremath{\mathcal R}(\ensuremath{\mathbf x})[\ensuremath{\mathbf h}] |\leq \nu^{1/2}( \nabla^2\ensuremath{\mathcal R}(\ensuremath{\mathbf x})[\ensuremath{\mathbf h},\ensuremath{\mathbf h}])^{1/2}
    \end{aligned}$$*
:::

where the third order differential is defined as: $$\begin{aligned}
\nabla^3\ensuremath{\mathcal R}(\ensuremath{\mathbf x})[\ensuremath{\mathbf h},\ensuremath{\mathbf h},\ensuremath{\mathbf h}] \stackrel{\text{\tiny def}}{=}\left.  \frac{\partial^3}{\partial t_1 \partial t_2 \partial t_3}  \ensuremath{\mathcal R}(\ensuremath{\mathbf x}+t_1 \ensuremath{\mathbf h}+t_2 \ensuremath{\mathbf h}+t_3 \ensuremath{\mathbf h})\right \vert_{t_1=t_2=t_3=0}
\end{aligned}$$ The Hessian of a self-concordant barrier induces a local
norm at every
$\ensuremath{\mathbf x}\in \text{int}(\ensuremath{\mathcal K})$, we
denote this norm by $||\cdot||_\ensuremath{\mathbf x}$ and its dual by
$||\cdot||_\ensuremath{\mathbf x}^{*},$ which are defined
$\forall \ensuremath{\mathbf h}\in {\mathbb R}^n$ by $$\begin{aligned}
\|\ensuremath{\mathbf h}\|_\ensuremath{\mathbf x}= \sqrt{\ensuremath{\mathbf h}^\top \nabla^2\ensuremath{\mathcal R}(\ensuremath{\mathbf x}) \ensuremath{\mathbf h}}, \qquad \|\ensuremath{\mathbf h}\|_\ensuremath{\mathbf x}^{*} = \sqrt{\ensuremath{\mathbf h}^\top (\nabla^2\ensuremath{\mathcal R}(\ensuremath{\mathbf x}))^{-1} \ensuremath{\mathbf h}}.
\end{aligned}$$ We assume that
$\nabla^2\ensuremath{\mathcal R}(\ensuremath{\mathbf x})$ always has
full rank. In BCO applications this is easy to ensure by adding a
fictitious quadratic function to the barrier, which does not affect the
overall regret by more than a constant.

Let $\ensuremath{\mathcal R}$ be a self-concordant barrier and
$\ensuremath{\mathbf x}\in \text{int}(\ensuremath{\mathcal K})$. The
*Dikin ellipsoid* is $$\begin{aligned}
 %\label{Definition:Dikin_ellipsoid}
{\mathcal E}_1(\ensuremath{\mathbf x}) :=\{\ensuremath{\mathbf y}\in{\mathbb R}^n : \|\ensuremath{\mathbf y}-\ensuremath{\mathbf x}\|_\ensuremath{\mathbf x}\leq 1\},
\end{aligned}$$ i.e., the $\|\cdot\|_\ensuremath{\mathbf x}$-unit ball
centered around $\ensuremath{\mathbf x}$, is completely contained in
$\ensuremath{\mathcal K}$.

In our next analysis we will need to bound
$\ensuremath{\mathcal R}(\ensuremath{\mathbf y}) - \ensuremath{\mathcal R}(\ensuremath{\mathbf x})$
for
$\ensuremath{\mathbf x},\ensuremath{\mathbf y}\in \text{int}(\ensuremath{\mathcal K})$,
for which the following lemma is useful:

::: {#Lemma:MinkowskiBarrier .lemma}
**Lemma 6.11**. *Let $\ensuremath{\mathcal R}$ be a $\nu$-self
concordant function over $\ensuremath{\mathcal K}$, then for all
$\ensuremath{\mathbf x},\ensuremath{\mathbf y}\in \text{int}(\ensuremath{\mathcal K})$:
$$\ensuremath{\mathcal R}(\ensuremath{\mathbf y})-\ensuremath{\mathcal R}(\ensuremath{\mathbf x})\leq \nu \log \frac{1}{1-\pi_{\ensuremath{\mathbf x}}(\ensuremath{\mathbf y})},$$
where
$\pi_\ensuremath{\mathbf x}(\ensuremath{\mathbf y}) = \inf\{t\geq 0: \ensuremath{\mathbf x}+t^{-1}(\ensuremath{\mathbf y}-\ensuremath{\mathbf x})\in\ensuremath{\mathcal K}\} .$*
:::

The function $\pi_\ensuremath{\mathbf x}(\ensuremath{\mathbf y})$ is
called the Minkowski function for $\ensuremath{\mathcal K}$, and its
output is always in the interval $[0,1]$. Moreover, as $y$ approaches
the boundary of $\ensuremath{\mathcal K}$ then
$\pi_\ensuremath{\mathbf x}(\ensuremath{\mathbf y})\to 1$.

Another important property of self-concordant functions is the
relationship between a point and the optimum, and the norm of the
gradient at the point, according to the local norm, as given by the
following lemma.

::: {#Lemma:DistanceAndGradients .lemma}
**Lemma 6.12**. *Let
$\ensuremath{\mathbf x}\in \text{int}(\ensuremath{\mathcal K})$ be such
that
$\|\nabla \ensuremath{\mathcal R}(\ensuremath{\mathbf x}) \| _\ensuremath{\mathbf x}^*  \leq \frac{1}{4}$,
and let
$\ensuremath{\mathbf x}^\star = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \ensuremath{\mathcal R}(\ensuremath{\mathbf x})$.
Then
$$\| \ensuremath{\mathbf x}- \ensuremath{\mathbf x}^\star\|_x \leq  2  \|\nabla \ensuremath{\mathcal R}(\ensuremath{\mathbf x}) \| _\ensuremath{\mathbf x}^* .$$*
:::

### A near-optimal algorithm

We have now set up all the necessary tools to derive a near-optimal BLO
algorithm, presented in algorithm
[\[alg:scrible\]](#alg:scrible){reference-type="ref"
reference="alg:scrible"}.

::: algorithm
[]{#alg:egmincut label="alg:egmincut"}

::: algorithmic
Input: decision set $\ensuremath{\mathcal K}$ with self concordant
barrier $\ensuremath{\mathcal R}$, set
$\ensuremath{\mathbf x}_1 \in \text{int}(\ensuremath{\mathcal K})$ such
that $\nabla \ensuremath{\mathcal R}(\ensuremath{\mathbf x}_1) = 0$,
parameters $\eta,\delta$. Let
$\ensuremath{\mathbf A_{t}}= \left[\nabla^2 \ensuremath{\mathcal R}(\ensuremath{\mathbf x}_t) \right]^{-1/2}$
. Pick $\ensuremath{\mathbf u}_t \in \ensuremath{\mathbb {S}}$
uniformly, and set
$\ensuremath{\mathbf y}_t  = \ensuremath{\mathbf x}_t +  \ensuremath{\mathbf A_{t}}\ensuremath{\mathbf u}_t$.
Play $\ensuremath{\mathbf y}_t$, observe and suffer loss
$f_t \left( \ensuremath{\mathbf y}_t \right)$. let
$\ensuremath{\mathbf g_{t}}=  n f_{t}\left(\ensuremath{\mathbf y}_{t}\right) \ensuremath{\mathbf A_{t}}^{-1 }\ensuremath{\mathbf u}_{t}$.
[]{#line:rftl label="line:rftl"} Update
$$\ensuremath{\mathbf x}_{t+1}= \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}_\delta} \left\{ \eta \sum_{\tau =1 }^t \ensuremath{\mathbf g_{\tau}}^\top \ensuremath{\mathbf x}+  \ensuremath{\mathcal R}(\ensuremath{\mathbf x})  \right\} .$$
:::
:::

::: theorem
**Theorem 6.13**. *For appropriate choice of $\eta,\delta$, the SCRIBLE
algorithm guarantees
$$\sum_{t=1}^T \mathop{\mbox{\bf E}}[ f_{t}(\ensuremath{\mathbf y}_{t}) ]-\min_{\ensuremath{\mathbf x}\in\mathcal{K}} \sum_{t=1}^T  f_{t} (\ensuremath{\mathbf x})  \leq O\left(\sqrt{T} \log T \right).$$*
:::

::: proof
*Proof.* First, we note that
$\ensuremath{\mathbf y}_t \in \ensuremath{\mathcal K}$ never steps
outside of the decision set. The reason is that
$\ensuremath{\mathbf x}_t \in \ensuremath{\mathcal K}$ and
$\ensuremath{\mathbf y}_t$ lies in the Dikin ellipsoid centered at
$\ensuremath{\mathbf x}_t$.

Further, by Corollary
[6.8](#Corollary:Gradient_Estimate_SinglePoint){reference-type="ref"
reference="Corollary:Gradient_Estimate_SinglePoint"}, we have that
$$\mathop{\mbox{\bf E}}[ \ensuremath{\mathbf g_{t}}] = \nabla \hat{f}_t (\ensuremath{\mathbf x}_t)  = \nabla f_t(\ensuremath{\mathbf x}_t),$$
where the latter equality follows since $f_t$ is linear, and thus its
smoothed version is identical to itself.

A final observation is that line
[\[line:rftl\]](#line:rftl){reference-type="ref" reference="line:rftl"}
in the algorithm is an invocation of the RFTL algorithm with the
self-concordant barrier $\ensuremath{\mathcal R}$ serving as a
regularisation function. The RFTL algorithm for linear functions is a
first order OCO algorithm and thus Lemma
[6.5](#Lemma:Flaxman_FirstOrderAlgos){reference-type="ref"
reference="Lemma:Flaxman_FirstOrderAlgos"} applies.

We can now bound the regret by $$\begin{aligned}
& \sum_{t=1}^T \mathop{\mbox{\bf E}}[ f_{t}(\ensuremath{\mathbf y}_{t}) ]-  \sum_{t=1}^T  f_{t} (\ensuremath{\mathbf x}^\star) \\
& \leq \sum_{t=1}^T \mathop{\mbox{\bf E}}[ \hat{f}_{t}(\ensuremath{\mathbf x}_{t}) ]- \sum_{t=1}^T  \hat{f}_{t} (\ensuremath{\mathbf x}^\star)  & \mbox{ $\hat{f}_t = f_t$, $\mathop{\mbox{\bf E}}[\ensuremath{\mathbf y}_t] = \ensuremath{\mathbf x}_t $  } \\
& \leq \ensuremath{\mathrm{{Regret}}}_{RFTL}( \ensuremath{\mathbf g_{1}} , ..., \ensuremath{\mathbf g_{T}})   & \mbox{ Lemma \ref{Lemma:Flaxman_FirstOrderAlgos} } \\
& \leq   \sum_{t=1}^T  \ensuremath{\mathbf g_{t}}^\top (\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}})  + \frac{\ensuremath{\mathcal R}(\ensuremath{\mathbf x}^\star) - \ensuremath{\mathcal R}( \ensuremath{\mathbf x}_1)}{\eta}   & \mbox{ Lemma \ref{lem:FTL-BTL}} \\
& \leq   \sum_{t=1}^T  \| \ensuremath{\mathbf g_{t}}\|_{t}^* \| \ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}} \|_{t}   + \frac{\ensuremath{\mathcal R}(\ensuremath{\mathbf x}^\star) - \ensuremath{\mathcal R}( \ensuremath{\mathbf x}_1)}{\eta}  . & \mbox{ Cauchy-Schwarz}
%& \leq  2 \eta \sum_{t=1}^T \| \gv\|_t^{*  \ 2} + \frac{\R(\x^\star) - \R( \x_1)}{\eta}   & \mbox{ Theorem \ref{thm:RFTLmain1}} \\
%& \leq  2 \eta n^2 T  + \frac{\R(\x^\star) - \R( \x_1) }{\eta}   & \mbox {$\|\gv\|^{* \ 2}_t \leq  n^2 $} .
\end{aligned}$$ Here we use our notation from the previous chapter for
the local norm
$\| \ensuremath{\mathbf h}\|_t = \| \ensuremath{\mathbf h}\|_{\ensuremath{\mathbf x}_t} = \sqrt{\ensuremath{\mathbf h}^\top \nabla^2 \ensuremath{\mathcal R}(\ensuremath{\mathbf x}_t) \ensuremath{\mathbf h}}$.

To bound the last expression, we use Lemma
[6.12](#Lemma:DistanceAndGradients){reference-type="ref"
reference="Lemma:DistanceAndGradients"}, and the definition of
$\ensuremath{\mathbf x_{t+1}} = \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \Phi_t(\ensuremath{\mathbf x})$
where
$\Phi_t(\ensuremath{\mathbf x}) =  \eta \sum_{\tau =1 }^t \ensuremath{\mathbf g_{\tau}}^\top \ensuremath{\mathbf x}+  \ensuremath{\mathcal R}(\ensuremath{\mathbf x})$
is a self-concordant barrier. Thus,
$$\| \ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x_{t+1}} \|_{t} \leq  2 \| \nabla \Phi_t(\ensuremath{\mathbf x_{t}}) \|_{t}^{*} = 2 \|  \nabla \Phi_{t-1}(\ensuremath{\mathbf x_{t}}) +  \eta \ensuremath{\mathbf g_{t}}\| _{t}^* = 2 \eta \| \ensuremath{\mathbf g_{t}}\|_{t}^* ,$$
since $\nabla \Phi_{t-1}(\ensuremath{\mathbf x_{t}}) = 0$ by definition
of $\ensuremath{\mathbf x_{t}}$. Recall that to use Lemma
[6.12](#Lemma:DistanceAndGradients){reference-type="ref"
reference="Lemma:DistanceAndGradients"}, we need
$\| \nabla \Phi_t(\ensuremath{\mathbf x_{t}})\|_{t}^* = \eta \| \ensuremath{\mathbf g_{t}}\|_{t} ^* \leq \frac{1}{4}$,
which is true by choice of $\eta$ and since
$$\|\ensuremath{\mathbf g_{t}}\|^{* \ 2}_{t} \leq n^2 \ensuremath{\mathbf u}_t^T \ensuremath{\mathbf A_{t}}^{-T} \nabla^{-2} \ensuremath{\mathcal R}(\ensuremath{\mathbf x}_t) \ensuremath{\mathbf A_{t}}^{-1} \ensuremath{\mathbf u}_t \leq n^2 .$$
Thus, $$\begin{aligned}
& \sum_{t=1}^T \mathop{\mbox{\bf E}}[ f_{t}(\ensuremath{\mathbf y}_{t}) ]-  \sum_{t=1}^T  f_{t} (\ensuremath{\mathbf x}^\star)  \leq  2 \eta \sum_{t=1}^T \| \ensuremath{\mathbf g_{t}}\|_{t}^{*  \ 2} + \frac{\ensuremath{\mathcal R}(\ensuremath{\mathbf x}^\star) - \ensuremath{\mathcal R}( \ensuremath{\mathbf x}_1)}{\eta}   \\
& \leq 2 \eta n^2 T  + \frac{  \ensuremath{\mathcal R}(\ensuremath{\mathbf x}^\star) - \ensuremath{\mathcal R}(\ensuremath{\mathbf x}_1) }{\eta}   .
\end{aligned}$$ It remains to bound the Bregman divergence with respect
to $\ensuremath{\mathbf x}^\star$, for which we use a similar technique
as in the analysis of algorithm
[\[algFKM\]](#algFKM){reference-type="ref" reference="algFKM"}, and
bound the regret with respect to $\ensuremath{\mathbf x}^\star_\delta$,
which is the projection of $\ensuremath{\mathbf x}^\star$ onto
$\ensuremath{\mathcal K}_\delta$. Using equation
[\[Lip_step\]](#Lip_step){reference-type="eqref" reference="Lip_step"},
we can bound the overall regret by: $$\begin{aligned}
& \sum_{t=1}^T \mathop{\mbox{\bf E}}[ f_{t}(\ensuremath{\mathbf y}_{t}) ]-  \sum_{t=1}^T  f_{t} (\ensuremath{\mathbf x}^\star) \\
& \leq  \sum_{t=1}^T \mathop{\mbox{\bf E}}[ f_{t}(\ensuremath{\mathbf y}_{t}) ]-  \sum_{t=1}^T  f_{t} (\ensuremath{\mathbf x}_\delta^*)  +  \delta T G D & \mbox{ equation \eqref{Lip_step} } \\
%& \leq  \eta n T  + \frac{B_\R(\x_1,\x^\star_\delta) }{\eta}   +  \delta T G D  & \mbox { above derivation} \\
%& = 2 \eta n T  + \frac{  \R(\x^\star_\delta) - \R(\x_1) }{\eta}   +  \delta T G D  & \mbox { Since $\nabla(\x_1) = 0$ } \\
& = 2 \eta n^2 T  + \frac{  \ensuremath{\mathcal R}(\ensuremath{\mathbf x}^\star_\delta) - \ensuremath{\mathcal R}(\ensuremath{\mathbf x}_1) }{\eta}   +  \delta T G D  &\mbox { above derivation}  \\
& \leq 2 \eta n^2 T  + \frac{\nu \log \frac{1}{1-\pi_{\ensuremath{\mathbf x}_1}(\ensuremath{\mathbf x}^\star_\delta)} }{\eta}   +  \delta T G D  & \mbox { Lemma \ref{Lemma:MinkowskiBarrier} } \\
& \leq 2 \eta n^2 T  + \frac{\nu \log \frac{1}{\delta }}{\eta}   +  \delta T G D  &  \ensuremath{\mathbf x}^\star_\delta \in \ensuremath{\mathcal K}_\delta  .
\end{aligned}$$ Taking $\eta = O(\frac{1}{\sqrt{T}})$ and
$\delta = O(\frac{1}{T})$, the above bound implies our theorem. ◻
:::

## Bibliographic Remarks {#bibliographic-remarks-3}

The Multi-Armed Bandit problem has history going back more than fifty
years to the work of @Robbins52, see the survey of @BubeckC12 for a much
more detailed history. The non-stochastic MAB problem and the EXP3
algorithm, as well as tight lower bounds were given in the seminal paper
of @AueCesFreSch03nonstochastic. The logarithmic gap in attainable
regret for non-stochastic MAB was resolved in [@AudibertB09].

Bandit Convex Optimization for the special case of linear cost functions
and the flow polytope, was introduced and studied by @AweKle08 in the
context of online routing. The full generality BCO setting was
introduced by @FlaxmanKM05, who gave the first efficient and low-regret
algorithm for BCO. Tight bounds for BCO were obtained by
@bubeck2015bandit for the one dimensional case, via an inefficient
algorithm by @hazan2016optimal, and finally with a polynomial time
algorithm in @bubeck2017kernel.

The special case in which the cost functions are linear, called Bandit
Linear Optimization, received significant attention. @DanHayKak07price
gave an optimal regret algorithm up to constants depending on the
dimension. @AbernethyHR08 gave an efficient algorithm and introduced
self-concordant barriers to the bandit setting. Self-concordant barrier
functions were devised in the context of polynomial-time algorithms for
convex optimization in the seminal work of @NesterovNemirovskii94siam.
Lower bounds for regret in the bandit linear optimization setting were
studied by [@shamir2015complexity].

In this chapter we have considered the expected regret as a performance
metric. Significant literature is devoted to high probability guarantees
on the regret. High probability bounds for the MAB problem were given in
[@AueCesFreSch03nonstochastic], and for bandit linear optimization in
[@AbernethyR09]. Other more refined metrics have been recently explored
in [@DekelTA12] and in the context of adaptive adversaries in
[@NeuGSA14; @YuMa09; @EvenDarKaMa09; @MannorSh03; @YuMaSh09].

For a recent comprehensive text on bandit algorithms see
[@lattimore2020bandit].

## Exercises

# Projection-Free Algorithms {#chap:FW}

In many computational and learning scenarios the main bottleneck of
optimization, both online and offline, is the computation of projections
onto the underlying decision set (see
§[2.1.1](#sec:projections){reference-type="ref"
reference="sec:projections"}). In this chapter we introduce
projection-free methods for online convex optimization, that yield more
efficient algorithms in these scenarios.

The motivating example throughout this chapter is the problem of matrix
completion, which is a widely used and accepted model in the
construction of recommendation systems. For matrix completion and
related problems, projections amount to expensive linear algebraic
operations and avoiding them is crucial in big data applications.

We start with a detour into classical offline convex optimization and
describe the conditional gradient algorithm, also known as the
Frank-Wolfe algorithm. Afterwards, we describe problems for which linear
optimization can be carried out much more efficiently than projections.
We conclude with an OCO algorithm that eschews projections in favor of
linear optimization, in much the same flavor as its offline counterpart.

## Review: Relevant Concepts from Linear Algebra

This chapter addresses rectangular matrices, which model applications
such as recommendation systems naturally. Consider a matrix
$X \in {\mathbb R}^{n \times m}$. A non-negative number
$\sigma \in {\mathbb R}_+$ is said to be a singular value for $X$ if
there are two vectors
$\ensuremath{\mathbf u}\in {\mathbb R}^n, \ensuremath{\mathbf v}\in {\mathbb R}^m$
such that
$$X^\top \ensuremath{\mathbf u}= \sigma \ensuremath{\mathbf v}, \quad X \ensuremath{\mathbf v}= \sigma \ensuremath{\mathbf u}.$$
The vectors $\ensuremath{\mathbf u},\ensuremath{\mathbf v}$ are called
the left and right singular vectors respectively. The non-zero singular
values are the square roots of the eigenvalues of the matrix $X X^\top$
(and $X^\top X$). The matrix $X$ can be written as
$$X = U \Sigma V^\top  \ , \ U \in {\mathbb R}^{n \times \rho} \ , \ V^\top  \in {\mathbb R}^{ \rho \times m} ,$$
where $\rho = \min\{n,m\}$, the matrix $U$ is an orthogonal basis of the
left singular vectors of $X$, the matrix $V$ is an orthogonal basis of
right singular vectors, and $\Sigma$ is a diagonal matrix of singular
values. This form is called the singular value decomposition for $X$.

The number of non-zero singular values for $X$ is called its rank, which
we denote by $k \leq \rho$. The nuclear norm of $X$ is defined as the
$\ell_1$ norm of its singular values, and denoted by
$$\|X \|_* = \sum_{i=1}^\rho \sigma_i .$$ It can be shown (see
exercises) that the nuclear norm is equal to the trace of the square
root of the matrix times its transpose, i.e.,
$$\|X\|_* = {\bf Tr}( \sqrt{ X^\top X}  )$$ We denote by $A \bullet B$
the inner product of two matrices as vectors in
${\mathbb R}^{n \times m}$, that is
$$A \bullet B = \sum_{i = 1}^n \sum_{j=1}^m A_{ij} B_{ij} = {\bf Tr}(AB^\top) .$$

## Motivation: Recommender Systems {#sec:recommendation_systems}

Media recommendations have changed significantly with the advent of the
Internet and rise of online media stores. The large amounts of data
collected allow for efficient clustering and accurate prediction of
users' preferences for a variety of media. A well-known example is the
so called "Netflix challenge"---a competition of automated tools for
recommendation from a large dataset of users' motion picture
preferences.

One of the most successful approaches for automated recommendation
systems, as proven in the Netflix competition, is matrix completion.
Perhaps the simplest version of the problem can be described as follows.

The entire dataset of user-media preference pairs is thought of as a
partially-observed matrix. Thus, every person is represented by a row in
the matrix, and every column represents a media item (movie). For
simplicity, let us think of the observations as binary---a person either
likes or dislikes a particular movie. Thus, we have a matrix
$M \in \{0,1,*\}^{n \times m}$ where $n$ is the number of persons
considered, $m$ is the number of movies at our library, and $0/1$ and
$*$ signify "dislike", "like" and "unknown" respectively: $$M_{ij} = {
\left\{
\begin{array}{ll}
    {0}, & {\mbox{person $i$ dislikes movie $j$}} \\\\
    {1}, & {\mbox{person $i$ likes movie $j$}} \\\\
    {*}, & {\mbox{preference unknown}}
\end{array}
\right. } .$$

The natural goal is to complete the matrix, i.e., correctly assign $0$
or $1$ to the unknown entries. As defined so far, the problem is
ill-posed, since any completion would be equally good (or bad), and no
restrictions have been placed on the completions.

The common restriction on completions is that the "true" matrix has low
rank. Recall that a matrix $X \in {\mathbb R}^{n \times m}$ has rank
$k < \rho = \min \{n,m\}$ if and only if it can be written as
$$X = U V \ , \ U \in {\mathbb R}^{n \times k} , V \in {\mathbb R}^{k \times m}.$$

The intuitive interpretation of this property is that each entry in $M$
can be explained by only $k$ numbers. In matrix completion this means,
intuitively, that there are only $k$ factors that determine a persons
preference over movies, such as genre, director, actors and so on.

Now the simplistic matrix completion problem can be well-formulated as
in the following mathematical program. Denote by $\| \cdot \|_{ob}$ the
Euclidean norm only on the observed (non starred) entries of $M$, i.e.,
$$\|X\|_{ob}^2 = \sum_{M_{ij} \neq *} X_{ij}^2.$$ The mathematical
program for matrix completion is given by $$\begin{aligned}
& \min_{X \in {\mathbb R}^{n \times m} } \frac{1}{2} \| X - M \|_{ob}^2 \\
& \text{s.t.} \quad \mathop{\mbox{\rm rank}}(X) \leq k.
\end{aligned}$$

Since the constraint over the rank of a matrix is non-convex, it is
standard to consider a relaxation that replaces the rank constraint by
the nuclear norm. It is known that the nuclear norm is a lower bound on
the matrix rank if the singular values are bounded by one (see
exercises). Thus, we arrive at the following convex program for matrix
completion: $$\begin{aligned}
 \label{eqn:matrix-completion}
& \min_{X \in {\mathbb R}^{n \times m} }  \frac{1}{2} \| X - M \|_{ob}^2 \\
& \text{s.t.} \quad \|X\|_* \leq k. \notag
\end{aligned}$$

We consider algorithms to solve this convex optimization problem next.

## The Conditional Gradient Method {#subsec:cond_grad_intro}

In this section we return to the basics of convex
optimization---minimization of a convex function over a convex domain as
studied in chapter [2](#chap:opt){reference-type="ref"
reference="chap:opt"}.

The conditional gradient (CG) method, or Frank-Wolfe algorithm, is a
simple algorithm for minimizing a smooth convex function $f$ over a
convex set $\ensuremath{\mathcal K}\subseteq {\mathbb R}^n$. The appeal
of the method is that it is a first order interior point method - the
iterates always lie inside the convex set, and thus no projections are
needed, and the update step on each iteration simply requires to
minimize a linear objective over the set. The basic method is given in
algorithm [\[alg:condgrad\]](#alg:condgrad){reference-type="ref"
reference="alg:condgrad"}.

::: algorithm
::: algorithmic
Input: step sizes $\{ \eta_t \in (0,1] , \ t \in [T]\}$, initial point
$\ensuremath{\mathbf x}_1 \in \ensuremath{\mathcal K}$.
$\ensuremath{\mathbf v}_{t} \gets \arg \min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \left\{\ensuremath{\mathbf x}^\top \nabla{}f(\ensuremath{\mathbf x}_t) \right\}$.
[]{#algstep:linearopt label="algstep:linearopt"}
$\ensuremath{\mathbf x}_{t+1} \gets \ensuremath{\mathbf x}_t + \eta_t(\ensuremath{\mathbf v}_t - \ensuremath{\mathbf x}_t)$.
:::
:::

Note that in the CG method, the update to the iterate
$\ensuremath{\mathbf x}_t$ may be not be in the direction of the
gradient, as $\ensuremath{\mathbf v}_t$ is the result of a linear
optimization procedure in the direction of the negative gradient. This
is depicted in figure [7.1](#fig:OFW){reference-type="ref"
reference="fig:OFW"}.

::: center
![Direction of progression of the CG algorithm
](images/fig_fw2.jpg){#fig:OFW width="3.5in"}
:::

The following theorem gives an essentially tight performance guarantee
of this algorithm over smooth functions. Recall our notation from
chapter [2](#chap:opt){reference-type="ref" reference="chap:opt"}:
$\ensuremath{\mathbf x}^\star$ denotes the global minimizer of $f$ over
$\ensuremath{\mathcal K}$, $D$ denotes the diameter of the set
$\ensuremath{\mathcal K}$, and
$h_t = f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^\star)$
denotes the suboptimality of the objective value in iteration $t$.

::: {#thm:offlineFW .theorem}
**Theorem 7.1**. *The CG algorithm applied to $\beta$-smooth functions
with step sizes $\eta_t =  \min\{1,\frac{2}{t}\}$ attains the following
convergence guarantee $$h_t \leq \frac{2 \beta  D^2 }{t}$$*
:::

::: proof
*Proof.* As done before in this manuscript, we denote
$\nabla_t = \nabla f(\ensuremath{\mathbf x}_t)$. For any set of step
sizes, we have $$\begin{aligned}
\label{old_fw_anal}
	&  f(\ensuremath{\mathbf x}_{t+1}) - f(\ensuremath{\mathbf x}^\star)
	= f(\ensuremath{\mathbf x}_t + \eta_t(\ensuremath{\mathbf v}_t - \ensuremath{\mathbf x}_t)) - f(\ensuremath{\mathbf x}^\star) \notag \\
	&\leq  f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^\star) + \eta_t(\ensuremath{\mathbf v}_t-\ensuremath{\mathbf x}_t)^{\top}\nabla_t + \eta_t^2 \frac{\beta}{2}\Vert{\ensuremath{\mathbf v}_t-\ensuremath{\mathbf x}_t}\Vert^2 &  \textrm{smoothness } \nonumber \\
	&\leq  f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^\star) + \eta_t(\ensuremath{\mathbf x}^\star-\ensuremath{\mathbf x}_t)^{\top}\nabla_t + \eta_t^2 \frac{\beta}{2}\Vert{\ensuremath{\mathbf v}_t-\ensuremath{\mathbf x}_t}\Vert^2 &  \textrm{$\ensuremath{\mathbf v}_t$ choice} \nonumber \\
	&\leq  f(\ensuremath{\mathbf x}_t) - f(\ensuremath{\mathbf x}^\star) + \eta_t(f(\ensuremath{\mathbf x}^\star)-f(\ensuremath{\mathbf x}_t)) + \eta_t^2 \frac{\beta}{2}\Vert{\ensuremath{\mathbf v}_t-\ensuremath{\mathbf x}_t}\Vert^2 & \textrm{convexity} \nonumber \\
	&\leq  (1-\eta_t)(f(\ensuremath{\mathbf x}_t)-f(\ensuremath{\mathbf x}^\star)) + \frac{\eta_t^2\beta}{2} D^2.
\end{aligned}$$ We reached the recursion
$h_{t+1} \leq (1- \eta_t) h_t + \eta_t^2\frac{  \beta D^2}{2}$, and by
Lemma [7.2](#lemma:FW-recursion){reference-type="ref"
reference="lemma:FW-recursion"} we obtain,
$$h_{t} \leq \frac{2 \beta D^2 }{t} .$$ ◻
:::

::: {#lemma:FW-recursion .lemma}
**Lemma 7.2**. *Let $\{ h_t \}$ be a sequence that satisfies the
recurence $$h_{t+1} \leq h_t (1 - \eta_t) + \eta_t^2 c .$$ Then taking
$\eta_t = \min\{1,\frac{2}{t}\}$ implies $$h_t \leq \frac{4c}{t} .$$*
:::

::: proof
*Proof.* This is proved by induction on $t$.

**Induction base.** For $t=1$, we have
$$h_2 \leq h_1 (1-\eta_1 ) + \eta_1^2 c = c \leq 4c .$$

**Induction step.** $$\begin{aligned}
 h_{t+1} & \leq (1- \eta_t) h_t + \eta_t^2 c  \\
 & \leq \left(1- \frac{2}{t} \right) \frac{4c}{ t} + \frac{4c}{t^2}  & \mbox{induction hypothesis}\\
 & = \frac{4c}{t} \left( 1 - \frac{1}{t} \right) \\
 & \leq  \frac{4c}{t} \cdot  \frac{t}{t+1} & \mbox{$\frac{t-1}{t} \leq \frac{t}{t+1} $ }   \\
 & = \frac{4c}{t+1} .

\end{aligned}$$ ◻
:::

### Example: matrix completion via CG

As an example of an application for the conditional gradient algorithm,
recall the mathematical program given by
[\[eqn:matrix-completion\]](#eqn:matrix-completion){reference-type="eqref"
reference="eqn:matrix-completion"}. The gradient of the objective
function at point $X^t$ is $$\label{eqn:matrix-gradient}
\nabla f(X^t) = (X^t - M)_{ob}  = {
\left\{
\begin{array}{ll}
    { X_{ij}^t - M_{ij} }, & {(i,j) \in OB} \\\\
    {0}, & {\text{otherwise}}
\end{array}
\right. } .$$ Over the set of bounded-nuclear norm matrices, the linear
optimization of line
[\[algstep:linearopt\]](#algstep:linearopt){reference-type="ref"
reference="algstep:linearopt"} in algorithm
[\[alg:condgrad\]](#alg:condgrad){reference-type="ref"
reference="alg:condgrad"} becomes, $$\begin{aligned}
&   \min     X \bullet \nabla_t \ , \quad \nabla_t = \nabla f(X_t)    \\
& \mbox{s.t.  }  \|X\|_* \leq k.
\end{aligned}$$ For simplicity, let's consider square symmetric
matrices, for which the nuclear norm is equivalent to the trace norm,
and the above optimization problem becomes $$\begin{aligned}
& \min  X \bullet \nabla_t  \\
& \mbox{s.t.  }   {\bf Tr}(X)  \leq k.
\end{aligned}$$ It can be shown that this program is equivalent to the
following (see exercises): $$\begin{aligned}
& \min_{\ensuremath{\mathbf x}\in {\mathbb R}^n}  \ensuremath{\mathbf x}^\top \nabla_t \ensuremath{\mathbf x}\\
& \mbox{s.t.  }   \|\ensuremath{\mathbf x}\|_2^2  \leq k.
\end{aligned}$$ Hence, this is an eigenvector computation in disguise!
Computing the largest eigenvector of a matrix takes linear time via the
power method, which also applies more generally to computing the largest
singular value of rectangular matrices. With this, step
[\[algstep:linearopt\]](#algstep:linearopt){reference-type="ref"
reference="algstep:linearopt"} of algorithm
[\[alg:condgrad\]](#alg:condgrad){reference-type="ref"
reference="alg:condgrad"}, which amounts to mathematical program
[\[eqn:matrix-completion\]](#eqn:matrix-completion){reference-type="eqref"
reference="eqn:matrix-completion"}, becomes computing
$v_{\max}(- \nabla f(X^t))$, the largest eigenvector of
$- \nabla f(X^t)$. Algorithm
[\[alg:condgrad\]](#alg:condgrad){reference-type="ref"
reference="alg:condgrad"} takes on the modified form described in
Algorithm
[\[alg:condgrad4matrixcompletion\]](#alg:condgrad4matrixcompletion){reference-type="ref"
reference="alg:condgrad4matrixcompletion"}.

::: algorithm
::: algorithmic
Let $X^1$ be an arbitrary matrix of trace $k$ in
$\ensuremath{\mathcal K}$.
$\ensuremath{\mathbf v}_{t} = \sqrt{k} \cdot v_{\max}(-\nabla_t )$.
$X^{t+1} = X^t + \eta_t(\ensuremath{\mathbf v}_t  \ensuremath{\mathbf v}_t^\top - X^t)$
for $\eta_t\in(0,1)$.
:::
:::

##### Comparison to other gradient-based methods. {#comparison-to-other-gradient-based-methods. .unnumbered}

How does this compare to previous convex optimization methods for
solving the same matrix completion problem? As a convex program, we can
apply gradient descent, or even more advantageously in this setting,
stochastic gradient descent as in §[3.4](#sec:sgd){reference-type="ref"
reference="sec:sgd"}. Recall that the gradient of the objective function
at point $X^t$ takes the simple form
[\[eqn:matrix-gradient\]](#eqn:matrix-gradient){reference-type="eqref"
reference="eqn:matrix-gradient"}. A stochastic estimate for the gradient
can be attained by observing just a single entry of the matrix $M$, and
the update itself takes constant time as the gradient estimator is
sparse. However, the projection step is significantly more difficult.

In this setting, the convex set $\ensuremath{\mathcal K}$ is the set of
bounded nuclear norm matrices. Projecting a matrix onto this set amounts
to calculating the SVD of the matrix, which is similar in computational
complexity to algorithms for matrix diagonalization or inversion. The
best known algorithms for matrix diagonalization are superlinear in the
matrices' size, and thus impractical for large datasets that are common
in applications.

In contrast, the CG method does not require projections at all, and
replaces them with linear optimization steps over the convex set, which
we have observed to amount to singular vector computations. The latter
can be implemented to take linear time via the power method or the
Lanczos algorithm (see bibliography).

Thus, the Conditional Gradient method allows for optimization of the
mathematical program
[\[eqn:matrix-completion\]](#eqn:matrix-completion){reference-type="eqref"
reference="eqn:matrix-completion"} with a linear-time operation
(eigenvector using power method) per iteration, rather than a
significantly more expensive computation (SVD) needed for gradient
descent.

## Projections versus Linear Optimization

The conditional gradient (Frank-Wolfe) algorithm described before does
not resort to projections, but rather computes a linear optimization
problem of the form $$\label{eqn:linopt}
 \arg \min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \left\{\ensuremath{\mathbf x}^\top \ensuremath{\mathbf u}\right\}.$$
When is the CG method computationally preferable? The overall
computational complexity of an iterative optimization algorithm is the
product of the number of iterations and the computational cost per
iteration. The CG method does not converge as well as the most efficient
gradient descent algorithms, meaning it requires more iterations to
produce a solution of a comparable level of accuracy. However, for many
interesting scenarios the computational cost of a linear optimization
step [\[eqn:linopt\]](#eqn:linopt){reference-type="eqref"
reference="eqn:linopt"} is *significantly* lower than that of a
projection step.

Let us point out several examples of problems for which we have very
efficient linear optimization algorithms, whereas our state-of-the-art
algorithms for computing projections are significantly slower.

##### Recommendation systems and matrix prediction. {#recommendation-systems-and-matrix-prediction. .unnumbered}

In the example pointed out in the preceding section of matrix
completion, known methods for projection onto the spectahedron, or more
generally the bounded nuclear-norm ball, require singular value
decompositions, which take superlinear time via our best known methods.
In contrast, the CG method requires maximal eigenvector computations
which can be carried out in linear time via the power method (or the
more sophisticated Lanczos algorithm).

##### Network routing and convex graph problems. {#network-routing-and-convex-graph-problems. .unnumbered}

Various routing and graph problems can be modeled as convex optimization
problems over a convex set called the flow polytope.

Consider a directed acyclic graph with $m$ edges, a source node marked
$s$ and a target node marked $t$. Every path from $s$ to $t$ in the
graph can be represented by its identifying vector, that is a vector in
$\lbrace{0,1}\rbrace^m$ in which the entries that are set to 1
correspond to edges of the path. The flow polytope of the graph is the
convex hull of all such identifying vectors of the simple paths from $s$
to $t$. This polytope is also exactly the set of all unit $s$--$t$ flows
in the graph if we assume that each edge has a unit flow capacity (a
flow is represented here as a vector in $\mathbb{R}^m$ in which each
entry is the amount of flow through the corresponding edge).

Since the flow polytope is just the convex hull of $s$--$t$ paths in the
graph, minimizing a linear objective over it amounts to finding a
minimum weight path given weights for the edges. For the shortest path
problem we have very efficient combinatorial optimization algorithms,
namely Dijkstra's algorithm.

Thus, applying the CG algorithm to solve **any** convex optimization
problem over the flow polytope will only require iterative shortest path
computations.

##### Ranking and permutations.  {#ranking-and-permutations. .unnumbered}

A common way to represent a permutation or ordering is by a permutation
matrix. Such are square matrices over $\{0,1\}^{n \times n}$ that
contain exactly one $1$ entry in each row and column.

Doubly-stochastic matrices are square, real-valued matrices with
non-negative entries, in which the sum of entries of each row and each
column amounts to 1. The polytope that defines all doubly-stochastic
matrices is called the Birkhoff-von Neumann polytope. The Birkhoff-von
Neumann theorem states that this polytope is the convex hull of exactly
all $n\times{n}$ permutation matrices.

Since a permutation matrix corresponds to a perfect matching in a fully
connected bipartite graph, linear minimization over this polytope
corresponds to finding a minimum weight perfect matching in a bipartite
graph.

Consider a convex optimization problem over the Birkhoff-von Neumann
polytope. The CG algorithm will iteratively solve a linear optimization
problem over the BVN polytope, thus iteratively solving a minimum weight
perfect matching in a bipartite graph problem, which is a well-studied
combinatorial optimization problem for which we know of efficient
algorithms. In contrast, other gradient based methods will require
projections, which are quadratic optimization problems over the BVN
polytope.

##### Matroid polytopes. {#matroid-polytopes. .unnumbered}

A matroid is pair $(E,I)$ where $E$ is a set of elements and $I$ is a
set of subsets of $E$ called the independent sets which satisfy various
interesting proprieties that resemble the concept of linear independence
in vector spaces. Matroids have been studied extensively in
combinatorial optimization and a key example of a matroid is the
graphical matroid in which the set $E$ is the set of edges of a given
graph and the set $I$ is the set of all subsets of $E$ which are
cycle-free. In this case, $I$ contains all the spanning trees of the
graph. A subset $S\in{I}$ could be represented by its identifying vector
which lies in $\lbrace{0,1}\rbrace^{\vert{E}\vert}$ which also gives
rise to the matroid polytope which is just the convex hull of all
identifying vectors of sets in $I$. It can be shown that some matroid
polytopes are defined by exponentially many linear inequalities
(exponential in $\vert{E}\vert$), which makes optimization over them
difficult.

On the other hand, linear optimization over matroid polytopes is easy
using a simple greedy procedure which runs in nearly linear time. Thus,
the CG method serves as an efficient algorithm to solve any convex
optimization problem over matroids iteratively using only a simple
greedy procedure.

## The Online Conditional Gradient Algorithm

In this section we give a projection-free algorithm for OCO based on the
conditional gradient method, which is projection-free and thus carries
the computational advantages of the CG method to the online setting.

It is tempting to apply the CG method straightforwardly to the online
appearance of functions in the OCO setting, such as the OGD algorithm in
§[3.1](#section:ogd){reference-type="ref" reference="section:ogd"}.
However, it can be shown that an approach that only takes into account
the last cost function is doomed to fail. The reason is that the
conditional gradient method takes into account the *direction* of the
gradient, and is insensitive to its *magnitude*.

Instead, we apply the CG algorithm step to the aggregate sum of all
previous cost functions with added Euclidean regularization. The
resulting algorithm is given formally in Algorithm
[\[alg:ocg\]](#alg:ocg){reference-type="ref" reference="alg:ocg"}.

::: algorithm
::: algorithmic
Input: convex set $\ensuremath{\mathcal K}$, $T$,
$\ensuremath{\mathbf x}_1 \in \mathcal{K}$, parameters
$\eta  , \ \{\sigma_t\}$. Play $\mathbf{x}_t$ and observe $f_t$.
[]{#eq:F_t-def label="eq:F_t-def"} Let
$F_t(\ensuremath{\mathbf x}) = \eta {\textstyle \sum}_{\tau=1}^{t-1} \nabla_\tau^\top \ensuremath{\mathbf x}+  \|\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_1\|^2$.
Compute
$\mathbf{v}_t = \arg \min_{\mathbf{x}\in \ensuremath{\mathcal K}} \{\nabla F_t(\mathbf{x}_t) \cdot \mathbf{x}\}$.
Set
$\mathbf{x}_{t+1} = (1 - \sigma_t)\mathbf{x}_{t} + \sigma_t \mathbf{v}_t$.
:::
:::

We can prove the following regret bound for this algorithm. While this
regret bound is suboptimal in light of the previous upper bounds we have
seen, its suboptimality is compensated by the algorithm's lower
computational cost.

::: {#thm:FWonline-main .theorem}
**Theorem 7.3**. *Online conditional gradient (Algorithm
[\[alg:ocg\]](#alg:ocg){reference-type="ref" reference="alg:ocg"}) with
parameters
$\eta = \frac{D}{2 G T^{3/4} }, \sigma_t = \min\{1,\frac{2}{t^{1/2}}\}$,
attains the following guarantee
$$\ensuremath{\mathrm{{Regret}}}_T = \sum_{t=1}^{T} f_t(\mathbf{x}_t) -\min_{\mathbf{x}^\star \in \ensuremath{\mathcal K}}\sum_{t=1}^{T}
f_t(\mathbf{x}^\star)\ \leq  8 D G T^{3/4}$$*
:::

As a first step in analyzing Algorithm
[\[alg:ocg\]](#alg:ocg){reference-type="ref" reference="alg:ocg"},
consider the points
$$\ensuremath{\mathbf x_{t}}^\star = \arg \min _{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} F_t(\ensuremath{\mathbf x}) .$$
These are exactly the iterates of the RFTL algorithm from chapter
[5](#chap:regularization){reference-type="ref"
reference="chap:regularization"}, namely Algorithm
[\[alg:RFTLmain\]](#alg:RFTLmain){reference-type="ref"
reference="alg:RFTLmain"} with the regularization being
$R(\ensuremath{\mathbf x}) = \| \ensuremath{\mathbf x}- \ensuremath{\mathbf x_{1}}\|^2$,
applied to cost functions with a shift, namely:
$$\tilde{f}_t = f_t( \ensuremath{\mathbf x}+ (\ensuremath{\mathbf x}_t^\star - \ensuremath{\mathbf x}_t) ) .$$
The reason is that $\nabla_t$ in Algorithm
[\[alg:ocg\]](#alg:ocg){reference-type="ref" reference="alg:ocg"} refers
to $\nabla f_t(\ensuremath{\mathbf x}_t)$, whereas in the RFTL algorithm
we have $\nabla_t = \nabla f_t(\ensuremath{\mathbf x}_t^\star)$. Notice
that for any point $\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}$
we have
$| f_t(\ensuremath{\mathbf x}) - \tilde{f}_t(\ensuremath{\mathbf x}) | \leq G \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}_t^\star\|$.
Thus, according to Theorem [5.2](#thm:RFTLmain1){reference-type="ref"
reference="thm:RFTLmain1"}, we have that $$\begin{aligned}
 \label{eqn:FW1}
 & \sum_{t=1}^{T} f_t(\ensuremath{\mathbf x}_t^\star) - \sum_{t=1}^{T} f_t(\mathbf{x}^\star) \notag \\
 & \leq 2 G\sum_t  \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}_t^\star\| +    \sum_{t=1}^{T} \tilde{f}_t(\ensuremath{\mathbf x}_t^\star) - \sum_{t=1}^{T} \tilde{f}_t(\mathbf{x}^\star )  \notag \\
 & \leq  2 G\sum_t  \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}_t^\star\| +   2 \eta G T + \frac{1}{\eta} D .
\end{aligned}$$

Using our previous notation, denote by
$h_t(\ensuremath{\mathbf x}) = {F_t(\ensuremath{\mathbf x}) - F_t(\ensuremath{\mathbf x}^\star_t)}$,
and by $h_t = h_t(\ensuremath{\mathbf x}_t)$. The main lemma we require
to proceed is the following, which relates the iterates
$\ensuremath{\mathbf x}_t$ to the optimal point according to the
aggregate function $F_t$.

::: {#lem:mainFW .lemma}
**Lemma 7.4**. *The iterates $\ensuremath{\mathbf x}_t$ of
Algorithm [\[alg:ocg\]](#alg:ocg){reference-type="ref"
reference="alg:ocg"} satisfy for all $t \ge 1$
$$h_t \leq  {  2 D^2} \sigma_t.$$*
:::

::: proof
*Proof.* As the functions $F_t$ are $1$-smooth, applying the offline
Frank-Wolfe analysis technique, and in particular Equation
[\[old_fw_anal\]](#old_fw_anal){reference-type="eqref"
reference="old_fw_anal"} to the function $F_t$ we obtain:
$$\begin{aligned}
 h_{t}(\ensuremath{\mathbf x}_{t+1}) & = F_{t} (\ensuremath{\mathbf x}_{t+1}) -  F_{t} (\ensuremath{\mathbf x}^\star_{t}) \\
&\leq  (1-\sigma_t)( F_t (\ensuremath{\mathbf x}_t)- F_t (\ensuremath{\mathbf x}^\star_t)) + \frac{D^2}{2} \sigma_t^2 & \mbox { Equation \eqref{old_fw_anal}} \\
& =  (1-\sigma_t)  h_t + \frac{D^2}{2} \sigma_t^2.
\end{aligned}$$

In addition, by definition of $F_t$ and $h_t$ we have $$\begin{aligned}
& h_{t+1}  \\
& = F_t(\ensuremath{\mathbf x}_{t+1})  - F_t(\ensuremath{\mathbf x}_{t+1}^\star) + \eta \nabla_{t+1}(\ensuremath{\mathbf x}_{t+1} - \ensuremath{\mathbf x}^\star_{t+1} ) \\
& \leq h_t(\ensuremath{\mathbf x}_{t+1})  + \eta \nabla_{t+1}(\ensuremath{\mathbf x}_{t+1} - \ensuremath{\mathbf x}^\star_{t+1}) & \mbox{$F_t(\ensuremath{\mathbf x}_t^\star) \leq F_t(\ensuremath{\mathbf x}_{t+1}^\star)$} \\
 & \leq\ h_t(\ensuremath{\mathbf x}_{t+1}) + \eta G \| \ensuremath{\mathbf x}_{t+1} - \ensuremath{\mathbf x}_{t+1}^\star\| .  & \mbox{Cauchy-Schwarz}
\end{aligned}$$ Since $F_t$ is $1$-strongly convex, we have
$$\| \ensuremath{\mathbf x}- \ensuremath{\mathbf x_{t}}^\star \|^2 \leq F_t(\ensuremath{\mathbf x}) - F_t(\ensuremath{\mathbf x_{t}}^\star)  .$$
Thus, $$\begin{aligned}
h_{t+1} & \leq\ h_t(\ensuremath{\mathbf x}_{t+1}) + \eta G \| \ensuremath{\mathbf x}_{t+1} - \ensuremath{\mathbf x}_{t+1}^\star\| \\
& \leq h_t(\ensuremath{\mathbf x}_{t+1}) + \eta G \sqrt{h_{t+1}}  \\
%& \leq h_t(\x_{t+1}) + \eta^{4/3} G^2 + \eta^{2/3} \|\x_{t+1} - \x_{t+1}^\star\|^2 \\
%& \leq h_t(\x_{t+1}) + \eta^{4/3} G^2 + \eta^{2/3} h_{t+1}  \\
%& \leq h_t(\x_{t+1}) +  G^2 \sigma_t^2 + \sigma_t h_{t+1}
 &  \leq h_t (1 - \sigma_t)   + \frac{1}{2} {D^2 } \sigma_t^2 + \eta G \sqrt{h_{t+1}}  & \mbox{above derivation} \\
& \leq h_t (1 - \frac{5}{6} \sigma_t)   + \frac{5}{8}{D^2 } \sigma_t^2. & \mbox{ equation \eqref{prop:fwhelperprop} below}
\end{aligned}$$

Above we used the following derivation, that holds by choice of
parameters $\eta = \frac{D}{2 G T^{3/4} }$ and
$\sigma_t = \min\{1,\frac{2}{t^{1/2}}\}$: since $\eta,G,h_t$ are all
non-negative, we have $$\begin{aligned}
\eta G \sqrt{h_{t+1}} &  = \left(  \sqrt{D} {G \eta} \right)^{2/3} \left( \frac{G  \eta}{D}  \right)^{1/3} \sqrt{h_{t+1}} \notag \\
& \leq \frac{1}{2} \left( { \sqrt{D} G \eta}{} \right)^{4/3} + \frac{1}{2}  \left( \frac{G \eta}{D}  \right)^{2/3} h_{t+1}  \notag \\
& \leq \frac{1}{8} D^2  \sigma_t^2 + \frac{1}{6}  \sigma_t h_{t+1} \label{prop:fwhelperprop}
\end{aligned}$$

We now claim that the theorem follows inductively. The base of the
induction holds since, for $t = 1$, the definition of $F_1$ implies
$$h_1 = F_1(\ensuremath{\mathbf x}_1) - F_1(\ensuremath{\mathbf x}^\star) = \| \ensuremath{\mathbf x}_1 - \ensuremath{\mathbf x}^\star\|^2 \leq  D^2 \leq  2 D^2 \sigma_1  .$$

Assuming the bound is true for $t$, we now show it holds for $t+1$ as
well: $$\begin{aligned}
h_{t+1} & \leq & h_t (1 - \frac{5}{6} \sigma_t)   + \frac{5}{8} {D^2 } \sigma_t^2 \\
& \leq &   2 D^2 \sigma_t  \left(1 -  \frac{5}{6} \sigma_t \right) + \frac{5}{8} { D^2}\sigma_t^2  \\
& \leq &  { 2 D^2 }\sigma_t  \left(1 -  \frac{\sigma_t}{2} \right) \\
& \le & {  2 D^2} \sigma_{t+1},
\end{aligned}$$ as required. The last inequality follows by the
definition of $\sigma_t$ (see exercises). ◻
:::

We proceed to use this lemma in order to prove our theorem:

::: proof
*Proof of Theorem [7.3](#thm:FWonline-main){reference-type="ref"
reference="thm:FWonline-main"}.* By definition, the functions $F_t$ are
$1$-strongly convex. Thus, we have for
$\ensuremath{\mathbf x_{t}}^\star = \arg \min_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} F_t(\ensuremath{\mathbf x})$:
$$\| \ensuremath{\mathbf x}- \ensuremath{\mathbf x_{t}}^\star \|^2 \leq F_t(\ensuremath{\mathbf x}) - F_t(\ensuremath{\mathbf x_{t}}^\star)  .$$
Let $\eta = \frac{D}{2G T^{3/4}}$, and notice that this satisfies the
constraint of Lemma [7.4](#lem:mainFW){reference-type="ref"
reference="lem:mainFW"}, which requires
$\eta G \sqrt{h_{t+1}}  \leq \frac{D^2}{2} \sigma_t^2$. In addition,
$\eta < 1$ for $T$ large enough. Hence, $$\begin{aligned}
f_t(\ensuremath{\mathbf x_{t}}) - f_t(\ensuremath{\mathbf x}^\star_t) & \leq G \| \ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x}^\star_t \|  \notag \\
& \leq {G} \sqrt{  F_t(\ensuremath{\mathbf x_{t}}) - F_t(\ensuremath{\mathbf x_{t}}^\star) } \notag \\
& \leq { 2 G D } \sqrt{\sigma_t} .  & \mbox{ Lemma \ref{lem:mainFW} }  \label{eqn:FW2}
\end{aligned}$$ Putting everything together we obtain:

$$\begin{aligned}
& \text{\em Regret}_T(\text{\em OCG})  = \sum_{t=1}^{T} f_t(\mathbf{x}_t) - \sum_{t=1}^{T}
f_t(\mathbf{x}^\star) \\
& = \sum_{t=1}^T \left[ f_t(\mathbf{x}_t) - f_t(\ensuremath{\mathbf x_{t}}^\star) + f_t(\ensuremath{\mathbf x_{t}}^\star) - f_t(\ensuremath{\mathbf x}^\star) \right] \\
& \leq  \sum_{t=1}^{T} 2 G {D} \sqrt{\sigma_t}  + \sum_t \left[  f_t(\ensuremath{\mathbf x_{t}}^\star) - f_t(\ensuremath{\mathbf x}^\star) \right] & \mbox{by \eqref{eqn:FW2}} \\
& \leq 4 G D {T}^{3/4} + \sum_t \left[  f_t(\ensuremath{\mathbf x_{t}}^\star) - f_t(\ensuremath{\mathbf x}^\star) \right]   \\
& \leq 4 G  D {T}^{3/4} + 2 G\sum_t  \|\ensuremath{\mathbf x}_t - \ensuremath{\mathbf x}_t^\star \| +   2 \eta G T + \frac{1}{\eta} D .  & \mbox{by \eqref{eqn:FW1}}   \\
 %2 \eta G^2 T  + \frac{D^2}{\eta}.  & \mbox { \eqref{eqn:FW1}}
\end{aligned}$$

We thus obtain: $$\begin{aligned}
	\ensuremath{\mathrm{{Regret}}}_T(\text{\em OCG})
	& \leq 4 G D {T}^{3/4} + 2 \eta G^2 T  + \frac{D^2}{\eta}  \\
	& \leq 4 G D T^{2/3} + DG T^{1/4} + 2 DG T^{3/4} \leq 8 D G T^{3/4}.
\end{aligned}$$ ◻
:::

## Bibliographic Remarks {#bibliographic-remarks-4}

The matrix completion model has been extremely popular since its
inception in the context of recommendation systems
[@SrebroThesis; @Rennie:2005; @salakhutdinov:collaborative; @lee:practical; @CandesR09; @ShamirS11].

The conditional gradient algorithm was devised in the seminal paper by
@FrankWolfe. Due to the applicability of the FW algorithm to large-scale
constrained problems, it has been a method of choice in recent machine
learning applications, to name a few:
[@Hazan08; @Jaggi10; @Jaggi13a; @Jaggi13b; @Dudik12a; @Dudik12b; @Hazan12; @ShalevShwartz11; @Bach12; @Tewari11; @Garber11; @Garber13; @Florina14].
In the context of matrix completion and recommendation systems, several
faster variants of the Frank-Wolfe method were proposed
[@garber2016faster; @allen2017linear]

The online conditional gradient algorithm is due to @Hazan12. An optimal
regret algorithm, attaining the $O(\sqrt{T})$ bound, for the special
case of polyhedral sets was devised in [@Garber13].

Recent works consider accelerating projection-free optimization using
variance reduction [@lan2016conditional; @hazan2016variance], and the
case of projection-free algorithms with stochastic gradient oracles
[@mokhtari2018stochastic; @chen2018projection; @xie2019stochastic].

For an analysis of the running time of the power and Lanczos methods for
computing eigenvectors see [@kuczynski1992estimating]. For modern
algorithms for fast computation of the singular value decomposition see
[@allen2016lazysvd; @musco2015randomized].

## Exercises

# Games, Duality, and Regret {#chap:games}

In this chapter we tie the material covered thus far to some of the most
intriguing concepts in optimization and game theory. We shall use the
existence of online convex optimization algorithms with sublinear regret
to prove two fundamental properties: convex duality in mathematical
optimization, and von Neumann's minimax theorem in game theory.

Historically, the theory of games was developed by von Neumann in the
early 1930's. In an entirely different scientific thread, the theory of
linear programming (LP) was advanced by Dantzig a decade later. Dantzig
describes in his memoir a notable meeting between himself and von
Neumann at Princeton in 1947. In this meeting, according to Dantzig,
after describing the geometric and algebraic versions of linear
programming, von Neumann essentially formulated and proved linear
programming duality:

> "I don't want you to think I am pulling all this out of my sleeve at
> the spur of the moment like a magician. I have just recently completed
> a book with Oscar Morgenstern on the theory of games. What I am doing
> is conjecturing that the two problems are equivalent. The theory that
> I am outlining for your problem is an analogue to the one we have
> developed for games.\"

At that time, the topic of discussion was not the existence and
uniqueness of equilibrium in zero-sum games, which is captured by the
minimax theorem. Both concepts were originally captured and proved using
very different mathematical techniques: the minimax theorem was
originally proved using machinery from mathematical topology, whereas
linear programming duality was shown using convexity and geometric
tools.

More than half a century later, Freund and Schapire tied both concepts,
which were by then known to be strongly related, to regret minimization.
We shall follow their lead in this chapter, introduce the relevant
concepts and give concise proofs using the machinery developed earlier
in this manuscript.

The chapter can be read with basic familiarity with linear programming
and little or no background in game theory. We define linear programming
and zero-sum games succinctly, barely enough to prove the duality
theorem and the minimax theorem. The reader is referred to the numerous
wonderful texts available on linear programming and game theory for a
much more thorough introduction and definitions.

## Linear Programming and Duality

Linear programming is a widely successful and practical convex
optimization framework. Amongst its numerous successes is the Nobel
prize award given on account of its application to economics. It is a
special case of the convex optimization problem from chapter
[2](#chap:opt){reference-type="ref" reference="chap:opt"} in which
$\ensuremath{\mathcal K}$ is a polyhedron (i.e., an intersection of a
finite set of halfspaces) and the objective function is a linear
function. Thus, a linear program can be described as follows, where
$(A \in \mathbb{R}^{n\times m})$: $$\begin{aligned}
 \min  \quad &  c^{\top} \ensuremath{\mathbf x}&      \\
 \text{s.t.} \quad &  A \ensuremath{\mathbf x}\geq b  & .
\end{aligned}$$ The above formulation can be transformed into several
different forms via basic manipulations. For example, any LP can be
transformed to an equivalent LP with the variables taking only
non-negative values. This can be accomplished by writing every variable
$x$ as $x = x^{+}  - x^{-}$, with $x^{+}, x^{-} \geq 0$. It can be
verified that this transformation leaves us with another LP, whose
variables are non-negative, and contains at most twice as many variables
(see exercises section for more details).

We are now ready to define a central notion in LP and state the duality
theorem:

::: theorem
**Theorem 8.1** (The duality theorem). *Given a linear program:
$$\begin{aligned}
 \min \quad & & c^{\top} \ensuremath{\mathbf x}\\
 \text{s.t.} \quad & & A \ensuremath{\mathbf x}\geq b , \\
 & & \ensuremath{\mathbf x}\geq 0 ,
\end{aligned}$$ its dual program is given by: $$\begin{aligned}
 \max \quad & & b^\top \ensuremath{\mathbf y}\\
 \text{s.t.} \quad & & A^{\top} \ensuremath{\mathbf y}\leq c, \\
   & & \ensuremath{\mathbf y}\geq 0 .
\end{aligned}$$ and the objectives of both problems are either equal or
unbounded.*
:::

Instead of studying duality directly, we proceed to define zero-sum
games and an analogous concept to duality.

## Zero-sum Games and Equilibria

The theory of games is an established research field in economic theory.
We give here brief definitions of the main concepts studied in this
chapter.

Let us start with an example of a zero-sum game we all know: the
rock-paper-scissors game. In this game each of the two players chooses a
strategy: either rock, scissors or paper. The winner is determined
according to the following table, where $0$ denotes a draw, $-1$ denotes
that the row player wins, and $1$ denotes a column player victory.

The rock-paper-scissors game is called a "zero-sum" game since one can
think of the numbers as losses for the row player (loss of $-1$
resembles victory, $1$ loss and $0$ draw), in which case the column
player receives a loss which is exactly the negation of the loss of the
row player. Thus the sum of losses which both players suffer is zero in
every outcome of the game.

Noticed that we termed one player as the "row player" and the other as
the "column player" corresponding to the matrix losses. Such a matrix
representation is far more general:

::: {#defn:zsg .definition}
**Definition 8.2**. *A two-player zero-sum-game in normal form is given
by a matrix $A \in [-1,1]^{n \times m}$. The loss for the row player
playing strategy $i \in [n]$ is equal to the negative loss (reward) of
the column player playing strategy $j \in [m]$ and equal to $A_{ij}$.*
:::

The fact that the losses were defined in the range $[-1,1]$ is
arbitrary, as the concept of main importance we define next is invariant
to scaling and shifting by a constant.

A central concept in game theory is equilibrium. There are many
different notions of equilibria. In two-player zero-sum games, a pure
equilibrium is a pair of strategies $(i,j) \in [n] \times [m]$ with the
following property: given that the column player plays $j$, there is no
strategy that dominates $i$ - i.e., every other strategy $k \in [n]$
gives higher or equal loss to the row player. Equilibrium also requires
that a symmetric property for strategy $j$ holds - it is not dominated
by any other strategy given that the row player plays $i$.

It can be shown that some games do not have a pure equilibrium as
defined above, e.g., the rock-paper-scissors game. However, we can
extend the notion of a strategy to a *mixed* strategy - a distribution
over pure strategies. The loss of a mixed strategy is the expected loss
according to the distribution over pure strategies. More formally, if
the row player chooses $\ensuremath{\mathbf x}\in \Delta_n$ and column
player chooses $\ensuremath{\mathbf y}\in \Delta_m,$ then the expected
loss of the row player (which is the negative reward to the column
player) is given by:

$$\textbf{E}[\text{loss}] = \sum_{i \in [n]}{\ensuremath{\mathbf x}_i \sum_{j \in [m]}{\ensuremath{\mathbf y}_j A_{ij}}} = \ensuremath{\mathbf x}^{\top} A \ensuremath{\mathbf y}.$$

We can now generalize the notion of equilibrium to mixed strategies.
Given a row strategy $\ensuremath{\mathbf x}$, it is dominated by
$\tilde{\ensuremath{\mathbf x}}$ with respect to a column strategy
$\ensuremath{\mathbf y}$ if and only if
$$\ensuremath{\mathbf x}^\top A \ensuremath{\mathbf y}> \tilde{\ensuremath{\mathbf x}}^\top A \ensuremath{\mathbf y}.$$
We say that $\ensuremath{\mathbf x}$ is dominant with respect to
$\ensuremath{\mathbf y}$ if and only if it is not dominated by any other
mixed strategy. A pair $(\ensuremath{\mathbf x},\ensuremath{\mathbf y})$
is an equilibrium for game $A$ if and only if both
$\ensuremath{\mathbf x}$ and $\ensuremath{\mathbf y}$ are dominant with
respect to each other. It is a good exercise for the reader at this
point to find an equilibrium for the rock-paper-scissors game.

At this point, some natural questions arise: Is there always an
equilibrium in a given zero-sum game? Can it be computed efficiently?
Are there natural repeated-game-playing strategies that reach it?

As we shall see, the answer to all questions above is affirmative. Let
us rephrase these questions in a different way. Consider the optimal row
strategy, i.e., a mixed strategy $\ensuremath{\mathbf x}$, such that the
expected loss is minimized, no matter what the column player does. The
optimal strategy for the row player would be:
$$\ensuremath{\mathbf x}^\star \in \mathop{\mathrm{\arg\min}}_{\ensuremath{\mathbf x}\in \Delta_n} {\max_{\ensuremath{\mathbf y}\in \Delta_m} \ensuremath{\mathbf x}^{\top}A \ensuremath{\mathbf y}}.$$
Notice that we use the notation $\ensuremath{\mathbf x}^\star \in$
rather than $\ensuremath{\mathbf x}^\star =$, since in general the set
of strategies attaining the minimal loss over worst-case column
strategies can contain more than a single strategy. Similarly, the
optimal strategy for the column player would be:
$$\ensuremath{\mathbf y}^\star \in \mathop{\mathrm{\arg\max}}_{\ensuremath{\mathbf y}\in \Delta_m} {\min_{\ensuremath{\mathbf x}\in \Delta_n} \ensuremath{\mathbf x}^{\top}A \ensuremath{\mathbf y}}.$$

Playing these strategies, no matter what the column player does, the row
player would pay no more than
$$\lambda_R = \min_{\ensuremath{\mathbf x}\in \Delta_n} \max_{\ensuremath{\mathbf y}\in \Delta_m} \ensuremath{\mathbf x}^{\top} A \ensuremath{\mathbf y}= \max_{\ensuremath{\mathbf y}\in \Delta_m} {\ensuremath{\mathbf x}^{\star}}^{\top} A \ensuremath{\mathbf y},$$
and column player would earn at least
$$\lambda_C =  \max_{\ensuremath{\mathbf y}\in \Delta_m} \min_{\ensuremath{\mathbf x}\in \Delta_n} \ensuremath{\mathbf x}^{\top} A \ensuremath{\mathbf y}= \min_{\ensuremath{\mathbf x}\in \Delta_n} {\ensuremath{\mathbf x}^{\top}} A \ensuremath{\mathbf y}^\star .$$

With these definitions we can state von Neumann's famous minimax
theorem:

::: theorem
**Theorem 8.3** (von Neumann minimax theorem). *In any zero-sum game, it
holds that $\lambda_R = \lambda_C$.*
:::

This theorem answers all our above questions on the affirmative. The
value $\lambda^\star = \lambda_C = \lambda_R$ is called the **value of
the game**, and its existence and uniqueness imply that any
$\ensuremath{\mathbf x}^\star$ and $\ensuremath{\mathbf y}^\star$ in the
appropriate optimality sets are an equilibrium.

We proceed to give a constructive proof of von Neumann's theorem which
also yields an efficient algorithm as well as natural repeated-game
playing strategies that converge to it.

### Equivalence of von Neumann Theorem and LP duality

The von Neumann theorem is equivalent to the duality theorem of linear
programming in a very strong sense, and either implies the other via
simple reduction. Thus, it suffices to prove only von Neumann's theorem
to prove the duality theorem.

The first part of this equivalence is shown by representing a zero-sum
game as a primal-dual linear program instance, as we do now.

Observe that the definition of an optimal row strategy and value is
equivalent to the following LP: $$\begin{aligned}
 \min \quad & & \lambda \\
\text{s.t.} \quad & & \sum{\ensuremath{\mathbf x}_i}=1 \\
   & & \forall i \in [m] \ . \ \ensuremath{\mathbf x}^{\top}A e_i \leq \lambda  \\
   & & \forall i \in [n] \ . \ \ensuremath{\mathbf x}_i \geq 0  .
\end{aligned}$$ To see that the optimum of the above $LP$ is attained at
$\lambda_R$, note that the constraint
$\ensuremath{\mathbf x}^{\top}A e_i \leq \lambda \quad \forall i \in [m]$
is equivalent to the constraint
$\forall \ensuremath{\mathbf y}\in \Delta_m \ . \ \ensuremath{\mathbf x}^\top A \ensuremath{\mathbf y}\leq \lambda$,
since: $$\begin{aligned}
\forall \ensuremath{\mathbf y}\in \Delta_m \ . \quad \ensuremath{\mathbf x}^\top A \ensuremath{\mathbf y}= \sum_{j=1}^m {\ensuremath{\mathbf x}^{\top}A e_j} \cdot \ensuremath{\mathbf y}_j   \leq  \lambda \sum_{j=1}^m {\ensuremath{\mathbf y}_j} = \lambda
\end{aligned}$$

The dual program to the above LP is given by $$\begin{aligned}
	\max  \quad  & & \mu \\
	\text{s.t.}  \quad  & & \sum{\ensuremath{\mathbf y}_i}=1 \\
	& & \forall i \in [n] \ . \ e_i^\top A \ensuremath{\mathbf y}\geq \mu  \\
	& & \forall i \in [m] \ . \ \ensuremath{\mathbf y}_i\geq0  .
\end{aligned}$$

By similar arguments, the dual program precisely defines $\lambda_C$ and
$\ensuremath{\mathbf y}^\star$. The duality theorem asserts that
$\lambda_R = \lambda_C = \lambda^\star$, which gives von Neumann's
theorem.

The other direction, i.e., showing that von Neumann's theorem implies LP
duality, is slightly more involved. Basically, one can convert any LP
into the format of a zero-sum game. Special care is needed to ensure
that the original LP is indeed feasible, as zero-sum games are always
feasible and linear programs need not be. The details are left as an
exercise at the end of this chapter.

## Proof of von Neumann Theorem

In this section we give a proof of von Neumann's theorem using online
convex optimization algorithms with sublinear regret.

The first part of the theorem, which is also known as weak duality in
the LP context, is rather straightforward:

**Direction 1 ($\lambda_R \geq \lambda_C$):**

::: proof
*Proof.* $$\begin{aligned}
\lambda_R &  =   \min_{\ensuremath{\mathbf x}\in \Delta_n} \max_{\ensuremath{\mathbf y}\in \Delta_m} \ensuremath{\mathbf x}^{\top} A \ensuremath{\mathbf y}\\
&  = \max_{\ensuremath{\mathbf y}\in \Delta_m} {\ensuremath{\mathbf x}^{\star}}^{\top} A \ensuremath{\mathbf y}& \mbox{ definition of $\ensuremath{\mathbf x}^\star$} \\
& \geq  \max_{\ensuremath{\mathbf y}\in \Delta_m} \min_{\ensuremath{\mathbf x}\in \Delta_n} \ensuremath{\mathbf x}^\top A \ensuremath{\mathbf y}\\
& =  \lambda_C.
\end{aligned}$$ ◻
:::

The second and main direction, known as strong duality in the LP
context, requires the technology of online convex optimization we have
proved thus far:

**Direction 2 ($\lambda_R \leq \lambda_C$):**

::: proof
*Proof.* We consider a repeated game defined by the $n \times m$ matrix
$A$. For $t=1,2,...,T$, the row player provides a mixed strategy
$\ensuremath{\mathbf x}_t \in \Delta_n$, column player plays mixed
strategy $\ensuremath{\mathbf y}_t \in \Delta_m$, and the loss of the
row player, which equals to the reward of the column player, equals
$\ensuremath{\mathbf x}_t^\top A \ensuremath{\mathbf y}_t$.

The row player generates the mixed strategies $\ensuremath{\mathbf x}_t$
according to an OCO algorithm --- specifically using the Exponentiated
Gradient algorithm [\[alg:eg\]](#alg:eg){reference-type="ref"
reference="alg:eg"} from chapter
[5](#chap:regularization){reference-type="ref"
reference="chap:regularization"}. The convex decision set is taken to be
the $n$ dimensional simplex
$\mathcal{K} =  \Delta_n = \{ \ensuremath{\mathbf x}\in \mathbb{R}^n \; | \; \ensuremath{\mathbf x}(i) \geq 0, \sum{\ensuremath{\mathbf x}(i)}=1 \}$.
The loss function at time $t$ is given by
$$f_t(\ensuremath{\mathbf x}) =  \ensuremath{\mathbf x}^{\top}A\ensuremath{\mathbf y}_t \mbox{\ \ \ ($f_t$ is linear with respect to $\ensuremath{\mathbf x}$) } .$$

Spelling out the EG strategy for this particular instance, we have
$$\ensuremath{\mathbf x}_{t+1}(i)  \gets \frac{  \ensuremath{\mathbf x}_t(i) e^{ -\eta A_i \ensuremath{\mathbf y}_t } } { \sum_j \ensuremath{\mathbf x}_{t}(i) e^{ -\eta A_j \ensuremath{\mathbf y}_t} }  \;.$$
Then, by appropriate choice of $\eta$ and Corollary
[5.7](#cor:eg){reference-type="ref" reference="cor:eg"}, we have
$$\begin{aligned}
 \label{eq:shalom5}
 \sum_t{f_t (\ensuremath{\mathbf x}_t)} & \leq  &  \min_{\ensuremath{\mathbf x}^\star \in \mathcal{K}}{\sum_t{f_t (\ensuremath{\mathbf x}^\star)}} + {\sqrt{2 T \log n }}  \;.
\end{aligned}$$

The column player plays her best response to the row player's strategy,
that is: $$\begin{aligned}
 \label{shalom2}
\ensuremath{\mathbf y}_t = \arg \max_{\ensuremath{\mathbf y}\in \Delta_m} \ensuremath{\mathbf x}_t^\top A \ensuremath{\mathbf y}.
\end{aligned}$$

Denote the average mixed strategies by:
$$\bar{\ensuremath{\mathbf x}} = \frac{1}{t} \sum_{\tau=1}^t {\ensuremath{\mathbf x}_\tau} \quad,\quad  \bar{\ensuremath{\mathbf y}} = \frac{1}{t} \sum_{\tau=1}^t {\ensuremath{\mathbf y}_\tau} \;.$$

Then, we have $$\begin{aligned}
\lambda_R & = \min_\ensuremath{\mathbf x}\max_\ensuremath{\mathbf y}\ \ensuremath{\mathbf x}^\top A \ensuremath{\mathbf y}\\
& \leq \max_\ensuremath{\mathbf y}\bar{\ensuremath{\mathbf x}}^\top A \ensuremath{\mathbf y}& \mbox{special case}\\
& = \frac{1}{T} \sum_t \ensuremath{\mathbf x}_t A \ensuremath{\mathbf y}^\star \\
& \leq \frac{1}{T} \sum_t \ensuremath{\mathbf x}_t A \ensuremath{\mathbf y}_t & \mbox{ by \eqref{shalom2} }\\
& \leq \frac{1}{T} \min_\ensuremath{\mathbf x}\sum_t \ensuremath{\mathbf x}^\top A \ensuremath{\mathbf y}_t + \sqrt{2 \log n /T} & \mbox{ by \eqref{eq:shalom5} } \\
& = \min_\ensuremath{\mathbf x}\ensuremath{\mathbf x}^\top A \bar{\ensuremath{\mathbf y}} + \sqrt{2 \log n /T} \\
& \leq \max_\ensuremath{\mathbf y}\min_\ensuremath{\mathbf x}\ensuremath{\mathbf x}^\top A \ensuremath{\mathbf y}+ \sqrt{2 \log n /T} & \mbox{special case}\\
& = \lambda_C	+ \sqrt{2 \log n /T}.
\end{aligned}$$ Thus $\lambda_R \leq \lambda_C + \sqrt{2 \log n /T}$. As
$T \rightarrow \infty$, we obtain part 2 of the theorem. ◻
:::

Notice that besides the basic definitions, the only tool used in the
proof is the existence of sublinear regret algorithms for online convex
optimization. The fact that the regret bounds for OCO algorithms were
defined without restricting the cost functions, and that they can be
adversarially chosen, is crucial for the proof. The functions $f_t$ are
defined according to $\ensuremath{\mathbf y}_t$, which is chosen based
on $\ensuremath{\mathbf x}_t$. Thus, the cost functions we constructed
are adversarially chosen after the decision $\ensuremath{\mathbf x}_t$
was made by the row player.

## Approximating Linear Programs

The technique in the preceding section not only proves the minimax
theorem, and thus linear programming duality, but also entails an
efficient algorithm. Using the equivalence of zero-sum games and linear
programs, this efficient algorithm can be used to solve linear
programming. We now spell out the details of this algorithm in the
context of zero-sum games.

Consider the following algorithm:

::: algorithm
::: algorithmic
Input: linear program in zero-sum game format, by matrix
$A \in {\mathbb R}^{n \times m}$. Let
$\ensuremath{\mathbf x}_1  =  [ 1/n ,1/n,...,1/n]$ Compute
$\ensuremath{\mathbf y}_t  =  \max_{\ensuremath{\mathbf y}\in \Delta_m} {\ensuremath{\mathbf x}_t^\top A \ensuremath{\mathbf y}}$
Update
$\forall i \ . \  \ensuremath{\mathbf x}_{t+1}(i)  \gets \frac{  \ensuremath{\mathbf x}_t(i) e^{ -\eta A_i \ensuremath{\mathbf y}_t } } { \sum_j \ensuremath{\mathbf x}_{t}(j) e^{ -\eta A_j \ensuremath{\mathbf y}_t} }$
$\bar{\ensuremath{\mathbf x}} = \frac{1}{T} \sum_{t=1}^T \ensuremath{\mathbf x}_t$
:::
:::

Almost immediately we obtain from the previous derivation the following:

::: lemma
**Lemma 8.4**. *The returned vector $\bar{\ensuremath{\mathbf x}}$ of
Algorithm [\[alg:simpleLP\]](#alg:simpleLP){reference-type="ref"
reference="alg:simpleLP"} is a
$\frac{\sqrt{2 \log n}}{\sqrt{T}}$-approximate solution to the zero-sum
game and linear program it describes.*
:::

::: proof
*Proof.* Following the exact same steps of the previous derivation, we
have $$\begin{aligned}
\max_\ensuremath{\mathbf y}\bar{\ensuremath{\mathbf x}}^\top A \ensuremath{\mathbf y}& = \frac{1}{T} \sum_t \ensuremath{\mathbf x}_t A \ensuremath{\mathbf y}^\star \\
& \leq \frac{1}{T} \sum_t \ensuremath{\mathbf x}_t A \ensuremath{\mathbf y}_t & \mbox{ by \eqref{shalom2} }\\
& \leq \frac{1}{T} \min_\ensuremath{\mathbf x}\sum_t \ensuremath{\mathbf x}^\top A \ensuremath{\mathbf y}_t + \sqrt{2 \log n /T} & \mbox{ by \eqref{eq:shalom5} } \\
& = \min_\ensuremath{\mathbf x}\ensuremath{\mathbf x}^\top A \bar{\ensuremath{\mathbf y}} + \sqrt{2 \log n /T} \\
& \leq \max_\ensuremath{\mathbf y}\min_\ensuremath{\mathbf x}\ensuremath{\mathbf x}^\top A \ensuremath{\mathbf y}+ \sqrt{2 \log n /T} & \mbox{special case}\\
& = \lambda^\star	+ \sqrt{2 \log n /T} .
\end{aligned}$$ Therefore, for each $i \in [m]$:
$$\bar{\ensuremath{\mathbf x}}^\top A e_i  \leq \lambda^\star + \frac{\sqrt{2 \log n}}{\sqrt{T}}$$ ◻
:::

Thus, to obtain an $\varepsilon$-approximate solution, one would need
$\frac{2 \log n}{\varepsilon^2}$ iterations, each involving a simple
update procedure.

## Bibliographic Remarks {#bibliographic-remarks-5}

Game theory was founded in the late 1920's-early '30s, whose cornerstone
was laid in the classic text "Theory of Games and Economic Behavior" by
@neumann44a.

Linear programming is a fundamental mathematical optimization and
modeling tool, dating back to the 1940's and the work of @kantorovich40
and @dantzig51. Duality for linear programming was conceived by von
Neumann, as described by Dantzig in an interview [@dantzig]. For in
depth treatment of the theory of linear programming there are numerous
comprehensive texts, e.g., [@BertsimasLP; @matousek2007understanding].

The beautiful connection between low-regret algorithms and solving
zero-sum games was discovered by @Freund199979. More general connections
of convergence of low-regret algorithms to equilibria in games were
studied by @hart2000simple, and more recently in
[@Even-dar:2009; @Roughgarden:2015].

Approximation algorithms that arise via simple Lagrangian relaxation
techniques were pioneered by @PST. See also the survey [@AHK-MW] and
more recent developments that give rise to sublinear time algorithms
[@CHW; @hazan2011beating].

## Exercises

# Learning Theory, Generalization, and Online Convex Optimization {#chap:online2batch}

In our treatment of online convex optimization so far we have only
implicitly discussed learning theory. The framework of OCO was shown to
capture applications such as learning classifiers online, prediction
with expert advice, online portfolio selection and matrix completion,
all of which have a learning aspect. We have introduced the metric of
regret and gave efficient algorithms to minimize regret in various
settings. We have also argued that minimizing regret is a meaningful
approach for many online prediction problems. However, the relation to
other theories of learning was not discussed thus far.

In this section we draw a formal and strong connection between OCO and
the theory of statistical learning. We begin by giving the basic
definitions of statistical learning theory, and proceed to describe how
the applications studied in this manuscript relate to this model. We
then continue to show how regret minimization in the setting of online
convex optimization gives rise to computationally efficient statistical
learning algorithms.

## Statistical Learning Theory

The theory of statistical learning addresses the problem of learning a
concept from examples. A concept is a mapping from domain ${\mathcal X}$
to labels ${\mathcal Y}$, denoted
$C : {\mathcal X}\mapsto {\mathcal Y}$.

As an example, consider the problem of optical character recognition. In
this setting, the domain ${\mathcal X}$ can be all $n \times n$ bitmap
images, the label set ${\mathcal Y}$ is the Latin (or other) alphabet,
and the concept $C$ maps a bitmap into the character depicted in the
image.

Statistical theory models the problem of learning a concept by allowing
access to labelled examples from the target distribution. The learning
algorithm has access to pairs, or samples, from an unknown distribution
$$(\mathbf{x},y) \sim {\mathcal D}\quad , \quad \mathbf{x}\in {\mathcal X}\ , \ y \in {\mathcal Y}.$$

The goal is to be able to predict $y$ as a function of $\mathbf{x}$,
i.e., to **learn** a hypothesis, or a mapping from ${\mathcal X}$ to
${\mathcal Y}$, denoted $h: {\mathcal X}\mapsto {\mathcal Y}$, with
small error with respect to the distribution ${\mathcal D}$. In the case
that the label set is binary ${\mathcal Y}= \{0,1\}$, or discrete such
as in optical character recognition, the *generalization error* of an
hypothesis $h$ with respect to distribution ${\mathcal D}$ is given by
$$\mathop{\mbox{\rm error}}(h) \stackrel{\text{\tiny def}}{=}\mathop{\mbox{\bf E}}_{(\mathbf{x},y)\sim {\mathcal D}} [ h(\mathbf{x}) \neq y ] .$$

More generally, the goal is to learn a hypothesis that minimizes the
loss according to a (usually convex) loss function
$\ell: {\mathcal Y}\times {\mathcal Y}\mapsto {\mathbb R}$. In this case
the generalization error of a hypothesis is defined as:
$$\mathop{\mbox{\rm error}}(h) \stackrel{\text{\tiny def}}{=}\mathop{\mbox{\bf E}}_{(\mathbf{x},y)\sim {\mathcal D}} [ \ell(h(\mathbf{x}), y) ] .$$

We henceforth consider learning algorithms ${\mathcal A}$ that observe a
sample from the distribution ${\mathcal D}$ , denoted
$S \sim {\mathcal D}^m$ for a sample of $m$ examples,
$S = \{(\ensuremath{\mathbf x}_1,y_1),...,(\ensuremath{\mathbf x}_m,y_m)\}$,
and produce a hypothesis
${\mathcal A}(S) : {\mathcal X}\mapsto {\mathcal Y}$ based on this
sample.

The goal of statistical learning can thus be summarised as follows:

::: center
*Given access to i.i.d. samples from an arbitrary distribution over
${\mathcal X}\times {\mathcal Y}$ corresponding to a certain concept,
learn a hypothesis $h : {\mathcal X}\mapsto {\mathcal Y}$ which has
arbitrarily small generalization error with respect to a given loss
function.*
:::

### Overfitting

In the problem of optical character recognition the task is to recognize
a character from a given image in bitmap format. To model it in the
statistical learning setting, the domain ${\mathcal X}$ is the set of
all $n \times n$ bitmap images for some integer $n$. The label set
${\mathcal Y}$ is the latin alphabet, and the concept $C$ maps a bitmap
into the character depicted in the image.

Consider the naive algorithm which fits the perfect hypothesis for a
given sample, in this case set of bitmaps. Namely, ${\mathcal A}(S)$ is
the hypothesis which correctly maps any given bitmap input
$\ensuremath{\mathbf x}_i$ to its correct label $y_i$, and maps all
unseen bitmaps to the character $``1."$

Clearly, this hypothesis does a very poor job of generalizing from
experience - all images that have not been observed yet will be
classified without regard to their properties, surely an erroneous
classification most times. However - the training set, or observed
examples, are perfectly classified by this hypothesis!

This disturbing phenomenon is called "overfitting," a central concern in
machine learning. Before continuing to add the necessary components in
learning theory to prevent overfitting, we turn our attention to a
formal statement of when overfitting can appear.

### No free lunch?

The following theorem shows that learning, as stated in the goal of
statistical learning theory, is impossible without restricting the
hypothesis class being considered. For simplicity, we consider the
zero-one loss in this section.

::: {#thm:nfl .theorem}
**Theorem 9.1** (No Free Lunch Theorem). *Consider any domain
$\mathcal{X}$ of size $|\mathcal{X}| = 2m > 4$, and any algorithm
${\mathcal A}$ which outputs a hypothesis ${\mathcal A}(S)$ given a
sample $S$ of size $m$. Then there exists a concept
$C: \mathcal{X} \rightarrow \{0,1\}$ and a distribution $\mathcal{D}$
such that:*

-   *The generalization error of the concept $C$ is zero.*

-   *With probability at least $\frac{1}{10}$, the error of the
    hypothesis generated by ${\mathcal A}$ is at least
    $\mathop{\mbox{\rm error}}(A(S)) \geq \frac{1}{10}$.*
:::

The proof of this theorem is based on the probabilistic method, a useful
technique for showing the existence of combinatorial objects by showing
that the probability they exist in some distributional setting is
bounded away from zero. In our setting, instead of explicitly
constructing a concept $C$ with the required properties, we show it
exists by a probabilistic argument.

::: proof
*Proof.* We show that for any learning algorithm, there is some learning
task (i.e., "hard" concept) that it will not learn well. Formally, take
$\mathcal{D}$ to be the uniform distribution over ${\mathcal X}$. Our
proof strategy will be to show the following inequality, where we take a
uniform distribution over all concepts ${\mathcal X}\mapsto \{0,1\}$

$$Q \overset{def}{=} \mathop{\mbox{\bf E}}_{C:{\mathcal X}\rightarrow\{0,1\}} [\mathop{\mbox{\bf E}}_{S\sim \mathcal{D}^m} [\mathop{\mbox{\rm error}}({\mathcal A}(S))]] \geq \frac{1}{4} .$$
After showing this step, we will use Markov's Inequality to conclude the
theorem.

We proceed by using the linearity property of expectations, which allows
us to swap the order of expectations, and then conditioning on the event
that $\ensuremath{\mathbf x}\in S$.

$$\begin{aligned}
Q & = \mathop{\mbox{\bf E}}_{S} [\mathop{\mbox{\bf E}}_{C} [\mathop{\mbox{\bf E}}_{\ensuremath{\mathbf x}\in \mathcal{X}} [{\mathcal A}(S)(\ensuremath{\mathbf x}) \neq C(\ensuremath{\mathbf x})]]]  \\
& = \mathop{\mbox{\bf E}}_{S,\ensuremath{\mathbf x}} [ \mathop{\mbox{\bf E}}_{C} [{\mathcal A}(S)(\ensuremath{\mathbf x}) \neq C(\ensuremath{\mathbf x})|\ensuremath{\mathbf x}\in S] \Pr [\ensuremath{\mathbf x}\in S] ]  \\ & + \mathop{\mbox{\bf E}}_{S,\ensuremath{\mathbf x}} [ \mathop{\mbox{\bf E}}_{C} [{\mathcal A}(S)(\ensuremath{\mathbf x}) \neq C(\ensuremath{\mathbf x})|\ensuremath{\mathbf x}\not \in S]  \Pr[ \ensuremath{\mathbf x}\not \in S] ].
\end{aligned}$$

All terms in the above expression, and in particular the first term, are
non-negative and at least $0$. Also note that since the domain size is
$2m$ and the sample is of size $|S| \leq m$, we have
$\Pr(\ensuremath{\mathbf x}\not \in S ) \geq \frac{1}{2}$. Finally,
observe that
$\Pr[ {\mathcal A}(S)(\ensuremath{\mathbf x}) \neq C(\ensuremath{\mathbf x})] = \frac{1}{2}$
for all $\ensuremath{\mathbf x}\not\in S$ since we are given that the
"true" concept $C$ is chosen uniformly at random over all possible
concepts. Hence, we get that:

$$Q \geq 0 + \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4},$$ which is the
intermediate step we wanted to show. The random variable
$\mathop{\mbox{\bf E}}_{S\sim \mathcal{D}^m} [\mathop{\mbox{\rm error}}({\mathcal A}(S))]$
attains values in the range $[0,1]$. Since its expectation is at least
$\frac{1}{4}$, the event that it attains a value of at least
$\frac{1}{4}$ is non-empty. Thus, there exists a concept such that
$$\mathop{\mbox{\bf E}}_{S\sim \mathcal{D}^m} [\mathop{\mbox{\rm error}}({\mathcal A}(S))] \geq \frac{1}{4}$$
where, as assumed beforehand, $\mathcal{D}$ is the uniform distribution
over ${\mathcal X}$.

We now conclude with Markov's Inequality: since the expectation above
over the error is at least one-fourth, the probability over examples
such that the error of ${\mathcal A}$ over a random sample is at least
one-tenth is at least
$$\Pr_{S \sim {\mathcal D}^m}  \left( \mathop{\mbox{\rm error}}({\mathcal A}(S) )  \geq \frac{1}{10}\right) \geq \frac{\frac{1}{4}-\frac{1}{10}}{1-\frac{1}{10}} > \frac{1}{10}.$$ ◻
:::

### Examples of learning problems

The conclusion of the previous theorem is that the space of possible
concepts being considered in a learning problem needs to be restricted
for any meaningful guarantee. Thus, learning theory concerns itself with
concept classes, also called hypothesis classes, which are sets of
possible hypotheses from which one would like to learn. We denote the
concept (hypothesis) class by
${\mathcal H}= \{h : {\mathcal X}\mapsto {\mathcal Y}\}$.

Common examples of learning problems that can be formalized in this
model and the corresponding definitions include:

-   Optimal character recognition: In the problem of optical character
    recognition the domain ${\mathcal X}$ consists of all $n \times n$
    bitmap images for some integer $n$, the label set ${\mathcal Y}$ is
    a certain alphabet, and the concept $C$ maps a bitmap image into the
    character depicted in it. A common (finite) hypothesis class for
    this problem is the set of all decision trees with bounded depth.

-   Text classification: In the problem of text classification the
    domain is a subset of Euclidean space, i.e.,
    ${\mathcal X}\subseteq {\mathbb R}^d$. Each document is represented
    in its bag-of-words representation, and $d$ is the size of the
    dictionary. The label set ${\mathcal Y}$ is binary, where one
    indicates a certain classification or topic, e.g.,"Economics", and
    zero others.

    A commonly used hypothesis class for this problem is the set of all
    bounded-norm vectors in Euclidean space
    ${\mathcal H}= \{ h_\mathbf{w}\ , \ \mathbf{w}\in {\mathbb R}^d \ , \ \|\mathbf{w}\|_2^2 \leq \omega \}$
    such that
    $h_\mathbf{w}(\ensuremath{\mathbf x}) = \mathbf{w}^\top \ensuremath{\mathbf x}$.
    The loss function is chosen to be the hinge loss, i.e.,
    $\ell(\hat{y},y) = \max\{0 , 1 - \hat{y} y \}$.

-   Recommendation systems: recall the online convex optimization
    formulation of this problem in section
    [7.2](#sec:recommendation_systems){reference-type="ref"
    reference="sec:recommendation_systems"}. A statistical learning
    formulation for this problem is very similar. The domain is a direct
    sum of two sets
    ${\mathcal X}= {\mathcal X}_1 \oplus {\mathcal X}_2$. Here
    $\ensuremath{\mathbf x}_1 \in {\mathcal X}_1$ is a certain media
    item, and every person is an item
    $\ensuremath{\mathbf x}_2 \in {\mathcal X}_2$. The label set
    ${\mathcal Y}$ is binary, where one indicates a positive sentiment
    for the person to the particular media item, and zero a negative
    sentiment.

    A commonly considered hypothesis class for this problem is the set
    of all mappings
    ${\mathcal X}_1 \times {\mathcal X}_2 \mapsto {\mathcal Y}$ that,
    when viewed as a matrix in
    ${\mathbb R}^{|{\mathcal X}_1| \times |{\mathcal X}_2|}$, have
    bounded algebraic rank.

### Defining generalization and learnability

We are now ready to give the fundamental definition of statistical
learning theory, called Probably Approximately Correct (PAC) learning:

::: {#def:learnability .definition}
**Definition 9.2** (PAC learnability). *A hypothesis class
${\mathcal H}$ is PAC learnable with respect to loss function
$\ell : {\mathcal Y}\times {\mathcal Y}\mapsto {\mathbb R}$ if the
following holds. There exists an algorithm ${\mathcal A}$ that accepts
$S_T = \{(\mathbf{x}_t,y_t), \ t \in [T]\}$ and returns hypothesis
${\mathcal A}(S_T) \in {\mathcal H}$ that satisfies: for any
$\varepsilon,\delta > 0$ there exists a sufficiently large natural
number $T  = T(\varepsilon,\delta)$, such that for any distribution
${\mathcal D}$ over pairs $(\mathbf{x},y)$ and $T$ samples from this
distribution, it holds that with probability at least $1-\delta$
$$\mathop{\mbox{\rm error}}( {\mathcal A}(S_T) ) \leq \varepsilon.$$*
:::

A few remarks regarding this definition:

-   The set $S_T$ of samples from the underlying distribution is called
    the training set. The error in the above definition is called the
    **generalization error**, as it describes the overall error of the
    concept as generalized from the observed training set. The behavior
    of the number of samples $T$ as a function of the parameters
    $\varepsilon,\delta$ and the concept class is called the **sample
    complexity** of ${\mathcal H}$.

-   The definition of PAC learning says nothing about computational
    efficiency. Computational learning theory usually requires, in
    addition to the definition above, that the algorithm ${\mathcal A}$
    is efficient, i.e., polynomial running time with respect to
    $\varepsilon,\log \frac{1}{\delta}$ and the representation of the
    hypothesis class. The representation size for a discrete set of
    concepts is taken to be the logarithm of the number of hypotheses in
    ${\mathcal H}$, denoted $\log |{\mathcal H}|$.

-   If the hypothesis ${\mathcal A}(S_T)$ returned by the learning
    algorithm belongs to the hypothesis class ${\mathcal H}$, as in the
    definition above, we say that ${\mathcal H}$ is **properly
    learnable**. More generally, ${\mathcal A}$ may return hypothesis
    from a different hypothesis class, in which case we say that
    ${\mathcal H}$ is **improperly learnable**.

The fact that the learning algorithm can learn up to *any* desired
accuracy $\varepsilon > 0$ is called the **realizability assumption**
and greatly reduces the generality of the definition. It amounts to
requiring that a hypothesis with near-zero error belongs to the
hypothesis class. In many cases, concepts are only approximately
learnable by a given hypothesis class, or inherent noise in the problem
prohibits realizability (see exercises).

This issue is addressed in the definition of a more general learning
concept, called **agnostic learning**:

::: {#def:agnosticlearnability .definition}
**Definition 9.3** (agnostic PAC learnability). *The hypothesis class
${\mathcal H}$ is agnostically PAC learnable with respect to loss
function $\ell : {\mathcal Y}\times {\mathcal Y}\mapsto {\mathbb R}$ if
the following holds. There exists an algorithm ${\mathcal A}$ that
accepts $S_T = \{(\mathbf{x}_t,y_t), \ t \in [T]\}$ and returns
hypothesis ${\mathcal A}(S_T)$ that satisfies: for any
$\varepsilon,\delta > 0$ there exists a sufficiently large natural
number $T  = T(\varepsilon,\delta)$ such that for any distribution
${\mathcal D}$ over pairs $(\mathbf{x},y)$ and $T$ samples from this
distribution, it holds that with probability at least $1-\delta$
$$\mathop{\mbox{\rm error}}( {\mathcal A}(S_T) ) \leq \min_{h \in {\mathcal H}} \{ \mathop{\mbox{\rm error}}(h) \} + \varepsilon.$$*
:::

With these definitions, we can state the fundamental theorem of
statistical learning theory for finite hypothesis classes:

::: theorem
**Theorem 9.4** (PAC learnability of finite hypothesis classes). *Every
finite concept class ${\mathcal H}$ is agnostically PAC learnable with
sample complexity that is
$\mathop{\mbox{\rm poly}}(\varepsilon,\delta, \log |{\mathcal H}|)$.*
:::

In the following sections we prove this theorem, and in fact a more
general statement that holds also for certain infinite hypothesis
classes. The complete characterization of which infinite hypothesis
classes are learnable is a deep and fundamental question, whose complete
answer was given by Vapnik and Chervonenkis (see bibliography). The
question of which (finite or infinite) hypothesis classes are
**efficiently** PAC learnable, especially in the improper sense, is
still at the forefront of learning theory today.

## Agnostic Learning using Online Convex Optimization

In this section we show how to use online convex optimization for
agnostic PAC learning. Following the paradigm of this manuscript, we
describe and analyze a reduction from agnostic learning to online convex
optimization. The reduction is formally described in Algorithm
[\[alg:reductionOCO2LRN\]](#alg:reductionOCO2LRN){reference-type="ref"
reference="alg:reductionOCO2LRN"}.

::: algorithm
::: algorithmic
Input: OCO algorithm ${\mathcal A}$, convex hypothesis class
${\mathcal H}\subseteq {\mathbb R}^d$, convex loss function $\ell$. Let
$h_1 \leftarrow {\mathcal A}(\emptyset)$. Draw labeled example
$(\mathbf{x}_t,y_t) \sim {\mathcal D}$. Let
$f_t(h) = \ell( h(\mathbf{x}_t) , y_t)$. Update
$$h_{t+1} = {\mathcal A}( f_1,...,f_t) .$$ Return
$\bar{h} = \frac{1}{T} \sum_{t=1}^T h_t$.
:::
:::

For this reduction we assumed that the concept (hypothesis) class is a
convex subset of Euclidean space. A similar reduction can be carried out
for discrete hypothesis classes (see exercises). In fact, the technique
we explore below will work for any hypothesis set ${\mathcal H}$ that
admits a low regret algorithm, and can be generalized to infinite
hypothesis classes that are known to be learnable.

Let
$h^\star = \arg\min_{h \in {\mathcal H}} \{ \mathop{\mbox{\rm error}}(h) \}$
be the hypothesis in the class ${\mathcal H}$ that minimizes the
generalization error. Using the assumption that ${\mathcal A}$
guarantees sublinear regret, our simple reduction implies PAC learning,
as given in the following theorem.

::: {#thm:OCO2LRN .theorem}
**Theorem 9.5**. *Let ${\mathcal A}$ be an OCO algorithm whose regret
after $T$ iterations is guaranteed to be bounded by
$\ensuremath{\mathrm{{Regret}}}_T({\mathcal A})$. Then for any
$\delta > 0$, with probability at least $1-\delta$, it holds that
$$\mathop{\mbox{\rm error}}(\bar{h})\le \mathop{\mbox{\rm error}}(h^\star) + \frac{ \ensuremath{\mathrm{{Regret}}}_T({\mathcal A}) }{T} +\sqrt{\frac{8\log (\frac{2}{\delta})}{T}}.$$
In particular, for
$T = O( \frac{1}{\varepsilon^2} \log \frac{1}{\delta}  + T_\varepsilon({\mathcal A}) )$,
where $T_\varepsilon({\mathcal A})$ is the integer $T$ such that
$\frac{\ensuremath{\mathrm{{Regret}}}_T({\mathcal A})}{T} \leq \varepsilon$,
we have
$$\mathop{\mbox{\rm error}}(\bar{h})\le \mathop{\mbox{\rm error}}(h^*) + \varepsilon.$$*
:::

How general is the theorem above? In the previous chapters we have
described and analyzed OCO algorithms with regret guarantees that behave
asymptotically as $O(\sqrt{T})$ or better. This translates to sample
complexity of $O(\frac{1}{\varepsilon^2} \log \frac{1}{\delta})$ (see
exercises), which is known to be tight for certain scenarios.

To prove this theorem we need some tools from probability theory, such
as the concentration inequalities that we survey next.

### Reminder: measure concentration and martingales

Let us briefly discuss the notion of a martingale in probability theory.
For intuition, it is useful to recall the simple random walk. Let $X_i$
be a Rademacher random variable which takes values $$X_i = {
\left\{
\begin{array}{ll}
    {1}, & {\text{with probability } \quad \frac{1}{2} } \\\\
    {-1}, & {\text{with probability } \quad \frac{1}{2} }
\end{array}
\right. } .$$ A simple symmetric random walk is described by the sum of
such random variables, depicted in figure
[9.1](#fig:randomwalk){reference-type="ref" reference="fig:randomwalk"}.
Let $X = \sum_{i=1}^T X_i$ be the position after $T$ steps of this
random walk. The expectation and variance of this random variable are
$\mathop{\mbox{\bf E}}[ X] = 0 \ , \ \mbox{Var}(X) = T$.

::: center
![Symmetric random walk: 12 trials of 200 steps. The black dotted lines
show the functions $\pm \sqrt{x}$ and $\pm 2 \sqrt{x}$, respectively.
](images/random_walk.jpg){#fig:randomwalk width="3.0in"}
:::

The phenomenon of measure concentration addresses the probability of a
random variable to attain values within range of its standard deviation.
For the random variable $X$, this probability is much higher than one
would expect using only the first and second moments. Using only the
variance, it follows from Chebychev's inequality that
$$\Pr\left[ |X| \geq c \sqrt{T} \right] \leq \frac{1}{c^2}.$$ However,
the event that $|X|$ is centred around $O(\sqrt{T})$ is in fact much
tighter, and can be bounded by the Hoeffding-Chernoff lemma as follows
$$\begin{aligned}
 \label{lem:chernoff}
\Pr[|X|  \ge c \sqrt{T}] \le 2  e^{\frac{-c^2}{2}} & \mbox  { Hoeffding-Chernoff lemma.}
\end{aligned}$$

Thus, deviating by a constant from the standard deviation decreases the
probability exponentially, rather than polynomially. This well-studied
phenomenon generalizes to sums of weakly dependent random variables and
martingales, which are important for our application.

::: definition
**Definition 9.6**. *A sequence of random variables $X_1, X_2,...$ is
called a *martingale* if it satisfies:
$$\mathop{\mbox{\bf E}}[X_{t+1}|X_{t}, X_{t-1}...X_{1}] = X_{t} \quad \forall \; t>0.$$*
:::

A similar concentration phenomenon to the random walk sequence occurs in
martingales. This is captured in the following theorem by Azuma.

::: theorem
**Theorem 9.7** (Azuma's inequality). *Let
$\big \lbrace X_{i} \big \rbrace _{i=1}^{T}$ be a martingale of $T$
random variables that satisfy $|X_{i} - X_{i+1}|  \leq {1}$. Then:
$$\Pr \left[ |X_{T} - X_{0}|>c \right]  \le 2 e^{\frac{-c^2}{2T}}.$$*
:::

By symmetry, Azuma's inequality implies, $$\label{eqn:azuma2}
\Pr \left[ X_{T} - X_{0}> c \right]  = \Pr \left[X_{0} - X_{T}> c \right] \le e^{  \frac{-c^2}{2T}}.$$

### Analysis of the reduction

We are ready to prove the performance guarantee for the reduction in
Algorithm
[\[alg:reductionOCO2LRN\]](#alg:reductionOCO2LRN){reference-type="ref"
reference="alg:reductionOCO2LRN"}. Assume for simplicity that the loss
function $\ell$ is bounded in the interval $[0,1]$, i.e.,
$$\forall \hat{y},y \in {\mathcal Y}\ , \ \ell(\hat{y}, y) \in [0,1].$$

::: proof
*Proof of Theorem [9.5](#thm:OCO2LRN){reference-type="ref"
reference="thm:OCO2LRN"}.* We start by defining a sequence of random
variables that form a martingale. Let
$$Z_{t} \stackrel{\text{\tiny def}}{=}\mathop{\mbox{\rm error}}(h_{t})- \ell(h_{t}(\ensuremath{\mathbf x}_{t}),y_{t})   , \quad  X_{t} \stackrel{\text{\tiny def}}{=}\sum_{i=1}^t Z_{i}.$$
Let us verify that $\{X_t\}$ is indeed a bounded martingale. Notice that
by definition of $\mathop{\mbox{\rm error}}(h)$, we have that
$$\mathop{\mbox{\bf E}}_{(\mathbf{x},y)\sim {\mathcal D}}[Z_{t} | X_{t-1}]  = \mathop{\mbox{\rm error}}(h_t) - \mathop{\mbox{\bf E}}_{(\mathbf{x},y)\sim {\mathcal D}} [\ell(h_t(\ensuremath{\mathbf x}) , y)] =  0  .$$
Thus, by the definition of $Z_t$, $$\begin{aligned}
\mathop{\mbox{\bf E}}[ X_{t+1}|X_{t},...X_{1} ] & = \mathop{\mbox{\bf E}}[Z_{t+1} | X_t]+ X_{t}
 = X_t.
\end{aligned}$$ In addition, by our assumption that the loss is bounded,
we have that (see exercises) $$\begin{aligned}
 \label{eqn:martingale-bound}
|X_{t} - X_{t-1}| = |Z_{t}|  \le 1.
\end{aligned}$$ Therefore we can apply Azuma's theorem to the martingale
$\{X_t\}$, or rather its consequence
[\[eqn:azuma2\]](#eqn:azuma2){reference-type="eqref"
reference="eqn:azuma2"}, and get

$$\Pr[X_{T}  > c] \le  e^{\frac{-c^2}{2T}}.$$ Plugging in the definition
of $X_{T}$, dividing by $T$ and using
$c = \sqrt{2T\log (\frac{2}{\delta})}$: $$\label{eqn:oco2lrn1}
\Pr\left[
\frac{1}{T}\sum_{t=1}^{T}\mathop{\mbox{\rm error}}(h_{t})- \frac{1}{T}\sum_{t=1}^{T}{\ell(h_{t}(\ensuremath{\mathbf x}_{t}),y_{t})}  > \sqrt{\frac{2\log (\frac{2}{\delta})}{T}}
\right] \le \frac{\delta}{2}.$$

A similar martingale can be defined for $h^\star$ rather than $h_t$, and
repeating the analogous definitions and applying Azuma's inequality we
get: $$\label{eqn:oco2lrn2}
\Pr\left [  \frac{1}{T}\sum_{t=1}^{T}\mathop{\mbox{\rm error}}(h^\star)- \frac{1}{T}\sum_{t=1}^{T}{l(h^\star(\ensuremath{\mathbf x}_{t}),y_{t})}  < - \sqrt{\frac{2\log (\frac{2}{\delta})}{T}}
\right] \le \frac{\delta}{2}.$$ For notational convenience, let us use
the following notation:
$$\Gamma_1  = \frac{1}{T}\sum_{t=1}^{T}\mathop{\mbox{\rm error}}(h_{t})- \frac{1}{T}\sum_{t=1}^{T}{\ell(h_{t}(\ensuremath{\mathbf x}_{t}),y_{t})},$$
$$\Gamma_2  =  \frac{1}{T}\sum_{t=1}^{T}\mathop{\mbox{\rm error}}(h^\star)- \frac{1}{T}\sum_{t=1}^{T}{l(h^\star(\ensuremath{\mathbf x}_{t}),y_{t})}.$$
Next, observe that $$\begin{aligned}
& \frac{1}{T}\sum_{t=1}^{T}\mathop{\mbox{\rm error}}(h_t)- \mathop{\mbox{\rm error}}(h^\star) \\
&  =   \Gamma_1 - \Gamma_2
+ \frac{1}{T}\sum_{t=1}^{T}{\ell(h_{t}(\ensuremath{\mathbf x}_{t}),y_{t})} -
\frac{1}{T}\sum_{t=1}^{T}{\ell(h^\star (\ensuremath{\mathbf x}_{t}),y_{t})}  \\
& \leq \frac{\ensuremath{\mathrm{{Regret}}}_T({\mathcal A})}{T} + \Gamma_1 - \Gamma_2, % & \mbox { $f_t(h) = \ell(h(\bx_t),y_t) $  } % \\
% & \leq |\Gamma_1 | + | \Gamma_2 | +  \regret_T(\mA) & \mbox { $\triangle$-inequality } \\
\end{aligned}$$ where in the last inequality we have used the definition
$f_t(h) = \ell(h(\mathbf{x}_t),y_t)$. From the above and Inequalities
[\[eqn:oco2lrn1\]](#eqn:oco2lrn1){reference-type="eqref"
reference="eqn:oco2lrn1"},
[\[eqn:oco2lrn2\]](#eqn:oco2lrn2){reference-type="eqref"
reference="eqn:oco2lrn2"} we get $$\begin{aligned}
& \Pr \left [ \frac{1}{T}\sum_{t=1}^{T}\mathop{\mbox{\rm error}}(h_{t})- \mathop{\mbox{\rm error}}(h^\star) >  \frac{\ensuremath{\mathrm{{Regret}}}_T({\mathcal A})}{T} +2\sqrt{\frac{2\log (\frac{2}{\delta})}{T}} \right]  \\
&
\le \Pr \left [   \Gamma_1 - \Gamma_2  >  2\sqrt{\frac{2\log (\frac{1}{\delta})}{T}} \right] \\
& \leq   \Pr \left [ \Gamma_1    >  \sqrt{\frac{2\log (\frac{1}{\delta})}{T}} \right]  + \Pr \left [   \Gamma_2  \le - \sqrt{\frac{2\log (\frac{1}{\delta})}{T}} \right] \\
& \leq \delta. \quad\quad\quad \mbox{Inequalities \eqref{eqn:oco2lrn1}, \eqref{eqn:oco2lrn2}}
\end{aligned}$$ By convexity we have that
$\mathop{\mbox{\rm error}}(\bar{h}) \le \frac{1}{T}\sum_{t=1}^{T}\mathop{\mbox{\rm error}}(h_{t})$.
Thus, with probability at least $1-\delta$,
$$\mathop{\mbox{\rm error}}(\bar{h}) \le \dfrac{1}{T}\sum_{t=1}^{T}\mathop{\mbox{\rm error}}(h_{t}) \le \mathop{\mbox{\rm error}}(h^\star) + \frac{ \ensuremath{\mathrm{{Regret}}}_T({\mathcal A}) }{T} +\sqrt{\frac{8\log (\frac{2}{\delta})}{T}}.$$ ◻
:::

## Learning and Compression

Thus far we have considered finite and certain infinite hypothesis
classes, and shown that they are efficiently learnable if there exists
an efficient regret-minimization algorithm for a corresponding OCO
setting.

In this section we describe yet another property which is sufficient for
PAC learnability: the ability to compress the training set. This
property is particularly easy to state and use, especially for infinite
hypothesis classes. It does not, however, imply efficient algorithms.

Intuitively, if a learning algorithm is capable to express an hypothesis
using a small fraction of the training set, we will show that it
generalizes well to unseen data. For simplicity, we only consider
learning problems that satisfy a variant of the realizablility
assumption, i.e., the compression scheme generates a hypothesis that
attains zero error.

More formally, we define the notion of a compression scheme for a given
learning problem as follows. The definition and theorem henceforth can
be generalized to allow for loss functions, but for simplicity, consider
only the zero-one loss function for this section.

::: definition
**Definition 9.8**. *(Compression Scheme) A distribution ${\mathcal D}$
over ${\mathcal X}\times {\mathcal Y}$ admits a compression scheme of
size $k$, realized by an algorithm ${\mathcal A}$, if the following
holds. For any $T > k$, let $S_T = \{(\mathbf{x}_t,y_t), \ t \in [T]\}$
be a sample from ${\mathcal D}$. There exists an
$S' \subseteq S_T \ , \ |S'| = k$, such that the algorithm
${\mathcal A}$ accepts the set of $k$ examples $S'$, and returns a
hypothesis ${\mathcal A}(S') \in \{ {\mathcal X}\mapsto {\mathcal Y}\}$,
which satsifies:
$$\mathop{\mbox{\rm error}}_{S_T}( {\mathcal A}(S') ) = 0 .$$*
:::

The main conclusion of this section is that a learning problem that
admits a compression scheme of size $k$ is PAC learnable with sample
complexity proportional to $k$. This is formally given the following
theorem.

::: {#thm:compression2generalization .theorem}
**Theorem 9.9**. *Let ${\mathcal D}$ be a data distribution that admits
a compression scheme of size $k$ realized by algorithm ${\mathcal A}$.
Then with probability at least $1-\delta$ over the choice of a training
set $|S_T|=T$, it holds that
$$\mathop{\mbox{\rm error}}( {\mathcal A}(S_T)) \leq \frac{8 k \log \frac{T}{\delta}}{T} .$$*
:::

::: proof
*Proof.* Denote by $\mathop{\mbox{\rm error}}_S(h)$ the error of an
hypothesis $h$ on a sample $S$ of i.i.d. examples, where the sample is
taken independently of $h$. Since the examples are chosen independently,
the probability that a hypothesis with
$\mathop{\mbox{\rm error}}(h) > \varepsilon$ has
$\mathop{\mbox{\rm error}}_S(h) = 0$ is at most $(1-\varepsilon)^{|S|}$.
Denote the event of $h$ satisfying these two conditions by
${h \in {\mbox{bad}}}$.

Consider a compression scheme for distribution ${\mathcal D}$ of size
$k$, realized by ${\mathcal A}$, and a sample of size $|S_T|=T \gg  k$.
By definition of a compression scheme, the hypothesis returned by
${\mathcal A}$ is based on $k$ examples chosen from the set
$S' \subseteq S_T$. We can bounds the probability of the event that
$\mathop{\mbox{\rm error}}_{S_T}({\mathcal A}(S')) = 0$ and
$\mathop{\mbox{\rm error}}({\mathcal A}(S')) > \varepsilon$, denoted by
${{\mathcal A}(S') \in \mbox{bad}}$, as follows, $$\begin{aligned}
& \Pr[ {{\mathcal A}(S') \in \mbox{bad}} ] \\
& = \sum_{S' \subseteq S_T, |S'|=k} \Pr[ {{\mathcal A}(S') \in \mbox{bad}}   ] \cdot  \Pr[ S'] & \mbox{law of total probability} \\
& \leq \binom{T}{k}  (1-\varepsilon)^T .
\end{aligned}$$

For $\varepsilon= \frac{8k \log \frac{T}{\delta} }{T}$, we have that
$$\binom{T}{k}  (1-\varepsilon)^T \leq T^k e^{-\varepsilon T} \leq \delta .$$
Since the compression scheme is guaranteed to return a hypothesis such
that $\mathop{\mbox{\rm error}}_{S_T}({\mathcal A}(S')) = 0$, this
implies that with probability at least $1-\delta$, the hypothesis
${\mathcal A}(S')$ has
$\mathop{\mbox{\rm error}}({\mathcal A}(S')) \leq \varepsilon$. ◻
:::

An important example of the use of compression schemes to bound the
generalization error is for the hypothesis class of hyperplanes in
${\mathbb R}^d$. It is left as an exercise to show that this hypothesis
class admits a compression scheme of size $d$.

## Bibliographic Remarks {#bibliographic-remarks-6}

The foundations of statistical and computational learning theory were
put forth in the seminal works of @Vapnik1998 and @Valiant1984
respectively. There are numerous comprehensive texts on statistical and
computational learning theory, see e.g., [@Kearns1994], and the recent
text [@shalev-shwartz_ben-david_2014].

Reductions from the online to the statistical (a.k.a. "batch") setting
were initiated by Littlestone [@Littlestone89]. Tighter and more general
bounds were explored in [@Cesa06; @CesaGen08; @Zhang05].

The probabilistic method is attributed to Paul Erdos, see the
illuminating text of Alon and Spencer [@AlonS92].

The relationship between compression and PAC learning was studied in the
seminal work of @LittlestoneW86. For more on the relationship and
historical connections between statistical learning and compression see
the inspiring chapter in [@avibook]. More recently @moran2016sample
[@david2016statistical] show that compression is equivalent to
learnability in general supervised learning tasks and give quantitative
bounds for this relationship.

The use of compression for proving generalization error bounds has been
applied in [@hanneke2019sample] for regression and in
[@gottlieb2018near; @kontorovich2017nearest] for nearest neighbor
classification. Another application is the recent work of
@bousquet2020proper which gives optimal generalization error bounds for
support vector machines using compression.

## Exercises

# Learning in Changing Environments {#chap:adaptive}

In online convex optimization the decision maker iteratively makes a
decision without knowledge of the future, and pays a cost based on her
decision and the observed outcome. The algorithms that we have studied
thus far are designed to perform nearly as well as the best single
decision in hindsight. The performance metric we have advocated for,
average *regret* of the online player, approaches zero as the number of
game iterations grows.

In scenarios in which the outcomes are sampled from some (unknown)
distribution, regret minimization algorithms effectively "learn\" the
environment and approach the optimal strategy. This was formalized in
chapter [9](#chap:online2batch){reference-type="ref"
reference="chap:online2batch"}. However, if the underlying distribution
changes, no such claim can be made.

Consider for example the online shortest path problem we have studied in
the first chapter. It is a well observed fact that traffic in networks
exhibits changing cyclic patterns. A commuter may choose one path from
home to work on a weekday, but a completely different path on the
weekend when traffic patterns are different. Another example is the
stock market: in a bull market the investor may want to purchase
technology stocks, but in a bear market perhaps they would shift their
investments to gold or government bonds.

When the environment undergoes many changes, standard regret may not be
the best measure of performance. In changing environments, the online
convex optimization algorithms we have studied thus far for strongly
convex or exp-concave loss functions exhibit undesirable "static\"
behavior, and converge to a fixed solution.

In this chapter we introduce and study a generalization of the concept
of regret called *adaptive* regret, to allow for a changing prediction
strategy. We start with examining the notion of adapting in the problem
of prediction from expert advice. We then continue to the more
challenging setting of online convex optimization, and derive efficient
algorithms for minimizing this more refined regret metric.

## A Simple Start: Dynamic Regret

Before giving the main performance metric studied in this chapter, we
consider the first natural approach: measuring regret w.r.t. any
sequence of decisions. Clearly, in general it is impossible to compete
with an arbitrary changing benchmark. However, it is possible to give a
refined analysis that shows what happens to the regret of an online
convex optimization algorithm vs. changing decisions.

More precisely, define the *dynamic regret* of an OCO algorithm with
respect to a sequence
$\ensuremath{\mathbf u}_1,\ldots,\ensuremath{\mathbf u}_T$ as:
$$\begin{aligned}
\ensuremath{\mathrm{{DynamicRegret}}}_T({\mathcal A},\ensuremath{\mathbf u}_1,\ldots,\ensuremath{\mathbf u}_T) & \stackrel{\text{\tiny def}}{=}&  \sum_{t=1}^T f_t(\ensuremath{\mathbf x}_t) - \sum_{t=1}^T f_t(\ensuremath{\mathbf u}_t)
\end{aligned}$$

To analyze the dynamic regret, some measure of the complexity of the
sequence $\ensuremath{\mathbf u}_1,\ldots,\ensuremath{\mathbf u}_T$ is
necessary. Let
${\mathcal P}(\ensuremath{\mathbf u}_1,\ldots,\ensuremath{\mathbf u}_T)$
be the path length of the comparison sequence defined as
$${\mathcal P}(\ensuremath{\mathbf u}_1,\ldots,\ensuremath{\mathbf u}_T) = \sum_{t=1}^{T-1} \|\ensuremath{\mathbf u}_t - \ensuremath{\mathbf u}_{t+1}\|  + 1.$$

It is natural to expect the regret to scale with the path length, as
indeed the following theorem shows. For a fixed comparator
$\ensuremath{\mathbf u}_t = \ensuremath{\mathbf x}^\star$, the path
length is one, and thus Theorem
[10.1](#thm:dynamic-regret){reference-type="ref"
reference="thm:dynamic-regret"} recovers the $O(\sqrt{T})$ standard
regret bound. For simplicity, we assume that the time horizon $T$ is
known ahead of time, and so is the path length of the comparator
sequence, although these limitations can be removed (see bibliographic
section).

::: {#thm:dynamic-regret .theorem}
**Theorem 10.1**. *Online Gradient Descent (algorithm
[\[alg:ogd\]](#alg:ogd){reference-type="ref" reference="alg:ogd"}) with
step size
$\eta \approx \sqrt{\frac{{\mathcal P}(\ensuremath{\mathbf u}_1,...,\ensuremath{\mathbf u}_T) }{T}}$
guarantees the following dynamic regret bound:
$$\ensuremath{\mathrm{{DynamicRegret}}}_T({\mathcal A},\ensuremath{\mathbf u}_1,\ldots,\ensuremath{\mathbf u}_T) = O( \sqrt{T  {\mathcal P}(\ensuremath{\mathbf u}_1,\ldots,\ensuremath{\mathbf u}_T) } )$$*
:::

::: proof
*Proof.* Using our notation, and following the steps of the proof of
Theorem [3.1](#thm:gradient){reference-type="ref"
reference="thm:gradient"}, $$\begin{aligned}
\|\mathbf{x}_{t+1}-\ensuremath{\mathbf u}_t\|^2\ \leq  \|\mathbf{y}_{t+1}-\ensuremath{\mathbf u}_t\|^2 = \|\mathbf{x}_t- \ensuremath{\mathbf u}_t\|^2 + \eta^2
\|\nabla_t\|^2 -2 \eta \nabla_t^\top (\mathbf{x}_t -\ensuremath{\mathbf u}_t) .
\end{aligned}$$ Thus, $$\begin{aligned}
2 \nabla_t^\top (\mathbf{x}_t-\ensuremath{\mathbf u}_t)\ &\leq  \frac{ \|\mathbf{x}_t-
\ensuremath{\mathbf u}_t\|^2-\|\mathbf{x}_{t+1}-\ensuremath{\mathbf u}_t\|^2}{\eta} + \eta G^2 %\\
%& \leq \frac{1}{\eta} \left( \|\x_t\|^2 - \|\x_{t+1}\|^2 +
\end{aligned}$$ Using convexity and summing this inequality across time
we get $$\begin{aligned}
& 2 \left( \sum_{t=1}^T f_t(\mathbf{x}_t)-f_t(\ensuremath{\mathbf u}_t) \right ) \leq 2\sum_{t=1}^T \nabla_t^\top (\ensuremath{\mathbf x_{t}}- \ensuremath{\mathbf x}^\star) \\
&\leq  \sum_{t=1}^T \frac{ \|\mathbf{x}_t-
	\ensuremath{\mathbf u}_t\|^2-\|\mathbf{x}_{t+1}-\ensuremath{\mathbf u}_t\|^2}{\eta} + \eta G^2 T     \\
& =   \frac{1}{\eta}  \sum_{t=1}^T  \left( \|\ensuremath{\mathbf x}_t\|^2 - \|\ensuremath{\mathbf x}_{t+1}\|^2 + 2 \ensuremath{\mathbf u}_t^\top (\ensuremath{\mathbf x}_{t+1} - \ensuremath{\mathbf x}_{t})  \right)
 + \eta  G^2 T  \\
&\leq \frac{2}{\eta} \left( D^2  +  \sum_{t=2}^{T}  \ensuremath{\mathbf x}_t^\top ( \ensuremath{\mathbf u}_{t-1} - \ensuremath{\mathbf u}_{t})   + \ensuremath{\mathbf u}_T^\top \ensuremath{\mathbf x}_{T+1} - \ensuremath{\mathbf u}_1^\top \ensuremath{\mathbf x}_1 \right)  + \eta G^2 T   \\
&\leq \frac{3}{\eta} \left(  D^2   + D \sum_{t=2}^{T}  \| \ensuremath{\mathbf u}_{t-1} - \ensuremath{\mathbf u}_{t} \|  \right)  + \eta G^2 T &  \ensuremath{\mathbf u}_t \in \ensuremath{\mathcal K}\\
& \leq \frac{3D^2 }{\eta} {\mathcal P}(\ensuremath{\mathbf u}_1,...,\ensuremath{\mathbf u}_{T} ) + \eta G^2 T . %\leq 3 DG \sqrt{T \mP(\uv_1,\ldots,\uv_T) }.
\end{aligned}$$ The theorem now follows by choice of $\eta$. ◻
:::

This simple modification to the analysis of online gradient descent
naturally extends to online mirror descent, as well as to other notions
of path distance of the comparison sequence.

We now turn to another metric of performance that requires more advanced
methods than we have seen thus far. This metric can be shown to be more
general than dynamic regret, in the sense that the bounds we prove also
imply low dynamic regret.

## The Notion of Adaptive Regret

The main performance metric we consider in this chapter is designed to
measure the performance of a decision maker in a changing environment.
It is formally given in the following definition.

::: {#def:adaptiveregret .definition}
**Definition 10.2**. *The adaptive regret of an online convex
optimization algorithm ${\mathcal A}$ is defined as the maximum regret
it achieves over any contiguous time interval. Formally,
$$\begin{aligned}
\ensuremath{\mathrm{{AdaptiveRegret}}}_T({\mathcal A}) & \stackrel{\text{\tiny def}}{=}& \sup_{I = [r,s] \subseteq [T]} \left\{   \sum_{t=r}^s f_t(\ensuremath{\mathbf x}_t) - \min_{x^*_I \in \ensuremath{\mathcal K}} \sum_{t=r}^s f_t(\ensuremath{\mathbf x}^*_I) \right\} \\
& =  & \sup_{I = [r,s] \subseteq [T]} \left\{ \ensuremath{\mathrm{{Regret}}}_{[r,s]}({\mathcal A}) \right\} .
\end{aligned}$$*
:::

As opposed to standard regret, the power of this definition stems from
the fact that the comparator is allowed to change. In fact, it is
allowed to change indefinitely with every interval of time.

For an algorithm with low adaptive regret, as opposed to standard
regret, how would its performance guarantee differ in a changing
environment? Consider the problem of portfolio selection, for which time
can be divided into disjoint segments with different characteristics:
bear market in the first $T/2$ iterations and bull market in the last
$T/2$ iterations. A (standard) sublinear regret algorithm is only
required to converge to the average of both optimal portfolios, clearly
an undesirable outcome. However, an algorithm with sublinear adaptive
regret bounds would *necessarily* converge to the optimal portfolio in
both intervals.

Not only does this definition make intuitive sense, but it generalizes
other natural notions. For example, consider an OCO setting that can be
divided into $k$ intervals, such that in each a different comparator is
optimal. Then an adaptive regret guarantee of
$\ensuremath{\mathrm{{AdaptiveRegret}}}_T = o(T)$ would translate to
overall regret of
$k \times \ensuremath{\mathrm{{AdaptiveRegret}}}_{T/k}$ compared to the
best $k$-shifting comparator.

### Weakly and strongly adaptive algorithms

The Online Gradient Descent algorithm over general convex losses, with
step sizes $O(\frac{1}{\sqrt{t}})$, attains an adaptive regret guarantee
of $$\ensuremath{\mathrm{{AdaptiveRegret}}}_T(OGD) = O(\sqrt{T}) ,$$ and
this bound is tight. This is a simple consequence of the analysis we
have already seen in chapter [3](#chap:first order){reference-type="ref"
reference="chap:first order"}, and left as an exercise. Unfortunately
this guarantee is meaningless for intervals of length $o(\sqrt{T})$.

Recall that for strongly convex loss functions, the OGD algorithm with
the optimal learning rate schedule attains $O(\log T)$ regret. However,
it does **not** attain any non-trivial adaptive regret guarantee: its
adaptive regret can be as large as $\Omega(T)$, and this is also left as
an exercise.

An OCO algorithm ${\mathcal A}$ is said to be *strongly adaptive* if its
adaptive regret can be bounded by its regret over the interval up to
logarithmic terms in $T$, i.e. $$\begin{aligned}
%& \sup_{I = [r,s] \subseteq [T]} \left\{   \sum_{t=r}^s f_t(\x_t) - \min_{x^*_I \in \K} \sum_{t=r}^s f_t(\x^*_I) \right\} \\
&  \ensuremath{\mathrm{{AdaptiveRegret}}}_T({\mathcal A})   = O(\ensuremath{\mathrm{{Regret}}}_I({\mathcal A})  \cdot \log^{O(1)} T).
\end{aligned}$$

The natural question is thus: are there algorithms that attain the
optimal regret guarantee, and simultaneously the optimal adaptive regret
guarantee? As we shall see, the answer is affirmative in a strong sense:
we shall describe and analyze algorithms that are optimal in both
metrics. Furthermore, these algorithms can be implemented with small
computational overhead over the non-adaptive methods we have already
studied.

## Tracking the Best Expert

Consider the fundamental problem studied in the first chapter of this
text, prediction from expert advice, but with a small twist. Instead of
a static best expert, consider the setting in which different experts
are the "best expert\" in different time intervals. More precisely,
consider the situation in which time $[T]$ can be divided into $k$
disjoint intervals such that each admits a different "locally best\"
expert. Can we learn to track the best expert?

This tracking problem was historically the first motivation to study
adaptivity in online learning. Indeed, as shown by Herbster and Warmuth
(see bibliographic section), there is a natural algorithm that attains
optimal regret bounds.

The Fixed Share algorithm, describe in Algorithm
[\[alg:fixed-share\]](#alg:fixed-share){reference-type="ref"
reference="alg:fixed-share"}, is a variant of the Hedge Algorithm
[\[alg:Hedge\]](#alg:Hedge){reference-type="ref" reference="alg:Hedge"}.
On top of the familiar multiplicative updates, it adds a uniform
exploration term whose purpose is to avoid the weight of any expert from
becoming too small. This provably allows a regret bound that tracks the
best expert in any interval.

::: algorithm
::: algorithmic
Input: parameter $\delta < \frac{1}{2}$. Initialize
$\forall i \in [N] , p_i^1 = \frac{1}{N}$. Play
$\ensuremath{\mathbf x}_t = \sum_{i=1}^N p_t^i \ensuremath{\mathbf x}_t^{i}$.
After receiving $f_t$, update for $1 \leq i \leq N$\
$$\hat{p}^{i}_{t+1} = \frac{p^{i}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{i}_t)}}{\sum_{j=1}^N p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)}}$$
Fixed-share step:
$$p_{t+1}^{i}  = (1 - \delta )\hat{p}^{i}_{t+1} + \frac{\delta}{N}$$
:::
:::

In line with the notation we have used throughout this manuscript, we
denote decisions in a convex decision set by
$\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}$. An expert $i$
suggests decision $\ensuremath{\mathbf x}_t^i$, and suffers loss
according to a convex loss function denoted
$f_t(\ensuremath{\mathbf x}_t^i)$. The main performance guarantee for
the Fixed Share algorithm is given in the theorem below.

::: {#thm:fixed-share .theorem}
**Theorem 10.3**. *Given a sequence of $\alpha$-exp-concave loss
functions, the Fixed-Share algorithm with $\delta = \frac{1}{2 T}$
guarantees
$$\sup_{I = [r,s] \subseteq [T]} \left\{   \sum_{t=r}^s f_t(\ensuremath{\mathbf x}_t) - \min_{i^* \in [N]} \sum_{t=r}^s f_t(\ensuremath{\mathbf x}^{i^*}_t) \right\} =  O\left(\frac{1}{\alpha} \log N T \right) .$$*
:::

Notice that this is a different guarantee than adaptive regret as per
Definition [10.2](#def:adaptiveregret){reference-type="ref"
reference="def:adaptiveregret"}, as the decision set is discrete.
However, it is a crucial component in the adaptive algorithms we will
explore in the next section.

As a direct conclusion from this theorem, it can be shown (see
exercises) that if the best expert changes $k$ times in a sequence of
length $T$, the overall regret compared to the best expert in every
interval is bounded by $$O \left( k \log \frac{NT}{k} \right) .$$

To prove this theorem, we start with the following lemma, which is a
fine-grained analysis of the multiplicative weights properties:

::: {#lem:round-reg .lemma}
**Lemma 10.4**. *For all $1 \leq i < N$,
$$f_t(\ensuremath{\mathbf x}_t) - f_t(\ensuremath{\mathbf x}^{i}_t) \leq \alpha^{-1} (\log \hat{p}^{i}_{t+1} - \log \hat{p}^{i}_{t} - \log (1 - \delta) ) .$$*
:::

::: proof
*Proof.* Using the $\alpha$-exp concavity of $f_t$, $$\begin{aligned}
e^{-\alpha f_t(\ensuremath{\mathbf x}_t)} & = & e^{-\alpha f_t(\sum_{j=1}^N p^{j}_t \ensuremath{\mathbf x}^{j}_t)}  \geq  \sum_{j=1}^N p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)}.
\end{aligned}$$ Taking the natural logarithm,
$$f_t(\ensuremath{\mathbf x}_t) \leq  -\alpha^{-1} \log \sum_{j=1}^N p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)} \nonumber$$
Hence, $$\begin{aligned}
  f_t(\ensuremath{\mathbf x}_t) - f_t(\ensuremath{\mathbf x}^{i}_t)  & \leq  \alpha^{-1}(\log e^{-\alpha f_t(\ensuremath{\mathbf x}^{i}_t)} - \log \sum_{j=1}^N p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)})  \\
& =  \alpha^{-1} \log \frac{e^{-\alpha f_t(\ensuremath{\mathbf x}^{i}_t)}}{\sum_{j=1}^N p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)}}  \\
& =  \alpha^{-1} \log \left(\frac{1}{p^{i}_t} \cdot
\frac{p^{i}_te^{-\alpha f_t(\ensuremath{\mathbf x}^{i}_t)}}{\sum_{j=1}^N p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)}} \right)  \\
& =  \alpha^{-1} \log \frac{\hat{p}^{i}_{t+1}}{p^{i}_t} =  \alpha^{-1} (\log \hat{p}^{i}_{t+1} - \log {p}^{i}_{t} )
\end{aligned}$$ The proof is completed observing that: $$\begin{aligned}
\log p^{i}_t & = \log \left( (1 - \delta)\hat{p}^{i}_{t} + \frac{\delta}{N}  \right) \\
& \geq  \log \hat{p}^{i}_t + \log (1 - \delta ) . %\geq \log \hat{p}^{i}_t  \log(1- \delta)   .
\end{aligned}$$ ◻
:::

Theorem [10.3](#thm:fixed-share){reference-type="ref"
reference="thm:fixed-share"} can now be derived as a corollary:

::: proof
*Theorem [10.3](#thm:fixed-share){reference-type="ref"
reference="thm:fixed-share"}.* By summing up over the interval
$I = [r,s]$, and using the lower bound on $p_t^i$, we have
$$\begin{aligned}
& \sum_{t \in I } f_t(\ensuremath{\mathbf x}_t) - \sum_{t \in I} f_t(\ensuremath{\mathbf x}^{i}_t ) \\
& \leq \sum_{t \in I} \alpha^{-1} (\log \hat{p}^{i}_{t+1} - \log \hat{p}^{i}_{t} - \log (1 - \delta) ) \\
& \leq \frac{1}{\alpha} \left[ \log \frac{1}{\hat{p}^i_r} - |I| \log (1-\delta) \right] \\
& \leq \frac{1}{\alpha} \left[ \log \frac{N}{\delta} + 2 \delta |I| \right] & \hat{p}^i_r \geq \frac{\delta}{N}, \delta < \frac{1}{2} \\
& \leq \frac{1}{\alpha} \log 2 N T + \frac{1}{\alpha}  & \delta = \frac{1}{2T}
\end{aligned}$$ ◻
:::

## Efficient Adaptive Regret for Online Convex Optimization {#sec:basic}

The Fixed-Share algorithm described in the previous section is extremely
practical and efficient for discrete sets of experts. However, to
exploit the full power of OCO we require an efficient algorithm for
continuous decision sets.

Consider for example the problems of online portfolio selection and
online shortest paths: naïvely applying the Fixed-Share algorithm is
computationally inefficient. Instead, we seek an algorithm which takes
advantage of the efficient representation of these problems in the
language of convex programming.

We present such a method called FLH, or Follow the Leading History. The
basic idea is to think of different online convex optimization
algorithms starting at different time points as experts, and apply a
version of Fixed Share to these experts.

::: algorithm
::: algorithmic
Let ${\mathcal A}$ be an OCO algorithm. Initialize $p_1^1 = 1$ Set
$\forall j \leq t \ , \ \ensuremath{\mathbf x}^{j}_t \leftarrow {\mathcal A}(f_j,...,f_{t-1})$
[]{#eqn:shalom12 label="eqn:shalom12"} Play
$\ensuremath{\mathbf x}_t = \sum_{j=1}^t p^{j}_t \ensuremath{\mathbf x}^{j}_t$.
After receiving $f_t$, update for $1 \leq i \leq t$\
$$\hat{p}^{i}_{t+1} = \frac{p^{i}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{i}_t)}}{\sum_{j=1}^t p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)}}$$
Mixing step: set $p^{t+1}_{t+1} = \frac{1}{t+1}$ and
$$\forall i \neq t+1 \ , \ p^{i}_{t+1} = \left(1 - \frac{1}{t+1} \right)\hat{p}^{i}_{t+1} .$$
:::
:::

The main performance guarantee is given in the following theorem.

::: {#thm:main-flh1 .theorem}
**Theorem 10.5**. *Let ${\mathcal A}$ be an OCO algorithm for
$\alpha$-exp-concave loss function with
$\ensuremath{\mathrm{{Regret}}}_T({\mathcal A})$. Then,
$$\mbox{\ensuremath{\mathrm{{AdaptiveRegret}}}}_T(FLH) \leq \ensuremath{\mathrm{{Regret}}}_T({\mathcal A})  + O(\frac{1}{\alpha} \log T) .$$
In particular, taking ${\mathcal A}\equiv ONS$ guarantees
$$\mbox{\ensuremath{\mathrm{{AdaptiveRegret}}}}_T =  O(\frac{1}{\alpha} \log
T) .$$*
:::

Notice that FLH invokes ${\mathcal A}$ at iteration $t$ at most $T$
times. Hence its running time is bounded by $T$ times that of
${\mathcal A}$. This can still be prohibitive as the number of
iterations grows large. In the next section, we show how the ideas from
this algorithm can give rise to an efficient adaptive algorithm with
only $O(\log T)$ computational overhead and slightly worse regret
bounds.

The analysis of FLH is very similar to that of Fixed Share, with the
main subtleties due to the fact that the time horizon $T$ is not assumed
to be known ahead of time, and thus the number of experts varies with
time.

Instead of giving the full analysis, which is deferred to the exercises,
we give a simplified version of FLH which does assume a-priory knowledge
of $T$, and whose analysis can be directly reduced to that of Theorem
[10.3](#thm:fixed-share){reference-type="ref"
reference="thm:fixed-share"}.

::: algorithm
::: algorithmic
Let ${\mathcal A}$ be an OCO algorithm. Set $N=T, \delta= \frac{1}{2T}$.
For all $i \leq t$, set
$\ensuremath{\mathbf x}^{i}_t \leftarrow {\mathcal A}(f_j,...,f_{t-1})$.
Otherwise, set $\ensuremath{\mathbf x}_t^i = \mathbf{0}$. Apply the
Fixed Share algorithm with expert predictions
$\ensuremath{\mathbf x}_t^i$.
:::
:::

The simplified version of FLH is given in Algorithm
[\[alg:simple-flh\]](#alg:simple-flh){reference-type="ref"
reference="alg:simple-flh"}, and it guarantees the following adaptive
regret bound.

::: {#thm:simple-flh .theorem}
**Theorem 10.6**. *Algorithm
[\[alg:simple-flh\]](#alg:simple-flh){reference-type="ref"
reference="alg:simple-flh"} guarantees:
$$\mbox{\ensuremath{\mathrm{{AdaptiveRegret}}}}_T(\mbox{Simple-FLH}) \leq \ensuremath{\mathrm{{Regret}}}_T({\mathcal A})  + O(\frac{1}{\alpha} \log T) .$$*
:::

::: proof
*Proof.* Applying Theorem [10.3](#thm:fixed-share){reference-type="ref"
reference="thm:fixed-share"} to the experts defined in Simple FLH,
guarantees for every interval in time $I = [r,s]$, and by choice of $N$,
for every $i \leq s$, $$\begin{aligned}
\sum_{t \in I } f_t(\ensuremath{\mathbf x}_t) - \sum_{t \in I} f_t(\ensuremath{\mathbf x}^{i}_t ) \leq  \frac{1}{\alpha} \log 2 N T + 1  = O(\frac{1}{\alpha} \log T) .
\end{aligned}$$ In particular, consider the sequence of predictions
given by the $r$'th expert, for which we have $$\begin{aligned}
\sum_{t \in I }  f_t(\ensuremath{\mathbf x}^{r}_t ) = \ensuremath{\mathrm{{Regret}}}_{s-r+1}({\mathcal A}) \leq \ensuremath{\mathrm{{Regret}}}_T({\mathcal A}).
\end{aligned}$$ The theorem now follows since this holds for every
iterval $I \subseteq [T]$. ◻
:::

## \*  Computationally Efficient Methods  {#sec:pruning}

In the previous section we studied adaptive regret, introduced and
analyzed an algorithm that attains near optimal adaptive regret bounds.
However, FLH suffers from a significant computational and memory
overhead: it requires maintaining $O(T)$ copies of an online convex
optimization algorithm. This computational overhead, which is
proportional to the number of iterations, can be prohibitive in many
applications. In this section our goal is to implement the algorithmic
template of FLH efficiently and using little space.

To be more precise, henceforth denote the running time per iteration of
algorithm ${\mathcal A}$ as $V_t({\mathcal A})$. Recall that at time
$t$, FLH stores all predictions
$\{\ensuremath{\mathbf x}_t^i  \ | \  i \in [t]\}$ and has to compute
weights for all of them. This requires running time of at least
$O(V_t({\mathcal A}) \cdot t)$.

The FLH2 algorithm, described in Algorithm
[\[alg:flh2\]](#alg:flh2){reference-type="ref" reference="alg:flh2"},
significantly cuts down this running time to being only logarithmic in
the current time iteration parameter $t$. To achieve this, FLH2 applies
a pruning method to cut down the number of active online algorithms from
$t$ to $O(\log t)$. However, its adaptive regret guarantee is slightly
worse, and suffers a multiplicative factor of $O(\log T)$ as compared to
FLH.

::: algorithm
::: algorithmic
Let ${\mathcal A}$ be an OCO algorithm. Initialize
$p_1^1 = 1, S_1 = \{1\}$

Set
$\forall j \in S_t \ , \ \ensuremath{\mathbf x}^{j}_t \leftarrow {\mathcal A}(f_j,...,f_{t-1})$
Play
$\ensuremath{\mathbf x}_t = \sum_{j \in S_t} p^{j}_t \ensuremath{\mathbf x}^{j}_t$.
[]{#algstep:mw label="algstep:mw"} After receiving $f_t$, perform update
for $i \in S_t$:\
$$\hat{p}^{i}_{t+1} = \frac{p^{i}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{i}_t)}}{\sum_{j \in S_t} p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)}}$$
[]{#algstep:mixing label="algstep:mixing"} Pruning: set
$S_{t+1} \leftarrow \mbox{Prune}(S_t) \cup \{t+1\}$. Set
$\hat{p}^{t+1}_{t+1}$ to $\frac{1}{t}$, and update:
$$\forall i \in S_{t+1} \ . \   p^{i}_{t+1} = \frac{ \hat{p}^{i}_{t+1}} {\sum_{j \in S_{t+1}} \hat{p}^j_{t+1} }$$
:::
:::

Before giving the exact details of this pruning method, we state the
performance guarantee for FLH2.

::: {#thm:flh2 .theorem}
**Theorem 10.7**. *Given an OCO algorithm ${\mathcal A}$ with regret
$\ensuremath{\mathrm{{Regret}}}_T({\mathcal A})$ and running time
$V_T(A)$, algorithm $FLH2$ guarantees:
$V_T(FLH2) \leq V_T({\mathcal A})  \log T$ and
$$\mbox{\ensuremath{\mathrm{{AdaptiveRegret}}}}_T(FLH2) \leq \ensuremath{\mathrm{{Regret}}}_T({\mathcal A}) \log T + O(\frac{1}{\alpha} \log^2 T) .$$*
:::

The main conclusion from this theorem is obtained by using FLH2 with
${\mathcal A}$ being the ONS algorithm from chapter
[4](#chap:second order-methods){reference-type="ref"
reference="chap:second order-methods"}. This gives adaptive regret of
$O(\frac{1}{\alpha} \log^2 T)$ and running time which is polynomial in
natural parameters of the problem and poly-logarithmic in the number of
iterations.

Before diving into the analysis, we explain the main new ingredient. At
the heart of this algorithm is a new method for incorporating history.
We will show that it suffices to store only $O(\log t)$ experts at time
$t$, rather than all $t$ experts as in FLH.

At time $t$, there is a working set $S_t$ of experts. In FLH, this set
can be thought of to contain $E^1,\cdots,E^t$, where each $E^i$ is the
algorithm ${\mathcal A}$ starting from iteration $i$. For the next
round, a new expert $E^{t+1}$ is added to get $S_{t+1}$. The complexity
and regret of FLH is directly related to the cardinality of these sets.

The key to decreasing the sizes of the sets $S_t$ is to also *remove*
(or *prune*) some experts. Once an expert is removed, it is never used
again. The algorithm will perform the multiplicative update and mixing
steps (steps [\[algstep:mw\]](#algstep:mw){reference-type="ref"
reference="algstep:mw"} and
[\[algstep:mixing\]](#algstep:mixing){reference-type="ref"
reference="algstep:mixing"} in algorithm
[\[alg:flh2\]](#alg:flh2){reference-type="ref" reference="alg:flh2"})
only on the working set of experts.

The problem of maintaining the set of active experts can be thought of
as the following abstract data streaming problem. Suppose the integers
$1,2,\cdots$ are being "processed\" in a streaming fashion. At time $t$,
we have "read\" the positive integers up to $t$ and maintain a very
small subset of them in $S_t$. At time $t$ we create $S_{t+1}$ from
$S_t$: we are allowed to add to $S_t$ only the integer $t+1$, and remove
some integers already in $S_t$. Our aim is to maintain a set $S_t$ which
satisfies:

1.  For every positive $s \leq t$,
    $[s,(s+t)/2] \cap S_t \neq \emptyset$.

2.  For all $t$, $|S_t| = O(\log T)$.

3.  For all $t$, $S_{t+1}\backslash S_t = \{t+1\}$.

The first property of the sets $S_t$ intuitively means that $S_t$ is
"well spread out\" in a logarithmic scale. This is depicted in
Figure [10.1](#fig-st){reference-type="ref" reference="fig-st"}. The
second property ensures computational efficiency.

::: center
![Illustration of the working set $S_t$](images/st.pdf){#fig-st
width="3in"}
:::

Indeed, the procedure "Prune\" maintains $S_t$ with these exact
properties, and is detailed after we prove Theorem
[10.7](#thm:flh2){reference-type="ref" reference="thm:flh2"}.

We proceed to prove the main theorem. We start with an analogue of Lemma
[10.4](#lem:round-reg){reference-type="ref" reference="lem:round-reg"}.

::: {#round-reg2 .proposition}
**Proposition 10.8**. *The following holds for all $i \in S_t$,*

1.  *$f_t(\ensuremath{\mathbf x}_t) - f_t(\ensuremath{\mathbf x}^{i}_t) \leq \alpha^{-1} (\log \hat{p}^{i}_{t+1} - \log \hat{p}^{i}_{t} + \log \frac{t-1}{t} )$*

2.  *$f_t(\ensuremath{\mathbf x}_t) - f_t(\ensuremath{\mathbf x}^{t}_t)\leq  \alpha^{-1} (\log \hat{p}^{t}_{t+1} + \log t)$*
:::

::: proof
*Proof.* Using the $\alpha$-exp concavity of $f_t$ - $$\begin{aligned}
e^{-\alpha f_t(\ensuremath{\mathbf x}_t)} & = & e^{-\alpha f_t(\sum_{j \in S_t} p^{j}_t \ensuremath{\mathbf x}^{j}_t)}  \geq  \sum_{j \in S_t} p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)}
\end{aligned}$$ Taking the natural logarithm,
$$f_t(\ensuremath{\mathbf x}_t) \leq  -\alpha^{-1} \log \sum_{j \in S_t} p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)} \nonumber$$
Hence, $$\begin{aligned}
  f_t(\ensuremath{\mathbf x}_t) - f_t(\ensuremath{\mathbf x}^{i}_t)  & \leq  \alpha^{-1}(\log e^{-\alpha f_t(\ensuremath{\mathbf x}^{i}_t)} - \log \sum_{j \in S_t} p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)})  \\
& =  \alpha^{-1} \log \frac{e^{-\alpha f_t(\ensuremath{\mathbf x}^{i}_t)}}{\sum_{j \in S_t} p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)}}  \\
& =  \alpha^{-1} \log \left( \frac{1}{p^{i}_t} \cdot
\frac{p^{i}_te^{-\alpha f_t(\ensuremath{\mathbf x}^{i}_t)}}{\sum_{j \in S_t} p^{j}_t e^{-\alpha f_t(\ensuremath{\mathbf x}^{j}_t)}}\right)  \\
& =  \alpha^{-1} \log \frac{\hat{p}^{i}_{t+1}}{p^{i}_t}
\end{aligned}$$ To complete the proof, we note the following two facts
that are analogous to the ones used in Claim
[10.4](#lem:round-reg){reference-type="ref" reference="lem:round-reg"}:

1.  For $1 \leq i < t$,
    $\log p^{i}_t \geq \log \hat{p}^{i}_t + \log \frac{t-1}{t}$

2.  $\log p^{t}_t \geq -\log t$

Proving these facts is left as an exercise. ◻
:::

Using this we can prove the following Lemma.

::: {#eff-int-reg .lemma}
**Lemma 10.9**. *Consider some time interval $I = [r,s]$. Suppose that
$E^r$ was in the working set $S_t$, for all $t \in I$. Then the regret
incurred in $I$ is at most
$\frac{1}{\alpha} \log (s) + \ensuremath{\mathrm{{Regret}}}_{T}({\mathcal A})$.*
:::

::: proof
*Proof.* Consider the regret in $I$ with respect to expert $E^r$,
$$\begin{aligned}
& \sum_{t = r}^{s} (f_t(\ensuremath{\mathbf x}_t) - f_t(\ensuremath{\mathbf x}^{r}_t)) \\
& =  (f_r(\ensuremath{\mathbf x}_r) - f_r(\ensuremath{\mathbf x}^{r}_r)) + \sum_{t = r+1}^{s} (f_t(\ensuremath{\mathbf x}_t) - f_t(\ensuremath{\mathbf x}^{r}_t)) \nonumber \\
& \leq  \alpha^{-1} \bigl(\log \hat{p}^{r}_{r+1} + \log r + \sum_{t = r+1}^{s} (\log \hat{p}^{r}_{t+1} - \log \hat{p}^{r}_{t} + \log \frac{t}{t-1} )\bigr) & \mbox{ Claim \ref{round-reg2}} \\
& =  \alpha^{-1} (\log r + \log \hat{p}^{r}_{s+1}	+ \sum_{t = r+1}^{s} \log \frac{t}{t-1}) \nonumber \\
& =  \alpha^{-1} (\log (s) + \log \hat{p}^{r}_{s+1}	) \nonumber
\end{aligned}$$

Since $\hat{p}^{r}_{s+1} \leq 1$, $\log \hat{p}^{r}_{s+1} \leq 0$. This
implies that the regret w.r.t. expert $E^r$ is bounded by
$\alpha^{-1} \log (s)$. Since $E^r$ has regret bounded by
$\ensuremath{\mathrm{{Regret}}}_I({\mathcal A}) \leq \ensuremath{\mathrm{{Regret}}}_T({\mathcal A})$
over $I$, the conclusion follows. ◻
:::

Given the properties of $S_t$, we can show that in any interval the
regret incurred is small.

::: {#aflh-reg .lemma}
**Lemma 10.10**. *For any interval $I$ the regret incurred by the FLH2
is at most\
$(\frac{1}{\alpha} \log(s) + \ensuremath{\mathrm{{Regret}}}_{T}({\mathcal A})) (\log_2 |I|+1)$.*
:::

::: proof
*Proof.* Let $|I| \in [2^q, 2^{q+1})$, and denote for simplicity
$R_T = \frac{1}{\alpha} \log(s) + \ensuremath{\mathrm{{Regret}}}_{T}({\mathcal A})$.
We will prove by induction on $q$.

**base case:** For $q=0$ the regret is bounded by
$$f_r(\ensuremath{\mathbf x}_r) \leq \ensuremath{\mathrm{{Regret}}}_T({\mathcal A}) \leq R_T$$

**induction step:** By the properties of the $S_t$'s, there is an expert
$E^i$ in the pool such that $i \in [r,(r+s)/2]$. This expert $E^i$
entered the pool at time $i$ and stayed throughout $[i,s]$. By
Lemma [10.9](#eff-int-reg){reference-type="ref"
reference="eff-int-reg"}, the algorithm incurs regret at most
$R_T = \frac{1}{\alpha} \log (s) + \ensuremath{\mathrm{{Regret}}}_{T}({\mathcal A})$
in $[i,s]$.

The interval $[r,i-1]$ has size at most
$\frac{|I|}{2} \in [2^{q-1},2^q)$, and by induction the algorithm has
regret of at most $R_T \cdot q$ on this interval. This gives a total of
$R_T(q+1)$ regret on $I$. ◻
:::

We can now prove Theorem [10.7](#thm:flh2){reference-type="ref"
reference="thm:flh2"}:

::: proof
*Theorem [10.7](#thm:flh2){reference-type="ref" reference="thm:flh2"}.*
The running time of FLH2 is bounded by $|S_t| \cdot V_T({\mathcal A})$.
Since $|S_t| = O(\log t)$, we can bound the running time by
$O(V_T({\mathcal A}) \log T)$. This fact, together with
Lemma [10.10](#aflh-reg){reference-type="ref" reference="aflh-reg"},
completes the proof. ◻
:::

### The pruning method {#section:streamsoln}

We now explain the pruning procedure used to maintain the set
$S_t \subseteq \{1,2,...,t\}$.

We specify the *lifetime* of integer $i$ - if $i = r2^k$, where $r$ is
odd, then the lifetime of $i$ is $2^{k+2}+1$. Suppose the lifetime of
$i$ is $m$. Then for any time $t \in [i,i+m]$, integer $i$ is *alive* at
$t$. The set $S_t$ is simply the set of all integers that are alive at
time $t$. Obviously, at time $t$, the only integer added to $S_t$ is
$t$ - this immediately proves Property (3). We now prove the other
properties.

::: proof
*Proof.* (Property (1)) We need to show that some integer in
$[s,(s+t)/2]$ is alive at time $t$. This is trivially true when
$t-s < 2$, since $t-1, t \in S_t$. Let $2^\ell$ be the largest power of
$2$ such that $2^\ell \leq (t-s)/2$. There is some integer
$x \in [s,(s+t)/2]$ such that $2^\ell | x$. The lifetime of $x$ is
larger than $2^\ell \times 2 + 1 > t-s$, so $x$ is alive at $t$. ◻
:::

::: proof
*Proof.* (Property (2)) For each $0 \leq k \leq \lfloor \log t \rfloor$,
let us count the number of integers of the form $r2^k$ ($r$ odd) alive
at $t$. The lifetimes of these integers are $2^{k+2}+1$. The only
integers alive lie in the interval $[t-2^{k+2}-1,t]$. Since all of these
integers of this form are separated by gaps of size at least $2^k$,
there are at most a constant number of such integers alive at $t$. In
total, the size of $S_t$ is $O(\log t)$. ◻
:::

## Bibliographic Remarks {#bibliographic-remarks-7}

Dynamic regret bounds for online gradient descent were proposed by
@Zinkevich03, and further studied in [@besbes2015non]. It was shown in
[@zhang2018dynamic] that adaptive regret bounds imply dynamic regret
bounds.

The study of learning in changing environments can be traced to the
seminal work of @HW in the context of tracking for the problem of
prediction from expert advice. Their technique was later extended to
tracking of experts from a small pool [@BW].

The problem of tracking a large set of experts efficiently was studied
using the Fixed-Share technique in
[@singer-portfolios; @asinger-portfolios; @Gyorgy05trackingthe].

The deviation from Fixed-Share to the FLH technique and the notion of
adaptive regret were introduced in @hazan2007adaptive. These techniques
were subject of later study and extensions
[@adamskiy2016closer; @zhang2019adaptive]. @daniely2015strongly study
adaptive regret for weakly convex loss functions and introduced the term
"strongly adaptive\", which differentiates the weakly and strongly
convex settings. They note that FLH is a strongly adaptive algorithm.

The use of an exponential look-back for prediction has roots in
information theory [@WillemsK97; @ShamirM06]. Efficient methods for
streaming, that were used in this chapter to maintaining a small set of
active experts, were studied in the steaming algorithms literature
[@gopalan2007estimating].

Adaptive regret algorithms were motivated by applications involving
changing environments, such as the portfolio selection problem. More
recently they were applied for time series prediction [@anava2013online]
and the control of dynamical systems [@gradu2020adaptive].

## Exercises

# Boosting and Regret {#chap:boosting}

In this chapter we consider a fundamental methodology of machine
learning: *boosting*. In the statistical learning setting, roughly
speaking, boosting refers to the process of taking a set of rough "rules
of thumb" and combining them into a more accurate predictor.

Consider for example the problem of Optical Character Recognition (OCR)
in its simplest form: given a set of bitmap images depicting
hand-written postal-code digits, classify those that contain the digit
"1" from those of "0".

::: center
![Distinguishing zero versus one from a single
pixel](images/mnist.pdf){width="3.3in"}
:::

Seemingly, discerning the two digits seems a difficult task taking into
account the different styles of handwriting, inconsistent styles even
for the same person, label errors in the training data, etc. However, an
inaccurate rule of thumb is rather easy to produce: in the bottom-left
area of the picture we'd expect many more dark bits for "1"s than if the
image depicts a "0". This is, of course, a rather inaccurate classifier.
It does not consider the alignment of the digit, thickness of the
handwriting, and numerous other factors. Nevertheless, as a rule of
thumb - we'd expect better-than-random performance and some correlation
with the ground truth.

The inaccuracy of the crude single-bit predictor is compensated by its
simplicity. It is not hard to implement a classifier based upon this
rule of thumb, which is very efficient indeed. The natural and
fundamental question which now arises is: can several such rules of
thumb be combined into a single, accurate and efficient classifier?

In the rest of this chapter we shall formalize this question in the
statistical learning theory framework. We then proceed to use the
technology developed in this manuscript, namely regret minimization
algorithms for online convex optimization, to answer this question in
the affirmative. Our development will be somewhat non-standard: we'll
describe a black-box reduction from regret-minimization to boosting.
This allows any of the OCO methods previously discussed in this text to
be used as the main component of a boosting algorithm.

## The Problem of Boosting

Throughout this chapter we use the notation and definitions of chapter
[9](#chap:online2batch){reference-type="ref"
reference="chap:online2batch"} on learning theory, and focus on
statistical learnability rather than agnostic learnability. More
formally, we assume the so called "realizability assumption", which
states that for a learning problem over hypothesis class ${\mathcal H}$
there exists some $h^\star \in \mathcal{H}$ such that its generalization
error is zero, or formally $\mathop{\mbox{\rm error}}(h^\star)=0.$

Using the notations of the previous chapter, we can define the following
seemingly weaker notion than statistical learnability.

::: definition
**Definition 11.1** (Weak learnability). *The concept class
${\mathcal H}: {\mathcal X}\mapsto {\mathcal Y}$ is said to be
$\gamma$-weakly-learnable if the following holds. There exists an
algorithm ${\mathcal A}$ that accepts $S_m = \{(\mathbf{x},y)\}$ and
returns an hypothesis in ${\mathcal A}(S_m)  \in {\mathcal H}$ that
satisfies:\
for any $\delta > 0$ there exists $m  = m(\delta)$ large enough such
that for any distribution ${\mathcal D}$ over pairs $(\mathbf{x},y)$,
for $y = h^\star(\ensuremath{\mathbf x})$, and $m$ samples from this
distribution, it holds that with probability $1 - \delta$,
$$\mathop{\mbox{\rm error}}( {\mathcal A}(S_m) ) \leq \frac{1}{2} - \gamma$$*
:::

This is an apparent weakening of the definition of statistical
learnability that we have described in chapter
[9](#chap:online2batch){reference-type="ref"
reference="chap:online2batch"}: the error is not required to approach
zero. The standard case of statistical learning in the context of
boosting is called "strong learnability". An algorithm that achieves
weak learning is referred to as a weak learner, and respectively we can
refer to a strong learner as an algorithm that attains statistical
learning, i.e., allows for generalization error arbitrarily close to
zero, for a certain concept class.

The central question of boosting can now be formalized: are weak
learning and strong learning equivalent? In other words, is there an
(efficient?) procedure that has access to a weak oracle for a concept
class, and returns a strong learner for the class?

Miraculously, the answer is affirmative, and gives rise to one of the
most effective paradigms in machine learning.

## Boosting by Online Convex Optimization

In this section we describe a *reduction* from OCO to boosting. The
template is similar to the one we have used in chapter
[9](#chap:online2batch){reference-type="ref"
reference="chap:online2batch"}: using one of the numerous algorithms for
online convex optimization we have explored in this manuscript, as well
as access to a weak learner, we create a procedure for strong learning.

### Simplification of the setting

Our derivation focuses on simplicity rather than generality. As such, we
make the following assumptions:

1.  We restrict ourselves to the classical setting of binary
    classification. Boosting to real-valued losses is also possible, but
    outside our scope. Thus, we assume the loss function to be the
    zero-one loss, that is: $$\ell(\hat{y}, y) = {
    \left\{
    \begin{array}{ll}
        {0}, & {y = \hat{y}} \\\\
        {1}, & {0/w}
    \end{array}
    \right. }$$

2.  We assume that the concept class is realizable, i.e., there exists
    an $h^\star \in {\mathcal H}$ such that
    $\mathop{\mbox{\rm error}}(h^\star) = 0$. There are results on
    boosting in the agnostic learning setting, these are surveyed in the
    bibliographic section.

3.  We denote the distribution over examples
    ${\mathcal X}\times {\mathcal Y}= \{(x,y)\}$, where
    $y = h^\star(\ensuremath{\mathbf x})$, as a point in
    $\Delta_{{\mathcal X}}$. That is, a point
    $\mathbf{p}\in \Delta_{{\mathcal X}}$ is a non-negative vector that
    integrates to one over all examples. For simplicity, we think of
    ${\mathcal X},{\mathcal Y}$ as a finite, and therefore
    $\mathbf{p}\in \Delta_{m}$ belongs to the $m$ dimensional simplex,
    i.e., is a discrete distribution over $m$ elements.

4.  We henceforth denote the weak learning algorithm by ${\mathcal W}$,
    and denote by ${\mathcal W}(\mathbf{p}, \delta )$ a call to the weak
    learning algorithm over distribution $\mathbf{p}$ that satisfies
    $$\Pr[  \mathop{\mbox{\rm error}}_{\mathbf{p}} ({\mathcal W}(\mathbf{p},\delta))   \geq \frac{1}{2}- \gamma ]  \leq \delta .$$

With these assumptions and definitions we are ready to prove the main
result: a reduction from weak learning to strong learning using an
online convex optimization algorithm with a sublinear regret bound.
Essentially, our task would be to find a hypothesis which attains zero
error on a given sample.

### Algorithm and analysis

Pseudocode for the boosting algorithm is given in Algorithm
[\[alg:boost1\]](#alg:boost1){reference-type="ref"
reference="alg:boost1"}. This reduction accepts as input a $\gamma$-weak
learner and treats it as a black box, returning a function which we'll
prove is a strong learner.

The reduction also accepts as input an online convex optimization
algorithm denoted ${\mathcal A}^{OCO}$. The underlying decision set for
the OCO algorithm is the $m$-dimensional simplex, where $m$ is the
sample size. Thus, its decisions are distributions over examples. The
cost functions are linear, and assign a value of zero or one, depending
on whether the current hypothesis errs on a particular example. Hence,
the cost at a certain iteration is the expected error of the current
hypothesis (chosen by the weak learner) over the distribution chosen by
the low-regret algorithm.

::: algorithm
::: algorithmic
**Input**: ${\mathcal H},\delta$, OCO algorithm ${\mathcal A}^{OCO}$,
$\gamma$-weak learning algorithm ${\mathcal W}$, sample
$S_m \sim {\mathcal D}$. Set $T$ such that
$\frac{1}{T} {\ensuremath{\mathrm{{Regret}}}_T(A^{OCO})} \leq \frac{\gamma}{2}$
Set distribution $\mathbf{p}_1 = \frac{1}{m} \mathbf{1}\in \Delta_m$ to
be the uniform distribution. Find hypothesis
$h_t \leftarrow {\mathcal W}(\mathbf{p}_t ,\frac{\delta}{2T} )$ Define
the loss function $f_t( \mathbf{p}) = \mathbf{r}_t^\top \mathbf{p}$,
where the vector $\mathbf{r}_t \in {\mathbb R}^m$ is defined as
$$\mathbf{r}_t( i) = {
\left\{
\begin{array}{ll}
    {1}, & { h_t(\ensuremath{\mathbf x}_i)=y_i } \\\\
    {0}, & {o/w}
\end{array}
\right. }$$ Update
$\mathbf{p}_{t+1} \leftarrow {\mathcal A}^{OCO} (f_1,...,f_t)$
$\bar{h}(\ensuremath{\mathbf x}) =\text{sign}(\sum_{t=1}^T h_t(\ensuremath{\mathbf x}))$
:::
:::

It is important to note that the final hypothesis $\bar{h}$ which the
algorithm outputs does not necessarily belong to ${\mathcal H}$ - the
initial hypothesis class we started off with.

::: {#thm:boosting-basic .theorem}
**Theorem 11.2**. *Algorithm
[\[alg:boost1\]](#alg:boost1){reference-type="ref"
reference="alg:boost1"} returns a hypothesis $\bar{h}$ such that with
probability at least $1-\delta$,
$$\mathop{\mbox{\rm error}}_S(\bar{h}) =0 .$$*
:::

::: proof
*Proof.* Given $h\in \mathcal{H}$, we denote its empirical error on the
sample $S$, weighted by the distribution $\mathbf{p}\in \delta_m$, by:
$$\begin{aligned}
 \nonumber
\mathop{\mbox{\rm error}}_{S,\mathbf{p}}(h) = \sum_{i=1}^m \mathbf{p}(i) \cdot \mathbf{1}_{ h(\ensuremath{\mathbf x}_i) \neq y_i } .

\end{aligned}$$ Notice that by definition of $\mathbf{r}_t$ we have
$\mathbf{r}_t^\top  \mathbf{p}_t  = 1 - \mathop{\mbox{\rm error}}_{S , \mathbf{p}_t} (h_t)$.
Since $h_t$ is the output of a $\gamma$-weak-learner on the distribution
$\mathbf{p}_t$, we have for all $t \in [T]$, $$\begin{aligned}
 \nonumber
\Pr[ \mathbf{r}_t^\top  \mathbf{p}_t  \leq \frac{1}{2} + \gamma ]  & = \Pr[ 1 - \mathop{\mbox{\rm error}}_{S , \mathbf{p}_t} (h_t) \leq \frac{1}{2}+\gamma ] \\
& = \Pr[  \mathop{\mbox{\rm error}}_{S , \mathbf{p}_t} (h_t) \geq \frac{1}{2}- \gamma ] \\
& \leq \frac{\delta}{2T} .

\end{aligned}$$ This applies for each $t$ separately, and by the union
bound we have
$$\Pr[ \frac{1}{T} \sum_{t=1}^T \mathbf{r}_t^\top  \mathbf{p}_t  \geq \frac{1}{2} + \gamma ]  \geq 1- \delta$$

Denote by $S_\phi \subseteq S$ be the set of all missclassified examples
by $\bar{h}$. Let $\mathbf{p}^*$ the uniform distribution over $S_\phi$.
$$\begin{aligned}
 \nonumber
  \sum_{t=1}^T \mathbf{r}_t^\top \mathbf{p}^* & = \sum_{t=1}^T \frac{1}{|S_\phi|}\sum_{(\ensuremath{\mathbf x},y) \in S_\phi} \mathbf{1}_{h_t(\ensuremath{\mathbf x}) = y}  \\
  & =       \frac{1}{|S_\phi|} \sum_{(\ensuremath{\mathbf x},y) \in S_\phi} \sum_{t=1}^T \mathbf{1}_{h_t(\ensuremath{\mathbf x}_j) = y_j} \\
  &  \leq  \frac{1}{|S_\phi|} \sum_{(\ensuremath{\mathbf x},y) \in S_\phi}  \frac{T}{2} & \mbox{ $\bar{h}(\ensuremath{\mathbf x}_j) \neq y_j $} \\
  & =\frac{T}{2} .

\end{aligned}$$ Combining the previous two observations, we have with
probability at least $1-\delta$ that $$\begin{aligned}
\frac{1}{2} +  \gamma & \leq \frac{1}{T}  \sum_{t=1}^T \mathbf{r}_t^\top  \mathbf{p}_t \\
& \leq  \frac{1}{T}  \sum_{t=1}^T \mathbf{r}_t^\top \mathbf{p}^* + \frac{1}{T}  \ensuremath{\mathrm{{Regret}}}_T({\mathcal A}^{OCO}) & \mbox { low regret of ${\mathcal A}^{OCO}$} \\
& \leq \frac{1}{2} + \frac{1}{T} \ensuremath{\mathrm{{Regret}}}_T({\mathcal A}^{OCO} ) \\
& \leq \frac{1}{2} + \frac{\gamma}{2}   .
\end{aligned}$$ This is a contradiction. We conclude that a distribution
$\mathbf{p}^*$ cannot exist, and thus all examples in $S$ are classified
correctly. ◻
:::

### AdaBoost

A special case of the template reduction we have described is obtained
when the OCO algorithm is taken to be the Multiplicative Updates method
we have come to know in the manuscript.

Corollary [5.7](#cor:eg){reference-type="ref" reference="cor:eg"} gives
a bound of $O(\sqrt{T\log m})$ on the regret of the EG algorithm in our
context. This bounds $T$ in Algorithm
[\[alg:boost1\]](#alg:boost1){reference-type="ref"
reference="alg:boost1"} by $O(\frac{1}{\gamma^2} \log m)$.

Closely related is the AdaBoost algorithm, which is one of the most
useful and successful algorithms in Machine Learning at large (see
bibliography). Unlike the Boosting algorithm that we have analyzed,
AdaBoost doesn't have to know in advance the parameter $\gamma$ of the
weak learners. Pseudo code for the AdaBoost algorithm is given in
[\[alg:adaboost\]](#alg:adaboost){reference-type="ref"
reference="alg:adaboost"}.

::: algorithm
::: algorithmic
**Input**: ${\mathcal H},\delta$, $\gamma$-weak-learner ${\mathcal W}$,
sample $S_m \sim{\mathcal D}$. Set $\mathbf{p}_1 \in \Delta_m$ be the
uniform distribution over $S_m$. Find hypothesis
$h_t \leftarrow {\mathcal W}(\mathbf{p}_t,\frac{\delta}{T})$ Calculate
$\varepsilon_t = \mathop{\mbox{\rm error}}_{S,\mathbf{p}_t}(h_t)$,
$\alpha_t = \frac{1}{2}\log(\frac{1-\varepsilon_t}{\varepsilon_t})$
Update,
$$\mathbf{p}_{t+1}(i) = \frac{\mathbf{p}_t(i) e^{ -\alpha_t y_i h_t(i)} } {\sum_{j=1}^m \mathbf{p}_t(j)e^{-\alpha_t y_j h_t(j)}}$$
$\bar{h}(\ensuremath{\mathbf x}) =\text{sign}(\sum_{t=1}^T \alpha_t h_t(\ensuremath{\mathbf x}))$
:::
:::

### Completing the picture

In our discussion so far we have focused only on the empirical error
over a sample. To show generalization and complete the Boosting theorem,
one must show that zero empirical error on a large enough sample implies
$\varepsilon$ generalization error on the underlying distribution.

Notice that the hypothesis returned by the Boosting algorithms does not
belong to the original concept class. This presents a challenge for
certain methods of proving generalization error bounds that are based on
measure concentration over a fixed hypothesis class.

Both issues are resolved using the implication that compression implies
generalization, as given in Theorem
[9.9](#thm:compression2generalization){reference-type="ref"
reference="thm:compression2generalization"}. We sketch the argument
below, and the precise derivation is left as an exercise.

Roughly speaking, boosting algorithm
[\[alg:boost1\]](#alg:boost1){reference-type="ref"
reference="alg:boost1"} runs on $m$ examples for
$T = O(\frac{\log m}{\gamma^2})$ rounds, returns a final hypothesis
$\bar{h}$ that is the majority vote of $T$ hypothesis, and classifies
correctly all $m$ examples of the training set.

Suppose that the weak learning algorithm has sample complexity of size
$k(\gamma,\delta)$: given $k = k(\gamma,\delta)$ examples, it returns a
hypothesis with generalization error at most $\frac{1}{2} - \gamma$ with
probability at least $1-\delta$. Further, suppose the original training
set of $m$ examples was sampled from distribution ${\mathcal D}$.

Since $\bar{h}$ classifies correctly the entire training set, it follows
that the distribution ${\mathcal D}$ has a compression scheme of size
$$Tk  = O\left( \frac{k(\gamma,\frac{\delta}{T}) \log m}{\gamma^2} \right) .$$

Therefore, using Theorem
[9.9](#thm:compression2generalization){reference-type="ref"
reference="thm:compression2generalization"}, we have that,
$$\mathop{\mbox{\rm error}}_{{\mathcal D}}(\bar{h}) \leq O\left( \frac{k \log^2 \frac{m}{\delta}} {\gamma^2 m} \right) .$$

Now one can obtain an arbitrary small generalization error by choosing
$m$ as a function of $k,\delta,\gamma$. Notice that this argument makes
an assumption only about the sample complexity of the weak learning
algorithm, rather than the hypothesis class ${\mathcal H}$.

## Bibliographic Remarks {#bibliographic-remarks-8}

The theoretical question of Boosting and posed and addressed in the work
of @Schapire90 [@freund1995boosting]. The AdaBoost algorithm was
proposed in the seminal paper of @FreundSch1997. The latter paper also
contains the essential ingredients for the reduction from general
low-regret algorithms to boosting.

Boosting has had significant impact on theoretical and practical data
analysis as described by the statistician Leo Breiman [@Breiman01]. For
a much more comprehensive survey of Boosting theory and applications see
the recent book [@schapire2012boosting].

The theory for agnostic boosting is more recent, and several different
definitions and settings exist, see
 [@kalai2008agnostic; @KalaiS05; @kanade2009potential; @feldman2009distribution; @bendavid2001agnostic],
the most general of which is perhaps by @kanade2009potential.

A unified framework for realizable and agnostic boosting, for both the
statistical and online settings, is given in [@brukhim2020online].

The theory of boosting has been extended to real valued learning via the
theory of gradient boosting [@friedman2002stochastic]. More recently it
was extended to online learning
[@leistner2009robustness; @chen2012online; @chen2014boosting; @beygelzimer2015optimal; @beygelzimer2015online; @agarwal2019boosting; @jung2017online; @jung2018online; @brukhim2020online2].

## Exercises

# Online Boosting {#chap:ocoboost}

This text considers online optimization and learning, and it is a
natural question to ask whether the technique of boosting has an
analogue in the online world? What is a "weak learner\" in online convex
optimization, and how can one strengthen it? This is the subject of this
chapter, and we shall see that boosting can be extremely powerful and
useful in the setting of online convex optimization.

## Motivation: Learning from a Huge Set of Experts

Recall the classical problem of prediction from expert advice from the
first chapter of this text. A learner iteratively makes decisions and
receives loss according to an arbitrarily chosen loss function. For its
decision making, the learner is assisted by a pool of experts. Classical
algorithms such as the Hedge algorithm
[\[alg:Hedge\]](#alg:Hedge){reference-type="ref" reference="alg:Hedge"},
guarantees a regret bound of $O(\sqrt{T \log N} )$, where $N$ is the
number of experts, and this is known to be tight.

However, in many problems of interest, the class of experts is too large
to efficiently manipulate. This is particularly evident in contextual
learning, as formally defined below, where the experts are *policies* --
functions mapping contexts to action. In such instances, even if a
regret bound of $O(\sqrt{T \log N})$ is meaningful, the algorithms
achieving this bound are computationally inefficient; their running time
is linear in $N$. This linear dependence is many times unacceptable: the
effective number of policies mapping contexts to actions is exponential
in the number of contexts.

The boosting approach to address this computational intractability is
motivated by the observation that it is often possible to design simple
*rules-of-thumb* that perform slightly better than random guesses.
Analogously to the weak learning oracles from chapter
[11](#chap:boosting){reference-type="ref" reference="chap:boosting"}, We
propose that the learner has access to an "online weak learner\" - a
computationally cheap mechanism capable of guaranteeing multiplicatively
approximate regret against a base hypotheses class.

In the rest of this chapter we describe efficient algorithms that when
provided weak learners, compete with the convex hull of the base
hypotheses class with near-optimal regret.

### Example: boosting online binary classification

As a more precise example to the motivation we just surveyed, we
formalize online boosting for binary prediction from expert advice. At
iteration $t$, a set of experts denoted $h \in {\mathcal H}$, observe a
context $\mathbf{a}_t$, and predict a binary outcome
$h(\mathbf{a}_t) \in \{-1 ,1 \}$. The loss of each expert is taken to be
the binary loss, $- h(\mathbf{a}_t) \cdot y_t$ for a true label
$y_t \in \{-1,1\}$. The Hedge algorithm from the first chapter applies
to this problem, and guarantees a regret of
$O(\sqrt{T \log |{\mathcal H}|})$ for a finite ${\mathcal H}$. However,
the case in which ${\mathcal H}$ is extremely large, maintaining the
weights is prohibitive computationally.

A weak online learner ${\mathcal W}$ in this setting is an algorithm
which is guaranteed to attain at most a factor $\gamma$ loss from the
best expert in class, for some $\gamma \in [0,1]$, up to an additive
sublinear regret term. Formally, for any sequence of contexts and labels
$\{\mathbf{a}_t,y_t\}$,
$$\sum_{t=1}^T y_t \cdot {\mathcal W}(\mathbf{a}_t)  \le \gamma \cdot \underset{h \in {\mathcal H}}{\min} \sum_{t=1}^T y_t \cdot h(\mathbf{a}_t)  + \ensuremath{\mathrm{{Regret}}}_{T}({\mathcal W}).$$

The online boosting question can now be phrased as follows: given access
to a weak online learning algorithm ${\mathcal W}$, can we design an
efficient online algorithm ${\mathcal A}$ that guarantees vanishing
regret over ${\mathcal H}$? More formally, let
$$\ensuremath{\mathrm{{Regret}}}_T({\mathcal A}) =  \sum_{t=1}^T y_t \cdot {\mathcal W}(\mathbf{a}_t)  -  \underset{h \in {\mathcal H}}{\min} \sum_{t=1}^T y_t \cdot h(\mathbf{a}_t)  .$$

Can we design an algorithm ${\mathcal A}$ that has
$\frac{\ensuremath{\mathrm{{Regret}}}_T({\mathcal A})}{T} \mapsto 0$,
without explicit access to ${\mathcal H}$? As we will see, the answer to
this question is affirmative in a strong sense: boosting does have an
online analogue which is a powerful technique in online learning. In the
next section we describe a more powerful notion of boosting that applies
to the full generality of online convex optimization. This in turn
implies an affirmative answer to this question for online binary
classification.

### Example: personalized article placement

In the problem of matching articles to visitors of a web-page on the
Internet, a number of articles are available to be placed in a given
web-page for a particular visitor. The goal of the decision maker, in
this case the article placer, is to find the most relevant article that
will maximize the probability of a visitor click.

It is usually the case that context is available, in the form of user
profile, preferences surfing history and so forth. This context is
invaluable in terms of placing the most relevant article. Thus, the
decision of the article-placer is to choose from a *policy*: a mapping
from context to article.

The space of policies is significantly larger than the space of articles
and space of contexts: its size is the power of articles to the
cardinality of contexts. This motivates the use of online learning
algorithms whose computational complexity is independent of the number
of experts.

The natural formulation of this problem is not binary prediction, but
rather multi-class prediction. Formulating this problem in the language
of online convex optimization is left as an exercise.

## The Contextual Learning Model

Boosting in the context of online convex optimization is most useful for
the contextual learning problem which we now describe.

Let us consider the familiar OCO setting over a general convex decision
set $\ensuremath{\mathcal K}\subseteq {\mathbb R}^d$, and adversarially
chosen convex loss functions
$f_1,...,f_t : \ensuremath{\mathcal K}\mapsto {\mathbb R}$. Boosting is
particularly important in settings that we have a very large number of
possible experts that makes running one of the algorithms we have
considered thus far infeasible. Concretely, suppose we have access to a
hypothesis class
${\mathcal H}\subseteq \{\mathbf{a}\} \mapsto \ensuremath{\mathcal K}$,
that given a sequence of contexts $\mathbf{a}_1, ...,\mathbf{a}_t$,
produces a new point
$h( \mathbf{a}_{t+1} ) \in \ensuremath{\mathcal K}$.

We have studied numerous methods capable of minimizing regret for this
setting in this text, all assumed that we have access to the set
${\mathcal H}$, and depend on its diameter in some way.

To avoid this dependence, we consider an alternative access model to
${\mathcal H}$. A weak learner for the OCO setting is defined as
follows.

::: {#online_agnostic_wl .definition}
**Definition 12.1**. *An online learning algorithm ${\mathcal W}$ is a
$\gamma$-**weak OCO learner (WOCL)** for ${\mathcal H}$ and
$\gamma \in (0,1)$, if for any sequence of contexts $\{ \mathbf{a}_t \}$
and **linear** loss functions $f_1,...,f_T$, for which
$\max_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} f_t(\ensuremath{\mathbf x}) - \min_{\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}} f_t(\ensuremath{\mathbf y}) \leq 1$
, we have $$\label{eq:wl-stepwise}
 \sum_{t=1}^T f_t({\mathcal W}(\mathbf{a}_t))  \le \gamma \cdot \underset{h \in {\mathcal H}}{\min} \sum_{t=1}^T f_t(h(\mathbf{a}_t))  + (1-\gamma) \sum_{t=1}^T f_t(\bar{\ensuremath{\mathbf x}}) + \ensuremath{\mathrm{{Regret}}}_{T}({\mathcal W}),$$
where
$\bar{\ensuremath{\mathbf x}} = \int_{\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}} \ensuremath{\mathbf x}$
is the center of mass of $\ensuremath{\mathcal K}$.*
:::

This definition differs in two aspects from the types of regret
minimization guarantees we have seen thus far. For one, the algorithm
competes with a $\gamma$-multiple of the best comparator in hindsight,
and is "weak\" in this precise manner.

Secondly, a multiplicative guarantee is not invariant for a constant
shift. This is the reason for the existence of an additional component,
$\sum_t f_t(\bar{\ensuremath{\mathbf x}})$, in the regret bound. This
can be thought of as the cost of a random, or naive predictor. A weak
learner must, at the very least, perform better than this naive and
non-anticipating predictor!

It is convenient to henceforth assume that the loss functions are
shifted such that $f_t(\bar{\ensuremath{\mathbf x}}) = 0$. Under this
assumption, we can rephrase $\gamma$-WOCL as $$\label{eqn:simpleWOLL}
 \sum_{t=1}^T f_t({\mathcal W}(\mathbf{a}_t))  \le \gamma \cdot \underset{h \in {\mathcal H}}{\min} \sum_{t=1}^T f_t(h(\mathbf{a}_t))  + \ensuremath{\mathrm{{Regret}}}_{T}({\mathcal W}).$$

## The Extension Operator

The main difficulty is coping with the approximate guarantee that the
WOCL provides. Therefore the algorithm we describe henceforth scales the
predictions returned by the weak learner by a factor of
$\frac{1}{\gamma}$. This means that the scaled decisions do not belong
to the original decision set anymore, and need to be projected back.

Here lies the main challenge. First, we assume that the loss functions
$f \in {\mathcal F}$ are defined over all of ${\mathbb R}^d$ to enable
valid decisions outside of $\ensuremath{\mathcal K}$. Next, we need to
be able to project onto $\ensuremath{\mathcal K}$ without increasing the
cost. It can be seen that some natural families of functions, i.e.,
linear functions, do not admit any such projection. To remedy this
situation, we define the extension operator of a function over a convex
domain $\ensuremath{\mathcal K}$ as follows.

First, denote the Euclidean distance function to a set
$\ensuremath{\mathcal K}$ as (see also section
[13.2](#sec:approach-dist){reference-type="ref"
reference="sec:approach-dist"}),
$${\bf Dist}(\cdot, \ensuremath{\mathcal K}) \ , \  {\bf Dist}(\ensuremath{\mathbf x},\ensuremath{\mathcal K}) = \min_{\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}} \|\ensuremath{\mathbf y}-\ensuremath{\mathbf x}\| .$$

::: {#defn:extension .definition}
**Definition 12.2**
($(\ensuremath{\mathcal K},\kappa,\delta)$-extension). *The extension
operator over $\ensuremath{\mathcal K}\subseteq {\mathbb R}^d$ is
defined as:
$$X_{\ensuremath{\mathcal K},\kappa,\delta}[f] : {\mathbb R}^d \mapsto {\mathbb R}\ \ , \ \ X[f ]  = S_\delta[  f  + \kappa \cdot {\bf Dist}(\cdot,\ensuremath{\mathcal K})  ] ,$$
where the smoothing operator $S_\delta$ was defined as per Lemma
[2.8](#lem:SmoothingLemma){reference-type="ref"
reference="lem:SmoothingLemma"}.*
:::

The important take-away from these operators is the following lemma,
whose importance is crucial in the OCO boosting algorithm
[\[alg:ocoboost\]](#alg:ocoboost){reference-type="ref"
reference="alg:ocoboost"}, as it projects infeasible points that are
obtained from the weak learners to the feasible domain.

::: {#lem:extensionX .lemma}
**Lemma 12.3**. *The $(\ensuremath{\mathcal K},\kappa,\delta)$-extension
of a function ${\ensuremath{\hat{f}}}= X[f]$ satisfies the following:*

1.  *For every point
    $\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}$, we have
    $\| {\ensuremath{\hat{f}}}(\ensuremath{\mathbf x}) - f(\ensuremath{\mathbf x}) \|_2 \leq \delta G$.*

2.  *The projection of a point, whose gradient is bounded by $G$, onto
    $\ensuremath{\mathcal K}$ improves the
    $(\ensuremath{\mathcal K},\kappa,\delta)$-extension function, for
    $\kappa = G$, value up to a small term,
    $${\ensuremath{\hat{f}}}\left( \prod_\ensuremath{\mathcal K}(\ensuremath{\mathbf x})\right) \leq {\ensuremath{\hat{f}}}(\ensuremath{\mathbf x}) + \delta G .$$*
:::

::: proof
*Proof.*

1.  Since
    ${\bf Dist}(\ensuremath{\mathbf x},\ensuremath{\mathcal K}) = 0$ for
    all $x \in \ensuremath{\mathcal K}$, this follows immediately from
    Lemma [2.8](#lem:SmoothingLemma){reference-type="ref"
    reference="lem:SmoothingLemma"}.

2.  Denote
    $\ensuremath{\mathbf x}_\pi =  \prod_\ensuremath{\mathcal K}(\ensuremath{\mathbf x})$
    for brevity. Then $$\begin{aligned}
        & {\ensuremath{\hat{f}}}(\ensuremath{\mathbf x}_\pi) - {\ensuremath{\hat{f}}}(\ensuremath{\mathbf x}) \\
        & \leq f(\ensuremath{\mathbf x}_\pi) - f(\ensuremath{\mathbf x}) - \kappa  {\bf Dist}(\ensuremath{\mathbf x},\ensuremath{\mathcal K})  + \delta G & \mbox{part 1} \\
        & \leq f(\ensuremath{\mathbf x}_\pi) - f(\ensuremath{\mathbf x}) - \kappa \| \ensuremath{\mathbf x}- \ensuremath{\mathbf x}_\pi\| + \delta G \\
        & = \nabla f(\ensuremath{\mathbf x}) (\ensuremath{\mathbf x}- \ensuremath{\mathbf x}_\pi) - \kappa \| \ensuremath{\mathbf x}- \ensuremath{\mathbf x}_\pi\| + \delta G  \\
        & \leq \| \nabla f(\ensuremath{\mathbf x}) \| \| \ensuremath{\mathbf x}- \ensuremath{\mathbf x}_\pi\| - \kappa \| \ensuremath{\mathbf x}- \ensuremath{\mathbf x}_\pi\| + \delta G & \mbox{Cauchy-Schwartz} \\
        & \leq G  \| \ensuremath{\mathbf x}- \ensuremath{\mathbf x}_\pi\| - \kappa \| \ensuremath{\mathbf x}- \ensuremath{\mathbf x}_\pi\| + \delta G \\
        & \leq \delta  G & \mbox{choice of $\kappa$} .

    \end{aligned}$$

 ◻
:::

## The Online Boosting Method

The online boosting algorithm we describe in this section is closely
related to the online Frank-Wolfe algorithm from chapter
[7](#chap:FW){reference-type="ref" reference="chap:FW"}. Not only does
it deliver in boosting WOCL to strong learning, but it gives an even
stronger guarantee: low regret over the convex hull of the hypothesis
class.

Algorithm [\[alg:ocoboost\]](#alg:ocoboost){reference-type="ref"
reference="alg:ocoboost"} efficiently converts a weak online learning
algorithm into an OCO algorithm with vanishing regret in a black-box
manner. The idea is to apply the weak learning algorithm on linear
functions that are gradients of the loss. The algorithm then recursively
applies another weak learner on the gradients of the residual loss, and
so forth.

::: algorithm
::: algorithmic
Input: $N$ copies of the $\gamma$-WOCL
${\mathcal W}^1, {\mathcal W}^2, \ldots, {\mathcal W}^N$, parameters
$\eta_1,...,\eta_T$,$\delta,\kappa=G$. Receive context $\mathbf{a}_t$,
choose $\ensuremath{\mathbf x}_t^0 = \mathbf{0}$ arbitrarily. Define
$\ensuremath{\mathbf x}_t^i = (1 - \eta_i) \ensuremath{\mathbf x}_t^{i-1} + \eta_i \frac{1}{\gamma} {\mathcal W}^i(\mathbf{a}_t)$.
Predict
$\ensuremath{\mathbf x}_t = \prod_{\ensuremath{\mathcal K}}[ \ensuremath{\mathbf x}_t^N ]$,
suffer loss $f_t(\ensuremath{\mathbf x}_t)$. Obtain loss function $f_t$,
create
${\ensuremath{\hat{f}}}_t = X_{\ensuremath{\mathcal K},\kappa,\delta}[f_t]$.
Define and pass to ${\mathcal W}^i$ the linear loss function $f_t^i$,
$$f_t^i(\ensuremath{\mathbf x}) = \nabla {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}_t^{i-1}) \cdot  \ensuremath{\mathbf x}.$$
:::
:::

However, the Frank-Wolfe method is not applied directly to the loss
functions, but rather to a proxy loss which defined using the extension
operation in [12.2](#defn:extension){reference-type="ref"
reference="defn:extension"}. Importantly, algorithm
[\[alg:ocoboost\]](#alg:ocoboost){reference-type="ref"
reference="alg:ocoboost"} has a running time that is independent of
$|{\mathcal H}|$.

Notice that if $\gamma=1$, the algorithm stills gives a significant
advantage as compared to the weak learner: the regret guarantee is vs.
the convex hull of ${\mathcal H}$, as compared to the best single
hypothesis.

::: {#thm:oco-boost-convex .theorem}
**Theorem 12.4** (Main). *The predictions $\ensuremath{\mathbf x}_t$
generated by
Algorithm [\[alg:ocoboost\]](#alg:ocoboost){reference-type="ref"
reference="alg:ocoboost"} with
$\delta = \sqrt{ \frac{ D^2}{\gamma N}  }, \eta_i = \min \{\frac{2}{i}, 1\}$
satisfy $$\begin{aligned}
\sum_{t=1}^T f_t(\ensuremath{\mathbf x}_t)\ - \min_{h^\star \in \mathbf{CH}({\mathcal H})} \sum_{t=1}^T f_t( h^\star(\mathbf{a}_t) ) \leq   \frac{ 5 d G  D T}{\gamma\sqrt{  N}}  +  \frac{2GD}{\gamma } \ensuremath{\mathrm{{Regret}}}_T({\mathcal W})  .
\end{aligned}$$*
:::

##### Remark 1:

It is possible to obtain tighter bounds by a factor of the dimension,
and other constant terms, using a more sophisticated smoothing operator.
References for these tighter results are given in the bibliographic
section at the end of this chapter.

##### Remark 2:

The regret bound of Theorem
[12.4](#thm:oco-boost-convex){reference-type="ref"
reference="thm:oco-boost-convex"} is nearly as good as we could hope
for. The first term approaches zero as the number of weak learners $N$
grows. The second term is sublinear as the regret of the weak learner.
It is scaled by a factor of $\frac{1}{\gamma}$, which we can expect due
to the approximate guarantee of the weak learner.

Before proving the theorem, let us define some notations we use. The
algorithm defines the extension of the loss functions as
$${\ensuremath{\hat{f}}}_t = X[f_t] = S_\delta [ f_t  +  G\cdot {\bf Dist}(\ensuremath{\mathbf x},\ensuremath{\mathcal K}) ] .$$
We apply the setting of $\kappa=G$, as required by Lemma
[12.3](#lem:extensionX){reference-type="ref"
reference="lem:extensionX"}, and by Lemma
[2.8](#lem:SmoothingLemma){reference-type="ref"
reference="lem:SmoothingLemma"}, ${\ensuremath{\hat{f}}}_t$ is
$\frac{dG }{\delta}$-smooth. Also, denote by
$\mathbf{CH}({\mathcal H}) = \{ \sum_{h \in {\mathcal H}} \mathbf{p}_h  h | \mathbf{p}\in \Delta_{\mathcal H}\}$
the convex hull of the set ${\mathcal H}$, and let
$$h^\star = \mathop{\mathrm{\arg\min}}_{h^\star \in \mathbf{CH}({\mathcal H})}\sum_{t=1}^T f_t(h^\star(\mathbf{a}_t))$$
to be the best hypothesis in the convex hull of ${\mathcal H}$ in
hindsight, i.e., the best convex combination of hypothesis from
${\mathcal H}$. Notice that since the loss functions are generally
convex and non-linear, this convex combination is not necessarily a
singleton. We define
$\ensuremath{\mathbf x}_t^\star = h^\star(\mathbf{a}_t)$ as the
decisions of this hypothesis.

The main crux of the proof is given by the following lemma.

::: {#lem:main-analysis .lemma}
**Lemma 12.5**. *For smoothed loss functions
$\{ {\ensuremath{\hat{f}}}_t \}$ that are ${\beta}$-smooth and $\hat{G}$
Lipschitz, it holds that $$\begin{aligned}
\sum_{t=1}^T {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}_t^N)\ -  \sum_{t=1}^T {\ensuremath{\hat{f}}}_t( \ensuremath{\mathbf x}^\star_t )  \leq  \frac{2 {\beta} D^2 T}{\gamma^2 N} +  \frac{\hat{G}D}{\gamma} \ensuremath{\mathrm{{Regret}}}_T({\mathcal W}) .
\end{aligned}$$*
:::

::: proof
*Proof.* Define for all $i = 0, 1, 2, \ldots, N$,
$$\Delta_i = \sum_{t=1}^T \left({\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}_t^i) - {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}^\star_t)\right) .$$
Recall that ${\ensuremath{\hat{f}}}_t$ is ${\beta}$ smooth by our
assumption. Therefore: $$\begin{aligned}
	  & \Delta_i  =  \sum_{t=1}^T \left[ {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}_t^{i-1} + \eta_i ( \frac{1}{\gamma} {\mathcal W}^i(\mathbf{a}_t) - \ensuremath{\mathbf x}_t^{i-1})) - {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}^\star_t) \right]\\
	\leq & \sum_{t=1}^T \Bigl[ {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}_t^{i-1}) - {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}^\star_t)  + \eta_i \nabla {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}_t^{i-1}) \cdot ( \frac{1}{\gamma} {\mathcal W}^i(\mathbf{a}_t) - \ensuremath{\mathbf x}_t^{i-1}) \\
	& + \frac{\eta_i^2{\beta}}{2} \| \frac{1}{\gamma} {\mathcal W}^i(\mathbf{a}_t) - \ensuremath{\mathbf x}_t^{i-1} \|^2  \Bigr] .

\end{aligned}$$ By using the definition and linearity of $f_t^i$, we
have $$\begin{aligned}
    \Delta_i \leq&  \sum_{t=1}^T \left[{\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}_t^{i-1}) - {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}^\star_t)  +\eta_i ( f_t^i( \frac{1}{\gamma} {\mathcal W}^i(\mathbf{a}_t))  - f_t^i(\ensuremath{\mathbf x}_t^{i-1}))  + \frac{\eta_i^2{\beta} D^2}{2 \gamma^2} \right]  \\
	 =& \Delta_{i-1}  + \sum_{t=1}^T \eta_i ( \frac{1}{\gamma} f_t^i(  {\mathcal W}^i(\mathbf{a}_t))  - f_t^i(\ensuremath{\mathbf x}_t^{i-1})) + \sum_{t=1}^T\frac{\eta_i^2{\beta} D^2}{2 \gamma^2} .
\end{aligned}$$ Now, note the following equivalent restatement of the
WOCL guarantee, which again utilizes linearity of $f_t^i$ to conclude:
linear loss on a convex combination of a set is equal to the same convex
combination of the linear loss applied to individual elements.
$$\begin{aligned}
    \frac{1}{\gamma} \sum_{t=1}^T f_t^i ({\mathcal W}^i(\mathbf{a}_t)) \leq &\min_{h^\star \in {\mathcal H}} \sum_{t=1}^T f_t^i (h^\star (\mathbf{a}_t)) + \frac{\hat{G}D \ensuremath{\mathrm{{Regret}}}_T({\mathcal W})}{\gamma}\\
    =& \min_{h^\star \in \mathbf{CH}({\mathcal H})} \sum_{t=1}^T f_t^i (h^\star(\mathbf{a}_t)) + \frac{\hat{G}D \ensuremath{\mathrm{{Regret}}}_T({\mathcal W})}{\gamma} .
\end{aligned}$$ Using the above and that
$h^\star\in \mathbf{CH}({\mathcal H})$, we have $$\begin{aligned}
     & \Delta_i  \\
     & \leq  \Delta_{i-1}  + \sum_{t=1}^T [ \eta_i \nabla {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}_t^{i-1}) \cdot (\ensuremath{\mathbf x}_t^\star - \ensuremath{\mathbf x}_t^{i-1})   + \frac{\eta_i^2{\beta} D^2}{2 \gamma^2 }     ] + \eta_i \frac{\hat{G}D}{\gamma} \ensuremath{\mathrm{{Regret}}}_T({\mathcal W}) \\
    & \leq  \Delta_{i-1}  (1 -  \eta_i )  + \frac{\eta_i^2{\beta} D^2 T }{2 \gamma^2 }  + \eta_i  {R_T} .
\end{aligned}$$ where the last inequality uses the convexity of
$\hat{f}_t$ and we denote
$R_T = \frac{\hat{G}D}{\gamma} \ensuremath{\mathrm{{Regret}}}_T({\mathcal W})$.
We thus have the recurrence
$$\Delta_i \leq \Delta_{i-1} (1 - \eta_i) +  \eta_i^2 \frac{{\beta} D^2 T }{2 \gamma^2 }  + \eta_i  {R_T} .$$
Denoting $\hat{\Delta}_i = \Delta_i - {R_T}$, we are left with
$$\hat{\Delta}_i \leq \hat{\Delta}_{i-1} (1 - \eta_i) +  \eta_i^2 \frac{{\beta} D^2 T }{2 \gamma^2 }   .$$

This is a recursive relation that can be simplified by applying Lemma
[7.2](#lemma:FW-recursion){reference-type="ref"
reference="lemma:FW-recursion"} from chapter
[7](#chap:FW){reference-type="ref" reference="chap:FW"}. We obtain that
$\hat{\Delta}_N \leq \frac{2 {\beta} D^2 T}{\gamma^2 N}$. ◻
:::

We are ready to prove the main guarantee of Algorithm
[\[alg:ocoboost\]](#alg:ocoboost){reference-type="ref"
reference="alg:ocoboost"}.

::: proof
*Proof of Theorem [12.4](#thm:oco-boost-convex){reference-type="ref"
reference="thm:oco-boost-convex"}.* Using both parts of Lemma
[12.3](#lem:extensionX){reference-type="ref" reference="lem:extensionX"}
in succession, we have $$\begin{aligned}
 \sum_{t=1}^T f_t(\ensuremath{\mathbf x}_t)\ -  \sum_{t=1}^T f_t( \ensuremath{\mathbf x}^\star_t )
& \leq \sum_{t=1}^T {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}_t)\ - \sum_{t=1}^T {\ensuremath{\hat{f}}}_t( \ensuremath{\mathbf x}^\star_t ) + 2 \delta G T \\
& \leq  \sum_{t=1}^T {\ensuremath{\hat{f}}}_t(\ensuremath{\mathbf x}_t^N)\ -  \sum_{t=1}^T {\ensuremath{\hat{f}}}_t( \ensuremath{\mathbf x}^\star_t ) + 3 \delta G T. \label{eqn:shalom4}
\end{aligned}$$ Next, recall by Lemma
[2.8](#lem:SmoothingLemma){reference-type="ref"
reference="lem:SmoothingLemma"}, that ${\ensuremath{\hat{f}}}_t$ is
$\frac{d G}{\delta}$-smooth. By applying Lemma
[12.5](#lem:main-analysis){reference-type="ref"
reference="lem:main-analysis"}, and optimizing $\delta$, we have
$$\begin{aligned}
 \sum_{t=1}^T f_t(\ensuremath{\mathbf x}_t)\ -  \sum_{t=1}^T f_t( \ensuremath{\mathbf x}^\star_t )
& \leq  3 \delta G T +  \frac{2  d G  D^2 T}{\delta \gamma^2 N} +  \frac{\hat{G}D}{\gamma} \ensuremath{\mathrm{{Regret}}}_T({\mathcal W}) \\
& =  \frac{ 5 \sqrt{d} G  D T}{\gamma \sqrt{N}}  +  \frac{\hat{G}D}{\gamma} \ensuremath{\mathrm{{Regret}}}_T({\mathcal W})  \\
& \leq \frac{ 5 d G  D T}{\gamma \sqrt{N}}  +  \frac{\hat{G}D}{\gamma} \ensuremath{\mathrm{{Regret}}}_T({\mathcal W}) ,
\end{aligned}$$ where the last inequality is only to obtain a nicer
expression.

It remains to bound $\hat{G}$, and we claim that $\hat{G} \leq 2G$. To
see this, notice that the function
${\bf Dist}(\ensuremath{\mathbf x},\ensuremath{\mathcal K})$ is
$1$-Lipschitz, since $$\begin{aligned}
&  {\bf Dist}(\ensuremath{\mathbf x}, \ensuremath{\mathcal K}) - {\bf Dist}(\ensuremath{\mathbf y}, \ensuremath{\mathcal K}) \\
& = \|\ensuremath{\mathbf x}-\Pi_\ensuremath{\mathcal K}(\ensuremath{\mathbf x})\|  - \|\ensuremath{\mathbf y}-\Pi_\ensuremath{\mathcal K}(\ensuremath{\mathbf y})\| \\
& \leq \|\ensuremath{\mathbf x}-\Pi_\ensuremath{\mathcal K}(\ensuremath{\mathbf y})\|  - \|\ensuremath{\mathbf y}-\Pi_\ensuremath{\mathcal K}(\ensuremath{\mathbf y})\| & \mbox{ $\Pi_\ensuremath{\mathcal K}(\ensuremath{\mathbf y})\in\ensuremath{\mathcal K}$}\\
& \leq \|\ensuremath{\mathbf x}-\ensuremath{\mathbf y}\| .  & \mbox{ $\Delta$-inequality}
\end{aligned}$$ Thus, by the definition of the extension operator and
the functions $f_t^i$, we have that
$\|\nabla f_t^i(\ensuremath{\mathbf x}_t^i)\|=\|\nabla \hat{f}_t(\ensuremath{\mathbf x}_t^i)\| \leq 2G$. ◻
:::

## Bibliographic Remarks {#bibliographic-remarks-9}

The theory of boosting, which we have surveyed in chapter
[11](#chap:boosting){reference-type="ref" reference="chap:boosting"},
originally applied to binary classification problems. Boosting for
real-valued regression was studied in the theory of gradient boosting by
@friedman2002stochastic.

Online boosting, for both the classification and regression settings was
studied much later
[@leistner2009robustness; @chen2012online; @chen2014boosting; @beygelzimer2015optimal; @beygelzimer2015online; @agarwal2019boosting; @jung2017online; @jung2018online; @brukhim2020online2].
The relationship to the Frank-Wolfe method was explicit in these works,
and also studied in [@10.1214/16-AOS1505; @wang2015functional]. A
framework which encapsulates both agnostic and realizable boosting, for
both offline and online settings, is given in [@brukhim2020online].

Boosting for the full online convex optimization setting, with a
multiplicative approximation and general convex decision set, was
obtained in [@hazan2021boosting]. The latter also give tighter bounds by
a factor of the dimension than those presented in this text using a more
sophisticated smoothing technique known as the Moreau-Yoshida
regularization [@beck2017first].

The contextual experts and bandits problems have been proposed by
@langford2008epoch as a decision making framework with large number of
policies. In the online setting, several works study the problem with
emphasis on efficient algorithms given access to an optimization oracle
[@rakhlin2016bistro; @syrgkanis2016improved; @syrgkanis2016efficient; @rakhlin2016bistro].
For surveys on contextual bandit algorithms and applications of this
model see [@zhou2015survey; @bouneffouf2019survey].

## Exercises

# Blackwell Approachability and Online Convex Optimization {#chap:approach}

The history of adversarial prediction started with the seminal works of
mathematicians David Blackwell and James Hannan. In most of the text
thus far, we have presented the viewpoint of sequential prediction and
loss minimization, taken by Hannan. This was especially true in chapter
[5](#chap:regularization){reference-type="ref"
reference="chap:regularization"}, as the FPL algorithm dates back to his
work. In this chapter we turn to a dual view of regret minimization,
called "Blackwell approachability\". Approachability theory originated
in the work of Blackwell, and was discovered simultaneously to that of
Hannan. A short historical account is surveyed in the bibliographic
materials at the end of this chapter.

For decades the relationship between regret minimization in general
convex games and Blackwell approachability was not fully understood. The
common thought was, in fact, that Blackwell approachability is a
stronger notion. In this chapter we show that approachability and online
convex optimization are equivalent in a strong sense: an algorithm for
one task implies an algorithm for the other with no loss of
computational efficiency.

As a side benefit to this equivalence, we deduce a proof of Blackwell's
approachability theorem using the existence of online convex
optimization algorithms. This proof applies to a more general version of
approachabilty, over general vector games, and comes with rates of
convergence that are borrowed from the OCO algorithms we have already
studied.

While previous chapters had a practical motivation and introduced
methods for online learning, this chapter is purely theoretical, and
devoted to give an alternate viewpoint of online convex optimization
from a game theoretic perspective.

## Vector-Valued Games and Approachability

Von Neumann's minimax theorem, that we have studied in chapter
[8](#chap:games){reference-type="ref" reference="chap:games"},
establishes a central result in the theory of two-player zero-sum games
by providing a prescription to both players. This prescription is in the
form of a pair of optimal mixed strategies: each strategy attains the
optimal worst-case value of the game without knowledge of the opponent's
strategy. However, the theorem fundamentally requires that both players
have a utility function that can be expressed as a *scalar*.

In 1956, in response to von Neumann's result, David Blackwell posed an
intriguing question: what guarantee can we hope to achieve when playing
a two-player game with a *vector-valued payoff*?

A vector-valued game is defined similarly to zero-sum games as we have
defined in Definition [8.2](#defn:zsg){reference-type="ref"
reference="defn:zsg"}, with reward/loss vectors replacing the scalar
rewards/losses.

::: {#defn:vectorgame .definition}
**Definition 13.1**. *A two-player vector game is given by a set of
$n \times m$ vectors
$\{ \ensuremath{\mathbf u}(i,j) \in {\mathbb R}^d \}$. The reward vector
for the row player playing strategy $i \in [n]$, and column player
playing strategy $j \in [m]$, is given by the vector
$\ensuremath{\mathbf u}(i,j) \in {\mathbb R}^d$.*
:::

Similar to scalar games, we can define mixed strategies as distributions
over pure strategies, and denote the expected reward vector for playing
mixed strategies by
$$\forall \ensuremath{\mathbf x}\in \Delta_n , \ensuremath{\mathbf y}\in \Delta_m \ . \ \ensuremath{\mathbf u}(\ensuremath{\mathbf x},\ensuremath{\mathbf y}) = \mathop{\mbox{\bf E}}_{i \sim \ensuremath{\mathbf x}, j \sim \ensuremath{\mathbf y}} \left[ \ensuremath{\mathbf u}(i,j) \right] .$$

We henceforth consider more general vector games than originally
considered in the literature. The additional generality allows for
uncountably many strategies for both players, and allows the strategies
to originate from bounded convex and closed sets in Euclidean space.

::: {#defn:generalizedvectorgame .definition}
**Definition 13.2**. *A generalized two-player vector game is given by a
set of vectors $\{ \ensuremath{\mathbf u}\in {\mathbb R}^d \}$, and two
bounded convex and closed decision sets
$\ensuremath{\mathcal K}_1,\ensuremath{\mathcal K}_2$. The reward vector
for the row player playing strategy
$\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}_1$, and column player
playing strategy $\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}_2$,
is given by the vector
$\ensuremath{\mathbf u}(\ensuremath{\mathbf x},\ensuremath{\mathbf y}) \in {\mathbb R}^d$.*
:::

The goal of a zero-sum game is clear: to guarantee a certain
loss/reward. What should be the vector game generalization? Blackwell
proposed to ask "can we guarantee that our vector payoff lies in some
closed convex set $S$?"

It is left as an exercise at the end of this chapter to show that an
immediate analogue of Von Neumann's theorem does not exist: there is no
single mixed strategy that ensures the vector payoff lies in a given
set. However, this does not rule out an asymptotic notion, if we allow
the game to repeat indefinitely, and ask whether there exists a strategy
to ensure that the *average* reward vector lies in a certain set, or at
least approaches it in terms of Euclidean distance. This is exactly the
solution concept that Blackwell proposed as defined formally below.

Using the notation we have used throughout this text, we denote the
(Euclidean) distance to a bounded, closed and convex set $S$ as
$${\bf Dist}(\mathbf{w},S) = \min_{\ensuremath{\mathbf x}\in S} \|\mathbf{w}-\ensuremath{\mathbf x}\| .$$

::: {#def:approach .definition}
**Definition 13.3**. *Given a generalized vector game
$\ensuremath{\mathcal K}_1,\ensuremath{\mathcal K}_2, \{\ensuremath{\mathbf u}(\cdot,\cdot)\}$,
we say that a set $S \subseteq {\mathbb R}^d$ is **approachable** if
there exists some algorithm ${\mathcal A}$, called an **approachability
algorithm**, which iteratively selects points
$\ensuremath{\mathcal K}_1 \ni \ensuremath{\mathbf x}_{t} \leftarrow {\mathcal A}(\ensuremath{\mathbf y}_{1}, \ensuremath{\mathbf y}_{2}, \ldots, \ensuremath{\mathbf y}_{t-1})$,
such that, for any sequence\
$\ensuremath{\mathbf y}_1, \ensuremath{\mathbf y}_2, \ldots , \ensuremath{\mathbf y}_T \in \ensuremath{\mathcal K}_2$,
we have
$${\bf Dist}\textstyle{\left(\frac 1 T \sum_{t=1}^T \ensuremath{\mathbf u}(\ensuremath{\mathbf x}_t, \ensuremath{\mathbf y}_t), S \right) }\to 0
		\quad \text{ as } \quad
		T \to \infty .$$*
:::

Under this notion, we can now allow the player to implement an adaptive
strategy for a repeated version of the game, and we require that the
average reward vector comes arbitrarily close to $S$. Blackwell's
theorem characterizes which sets in Euclidean space are approachable. We
give it below in generalized form,

::: {#thm:blackwell .theorem}
**Theorem 13.4** (Blackwell's Approachability Theorem). *For any vector
game
$\ensuremath{\mathcal K}_1,\ensuremath{\mathcal K}_2,\{\ensuremath{\mathbf u}(\cdot,\cdot)\}$,
the closed, bounded and convex set $S \subseteq {\mathbb R}^d$ is
approachable if and only if the following condition holds:
$$\forall \ensuremath{\mathbf y}\in \ensuremath{\mathcal K}_2 \ , \ \exists \ensuremath{\mathbf x}\in \ensuremath{\mathcal K}_1 \ , \mbox{s.t. } \ensuremath{\mathbf u}(\ensuremath{\mathbf x},\ensuremath{\mathbf y}) \in S .$$*
:::

The approachability condition spelled out in the equation above is both
necessary and sufficient. The necessity of this condition is left as an
exercise, and the more interesting implication is that any set that
satisfies this condition is, in fact, approachable. Our reductions
henceforth give an explicit proof of Blackwell's theorem, and we leave
it as an exercise to draw the explicit conclusion of this theorem from
the first efficient reduction.

The relationship between Blackwell approachability in vector games and
OCO may not be evident at this point. However, we proceed to show that
the two notions are in fact algorithmically equivalent.

In the next section we show that any algorithm for OCO can be
efficiently converted to an approachability algorithm for vector games.
Following this, we show the other direction as well: an approachability
algorithm for vector games gives an OCO algorithm with no loss of
efficiency!

## From Online Convex Optimization to Approachability {#sec:approach-dist}

In this section we give an efficient reduction from OCO to
approachability. Namely, assume that we have an OCO algorithm denoted
${\mathcal A}$, that attains sublinear regret. Our goal is to design a
Blackwell approachability algorithm for a given vector game and closed,
bounded convex set $S$. Thus, the reduction in this section shows that
OCO is a stronger notion than approachabiliy. This direction is perhaps
the more surprising one, and was discovered more recently, see
bibliographic section for an historical account of this development.

Since we are looking to approach a given set, it is natural to consider
minimizing the distance of our reward vector to the set. Recall we
denote the (Euclidean) distance to a set as
${\bf Dist}(\mathbf{w},S) = \min_{\ensuremath{\mathbf x}\in S} \|\mathbf{w}-\ensuremath{\mathbf x}\|$.
The support function of closed convex set $S$ is given by
$$h_S(\mathbf{w}) = \max_{\ensuremath{\mathbf x}\in S} \{ \mathbf{w}^\top \ensuremath{\mathbf x}\}.$$
Notice that this function is convex, since it is a maximum over linear
functions.

::: {#lem:dist-equivalence .lemma}
**Lemma 13.5**. *The distance to a set can be written as
$${\bf Dist}(\ensuremath{\mathbf u},S) = \max_{\|\mathbf{w}\|\leq 1} \left\{ \mathbf{w}^\top \ensuremath{\mathbf u}- h_S(\mathbf{w}) \right\} .$$*
:::

::: proof
*Proof.* Using the definition of the support function, $$\begin{aligned}
& \max_{\|\mathbf{w}\|\leq 1} \left\{  \mathbf{w}^\top \ensuremath{\mathbf u}- h_S(\mathbf{w}) \right\} \\
 & =
 \max_{\|\mathbf{w}\|\leq 1} \left\{ \mathbf{w}^\top \ensuremath{\mathbf u}- \max_{\ensuremath{\mathbf x}\in S} \mathbf{w}^\top \ensuremath{\mathbf x}\right\} \\
 & = \max_{\|\mathbf{w}\|\leq 1} \min_{\ensuremath{\mathbf x}\in S} \left\{ \mathbf{w}^\top \ensuremath{\mathbf u}-   \mathbf{w}^\top \ensuremath{\mathbf x}\right\} & \mbox{negation}\\
 & =  \min_{\ensuremath{\mathbf x}\in S} \max_{\|\mathbf{w}\|\leq 1} \left\{ \mathbf{w}^\top \ensuremath{\mathbf u}-   \mathbf{w}^\top \ensuremath{\mathbf x}\right\}  & \mbox{minimax theorem} \\
 & = \min_{\ensuremath{\mathbf x}\in S} \| \ensuremath{\mathbf x}- \ensuremath{\mathbf u}\| \\
& =   {\bf Dist}(\ensuremath{\mathbf u},S) .
\end{aligned}$$ ◻
:::

Blackwell's theorem characterizes approachable sets: it is necessary and
sufficient to be able to find a best response
$\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}_1$, to any
$\ensuremath{\mathbf y}\in \ensuremath{\mathcal K}_2$, such that
$\ensuremath{\mathbf u}(\ensuremath{\mathbf x},\ensuremath{\mathbf y}) \in S$.
To proceed with the reduction, we need an equivalent condition stated
formally as follows.

::: {#lem:blackwell-1 .lemma}
**Lemma 13.6**. *For a generalized vector game
$\ensuremath{\mathcal K}_1,\ensuremath{\mathcal K}_2,\{\ensuremath{\mathbf u}\}$,
the following to conditions are equivalent:*

1.  *There exists a feasible best response,
    $$\forall \ensuremath{\mathbf y}\in \ensuremath{\mathcal K}_2 \ , \ \exists \ensuremath{\mathbf x}\in \ensuremath{\mathcal K}_1 \ , \mbox{s.t. } \ensuremath{\mathbf u}(\ensuremath{\mathbf x},\ensuremath{\mathbf y}) \in S .$$*

2.  *For all $\mathbf{w}\in {\mathbb R}^d, \|\mathbf{w}\|\leq 1$, there
    exists $\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}_1$ such
    that
    $$\forall \ensuremath{\mathbf y}\in \ensuremath{\mathcal K}_2 \ \ , \ \  \mathbf{w}^\top \ensuremath{\mathbf u}(\ensuremath{\mathbf x},\ensuremath{\mathbf y}) - h_S(\mathbf{w}) \leq 0 .$$*
:::

::: proof
*Proof.* Consider the scalar zero sum game
$$\min_{\ensuremath{\mathbf x}} \max_{\ensuremath{\mathbf y}}  {\bf Dist}(\ensuremath{\mathbf u}(\ensuremath{\mathbf x},\ensuremath{\mathbf y}) , S) = \lambda .$$
Blackwell's theorem asserts that $\lambda = 0$ if and only if $S$ is
approachable. Using Sion's generalization to the Von Neumann minimax
theorem from chapter [8](#chap:games){reference-type="ref"
reference="chap:games"}, $$\begin{aligned}
& \lambda  = \min_{\ensuremath{\mathbf x}} \max_\ensuremath{\mathbf y}{\bf Dist}(\ensuremath{\mathbf u}(\ensuremath{\mathbf x},\ensuremath{\mathbf y}) , S) \\
& = \min_\ensuremath{\mathbf x}\max_\ensuremath{\mathbf y}\max_{\| {\mathbf{w}} \|\leq 1} \left\{ {\mathbf{w}}^\top \ensuremath{\mathbf u}(\ensuremath{\mathbf x},\ensuremath{\mathbf y}) - h_S ({\mathbf{w}}) \right\} & \mbox{Lemma \ref{lem:dist-equivalence}} \\
& =  \max_{\|{\mathbf{w}}\|\leq 1} \min_\ensuremath{\mathbf x}\max_\ensuremath{\mathbf y}\left\{ {\mathbf{w}}^\top \ensuremath{\mathbf u}(\ensuremath{\mathbf x},\ensuremath{\mathbf y}) - h_S ({\mathbf{w}}) \right\} & \mbox{minimax}
%& \geq   \min_\x \max_\y \left\{ \w^\top \uv(\x,\y) - h_S (\w) \right\}.  & \mbox{particular $\w$ }
\end{aligned}$$ Thus, the second statement of the lemma is satisfied if
and only if $\lambda = 0$. ◻
:::

As mentioned previously, the necessity of Blackwell's condition is left
as an exercise. To prove sufficiency, we assume that form (2) of
Blackwell's condition as in Lemma
[13.6](#lem:blackwell-1){reference-type="ref"
reference="lem:blackwell-1"} is satisfied. Formally, we henceforth
assume that the vector game and set $S$ are equipped with a best
response oracle ${\mathcal O}$, such that $$\label{eqn:blackwell-oracle}
\forall \ensuremath{\mathbf y}\in \ensuremath{\mathcal K}_2 \ , \ \  \mathbf{w}^\top \ensuremath{\mathbf u}( {\mathcal O}(\mathbf{w}),\ensuremath{\mathbf y}) - h_S(\mathbf{w}) \leq 0  .$$

We proceed with the formal proof of sufficiency, constructively
specified in Algorithm
[\[alg:oco2approach\]](#alg:oco2approach){reference-type="ref"
reference="alg:oco2approach"}. Notice that in this reduction, the
functions $f_t$ are concave, and the OCO algorithm is used for
maximization.

::: algorithm
::: algorithmic
Input: generalized vector game
$\ensuremath{\mathcal K}_1,\ensuremath{\mathcal K}_2,\{ \ensuremath{\mathbf u}(\cdot,\cdot)\}$,
set $S$, best response oracle ${\mathcal O}$, OCO algorithm
${\mathcal A}$ Set
$\ensuremath{\mathcal K}=  \mathbb{B}\in {\mathbb R}^d$ to be the unit
Euclidean ball, as decision set for ${\mathcal A}$. set
$f_t(\mathbf{w}) =  \mathbf{w}^\top \ensuremath{\mathbf u}_{t-1}  - h_S(\mathbf{w})$
Query ${\mathcal A}$:
$\mathbf{w}_t \leftarrow {\mathcal A}(f_1, \ldots, f_{t-1})$ Query
${\mathcal O}$:
$\ensuremath{\mathbf x}_t \leftarrow {{\mathcal O}}(\mathbf{w}_t)$
Observe $\ensuremath{\mathbf y}_t$ and let
$\ensuremath{\mathbf u}_t = \ensuremath{\mathbf u}(\ensuremath{\mathbf x}_t,\ensuremath{\mathbf y}_t)$
$\bar{\ensuremath{\mathbf u}}_T = \frac{1}{T} \sum_{t=1}^T \ensuremath{\mathbf u}(\ensuremath{\mathbf x}_t,\ensuremath{\mathbf y}_t)$
:::
:::

::: theorem
**Theorem 13.7**. *Algorithm
[\[alg:oco2approach\]](#alg:oco2approach){reference-type="ref"
reference="alg:oco2approach"}, with input OCO algorithm ${\mathcal A}$,
returns the vector
$\bar{\ensuremath{\mathbf u}}_T = \frac{1}{T} \sum_{t=1}^T \ensuremath{\mathbf u}(\ensuremath{\mathbf x}_t,\ensuremath{\mathbf y}_t)$
that approaches the set $S$ at a rate of
$${\bf Dist}( \bar{\ensuremath{\mathbf u}}_T , S ) \leq \frac{\ensuremath{\mathrm{{Regret}}}_T({\mathcal A})}{T}$$*
:::

::: proof
*Proof.* Notice that equation
[\[eqn:blackwell-oracle\]](#eqn:blackwell-oracle){reference-type="eqref"
reference="eqn:blackwell-oracle"} implies that for $\mathbf{w}_t$ as
defined in the algorithm, we have
$$\forall \ensuremath{\mathbf y}\in \ensuremath{\mathcal K}_2 \ \ , \ \      \mathbf{w}_t^\top \ensuremath{\mathbf u}({\mathcal O}(\mathbf{w}_t),\ensuremath{\mathbf y})  - h_S(\mathbf{w}_t)   \leq 0  .$$
This implies that for any $t$, $$\label{eqn:blackwell-e1}
f_t(\mathbf{w}_t) =    \mathbf{w}_t^\top  \ensuremath{\mathbf u}({\mathcal O}(\mathbf{w}_t),\ensuremath{\mathbf y}_t)  - h_S(\mathbf{w}_t)  \leq 0 .$$

Therefore we have, using Lemma
[13.5](#lem:dist-equivalence){reference-type="ref"
reference="lem:dist-equivalence"}, $$\begin{aligned}
{\bf Dist}(\bar{\ensuremath{\mathbf u}}_T, S) & = \max_{\|\mathbf{w}\|\leq 1} \left\{  \mathbf{w}^\top \bar{\ensuremath{\mathbf u}}_T  - h_S(\mathbf{w}) \right\}  \\
& = \max_{\mathbf{w}^\star \in \ensuremath{\mathcal K}}  \frac{1}{T} \sum_t f_t(\mathbf{w}^\star) & \mbox{definition of $f_t$} \\
& \leq \frac{1}{T} \sum_t f_t(\mathbf{w}_t) + \frac{\ensuremath{\mathrm{{Regret}}}_T({\mathcal A})}{T} & \mbox{OCO guarantee of ${\mathcal A}$} \\
& \leq \frac{\ensuremath{\mathrm{{Regret}}}_T({\mathcal A})}{T} & \mbox{equation \eqref{eqn:blackwell-e1}}
\end{aligned}$$ ◻
:::

This theorem explicitly relates OCO with approachability, and since we
have already proved the existence of efficient OCO algorithms in this
text, it can be used to formally prove Blackwell's theorem. Completing
the details is left as an exercise.

## From Approachability to Online Convex Optimization

In this subsection we show the converse reduction: given an
approachability algorithm, we design an OCO algorithm with no loss of
computational efficiency. This direction was essentially shown by
Blackwell for discrete decision problems, as described in more detail in
the bibliographic section. We prove it here in the full generality of
OCO.

Formally, given an approachability algorithm ${\mathcal A}$, denote by
${\bf Dist}_T({\mathcal A})$ an upper bound on its rate of convergence
to the set $S$ as a function of the number of iterations $T$. That is,
for a given vector game, denote by
$\bar{\ensuremath{\mathbf u}}_T = \frac{1}{T} \sum_{t=1}^T \ensuremath{\mathbf u}(\ensuremath{\mathbf x}_t,\ensuremath{\mathbf y}_t)$
the average reward vector. Then ${\mathcal A}$ guarantees
$${\bf Dist}(\bar{\ensuremath{\mathbf u}}_T,S) \leq {\bf Dist}_T({\mathcal A}) \ , \ \lim_{T \mapsto \infty} {\bf Dist}_T({\mathcal A}) = 0 .$$
Given an approachability algorithm ${\mathcal A}$, we henceforth create
an OCO algorithm with vanishing regret.

### Cones and polar cones

Approachability is in a certain geometric sense a dual to OCO. To see
this, we require several geometric notions, that are explicitly required
for the reduction from approachability to OCO.

For a given convex set $\ensuremath{\mathcal K}\subseteq {\mathbb R}^d$,
we define its cone as the set of all vectors in
$\ensuremath{\mathcal K}$ multiplied by a non-negative scalar,
$$\mbox{cone}(\ensuremath{\mathcal K}) = \{ c \cdot \ensuremath{\mathbf x}\ | \ \ensuremath{\mathbf x}\in \ensuremath{\mathcal K}, 0 \leq c \in {\mathbb R}\} .$$

The notion of a convex cone is not strictly required for the proofs
below, but they are commonly used in the context of approachability. The
polar set to a given set
$\ensuremath{\mathcal K}\subseteq {\mathbb R}^d$ is defined to be
$$\ensuremath{\mathcal K}^0 \stackrel{\text{\tiny def}}{=}\{ \ensuremath{\mathbf y}\in {\mathbb R}^{d} \ \mbox{ s.t. } \  \forall \ensuremath{\mathbf x}\in \ensuremath{\mathcal K}\ , \ \ensuremath{\mathbf x}^\top  \ensuremath{\mathbf y}\leq 0 \} .$$
It is left as an exercise to prove that $\ensuremath{\mathcal K}^0$ is a
convex set, and that for cones, the polar to the polar is the original
set.

Henceforth we need the extension of a convex set defined as follows.
Denote by $1 \oplus \ensuremath{\mathcal K}$ as the direct sum of the
scalar one and the set $\ensuremath{\mathcal K}$, i.e., all vectors of
the form
$\tilde{\ensuremath{\mathbf x}} = 1 \oplus \ensuremath{\mathbf x}$ for
$\ensuremath{\mathbf x}\in \ensuremath{\mathcal K}$. Denote the bounded
polar extension of a set $\ensuremath{\mathcal K}$ by
$$Q(\ensuremath{\mathcal K}) = (1 \oplus \ensuremath{\mathcal K})^0 .$$
That is, we take all points in the polar set to the direct sum
$1 \oplus \ensuremath{\mathcal K}$.

This definition of the polar set gives rise to the following
quantitative characterization.

::: {#lem:dual-dist .lemma}
**Lemma 13.8**. *Let $\ensuremath{\mathbf y}\in {\mathbb R}^{d+1}$ be
such that
${\bf Dist}(\ensuremath{\mathbf y},Q(\ensuremath{\mathcal K})) \leq \varepsilon$.
Then, denoting by $D$ the diameter of $\ensuremath{\mathcal K}$,
$$\forall \tilde{\ensuremath{\mathbf x}} \in 1 \oplus \ensuremath{\mathcal K}\ ,  \ \ensuremath{\mathbf y}^\top \tilde{\ensuremath{\mathbf x}} \leq \varepsilon(D+1) .$$*
:::

::: proof
*Proof.* By definition of distance to a set, we have that
${\bf Dist}(\ensuremath{\mathbf y},Q(\ensuremath{\mathcal K})) \leq \varepsilon$,
implies the existence of a point
$\ensuremath{\mathbf z}\in Q(\ensuremath{\mathcal K})$ such that
$\| \ensuremath{\mathbf y}- \ensuremath{\mathbf z}\| \leq \varepsilon$.
Thus, for all
$\tilde{\ensuremath{\mathbf x}} \in 1 \oplus \ensuremath{\mathcal K}$,
we have $$\begin{aligned}
\ensuremath{\mathbf y}^\top \tilde{\ensuremath{\mathbf x}} & = (\ensuremath{\mathbf y}- \ensuremath{\mathbf z}+ \ensuremath{\mathbf z})^\top \tilde{\ensuremath{\mathbf x}} \\
& \leq \| \ensuremath{\mathbf y}- \ensuremath{\mathbf z}\| \|\tilde{\ensuremath{\mathbf x}} \| + \ensuremath{\mathbf z}^\top \tilde{\ensuremath{\mathbf x}} & \mbox{Cauchy-Schwartz} \\
& \leq \varepsilon\|\tilde{\ensuremath{\mathbf x}} \| + \ensuremath{\mathbf z}^\top \tilde{\ensuremath{\mathbf x}} & \|\ensuremath{\mathbf y}-\ensuremath{\mathbf z}\|\leq \varepsilon\\
& \leq \varepsilon\|\tilde{\ensuremath{\mathbf x}}\| + 0 & \tilde{\ensuremath{\mathbf x}} \in 1 \oplus \ensuremath{\mathcal K}, \ensuremath{\mathbf z}\in (1\oplus \ensuremath{\mathcal K})^0 \\
& \leq \varepsilon(1+D) .
\end{aligned}$$ ◻
:::

### The reduction

Algorithm [\[alg:bwa_to_lra\]](#alg:bwa_to_lra){reference-type="ref"
reference="alg:bwa_to_lra"} takes as an input a Blackwell
approachability algorithm that guarantees, under the necessary and
sufficient condition, convergence to a given set. It also takes as an
input a set $\ensuremath{\mathcal K}$ for OCO.

The reduction considers a vector game with decision sets
$\ensuremath{\mathcal K},{\mathcal F}$ and approachability set
$S =  Q(\ensuremath{\mathcal K})$, and generates a sequence of decisions
that guarantee low regret as we prove next.

Since this reduction creates the approachability set $S$ as a function
of $\ensuremath{\mathcal K}$, we need to prove that indeed the set $S$
is approachable. We show this in the next subsection.

::: algorithm
::: algorithmic
Input: closed, bounded and convex decision set
$\ensuremath{\mathcal K}\subset {\mathbb R}^d$, approachability oracle
${\mathcal A}$. Let: vector game w.
$\ensuremath{\mathcal K}_1 = \ensuremath{\mathcal K}$,
$\ensuremath{\mathcal K}_2 = {\mathcal F}$, and set
$S := Q(\ensuremath{\mathcal K})$.

Query ${\mathcal A}$:
$\ensuremath{\mathbf x}_t \leftarrow {\mathcal A}(f_1, \ldots, f_{t-1})$
Let: ${\mathcal L}(f_1, \ldots, f_{t-1}) := \ensuremath{\mathbf x}_t$
Receive: cost function $f_t$ Construct reward vector
$\ensuremath{\mathbf u}(\ensuremath{\mathbf x}_t ,f_t) := \nabla_t^\top  \ensuremath{\mathbf x}_t  \oplus (-\nabla_t)$
:::
:::

::: {#thm:blackwell_to_olo .theorem}
**Theorem 13.9**. *The reduction defined in
Algorithm [\[alg:bwa_to_lra\]](#alg:bwa_to_lra){reference-type="ref"
reference="alg:bwa_to_lra"}, for any input algorithm ${\mathcal A}$,
produces an OLO algorithm ${\mathcal L}$ such that
$$\ensuremath{\mathrm{{Regret}}}({\mathcal L}) \leq  T (D +1) \cdot {{\bf Dist}_T({\mathcal A})} .$$*
:::

::: proof
*Proof.* The approachability algorithm guarantees
${\bf Dist}( \bar{\ensuremath{\mathbf u}_T} , S ) \leq  {\bf Dist}_T({\mathcal A})$.
Using the definition of $S$ and Lemma
[13.8](#lem:dual-dist){reference-type="ref" reference="lem:dual-dist"}
we have

$$\begin{aligned}
& \forall \ensuremath{\mathbf {\tilde{x}}_{}} \in Q(\ensuremath{\mathcal K}) \ . \ (D+1) \cdot {\bf Dist}_T({\mathcal A}) \\
& \geq  ( \frac{1}{T} \sum_{t=1}^T \ensuremath{\mathbf u}(\ensuremath{\mathbf x}_t, f_t)) ^\top \ensuremath{\mathbf {\tilde{x}}_{}} \\
& \geq ( \frac{1}{T} \sum_{t=1}^T \ensuremath{\mathbf u}(\ensuremath{\mathbf x}_t, f_t)) ^\top (1 \oplus \ensuremath{\mathbf x}^\star )  \\
& =   \frac{1}{T} \sum_{t=1}^T   \nabla_t^\top \ensuremath{\mathbf x}_t  -  \frac{1}{T} \sum_{t=1}^T \nabla_t^\top \ensuremath{\mathbf x}^\star   \\
		& \geq  \frac{1}{T}   \ensuremath{\mathrm{{Regret}}}_T({\mathcal L}),
\end{aligned}$$ where the second inequality holds since the first
inequality holds for every $\ensuremath{\mathbf {\tilde{x}}_{}}$, in
particular for the vector $1 \oplus \ensuremath{\mathbf x}^\star$. ◻
:::

### Existence of a best response oracle

Notice that the reduction of this section from approachability to OCO
does not require the best response oracle. However, Blackwell's
approachability theorem does require this oracle as sufficient and
necessary, and thus for the set $S$ we constructed to be approachable at
all, such an oracle needs to exist. This is what we show next.

Consider the vectors $\ensuremath{\mathbf u}_t$ constructed in the
reduction. A best response oracle finds, for every vector
$\ensuremath{\mathbf y}$, a vector $\ensuremath{\mathbf x}$ that
guarantees
$\ensuremath{\mathbf u}(\ensuremath{\mathbf x},\ensuremath{\mathbf y}) \in S$.
In our case, this translates to the condition
$$\forall f \in {\mathcal F}\ , \ \exists \ensuremath{\mathbf x}\in  \ensuremath{\mathcal K}\ ,  \ \nabla f(\ensuremath{\mathbf x})^\top \ensuremath{\mathbf x}\oplus (- \nabla f(\ensuremath{\mathbf x})) \in  (1 \oplus \ensuremath{\mathcal K})^0 .$$
By definition of the polar set, this implies that for all
$\tilde{\ensuremath{\mathbf x}} \in \ensuremath{\mathcal K}$, we have
$$\nabla f(\ensuremath{\mathbf x})^\top \ensuremath{\mathbf x}- \nabla f(\ensuremath{\mathbf x})^\top \tilde{\ensuremath{\mathbf x}}  \leq 0 .$$
In other words, the best response oracle corresponds to a procedure that
given $f$, finds a vector $\ensuremath{\mathbf x}^\star$ such that
$$\forall \ensuremath{\mathbf x}\in \ensuremath{\mathcal K}\ . \ f(\ensuremath{\mathbf x}^\star) - f(\ensuremath{\mathbf x}) \leq \nabla f(\ensuremath{\mathbf x}^\star)^\top (\ensuremath{\mathbf x}^\star - \ensuremath{\mathbf x}) \leq 0 .$$
This is an optimization oracle for the set $\ensuremath{\mathcal K}$!

## Bibliographic Remarks {#bibliographic-remarks-10}

David Blackwell's celebrated Approachability Theorem was published in
[@blackwell_analog_1956]. The first no-regret algorithm for a discrete
action setting was given in a seminal paper by James Hannan in
[@Hannan57] the next year. That same year, Blackwell pointed out
[@blackwell1954controlled] that his approachability result leads, as a
special case, to an algorithm with essentially the same low-regret
guarantee proven by Hannan. For Hannan's account of events see
[@gilliland2010conversation].

Over the years several other problems have been reduced to Blackwell
approachability, including asymptotic calibration
[@foster_asymptotic_1998], online learning with global cost functions
[@even-dar_online_2009] and more [@mannor2008regret]. Indeed, it has
been presumed that approachability, while establishing the existence of
a no-regret algorithm, is strictly more powerful than
regret-minimization; hence its utility in such a wide range of problems.

However, this was recently shown not to be the case.
@abernethy2011blackwell showed that approachability is in fact
equivalent to OCO. This result is the basis of the material presented in
this chapter. One side of their reduction was simplified and generalized
in [@shimkin2016online].

## Exercises