Convex Interpolation and Performance Estimation ofFirst-order Methods for Convex Optimization

Adrien Taylor

Thesis submitted in partial fulfillment of the requirements for the degree ofDocteur en sciences de l’ingénieur

Dissertation committee:
Coralia Cartis (Oxford University)
Etienne de Klerk (Tilburg University and Delft University of Technology)
François Glineur (Université catholique de Louvain, advisor)
Julien Hendrickx (Université catholique de Louvain, advisor)
Philippe Lefèvre (Université catholique de Louvain, chairman)
Yurii Nesterov (Université catholique de Louvain)Abstract
The goal of this thesis is to show how to derive in a completely automatedway exact and global worst-case guarantees for first-order methods in convex
optimization. To this end, we formulate a generic optimization problem looking
for the worst-case scenarios.
The worst-case computation problems, referred to as performance estimation
problems (PEPs), are intrinsically infinite-dimensional optimization problems
formulated over a given class of objective functions. To render those problems
tractable, we develop (smooth and non-smooth) convex interpolation framework, which provides necessary and sufficient conditions to interpolate ourobjective functions.
With this idea, we transform PEPs into solvable finite-dimensional semidefinite programs, from which one obtains worst-case guarantees and worst-casefunctions, along with the corresponding explicit proofs.
PEPs already proved themselves very useful as a tool for developing convergence analyses of first-order optimization methods. Among others, PEPs allow
obtaining exact guarantees for gradient methods, along with their inexact, projected, proximal, conditional, decentralized and accelerated versions.

iii

Acknowledgements
A thesis is a long journey that can hardly be completed without support. I wasvery lucky for the work and personal environment I have been living in during
those last years, and I wish to thank the many persons who contributed to it.
First of all, I would like to thank my two advisers, François and Julien, for givingme the opportunity to work within the mathematical engineering department,
for their support and guidances. Also, I am very grateful to Moritz Diehl,
Etienne de Klerk, Coralia Cartis, Yurii Nesterov and Philippe Lefèvre for theirinsightful feedbacks, and for the time they spent for me. In addition, I am
greatly indebted to other researchers I have met during those last years, for all
the scientific discussions, motivations and advise. Among others, big thanks to
Matthew Philippe, Romain Hollanders, François Gonze, Pierre-Yves Chevalier,Andrea Simonetto, Nicolas Boumal, Ion Necoara and Frank Iutzeler.
I wish to thank the colleagues and friends with whom I had the pleasure tospend time at the office. Among the many others, my two former officemates
Pyc and François certainly deserve a particular attention for bearing my (often
unstoppable) long speeches. Also, I could not emphasize enough the importanceof the technical and administrative staff of the department and of all the timethey spend making the Euler so enjoyable: thanks to Isabelle, Marie-Christine,
Nathalie, Carine, Etienne, François and Ludovic.
Those last years, I also had the chance to live in those great places that werele Petit-Ry and la rue Philippe Baucq, with passionate persons sharing similar
convictions, ways, jobs, and hobbies; for that, huge thanks to Bouny, Antoinette, Alex, Ade, Bob, Anais, Laulau, Tom, Coco, Juju (J.), Oli, Théthé,Pyf, Ju (C.), Ol., Delphine, Vio, Adri (D.), Stéphanie and Eliane.
I would also like to thank different groups of persons who contributed in one
way or another to the accomplishment of this work, through support, drinks,
sport, music, board games, robotic design, holidays, debates, or simply by enjoying good time together. Among them, I wish to particularly thank Adri (C.),
Gauthier, Nono, Ludi, Thomas (R.), Bigno, Coin, Emilie, Carl, Barnab, Aileen,
Marie, Mehdi, Roxane, Ben and Thomas (G.) for their constant enthusiasm,
smiles and precious advise. Finally, I could not finish those lines without thanking the old-timers Gi, Jeje and Domi for their caring presence along the years.
Pour finir, un énorme merci à Emily et Jojo, pour leur patience, leurs attentions, et pour le fait de toujours parvenir à supporter après toutes ces années.
Un grand merci également à mes parents Sabine et Olivier, ainsi qu’à leurs compagnons respectifs Pascal et Maria-Paola, pour leurs oreilles attentives, leursconseils et leur bienveillance.

iii

iv

Contents
1 Introduction
1.1 What is optimization ? . . . . . . . . . . . . . . . . . . . . . . .
1.2 Recent trends in large-scale optimization . . . . . . . . . . . . .
1.3 Worst-case analyses of numerical schemes . . . . . . . . . . . .
1.4 Organization of the thesis . . . . . . . . . . . . . . . . . . . . .
1.5 Overview of problem classes and algorithms . . . . . . . . . . .

1
1
2
4
11
12

I

Discrete Representations of Convex Functions

13

2 Elements of Convex Analysis
2.1 Spaces, norms and scalar products . . . . . . . . . . . . . . . .
2.2 Convex sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Convex functions . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 Legendre-Fenchel conjugation . . . . . . . . . . . . . . . . . . .
2.5 Functional classes . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6 Local and global smoothness . . . . . . . . . . . . . . . . . . .

15
16
17
18
22
26
38

3 Convex Interpolation
3.1 Problem and motivations . . . . . . . . . . . . . . . . . . . . .
3.2 Motivating counterexamples . . . . . . . . . . . . . . . . . . . .
3.3 Convex interpolation . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Interpolation without function values . . . . . . . . . . . . . . .
3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

43
44
45
47
61
65

II

67

Performance Estimation Problems

4 Performance Estimation Problems
4.1 Introduction to performance estimation . . . . . . . . . . . . .
4.2 A convex formulation for performance estimation . . . . . . . .
4.3 Study of standard first-order methods . . . . . . . . . . . . . .
4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
v

69
70
74
85
103

Contentsvi

Appendices
105
4.A Tight worst-case of a gradient step . . . . . . . . . . . . . . . . 105
5 Performance Estimation Problems for Composite Convex Optimization
109
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.2 Performance estimation framework for first-order methods . . . 114
5.3 Algorithm analysis . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Appendices
137
5.A Proof of Theorem 5.13 . . . . . . . . . . . . . . . . . . . . . . . 137
5.B Proof of Theorem 5.15 . . . . . . . . . . . . . . . . . . . . . . . 141
6 Steepest Descent with Exact Line Search147
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
6.2 Background results . . . . . . . . . . . . . . . . . . . . . . . . . 151
6.3 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
6.4 Noisy Search Directions . . . . . . . . . . . . . . . . . . . . . . 156
6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
7 Proximal Gradient Method
163
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
7.2 Convergence in distance, gradient and function accuracy . . . . 167
7.3 Mixed performance measures . . . . . . . . . . . . . . . . . . . 177
7.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

III

Conclusion

181

8 Further Developments in Performance Estimation
183
8.1 Monotone operators and splitting methods . . . . . . . . . . . . 184
8.2 Further developments . . . . . . . . . . . . . . . . . . . . . . . 188
8.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
9 Research Outcomes and Perspectives
Bibliography

193
199

vii

Contents

New answers to simple questions
This section is intended for the informed readers: we present a few spoilerson (perhaps surprising) results taken from the following chapters, expressed intheir simplest forms. In order to do that, let f be a smooth (strongly) convex
function, and consider the corresponding unconstrained minimization problem
min f (x).

x∈Rd

Given an iterative algorithm and an initial point x0 , the performance estimation
problem that is the main focus of this thesis consists in identifying the functions
achieving the worst value of objective function accuracy after N iterations, i.e.
that maximize f (xN ) − f (x∗ ).
What are the worst-case functions for the fixed-step gradient, fast gradient and optimized gradient methods in smooth convex unconstrained
minimization ?
One-dimensional Huber losses and quadratics. (Chapter 4)
For smooth strongly convex unconstrained minimization with the gradientmethod, is there a benefit of using exact line search as compared to fixed
step sizes ?
With an appropriate step size, the worst-cases are the same.
(Chapter 6)
In smooth strongly convex unconstrained minimization, how does theworst-case guarantee change when inaccurate search directions are tolerated within an exact line search steepest descent scheme ?
Linear convergence is preserved, albeit with a worsened condition
number. (Chapter 6)
When analyzing the global worst-case guarantee of simple first-order methods, is there an upper bound on the number of (in)equalities required in
the proof ?
Yes, we only need to consider interpolation conditions (from the classof problems under consideration). The number of such conditions
required to write a tight convergence proof depends on the numberof iterations, and on the class of functions of interest. It is a priori O(N 2 ) for non-smooth and smooth (strongly) convex problems.
(Chapter 4 and Chapter 5)

Contents

viii

Chapter 1

Introduction
1.1

What is optimization ?

Mathematical optimization is the science of making decisions in a rational way.
It drives many of today’s theories in physics (for example least-action principles dictating equations of motions, or physical systems going to their states ofminimal energy), in biology (Darwin’s theory of evolution favoring survival ofindividuals with most beneficial traits) or even in economy (individual behaviors are predicted by assuming they maximize their utilities).
Due to the rise of larger computational capabilities, optimization has also become a very powerful tool for making decisions in practice. Formally, the aimof mathematical optimization is to chose some parameter x within a set X suchthat f (x) is as small as possible:
min f (x).

x∈X

(OPT)

Writing an optimization problem in this form is referred to as modelling. A lotof practical applications can be modelled and are actually being solved using
this formalism — we refer to [RM09, SNW12, PE10, BV04] for examples of uses
in automatic control, machine learning, signal processing, inverse problems,
statistical modelling and other engineering applications.
Although very general, the modelling process should never be underestimated.
In fact, almost every problem in our daily life can be modelled in the optimization framework. However, there are relatively few instances that we canactually solve in practice, as there exists no universal way to solve (OPT)within a reasonable time. That is, even by adding some assumptions (e.g.,differentiability), it is generally possible to design problems of the form (OPT)such that the computational cost for solving them is just too high to be even1CHAPTER 1. INTRODUCTION

2

considered. Therefore, it is commonly admitted that a good modelling of the
problem is usually the key for solving it, even approximately.
As there is no guarantee for solving (OPT) in general, optimizers commonly
add assumptions on both X and f in order to design algorithms with working
certifications. Those assumptions should be motivated in two aspects: on the
one hand they should allow for designing efficient algorithms for solving thecorresponding (OPT), and on the other hand, they should contain a sufficiently
rich class of problems to embed as many applications as possible — those areof course conflicting goals, but both aspects are nevertheless essential.
In what follows, we focus on continuous optimization problems, that is, when
variables are allowed to take their values within a continuum. More precisely,
we mostly focus on the case where X is a convex set, and f a convex function.
Those assumptions are motivated on the one hand by a variety of applications(see e.g., [PE10, BV04] and the references therein), and on the other handby strong theoretical guarantees. Also, we focus on the class of first-order
optimization algorithms, which proved themselves particularly successful inthe context of large-scale decision making, that is, when the decision space X
is high-dimensional (more details follow in the next sections).
The goal of this thesis is to provide a framework for automatically analyzing
the worst-case performances of first-order methods in convex optimization. We
devote the next section to briefly describe the broader context in which thoseanalyses take place.

1.2

Recent trends in large-scale optimization

Since the beginning of the modern computational optimization era, starting
after the second world war with among others the work of Dantzig on linear
programming and the simplex method [Dan98], optimization has undergone
several paradigm shifts. This resulted in a vast literature on large families ofalgorithms tailored for particular classes and instances of (OPT).
In the context of continuous optimization, two families of algorithms are particularly used: methods using first-order derivatives (gradients) on the one hand,and methods using second-order derivatives (Hessian) on the other one. In thesequel, we only treat first-order methods; however, depending on the specificapplication, it may be much more appropriate to use second-order schemes.
 Second-order (or Newton) methods are usually the algorithms of choicefor obtaining highly-accurate solutions (see e.g., [NW06, BV04]). However, this usually comes at the price of large iteration costs.
Numerous numerical schemes were developed for trying to alleviate thecomputational burden coming with second-order information, while keeping their accuracy. For example, it is common to use approximate second-3

1.2. RECENT TRENDS IN LARGE-SCALE OPTIMIZATION

order information or to exploit specific problem structures (e.g., [NW06,
Gon12b, FG16]). In convex optimization, second-order schemes are particularly studied for interior-point methods (see e.g., [Gon12a, Ren01,
NN94]).
 On the other hand, much attention is currently being given to first-ordermethods. Those methods have an old history, but were usually not viewed
as methods of choice, especially due to their difficulties in obtaining very
accurate solutions. Contrasting with second-order methods, first-ordermethods generally benefits from a much cheaper cost per iteration.
First-order methods are prominent in today’s computational optimization
practices. This is due to their low iteration cost, but also to a quantityof practical large-scale problems that do not actually require accurate
solutions. For example, a lot of objective functions arising in machinelearning, signal and image processing problems can seen as approximations (see e.g., [BB08]). For those problems, it is unnecessary to obtainsolutions which are more accurate than the objectives themselves.
The previous points motivated a lot of different research tracks for renderinglarge-scale computations computationally feasible and as light as possible. Inthat direction, much attention is currently being given to structural optimization (we borrow this term from [Nes04, Nes08]), whose focus is to exploit theparticular structure of common optimization problems for designing efficient
tailored methods. In that context, very common examples include: block-coordinate descent schemes which treat a subset of the variables at
a time in order to maintain cheap iteration costs (see e.g. [RT14, FR15,
Nes12a]). Those methods are particularly suited for problems with low
interactions between blocks of variables.
 Incremental or stochastic gradient descent schemes, which use partial
knowledge on the objective function at each iteration (see e.g., [Ber10,
SSBD14, Kiw04] and [RM51] for the original presentation). Those methods are designed for objective functions written as averages of simple
functions.
 Distributed and/or decentralized methods, which use particular structures in order to split the optimization process among different computational units, either in order to lighten the computational burden of asingle unit, or because one can not rely on a central computational unit
(see e.g., [BPC+ 11, NO09]).
 Numerous other techniques exist for improving either the convergence
rates and/or the cost per iteration. As examples, we cite smoothing
techniques (see e.g., [DGN12, BT12, Nes05]) whose aim is to improve theconvergence rates of first-order schemes on well-structured non-smoothproblems; and sparsity-exploiting methods that may render iteration cost
sublinear in the dimension of the problem (see e.g., [Nes14]).

CHAPTER 1. INTRODUCTION

1.3

4

Worst-case analyses of numerical schemes

The main contributions of this work take place in the context of worst-caseanalyses of first-order optimization algorithms. Our goal is to obtain global
guarantees on various measures of accuracy achieved by optimization methods,
for a given computational cost. Those guarantees are essential as they allow
comparing and choosing the most appropriate methods for solving (OPT).
Comparing the efficiency of optimization algorithms can be carried out in avariety of different manners. In this work, we focus on the concepts of absoluteinaccuracy and efficiency (or worst-case inaccuracy). For that purpose, we
consider three ingredients:
 a class of problems instances P containing the problems of interest (f, X).
 A method M that produces an approximate solution xM (f,X) when it
is given a problem instance (f, X) ∈ P (i.e., M has a built-in stopping
criterion).
 A performance measure, typically f (xM (f,X) ) − f (x∗ ) withx∗ = argmin f (x)x∈X(for other performance measures, see Chapter 4 and Chapter 5).
From those elements, we define the absolute inaccuracy of method M on an
instance (f, X) as the value of the performance measure evaluated at the outputof M :ε(M (f, X), f, X) = f (xM (f,X) ) − f (x∗ ).
In the same way, the efficiency of M is defined as its worst-case absoluteinaccuracyε(M, P) = sup ε(M (f, X), f, X).
(f,X)∈P

The efficiency allows comparing methods in terms of the worst accuracy of
their output on any instance of P. However, comparing methods solely on
basis of their efficiencies is not fair so far. Indeed, the computational cost of
two methods may be very different, and comparing efficiencies only make sense
for methods with similar costs.
In the context of first-order methods, the main computationally demanding
steps performed by the optimization algorithms are usually the evaluations offunction values f (x) and gradients ∇f (x) at different points. Therefore, thecomputational cost is commonly modelled as proportional to the number offunction and gradient evaluations (this is usually approached using the conceptof black-box oracle, which is made more precise in Chapter 4 and Chapter 5).
Using this model, we are able to rigorously compare methods using their computational costs and their corresponding efficiencies. That is, given a number of

5

1.3. WORST-CASE ANALYSES OF NUMERICAL SCHEMES

evaluations, a method M1 performs better than a method M2 if the worst-caseinaccuracy of M1 is smaller than that of M2 , i.e., when ε(M1 , P) ≤ ε(M2 , P).
Performing a global worst-case analysis of a numerical method consists in characterizing the evolution of its efficiency as a function of its computational cost.
Worst-case absolute accuracies are expected to be decreasing functions of thecomputational cost, and the corresponding decrease rates are usually referred
to as the global convergence rates.
A standard alternative to the viewpoint taken in the thesis is to consider thecomputational cost as a function of the required accuracy (see e.g., [NY83]).
This is often referred to as complexity analysis. For example, given an iterative
method M (we denote by M (., .; N ) the method stopped after N iterations)whose computational cost is proportional to the number of iterations N , theaim of global worst-case analyses is to obtain a guarantee on the worst-caseabsolute accuracy ε(M (., .; N ), P) (as a function of N ). On the other hand,complexity analysis focuses on obtaining guarantees on the computational cost
(number of iterations) required to achieve a certain accuracy  > 0:
Nmin (M, P; ) = min{N : ε(M (f, X; N ), f, X) ≤  ∀(f, X) ∈ P}.
In the sequel, we use the worst-case analysis point of view, but all results canbe transposed in terms of complexity analysis.
Classical approaches to convergence analysis are covered in details in numerous seminal references, such as the books of Yudin and Nemirovski [NY83],
Polyak [Pol87], Nesterov [Nes04] and the more recent book of Bertsekas [Ber15].

1.3.1

Novel methodologies for global worst-case analyses

As previously underlined, this work is about computing the worst-case absoluteinaccuracy of first-order optimization methods. That is, denoting by M somefirst-order method performing N function and gradient evaluations and by P
the class of optimization problems (f, X) of interest, we want to computeε(M, P) =sup ε(M (f, X), f, X)(PEP)(f,X)∈P

The idea of solving (PEP) rose through the work of Drori and Teboulle [DT14].
We believe the main achievements in the development of this novel approach
are the following (in historical ordering).
 Drori and Teboulle [DT14] introduced the idea of performance estimation
problems for smooth unconstrained convex minimization. The idea is touse semidefinite programming to solve relaxations of the worst-case computation problem (PEP). This allowed to automatically obtain upperCHAPTER 1. INTRODUCTION

6bounds on (PEP) for various optimization schemes in a fully computational way.
 In the same work, Drori and Teboulle [DT14] managed to devise a new
method, by numerically optimizing its worst-case (more precisely, upperbounds on its worst-case).
 Kim and Fessler [KF16d] managed to find an analytical form for the
optimized gradient method (OGM) obtained by Drori and Teboulle. This
scheme is shown to have twice better theoretical guarantees compared
to standard accelerated methods. However, as only upper bounds wereinvolved so far, it was impossible to conclude whether OGM was optimal
or not.
 In parallel, Lessard, Recht and Packard [LRP16] proposed a cheaper
methodology based on control theory for analyzing first-order schemes.
This methodology is particularly suited for studying linear convergence
rates, and involves semidefinite programs of much smaller dimensions (seediscussion in the following section).
 We (T., Hendrickx and Glineur [THG16a]) proposed a generic way forformulating and solving the performance estimation problems of Drori
and Teboulle [DT14] in order to have guaranteed tight results. The new
methodology also automatically generates worst-case functions, and not
only upper bounds. The main difference with Drori and Teboulle’s original approach is essentially threefold: first, we rely on convex interpolation, which allows working with finite versions of the convex functionsof interest with tightness guarantees. Second, a lifting procedure allows
us to work in the primal space (space of functions) and to keep a verytransparent approach not relying on multiple relaxations and dualizations. Finally, our approach naturally generalizes to a broader class ofproblem classes (e.g., involving constraints and non-smooth terms) andmethods.
In the next sections, we survey contributions to the performance estimation
framework and classify them in three categories. First, we summarize contributions aiming at developing the methodology itself. Then, we survey newly
developed algorithms based on the performance estimation framework. Finally, we list other contributions related convergence analyses of optimization
methods due to performance estimations.
As this field is very young, this survey is essentially exhaustive. Our main
contributions are emphasized throughout the summary.

1.3.2

Survey on the performance estimation framework

Drori and Teboulle [DT14] were first to consider the notion of a performance
estimation problem. They focus exclusively on the case of smooth convex

7

1.3. WORST-CASE ANALYSES OF NUMERICAL SCHEMES

functions equipped with the performance criterion f (xN ) − f∗ , and introduce
the idea of reducing (PEP) to a finite-dimensional problem involving only theiterates xi , their gradients gi and function values fi , along with an optimal
point x∗ and optimal value f∗ . They treat several standard first-order algorithms, namely, the standard fixed-step gradient algorithm, the heavy-ballmethod [Pol64] and the accelerated gradient method [Nes83]. In their approach,(PEP) is expressed as a non-convex quadratic matrix program [Bec07], whichis then relaxed and dualized. The resulting convex problem is then used toprovide bounds on the worst-case performance (and, in some cases, is solved
analytically). As will be shown later in this work, because of the use of a relaxation and the dualization of a non-convex problem, these bounds are in general
not tight, although they turned out to be tight in surprisingly many situations(see Section 4.3). The approach of Drori and Teboulle [DT14] was originally
tailored for first-order algorithms minimizing a single smooth convex function
over Rd , but an extension to provide upper bounds for the fixed-step projectedgradient method is also provided in Drori’s thesis [Dro14].
Another computational approach for the analysis and design of first-order algorithms is proposed in [LRP16], in which optimization procedures are regarded
as dynamical systems. Integral quadratic constraints (IQC), which are usually used to obtain stability guarantees on complicated dynamical systems, areadapted in order to obtain sufficient conditions for the convergence of optimization algorithms. In a few words, the core idea is to formulate (fixed-step,time-invariant) algorithms as (linear) dynamical systems of the form
ξk+1 = Aξk + Buk ,yk = Cξk + Duk ,uk = ∇f (xk ),where ξk are internal states of the dynamical system at time k, yk are its
outputs, and uk are its inputs. From this reformulation, the idea is to replace
the non-linearity coming from the input (∇f (xk )) by (necessary) quadratic
constraints (different families of such constraints are developed, see [LRP16,
Definition 3]). Then, one can formulate sufficient conditions for the first-ordermethod to converge with rate ρ (in terms of kxk − x∗ k) as LMI feasibility
problems (see [LRP16, Theorem 4]), which can be solved with appropriate
solvers and bisection schemes.
This methodology is capable of establishing iteration-independent linear rates
of convergence by solving series of small semidefinite programs. However thosebounds, valid for any number of iterations, are in general not tight, i.e., moreconservative than ours and those of [DT14] when used to estimate worst-caseperformance after a given finite number of iterations (see Subsection 4.3.1 foran example). In addition, while this methodology is well-suited for studying thelinear convergence rates of algorithms for smooth strongly convex optimization,
it fails to recover the exact sublinear rates in the non-strongly convex case.

CHAPTER 1. INTRODUCTION8

Our contributions (1) — Chapter 4. We (T., Hendrickx, Glineur) formalize the concept of performance estimation problems for smooth unconstrained
minimization in [THG16a]. In this work, we use the concept of convex interpolation and smooth (strongly) convex interpolation in order to transform (PEP)
into a semidefinite program without performing any relaxation. That is, any
solution to those new problems can be converted to solutions to (PEP). Moreover, the concept of convex interpolation allows us to guarantee that if there
exists a convergence proof for a given fixed-step algorithm, then the proof canbe obtained using the set of interpolation conditions only.
Our contributions (2) — Chapter 5. Performance estimation problems
with tightness guarantees were later extended to larger classes of algorithms,
functions and convergence measures in our work [THG16b] (T., Hendrickx,
Glineur). In particular, tight guarantees can be obtained for fixed-step algorithm involving projected, proximal and conditional (sub)gradient steps.

1.3.3

Survey on optimized methods

A section of Drori and Teboulle’s work [DT14] is also devoted to the optimization of the coefficients of a fixed-step first-order black-box method for smooth
unconstrained convex minimization. More precisely, a numerical solver is used
to identify a method performing best according to their relaxation of the performance estimation problem, for a known given number of iterations. Thisapproach is taken further in [KF16d], which provides an analytical descriptionof this optimized method. Again we stress that, due to the non-tightness of therelaxation in general, these optimized methods were not guaranteed to havethe best possible performances.
Although it is quite remarkable that the optimized gradient method (OGM) has
a compact representation similar to the standard fast gradient method (FGM),
OGM suffers from a dependence on the number of iterations (more precisely,
only the last iterate depends on it). Kim and Fessler [KF16c] therefore studied
the convergence of the variants of OGM, which do not depend on the numberof iterations, and whose worst-case performances are not significantly worse
than that of the original OGM.
Very recently, Drori [Dro16] provided new results related to the optimized
gradient method (OGM). In this work, he managed to compute exactly theminimax risk (i.e., the best achievable absolute inaccuracy as a function of
the number of iterations) of smooth unconstrained convex minimization. This
lower bound appears to match the upper complexity bound obtained by Kim
and Fessler [KF16d] for OGM, which has therefore the optimal worst-case (forlarge-scale problems).
After that, Kim and Fessler further studied the worst-case guarantees of optimized gradient-type methods (OGM) and fast gradient methods (FGM) in91.3. WORST-CASE ANALYSES OF NUMERICAL SCHEMES

terms of residual gradient norm [KF16b] and in their proximal variants [KF16a].
Among others, they show that FGM achieves a O(N −3/2 ) convergence rate forthe best gradient norm among iterates (this was premised in our work [THG16a]and verified for a less practical variant of FGM in [Nes12b]), and that it was
actually possible to design optimized gradient schemes with better worst-caseguarantees than FGM both in function values and residual gradient norms.
However, those results do not manage to beat the better O(N −2 ) bound obtained with the regularization technique proposed in [Nes12b] for minimizing
the residual gradient norm.
Finally, Drori and Teboulle [DT16] devised an optimal variant of Kelley’s cutting plane method for solving non-smooth unconstrained convex minimization
problem with Lipschitz objectives. At each iteration of this method, it is possible to chose between two kinds of steps. The first possibility is to solve anintermediate bundle-like step, whereas the second one is to perform a simple
subgradient step with pre-determined step size (depending on the number of
iterations and on the previous bundle-like steps).
Remark 1.1. Note that this research trend that looks towards optimized firstorder methods can actually be placed in the broader context of the developmentof accelerated first-order methods [Nes83, Nes04] and their intuitions. This
topic recently attracted much attention. Among others, we refer the reader
to the recent works [DFR16, BLS15, AZO14, SBC14, APR15, AP15] aiming
at obtaining more intuitions on Nesterov’s acceleration both in the smooth
strongly convex case and in the degenerate smooth convex case.

1.3.4

Computer-aided convergence analyses

Finally, the performance estimation methodology was used in different works
for obtaining new improved (analytical) worst-case guarantees for well-known
optimization algorithms; those are summarized in the following paragraphs.
Our contributions (3) — Chapter 6. We (de Klerk, Glineur and T.)
analyze in [dKGT16] the steepest descent method with exact line search forminimizing a smooth strongly convex function. In this work, the natural formulation of (PEP) is nonconvex. Therefore, we perform a convex relaxation inorder to obtain upper bounds on the worst-case behavior. Those upper bounds
turn out to be tight in the case where the line search direction is the gradient and in the case where a relative tolerance on the choice of the direction isallowed.
Our contributions (4) — Chapter 7. We (T., Hendrickx and Glineur)
recently applied the performance estimation framework to obtain tight convergence rates for the proximal gradient method in smooth strongly convex

CHAPTER 1. INTRODUCTION

10composite minimization [THG16c]. In this work, we emphasize the importance and the differences that may arise by considering different kinds of initialknowledge on the optimization problem under consideration.
Very recently, Shi and Liu [SL16] also used the approach for analyzing cyclic
block coordinate descent schemes for smooth unconstrained convex minimization. This work relies on a non-convex formulation and on similar dualization
and relaxations as in the original work of Drori and Teboulle [DT14].

11

1.4. ORGANIZATION OF THE THESIS

1.4

Organization of the thesis

As previously underlined, this thesis is mainly about computing the worst-caseguarantees of numerical optimization schemes. The work is organized in threemain parts. The first part is dedicated to convex analysis.
 In Chapter 2, we review the basic definitions, tools and classes of convex functions we use in the thesis. We focus on providing the different
elements using a unified approach.
 Chapter 3 focuses on the development of convex interpolation and integration theorems, which are necessary tools for the subsequent chapters.
The second part is dedicated to the analysis and development of optimization
algorithms based on the performance estimation idea and on the tools provided
by the convex interpolation results.
 In Chapter 4, we present the performance estimation framework for the(simpler) smooth unconstrained convex minimization case. This work is
published as [THG16a].
 Chapter 5 focuses on forming an unified framework to treat as much firstorder methods as possible in a tight way in the performance estimation
framework. This work is contained in [THG16b].
 In Chapter 6, we provide a tight analysis for the steepest descent algorithm with exact line search (with possibly noise in the search directions)for smooth strongly convex unconstrained minimization. This work is
contained in [dKGT16].
 In Chapter 7, we provide tight convergence analyses of the proximal gradient method for the smooth strongly convex composite minimization.
This work is contained in [THG16c].
Part 3 is dedicated to conclusions and further developments.
 In Chapter 8, we provide new elements and example for further extending the performance estimation framework. The examples include thetreatment of monotone inclusions, decentralization, randomness, noise
and nonconvexities.
 Perspectives and conclusions are then drawn in Chapter 9.

CHAPTER 1. INTRODUCTION

1.5

12

Overview of problem classes and algorithms

The different classes of problems approached throughout the thesis are summarized in Table 1.1. Extensions to decentralized, randomized and second-orderalgorithms, and to the minimization of finite sums and of nonconvex problems
are further discussed in Chapter 8.
Problem type

Algorithm
Gradient method (4.3.1)

Smooth convex unconstrained
minimization

Fast gradient method (4.3.2)
Optimized gradient method (4.3.2)
Gradient method (4.3.1, 7.1)

Smooth strongly convex
unconstrained minimization

Steepest descent (6.1)
Inexact direction steepest descent (6.4)

Smooth convex constrained
minimization

Conditional gradient method (5.3.5)
Proximal point algorithm (5.3.1)
Inexact proximal point algorithm (5.3.1)

Non-smooth convex
minimization

Projected subgradient method (5.3.2)
Proximal gradient method (7.1)
Fast proximal gradient method (5.3.3)
Proximal optimized gradient method (5.3.4)

Convex set intersection

Monotone inclusions

Alternate projection method (5.3.6)
Dykstra alternate projection method (5.3.6)
Forward-backward splitting (8.1)
Douglas-Rachford splitting (8.1)

Table 1.1: Summary of methods and problem classes. Most problems are particular instances of the composite convex minimization class (see Section 5.1),
and most methods are instances of linear fixed step first-order methods (seeSection 5.2.3).

Part I

Discrete Representations ofConvex Functions

Chapter 2

Elements of Convex
Analysis
This chapter introduces the tools necessary for characterizing discrete representations of convex functions. As it mainly contains standard elements fromconvex analysis, the reader familiar with the field may safely skip it1 .
We provide basic definitions for the classes of functions and operators of interest for the sequel with a low level of details — we refer the reader to thestandard references for convex analysis [HUL96, Roc96, RW98], and to theseminal [BV04, Nes04, Rus06, BL10] (more specifically tailored for optimization) for more detailed treatments and further examples. In this chapter, moredetails are provided on Legendre-Fenchel duality results, which are used in
Chapter 3, and on the different points of view on smoothness and convexity,
which are heavily used in the subsequent chapters.
This chapter is organized as follows: Section 2.1 introduces the basic setting we mainly work in for the following chapters: finite dimensional real vector spaces with primal and dual
Euclidean structures.
 In Section 2.2, we provide two standard points of view for approaching
convexity of closed sets. Those are termed as inner and outer views.
 After that, we treat closed convex functions in Section 2.3 using the twodifferent set convexity points of view.
 Section 2.4 provides basic intuitions for understanding Legendre-Fenchel
conjugation, which plays an important role in Chapter 3.
1 Except the promotors.

15

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

16

 Section 2.5 introduces standard classes of functions we work with in thesequel. Those classes are omnipresent in first-order optimization theory.
 Finally, our main characteristics of interest — smoothness and strong
convexity — are strong requirements which are only locally satisfied bymost functions. In Section 2.6, we show that it is always possible toextend locally smooth functions by globally smooth ones. With this reasoning, we can use the results related to smooth function with functions
that are only locally smooth (which largely widens the class of problems
concerned with subsequent analyses).
The main contributions of this chapter are on the one side the presentationof standard results from convex analysis in a unified manner (inner and outer
points of view), and the possibility of extending the local strong convexity andsmoothness properties of a function to global ones.

2.1

Spaces, norms and scalar products

In the sequel, we work in a finite dimensional real vector space E and thecorresponding dual space E∗ formed by all linear functions on E. The dual
pairing is denoted h., .i : E∗ × E → R (so that for any s ∈ E∗ , the corresponding
linear function is denoted hs, .i : E → R) and satisfies∀x ∈ E\ {0} , ∃s ∈ E∗ such that hs, xi =6 0,∀s ∈ E∗ \ {0} , ∃x ∈ E such that hs, xi =6 0.
We also consider a self-adjoint positive definite linear operator B : E → E∗ forh., .i, that is, an operator satisfying
hBx, yi = hBy, xi ∀x, y ∈ E (self-adjoint),
hBx, xi > 0 ∀x ∈ E\ {0} (positive definite).
This allows us to define the following primal and dual Euclidean norms:
2

kxkE = hBx, xi, ∀x ∈ E,

2

kskE∗ = s, B −1 s , ∀s ∈ E∗ ;those norms result from the primal and dual scalar products hx, yiE = hBx, yi
for x, y ∈ E and hx, yiE∗ = x, B −1 y for x, y ∈ E∗ .
Example 2.1. Let E = E∗ = Rd and hx, yi the standard inner producthx, yi = x>y ∀x, y ∈ Rd . Any positive definite matrix B ∈ Sd++ induces a pair22of primal and dual norms: kxkE = x>Bx and kxkE∗ = x>B −1 x. The most standard case is to chose B as the identity operator; however, choosing a different
scaling matrix B can for example be useful for studying second-order methods

17

2.2. CONVEX SETS

(where the norm is typically defined using the Hessian, see e.g., [Ren01]), or forconditioning purposes (see e.g., [Nes12a] for an application to block coordinate
descent-type methods).
Remark 2.2. Note that E endowed with the inner product h., .iE (and thecorresponding induced norm) is a Hilbert space, and so is E∗ with h., .iE∗ . An
alternative but equivalent approach to introduce the different inner products
is to start with the space E, its reference inner product h., .iE and the linear
operator B, and then define h., .i and h., .iE∗ as by-products.

2.2

Convex sets

The notion of set convexity plays a huge role in optimization theory. The
standard intuitive view on convexity of a set is that the set should entirelycontain the segment joining any two points of that set. This point of view isreferred to as the inner characterization of convexity in the sequel.
Definition 2.3. A set Q ⊆ E is convex if and only if for any x, y ∈ Q and forany λ ∈ [0, 1]:
λx + (1 − λ)y ∈ Q.

In what follows, we mostly consider non-empty closed convex sets. Those sets
have numerous nice properties which renders them very convenient to workwith. As an example, the projection operation of any x ∈ E onto a non-empty
closed convex set Q
ΠQ (x) = argmin ky − xkE ,y∈Q

is always well defined and unique. On the other hand, non-closed sets areoften very impractical to work with. Among many possible reasons for that,
non-closed sets may turn simple problems into ill-defined ones (i.e., having nosolution).
For closed sets, it is possible to use another standard definition of convexity.
This alternative to Definition 2.3 uses outer point of view, which relies on
supporting closed half-spaces2 .
Theorem 2.4 (Supporting hyperplanes). Let Q ⊆ E be a closed set with
non-empty interior. Then Q is convex if and only if for every point x0 of its
boundary, there exists an hyperplane {x ∈ E | ha, xi = b} (with some b ∈ R,a ∈ E∗ , a 6= 0) such that ha, xi ≤ b ∀x ∈ Q and ha, x0 i = b.
2 Closed half-spaces are defined as sets of the form {x ∈ E | ha, xi ≤ b}, for some a ∈ E∗
(a 6= 0) and b ∈ R.

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

18

This result is generally referred to as the supporting hyperplane theorem3 , werefer to [BV04, Section 2.5.2] and to [Roc96, Corollary 11.6.1 and Theorem
18.8] for further details.

Example 2.5. Consider the set Q = (x, y) ∈ R2 | x2 + 2y 2 ≤ 1 . This set isclearly closed and convex, and therefore has a supporting hyperplane on anyof its boundary point, as illustrated in Figure 2.1.

•

•
•

•

•

•

•

•
(a) Inner point of view on convexity: every segment joining pointsof the convex set is in the set.

(b) Outer point of view on convexity: every point on the boundary of
the closed convex set has a supporting hyperplane.

Figure 2.1: Illustration of inner and outer points of view on convexity (seeExample 2.5).

2.3

Convex functions

As for convex sets, convex functions play a major role in optimization theory.
This is essentially because local properties of convex functions (e.g., derivatives)
provide global information on the function itself. In general, such global information significantly improves the possibilities for solving the corresponding
optimization problems. In this section, we provide two alternative ways (moreprecisely conjugate, or dual ways) for approaching the convexity of closed functions. Those alternatives are essentially the same as for convex sets and rely on
the standard definition of convexity (inner view) on the one hand (see Definition 2.3), and on supporting hyperplanes (outer view) on the other hand (seeTheorem 2.4).
Let us start with the basic ingredients for defining convex functions; for any
function f : E → R ∪ {∞}, we associate two sets to it: its domain and its
epigraph.
Definition 2.6. Let f : E → R ∪ {∞} be a function. The (effective) domainof f , denoted by dom f is defined as the set:dom f = {x ∈ E : f (x) < ∞} .
3 More precisely, this is a converse supporting hyperplane theorem — the set Q is not

required to be non-empty for guaranteeing the existence of supporting hyperplanes.

19

2.3. CONVEX FUNCTIONSDefinition 2.7. Let f : E → R ∪ {∞} be a function. The epigraph of f ,denoted by epi f is defined as the set:
epi f = {(x, t) ∈ E × R : t ≥ f (x)} .
In order to define convex functions, one first possibility is to rely on the standard
idea of using the definition of convexity of a set (see Definition 2.3), which weapply on the epigraph of f .
Definition 2.8. A function f : E → R ∪ {∞} is convex if its epigraph is aconvex set.
As for convex sets, we mostly restrict ourselves to closed proper functions in
the following. Those properties can be defined from the epigraph.
Definition 2.9. A function f : E → R ∪ {∞} is proper if its epigraph (orequivalently its domain) is a non-empty set.
Definition 2.10. A function f : E → R ∪ {∞} is closed if its epigraph is a
closed set.
Example 2.11. Essentially, non-closed convex functions can only have nonclosedness caused by some points on the boundary of their domain (otherwise,
the function cannot be convex). Consider for example the following convex
function f : R → R (see Figure 2.2)
 2
 x if |x| < 1,
2
if |x| = 1,
f (x) =
∞ otherwise.
Clearly, f is a convex function whose epigraph is not a closed set. The functionf˜ whose epigraph is the closure of epi f is given by 2
x if |x| ≤ 1,˜
f (x) =∞ otherwise.

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

f

•

•

◦

◦

20

f

•

•

x

x

(a) Non-closed convex function f .

(b) Convex function f˜, closure of f .

Figure 2.2: Non-closed convex function f and its closure f˜ (see Example 2.11).

Note that for convex functions, the closedness property is equivalent to the socalled lower semi-continuity (e.g., see [Roc96, Theorem 7.1] or [Rus06, Lemma
2.62]).
As for closed convex sets, it is also possible to use an outer approach to define
convex functions. This relies on the use of supporting hyperplanes applied tothe epigraph (see Theorem 2.4). For convex functions, we usually use nonvertical hyperplanes only, which are in one-to-one correspondence with theso-called subgradients (i.e., a non-vertical supporting hyperplane of epi f ischaracterized by its normal (a, af ) ∈ E∗ × R with af 6= 0 and a constantb ∈ R, whereas the corresponding subgradient of f is a ∈ E∗ ; that is, we
discard the dimension corresponding to the values of f in the epigraph). Those
subgradients are often used in order to characterize convex functions in the(relative) interior of their domain.
Definition 2.12. An element g ∈ E∗ is a subgradient of f : E ∪ {∞} → R at
x if it satisfies ∀y ∈ E
f (y) ≥ f (x) + hg, y − xi,(2.1)

and we denote by ∂f (x) the subdifferential of f at x — i.e., the set of all
subgradients of f at x.
Intuitively, subgradients correspond to global linear under-estimators of thefunction, or more geometrically to non-vertical supporting hyperplanes of theepigraph of f at (x, f (x)) — note that f may only have a non-empty subdifferential for points of its domain. Conversely, for any convex function f , thesubdifferential ∂f (x) is a non-empty set for any x ∈ relint (dom f ) (but may
be empty on the boundary, as it may be that the epigraph only has a verticalsupporting hyperplane there).

21

2.3. CONVEX FUNCTIONS

Example 2.13. Let us consider the closed convex function f : R → R √− −x if x ≤ 0,
f (x) =∞
else.
This function has no subgradient at x = 0 (only a vertical supporting hyperplane of epi f is available here). This can be seen on Figure 2.3 and via its
df= 2√1−x .
derivative dxx0√f = − −x
Figure 2.3: Convex function with empty subdifferential at the origin (see Example 2.13).

Note that Definition 2.8 provides a description of a convex function using upperbounds on its values based on neighboring values, whereas it is also possible
to define convex functions using lower bounds. This fact is emphasized by
the following theorem; as the necessity part of this formulation is not entirely
standard, we provide it with a simple proof.
Theorem 2.14. A function f : E → R ∪ {∞} is convex if and only if ∀x ∈relint (dom f ), the set ∂f (x) is non-empty.
Proof. We only prove the necessity part, as it is well-known that if f is convex
then ∀x ∈ relint (dom f ), the set ∂f (x) is non-empty4 — this is essentiallythe supporting hyperplane theorem (see Theorem 2.4), which imposes that∀x ∈ relint (dom f ) there exists a non-vertical supporting hyperplane, combinedwith the fact f is closed on any closed subset of relint (dom f ).
(Necessity) Let us prove that ∂f (x) 6= ∅ ∀x ∈ relint (dom f ) implies that f isconvex. For any x1 , x2 ∈ dom f and λ ∈ (0, 1) we have y = λx1 + (1 − λ)x2 ∈relint (dom f ). Therefore, ∃g ∈ ∂f (y) andf (x1 ) ≥ f (y) + hg, x1 − yi,f (x2 ) ≥ f (y) + hg, x2 − yi.
By combining those inequalities respectively with the coefficients λ and 1 − λ
4 Among others, one can refer to [Roc96, Theorem 23.4] or [Rus06, Theorem 2.74].

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

22

we obtainf (y) ≤ λf (x1 ) + (1 − λ)f (x2 ),which proves the statement for λ ∈ (0, 1). In order to conclude, it is sufficient to
notice that the previous inequality trivially holds true for λ = 0 and λ = 1.
The following theorem is central in convex optimization theory; for convex
functions, any local optimum is a global optimum.
Theorem 2.15 (Optimality conditions). Let f : E → R ∪ {∞} be proper andconvex, then 0 ∈ ∂f (x∗ ) if and only if x∗ = argmin f (x).
x∈E

Proof. Requiring x∗ to be optimal is equivalent to requiref (x) ≥ f (x∗ ) + h0, x − x∗ i ∀x ∈ E.

2.4

Legendre-Fenchel conjugation

In this section, we introduce the Legendre-Fenchel conjugation, which is essential for the remaining of this work. The common idea behind standard transformations As for common transforms, the use of conjugation is motivated by thefact it renders some operations easier. As examples of such simplifications in
the context of other transforms, note that logarithms transform multiplications
into additions and Fourier and Laplace transform convolutions into products5 .
The Fenchel-Legendre conjugation on the other hand allows an easier treatment of properties related to differentiability when working in the conjugate
space. This transformation dates back to the fifties with the work of Werner
Fenchel [Fen49, FB53] generalizing the Legendre transform6 .
In order to introduce the concept, we start with a simple example emphasizing
the interpretation of the Legendre transform of a function f as the corresponding function f ∗ whose gradient ∇f ∗ is roughly the inverse application of ∇f :y = ∇f (x) ⇔ x = ∇f ∗ (y).
In other words, f and f ∗ are such that the roles of coordinates and gradientsare switched (coordinates and gradients of f respectively becomes gradients
5 The corresponding simplified operations for the Fenchel-Legendre conjugation are theinfimal convolutions, which are transformed into sums.
6 More precisely, the Legendre-Fenchel transform is applicable for both convex and nonconvex functions, as well as for differentiable and non-differentiable ones. It actually reduces
to the Legendre transform in the case of differentiable convex functions (see e.g., [ZRM09]
for a recent pedagogical overview with applications in physics).

23

2.4. LEGENDRE-FENCHEL CONJUGATION

and coordinates of f ∗ ). We make this statement more precise and introduce
the more general Legendre-Fenchel conjugation after the following example.
(def.)

df
Example 2.16. Consider the function f (x) = a2 x2 , we note s(x) = dx
(x) =
1 2
s
ax. By inverting s(x), we obtain x(s) = a , and f (x(s)) = 2a s , which correspond to the value of f given its derivatives.
(def.)The Legendre transform of f is a function of s: f ∗ (s) = sx(s) − f (x(s)) =1 2
2a s . This particular definition guarantees that
dx
df
dx
df ∗
(s) = x(s) + s (s) −(x(s)) (s).
ds
ds
dx
ds
∗

For the case f (x) = a2 x2 , this corresponds to x(s) = dfds (s) = as whendf
s = dx(x(s)) = ax(s). In this case, we indeed conclude that the announced interpretation of the Legendre transform as the transformation linking functions
whose derivative are inverse to each others is valid:
−1

df ∗
df
(x(s))=(s).
dx
dsMore precise statements follows from the formal definition of the LegendreFenchel conjugation.
Definition 2.17. Let f : E → R ∪ {∞} be a function, its Legendre-Fenchel
conjugate f ∗ : E∗ → R ∪ {∞} is defined asf ∗ (s) = sup hs, xi − f (x).

(2.2)

x∈E

Also, we denote by f ∗∗ : E → R ∪ {∞} the biconjugate functionf ∗∗ (x) = sup hs, xi − f ∗ (s).
s∈E∗

Note that conjugate functions are always closed and convex, as they correspondto the maximum of linear functions of s indexed by x. Indeed, the epigraphof f ∗ is the intersection of the epigraphs of all linear functions of s (indexedby x) hx, si − f (x) — that is, the epigraph of f ∗ is an intersection of (possibly
infinitely many) half-spaces. As the intersection of (either finitely or infinitely
many) closed sets is closed, and as the same holds true for the intersection of
convex sets, we have that epi f ∗ is closed and convex.
Theorem 2.18. Let f : E → R ∪ {∞} be a function, its Legendre-Fenchel
conjugate f ∗ : E∗ → R ∪ {∞} is closed and convex.
Notations 2.19. We denote the set of closed proper convex functions on E
by F0,∞ (E). The reason for the 0 and ∞ will become clear later, when dealing
with smoothness and strong convexity.

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

24

Conjugation is often interpreted in terms of affine minorizations. Let f ∈F0,∞ (E) andf ∗ (z ∗ ) = sup hz ∗ , zi − f (z).
z∈E
∗

∗

For any z ∈ E such that the supremum is finite, we havef (x) ≥ hz ∗ , xi − f ∗ (z ∗ ) ∀x ∈ E,
which provides us with a nice interpretation of −f ∗ (z ∗ ) as the largest value
such that hz ∗ , .i − f ∗ (z ∗ ) is a global affine minorant of f . In particular,
f (x) ≥ hz ∗ , xi − f ∗ (z ∗ ) = f (z) + hz ∗ , x − zi,
when the supremum is attained at z ∈ argmax hz ∗ , zi − f (z) (i.e., z ∗ ∈ ∂f (z)).
z∈E

Note that by definition of f ∗ , we also havef (x) ≥ sup hz ∗ , xi − f ∗ (z ∗ ),z ∗ ∈E∗or equivalently f ≥ f ∗∗ , with f ∗∗ having the same affine minorants as f . Therefore, in the case where f is equal to the supremum of all its affine minorants,
we have f = f ∗∗ , which is the case for any closed convex function (using the
supporting hyperplane theorem, see Theorem 2.4).
In other words, conjugation forms a one-to-one correspondence (an involution)
for the class F0,∞ . This is formalized by the following theorem, which is often
referred to as the Fenchel-Moreau or the Fenchel biconjugation theorem (seee.g., [Roc96, Theorem 12.2], [Rus06, Theorem 2.95] or [BL10, Theorem 4.2.1]).
Theorem 2.20. Let f ∈ F0,∞ (E). Then, f ∗ ∈ F0,∞ (E∗ ) and f ∗∗ = f .
As for the Legendre transform (see Example 2.16), a very standard interpretation of the Legendre-Fenchel conjugation is as an operation reversing the rolesof the coordinates and the subgradients: any subgradient (resp. coordinate) of
the original function becomes a coordinate (resp. subgradient) of its conjugate.
This interpretation directly results from the application of first-order optimality
conditions (Theorem 2.15) to the definition of the conjugation (2.2).
Theorem 2.21. Let f ∈ F0,∞ (E) and f ∗ ∈ F0,∞ (E∗ ), the following propositions are equivalent conditions on x ∈ E and x∗ ∈ E∗ :(a) f (x) + f ∗ (x∗ ) = hx∗ , xi,(b) x∗ ∈ ∂f (x),(c) x ∈ ∂f ∗ (x∗ ).
Proof. Let x ∈ argmax hx∗ , zi−f (z). By definition of conjugate function (2.2),
z∈E

25

2.4. LEGENDRE-FENCHEL CONJUGATION

we have the following equivalence:x ∈ argmax hx∗ , zi − f (z) ⇔ f (x) + f ∗ (x∗ ) = hx∗ , xi.
z∈EIn order to prove (a) ⇔ (b), we can just invoke the necessary and sufficient
first-order optimality condition for the optimality of x:x ∈ argmax hx∗ , zi − f (z) ⇔ 0 ∈ x∗ − ∂f (x) ⇔ x∗ ∈ ∂f (x).
z∈E

In order to prove the equivalence (a) ⇔ (c), we use again the definition of
conjugate function, which implies
f (x) ≥ sup hs, xi − f ∗ (s).
s∈E∗

Therefore, Condition (a) is equivalent to x∗ ∈ argmax hs, xi − f ∗ (s), which ims∈E∗

plies (a) ⇔ (c) by invoking the necessity and sufficiency of first-order optimality
conditions again x∗ ∈ argmax hs, xi − f ∗ (s) ⇔ x ∈ ∂f ∗ (x∗ ).
s∈E∗

Before going on into the next theoretical facts about conjugation, let us provide
several examples.
Example 2.22. (a) Let a ∈ Rd , the affine function f (x) = ha, xi+b is closed
and convex, and its conjugate is simply given by−b
if s = a,f ∗ (s) =+∞ else.
(b) Let f : Rd :→ R be the quadratic function f (x) = 12 hAx, xi with A ∈ Sd++
some positive definite matrix. The function f is closed and convex, andits conjugate is given by f ∗ (s) = 21 s, A−1 s .
(c) Let E = E∗ = R, with the standard inner product hx, yi = xy for x, y ∈ R.
Also consider the `1 -norm f (x) = |x|. Its conjugate is0
if |s| ≤ 1,f ∗ (s) =∞ else.
Example 2.23. Let us consider the pair of norms k.kE andkskE∗ = sup {hs, xi : kxkE ≤ 1} ,x∈Ethe corresponding conjugate norm and some constant c > 0. As we will need it
2
1
in the sequel, let us prove that 2ck.kE∗ can equivalently be seen as the conjugate

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

26

2

of 2c k.kE . Indeed, we have the following:c
α2 c
2sup hs, xi − kxkE =max αhs, ui −,22
α∈R,kukE ≤1x∈Ewith αc = hs, ui by first-order optimality conditions on α. Therefore, we havec
1
2
sup hs, xi − kxkE =max (hs, ui)2 ,2
2c kukE ≤1
x∈E

2
1
max hs, ui ,=2c kukE ≤1
1
2= kskE∗ ,2cwhere we used the fact that there exists a solution u such that hs, ui ≥ 0 (e.g.,
u = 0) and therefore the maximum is also non-negative, along with the factthat f (x) = x2 is monotonically increasing on x ≥ 0.
Before going into the next section, let us remark that conjugation also reverses
inequalities between functions. This property is crucial as it allows converting
lower and upper bounds on closed proper convex functions to respectively upperand lower bounds on their conjugates, and vice versa.
Lemma 2.24. Let f (x) ≥ g(x) ∀x ∈ E, then g ∗ (s) ≥ f ∗ (s) ∀s ∈ E∗ .
Proof.
f ∗ (s) = sup hs, xi − f (x) ≤ sup hs, xi − g(x) = g ∗ (s).
x∈E

x∈E

Using this property, one can note that the definition of convex functions involving upper bounds (Definition 2.8) corresponds to the definition of convex
functions using lower bounds (Theorem 2.14) on its conjugate, and reciprocally(more details in the sequel, see Remark 3.5).

2.5

Functional classes

In this section, we introduce the main classes of functions we use in the following chapters; their characteristics were chosen because they are very present
nowadays in first-order optimization theory (see e.g., [BTN01, Ber99, Ber09,
Ber15, Nes04, Rus06]). The main characteristics of interest for us concernsmoothness, strong convexity and gradient and domain boundedness.

27

2.5. FUNCTIONAL CLASSES

2.5.1

Smoothness and strong convexity

The main focus of this section concerns proper closed convex functions satisfying both a smoothness condition and a strong convexity condition. Given two
parameters µ and L satisfying 0 ≤ µ ≤ L ≤ ∞, we will denote by L the constant characterizing the smoothness (i.e., Lipschitz constant on the gradient),
and by µ the strong convexity constant. We will also explicitly allow the case
L = ∞ to include nonsmooth functions as well, while µ on the other hand isalways assumed to be finite. We use the conventions 1/∞ = 0 and ∞ − µ = ∞to deal with the case L = ∞. The main focus of this section (presenting innerand outer points of view for smoothness and strong convexity) is summarizedin Table 2.1.
Class

Definition

Inner view

Outer view

Convex set

Definition 2.3

Definition 2.3

Theorem 2.4

Smooth convex

Definition 2.26

Theorem 2.29

Theorem 2.27

Strongly convex

Definition 2.30

Definition 2.30

Theorem 2.32

Table 2.1: Summary of inner and outer points of view on smoothness andstrong convexity (for proper, closed and convex functions).
Notations 2.25. Let L ∈ R+ ∪ {∞} and µ ∈ R+ be two constants satisfyingµ ≤ L. We denote by Fµ,L (E) the set of L-smooth µ-strongly convex closed
proper functions f : E → R ∪ {∞}. As already introduced, we recall the reader
that the class of proper closed convex functions is denoted by F0,∞ (E).
Definition 2.26. Let f ∈ F0,∞ (E), and a constant L ∈ R+ ∪ {∞}. We saythat f is L-smooth (notation f ∈ F0,L (E)) if it satisfies
1
kg1 − g2 kE∗ ≤ kx1 − x2 kE
L

(2.3)

for all pairs x1 , x2 ∈ E and corresponding subgradients g1 , g2 ∈ E∗ (i.e., suchthat g1 ∈ ∂f (x1 ) and g2 ∈ ∂f (x2 )).
This definition is not entirely standard, as it involves subgradients (even when
the function is differentiable) and allows the constant L to be equal to ∞. In
the case of a finite L, Condition (2.3) immediately implies uniqueness of thesubgradient at each point, hence differentiability of the function, and we recover
the well-known Lipschitz condition on the gradient of a smooth function
1k∇f (x1 ) − ∇f (x2 )kE∗ ≤ kx1 − x2 kE .
L
On the other hand, when L = ∞, the condition becomes vacuous, and theCHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

28

function can be non-differentiable. The reason for this slightly non-standard
definition allowing to choose L = ∞ will become clear when dealing with
Legendre-Fenchel conjugation of smooth convex functions. An alternative andequivalent way of defining smoothness is to require the function to be upperbounded by its first-order development plus a quadratic term7 .
Theorem 2.27. Let f ∈ F0,∞ (E), we have f ∈ F0,L (E) if and only if ∀g ∈ E∗ ,∀x ∈ E such that g ∈ ∂f (x), we have ∀y ∈ E:
f (y) ≤ f (x) + hg, y − xi +L2kx − ykE .
2

(2.4)

Note that when a smooth convex function is proper, it has a non-empty relativeinterior, and hence at least one point satisfying inequality (2.4). Therefore, thisfunction is also defined everywhere (the upper bound (2.4) is finite ∀x ∈ E), so
dom f = E and hence ∂f (x) 6= ∅ ∀x ∈ E. In addition to that, combining (2.4)
with the subgradient inequality (2.1) directly implies uniqueness of g and hence
differentiability g = ∇f (x) when L < ∞. This shows that when L < ∞,we have (2.4) being strictly equivalent to the well-known inequality usuallycharacterizing smoothness
f (y) ≤ f (x) + h∇f (x), y − xi +L2kx − ykE .
2
2

In addition to that, when L = ∞, the condition (2.4) becomes 0 ≤ kx − ykE
and hence void (i.e., always satisfied).
As for general convex functions, there are inner and outer points of view onsmoothness (regarding the epigraph, so they respectively correspond to upperand lower bounds on f ) — Theorem 2.27 provides an inner point of view. We
start with a simple condition allowing to easily obtain the desired equivalentouter result.
2

Theorem 2.28. Let f ∈ F0,∞ (E), we have f ∈ F0,L (E) if and only if L2 kxkE −f (x) ∈ F0,∞ (E).
2

Proof. Let h(x) = L2 kxkE − f (x), gf ∈ ∂f (x) and gh = LBx − gf . First notethat f is closed and proper if and only if h is closed and proper. Then, theresult follows from the verification of the equivalencef (y) ≤ f (x) + hgf , y − xi +L2kx − ykE ⇔ h(y) ≥ h(x) + hgh , y − xi
2∀x, y ∈ E,

7 We do not provide it with a proof, as it follows from the same reasoning as in standard
references — that is, re-write the function with its zeroth-order development plus an integral involving the first order term, and then bound the first-order term using (2.26) — seee.g., [Nes04, Lemma 1.2.3, Theorem 2.1.5].

29

2.5. FUNCTIONAL CLASSES

with gh ∈ ∂h(x) and ∂h(x) 6= ∅ ∀x ∈ E(= dom h) if and only if h ∈ F0,∞ (seeTheorem 2.14).
The following theorem provides an outer description of smooth functions; it isillustrated in Figure 2.4(a),Theorem 2.29. Let f ∈ F0,∞ (E). We have that f ∈ F0,L (E) if and only if∀x, y ∈ E:
L2
f (λx + (1 − λ)y) ≥ λf (x) + (1 − λ)f (y) − λ(1 − λ) kx − ykE .
2

(2.5)

2

Proof. Let f ∈ F0,∞ (E) and denote h(x) = L2 kxkE − f (x) (with h ∈ F0,∞ if
and only if f ∈ F0,L by Theorem 2.28). The results follows from the following
equivalence:h(λx+(1 − λ)y) ≤ λh(x) + (1 − λ)h(y)L2⇔f (λx + (1 − λ)y) ≥ λf (x) + (1 − λ)f (y) − λ(1 − λ) kx − ykE ,2∀x, y ∈ E and ∀λ ∈ [0, 1].
Strong convexity, on the other hand, is a strengthening of the convexity condition, that can also be seen both in terms of outer and inner bounds defining
convex functions. Let us start with the inner bound.
Definition 2.30. Let f ∈ F0,∞ (E) and a constant µ ∈ R+ . We say that f isµ-strongly convex (notation f ∈ Fµ,∞ (E)) if it satisfies for any x, y ∈ E andfor any λ ∈ [0, 1]:µ2
f (λx + (1 − λ)y) ≤ λf (x) + (1 − λ)f (y) − λ(1 − λ) kx − ykE .
2

(2.6)

The following theorem proposes an alternative characterization for strongly
convex functions, and can be deduced from Definition 2.6 along with the Euclidean structure of the primal norm k.kE .
2

Theorem 2.31. f ∈ Fµ,∞ (E) if and only if f (x) − µ2 kxkE ∈ F0,∞ (E).
This alternative definition can be used to obtain a third way of seeing strongly
convex functions, using an outer characterization, similarly to what is proposedby Theorem 2.14 for convex functions.
Theorem 2.32. Let f ∈ F0,∞ (E), then f ∈ Fµ,∞ (E) if and only if
f (y) ≥ f (x) + hg, y − xi +µ2kx − ykE ,2∀y ∈ E and ∀x ∈ E, ∀g ∈ E∗ such that g ∈ ∂f (x).

(2.7)

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

30

Using the previous definitions, one can readily extend Theorem 2.31 for handling smoothness, in addition to strong convexity. This can easily be obtained
again by using the Euclidean structure of k.kE , along with the upper bound 2.4
coming from the smoothness assumption.
Theorem 2.33. Consider a function f ∈ F0,∞ (E). We have that f ∈ Fµ,L (E)2
if and only if f (x) − µ2 kxkE ∈ F0,L−µ (E).
f•Theorem 2.27••

Theorem 2.29x(a) Inner and outer point of view on smoothness for closed
convex functions.

f•Definition 2.30•
Theorem 2.32•x(b) Inner and outer point of view on strong convexity forclosed convex functions.

Figure 2.4: Inner and outer views on strong convexity and smoothness.
Finally, smoothness and strong convexity are closely tied by the LegendreFenchel duality — this is a key element for the developments presented in
the following chapters. Although very standard8 , we provide a more compact
proof of this duality result here, which totally fits our original smoothness andstrong convexity definitions — although very similar, the results presented in[KSST09, Theorem 6] do not share exactly the same setting.
Theorem 2.34. Consider a function f ∈ F0,∞ (E). We have f ∈ F0,L (E) if
and only if f ∗ ∈ F1/L,∞ (E∗ ).
Note that in the case L = ∞, this theorem reduces to Theorem 2.20. Thisduality result explains why we need to include the case L = ∞ in our original
8 We refer to [RW98, Proposition 12.60] for the standard setting with self-conjugate norms
k.kE = k.kE∗ and to [KSST09, Theorem 6] for the dual norm setting — using slightly different
definitions.

31

2.5. FUNCTIONAL CLASSES

definition of smoothness: this is so that we can include the conjugates of smooth
but non-strongly convex functions in F0,L (E).

Proof. The idea of the proof is to show the equivalence between having aquadratic upper bound on f and having a quadratic lower bound on f ∗ .
The key ingredient we use in the proof is the result from Lemma 2.24 stating that lower and upper bounds are reversed when using conjugation. Thatis, ∀f, g : E → R ∪ {∞} such that f (x) ≥ g(x) ∀x ∈ E, we have g ∗ (y) ≥ f ∗ (y)∀y ∈ E∗ .
The proof is divided in three steps.
(a) We start by defining the function h(∆x) = f (x + ∆x) − (f (x) + hg, ∆xi),
for any x and any g such that g ∈ ∂f (x). Note that a L-smoothness
2
assumption on f corresponds to h(∆x) ≤ L2 k∆xkE , ∀∆x ∈ E (see Theorem 2.27).
(b) Using together Lemma 2.24 along with norm conjugation (see Example 2.23), one can note that this upper-bound requirement on h is equiv2
1
alent to the lower-bound requirement h∗ (∆g) ≥ 2L
k∆gkE∗ , ∀∆g ∈ E∗ .
21(c) We show that the condition h∗ (∆g) ≥ 2L
k∆gkE∗ is equivalent to the∗
1/L-strong convexity of f .

From the reasoning, it remains to prove (c); we focus on proving the equiv2
1k∆gkE∗ and 1/L-strong convexity of f ∗ using
alence between h∗ (∆g) ≥ 2L
Theorem 2.32. That is, we have to show that9f ∗ (g + ∆g) ≥ f ∗ (g) + h∆g, xi +1
2
k∆gkE∗ ,2L∀∆g ∈ E∗ and ∀g ∈ E∗ , ∀x ∈ E such that x ∈ ∂f ∗ (g) (or equivalently ∀x ∈E,∀g ∈ E∗ such that g ∈ ∂f (x) by Theorem 2.21). To this aim, we only haveto obtain an expression of h∗ in terms of f ∗ . The following lines are valid

9 We only show the strong convexity requirement, as we already know that f ∗ is closed
and convex (Theorem 2.20) and therefore already satisfies the condition ∂f ∗ (g) 6= ∅ ∀g ∈relint (dom f ∗ ) (Theorem 2.14).

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

32∀x ∈ E, ∀g ∈ E∗ such that g ∈ ∂f (x):h∗ (∆g) = sup h∆g, ∆xi − h(∆x),∆x∈E= f (x) + sup hg + ∆g, ∆xi − f (x + ∆x), (definition of h)∆x∈E= f (x) + sup hg + ∆g, x0 − xi − f (x0 ), (new variable x0 = x + ∆x)x0 ∈E= f (x) − hg + ∆g, xi + sup hg + ∆g, x0 i − f (x0 ),x0 ∈E∗= f (x) − hg + ∆g, xi + f (g + ∆g) , (definition of f ∗ )= f ∗ (g + ∆g) − f ∗ (g) − h∆g, xi(using the assumption g ∈ ∂f (x), equivalent to f (x) − hx, gi = −f ∗ (g)by Theorem 2.21).
Therefore, we have the claim
1
2
k∆gkE∗ ≤ f ∗ (g + ∆g) − f ∗ (g) − h∆g, xi,
2L∀∆g ∈ E∗ and ∀g ∈ E∗ ,∀x ∈ E such that x ∈ ∂f ∗ (g).
Corollary 2.35. Consider a function f ∈ F0,∞ (E). We have f ∈ Fµ,L (E) if
and only if f ∗ ∈ F1/L,1/µ (E∗ ).
Example 2.36. Let us consider the case E = E∗ = Rd , with the standard
inner product hx, yi = x>y for x, y ∈ Rd . Also consider the positive definite
matrix A  0, the corresponding quadratic function f (x) = 12 x>Ax and its
conjugate f ∗ (z) = 21 z >A−1 z (see Example 2.22).
By denoting µf = λmin (A) and Lf = λmax (A) respectively the minimum andmaximum eigenvalues of A, one can note that f is µf -strongly convex andLf -smooth.
Similarly, by denoting µf ∗ = λmin (A−1 ) and Lf ∗ = λmax (A−1 ), we have that−1∗f ∗ is µf ∗ -strongly convex and Lf ∗ -smooth, with µf ∗ = L−1
f and Lf = µf .
Example 2.37. Fenchel duality between strong convexity and smoothness isa fundamental principle used for example for smoothing techniques arising in
optimization (i.e., approximating a non-smooth convex objective function bya smooth one); see for example [Nes05, DGN12, BT12]. Essentially, the idea
is to regularize the conjugate f ∗ of the initially non-smooth function f witha strongly convex function and then to approximate the original function by
the conjugate of the regularized f ∗ . For instance, regularizing f ∗ by choosing2f˜∗ (g) = f ∗ (g) + τ2 kgkE∗ and then approximate f by f˜ = f˜∗∗ .
As an example, let E = E∗ = R, with the standard inner product hx, yi = xy
for x, y ∈ R. Also consider the `1 -norm f (x) = |x| as the non-smooth function.

33

2.5. FUNCTIONAL CLASSES

Its conjugate isf ∗ (z) =0
if |z| ≤ 1,
∞ else.

We denote by h∗ (z) = f ∗ (z) + µ2 |z|2 a regularization of f ∗ with µ ≥ 0. The
corresponding conjugate is:( 2x
if |x| < µ,2µh(x) =µ|x| − 2 else,
which is 1/µ-smooth (see Figure 2.37). This kind of functions is usually referred
to as a Huber loss (see e.g. [Hub64]); it is commonly used in regressions forapproximating the `1 -regularization.

fx
Figure 2.5: Absolute value (black) and its smoothing (red).

Finally, the following theorem provide a simple criterion to check whether atwice differentiable function is µ-strongly convex and L-smooth. For that purpose, we respectively consider the lowest and largest eigenvalues of a linear
application A : E → E∗ : respectively λmin (A) = minx∈E hAx,xiand λmax (A) =kxk2
Emaxx∈E hAx,xi. The proof is very standard, and we do not provide it. The
kxk2E
interested reader can refer to e.g., [Nes04, Lemma 1.2.2] for the smooth part,and use the same idea for the strongly convex one.
Theorem 2.38. Let f : E → R being twice continuously differentiable. Thenf ∈ Fµ,L (E) if and only if µ ≤ λmin (∇2 f (x)) and λmax (∇2 f (x)) ≤ L ∀x ∈ E.

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

2.5.2

34

Domain and subgradient boundedness

Definition 2.39. Let f ∈ F0,∞ (E) and a constant M ∈ R+ ∪ {∞}. We saythat f has M -bounded subgradients (resp. subgradient variations) if it satisfies
kgkE∗ ≤ M (resp. kg1 − g2 kE∗ ≤ M ),∀g ∈ ∪x∈E ∂f (x)(= dom f ∗ ) (resp. ∀g1 , g2 ∈ dom f ∗ ).
Closed convex functions with bounded subgradient are often referred to as
Lipschitz functions, as they are often defined using the relation ∀x, y ∈ dom f :|f (x) − f (y)| ≤ M kx − ykE .
(this can simply be obtained from the subgradient inequality and Definition 2.39).
The class of closed and proper convex functions with bounded subgradientvariations naturally appears as the class of function conjugate to the class offunction with bounded domain, whereas having bounded gradients instead corresponds to the class conjugate to functions with bounded domain centered at
the origin (more details in the sequel).
Remark 2.40. Convex functions with bounded subgradient variations arewithin the family of convex functions with bounded subgradients (with different constants), and reciprocally — this can be obtained by an appropriate
use of triangle inequalities along with Definition 2.39).
Definition 2.41. Let f ∈ F0,∞ (E) and a constant M ∈ R+ ∪ {∞}. We saythat f has a radius M (resp. a diameter M ) if it satisfies
kxkE ≤ M (resp. kx1 − x2 kE ≤ M ),for any x ∈ dom f (resp. x1 , x2 ∈ dom f ).
Note that as in the case of smoothness, the boundedness constant M is also
allowed to take the value ∞, in order to embed the unbounded (domain or
gradient) cases.
0
Notations 2.42. We denote by CM,L (E) (resp. CM,L(E)) the class of closed
proper convex L-smooth functions with M -bounded subgradients (resp. sub0
gradient variations) and by SM,µ (E) (resp. SM,µ(E)) the class of closed properµ-strongly convex functions with a radius (resp. diameter) M .

As for smoothness and strong convexity, domain and gradient boundedness areclosely related via Legendre-Fenchel conjugation. This is very natural in viewof the interpretation of conjugation as an operation reversing the roles of coordinates and subgradients (see Theorem 2.21), and is in particular emphasized
by the next theorem.

35

2.5. FUNCTIONAL CLASSES

0Theorem 2.43. Let f ∈ F0,∞ (E). We have f ∈ SM,0 (E) (resp. f ∈ SM,0
(E))
∗
∗
∗
0
∗
if and only if f ∈ CM,V (E ) (resp. f ∈ CM,∞ (E )).

Proof. We focus on the case M < ∞, as the result trivially holds otherwise.
Also, we only detail the case of f ∈ SM,0 (E) and f ∗ ∈ CM,∞ (E∗ ) as the corresponding results with bounded variations and domains follow from a similarreasoning.
Let us start with the case f ∈ SM,0 (E). In that setting we have that ∀x ∈E, ∀g ∈ E∗ such that g ∈ ∂f (x), x is such that kxkE ≤ M (because ∂f (x) 6= ∅only for x ∈ dom f ). The previous statement is equivalent to ∀x ∈ E, ∀g ∈ E∗
such that x ∈ ∂f ∗ (g) we have kxkE ≤ M . Hence, f ∈ SM,0 (E) ⇒ f ∗ ∈CM,∞ (E∗ ).
Second, let f ∗ ∈ CM,∞ (E∗ ). In that setting we have that ∀x ∈ E, ∀g ∈ E∗ suchthat x ∈ ∂f ∗ (g), x is such that kxkE∗ ≤ M . This is equivalent to ∀x ∈ E, ∀g ∈E∗ such that g ∈ ∂f (x), kxkE ≤ M . This proves that ∀x ∈ relint (dom f ) wehave kxkE ≤ M (see Theorem 2.14).
In order to prove that kxkE ≤ M ∀x ∈ dom f , we consider two cases: (a)relint (dom f ) 6= ∅ and (b) relint (dom f ) = ∅.
(a) We proceed by contradiction. In this case, let us consider some x ∈ E:
kxk > M and let some y ∈ relint (dom f ). Then, there exists λ ∈ (0, 1)
such that kzkE > M with z = λx + (1 − λ)y (so z ∈ relint (dom f )).
Therefore, there exists g ∈ ∂f (z) (see Theorem 2.14), and by conjugationz ∈ ∂f ∗ (g) is such that kzk > M which is a contradiction with theassumption f ∗ ∈ CM,∞ (E∗ ).
(b) Let us consider an empty relative interior. That is, dom f is a singleton.
Then, for x ∈ dom f and ∀g ∈ E we have g ∈ ∂f (x), and thereforex ∈ ∂f ∗ (g) and hence kxkE ≤ M by assumption.
Hence when f ∈ F0,∞ (E) we have f ∗ ∈ CM,∞ (E∗ ) ⇒ f ∈ SM,0 (E), whichconcludes the proof.

Corollary 2.44. Consider a function f ∈ F0,∞ (E). We have f ∈ CM,L (E)00(resp. f ∈ CM,L(E)) if and only if f ∗ ∈ SM,1/L (E∗ ) (resp. f ∗ ∈ SM,1/L(E∗ )).
Remark 2.45. Note that not all those properties — that is, smoothness,strong convexity and gradient and domain boundedness — are compatible
with each other. For example, smoothness and bounded gradient imposes thatdom f = E, and can therefore clearly not be combined with domain boundedness (with M < ∞). On the other hand, the strong convexity requirement is
not compatible with bounded subgradients, as for any strongly convex functionf ∈ Fµ,∞ (with µ > 0) and for any vector g ∈ E∗ , there exists x ∈ E such thatg ∈ ∂f (x) — which cannot be combined with the fact that any subgradientshould have a bounded norm. Another way to understand the incompatibility

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

36

between strong convexity and bounded gradients is to remark that its conjugate
would be a smooth convex function with bounded domain, which is impossible.
Those incompatibilities originate from the fact we require the functions to globally satisfy boundedness, smoothness, and strong convexity properties, insteadof only locally (local versions of smoothness and strong convexity are handled
in Section 2.6).
As examples of smooth functions with bounded subgradients and strongly convex functions with bounded domain, one can consider the Huber function h(x)of Example 2.37, and its conjugate h∗ (x).

2.5.3

Indicator and support functions

Constraints are so recurrent in optimization that we dedicate the next lines
specifically to them. For that, we first introduce the class of indicator function, and then the class of support functions, the class of convex conjugates toindicator functions.
Definition 2.46. Let Q ⊆ E be a closed convex set and define iQ : E → {0, ∞}:0
if x ∈ Q,
iQ (x) =∞ otherwise.
We call iQ : E → {0, ∞} the indicator function of Q.
Notations 2.47. We denote by IM (E) the class of closed convex indicator
0(E) forfunctions that are bounded in terms of radius, and alternatively IM
those bounded in terms of diameter.
Definition 2.48. Let Q ⊆ E be a closed convex set and define σQ : E∗ →R ∪ {∞}σQ (s) = sup hs, xi.
x∈Q

We call σQ : E∗ → R ∪ {∞} the support function of Q.
Intuitively, support functions provide correspondences between a supporting
hyperplane s ∈ E∗ and the distance (in the direction of s) to the origin of thepoint on which it supports the convex set Q.
Note that support functions can be seen as naturally defined as convex conjugate to indicator functions, as we can write:
σQ (s) = sup hs, xi = sup hs, xi − iQ (x).
x∈Q

x∈E

As a consequence, M -bounded indicator function are convex conjugate to support functions with M -bounded subgradients (by Theorem 2.43).

37

2.5. FUNCTIONAL CLASSES

Notations 2.49. The set of support functions with bounded subgradients is∗0∗denoted by IM(E∗ ) and IM(E∗ ) for the support functions that have bounded
subgradient variations.
Example 2.50. Every norm k.kE∗ is the support function of the unit ball of its
conjugate norm k.kE by definition of conjugate norm: kzkE∗ = supkxkE ≤1 hz, xi.

2.5.4

Non-convex smooth functions

Let us now leave the convex world for a moment. We introduce a class of differentiable non-convex functions. This class of function has the particularity of
being definable using convex functions, and is also very present in optimization
theory and practice.
Definition 2.51. A differentiable function f : E → R is L-smooth if and only
if it satisfies the following condition ∀x, y ∈ E:
|f (x) + h∇f (x), y − xi − f (y)| ≤L2kx − ykE .
2

(2.8)

Also note that it is alternatively possible to define this class of function using
a standard Lipschitz condition on the gradient ∀x, y ∈ E (when L < ∞):1k∇f (x) − ∇f (y)kE∗ ≤ kx − ykE .
L
Notations 2.52. In order to denote smooth non-linear functions, we overload
the notation used for closed proper smooth strongly convex functions. That is,the class of L-smooth non-linear functions over E is denoted f ∈ F−L,L (E).
The following lemma allows obtaining simple alternative definition for this classof function using the class of smooth convex functions.
2

Lemma 2.53. Let f : E → R ∪ {∞}. We have f ∈ F−L,L (E) ⇔ f + L2 kxkE ∈F0,2L (E).
2

Proof. Let f : E → R ∪ {∞} and define h(x) = f (x) + L2 kxkE . We have that∇h(x) = ∇f (x) + LBx, and ∀x, y ∈ E:L
2
kx − ykE ⇔ h(y) ≥ h(x) + h∇h(x), y − xi,
2L2−f (x) − h∇f (x), y − xi + f (y) ≤ kx − ykE ⇔ h(y) ≤ h(x) + h∇h(x), y − xi22
+ Lkx − ykE ,f (x) + h∇f (x), y − xi − f (y) ≤CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

38

where the equivalences are obtained by expressing f and ∇f in terms of h and∇h (or reciprocally), which proves our statement.
Remark 2.54. Note that smooth convex functions from Definition 2.51 could
alternatively be defined using different constants for characterizing upper andlower quadratic bounds.
The following theorem proves an useful characterization of twice differenciable
L-smooth functions.
Theorem 2.55. Let f : E → R be twice continuously differentiable. Thenf ∈ F−L,L (E) if and only if −LI ≤ λmin (∇2 f (x)) and λmax (∇2 f (x)) ≤ L∀x ∈ E.
The proof of this theorem is very standard and we therefore not provide it here
(as for Theorem 2.38, the reader can refer to e.g., [Nes04, Lemma 1.2.2]).
Example 2.56. Consider the case E = E∗ = Rd with the standard Euclidean
inner product. Let f be a quadratic function f (x) = 21 x>Ax defined from thesymmetric matrix A ∈ Sd with bounded eigenvalues −LId  A  LId . We
have that f ∈ F−L,L (Rd ).

2.6

Local and global smoothness

For practical optimization problems, smoothness and strong convexity can beseen as very strong requirements. However, we can in general be satisfied
even when those requirements are only locally met on a subset of the domain.
One common example of that situation is the case of constrained optimization
problems for which the objective function satisfies some smoothness condition
on its domain (but possibly not outside of it). This situation is very common,as for example any differentiable function defined on a compact set locallysatisfies a smoothness condition.
In order to show that this local property suffices, we provide a construction
for generating a smooth extension to the function. We start by assuming thatthe function f satisfies the following smoothness condition (2.4) on some closed
convex domain Q:
f (y) ≤ f (x) + h∇f (x), y − xi +L2kx − ykE :2∀x, y ∈ Q.

We show that it is also valid for the other definitions of smoothness and discuss similar ideas for obtaining global extensions for strong convexity and nonconvex smooth functions afterwards (see Corollary 2.60 and Corollary 2.61 in
the sequel).

39

2.6. LOCAL AND GLOBAL SMOOTHNESSTheorem 2.57. Let f : E → R ∪ {∞} be closed and proper, and Q ⊆ E bea closed convex set. Also, assume f being finite on Q and such that for allx ∈ Q, there exists gx ∈ E∗ satisfying
f (y) ≥ f (x) + hgx , y − xi: ∀y ∈ Q,

(2.9)

L
2
f (y) ≤ f (x) + hgx , y − xi + kx − ykE2: ∀y ∈ Q.

(2.10)

Then, there exists a function f˜ ∈ F0,L (E) such that f˜(x) = f (x) and ∇f˜(x) =gx for all x ∈ Q.
Proof. First, note that for any x ∈ Q, gx is necessarily unique; so we denote
gx = ∇f (x) in the following.
Let y ∈ Q and x ∈ E, we define the local upper bound on f obtained from y:qy (x) = f (y) + h∇f (y), x − yi +L2kx − ykE .
2

Also, we define the function f˜(x):f˜(x) = convh {qy (x) : y ∈ Q} = inf {t : (x, t) ∈ convh {epi qy (x) : y ∈ Q}} ,t

We prove that f˜ ∈ F0,L (E) with f˜(x) = f (x) ∀x ∈ Q. In order to do so, we
proceed in the following way:(a) we prove that f˜(x) = f (x) ∀x ∈ Q,(b) we show that f˜∗ ∈ F1/L,∞ (E∗ ) and f˜ ∈ F0,∞ (E).
Note that those elements also implies that ∇f˜(x) = ∇f (x) for all x ∈ Q.
We start with (a). As the function is locally smooth on Q, we havef (x) ≤ qy (x), ∀x, y ∈ Q.
Therefore, f (x) ≤ f˜(x) ∀x ∈ Q. In addition, we have qx (x) = f (x) ∀x ∈ Q andhence f˜(x) ≤ f (x) ∀x ∈ Q. Therefore, ∀x ∈ Q we have f˜(x) ≤ f (x) ≤ f˜(x)and hence f (x) = f˜(x).
For proving (b), we use the following identitiesf˜∗ (s) = sup qy∗ (s) , f˜∗∗ (s) = convh qy∗∗ (s) : y ∈ Q .
y∈Q(the proofs of those identities rely on the affine minorization interpretation of
conjugation: the convex hull operation corresponds to intersecting the set ofaffine minorants in the conjugate space; see e.g., [Roc96, Theorem 16.5].)

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

40

Since qy = qy∗∗ (closed convex functions), we have f˜∗∗ = f˜ ∈ F0,∞ (E). In orderto prove that f˜∗ ∈ F1/L,∞ , we compute qy∗ (s):qy∗ (s) = hs, yi − f (y) += hs, yi −12ks − ∇f (y)kE∗
2L

1
1
1
2
2
hs, ∇f (y)iE∗ − f (y) +k∇f (y)kE∗ +kskE∗ .
L
2L
2L

Hence,


1
1
1
2
2
∗
˜
k∇f (y)kE∗ +kskE∗ ,
f (s) = sup hs, yi − hs, ∇f (y)iE∗ − f (y) +
L
2L
2L
y∈Q
which is 1/L-strongly convex by Theorem 2.31. Therefore, the statement isproved as f˜(x) = f (x) ∀x ∈ Q and as f˜ ∈ F0,L (E).

Remark 2.58. The functional extension proposed in Theorem 2.57 satisfies
other definitions of smoothness (see Definition 2.26, Theorem 2.29), as they areequivalent for any smooth convex function.
Note that on the other hand, one can easily prove that a local satisfaction ofDefinition 2.26 or Theorem 2.29 implies a local satisfaction of Theorem 2.27;
therefore, any of those criterion may be checked locally in order to conclude
the existence of a smooth convex extension.
Remark 2.59. Local strong convexity is easily handled, as f (x)+iQ (x) would
be a global strongly convex extension to f . In order to obtain a smooth strongly
convex extension to some f , one can perform the standard transformation:2
h(x) = f (x) − µ2 kxkE should be convex and L − µ-smooth on Q. The extension2h̃(x) can then be regularized h̃(x)+ µ2 kxkE in order to obtain a smooth strongly
convex extension to f .
The same tip can be used for obtaining a smooth extension to the non-convex
locally smooth (on some closed convex set Q) function f : the regularization22h(x) = f (x) + L2 kxkE is locally convex and smooth, and h̃(x) − L2 kxkE is a
globally smooth non-convex extension to f .
Corollary 2.60. Let f : E → R ∪ {∞} be closed and proper, and Q ⊆ E be
closed and convex. In addition, let f be differentiable on an open set containing
Q, and let f satisfy Conditions (2.4) (smoothness) and (2.7) (strong convexity)
on Q. Then, there exists a function f˜ ∈ Fµ,L (E) such that f˜(x) = f (x) and∇f˜(x) = ∇f (x) for all x ∈ Q.

41

2.6. LOCAL AND GLOBAL SMOOTHNESSCorollary 2.61. Let f : E → R ∪ {∞} be closed and proper, and Q ⊆ E be
closed and convex. In addition, let f be differentiable on an open set containing
Q, and let f satisfy Condition (2.8) (smoothness) on Q. Then, there exists
a function f˜ ∈ F−L,L (E) such that f˜(x) = f (x) and ∇f˜(x) = ∇f (x) for
all x ∈ Q.
Example 2.62. Consider the function domain Q = [−1, 1] ⊂ R and the function f : R → R x2
if |x| ≤ 1,
2
f (x) =−|x| else.
We note that f is smooth and convex on Q (illustration on Figure 2.6), and its
2smooth convex extension from Theorem 2.57 is simply x2 .
fxFigure 2.6: Locally smooth convex function f (black), and the smooth convex
extension f˜ (dashed, red).

CHAPTER 2. ELEMENTS OF CONVEX ANALYSIS

42

Chapter 3

Convex Interpolation
Throughout this chapter, we study two types of finite representations of convex
functions. Also, we provide constructions and geometric interpretations forrecovering functions of the desired types from their finite representations.
For a good overview of the chapter, we suggest the reader to thumb throughSection 3.1 and Section 3.2 for understanding the main underlying concepts
and motivations, and Section 3.3.2 and Section 3.3.1 for the application of theconcept to the non-smooth and smooth convex interpolation. The remaining
parts can be seen as a collection of similar results for other classes of functions.
The main contributions of the chapter are the following.
 The concept of convex interpolation, which allows among others to reformulate infinite dimensional optimization problems over those functional
spaces (e.g., performance estimation problems) in finite dimension.
 The derivation of necessary and sufficient conditions for convex interpolation in the cases of different families of convex functions. Amongothers, we present interpolation conditions for classes of functions involving smoothness and strong convexity. We extend the idea to smooth
non-convex functions, to indicator and support functions, and to convex
functions with bounded domain or subdifferentials.
 Extensions of the convex integration problem — and the corresponding cyclic monotonicity conditions — for classes of functions involving
smoothness or strong convexity.
This chapter is organized as follows.
 In Section 3.1, we present the problem and its motivations.
 Section 3.2 illustrates that commonly used sets of inequalities are generally not sufficient for guaranteeing convex interpolability, especially when43

CHAPTER 3. CONVEX INTERPOLATION

44

the functions to be interpolated are required to satisfy some smoothness
requirements.
 In Section 3.3, we present the nonsmooth and smooth convex interpolation conditions. We follow a principled approach in order to require
the interpolated function to possibly satisfy smoothness, strong convexity and domain and gradient boundedness. Also, we specifically treat thecases of support and indicator functions, and provide an adaptation of
the results for handling non-linear smooth functions.
 Section 3.4 provides the results for the related convex integration problems, and provide a link with former results related to cyclic monotonicity.
Also, note that the subsequent text is based on sections of [THG16b, THG16a].

3.1

Problem and motivations

In this chapter, we study convex functions described by only a finite set ofpoints. The main underlying motivation is the ability to formulate performance
estimation problems (which are optimization problems over spaces of functions)in tractable ways (see Chapter 4 and Chapter 5).
Definition 3.1. Let I be a finite index set and F be a class of convex functions
over E, and consider the set of triples S = {(xi , gi , fi )}i∈I where xi ∈ E, gi ∈ E∗
and fi ∈ R for all i ∈ I. The set S is (first-order) F(E)-interpolable if and only
if there exists a function F ∈ F such that both gi ∈ ∂F (xi ) and F (xi ) = fi
hold for all i ∈ I.
The main idea is to develop necessary and sufficient conditions for a set of
triplets {(point, gradient, function value)} to be interpolable by a function F
within some specified class of (convex) functions F. That is, given a set of
triplets {(xi , gi , fi )}i ⊂ E×E∗ ×R we aim at finding F ∈ F such that fi = F (xi )and gi ∈ ∂F (xi ). One of the main underlying challenges is the incorporationof smoothness constraints into the class of functions F, i.e., to require theexistence of a differentiable interpolating function F , i.e., with a Lipschitz
condition on its gradient.
In particular cases, it may seem more convenient not to use function values fi
as variables. The problem of recovering a convex function based only on points
and gradient values at those points is referred to as the convex integration
problem.
Definition 3.2. Let I be a finite index set and F be a class of convex functions,and consider the set of triples S = {(xi , gi )}i∈I where xi ∈ E, gi ∈ E∗ for alli ∈ I. The set S is (first-order) F(E)-integrable if and only if there exists afunction F ∈ F(E) such that gi ∈ ∂F (xi ) holds for all i ∈ I.

45

3.2. MOTIVATING COUNTEREXAMPLESThis problem is related to the so-called cyclic monotonicity conditions. Finally,note that subdifferentials are particular cases of monotone operators, which we
discuss in Section 8.1.
Notations 3.3. We generally refer to F-interpolation or F-integration in thefollowing, without specifying the space it refers to (that is, instead of F(E) or
F(E∗ )), as the corresponding space is generally clear from the context.

3.2

Motivating counterexamples

In this section, we illustrate that the convex interpolation problem is in general
not handled using naive approaches, and in particular for requiring the convex
interpolated functions to satisfy a smoothness requirement.
For the following counterexamples, we restrict ourselves to the standard Euclidean setting E = E∗ = Rd with hx, yi = x>y ∀x, y ∈ Rd and k.kE = k.kE∗ =k.k2 the standard Euclidean 2-norm.
Finding necessary conditions for smooth convex interpolation is relatively easy:
starting from any set of conditions that hold over pairs of points from the wholedomain of any smooth convex function (for example conditions from Definition 2.26 or Theorem 2.27), one can simply restrict this set to the conditions
involving only points xi with i ∈ I (i.e., to discretize it). For example, it is wellknown that the class of L-smooth convex functions F0,L (Rd ) is characterized
by the pair of inequalities
f (y) ≥ f (z) + ∇f (z)>(y − z),∀ y, z ∈ Rd ,||∇f (y) − ∇f (z)||2 ≤ L||y − z||2 ,(C1)d∀ y, z ∈ R .

Therefore, specializing those conditions for y = xi and z = xj with i, j ∈ Ileads to the following set of inequalities, which is necessary for the existence ofan interpolating function in F0,L :fi ≥ fj + gj>(xi − xj ),||gi − gj ||2 ≤ L||xi − xj ||2 ,∀i, j ∈ I,

(C1f)∀i, j ∈ I.

Now, perhaps surprisingly, it turns out that this latter set of conditions is not
sufficient to guarantee F0,L -interpolability, despite the fact that the originating conditions (C1) are sufficient to guarantee that f ∈ F0,L (Rd ). In order to
see that, consider the following example with I = {1, 2} and d = 1:
(x1 , g1 , f1 ) = (−1, −2, 1) and (x2 , g2 , f2 ) = (0, −1, 0).
This set satisfies Conditions (C1f) with L = 1 but cannot be interpolated bya smooth convex function for any finite value of L: the convexity requirement

CHAPTER 3. CONVEX INTERPOLATION

46

forces the interpolating convex function to lie entirely above its linear underapproximations, which lead to an unavoidable non-differentiability at x1 , asillustrated on Figure 3.1. Therefore Conditions (C1f) are not sufficient to guarantee smooth convex interpolation.
f
•

x1
•
x2
x

Figure 3.1: Example (x1 , g1 , f1 ) = (−1, −2, 1) and (x2 , g2 , f2 ) = (0, −1, 0) for
I = {1, 2} and d = 1.
Similarly, we can carry out the same exercise for the following conditions, also
well-known to be equivalent to inclusion on F0,L (Rd ) when imposed on thewhole space:fi ≥ fj + gj>(xi − xj ),∀ i, j ∈ I,Lfi ≤ fj + gj>(xi − xj ) + ||xi − xj ||22 ,2(C2f)∀ i, j ∈ I.

With an appropriate use of an additional dimension (d = 2), one can readily observe that some information may be hidden to this pair of inequalities.
Consider the example
        
0
1
1
1(x1 , g1 , f1 ) =,, 0 and (x2 , g2 , f2 ) =
,
,1 ,
0
0
0
1
from which no smooth convex interpolation can be made (again, unavoidable
non-differentiability at both x1 and x2 ). However, both Conditions C1f and C2f
are satisfied with L = 1.
Those examples illustrate the weakness of a naive approach that consists indiscretizing standard necessary and sufficient conditions defined on the whole
space. If those discretized conditions were to replace a constraint f ∈ F0,L
in an optimization problem, they would implicitly allow functions that do notbelong to the class F0,L to be obtained, which would correspond to the solution
to a relaxation of the original optimization problem. To conclude this section,note that any set of necessary and sufficient conditions for smooth convex
interpolability must be a subset of some necessary and sufficient conditions on
the whole domain (since the interpolation conditions have to be satisfied forany discretization of any function within F0,L ), whereas the previous examples
precisely show that the converse is not true.

47

3.3. CONVEX INTERPOLATIONIn the next subsections, we follow a more principled approach in order to tackle
the Fµ,L -interpolation problem. We start with a special case of convex interpolation, that of proper convex functions with no smoothness or strong convexity
requirement (i.e., the class F0,∞ ).

3.3

Convex interpolation

3.3.1

Non-smooth convex interpolation

We begin by constructing interpolation conditions for the simpler class of nonsmooth convex functions F0,∞ (E). This result is not new but we provide a
simple constructive proof of it because it is one of the main building blocks formost following results of this chapter.
Theorem 3.4. The set {(xi , gi , fi )}i∈I is F0,∞ -interpolable if and only if
fi ≥ fj + hgj , xi − xj i

∀i, j ∈ I.

(3.1)

Proof. (Necessity) Assume there exists a convex function f : E → R suchthat fi = f (xi ) and gi ∈ ∂f (xi ) ∀i ∈ I. The definition of subgradient then
immediately implies
fi ≥ fj + hgj , xi − xj i

∀i, j ∈ I.

(Sufficiency) Define the following piecewise-linear convex function
f (x) = max {fj + hgj , x − xj i} .
j∈ISince f is the pointwise maximum of a finite number of affine functions, its
epigraph is a non-empty polyhedron, and hence f is convex, closed and proper.
In addition, f (xi ) = fi holds by construction. Indeed, we first see thatfi = fi + hgi , xi − xi i, ≤ max {fj + hgj , xi − xj i} = f (xi ).
j∈I

Therefore, we have fi ≤ f (xi ). In addition to this, we havef (xi ) = max {fj + hgj , xi − xj i} ≤ fij∈Iusing Condition (3.1) for each j,which allows to conclude that f (xi ) = fi . The construction also implies that

CHAPTER 3. CONVEX INTERPOLATION

48

gi ∈ ∂f (xi ) because
f (x) = max {fj + hgj , x − xj i}j∈I≥ fi + hgi , x − xi i

∀x ∈ E,∀i ∈ I, x ∈ E,

≥ f (xi ) + hgi , x − xi i∀i ∈ I, x ∈ E.

Remark 3.5. Interpolating functions are typically not unique. Two such interpolating functions are particularly remarkable: one is (pointwise) lower than
all the others, and one is (pointwise) higher than all the others. Those twofunctions naturally arise from the different possibilities for defining a convex
function (see Definition 2.8 for the definition in terms of epigraph and Theorem 2.14 for the alternative in terms of subgradients).
Let {(xi , gi , fi )}i∈I be satisfying the conditions from Theorem 3.4, we respectively have for the lowest and highest interpolating functions (illustration on
Figure 3.2):
fl (x) = max {fi + hgi , x − xi i} ,
i∈I
X
X
X
fh (x) = min
λi fi s.t.
λi = 1,λi xi = x,
λi ≥0

i∈I

i∈I

i∈I

(with the convention fh (x) = ∞ if the previous problem is unfeasible, i.e., if
x 6∈ conv{xi }).
Also, one should note that those interpolation procedures are in fact convex
conjugates to each others. For seeing that, we recall that (Theorem 2.21) for afunction f ∈ F0,∞ we have the following equivalencesg ∈ ∂f (x) ⇔ x ∈ ∂f ∗ (g) ⇔ f (x) + f ∗ (g) = hg, xi.
Therefore, a function f ∈ F0,∞ interpolates the set S = {(xi , gi , fi )} if and only
if its conjugate f ∗ ∈ F0,∞ interpolates the (conjugate) set S ∗ = {(gi , xi , hgi , xi i−fi )}, and f = f ∗∗ . Hence, F0,∞ -interpolation conditions on S are equivalent
to F0,∞ -interpolation of the conjugate set S ∗ .
As a conclusion, it can be proved that it is equivalent to interpolate the set S
with a construction fl : E → R ∪ {∞} (i.e., the lower bounding function) or
to interpolate the set S ∗ with fh : E∗ → R ∪ {∞} (i.e., the upper bounding
function) — that is, fl = fh∗ . Note that this can also be understood from
Lemma 2.24, as ∀f, g ∈ F0,∞ , we have f ≥ g ⇔ g ∗ ≥ f ∗ .

49

3.3. CONVEX INTERPOLATION

•

•
•

•

•

•

•

(a) Representation of a set of four coordinates, with associated subgradients andfunction values.

•(b) Lower interpolating function for the set
provided by Figure 3.2(a). This function
takes finite values for every point of E.

•

•
•

•

(c) Upper interpolating function for theset provided by Figure 3.2(a). The vertical lines on the extreme points of theinterval mean that the function takes
the value ∞ outside the convex hull of{xi }i∈I .

Figure 3.2: Upper and lower interpolating convex functions.

3.3.2

Smooth strongly convex interpolation

We now transform the smooth strongly convex interpolation problem into a
convex interpolation one. This is achieved using two previously defined operations: conjugation and minimal curvature subtraction. The reasoning is thefollowing:(i) Reformulate the Fµ,L interpolation problem into a F0,L−µ interpolation
problem using minimal curvature subtraction (Theorem 2.33).
(ii) Write the F0,L−µ interpolation problem into a F1/(L−µ),∞ interpolation
problem using (Legendre-Fenchel) conjugation (Theorem 2.34).
(iii) Transform the F1/(L−µ),∞ interpolation problem into a F0,∞ interpolation problem using again minimal curvature subtraction (Theorem 2.31).
The effect of minimal curvature subtraction on our interpolation problem, usedin steps (i) and (iii), is described by the following lemma1 .

1 In this section, we restrict ourselves to the case 0 ≤ µ < L ≤ ∞ for convenience.
However, the results can easily be adapted to the case L = µ. Note that the class FL,L (E)only contains quadratic functions of the form f (x) = L
kx − ck2E for some c ∈ E.
2

CHAPTER 3. CONVEX INTERPOLATION

50

Lemma 3.6. Consider a set {(xi , gi , fi )}i∈I with xi ∈ E, gi ∈ E∗ and fi ∈ R.
The following propositions are equivalent for any constants 0 ≤ µ < L ≤ +∞:(a) {(xi , gi , fi )}i∈I is Fµ,L -interpolable,
n
o
2
(b)
xi , gi − µBxi , fi − µ2 kxi kE
is F0,L−µ -interpolable.
i∈I

Proof. [(a) ⇒ (b)] It follows from Theorem 2.33 that if there exists f ∈ Fµ,L (E)2interpolating the set, then h(x) = f (x) − µ2 kxkE satisfies h ∈ F0,L−µ (E) and,∀i ∈ I:µ2h(xi ) = fi − kxi kE , gi − µBxi ∈ ∂h(xi ).
2
n
o
2
The setxi , gi − µBxi , fi − µ2 kxi kE
is therefore interpolated by thei∈Ifunction h ∈ F0,L−µ (E).
[(a) ⇐ (b)] If such a h ∈ F0,L−µ (E) exists and satisfies the interpolation condi2
tions (b), then one can reconstruct a function f (x) = h(x)+ µ2 kxkE , f ∈ Fµ,L (E)which interpolates the set {(xi , gi , fi )}i∈I .
The effect of conjugation in step (ii) of the reduction procedure is precisely
described in the following lemma.
Lemma 3.7. Consider a set {(xi , gi , fi )}i∈I with xi ∈ E, gi ∈ E∗ and fi ∈ R.
The following propositions are equivalent ∀L : 0 < L ≤ +∞:(a) {(xi , gi , fi )}i∈I is F0,L -interpolable,(b) {(gi , xi , hgi , xi i − fi )}i∈I is F1/L,∞ -interpolable.
Proof. [(a) ⇒ (b)] It follows from Theorem 2.34 that if there exists f ∈ F0,L (E)then f ∗ exists and satisfies f ∗ ∈ F1/L,∞ (E∗ ). In addition to that, if both f andf ∗ exists, then they satisfy ∀i ∈ I the three conditions (see Theorem 2.21):
f (xi ) + f ∗ (gi ) = hgi , xi i,gi ∈ ∂f (xi ),xi ∈ ∂f ∗ (gi ).

[(b) ⇒ (a)] If a function f ∗ ∈ F1/L,∞ (E∗ ) exists and satisfies the interpolation
conditions (b), then the conjugate f ∗∗ (which is convex, proper and closed byconstruction) satisfies f ∗∗ ∈ F0,L (E) by Theorem 2.34, as well as the interpolation conditions (see Theorem 2.21) ∀i ∈ I:
f ∗∗ (xi ) + f ∗ (gi ) = hgi , xi i,gi ∈ ∂f ∗∗ (xi ),xi ∈ ∂f ∗ (gi ).

We obtain the desired result by choosing f = f ∗∗ .
We are now properly armed to define all interpolation equivalences.

51

3.3. CONVEX INTERPOLATION

Theorem 3.8. The set {(xi , gi , fi )}i∈I is Fµ,L -interpolable if and only if thefollowing set of conditions holds for every pair of indices i ∈ I and j ∈ I1
12
fi − fj − hgj , xi − xj i ≥(3.2)
kgi − gj kE∗
2(1 − µ/L) Lµ2+µkxi − xj kE − 2 hgj − gi , xj − xi i .
LProof. We begin by showing the following equivalences:(a) {(xi , gi , fi )}i∈I is Fµ,L -interpolable,
(b)
(c)
(d)

o
n
2
xi , gi − µBxi , fi − µ2 kxi kEi∈I

is F0,L−µ -interpolable,no2gi − µBxi , xi , hgi , xi i − fi − µ2 kxi kEi∈I

is F1/(L−µ),∞ -interpolable,

n
o
−1
µLkxi k2E
kgi k2E∗
Lxi
i ,xi igi − µBxi , L−µ− BL−µgi , Lhg
L−µ − fi − 2(L−µ) − 2(L−µ)
is F0,∞ -interpolable,

(e)

n

µLkxi k2E
kgi k2E∗µhgi ,xi i
B −1 gi
Lxi
L−µ − L−µ , gi − µBxi , L−µ + fi − 2(L−µ) − 2(L−µ)

i∈I

o

is F0,∞ -interpolable.

i∈I(a) ⇔ (b) and (c) ⇔ (d) are direct applications of Lemma 3.6, whereas (b) ⇔ (c)and (d) ⇔ (e) are direct applications of Lemma 3.7. Theorem 3.8 follows
from equivalence between propositions (a) and (e) applied to the necessary
and sufficient conditions for convex interpolation of Theorem 3.4. Finally, it
is straightforward to check that condition (e) reduces to the statement of thetheorem.
It is straightforward to establish the equivalent interpolation conditions forboth the smooth but non-strongly convex case (µ = 0) and the nonsmooth
strongly convex case (L = +∞). In the first case — given by Corollary 3.9— we find the discrete version of the well-known inequality characterizing Lsmooth convex functions, which turns out to be necessary and sufficient:f (x) ≥ f (y) + h∇f (y), x − yi +12k∇f (x) − ∇f (y)kE∗ .
2L

Corollary 3.9. The set {(xi , gi , fi )}i∈I is F0,L -interpolable if and only if
fi ≥ fj + hgj , xi − xj i +12
kgi − gj kE∗ ,2L∀i, j ∈ I.

CHAPTER 3. CONVEX INTERPOLATION

52

Nonsmooth strongly convex interpolation conditions are given in Corollary 3.10,
which corresponds to the well-known inequality characterizing the subgradientsof strongly convex functions.
Corollary 3.10. The set {(xi , gi , fi )}i∈I is Fµ,∞ -interpolable if and only if
fi ≥ fj + hgj , xi − xj i +µ2kxi − xj kE ,2∀i, j ∈ I.

Remark 3.11. Note that one can also easily construct an interpolating function f (x) for the original set of points from Theorem 3.8(a).n
It followsofrom˜Theorem 3.4 that a possible interpolating function for the set
x̃i , g̃i , fii∈Iof Theorem 3.8(c) is given byh(x̃) = max f˜i + hx̃ − x̃i , g̃i +i

12kx̃ − x̃i kE∗2(L − µ)= max hi (x̃).
iThis can be conjugated into an interpolating function h∗ (x) of the set given by
Theorem 3.8(b). Using [Roc96, Theorem 16.5], this can equivalently be written
in the formh∗ (x) = convh (h∗i (x)) ,where the h∗ (x) is the function whose epigraph is the convex hull of theepigraphs of the h∗i ’s. Hence an interpolating function for the original set{(xi , gi , fi )}i∈I is given by
f (x) = convh (h∗i (x)) +µ2kxkE .
2

We provide an example of such an interpolating function on Figure 3.3.
Remark 3.12. Finding interpolation condition involving second-order derivatives seems a lot more challenging. Nevertheless, if such interpolation conditions involving second-order derivatives exist, they should rely on second-orderconvex analysis [Roc99], in which Fenchel-Legendre conjugation plays a similarrole as for first-order derivatives (see e.g., [Cro77]).
Remark 3.13. As in the non-smooth convex case (see Remark 3.5), we provide (pointwise) highest and lowest interpolating functions for smooth convex
interpolation. The upper bounding function has already been mentioned as theconvex hull of the upper bounding quadratic functions (see Remark 3.11 andFigure 3.3). The value of this function at any point can be obtained by solving

53

3.3. CONVEX INTERPOLATION

Figure 3.3: Example of an interpolating function; the data triples to beinterpolated by a 1-smooth convex function are (x1 , g1 , f1 ) = (2, 2, 3) and(x2 , g2 , f2 ) = (−3, −1, 1). Figure shows the upper-bounding quadratic functions h∗i (x) (red, left), the interpolating function f (x) = convh (h∗i (x)) (dashedblue) and the gradients (black tangents).
the following convex quadratically constrained quadratic program2 (QCQP):
fh (x) =max

d∈E∗ ,yi ∈E,c∈R

hd, xi + c,s.t. d = LB(yi − xi ) + giLL2
2
kyi kE + c + hgi , xi i − kxi kE − fi ≤ 0
2
2

(3.3)
∀i ∈ I,∀i ∈ I,

where the variables are d, c and {yi }i∈I .
The interpretation of those variables is the following: d corresponds to thegradient of fh (x) at x, c corresponds to the value of the intercept of the corresponding linear function to have fh (xi ) = hd, xi + c. On the other hand,yi corresponds to the point where the quadratic upper bound from point xi
has the slope d (required by the set of equality constraints). Finally, the setof inequality constraints requires that the quadratic upper bounds are above
hd, xi + c at point yi (and therefore that the quadratic upper bounds are always
above the linear function hd, xi + c).
The lowest interpolating function can be obtained by solving the alternative
convex QCQP:
fl (x) =min

Ld∈E∗ ,yi ∈E,c∈R 2

2

kxkE + hd, xi + c,s.t. d = gi − LByiL
2
kyi kE − hgi , xi i + fi − c ≤ 0
2(3.4)∀i ∈ I,∀i ∈ I.

2 This formulation can be further simplified, we provide it in this form for readability.

CHAPTER 3. CONVEX INTERPOLATION

54

The variables c, d, yi have the same interpretation as in the previous case. The
main difference with the upper bounding function is that this program focuses
on expressing fl (x) as the lower quadratic function with curvature L whichis globally underestimated by all first-order approximations fi + hgi , x − xi i
— whereas fu was obtained as the highest linear function which is globally2overestimated by all quadratic upper bounds fi + hgi , x − xi i + L2 kx − xi kE .
An example of lower interpolating function is provided on Figure 3.4.

Figure 3.4: Example of a lower interpolating function; the data triples to beinterpolated by a 1-smooth convex function are (x1 , g1 , f1 ) = (2, 2, 3) and(x2 , g2 , f2 ) = (−3, −1, 1). Figure shows the upper-bounding quadratic functions h∗i (x) (red, left), the interpolating function f (x) = fl (x) (dashed blue)— see (3.4) — and the gradients (black tangents).

From Remark 3.13, we arrive to the following theorem, which proposes anequivalent and more efficient QCQP for computing the highest and lowest interpolating functions.
Theorem 3.14. Let the set S = {(xi , gi , fi )}i∈I be Fµ,L -interpolable. Then,
any interpolating function f ∈ Fµ,L of S satisfies fl (x) ≤ f (x) ≤ fh (x) for allx ∈ E, withfl (x) =min f,g∈E∗ ,f ∈R1
1
2
kg − gj kE∗
s.t. f − fj − hgj , x − xj i ≥2(1 − µ/L) Lµ2+µkx − xj kE − 2 hgj − g, xj − xiL∀j ∈ I.

55

3.3. CONVEX INTERPOLATION

fu (x) =min f,g∈E∗ ,f ∈R

s.t. fi − f − hg, xi − xi ≥12(1 − µ/L)12
kgi − gkE∗Lµ2+µkxi − xkE − 2 hg − gi , x − xi i
L∀i ∈ I.

In addition, fh , fl ∈ Fµ,L are themselves interpolating functions for S.
Remark 3.15. Before going into the next section, we provide a simple andintuitive geometric interpretation for the smooth convex interpolation condition
from Corollary 3.9
f (x) ≥ f (y) + h∇f (y), x − yi +12k∇f (x) − ∇f (y)kE∗ .
2L

Let f be a L-smooth convex function and a given triple (x, ∇f (x), f (x)).
We want to find conditions characterizing the possible values for the triplet(y, ∇f (y), f (y)). For doing that, remark that the linear lower bound generatedby (y, ∇f (y), f (y)) on f should always be below the quadratic upper boundgenerated by (x, ∇f (x), f (x)). This condition is the following:f (y) + h∇f (y), z − yi ≤ f (x) + h∇f (x), z − xi +L2kz − xkE , ∀z ∈ E.
2

Rewriting this expression, one can obtain the equivalent form
0 ≤ f (x) − f (y) + h∇f (y), y − xi + h∇f (x) − ∇f (y), z − xi +L2kz − xkE ,2which should hold for every value of z ∈ E, and therefore also for the minimum(with respect to z) of the right-hand term:
0 ≤ f (x) − f (y) + h∇f (y), y − xi −12k∇f (x) − ∇f (y)kE∗ ,2Lwhich is exactly the condition from Corollary 3.9.
Example 3.16. Suppose we want to minimize a L-smooth convex function f .
We propose a simple geometric interpretation of the interpolation conditions
for finding the possible optimal points. We call this the feasible optimal set.
Let S = {(xi , ∇f (xi ), fi )}i∈I be a set of points which were already evaluated.
A new point is possibly optimal for the problem minx∈E f (x) if and only if theset S ∪ {(x, 0, f )} can still be interpolated by a L-smooth convex function (theset S is already Fµ,L -interpolable as it comes from the evaluation of f and its
gradient). That is, x is possibly optimal if and only if
fj −

1
1
2
2
k∇f (xj )kE∗ ≥ f ≥ fi + h∇f (xi ), x − xi i +k∇f (xi )kE∗ , ∀i, j ∈ I
2L
2L

CHAPTER 3. CONVEX INTERPOLATION

56

or equivalently if and only ifmin{fj −
j∈I

1
1
2
2
k∇f (xj )kE∗ } ≥ fi + h∇f (xi ), x − xi i +k∇f (xi )kE∗ , ∀i ∈ I.
2L
2L

Therefore, given the set S, it is possible that x is optimal if and only if itsatisfies the previous linear inequalities (illustration on Figure 3.5). Note that
there are two possibilities for shrinking the feasible optimal set. First, one candiminish the value of the upper bound on the optimal value f (x∗ ):f (x∗ ) ≤ min{fj −j∈I12k∇f (xj )kE∗ },2L

and on the other hand, one can add new points to the set S.
•

•

x1

•
x∗

x2

Figure 3.5: Example of feasible optimal set (see Example 3.16). The linear
constraint due to xi is orthogonal to the gradient direction ∇f (xi ).

3.3.3

Domain and gradient boundedness

In this section, we deal with SD,µ -interpolability (µ-strongly convex functions
with D-bounded domain), which will then serve to obtain CM,L -interpolation(L-smooth convex functions with M -bounded gradients) conditions using LegendreFenchel conjugation.
Theorem 3.17 (SD,µ -interpolability). The set {(xi , gi , fi )}i∈I is SD,µ (resp.
0SD,µ) interpolable if and only if the following set of conditions holds for every
pair of indices i ∈ I and j ∈ Iµ
2kxi − xj kE ,2
(resp. kxj − xi kE ≤ D).

fi − fj − hgj , xi − xj i ≥kxj kE ≤ D

57

3.3. CONVEX INTERPOLATION

0
Proof. (Necessity) Every function f ∈ SD,µ (E) (resp. f ∈ SD,µ(E)) satisfies
the conditions.

(Sufficiency) Consider the following construction:(no2
maxi fi + hgi , x − xi i + µ2 kx − xi kE
f (x) =+∞

if x ∈ conv {xi }i∈I
else,



One can note that f is µ-strongly convex (convex domain, and maximum of µstrongly convex functions), and that it indeed interpolates the set {(xi , gi , fi )}i∈I(see proof of Theorem 3.4).
Also, we note that conv {xi }i∈I ⊆ BE (0, D), with BE (0, D) the ball
Pof norm
k.kE centered at P
the origin and with radius D. Indeed, choose z = i∈I λi xi
with λi ≥ 0 and i∈I λi = 1, we havekzkE =

X

≤

λi xi

i∈I

E

X

λi kxi kE ≤ D,i∈I

and f has a bounded domain of radius D. Hence {(xi , gi , fi )}i∈I is SD,µ interpolable, which concludes the proof for the SD,µ part. To obtain
P the same0
result
for
S
,
note
that
∀y,
z
∈
conv({x
}
),
we
can
write
y
=
i
D,µ
i λi xi andPPPi∈Iz = i γi xi with λi , γi ≥ 0 and i λi = i γi = 1. Hence,ky − zkE =

X
i

≤

X
i

≤

λi (xi − z)
E

λi

X

X

λi kxi − zkE =

i

X

λi

i

X
j

γj (xi − xj )E

γj kxi − xj kE ≤ D.

j

This interpolation result can directly be used for developing interpolation conditions for the class of convex functions with bounded gradient, using the conjugate duality between smoothness and strong convexity on the one hand, andgradient and domain boundedness on the other hand.
Theorem 3.18 (CM,L -interpolability). The set {(xi , gi , fi )}i∈I is CM,L (resp0CM,L) interpolable if and only if the following set of conditions holds for every
pair of indices i ∈ I and j ∈ I
1
2
kgi − gj kE∗ ,2L(resp. kgj − gi kE∗ ≤ M ).

fi − fj − hgj , xi − xj i ≥(3.5)

kgj kE∗ ≤ M(3.6)

CHAPTER 3. CONVEX INTERPOLATION

58

0
Proof. Note that a function f ∈ CM,L (E) (resp. f ∈ CM,L(E)) interpolates theset {(xi , gi , fi )}i∈I if and only if there exists a corresponding conjugate function0f ∗ ∈ SM,1/L (E∗ ) (resp. f ∗ ∈ SM,1/L(E∗ )) interpolating the setno{(gi , xi , hxi , gi i − fi )}i∈I = (x̃i , g̃i , f˜i )i∈I(using exactly the same idea as for Lemma 3.7 along with Corollary 2.44).
Using interpolation conditions from Theorem 3.17, such a f ∗ exists if and onlyif
1
2
kx̃i − x̃j kE∗ ,f˜i − f˜j − hg̃j , x̃i − x̃j i ≥2Lkx̃j kE∗ ≤ M (resp. kx̃j − x̃i kE∗ ≤ M ),which are equivalent to conditions (3.5) and (3.6).

3.3.4

Indicator and support functions

In this short section, we specifically focus on indicator and support functions.
We refer to Section 2.5.3 for definitions and notations.
Indicator functions. Let us now consider the special case of interpolating
indicator functions. Basically, this problem is a particular case of the SD,0 (or0)-interpolation problem. This class of function is particularly interesting
SD,0
for projections in the context of performance estimation.
0)
Theorem 3.19 (ID -interpolability). The set {(xi , gi , fi )}i∈I is ID (resp. ID
interpolable if and only if the following inequalities hold ∀i, j ∈ I:

fi = 0,hgj , xi − xj i ≤ 0,
kxi kE ≤ D(3.7)(resp. kxj − xi kE ≤ D).

0
Proof. (Necessity) Any function f ∈ ID (E) (resp. f ∈ ID(E)) satisfies thoseconditions.

(Sufficiency) Let us construct a convex set whose indicator function interpolate
for the set {(xi , gi , 0)}. That is, we construct a closed convex set Q containing
all xi ’s, and such that ∀x ∈ Q we have hgi , x − xi i ≤ 0 and such that kxkE ≤ D∀x ∈ Q (resp. kx − ykE ≤ D ∀x, y ∈ Q). We start with the simpler caseD = ∞, by considering the polyhedral set
Q = {x ∈ E | haj , xi ≤ bj ∀j ∈ I} ,

59

3.3. CONVEX INTERPOLATION

with aj = gj and bj = hgj , xj i. The construction guarantees that xi ∈ Q.
Indeed, by Condition (3.7) we have:
hgj , xi i ≤ hgj , xj i,which is equivalent to haj , xi i ≤ bj using the definitions of aj and bj , andtherefore guarantees that xi ∈ Q. In order to add the boundedness requirement,
we modify the set Q in the following way:Q̃ = Q ∩ conv({xi }i∈I ).
This new set is still convex (intersection of two convex sets), it also triviallystill satisfies xi ∈ Q̃ (which are by construction both contained in Q andconv({xi }i )) and hgi , x − xi i ≤ 0 ∀x ∈ Q̃ (since Q̃ ⊆ Q). In addition, Q
has a radius bounded above by D, because D is an upper bound on the radius(resp. diameter) of conv({xi }i ). It is therefore clear that the indicator function
0
IQ̃ ∈ ID (E) (resp. ID(E)) interpolates {(xi , gi , 0)}i∈I as the convex hull has aradius (resp. diameter) D (see proof of Theorem 3.17).
Support functions. Interpolation conditions for support functions very naturally follows from those for indicator functions (see Section 2.5.3 for moredetails).
∗
0∗
Indeed, requiring a set S = {(xi , gi , fi )}i∈I to be IM(resp. IM)-interpolable
is equivalent to require the set S̃ = {(gi , xi , hxi , gi i − fi )}i∈I to be IM (or0IM)-interpolable.
∗∗
Corollary 3.20 (IM-interpolability). The set {(xi , gi , fi )}i∈I is IM(resp.
0∗IM )-interpolable if and only if the following inequalities hold ∀i, j ∈ I:

hgi , xi i − fi = 0,
hgi − gj , xj i ≤ 0,
kgi kE∗ ≤ M,

3.3.5

(resp. kgi − gj kE∗ ≤ M ).

Smooth non-convex interpolation

In this short section, we show how to extend convex interpolation to smooth
non-convex interpolation. From Lemma 2.53, it is now straightforward to establish the desired interpolation conditions.

CHAPTER 3. CONVEX INTERPOLATION

60

Theorem 3.21. Let L ∈ R++ , the set {(xi , gi , fi )}i∈I is F−L,L - interpolable
if and only if the following inequality holds ∀i, j ∈ I:
fi ≥ fj −

L
1
1
2
2
kxi − xj kE + hgi + gj , xj − xi i +kgi − gj kE∗ .
4
2
4L

Proof. As L is positive and finite, it follows from the equivalence of F−L,L interpolability of {(xi , gi , fi )}i∈I and the F0,2L -interpolability of {( xi , gi +2
LBxi , fi + L2 kxi kE )}i∈I .
As in the case of smooth convex interpolation, there exists lower and upperbounding interpolating functions (in fact, one can adapt every part of Remark 3.13 to cope with the non-convex case). We provide illustrations of those
upper and lower bounding interpolating function on Figure 3.6 and Figure 3.7.

Figure 3.6: Example of an upper interpolating smooth non-convex function;
the data triples to be interpolated by a 1/2-smooth function are (x1 , g1 , f1 ) =(2, 1, 2) and (x2 , g2 , f2 ) = (−3, −1.25, 0.5). Figure shows the upper and lowerbounding quadratic functions (red, left), the interpolating function (dashedblue) and the gradients (black tangents).

Figure 3.7: Example of a lower interpolating smooth non-convex function; thedata triples to be interpolated by a 1/2-smooth function are (x1 , g1 , f1 ) =(2, 1, 2) and (x2 , g2 , f2 ) = (−3, −1.25, 0.5). Figure shows the upper and lowerbounding quadratic functions (red, left), the interpolating function (dashedblue) and the gradients (black tangents).

61

3.4

3.4. INTERPOLATION WITHOUT FUNCTION VALUESInterpolation without function values

Our interpolation problems are extensions of the classical finite convex integration problem, which is concerned with the recovery of a convex function froma set of points xi , each associated with a subgradient gi (i.e., function values
are not specified). Finite convex integration is treated in details in [LCNS04]
in the convex case µ = 0 and L = ∞. It is the finite version of the continuous
convex integrability problem, which is treated in [Roc96].
A direct necessary and sufficient set of conditions for deciding whether a set
S = {(xi , gi )}i∈I is convex integrable is to require the existence of function
values fi for which the set {(xi , gi , fi )}i∈I is convex interpolable (see Theorem 3.4). It is however also possible to derive a set of inequalities that does not
involve unknown function values fi , using so-called cyclic monotonicity conditions. However, we will see that those conditions involve a much larger set
of inequalities, and that adding the function values as variables to the convex
integration problem is a good example of an extended formulation that renders
it tractable.
In what follows, we first provide a simple proof for the well-known finite convex
integration problem (see e.g., [LCNS04]) and then extend the finite convex
integration problem to possibly handle smoothness and strong convexity. On
the other hand, domain and gradient boundedness are not treated as they easily
follows from the other results.

3.4.1

Rockafellar’s cyclic monotonicity

Notations 3.22. Let S = {(xi , gi )}i∈I for some index set I. For avoiding thepain of using double indices in the sequel, we use the notation (z, z ∗ ) ∈ S for
meaning that z = xi and z ∗ = gi for some i ∈ I.
Definition 3.23. The set S = {(xi , gi )}i∈I is cyclically monotone if for every∗∗∗
cyclic sequence (z1 , z1∗ ), . . . , (zm , zm
) ∈ S with (zm+1 , zm+1
) =), (zm+1 , zm+1∗
(z1 , z1 ), the following (monotonicity) condition is satisfied:mXhzi∗ , zi+1 − zi i ≤ 0.

i=1

Note that the cyclic monotonicity conditions are satisfied if and only if they
are satisfied ∀m ≤ |I| (as for any cycle with m > |I|, the cycle is composedof at least two consecutive cycles of sizes m ≤ |I|). The following theorem isa slightly simplified version of an old result by Rockafellar [Roc96, Theorem
24.8].
Theorem 3.24. The set S is F0,∞ -integrable if and only if it is cyclicallymonotone.

CHAPTER 3. CONVEX INTERPOLATION

62

Proof. (Necessity) The necessity part is clear as all monotonicity conditions canbe obtained from the definition of subgradient. Indeed, let us chose a cyclic∗∗sequence (z1 , z1∗ ), . . . , (zm , zm), (zm+1 , zm+1
) ∈ S with m ≥ |I| and zm+1 = z1 .
We have that:f (z2 ) − f (z1 ) ≥ hz1∗ , z2 − z1 i
f (z3 ) − f (z2 ) ≥ hz2∗ , z3 − z2 i..
.
∗
f (z1 ) − f (zm ) ≥ hzm, z1 − zm i.

Summing those inequalities produces the monotonicity condition for the cycle:mXhzi∗ , zi+1 − zi i ≤ 0,

i=1

hence necessity.
(Sufficiency) Assuming the cyclic monotonicity conditions hold, we show that
there exists a function f ∈ F0,∞ such that gi ∈ ∂f (xi ) ∀i ∈ I. We construct f
in the following way: ∗∗f (x) =
max
hzm , x − zm i + zm−1
, zm − zm−1 + . . .
∗∗(z1 ,z1 ),...,(zm ,zm )∈S+hz1∗ , z2 − z1 i + hg0 , z1 − x0 i} ,where m ≥ |I| − 13 .
Note that by cyclic monotonicity, this (arbitrary) choice for f implies that∗) = (x0 , g0 ) we also havef (x0 ) ≥ 0, and by the choice (z1 , z1∗ ) = . . . = (zm , zm
that f (x0 ) ≤ 0, and therefore, f (x0 ) = 0, so f is proper. Also, note thatf ∈ F0,∞ , as it is the maximum of affine functions (i.e., its epigraph is theintersection of epigraphs of closed functions). In order to prove the desired
result, it remains to show that gi ∈ ∂f (xi ) ∀i ∈ I.
By definition of f , we have∗f (xi ) = hzm, xi − zm i + . . . + hg0 , z1 − x0 i∗
for some (z1 , z1∗ ), . . . , (zm , zm
) ∈ S. Therefore, we can build a global linear
underestimate for f , as ∀y ∈ E:

f (y) =max∗ )∈S
(w1 ,w1∗ ),...,(wm ,wm∗{hwm, y − wm i + . . . + hg0 , w1 − x0 i} ,∗≥ hgi , y − xi i + hzm, xi − zm i + . . . + hg0 , z1 − x0 i,
3 More precisely, m = |I| − 1 is sufficient, but it does not change much, as one can use
different copies of the same xi among the zj ’s.

63

3.4. INTERPOLATION WITHOUT FUNCTION VALUES

which shows that gi ∈ ∂f (xi ) and allows concluding the proof.

3.4.2

Smoothness and strong convexity

Using the cyclic monotonicity conditions for convex integration, we note that
Theorem 3.8 can also readily be extended to handle the finite (and continuous) integration problems for L-smooth µ-strongly convex functions (i.e., interpolation without function values). Indeed, summing inequality (3.2) from∗), (z1 , z1∗ ) ∈ S also
Theorem 3.8 over any cyclic sequence (z1 , z1∗ ), . . . , (zm , zm
produces a necessary inequality that does not involve function values fi . Moreover, we show in the next section that the set of those inequalities for all possible
sequences is necessary and sufficient for finite convex integration of L-smoothµ-strongly convex functions, generalizing the standard cyclic monotonicity conditions. As an illustration, note that the following inequality


1
1
2
2
hgi − gj , xi − xj i ≥kgi − gj kE∗ + µkxi − xj kE ,(3.8)
1 + µ/L L
is standard in the analysis of gradient methods on smooth strongly convex
functions (see e.g., [Nes04, Theorem 2.1.12]) and corresponds to cycles oflength 2. The set of all such inequalities is necessary but not sufficient4 , as itomits longer cycles.
In order to incorporate smoothness (both in the non-convex and (strongly)
convex cases), the exact same idea as for non-smooth convex integration canbe used, as for other classes of functions. Therefore, we only approach smooth
strongly convex integration in what follows.
Lemma 3.25. The set {(xi , gi )}i∈I is Fµ,L -integrable if and only if the setnoB −1 giLxi
is cyclically monotone.
L−µ − L−µ , gi − µBxi
i∈I

Proof. The following conditions are equivalent (see Theorem 3.8):(a) {(xi , gi )}i∈I is Fµ,L -integrable,(b)n

B −1 gi
Lxi
L−µ − L−µ , gi − µBxi

oi∈I

is F0,∞ -integrable.

This last theorem allows explicitly formulating the cyclic monotonicity condition for smooth strong convex functions.
4 A very classical example (for the case L = ∞ and µ = 0) is to consider a rotation
operator. It satisfies the monotonicity conditions but not the cyclic monotonicity ones. Also,
any sampling of points {(xi , gi )} taken from this operator satisfies the monotonicity condition.
Using the transformations from Theorem 3.8, one can adapt this example to be valid for any
values 0 ≤ µ < L ≤ ∞.

CHAPTER 3. CONVEX INTERPOLATION

64

Theorem 3.26. Let S = {(xi , gi )}i∈I . The set S is Fµ,L -integrable if and∗only if for any cyclic sequence (z1 , z1∗ ), . . . , (zm , zm), (z1 , z1∗ ) ∈ S we have
m h
X

1 ∗ ∗
hz , z− zi∗ iE∗ + µhzi , zi+1 − zi iE
L i i+1iµ ∗+
zi+1 − zi∗ , zi ≥ 0.
L

hzi∗ , zi − zi+1 i +i=1For the special values L = ∞ and µ = 0 admits as particular cases the smoothconvex and strong convex integration results.
Corollary 3.27. The set S = {(xi , gi )} is F0,L -integrable if and only if for∗∗), (zm+1 , zm+1
) ∈ S with zm+1 = z1 ,any cyclic sequence (z1 , z1∗ ), . . . , (zm , zm
we havem hXhzi∗ , zi − zi+1 i +

i=1

i
1 ∗ ∗
hzi , zi+1 − zi∗ iE∗ ≥ 0.
L

Corollary 3.28. The set S = {(xi , gi )} is Fµ,∞ -integrable if and only if for∗∗), (zm+1 , zm+1
) ∈ S with zm+1 = z1 ,any cyclic sequence (z1 , z1∗ ), . . . , (zm , zm
we havem hXihzi∗ , zi − zi+1 i + µhzi , zi+1 − zi iE ≥ 0.

i=1

Example 3.29. Let us consider the problem of minimizing a L-smooth µstrongly convex function f . We consider again the problem of characterizing
the feasible optimal region (subset where it is possible to find the optimal point,
see Example 3.16). For doing that, we use the monotonicity condition from
Equation (3.8). Also, let (x1 , ∇f (x1 )) be a point we evaluated; a point x is
possibly optimal if and only if the set {(x1 , ∇f (x1 )), (x, 0)} is Fµ,L -integrable.
This is equivalent to require the following


1
1
2
2
h∇f (x1 ), x1 − xi ≥k∇f (x1 )kE∗ + µkx1 − xkE ,1 + µ/L L
which is equivalent to



 2
1 1
1
(L − µ)22x − x1 −+B −1 ∇f (x1 )≤k∇f (x1 )kE∗ .
2
2 L µ
4(Lµ)
E
Hence the optimal point x∗ is within a ball of center x1 − 21 ( L1 + µ1 )B −1 ∇f (x1 )and of radius (L−µ)
2Lµ k∇f (x1 )kE∗ . In addition, this is the exact feasible optimal
set under the information provided by only the evaluation of (x1 , ∇f (x1 ), f (x1 )).

65

3.5

3.5. CONCLUSION

Conclusion

In this chapter, we explored two different ways of representing convex and/or
differentiable functions in a discrete fashion: with or without function values.
Also, we provided procedures for reconstructing the corresponding functions
along with corresponding geometrical interpretations.
The following chapters make use of those conditions in order to study theconvergence of well-known optimization methods.

CHAPTER 3. CONVEX INTERPOLATION

66

Part II

Performance Estimation
Problems

Chapter 4

Performance Estimation
Problems
The main contributions of the chapter are the following.
 We provide a methodology for computing the exact worst-case of fixedstep first-order methods for smooth convex (possibly strongly) unconstrained optimization. The methodology relies on reformulating the worstcase computation problem as a convex semidefinite program. As theworst-case computation problem is itself an optimization problem over aclass of functions, our convex SDP reformulation relies on using appropriate interpolation conditions (see Chapter 3).
 We apply our approach to different standard first-order methods, namely
the fixed-step gradient method (GM) for smooth (strongly) convex unconstrained optimization, the fast gradient method (FGM) and the optimized
gradient method (OGM).
For pedagogical reasons, the approach is described in the simpler case of thestandard Euclidean structure E = E∗ = Rd with hx, yi = x>y ∀x, y ∈ Rd and
we also exclusively focus on smooth (strongly) convex objective function in to
improve the readability of the chapter. A general approach (allowing among
others to consider constraints, regularization terms and different primal anddual Euclidean structures) is presented in Chapter 5.
 In Section 4.1, we review the concept of performance estimation problem for smooth convex unconstrained optimization. For a short reviewof the performance estimation history and other recent state-of-the-art
methodologies for analyzing first-order optimization methods, we refer to
Section 1.3.1.
 In Section 4.2, we show how the worst-case computation problem canbe formulated exactly as a (convex) semidefinite optimization problem,
69

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

70

which provides the first tractable and provably exact formulation of theperformance estimation problem. We allow consideration of both smoothconvex and smooth strongly convex functions, as well as a large class of
performance criteria (a larger class of settings is presented in Chapter 5).
We conclude the section with a tight analysis of one iteration of thegradient method (the analysis of one gradient step in the smooth strongly
convex case for general step sizes can be found in Appendix 4.A).
 Section 4.3 then tests our approach numerically on several standard firstorder methods for smooth unconstrained minimization, including theconstant-step gradient method, the fast gradient method and the optimized gradient method from [KF16d]. We are able to confirm several
bounds appearing previously in [DT14], and to conjecture several new
worst-case performance bounds, including bounds for strongly convex
functions, and bounds on the gradient norm (either for the final iterate,or the smallest norm among all iterates). Another byproduct of our results is a tight estimate of the optimal step size for the gradient methodon smooth convex and smooth strongly convex functions.
This chapter is based on sections of the paper [THG16a].

4.1

Introduction to performance estimation

Consider the standard unconstrained minimization problem
min f (x),x∈Rdwhere f is a smooth convex function, possibly strongly convex. First-order
black-box methods, which only rely on the computation of f and its gradient at
a sequence of iterates, can be designed to solve this type of problem iteratively.
A central question is then to estimate the accuracy of solutions computed bysuch a method. More precisely, given a class of problems and a first-ordermethod, one wishes to establish the worst-case accuracy of solutions that canbe obtained after applying a given number of iterations, i.e., the performanceof the method on the given class of problems.
Many first-order algorithms have been proposed in the literature for smoothconvex or smooth strongly convex functions, for which one usually provides atheoretical upper bound on the global worst-case accuracy after a number of
iterations (see e.g., [Nes04] or [Ber09, Chap.6] for recent overviews). However,many analyses focus on the order of convergence of these bounds, rather thantrying to compute exact numerical values. Similarly, lower bounds on the performance of first-order black-box methods on given classes of problems can befound in the literature (see e.g., the seminal [NY83]), again often with a focus

71

4.1. INTRODUCTION TO PERFORMANCE ESTIMATION

on order of convergence. In many situations, the order of convergence of thebest available methods match those lower bounds.
Nevertheless, the exact numerical value of the worst-case performance of a given
method is usually unknown. This is because upper bounds are not assessed precisely, i.e., are known only up to a (possibly unspecified) constant. Anotherreason is that lower bounds for specific methods are not very frequently developed, and that general lower bounds (valid for all methods) can be quite weak
for specific methods, especially if those methods do not feature the best possible order of convergence. Finally, even if exact numerical values are known forboth lower and upper bounds, and share the same (optimal) order of convergence, a significant gap between the numerical values of those lower and upperbounds can subsist. If one cares about the worst-case efficiency of a first-ordermethod in practice, this gap can translate into a very large uncertainty on theconcrete behavior of a method.
This work is not concerned with orders of convergence. It will focus on thecomputation of the exact global worst-case performance of a given first-order
black-box method, on a given class of functions, after a given number of iterations. We prove that this question can be formulated and solved exactly as a(finite-dimensional) convex optimization problem when the dimension d of theoriginal problem is large, with the following attractive features:
 Our formulation is a semidefinite optimization problem whose dimension
is proportional to the square of the number of iterations of the method
to be analyzed.
 Any dual feasible solution of our formulation provides an upper bound on
the worst-case performance. This solution can be easily converted into astandard proof establishing a bound on the performance (i.e., a series ofvalid inequalities).
 Any primal feasible solution of our formulation provides a lower boundon the worst case performance. This solution can be easily converted
into a concrete function on which the method exhibits the corresponding
performance.
 Hence our formulation is exact, i.e., its optimal value provides the exact
worst-case performance.
Our formulation covers both smooth convex functions and smooth strongly
convex functions in a unified fashion. It covers a large class of first-order methods which includes the majority of standard methods for smooth unconstrained
convex optimization. It can be applied to a variety of performance measures,such as objective function accuracy, gradient norm, or distance to an optimal
solution.

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

4.1.1

72

Formal definition

Our goal is to express the worst-case performance of an optimization algorithm
as the solution of an optimization problem. This approach was pioneered by
Drori and Teboulle [DT14], who called it a performance estimation problem(PEP). We now provide a formal definition for this problem.
We consider unconstrained minimization problems involving a given class ofobjective functions, and only treat first-order black-box methods. This meansthat the method can only gather information about the objective function
using an oracle Of , which returns first-order information about specific points,
i.e., Of (x) = {f (x), ∇f (x)}. Formally, the first N iterates generated by afirst-order black-box method M (which correspond to N calls of the oracle),
starting from an initial point x0 , can be described withx1 = M1 (x0 , Of (x0 )) ,x2 = M2 (x0 , Of (x0 ), Of (x1 )) ,
..
.

(4.1)

xN = MN (x0 , Of (x0 ), . . . , Of (xN −1 )) ,where Mi outputs the iterate after the ith iteration of M.
In order to measure the performance of a given method M on a specific function
f with a specific starting point, we introduce a performance criterion P to beminimized, that will only depend on the function f and the sequence of theiterates {x0 , x1 , . . . , xN } generated by the method. Since we are in a black-box
setting, we require that the criterion can be computed from the output of theoracle Of , which has only access to the iterates as well as to an additional pointx∗ , defined to be any minimizer of function f (the latter being necessary if thecriterion has to compare iterates to an optimal solution).
Examples of this performance criterion P(Of , x0 , . . . , xN , x∗ ) include the objective function accuracy f (xN ) − f (x∗ ), the norm of the gradient k∇f (xN )k,or the distance to an optimal solution kxN − x∗ k (see also Section 4.3.3 for an
example of criterion that does not only depend on the last iterate xN ).
Finally, we consider a given class F of smooth convex or smooth strongly convex
functions over Rd , over which we wish to estimate the worst-case performanceof a method after N iterations. As we will see in the sequel, specifying thedimension of the class F to d leads to two-regime results: the so-called small
scale regime on the one hand (when d is small compared to N ), and the largescale regime on the other one (when d is sufficiently large compared to N ).
As methods try to minimize the performance criterion, their worst-case perfor-

73

4.1. INTRODUCTION TO PERFORMANCE ESTIMATION

mance is obtained by maximizing P over functions in F, which can be written as
w(F, R, M, N, P) =sup

P(Of , x0 , . . . , xN , x∗ )(PEP)f,x0 ,...,xN ,x∗such that f ∈ Fx∗ is optimal for f,
x1 , . . ., xN is generated from x0 by method M with Of ,kx0 − x∗ k2 ≤ R.
Parameter R was introduced to bound the distance between the initial point x0
and the optimal solution x∗ . Indeed, it is well-known that in most situations,performance of a first-order method cannot be sensibly assessed without sucha constraint (see also the discussion of Section 4.2.5).

4.1.2

Finite-dimensional formulation using interpolation

Because it involves an unknown function f as a variable, problem (PEP) is
infinite-dimensional. Nevertheless, using the black-box property of the method(and of the performance criterion), we will show that a completely equivalentfinite-dimensional problem can readily be formulated by restricting the variable
f to the knowledge of the output of its oracle Of on the iterates {x0 , x1 , . . . , xN }and x∗ . Indeed, denoting the output of the oracle at each iterate xi by Of (xi ) ={fi , gi }, method M defined by (4.1) can be equivalently rewritten as
x1 = M1 (x0 , f0 , g0 ) ,x2 = M2 (x0 , f0 , g0 , f1 , g1 ) ,
..
.

(4.2)

xN = MN (x0 , f0 , g0 , . . . , fN −1 , gN −1 ) .
Now, defining a set I = {0, 1, 2, . . . , N, ∗} for the indices of the iterates, we
can reformulate (PEP) into a problem involving only the iterates {xi }i∈I , theirfunction values {fi }i∈I and their gradients {gi }i∈I as (using equivalence between optimality of x∗ and constraint g∗ = 0, as our problem is unconstrained)
wf (F, R, M, N, P) =sup
P {xi , gi , fi }i∈I ,(f-PEP){xi ,gi ,fi }i∈Isuch that there exists f ∈ F such that Of (xi ) = {fi , gi } ∀i ∈ I,g∗ = 0,x1 , . . ., xN is generated from x0 by method Mwith {fi , gi }i∈{0,...,N −1} ,kx0 − x∗ k2 ≤ R.

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

74

The crucial part of this reformulation is the first constraint, which can be understood as requiring that the set of variables {xi , gi , fi }i∈I can be interpolated
by a function belonging to the class F. This optimization problem is strictly
equivalent to the original (PEP) in terms of optimal value, since every solution
to (f-PEP) can be interpolated by a solution of (PEP) and, reciprocally, every
solution of (PEP) can be discretized to provide a solution to (f-PEP). Fromthat it is clear that w(F, R, M, N, P) = wf (F, R, M, N, P).

4.2

A convex formulation for performance estimation

As explained in the introduction, our performance estimation problem can now
be expressed in terms of the iterates and optimal point {xi , gi , fi }i∈{0,...,N,∗}
only, using the interpolation conditions given by Theorem 3.8.
As our class of functions Fµ,L (Rd ) and the first-order methods we study areinvariant with respect to both an additive shift in the function values anda translation in their domain, we can assume without loss of generality thatx∗ = 0 and f∗ = 0, which will simplify our derivations. We can also assumeg∗ = 0, from the optimality conditions of unconstrained optimization. The
problem can now be stated in its finite-dimensional formulation:(d)wµ,L (R, M, N, P) =
N +2 P {xi , gi , fi }i∈I , (d-PEP)

sup{xi ,gi ,fi }i∈I ∈ Rd ×Rd ×R

such that {xi , gi , fi }i∈I is Fµ,L -interpolable,
x1 , . . ., xN is generated from x0 by method M with (4.2),{x∗ , g∗ , f∗ } = {0d , 0d , 0} and kx0 − x∗ k2 ≤ R.
Problem (d-PEP) is an instance of (f-PEP) where the function class F is chosen
to be Fµ,L (Rd ), the set of d-dimensional L-smooth µ-strongly convex functions,(d)hence we have w(Fµ,L (Rd ), R, M, N, P) = wµ,L (R, M, N, P). Interestingly, in(d)most situations of interest, quantity wµ,L (R, M, N, P) is monotonically increasing with d, as a higher dimensional function can usually mimic a lower
dimensional one (see Theorem 4.2 and subsequent results for a discussion on
finite convergence of this sequence).
Finally, note that problem (d-PEP) is not convex, as it involves several nonconvex quadratic constraints (e.g., gj>xi terms in the interpolation conditions).
In the next section, we show how (d-PEP) can be cast as a convex semidefinite program [VB94] when dealing with a certain class of first-order black-box
methods, those based on fixed steps.

75

4.2.1

4.2. A CONVEX FORMULATION FOR PERFORMANCE ESTIMATION

Fixed-step first-order methods

We hereby restrict ourselves to the class of fixed-step first-order methods, where
each iterate is obtained by adding a series of gradients steps with fixed step
sizes to the starting point x0 .
Definition 4.1. A method M is called a fixed-step method if its iterates arecomputed according toi−1Xxi = x0 −hi,k gk .
k=0with fixed scalar coefficients hi,k .
A fixed-step method performing N steps is completely defined by the lower
triangular N × N matrix H = {hi,k }1≤i≤N,0≤k≤N −1 (where hi,k is defined to
be zero if k ≥ i). Many classical methods such as the gradient method withconstant step size (GM) and the fast gradient method (FGM) are included inthis class of algorithms (see the details in Section 4.3).

4.2.2

A convex reformulation using a Gram matrix

In order to obtain a convex formulation for (d-PEP), we introduce a Grammatrix1 to describe the iterates and their gradients. Denoting
P = [g0 g1 . . . gN x0 ]we define the symmetric (N + 2) × (N + 2) Gram matrix G = P >P ∈ SN +2 ,which is equivalent toGi,j = gi> gj for any 0 ≤ i, j ≤ N ,
GN +1,j = x>
for any 0 ≤ j ≤ N ,0 gj
G = {Gi,j }0≤i,j≤N with
>
G
=
g
x
for any 0 ≤ i ≤ N ,
i,N +1i 0GN +1,N +1 = x>x
0 0(note that the size of this matrix does not depend on the dimension of iterate
x0 and gradients gi ).
The constraints in problem (d-PEP) can now be entirely formulated in termsof the entries of the Gram matrix G along with the function values fi . Indeed
all iterates apart from x0 can be substituted out of the formulation using Definition 4.1 of a fixed-step method, and the resulting formulation only involves
function values fi and inner products between x0 and all gradients gi .
1 This sort of lifted representation was made famous among others for the MAX-CUT
problem [GW95].

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

76

Note that the initial iterate x0 and successive gradients gi of any solution to
problem (d-PEP) can be transformed into a symmetric and positive semidefinite Gram matrix G. Moreover, since vectors x0 and gi belong to Rd , matrix G
has rank at most d. In the other direction, it is easy to see that any symmetricand positive semidefinite Gram matrix G of rank at most d can be converted
back (using Cholesky decomposition for example) into N + 2 vectors x0 ∈ Rd
and gi ∈ Rd which describe the initial iterate and successive gradients of a ddimensional function (this transformation is however not unique). From thoseobservations we can anticipate that an equivalent formulation of (d-PEP) will
rely on imposing that G is symmetric and positive semidefinite, which is aconvex constraint and will naturally lead to a semidefinite program.

4.2.3

Exact worst-case performance of fixed-step firstorder methods as a semidefinite program

For notational convenience, we define vectors hi ∈ RN +2 for any i between 0
and N and h∗ ∈ RN +2 as follows (see Definition 4.1)h>i = [−hi,0 −hi,1 . . . −hi,i−1 0 . . . 0 1],h>∗ = [0 . . . 0],

so that we have xi = P hi . In order to lighten the notations we also define
ui = ei+1 ∈ RN +2 , the canonical basis vectors, and u∗ the vector of zeros. Using
those notations, we rewrite the interpolation constraints (3.2) from Theorem 3.8
in the following form for all i, j ∈ I:
1
L
(u>Ghi − u>(ui − uj )>G(ui − uj )j Ghj ) +L−µ j
2(L − µ)
µ
Lµ
>
+
(u>
(hi − hj )>G(hi − hj ).
i Ghj − ui Ghi ) +L−µ2(L − µ)fi ≥fj +We can equivalently formulate all constraints using the trace operator, andadd the distance constraint kx0 − x∗ k2 ≤ R on the starting point as well as
the positive semidefiniteness constraint for G. Defining matrices Aij and AR
in the following way for all i, j ∈ I:L
1
uj (hi − hj )> + (hi − hj )u>(ui − uj )(ui − uj )>
j +
L−µ
L−µ

µ
Lµ
+
ui (hj − hi )> + (hj − hi )u>(hi − hj )(hi − hj )>,
i +
L−µ
L−µ
AR = uN +1 u>
N +1 .

2Aij =

77

4.2. A CONVEX FORMULATION FOR PERFORMANCE ESTIMATION

We obtain the following compact formulation for the feasible region that islinear in its variables f ∈ RN +1 and G ∈ SN +2
fj − fi + Tr (GAij ) ≤ 0,

for all i, j ∈ I,

2

Tr (GAR ) − R ≤ 0,G0,with an additional non-convex rank constraint
rank(G) ≤ d,for imposing the dimension of the original problem.
From the discussion at the end of the previous section, it is easy to see thatany d-dimensional function f and starting point x0 ∈ Rd produce a feasible
solution (f, G) where matrix G has rank at most d. On the other hand, any
feasible solution (f, G) where G has rank at most d can be interpolated intoa d-dimensional function f ∈ Fµ,L (Rd ) and a starting point x0 ∈ Rd . Indeed,matrix G = P > P provides x0 ∈ Rd and N + 1 successive gradients gi ∈ Rd ,while the other iterates xi derive from the definition of the method. Ourinterpolating conditions ensure that a function compatible with these data
triples {xi , gi , fi }i∈I exists.
Considering finally the performance criterion P, we observe that any concave
semidefinite-representable function in G and f leads to a worst-case estimation
problem that can be cast as a convex semidefinite optimization problem (seee.g., [BTN01]) plus a rank constraint. In particular, linear functions of theentries of f and G are suitable. Classical performance criteria such as f (xN ) −22f∗ , k∇f (xN )k2 and kxN − x∗ k2 are indeed covered by this formulation. We
focus below on the case of a linear performance criterion, but note that othercriteria can be useful (see for example a concave piecewise linear criteria usedin Section 4.3.3).
We can now state the main result of this chapter.
Theorem 4.2. Consider the class Fµ,L (Rd ) of L-smooth µ-strongly convex
functions with 0 ≤ µ < L ≤ ∞, a fixed-step first-order method that computes N iterates according to matrix H ∈ RN ×N , and a performance criterion
Pb,C (f, G) = b> f + Tr (CG) that depends linearly on the function values at
those iterates and quadratically on the iterates and their gradients (b ∈ RN +1
and C ∈ SN +2 ).
The worst-case performance after N iterations of method H applied to somefunction in Fµ,L (Rd ) is equal to the optimal value of the following rank-

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

78

constrained semidefinite program(d)wµ,L (R, H, N, b, C) =b>f + Tr (CG)sup(sdp-PEP(d))

G∈SN +2 ,f ∈RN +1

such that fj − fi + Tr (GAij ) ≤ 0,i, j ∈ I,

2

Tr (GAR ) − R ≤ 0,
G  0,
rank(G) ≤ d.
Alternatively, if N ≤ d − 2, the worst-case performance after N iterations ofmethod H applied to some function in Fµ,L (Rd ) is equal to the optimal valueof the following convex semidefinite program
sdp
wµ,L(R, H, N, b, C) =b>f + Tr (CG)sup(sdp-PEP)G∈SN +2 ,f ∈RN +1

such that fj − fi + Tr (GAij ) ≤ 0,i, j ∈ I,

2

Tr (GAR ) − R ≤ 0,
G  0,
with matrices Aij and AR as defined above. In others words,(d)sdp(R, H, N, b, C) = wµ,L (R, H, N, b, C) for any d ≥ N + 2.
wµ,LProof. We have already shown the two-way correspondence between functions
in Fµ,L (Rd ) and feasible solutions of this problem where matrix G has rank at
most d. Since matrix G has size N + 2, it has rank at most N + 2, which establishes that this semidefinite program is a correct formulation of the performance
estimation problem when d ≥ N + 2.
sdp
The optimal value wµ,L(R, H, N, b, C) of (sdp-PEP(d)) is not necessarily finite
or attained at some feasible point. However, when L is finite, any continuous
performance criterion P will force the optimal value to be attained and finite.

Proposition 4.3. Under the assumptions of Theorem 4.2, the optimum valueof (sdp-PEP(d)) is attained and finite when L < ∞.
Proof. To show that the solution of (sdp-PEP(d)) is attained and finite, itsuffices to prove that its feasible region is compact (since the objective is continuous). We first prove that the iterates of method H applied to any function
in Fµ,L (Rd ) with L < ∞ are bounded, as well as their gradients.
Note that the Lipschitz condition on the gradients (C1f) with j = ∗ shows that
if iterate xi is bounded, gradient gi is also bounded. We proceed by recurrence.
We start with the fact that x0 is bounded, using the assumption that x∗ = 0
and constraint kx0 − x∗ k2 ≤ R. This implies that g0 is bounded, hence that79

4.2. A CONVEX FORMULATION FOR PERFORMANCE ESTIMATION

x1 is bounded using Definition 4.1 of a fixed-step method. This implies in turnthat g1 is bounded, then that x2 is bounded, and so on until we have shownthat all iterates and gradients are bounded.
Condition (C2f) with j = ∗ then implies that function values fi are bounded.
Therefore all entries in variables f and G are bounded which, combined withcloseness of the feasible region, establishes the claim.
Remark 4.4. When L = +∞ (recall the conventions 1/+∞ = 0 and +∞−µ =+∞ used in this chapter), the feasible region may be unbounded and it is possible to design feasible functions which drive standard performance criteria
arbitrarily away from 0. Nevertheless, performance estimation on such nonsmooth functions could still be tackled after introduction of another appropriate
Lipschitz condition on the class of functions, such as ||gi ||2 ≤ L. We leave thisas a topic for further research and, in the rest of this chapter, restrict ourselvesto the smooth case L < +∞. See Section 5.3 for examples with L = +∞.
Our formulation (sdp-PEP) is dimension-independent (i.e., it does not dependon the value of d), and computes the exact worst-case performance of a firstorder method with N steps as long as the class of functions of interest contains
functions of dimension at least N + 2. This corresponds to the so-called largescale optimization setting, which is usually assumed when analyzing the worstcase of first-order methods.
Using the structure of (sdp-PEP(d)), it is straightforward to establish that the(d)sequence {wµ,L (R, H, N, b, C)}d=1,2,... is monotonically increasing, and that itconverges for a finite value of d.
Corollary 4.5. The worst-case performance after N steps of a fixed-stepmethod on a L-smooth (µ-strongly) convex function is achieved by an N + 2dimensional function.
Finally, note that, when applied to the gradient method in the non-strongly
convex case (µ = 0), problem (sdp-PEP) is equivalent to one of the formulations
proposed by Drori and Teboulle in [DT14], more specifically to their problem(G). Theorem 4.2 establishes that this relaxation is in fact exact under largescale assumptions. In addition, note that the large-scale assumption N ≤ d − 2
may be conservative, as there may exist low-rank solutions to the SDP.
Remark 4.6. Existence of low-rank solutions in semidefinite programming is
an important issue which is addressed in numerous references. As examples, theseminal [Pat98, Bar01] discuss the existence of low-rank solutions depending on
the number of constraints appearing in the SDP. More precisely, for a feasible
SDP
with m affine constraints, there exists an optimal solution of rank at most√
b 8m+1−1c. As far as we know, those results do not yield interesting conclu2
sions in the case of our general formulation sdp-PEP (which has a quadratic
number of constraints). However, other frameworks for studying the properties

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

80

of the solutions of (sdp-PEP) using its structure could potentially be of application here, for example existence of low-rank solutions due to graph structures(see e.g.,[LV14]), or dimensionality reduction using lower-dimensional matrix
algebra structures (see e.g.,[DK10]). We leave further investigations in those
directions for future research.

4.2.4

A dual SDP to generate upper bounds

In general, it is not easy to find an analytical optimal solution to (sdp-PEP).
Hence, we are also interested in a generic and easier way of obtaining analytical
upper bounds on the performance of a given algorithm. A classical way of doing
so is to work with the Lagrangian dual of (sdp-PEP):X
inf τ R2 such that τ AR − C +λij Aij  0,(d-sdp-PEP)
λij ,τ

i,j∈I

b−

X

λij (uj − ui ) = 0,i,j∈Iλij ≥ 0,i, j ∈ I,τ ≥ 0,
whose feasible solutions will provide theoretical upper bounds on the worst-casebehavior of every fixed-step first-order method (using weak duality). Note thatthe final dual formulation used in [DT14], which deals with the case µ = 0, canbe recovered by taking λij = 0 for i + 1 6= j or i 6= ∗ in our dual, i.e., it is a
restriction of (d-sdp-PEP) with a potentially larger optimal value.
The next theorem guarantees that no duality gap occurs between (sdp-PEP)and (d-sdp-PEP) under the technical assumption hi,i−1 6= 0 (i ∈ {1, . . . , N }).
This assumption is reasonable as it only implies that, at each iteration, the mostrecent gradient obtained from the oracle has to be used in the computationof the next iterate. The theorem will also guarantee the existence of a dual
feasible point attaining the optimal value of the primal-dual pair of estimation
problems (sdp-PEP) and (d-sdp-PEP), i.e., a tight upper bound on the worstcase performance of the considered method.
Theorem 4.7. The optimal value of the dual problem (d-sdp-PEP) with 0 ≤sdpµ < L < ∞ is attained and equal to wµ,L(R, H, N, b, C) under the assumptions
that hi,i−1 6= 0 for all i ∈ {1, . . . , N }.
Proof. We use the classical Slater condition [BV04] on the primal problem inorder to guarantee a zero duality gap — that is, we show that (sdp-PEP) has
a feasible point with G  0. The reasoning is divided in two parts; we consider
first the case µ = 0 and L = 2+2 cos(π/(N +2)), and we generalize it to generalµ < L afterwards. Consider the quadratic function f (x) = 12 x> Qx with the81

4.2. A CONVEX FORMULATION FOR PERFORMANCE ESTIMATION

following tridiagonal positive definite matrix Q

2
1

Q=
0

..
.

1

0

0

...

2

1

0

1

2
..
.

1
..
.

...
..
.
..
.


0
0

.. 
 0.
.


We show how to construct a full-rank G feasible for (sdp-PEP) using the valuesof the quadratic function f . In order to do so, we exhibit a full-rank matrix
P = [x0 g0 g1 . . . gN ]corresponding to the application of a given method (with hi,i−1 6= 0) to thequadratic function f . Indeed, choosing x0 = Re1 , we can show that P is upper
triangular with non-zero diagonal entries. Then we have
g0 = Qx0 = 2e1 + e2 ,x1 = x0 − h1,0 g0 ,g1 = Qx1 = g0 − h1,0 Qg0 = 2e1 + e2 − h1,0 (4e1 + 4e2 + e3 ).

Hence g1 has a non-zero element associated with e3 whereas the only nonzero elements of g0 are associated with e1 and e2 . Now, we assume that gi−1
has a non-zero element corresponding to ei+1 and zero elements corresponding
to ek for all k > i + 1, while all previous gradients have zero components
corresponding to ek for all k > i. Then we have>gi>ei+2 = x>i Qei+2 = xi (ei+1 + 2ei+2 + ei+3 ),>>with x>i ei+2 = xi ei+3 = 0 and xi ei+1 6= 0 because of the recurrence assumption
and the iterative form of the algorithm:>x>
i ei+1 = x0 ei+1 −| {z }=0>x>
i ei+2 = x0 ei+2 −| {z }=0>x>
i ei+3 = x0 ei+3 −| {z }=0i−2X>hi,k gk>ei+1 −hi,i−1 gi−1
ei+1 ,| {z }
|
{z }
k=0
=0

6=0

i−2
X

>
hi,k gk>ei+2 −hi,i−1 gi−1
ei+2 ,| {z }
{z }
|
k=0
=0

=0

i−2
X

>
hi,k gk>ei+3 −hi,i−1 gi−1
ei+3 .
| {z }
|
{z }
k=0
=0

=0

Hence, gi has a non-zero element associated with ei+2 . We deduce that theCHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

82

following components are equal to zero by computing gi>ei+2+k for k > 0:>gi>ei+2+k = x>i Qei+2+k = xi (ei+1+k + 2ei+2+k + ei+3+k ),which is zero because of the algorithmic structure of xi , i.e.,>x>
i ei+1+k = x0 ei+1+k −| {z }=0i−2X>
ei+1+k .
hi,k gk>ei+1+k −hi,i−1 gi−1| {z }
|
{z }
k=0
=0

=0

Hence, P is an upper triangular matrix with positive entries on the diagonal,
and is therefore full-rank. In order to make this statement hold for generalµ < L, observe that the structure of the matrix is preserved using the operation(IN +2 is the identity matrix)
Q0 = (Q − λmin (Q)IN +2 )(L − µ)+ µIN +2 .
λmax (Q) − λmin (Q)The corresponding quadratic function is easily seen to be L-smooth and µstrongly convex. Therefore, the interior of the domain of (sdp-PEP) is nonempty and Slater’s condition applies for µ < L, ensuring that no duality gap
occurs and that the dual optimal value is attained.

One can note that Theorem 4.7 guarantees the existence of a fully explicit proof(i.e., a combination of valid inequalities, or equivalently, a dual feasible solution)
for any worst-case function (see the example at the end of this section).

4.2.5

Homogeneity of the optimal values with respect to
L and RWe observe that, for most performance criteria, one can predict how the worstcase performance depends from parameters L and R, provided the fixed step
sizes contained in H are scaled appropriately (i.e., inversely proportional to L).
In the rest of this chapter we will only consider such scaled (normalized) step
sizes. Therefore, the corresponding performance estimation problems have only
to be solved numerically in the case R = 1 and L = 1, from which a generalbound valid for any L and R can be deduced.
More specifically, a classical reasoning involving appropriate scaling operations
easily leads to the following homogeneity relations for the standard criteriaf (xN )−f∗ , k∇f (xN )k2 and kxN − x∗ k2 (a proof of the first relation is provided

83

4.2. A CONVEX FORMULATION FOR PERFORMANCE ESTIMATION

hereafter):
(d)

(d)

(d)

(d)

wµ,L (R, H/L, N, f (xN ) − f∗ ) = LR2 wκ,1 (1, H, N, f (xN ) − f∗ ),wµ,L (R, H/L, N, k∇f (xN )k2 ) = LR wκ,1 (1, H, N, k∇f (xN )k2 ),
(d)

(d)

wµ,L (R, H/L, N, kxN − x∗ k2 ) = R wκ,1 (1, H, N, kxN − x∗ k2 ),where κ = µ/L is the inverse condition number and H/L describes the fixedstep method obtained by dividing all step sizes hi,j by the Lipschitz constant
L. Results in the rest of this chapter implicitly rely on these relations.
Proof. Let us provide a proof for the relation
(d)

(d)

wµ,L (R, H/L, N, f (xN ) − f∗ ) = LR2 wκ,1 (1, H, N, f (xN ) − f∗ )to serve as an example. For doing that, we start by defining a constant α > 0
and two scaling operations A1 (α), A2 (α) : F0,∞ (Rd ) → F0,∞ (Rd ):x.
A1 (α) : f (x) → αf (x) ,A2 (α) : f (x) → α2 f
αWe make the following observations.
 First, for all f ∈ Fµ,L , we have:
A1 (α) [f ] ∈ Fαµ,αL ,A2 (α) [f ] ∈ Fµ,L .

 Second, if x1 , . . . , xN are the iterates obtained by a fixed-step first-ordermethod started at x0 on f ∈ Fµ,L , then the same iterates are generated
from x0 by the same (scaled) algorithm (i.e., whose coefficients are given
by H/(αL)) on f1 = A1 (α) [f ]. Hence, we have f1 (xN ) = αf (xN ) andf1 (xN ) − f1 (x∗ ) = α(f (xN ) − f (x∗ )), with x∗ being optimal for f and f1 .
 Similarly to the second observation, if x1 , . . . , xN are the iterates obtained by a fixed-step first-order method starting from x0 on f ∈ Fµ,L ,then the iterates αx1 , . . . , αxN are generated by the same algorithm on
f2 = A2 (α) [f ] from αx0 . Therefore, we have f2 (αxN ) = α2 f (xN ) andf2 (αxN ) − f2 (αx∗ ) = α2 (f (xN ) − f (x∗ )), with x∗ being optimal for f andαx∗ for f2 .
In order to conclude, assume that f ∈ Fµ,L generate the worst-case value off (xN ) − f∗ for a fixed-step first-order method, for some values of R, L and µ.
Then f1 = A1 (α) [f ] must generate the worst-case value for the same criterionwith R, αL and αµ. Indeed, if it was not the case, we could choose the function
reaching the worst-case of those new parameters (R, αL and αµ) and use theapplication A1 (α−1 ) in order to generate something worse than f in the original
setting, which would contradict the assumption that f (xN ) − f∗ is the worstcase. Hence, the worst-case value for the criteria f (xN ) − f∗ scales with L.

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

84

Using a similar argument with f2 , we conclude that f (xN ) − f∗ scales with R2 .

4.2.6

A simple example

Consider the very simple case of a method performing a single gradient step us3
3
ing the non-standard step-size 2L, i.e., x1 = x0 − 2L
∇f (x0 ) (this is actually thebest possible step size for a single step for smooth convex unconstrained minimization, see Appendix 4.A for a more detailed treatment of a single iterationand see Section 4.3.1 for more iterations). One wishes to estimate the worst-caseobjective function accuracy after taking that step, i.e., maximize f (x1 ) − f∗ ,over all L-smooth convex functions. Solving the corresponding
semidefinite
formulation (sdp-PEP) with µ = 0, N = 1, H = 23 and Pb,C (f, G) = f1
provides the optimal value
 



LR2
0
sdp
, 03×3 =,w0,LR, 23 , 1,
1
8
attained by the following optimal solution with rank one Gram matrix G


L
−L/2
1
LR2
LR2
f0 =
, f1 =and G = LR2 −L/2 L/4 −1/2  0.
2
8
1
−1/2 1/L
2This means that f (x1 ) − f∗ ≤ LR
holds for any f ∈ F0,L (Rd ) for any d and8provided that kx0 − x∗ k ≤ R. It is easy to check that function f (x) = L2 x2 ∈F0,L (R) achieves this worst-case when started from x0 = R. Indeed one can2
3
successively evaluate f0 = f (x0 ) = LR
2 , g0 = ∇f (x0 ) = LR, x1 = R − 2 R =2− R2 , f1 = f (x1 ) = LR
and g1 = − LR
8
2 . This function is one-dimensional
since the optimal G has rank one (note that Corollary 4.5 only guaranteed theexistence of a three-dimensional worst-case).
2

Solving the dual problem (d-sdp-PEP) leads to the same optimal value LR
8 ,attained by optimal multipliers λ01 = λ∗0 = λ∗1 = 21 ≥ 0 and τ = L8 . The
corresponding dual slack matrix is




1/L
1/L −1/2
−1/L

1
L
1/L
1/L −1/2 = −1/L −1/L −1/L 1/2  0.
S=
2
2
−1/2 −1/2 L/4
1/2
From this dual solution, a fully explicit proof of the worst-case performance canbe derived, which can be checked independently without any knowledge about
our approach. Indeed, linear equalities in the dual imply that the objective
accuracy f (x1 ) − f∗ can be written exactly as follows

85

4.3. STUDY OF STANDARD FIRST-ORDER METHODS

f (x1 ) − f (x∗ )


1
1
2
>
=
f (x1 ) − f (x0 ) + ∇f (x1 ) (x0 − x1 ) +k∇f (x0 ) − ∇f (x1 )k2
2
2L


1
1
2
>
f (x0 ) − f (x∗ ) + ∇f (x0 ) (x∗ − x0 ) +k∇f (x0 ) − ∇f (x∗ )k2
+
2
2L


1
1
2
>
f (x1 ) − f (x∗ ) + ∇f (x1 ) (x∗ − x1 ) +k∇f (x1 ) − ∇f (x∗ )k2
+
2
2L
L 1
∇f (x0 ) ∇f (x1 )L(x0 − x∗ ) −−+ kx0 − x∗ k2 −
8
2 2
L
L

2

(where for the last term we write the quadratic form Tr (SG) as a square,since S is rank-one). This equality, which is straightforward to check using3
x1 = x0 − 2L
∇f (x0 ) and ∇f (x∗ ) = 0, immediately implies inequality f (x1 ) −Lf∗ ≤ 8 kx0 − x∗ k2 , since the first three bracketed expressions are nonpositive
because of inequalites from Corollary 3.9 valid for all functions in F0,L .

4.3

Study of standard first-order methods

In this section we apply the convex PEP formulation to study convergenceof the fixed-step gradient method (GM), the standard fast gradient method(FGM) and the optimized gradient method (OGM) proposed by [KF16d].
We begin with the GM for smooth convex optimization, whose worst-case isconjectured in [DT14] to be attained on a simple one-dimensional function. Numerical experiments with our exact formulation confirm this conjecture. Further experiments on the worst-case complexity for different methods, problem
classes and performance criteria lead to a series of conjectures based on worstcase functions possessing a similar shape. We conclude this section with thestudy of a nonlinear performance criteria corresponding to the smallest gradientnorm among all iterates computed by the method.
The results were obtained using the large-scale regime formulation of PEP(without rank constraint), and are essentially numerical. They were obtainedon an Intel 3.5Ghz desktop computer using a combination of the YALMIP
modeling environment in MATLAB [L0̈4], the MOSEK [Mos10] and SeDuMi[Stu99] semidefinite solvers and the VSDP (verified semidefinite programming)
toolbox [HJL12].
Remark 4.8. Note that OGM has now been extensively studied by Kim andFessler in [KF16c, KF16b] (see survey in Section 1.3.2), by Drori in [Dro16] (seeSection 1.3.2) and been extended for composite minimization (see Chapter 5
and [KF16a, THG16b]). In addition, a projected version of the gradient method

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

86

was studied in Drori’s thesis [Dro14] (see Section 1.3.2), whereas its proximal
version for smooth (possibly strongly) convex composite optimization is further
studied in Chapter 7 (or [THG16c]).

4.3.1

Gradient method

As previously underlined, we begin by a numerical validation of a recent conjecture by Drori and Teboulle [DT14] on the behavior of the gradient method forsmooth convex unconstrained minimization for the objective function accuracyf (xN ) − f (x∗ ).
After that, we extend the conjecture to the smooth strongly convex unconstrained minimization setting in both function value accuracy f (xN ) − f (x∗ )and residual gradient norm k∇f (xN )k. On the way, we discuss the optimalvalues of the step size parameter for the different settings.
For doing that, we rely on the formulation (sdp-PEP). Extensive numerical validations suggest that the corresponding SDP provides us with one-dimensional
worst-case functions and hence, that they are also the solutions to sdp-PEP(d)for any d ≥ 1.

Conjecture on smooth convex functions by Drori and Teboulle [DT14]
Consider the classical fixed-step gradient method (GM) with constant step sizes
applied to a smooth convex function in F0,L (Rd ). Following the discussion inh, inversely proportional to L.
section 4.2.5 we use normalized step sizes L
Gradient Method (GM)
Input: f ∈ F0,L (Rd ), x0 ∈ Rd , y0 = x0 .
For i = 0 : N − 1
xi+1 = xi −h∇f (xi )L

The following conjecture on the convergence of the worst-case objective function
values was made in [DT14].
Conjecture 4.9 ([DT14], Conjecture 3.1.). Any sequence of iterates {xi }generated by the gradient method GM with constant normalized step size
0 ≤ h ≤ 2 on a smooth convex function f ∈ F0,L (Rd ) satisfies


LR2
1
2N
max
, (1 − h).
f (xN ) − f∗ ≤2
2N h + 1

87

4.3. STUDY OF STANDARD FIRST-ORDER METHODS

A proof of the conjecture is provided in [DT14] for step sizes 0 ≤ h ≤ 1, leaving
the case 1 < h < 2 open. We also recall that the upper bound in this conjecture
cannot be improved, as it matches the performance of the GM on two specificone-dimensional functions. Indeed, define(LR2LR|x| − 2(2N
if |x| ≥ 2NRh+1 ,
2N
h+1
h+1)2
f1 (x) =
L 2
else,
2x
L
f2 (x) = x2 .
2
It is straightforward to check that the final objective value accuracy of GM on
2
1
LR2
2N
f1 is equal to LR
on f2 . This
2 2N h+1 , and that it is equal to2 (1 − h)means that the conjecture can be reformulated as saying that the worst-casebehavior of the GM according to objective function accuracy is achieved byfunction f1 or f2 , depending on which of the two is worst (which will depend
only on the normalized step size h and number of iterations N ).
Intuitively, the behavior of GM on piecewise affine-quadratic f1 corresponds
to a situation in which iterates slowly approach the optimal value without
oscillating around it (i.e., no overshooting), whereas GM applied on purely
quadratic f2 generates a sequence oscillating around the optimal point. Thosebehaviors are illustrated on Figure 4.1. We also note that iterates for f1 stayon the affine piece of the function, and even never come close to the quadraticpiece. Interestingly, the existence of a one-dimensional worst-case function witha simple affine-quadratic shape will also be observed for the other algorithms
studied in this section, both in the smooth convex and in the smooth strongly
convex cases.

Figure 4.1: Behavior of the gradient method on f1 (left) and f2 (right), for
L = R = 1. We observe that GM does not overshoot the optimal solution on
f1 , while it does so at each iteration on f2 .
Empirical results from the numerical resolution of (sdp-PEP) strongly support Conjecture 4.9. Indeed, when comparing its predictions with numericallycomputed worst-case bounds, we obtained a maximal relative error of magni-

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

88

tude 10−7 (all pairs of values N ∈ {1, 2, . . ., 30} and h ∈ {0.05, 0.10, . . ., 1.95}
were tested). It is also worth pointing out that the Gram matrices computed
numerically correspond to the one-dimensional worst-case functions f1 and f2
introduced above.

Optimal step sizes. Before going into the details of other methods, we underline another observation coming from [DT14]: Conjecture 4.9 also suggests
the existence of an optimal step size hopt (N ) for the GM — optimal in the senseof achieving the lowest worst-case. That is, if you know in advance how many
iterations of the GM you will perform, it suggests using a step size hopt (N )that is the unique minimizer of the right-hand side of the Conjecture 4.9 for a
fixed value of N . It is obtained by solving2 the following non-linear equation
in hopt (for which no closed-form solution seems to be available):
1
2N hopt + 1= (1 − hopt )2N .

This optimal step size can be interpreted in terms of the trade-off between what
we obtain on functions f1 and f2 . On the one hand, we ensure that we are notgoing too slowly to the optimal point on f1 , and on the other hand we do notwant to overshoot too much on f2 .
Assuming Conjecture 4.9 holds true, one can show that the optimal step size
is an increasing function of N with 3/2 ≤ hopt (N ) < 2 and hopt (N ) → 2 as
N → ∞. More precisely, working out the expression defining hopt gives thefollowing tight lower and upper estimates3 :2−

log 4N∼ 1 + (1 + 4N )−1/(2N )2N≤ hopt (N ) ≤ 1 + (1 + 2N )−1/(2N ) ∼ 2 −log 2N.
2N(4.3)

It is interesting to compare the results from the relaxation (G’) proposed for
GM in [DT14] with ours, for values of the normalized step size h that are close
to hopt . Indeed, while the results of the two formulations are quite similar for
most values of h, it turns out that those from [DT14] are significantly more conservative in the zone around hopt , as presented in Table 4.1 for different values
2 This equation possesses several solutions, but the optimum is the unique point where
the two terms feature derivatives of opposite signs (a necessary and sufficient condition forthe maximum of two convex functions of one variable). This point can easily be computed
numerically with an appropriate bisection method.
3 Note that a bit of sensitivity analysis shows that it is preferable to use a lower boundon hopt (N ) rather than an upper bound. Hence, the use of the approximate hopt (N ) =1 + (1 + 4N )−1/(2N ) should be favored over hopt (N ) = 1 + (1 + 2N )−1/(2N ) . Also, it ispreferable to underestimate the maximum number of iterations (and therefore to use a step
size smaller than hopt ) than to overestimate it.

89

4.3. STUDY OF STANDARD FIRST-ORDER METHODS

of N . This also formally establishes the fact that the formulation from [DT14]
is a strict relaxation of the performance estimation problem.
These numerical results have been obtained with MOSEK, a standard semidefinite optimization solver. Despite convexity of the formulation, it might happen that the solution returned by such as solver is inaccurate, and in particular(slightly) infeasible. In that case, the objective value of the approximate primal(resp. dual) solution is no longer guaranteed to be a lower (resp. upper) boundon the exact optimal value, hence potentially negating the advantage of an exact convex formulation. For this reason, all numerical results reported in thissection have been double checked with an interval arithmetic-based semidefinite optimization solver [HJL12] that returns an interval that is guaranteed tocontain the optimal value. These guaranteed bounds are reported in Table 4.2
for the case h = 1.5, which compares them with Conjecture 4.9.
We can observe that the use of a verified solver does not impact our conclusions
about the validity of the conjecture. Moreover, this table is typical of what we
observed for all conjectures in this section: all numerical results reported werevalidated4 , and in what follows we will no longer mention this verification
explicitly.
Finally, we compare results obtained with Conjecture 4.9 with classical analytical bounds from the literature for the GM with unit normalized step size
h = 1 (which is usually recommended, and sometimes called optimal). The
best analytical bound we could find, e.g. in [Ber15, Proposition 6.1.7], states
that
LR2
.
(4.4)
f (xN ) − f∗ ≤2N
This analytical bound is asymptotically worse by a factor of 2 than the boundpredicted by Conjecture 4.9 with h = 1. Similarly, one can investigate theeffect of choosing the optimal normalized step size hopt (N ) instead of h = 1:
Conjecture 4.9 then predicts another improvement by a factor of 2. Theseobservations follow from the asymptotic (large N ) behaviors of the different
worst-case bounds on f (xN ) − f∗ , which can easily be computed:
Conjecture 4.9 with h = 1 :max{f (xN ) − f (x∗ )}LR2
1
2 2N +1

Conjecture 4.9 with h = hopt (N ) : lim

N →∞

= 1,

max{f (xN ) − f (x∗ )}LR2
1
2 4N +1= 1.

4 Except for tests where validation encountered numerical difficulties, i.e for which VSDP
returned no valid interval, which occurred more and more frequently as the value of theworst-case bound became closer to zero.

1.5000

1.60581.7471

1.8341

1.8971

1.92381.93881.94861.9705

1

2

5

10

20

30

40

50

100

LR /790.22

2

LR /391.72

2

LR /312.21

2

LR /232.85

2

LR /153.77

2

LR /75.36

LR /451.72

2

LR /244.70

2

LR /201.10

2

LR /156.23

2

LR /109.58

2

LR /59.80

LR2 /32.57

LR2 /36.94
2

LR2 /14.54

2

LR /8.00

LR2 /14.85

2

Value computed in [DT14]

LR /8.00

2

Conjecture 4.9

7e-01

6e-01

6e-01

5e-01

4e-01

3e-01

1e-01

2e-02

0.00

Rel. error

LR /790.22

2

LR /391.72

2

LR /312.21

2

LR /232.85

2

LR /153.77

2

LR /75.36

2

LR2 /36.94

LR2 /14.85

LR /8.00

2

Value from (sdp-PEP)

1e-07

1e-07

3e-08

7e-08

6e-08

3e-08

1e-08

5e-09

7e-09

Rel. error

12e-09
1.2e-01
2e-09

2
7e-10
7.1e-02
3e-09

5
2e-09
3.1e-02
9e-09

10
1e-09
1.6e-02
9e-08

15
9e-10
1.1e-02
2e-07

20
1e-09
8.2e-03
3e-07

30
9e-10
5.5e-03
9e-07

Table 4.2: Gradient method with relative step size h = 1.5: numerical values from Conjecture 4.9 and relative error for theupper and lower limits of the guaranteed interval obtained numerically with VSDP [HJL12] and SeDuMi [Stu99].

N
Relative error (upper limit)
Conjecture
Relative error (lower limit)

Table 4.1: Gradient Method with µ = 0, worst-case computed with relaxation from [DT14] and worst-case obtained by exact
formulation (sdp-PEP) for the criterion f (xN ) − f ∗ . Error is measured relatively to the conjectured result. Results obtainedwith MOSEK [Mos10].

hopt

N

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS
90

91

4.3. STUDY OF STANDARD FIRST-ORDER METHODS

A generalized conjecture for strongly convex functions
In view of the encouraging results obtained for the GM in the smooth case,
we now study the behavior of the GM on the class of strongly convex functions Fµ,L (Rd ) using our formulation (sdp-PEP) with the same performance
criterion, objective function accuracy. It turns out that the solution for every
problem consisted again in a one-dimensional worst-case function (rank G = 1)of the same piecewise quadratic type. We therefore introduce the following
general definitions for functions f1,τ and f2 :( µ2
if |x| ≥ τ
2 x + aτ |x| + bτ
f1,τ (x) =
L 2
else,
2x
f2 (x) =

L 2
x ,
2

 2
where scalars aτ = (L−µ)τ and bτ = − L−µτ are chosen to ensure continuity
2of f1,τ and its first derivative, and τ is a parameter that controls the radiusof the central quadratic piece (with the largest curvature). Although the valueof parameter τ could in principle be estimated from the numerical solutions ofour problems, it turns out it can be computed analytically by maximizing thefinal objective value f1,τ (xN ) (assuming that all iterates stay in the affine zone|x| ≥ τ ), which then leads toτ=Rκ(κ − 1) + (1 − κh)−2N

(4.5)

µ
where κ = L
is the inverse condition number of the problem class f ∈ Fµ,L (Rd ).
We are now able to extend Conjecture 4.9 to the GM applied to strongly convex
functions.

Conjecture 4.10. Any sequence of iterates {xi } generated by the gradientmethod GM with constant normalized step sizes 0 ≤ h ≤ 2 on a smooth
strongly convex function f ∈ Fµ,L (Rd ) satisfies


κ
LR2
2N
f (xN ) − f∗ ≤
max
,
(1
−
h)
.
2
(κ − 1) + (1 − κh)−2N

As in the previous section, this conjecture states that the worst-case behaviorof the GM according to objective function accuracy is achieved by function f1,τor f2 , depending on which of the two is worse. Proceeding now to numericalvalidations, we first point out that our results are intrinsically limited to theaccuracy that can be reached by numerical SDP solvers. For this reason, weonly report on situations for which Conjecture 4.10 predicts a final accuracy
larger than 10−6 , ensuring a few significant digits for the numerical results.

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

92

The resulting estimated relative differences between Conjecture 4.10 and thenumerical results obtained with (sdp-PEP) are given in Table 4.3, for different
values of κ. We observe that the conjecture is very well supported by ournumerical results, with a largest relative error around 10−6 , reached for thelargest value of κ considered here. This is expected as GM tends to performbetter as κ increases (i.e., final accuracy f (xN ) − f∗ approaches zero), which
renders a precise comparison between numerical results and the conjecture more
and more difficult.
κ

0

.001

.005

.010

.015

0.1

0.2

0.5

Relative error

6e-10

7e-10

4e-10

6e-10

8e-10

2e-07

9e-08

1e-06

Table 4.3: Maximum relative estimated differences between Conjecture 4.10
and corresponding numerical results obtained with SeDuMi [Stu99]. The maximum is taken over all N ∈ {1, . . . , 30} and h ∈ {0.05, . . . , 1.95} for which theconjecture predicts a worst-case larger than 10−6 .
We now investigate some consequences of our conjecture. First, we note that
Conjecture 4.10 tends to Conjecture 4.9 as µ tends to zero. This is a consequence of the fact that τ tends to 2NRh+1 as κ tends to zero (one can also
check that function f1,τ tends to function f1 introduced earlier). Hence ourformulation (sdp-PEP) closes an apparent gap between worst-case analyses ofthe smooth convex and the smooth strongly convex cases. Indeed, to the bestof our knowledge, existing worst-case bounds for the smooth strongly convex
case do not converge to the smooth case as µ → 0.
It is also interesting to compare our results to those obtained with the IQC
methodology of [LRP16]. If we only care about asymptotic linear rates of
convergence, Conjecture 4.10 predicts
f (xN ) − f∗ ≤


LR2
2N
max κ ρ2N
1 , ρ2
2

with ρ1 = |1 − κh| and ρ2 = |1 − h|(the first term in the max was obtained by neglecting (κ − 1) in the denominator). On the other hand [LRP16, Section 4.4] proves that the distance to thesolution converges linearly according tokxN − x∗ k ≤ ρN kx0 − x∗ k with a factor ρ = max{ρ1 , ρ2 }with the same values for ρ1 and ρ2 . This matches our asymptotic rate up to a
multiplicative constant.

Optimal step sizes. As for Conjecture 4.9, our new Conjecture 4.10 suggests
optimal step sizes hopt (N, κ), which can be obtained by solving the equation

93

4.3. STUDY OF STANDARD FIRST-ORDER METHODS(for 0 < κ < 1)
κ= (1 − hopt )2N(κ − 1) + (1 − κhopt )−2N

(4.6)

(note that one recovers the previous equation for hopt when µ tends to zero).
For a given N , as κ increases from 0 to 1, those optimal step sizes decrease fromhopt (N, 0) (optimal step size in the smooth case) to hopt (N, 1) = 1 (the latterbeing expected since it can only correspond to the case of function f2 in theoriginal (PEP), for which the GM with h = 1 converges in one iteration). For
a given κ, we find that hopt (N, κ) increases as N increases, as in the smoothconvex case, according to the following lower and upper estimates

1+

κ−1
1  1 + κ 2N
+
κ
κ 1−κ
(

1
− 2N

≤ hopt (N, κ) ≤ min 1 +1(κ − 1)+ (1 − κ)−2N
κ
κ

1
− 2N

2
,
1+κ

) (4.7)2
which both tend to 1+κ
as N increases (the first term appearing in the min2of the upper bound tends to 2 − κ, which is always greater than 1+κ
). This2
2
limiting normalized step size 1+κ corresponds to step size L+µ that is often
recommended for the GM, and sometimes called optimal.

We now illustrate the improvements provided by Conjecture 4.10 with respect
to the classical analytical worst-case bound found in the literature. When2
using normalized step size h = 1+κ
, iterates from GM applied to functions indFµ,L (R ) are known to satisfy (see [Nes04, Theorem 2.1.14] for example)
LR2
f (xN ) − f∗ ≤
2



1−κ
1+κ

2N
.

(4.8)

On the other hand, as the number of steps N tends to infinity, the true worstcase predicted by Conjecture 4.10 for the same step size asymptotically tends2
1−κ 2N
to LR, which is exactly the same as (4.8). Indeed, one can check that2
1+κthis rate is equal to the second term appearing in the max of Conjecture 4.10,
2
1−κ 2N
while the first term tends to LR
which is always smaller.
2 κ 1+κ
One can however do better using the optimal step size hopt . Since it is not
closed-form, we use the following approximate expression obtained after solving
a suitable approximation of equation (4.6)
1h̃opt (N ) =1 + κ 2N
1

1 + κ1+ 2N

2(note that h̃opt (N ) tends to 1+κ
as N grows), and find that Conjecture 4.10

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

predicts a worst-case with asymptotic convergence rate



1−κ
1+κ

2

94

:

1
max{f (xN ) − f (x∗ )}= κ 1+κ


2N
N →∞

lim

LR2
2

1−κ
1+κ

which improves the asymptotic rate by a factor
to lie between 34 κ1 and κ1 ).

1
κ

1
 1+κ

(which can be shown

A conjecture on the gradient norm
We now consider a different performance criterion, given by the norm of the gradient computed at the last iterate. Numerical experiments with our formulationsuggest that results similar to those presented in the previous sections can beobtained both in the smooth convex and smooth strongly convex cases, based
again on one-dimensional piecewise quadratic worst-case functions. Using thesame definition for functions f1,τ and f2 , and choosing now the parameter τ
according to
Rκ
τ=
,
(4.9)
(κ − 1) + (1 − κh)−N
we propose the following conjecture.
Conjecture 4.11. Any sequence of iterates {xi } generated by the gradientmethod GM with constant normalized step sizes 0 ≤ h ≤ 2 on a smooth
strongly convex function f ∈ Fµ,L (Rd ) satisfiesκNk∇f (xN )k2 ≤ LR max, |1 − h|.
(κ − 1) + (1 − κh)−NAs for Conjecture 4.10, we limit our numerical validation to the cases where
the worst-case values predicted by the Conjecture are larger than 10−6 ; thelargest relative error is about 10−7 .
We note that, as κ tends to zero (i.e., the smooth case), Conjecture 4.11 tendsto1k∇f (xN )k2 ≤ LR max, |1 − h|N .
Nh + 1
Optimal step sizes. From that, we see that the optimal step size h∇opt (N, 0)√∇
for the GM is again an increasing function of N with 2 ≤ hopt (N, 0) < 2 andh∇opt (N, 0) → 2 as N → ∞. In the strongly convex case κ > 0, the optimal step
size is a decreasing function of κ and satisfies h∇opt (N, κ) → 1 as κ → 1. As in

95

4.3. STUDY OF STANDARD FIRST-ORDER METHODS

2
the previous case, h∇
opt (N, κ) is bounded above by 1+κ , which we can confirm
with the following lower and upper bounds on h∇
opt :


1+

−1/N
κ − 1  1 + κ N
+
κ
1−κ
(


≤ h∇
opt (N, κ) ≤ min

1+

κ−1
1
+ (1 − κ)−N
κ
κ

−1/N

2
,
1+κ

)
.

In the smooth case, those bounds reduce to the simpler expression
2−

log N
log 2N−1/N−1/N∼ 1 + (1 + 2N )
≤ h∇
∼2−
.
opt ≤ 1 + (1 + N )NN

We now compare with a standard analytical worst-case bound. The iterates of2
the GM method with normalized step size 1+κ
are known to satisfykxN − x∗ k2 ≤ R

1−κ
1+κ

N
and


k∇f (xN )k2 ≤ LkxN − x∗ k2 ≤ LR

1−κ
1+κ

N

(4.10)(see for example [Nes04] for the left inequality, and use the L-Lipschitz propertyof the gradient along with ∇f (x∗ ) = 0 to derive the right inequality). The latter
estimate is tight according to Conjecture 4.11. Using the following approximate
optimal step size
1
1 + κN
h̃∇
(N
)
=
1
opt
1 + κ1+ N
2
as N grows) can be shown to improve the conjectured(which tends to 1+κ
1

asymptotic rate by the same factor κ− 1+κ as for convergence in function values.

4.3.2

Fast gradient and optimized gradient methods

In this section we assess the performance in the smooth convex case (µ = 0) of
two accelerated first-order methods: the so-called fast gradient method (FGM)
due to Nesterov [Nes83], and an optimized gradient method (OGM) recently
proposed by Kim and Fessler [KF16d].

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

96

Fast Gradient Method (FGM)
Input: f ∈ F0,L (Rd ), x0 ∈ Rd , y0 = x0 , θ0 = 1.
For i = 0 : N − 11
yi+1 = xi − ∇f (xi )Lp
1 + 4θi2 + 1
θi+1 =
2
θi − 1xi+1 = yi+1 +(yi+1 − yi )
θi+1

Optimized Gradient Method (OGM)
Input: f ∈ F0,L (Rd ), x0 ∈ Rd , y0 = x0 , θ0 = 1.
For i = 0 : N − 11
yi+1 = xi − ∇f (xi )L√ 1+ 4θi2 +1, i≤N −2√2θi+1 =
 1+ 8θi2 +1 , i = N − 12xi+1 = yi+1 +θi − 1θi(yi+1 − yi ) +(yi+1 − xi )
θi+1
θi+1

Both of these algorithms are defined in terms of two sequences: {yi }i is a
primary sequence, and {xi }i is a secondary sequence, where the gradient isevaluated. We first show that both of these algorithms can be expressed as
fixed-step first-order methods, which we defined as
xi = x0 −i−1Xhi,k ∇f (xk )(for L = 1).

k=0

One way to proceed is to focus on the secondary sequence {xi }i and substitutethe yi ’s in the algorithm formulation. For FGM, we haveθi − 1 gi
gi−1 
gi+xi − xi−1 − +
,
L
θi+1
L
L


θi − 1
θi − 1gi
θi − 1 gi−1
= xi +(xi − xi−1 ) −
+1
+
,
θi+1
θi+1
L
θi+1 Lxi+1 = xi −

97

4.3. STUDY OF STANDARD FIRST-ORDER METHODS

which allows to obtain the step sizes relative to x0 by recurrence:

θi −1

 hi,k + θi+1 (hi,k − hi−1,k ) if k ≤ i − 2,
−1
hi,k + θθii+1(hi,k − 1)
if k = i − 1,
hi+1,k = θi −1 + 1
if k = i,θi+1
with initial conditions h1,0 = 1, h1,k = 0 if k < 0 and h0,k = 0 for all k.
Similarly, we have for OGM

θi −1

 hi,k + θi+1 (hi,k − hi−1,k ) if k ≤ i − 2,
−1
hi,k + θθii+1(hi,k − 1)
if k = i − 1,
hi+1,k = 2θi −1 + 1
if k = i,θi+1
with the same initial conditions. This approach will provide estimates for thelast secondary iterate xN . If an estimate for last primary iterate yN is needed,
one just has to replace the expression of xN by yN , which is done by using thefollowing alternative coefficients for the last step:hN −1,k if k ≤ N − 2,hN,k =
1
if k = N − 1,for both FGM and OGM.
Again, our numerical experiments strongly suggest the same assumption about
the shape of the worst-case functions, i.e., one-dimensional and piecewise quadratic(with iterates staying in the affine zone of f1,τ ). Using this property, we areable to compute the following values of τ achieving the worst-case final objective accuracy, which surprisingly hold for both the classical FGM and the more
recent OGM (a coincidence for which we can offer no explanation)
τ1 =R

for the primary sequence,hN −1,k + 3Rτ2 = PN −1
for the secondary sequence.
2 k=0 hN,k + 1
2

PN −2
k=0

Our numerical results suggest the following two conjectures (validations forboth conjectures were performed for values of N ∈ {1, . . . , 100} and displayeda relative error less than 10−4 ).
Conjecture 4.12. Any (primary) sequence of iterates {yi } generated by thefast gradient methd FGM (resp. optimized gradient method OGM) on asmooth convex function f ∈ F0,L (Rd ) satisfies
f (yN ) − f∗ ≤ f1,τ1 (y1,N ) =

LR2
1
,
PN −2
2 2 k=0hN −1,k + 3

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

98

where y1,N is the final (primary) iterate computed by FGM (resp. OGM)
applied to f1,τ1 starting from x0 = R, and quantities hN −1,k are the fixedcoefficients of the last step of FGM (resp. OGM).
Conjecture 4.13. Any (secondary) sequence of iterates {xi } generated by
the fast gradient methd FGM (resp. optimized gradient method OGM) on asmooth convex function f ∈ F0,L (Rd ) satisfies
f (xN ) − f∗ ≤ f1,τ2 (x1,N ) =

1
LR2
,
PN −1
2 2 k=0 hN,k + 1

where x1,N is the final (secondary) iterate computed by FGM (resp. OGM) applied to f1,τ2 starting from x0 = R, and quantities hN,k are the fixed coefficientsof the last step of FGM (resp. OGM).
Note that Conjecture 4.13 has now been proved by Kim and Fessler [KF16d](revised version) in the case of OGM.
The worst-case bounds in these two conjectures involve the normalized step
sizes of the FGM and OGM methods. It turns out these can be computed inclosed form for OGM (see also [KF16c]), and give (N ≥ 1)
f (yN ) − f∗ ≤

LR2
2
LR2
≤
,
2
2+2
4θN
+
2
2
(N
+
1)
−1

f (xN ) − f∗ ≤

LR2
LR2
2
√
≤
2
2θN
2 (N + 1)(N + 1 + 2)
2

and

√

(N +1)
22(where the inequalities rely on θN≥ (N +1)(N2+1+ 2) ). We
and θN−1 ≥4
were not able to obtain similar closed-form bounds for the FGM.

We now compare the numerical values obtained with Conjectures 4.12 and 4.13
with analytical bounds known for the FGM. We use for the primary sequencef (yN ) − f∗ ≤2LR2,
(N + 1)2(4.11)

which can be found in [BT09b, Theorem 4.4], and for the secondary sequencef (xN ) − f∗ ≤2LR2
(N + 2)2(4.12)

which was very recently derived in [KF16d, Theorem 1]. The comparison isdisplayed on Figure 4.2. The asymptotic behaviors of both sequences are well
captured by the analytical bounds (4.11) and (4.12), but we observe that theestimation of the transient worst cases are improved by our conjectures: a factor
approximately equal to 1.15 is gained for both sequences after 30 iterations.

99

4.3. STUDY OF STANDARD FIRST-ORDER METHODS

Before going into the next section, we comment on the applicability of our results to monotone variants of first-order methods, i.e. methods which guaranteef (yi+1 ) ≤ f (yi ). Consider for example FISTA [BT09b], which is equivalent to
FGM when applied to smooth unconstrained minimization. MFISTA [BT09a],a monotone variant of FISTA. As FISTA happens to generate a monotonically decreasing sequence {f (yi )}i when applied to our worst-case function
f1,τ1 from x0 = R, the corresponding lower bound from Conjecture 4 also
applies to MFISTA.

4.3.3

Estimation of the smallest gradient norm among all
iterates

First-order methods are often used in dual approaches where, in addition toobjective function accuracy, gradient norm plays an important role. Indeed,this quantity controls primal feasibility of the iterates (see e.g., [DGN12]).
Considering for example the accelerated FGM in the smooth case, we know
from the previous section that the classical analytical bound on the worst-case
2LR2
accuracy for a function in F0,L (Rd ) is given by (N+1)2 . From that bound, it iseasy to obtain a similar bound on the last gradient norm, using Corollary 3.9(convexity and smoothness):k∇f (yN )k2 ≤p
2LR
2L(f (yN ) − f∗ ) ≤.
N +1(4.13)

Observe that this asymptotic rate is significantly worse than that of the objective function accuracy, and not better than that of the gradient method GM(see Conjecture 4.11).
However, it is well-known that the norm of the gradient is not decreasing monotonically among iterates of the FGM. Hence, in this section, we will estimate
the worst-case performance of FGM according to the smallest observed gradientnorm among all iterates:mini∈{0,...,N }k∇f (yi )k2 .

In order to do so, only a slightly modified version of (sdp-PEP) is needed:
this min-type objective function is representable using a new variable t for the2
objective and N + 1 additional linear inequalities t ≤ k∇f (yi )k2 ⇔ t ≤ Gi,ifor all 0 ≤ i ≤ N . Note that the maximum is still attained since this concave
piecewise linear objective function is continuous.
This criterion was suggested in [Nes12b], which proposes a variant of FGM thatconsists in performing N/2 steps of the standard FGM followed by N/2 stepsof the GM with h = 1. It is then theoretically established that this variant ofCHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

100

FGM, which we denote by MFGM, satisfiesmini∈{0,...,N }k∇f (yi )k2 ≤

8LR
,
N 3/2

(4.14)

an improvement compared to the rate of convergence of the gradient of the last
iterate.
We now compare FGM with this modified variant MFGM using our performance estimation formulation. Figure 4.3 compares the behaviors of thosemethods in both their last (for FGM) and best iterates, as well as the above
analytic bounds (4.13) and (4.14).
This experiment confirms that the gradient norm of the last iterate of FGM
decreases according to the slower O(N −1 ) rate of (4.13). We also observe thatboth the MFGM and the original FGM achieve the same O(N −3/2 ) convergence
rate for the smallest gradient norm, which was not known before for FGM. In
addition, numerical results reported in Table 4.4 suggest that FGM performs
slightly better than MFGM. Note that the convergence rate O(N −3/2 ) of FGM
has now been proved analytically by Kim and Fessler in [KF16b], using cleverrelaxations of the performance estimation problem.
A regularization technique is also described in [Nes12b], featuring a O(N −2 )convergence rate up to a logarithmic factor. A drawback of this approach isthat it requires a bound on the distance to the optimal solution, and thatthe coefficients of the method explicitly depend on this bound. No fixed-stepmethod achieving the same O(N −2 ) seems to be known.
0Upper bound on fN−f* (log scale)

10

−1

10

−2

10

−3

10

−4

10

0

10

1

10
Iteration count (log scale)2

10Figure 4.2: Comparison of the worst-case performance of the FGM: analytical bound (4.11) (dashed red) versus Conjecture 4.12 (red) and analyticalbound (4.12) (dashed blue) versus Conjecture 4.13 (blue).

101

4.3. STUDY OF STANDARD FIRST-ORDER METHODS

1

10

0

||∇ f(xi)|| (log scale)

10

−1

10

−2

10

−3

10

−4

10

0

10

1

2

10
10
Iteration count (log scale)3

10

Figure 4.3: Comparison of gradient norm convergence rates for the FGM andthe MFGM from [Nes12b]. Theoretical guarantees are dashed. Analyticalbound on FGM (4.13) in its last iterate (dashed blue); numerical worst-casefor FGM at its last iterate (blue); numerical worst-case for FGM at its best
iterate (red); analytical bound on MFGM (4.14) for the best iterate (dashedblack); numerical worst-case for MFGM at its best iterate (black).

LR/1.50

LR/2.50

LR/5.50

LR/10.50

LR/15.50

LR/20.50

LR/25.50

LR/50.50

LR/100.50

2

4

10

20

30

40

50

100

200

LR/205.77

LR/105.49

LR/55.25

LR/45.19

LR/35.13

LR/25.08

LR/15.14

LR/5.84

LR/3.00

FGM, last

LR/850.59

LR/311.34

LR/117.08

LR/86.17

LR/58.50

LR/34.49

LR/15.62

LR/5.84

LR/3.00

FGM, best

LR/353.55

LR/125.00

LR/44.19

LR/31.62

LR/20.54

LR/11.18

LR/3.95

LR/1.00

LR/0.35

MFGM, analytic (4.14)

LR/791.87

LR/296.58

LR/119.63

LR/86.41

LR/55.38

LR/30.77

LR/12.66

LR/5.00

LR/3.00

MFGM, best

Table 4.4: FGM and MFGM: comparison between theoretical bounds and numerical results for the criteria k∇f (xN )k2 (last)and mini k∇f (xi )k2 (best). Results obtained with [Mos10].

FGM, analytic (4.13)

N

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS
102

103

4.4

4.4. CONCLUSION

Conclusion

The contribution of this chapter is threefold: first, we presented why it was
crucial to develop necessary and sufficient conditions for smooth strongly convex interpolation. Those conditions were derived by showing an explicit way
of constructing the interpolating functions in Chapter 3. Second, we showthat the exact worst-case performance of any fixed-step first-order algorithm
for smooth strongly convex unconstrained optimization can be formulated asa convex problem. In this context, our interpolation procedure also provides
explicit functions achieving the worst-case bounds computed by our approach.
Third, we test of our formulation numerically on a variety of functions classes,
first-order methods and performance criteria, establishing on the way a seriesof conjectures on the corresponding worst-case behaviors. In particular, wesuggest new tight estimates of the optimal step size for the fixed-step gradientmethod with constant step size, which depend on the number of iterations andthe condition number.
Our performance estimation problem provide a generic tool to analyze fixedstep first-order methods. It allows computing both exact worst-case guarantees
and functions reaching them, and provides a unified algorithmic analysis forsmooth convex functions and smooth strongly convex functions.
The exact worst-case values provided by our approach require solving a convex
semidefinite program whose size grows as the square of the number of iterations
considered, which may become prohibitive when this number of iterations islarge. This can be avoided using iteration-independent bounds, as proposedin [LRP16], but at the cost of obtaining poorer worst-case guarantees.
Further extensions of the results and the methods presented in this chapter
have already been published. Among others, the gradient method with exact
line search has now been studied using PEP (see Chapter 6 or [dKGT16]).
Also, Drori partially studied the projected GM in his thesis [Dro14], and theframework has been adapted to cope with composite objective functions (seeChapter 5 or [THG16b]). Also, a cyclic coordinate descent for unconstrained
minimization was studied in [SL16] using a relaxed performance estimation
approach. More extensions of the PEP framework can be found in Chapter 8.
In addition, various results related to optimized methods have appeared. First,
as we already emphasized, the optimized gradient method for smooth unconstrained convex optimization has now been proved to be optimal by Droriin [Dro16]. Also, different extensions to OGM appeared: Kim and Fessler havefurther studied optimized methods for other convergence measures in [KF16b],
whereas they propose a proximal extension in [KF16a].

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

104

Appendix
4.A

Tight worst-case of a gradient step

Theorem 4.14. Any iterate x1 generated by the gradient method (GM) withconstant normalized step size 0 ≤ h ≤ 2 on a smooth strongly convex functionf ∈ Fµ,L (Rd ) satisfies
κ
L||x0 − x∗ ||2
2
max
,
(1
−
h)
,
f (x1 ) − f∗ ≤2
(κ − 1) + (1 − κh)−2µwith κ = L
the inverse condition number.
√

2

Note that this implies the following value for hopt (1) = κ+1− κκ −κ+1 (which
3tends to 2L
as µ → 0).
Proof. The theorem can be deduced by combining the following (interpolation)
inequalities with the appropriate coefficients λ0 , λ1 and λ2
0:

1:

2:

1
2
kg1 − g0 k2
2L
2
µ
1
+
x1 − x0 − (g1 − g0 ) ,2(1 − µ/L)
L
2
1
2
kg0 k2f∗ ≥ f0 + g0> (x∗ − x0 ) +
2L
2
µ
1
+
x0 − x∗ − g0 ,2(1 − µ/L)
L
2
1
2
>
f∗ ≥ f1 + g1 (x∗ − x1 ) +kg1 k22L
2µ1+x1 − x∗ − g1 .
2(1 − µ/L)L2
f0 ≥ f1 + g1> (x0 − x1 ) +halong with x1 = x0 − γg0 (and γ = L
the step size). Indeed, using the105

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

106

coefficients (we term this solution as the small step size regime in the sequel)λ0 = λ1 =1 − γµ,2 − γµλ2 =1,2 − γµor (we call this solution the large step size regime in the sequel)λ0 = λ1 = γL − 1,

λ2 = 2 − γL,

respectively allows to obtain2

f1 − f∗ ≤Lkx0 − x∗ k2
κ−22(1 − γµ) + (κ − 1)2

p1 (γ)  g0
g1 
L2
x0 − x∗ ++2(L − µ)p1 (γ)(γµ − 2) L
L 2
p2 (γ)
2−k(γµ − 1)g0 + g1 k2 ,2(L − µ)(2 − γµ)2

−

or
2

f1 − f∗ ≤Lkx0 − x∗ k2(1 − γL)22
2
L2 p3 (γ)
1 + γ 2 Lµ − γ(L + 2µ)
γL − 2−x0 − x∗ +g0 +g1
2(L − µ)Lp3 (γ)Lp3 (γ)2−p2 (γ)2−k(γL − 1)g0 + g1 k2 ,2(L − µ)p3 (γ)

with p1 (γ) = 1 + 2γ(L − µ) − γ 2 µ(L − µ) (which is nonnegative on the interval
0 ≤ γ ≤ L2 , as a simple analysis shows that the the interval [−µ, µ2 ] lies between
the roots, and that this interval contains both 0 and L2 ), p2 (γ) = 3 + γ 2 µL −2γ(L + µ) and p3 (γ) = 1 − 2γ(L − µ) + γ 2 L(L − µ) (which is positive everywhereas L ≥ µ).
In order to conclude, we have to verify that:(a) there is always one of the two solutions that is valid ∀γ: 0 ≤ γ ≤ L2 ,(b) the valid solution is the one producing the worst value (i.e. maximum
value).
We start by treating (a); in order to determine which solution is valid fora particular value of γ, let us denote by γmin and γmax the roots of p2 (γ)(previously defined positive definite quadratic function). They respectively
take the valuespp
L + µ − L2 − Lµ + µ2
L + µ + L2 − Lµ + µ2
γmin =, γmax =
.
Lµ
Lµ

107

4.A. TIGHT WORST-CASE OF A GRADIENT STEP

One can easily obtain that L1 ≤ γmin ≤ µ1 and L2 ≤ γmax .
Concerning the region of validity of the two regimes, we have
 the small step size regime is valid when both γ ≤ µ1 (multipliers should bepositive) and p2 (γ) ≥ 0 (coefficients of the norms should be positive), so
γ should be outside of the interval [γmin , γmax ]. This regime is therefore
valid for any γ such that 0 ≤ γ ≤ γmin ≤ µ1 .
 The large step size regime is valid when both γ ≥ L1 (multipliers should
be positive) and p2 (γ) ≤ 0 (coefficients of the norms should be positive),so γ should be inside the interval [γmin , γmax ]. This regime is therefore
valid for any γ such that L1 ≤ γmin ≤ γ ≤ L2 ≤ γmax .
Therefore, the small step size regime is valid ∀γ such that 0 ≤ γ ≤ γmin and thelarge step size regime is valid in the complementary region, when γmin ≤ γ ≤ L2 .
In order to check which solution dominates the other, we distinguish threecases:
 in the case 0 ≤ γ ≤ L1 , we haveκ≥ (1 − µγ)2 ≥ (1 − γL)2 ,(κ − 1) + (1 − µγ)−2so the small step size regime is indeed the one producing the larger value(in addition to being the only valid one).
 When L1 ≤ γ ≤ µ1 , there can be only one intersection between theκ
two regimes (that is, only one value of γ such that (κ−1)+(1−µγ)−2 =2(1 − Lγ) ), since the first is a decreasing function of γ, the second is anincreasing function of γ and the first is larger than the second at γ = L1 .
Also note that γmin is exactly the value of the intersection.
µ≥ 12 ), only the large step regime When µ1 ≤ γ ≤ L2 (only possible when L
is valid. It dominates the small step regime as the value of the small step1d
κ
size regime is smaller at γ = µ and as dµ (κ−1)+(1−κh)−2 ≥ 0 whenκ2as κ → 1.
γ ≥ µ1 , with (κ−1)+(1−µγ)−2 → (1 − Lγ)

CHAPTER 4. PERFORMANCE ESTIMATION PROBLEMS

108

Chapter 5

Performance Estimation
Problems for Composite
Convex Optimization
The main contributions of the chapter are the following.
 We further study the performance estimation framework, and extend
it with the possibility of handling a large class of algorithms, function
classes, convergence measures and initial conditions. As in Chapter 4,
the approach allows formulating the worst-case estimation problem as aconvex SDP using the convex interpolation framework (see Chapter 3).
 We apply the method to standard first-order methods, namely the proximal point algorithm (PPA), fixed-step projected subgradient method(PSM), different variants of fast proximal gradient methods (FPGM),
a conditional gradient method (CGM) and to two alternate projection
methods (APM).
Concerning the use of notations for primal and dual spaces, norms and scalar
products, we refer to Section 2.1.
This chapter is divided into three main parts.
 First, Section 5.1 introduces the composite optimization problems forwhich the performance estimation framework of this chapter is tailored,
and reviews the necessary concepts for handling those problems.
 Section 5.2 is concerned with putting in place the performance estimation
framework for large classes of first-order algorithms, objective functions,performance criteria and initial conditions. The main idea of this section
is to require every element of the performance estimation problem to be
109

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

110

linearly Gram-representable. This section contains multiple examples ofstandard settings for which the methodology applies — including the settings of (sub)gradient methods (along with their projected and proximal
counterparts), and conditional gradient methods.
 Finally, Section 5.3 is concerned with the application of the methodology
to several concrete first-order algorithms. We provide several numerical and analytical improvements on the analysis of well-known methods, including the proximal point algorithm and the conditional gradientmethod.
The subsequent text is based on sections of the following preprint [THG16b].

5.1

Introduction

Consider the convex composite1 minimization problem
(
)
n
X
(k)
min F (x) ≡F (x) ,
x∈E

(CM)

k=1

where E is a finite dimensional real vector space and each functional component
F (k) : E → R ∪ {∞} is a convex function belonging to some class Fk (E) —e.g., smooth or non-smooth, strongly convex or not, indicator functions, etc.
— for which some operations are assumed to be available in closed-form (e.g.
computing a gradient, projecting on the domain, computing a proximal step,etc.).
We are interested in the composite optimization problem (CM) because it naturally allows representing and exploiting a lot of the structure in many problems,
which can play a major role in our ability to efficiently solve them (see [Nes13]
among others). In addition, the class of composite convex optimization problems arises very commonly in practice, as it contains for example constrained,`1 and `2 -regularized convex optimization problems.
We focus on black-box oracle-based algorithms that use first-order information
to approximately solve (CM), and in particular on obtaining exact and global
worst-case guarantees on their performances. That is, for a given algorithm, we
simultaneously seek to obtain worst-case guarantees — for example on objectivefunction accuracy — and an instance of (CM) on which the algorithm behaves
as such. In this work, we treat the case of fixed-step linear first-order methods,
which includes among others fixed-step projected, proximal, conditional andinexact (sub)gradient methods.
1 We term this objective function as composite because the terms may be of different
natures (smooth, non-smooth, indicator, etc.); this contrasts with minimization of finite
sums where all the terms share similar properties (see e.g., [Ber10]).

111

5.1. INTRODUCTION

This work builds on the recent idea of performance estimation, first developedby Drori and Teboulle in [DT14] and followed-up by Kim and Fessler [KF16d]
and the authors [THG16a]. The approach was initially tailored for obtaining
upper bounds on the worst-case behavior of fixed-step gradient methods forunconstrained minimization of a single smooth convex objective function. Motivated by follow-up results (see among others [KF16c, KF16d]) we extend theframework of performance estimation to the composite case involving a muchbroader class of algorithms and function classes (see Section 1.3.1 for moredetails about previous works).
Our performance estimation framework relies on formulating the worst-casecomputation problem as a tractable semidefinite program (SDP), which can betackled by using standard solvers [L0̈4, Mos10, Stu99]. It enjoys the following
attractive features:
 any primal feasible solution to this SDP leads to a lower bound on theworst-case performance of the method under consideration, by exhibitinga particular instance of (CM), any dual feasible solutions to this SDP corresponds to an upper boundon the worst-case performance of the method under consideration, thatcan be converted into an explicit proof based on a combination of valid
inequalities.

5.1.1

Performance estimation problems

In Chapter 4, we introduced a formal definition for the performance estimation
problem in the case of a black-box first-order methods for unconstrained minimization of a single convex function F . We now formalize the performance
estimation framework for handling multiple components in the objective function.
First, we consider black-box methods formalized using the concept of black-box
oracles. That is, methods are only allowed to access the different componentsof the objective function via calling some routines returning some information
about them at a given point. In particular,
we focus oon the standard firstn
(k)
(k)
(k)
˜
order oracle for F : OF (k) (x) = F (x), ∇F (x) in the sequel. The
general formalism of the approach is nevertheless also valid for other standard
oracles, as for examples zeroth-order
or second-order ones — that is, OF (k) (x) =F (k) (x) or OF (k) (x) = F (k) (x), ∇F (k) (x), ∇2 F (k) (x) . However, as we willsee, our ability to solve the corresponding performance estimation problems inan exact way is limited to first-order oracles at the moment.
Second, we consider a sequence of N +1 iterates {xi }0≤i≤N ⊂ E, corresponding
to a method that performs N steps from an initial iterate x0 . For each of thoseCHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

112

iterates we consider the set of calls to the oracle for each functional component2
OF (k) : {OF (k) (xi )}i . For notational convenience we denote by K = {1, . . . , n}
the set of indices corresponding to the different components F (k) .
Third, we consider a method M whose iterates can be computed by combining
past and current oracle information about F . This means that after i − 1 steps
have been performed by the method, the next iterate xi should be computable
as a solution to an equation of the form:
Equation(x0 , {OF (k) (x0 )}k∈K , x1 , {OF (k) (x1 )}k∈K , . . . , xi , {OF (k) (xi )}k∈K ).
(5.1)
Note that the only unknown in this equation is xi , and that it thus provides
an implicit definition for the next step. We will see later that this assumption
on M includes a large number of existing methods for composite optimization.
Finally, we consider a real-valued performance criterion P, for which we assume that lower values are better. In our framework, this criterion is allowed
to depend on information returned by the oracles OF (k) at all the iterates{xi }0≤i≤N , but also at an extra point x∗ ∈ E assumed to be an optimal solution to problem (CM). The latter addition is necessary to allow criteria suchthe usual objective function accuracy at the last iterate F (xN ) − F∗ (whereF∗ = F (x∗ )). We also allow the performance criterion P to depend on thoseiterates themselves, which allows for example the distance to an optimal solu2
tion kxN − x∗ kE . For notational convenience we introduce an index set for all
iterates (including optimal solution) I = {0, 1, . . . , N, ∗}.
The worst-case performance of method M on (CM) is then the optimal valueof the following optimization problem, with both functions F (k) k∈K and
iterates {xi }i∈I as variables, which we call a performance estimation problem(PEP).
sup
P({OF (k) (xi )}i∈I,k∈K , {xi }i∈I ){F (k) }k∈K ,{xi }i∈I

(PEP)such that F (k) ∈ Fk (E) for all k ∈ K,
x0 satisfies some initialization condition,
xi+1 is computed by M according to (5.1) for all 0 ≤ i ≤ N − 1,x∗ is optimal for F (x).
That is, a solution to (PEP) corresponds to an instance of problem (CM) on
which method M behaves as badly as possible with respect to the performance
criterion P. The initialization condition on x0 is required as most methods
exhibit unbounded worst-case performance without it. In the sequel we will
mostly restrict ourselves to the classical approach which consists in bounding
2 That is, we chose to associate a call to each oracle to every iterate. This is mostly fornotational convenience and does not induce any loss of generality. Indeed, a method canalways choose not to use the information returned by one of the oracles at some iterations.

113

5.1. INTRODUCTION

the initial distance to an optimal solution with a constant R, i.e., assume
kx0 − x∗ kE ≤ R.
Note that (PEP) is inherently an infinite-dimensional optimization problem,
as functions F (k) appear as variables. However, a crucial observation is that,
due to the black-box assumption on the objective components, this problem
can be cast completely equivalently in a finite-dimensional fashion. Indeed,(k)introducing the outputs of the oracle calls as variables, namely Oi = OF (k) (xi )for all iterates i ∈ I and oracles k ∈ K, we observe that steps of method M(k)can be still be computed using only information contained in variables Oi , sothat we can reformulate (PEP) as
n

o
(k)
sup
P
Oi
, {xi }i∈I ,(PEP2)no(k)Oii∈I,k∈Ki∈I,k∈K,{xi }i∈I(k)such that ∃F (k) ∈ Fk (E) satisfying OF (k) (xi ) = Oi

for all i ∈ I, k ∈ K,

x0 satisfies some initialization condition.
xi+1 is computed by M according to (5.1) for all 0 ≤ i ≤ N − 1,x∗ is optimal for F.
Note the central role played by the interpolation conditions, which enforce theexistence of functions F (k) compatible with the output of the oracles. In thenext subsection we describe situations for which this formulation is tractable.

5.1.2

First-order methods and convex interpolation

In the remainder of this chapter, we restrict ourselves to first-order oracles andmethods. We now investigate the concept of (first-order) convex interpolability,
in order to make existence constraints from (PEP2) tractable — more precise
requirements are detailed in Section 5.2. From the assumptions, the existence
constraint for function F (k)(k)∃F (k) ∈ Fk (E) satisfying OF (k) (xi ) = Oi

for all i ∈ I,

found in (PEP2) may be expressed in terms of first-order information only.
Considering now oracles returning first-order information only OF (k) (x) =˜ (k) (x)}, we denote their output at point xi by OF (k) (xi ) = O(k) ={F (k) (x), ∇Fi(k) (k){fi , gi }. The above existence constraint can be rephrased into the following
set of interpolation conditions(k)∃F (k) ∈ Fk (E) satisfying F (k) (xi ) = fi(k)˜ (k) (xi ) = g
and ∇Fi(INT)

which is exactly the concept of convex interpolation from Definition 3.1.

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION114

The notion of F-interpolation can be considered for any class of convex functions (see Chapter 3 for more details). It allows us to formulate our performance
estimation problem in its final formno(k) (k)
P
f
,
g
,
{x
}
sup
i
i
i
i∈I ,
o
n
(k)

fi

(k)

,gi

i∈I,k∈K,{xi }i∈I

i∈I,k∈K(f-PEP)no(k)(k)
such that (xi , gi , fi )i∈I

is Fk -interpolable for all k ∈ K,

x0 satisfies some initialization condition.
xi+1 is computed by M according to (5.1) for all 0 ≤ i ≤ N − 1,x∗ is optimal for F,
We conclude that identifying explicit conditions for convex interpolability bya given class of functions will be the key to eliminate the infinite-dimensional
functional variables from (PEP) and transform it into a tractable estimation
problem (exactly as in Chapter 4).

5.2

Performance estimation framework for firstorder methods

We start this section by formulating (f-PEP) in terms of a Gram matrix. This
allows obtaining a tractable convex formulation for (f-PEP) — by making appropriate assumptions on the classes of objective function components, methods, performance criteria and initialization conditions. Those assumptions are
motivated by practical applications, which we also provide in the following
lines. Note that the main point underlying those assumptions is to ensure thatevery element of the performance estimation problem can be formulated in a
linear way in terms of the entries of a Gram matrix and the function values at
the iterates.

5.2.1

Gram representation of iterates and objective function

Let us consider N + 1 iterates x0 , . . . , xN and an optimal solution x∗ , and the(k) (k)set of corresponding oracle outputs {(fi , gi )}i∈I,k∈K .
The accumulated information after those N + 1 calls can be gathered into and × (n + 1)(N + 2) matrix3 PN (using a slight abuse of notations) and a vector
3 We remind the reader that B : E → E∗ is a positive definite operator which is chosen as
the identity operator in standard situations (see Section 2.1).

115

5.2. PEP FRAMEWORK FOR FIRST-ORDER METHODSFN of length n(N + 2):(1)

PN = [Bx0 . . . BxN | Bx∗ | g0
(1)

FN = [ f 0(n). . . f0

(1)

| . . . | fN(n)

. . . g0(n)

. . . fN

(1)

| . . . | gN(n)

. . . gN | g∗(1) . . . g∗(n) ],(5.2)| f∗(1) . . . f∗(n) ].

(5.3)

We also denote by B −1 PN the matrix
B −1 PN = [x0 . . . xN | x∗ | B −1 g0(1). . . B −1 g∗(n) ].

In order to formulate (PEP) in a tractable way for first-order methods, we use
a Gram matrix. That is, we define the symmetric (n+1)(N +2)×(n+1)(N +2)
Gram matrix GN ∈ S(n+1)(N +2) , using the following construction :

hx0 , x0 iE

.

.

.

hx , x i
 N 0 E
GN =  hx∗ , x0 iE
 (0) hg , x0 i
 0

.

.

.
(n)
hg∗ , x0 i

...
..
.
...
...
...
..
.
...

hx0 , xN iE.
.
.
hxN , xN iE
hx∗ , xN iE(0)hg0 , xN i
.
.
.
(n)
hg∗ , xN i

hx0 , x∗ iE.
.
.
hxN , x∗ iE
hx∗ , x∗ iE(0)hg0 , x∗ i
.
.
.
(n)
hg∗ , x∗ i(0)

hg0 , x0 i
.
.
.
(0)
hg0 , xN i(0)hg0 , x∗ i(0)(0)
hg0 , g0 iE∗
.
.
.
(0)
(n)
hg∗ , g0 iE∗

...
...
...
..
.


hg∗(n) , x0 i

.

.

.

(n)
hg∗ , xN i hg∗(n) , x∗ i   0.

(0)
(n)
hg0 , g∗ iE∗ 


.

.

.

...

hg∗(n) , g∗(n) iE∗

...
..
.

This can be written more compactly as[GN ]ij = hPN ei , B −1 PN ej i = hPN ei , PN ej iE∗ ,where PN ek corresponds to the kth column of PN . Also, note that the size of
this matrix does not depend on the dimension d of the spaces we are working in.
Remark 5.1. Note that Gram matrix GN is positive semidefinite for any
matrix PN (of the form (5.2)). The number of linearly independent columnsof PN is equal to the rank of GN . Hence this rank is upper bounded by thedimension d of the ambient space of the iterates. On the other hand, it ispossible to recover a matrix PN of the form4 (5.2) from any Gram matrix
GN  0 satisfying Rank GN ≤ d.
Our goal for the next subsections is to show that in a lot of situations, the performance estimation problem (f-PEP) can be expressed exactly as a semidefinite
program in the FN and GN variables:supGN 0,FN

c>FN + Tr CGN s.t. ai + b>
i FN + Tr Di GN ≤ 0∀i ∈ S(SDP-PEP)

with S some index set related to the constraints, and elements ai , bi , c, Di and
4 In the case E = E∗ = Rd with the usual inner product hx, yi = x> y and B the identity
operator, this can be done using the standard Cholesky factorization. In the general cases
the exact same idea can be used, using the chosen inner product h., .iE∗ in the process.

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

116

C of appropriate dimensions for writing the constraints and objective function linearly in terms of the Gram matrix GN and of the objective function
values FN .

5.2.2

Tractable formulation of the performance estimation problem

In this section, we present our main result, stating that computing the exact
worst-case performance of a method on a class of functions is tractable andcan, in many cases, be formulated as (SDP-PEP). We start with the concept of Gram-representability for the different ingredients of the performance
estimation problem.
Definition 5.2. A class of functions is Gram-representable (resp. linearly
Gram-representable) if and only if its interpolation conditions (INT) can beformulated using a finite number of convex (resp. linear) constraints involving
only the matrix GN and the corresponding function values FN .
The functional classes of smooth strongly convex functions, smooth convex
functions with bounded (sub)gradients, and strongly convex functions withbounded domain are linearly Gram-representable. In addition, the particularsubclasses of support and indicator convex functions share this same advantageous property. The details and proofs of these results are postponed to
Chapter 3).
Definition 5.3. A performance measure is Gram-representable (resp. linearly
Gram-representable) if and only if it can be expressed as a concave (resp.
linear) function involving only the matrix GN and the corresponding function
values FN .
The class of linearly Gram-representable performance criteria contains a large
variety of choices, including most standard measures we are aware of. For
example, it is easy to check that standard optimality criteria in function val2
˜ (xN ) , distance toues F (xN ) − F (x∗ ), in residual subgradient norm ∇F
2

E∗

2

optimality kxN − x∗ kE , and distance to feasibility kxN − ΠQ (xN )kE can behandled.
On the other hand, multiple examples of non-linear Gram-representable performance criteria can also be handled with no difficulty. This includes performance measures involving the best values among all iterates, for example
min0≤i≤N F (xi )−F (x∗ ), or the best residual gradient norm among the iterates
2
min0≤i≤N k∇F (xi )kE∗ (see also [THG16a, Sect. 4.3] or Section 4.3.3).
Definition 5.4. An initialization condition is Gram-representable (resp. linearly Gram-representable) if and only if it can be expressed using a finite num-117

5.2. PEP FRAMEWORK FOR FIRST-ORDER METHODS

ber of convex (resp. linear) constraints involving only the matrix GN and thecorresponding function values FN .
Standard examples of valid initial conditions include the classical bounds on2
the initial distance to optimality kx0 − x∗ kE ≤ R2 , on the initial function value
2
F0 − F∗ ≤ R, and on initial gradient value k∇F (x0 )kE∗ ≤ R2 .
Definition 5.5. A first-order method is Gram-representable (resp. linearly
Gram-representable) if and only if the computation of its iterates, implicitly
defined by an equation of type (5.1), can be expressed using a finite number of convex (resp. linear) constraints involving only the matrix GN and thecorresponding function values FN .
We refer to the next section for examples of linearly Gram-representable methods.
We can now state our main results concerning Gram-representable situations.
In the sequel, we recall that we use the notation FK (E) to denote the set offunctions of the form (CM) with components F (k) ∈ Fk (E) ∀k ∈ K — i.e.,
F ∈ FK (E).
Proposition 5.6. Consider a class of composite objective functions FK (E)with n components, a first-order method M, a performance measure P and an
initial condition I which are all Gram-representable.
Computing the worst-case for criterion P of method M after N iterations on
objective functions in class FK (E) with initial condition I can be formulated asa convex program when the dimension d of the space E satisfies d ≥ (n+1)(N +2). Otherwise, it can be formulated as a convex program plus an additional
non-convex rank constraint Rank GN ≤ d.
If in addition FK (E), M, P and I are linearly Gram-representable, then thecorresponding optimization problem is a SDP of the form (SDP-PEP), whose
variables are FN ∈ Rn(N +2) and GN ∈ S(n+1)(N +2) .
Proof. The result directly follows from Remark 5.1 and from the definitions of(linear) Gram-representability for the class of functions, first-order methods,
performance measures and initialization conditions: any solution to the corresponding optimization problem can be transformed into a particular instanceof (CM) and vice versa.
Remark 5.7. The optimal value of (PEP) increases with dimension d. When
(PEP) with Gram-representable elements attains a finite optimal value, Proposition 5.6 implies the existence of a function with dimension at most (n+1)(N +2) that achieves the worst-case value.
Remark 5.8. The assumption d ≥ (n + 1)(N + 2) is referred to as the largescale assumption in the sequel. In terms of performance estimation problems, this assumption allows us to discard the non-convex rank constraint andCHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

118

lead to a tractable semidefinite programming problem, which can be solved
to global optimality efficiently (see e.g., [VB94]). Without that assumption,
our performance estimation problem is a nonconvex rank-constrained semidefinite program, equivalent to a quadratic programming problem that is NP-hardin general (e.g., it has MAX-CUT [GW95] and other non-convex quadratic
programs [PV91, Sah74] as particular cases). Approaches to handle rank constraints exist (e.g., via augmented Lagrangian techniques [BM03], via manifold
optimization [JBAS10] or via Newton-like methods [OHM06]), but in general
only guarantee convergence to stationary points. This is not useful in the caseof (SDP-PEP), as this only provides lower bounds on the worst-case performance.
Remark 5.9. The worst-case results provided by the SDP from Proposition 5.6 provide a tight worst-case achievable for any operator B and any dual
pairing h., .i.
Remark 5.10. The necessary and sufficient condition for x∗ to be optimal for˜ (x∗ ) =F is linearly Gram-representable. Indeed, it corresponds to requiring ∇F0, i.e.
2

X

˜
∇F

k∈K

(k)

(x∗ ) =

X(k)g∗ = 0 ⇔

k∈K

(k)
g∗

X
k∈K

=h
E∗

X

(k)

g∗ ,

k∈K

X

(k)

g∗ i

k∈K

= 0,
E∗

where the last condition is linear in the entries of GN .

5.2.3

Linearly Gram-representable first-order methods

This class of first-order methods contains as particular cases what we call in thefollowing the class of fixed-step linear first-order methods (FSLFOM), whose
iterations are defined by a linear equation (with known constant coefficients)
involving the iterates and the corresponding (sub)gradients.
Definition 5.11. A fixed-step linear first-order method (FSLFOM) is a method
which computes iterate xi+1 as the solution of 5
ti+1,i+1 Bxi+1 +

X

(k)
(k)
hi+1,i+1 gi+1 =

k∈K

i
X
j=0

ti+1,j Bxj +

i X
X

(k)

(k)

hi+1,j gj ,j=0 k∈K(FSLFOM)(k)where all coefficients hi+1,j , ti+1,j ∈ R are fixed beforehand.
Note the class of FSLFOM is exactly the class of methods whose iterations
5 Note that the iteration is written as an equality on E, but it is possible and totallyequivalent to write it on E∗ using the operator B −1 , as B is invertible by assumption.

119

5.2. PEP FRAMEWORK FOR FIRST-ORDER METHODS

can be written in the form (using first-order optimality conditions, and theconvexity of F (k) ):(X (k)ti+1,i+12
xi+1 = argmin
hi+1,i+1 F (k) (x) +
kxkE
2
x∈E
k∈K
* i
+
i X

X
X
(k)
−
ti+1,j Bxj +hi+1,j ∇F (k) (xj ), x;

j=1

j=0 k∈K

which in some sense represents the most general method allowed in our framework. Those iterations can also be written by linearly combining the columnsof the matrix PN containing all the harvested first-order information about the
problem:
0 = PN α k ,with αk ∈ R(n+1)(N +2) a vector containing appropriate coefficients. Therefore,
we note that any FSLFOM is linearly Gram-representable using the following
formulation:
2

0 = PN αk ⇔ 0 = kPN αk kE∗ = PN αk , B −1 PN αk ,(5.4)

which is clearly linear in terms of the Gram matrix GN . Note that this can also
be extended to cope with the more general class of linearly Gram-representable
first-order methods6 :(low)>

ck
(low)

(low)

(low)

FN + bk
(up)

(up)>

≤ α>
k Gαk ≤ ck

(up)

FN + bk

,

(5.5)

(up)

where ck , bk
and ck , bk
are some fixed parameters. Those can for
example be used in order to require a sufficient decrease condition, or an inexact
version of (FSLFOM):α>k Gαk ≤ εk ,(Inexact FSLFOM)

with εk ≥ 0 some accuracy parameter for the computation of (FSLFOM).

Examples of FSLFOM. Before going into the details of the performance
estimation problems for our class of linear fixed-step methods and over thedifferent classes of convex functions, let us give several examples of methods
fitting into the model provided by (FSLFOM) and (Inexact FSLFOM).
- Fixed-step subgradient and gradient algorithms: fixed-step subgradientmethods for minimizing a convex function F are naturally described as
6 This formulation is just provided as an illustration to show that more general methods
than (FSLFOM) can still be considered.

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

120

xi = xi−1 − αi B −1 gi−1 with αi some step size, and gi−1 ∈ ∂F (xi−1 ). The
method is clearly in the class of FSLFOM and its linear Gram matrix
representation can be obtained using formulation (5.4).
- Proximal methods and proximal gradient methods: fixed-step proximal
gradient methods for minimizing F (1) + F (2) is usually described as doing
an explicit (sub)gradient step on F (1) followed by a minimization step on
F (2) :˜ (1) (xi−1 )xi = pαi F (2) xi−1 − αi B −1 ∇F21˜ (1) (xi−1 ) − x
xi−1 − αi B −1 ∇F.
= argmin αi F (2) (x) +2
E
x∈E
Optimality conditions on this last term allow writing each iterations as
˜ (2) (xi ) = Bxi−1 − αi ∇F˜ (1) (xi−1 ),Bxi + αi ∇F˜ (2) (xi ) ∈ ∂F (2) (xi ). This method is clearly a FSLFOMwith some ∇F
and therefore fits in the framework. Also, note that projected gradientmethods are obtained using the same technique, but on the particularclass of convex indicator functions, whereas proximal point algorithms
correspond to the case where F (1) = 0.
- Conditional gradient methods do also fit into the model provided by
Equation (FSLFOM). Indeed, the iterations take the following form:nDEo˜ (1) (zi ) + F (2) (z) ,yi = argminz − zi , ∇Fz∈Ezi+1 = (1 − λi )zi + λi yi ,with λi ∈ [0, 1] chosen beforehand. Now, by imposing yi using firstorder necessary and sufficient optimality conditions on the intermediate
optimization problem, we obtain
˜ (1) (zi ) = −∇F
˜ (2) (yi ).
∇F
Note that for conditional gradient-type methods, F (2) is usually chosen
as the indicator function of some closed convex set Q. This algorithm
can also clearly be written as a FSLFOM by artificially denoting for
i = 0, 1, . . . the iterates x2i = zi and x2i+1 = yi .
- Inexact (sub)gradient methods for a convex function F (1) , with xi+1 =˜ (1) (xi ) + εi ) and kεi k ∗ ≤ i for some i ≥ 0 the tolerancexi − αi B −1 (∇FEon the (sub)gradient computation. This can be written in the inexact
FSLFOM format:
˜ (1) (xi )αi (xi+1 − xi ) + ∇F2E∗

≤ 2i .

121

5.2. PEP FRAMEWORK FOR FIRST-ORDER METHODS

Also, note that other noise models, as for example the one proposed byd’Aspremont [d’A08] can also easily be used in the framework. On theother hand, the inexact (δ, L)-oracles developed by Devolder et al. [DGN14]
do not seem to easily fit into the approach7 .
Note that a broad class of methods can be modelled using those operations, justby requiring the functions on which it is applied to belong to certain classes. As
an example, alternate projection-type algorithms are special cases of proximal
methods, applied on the class of convex indicator functions. Therefore, they
can be represented in the FSLFOM format.

5.2.4

Simplified performance estimation problems

Note that for standard algorithms such as the previous examples of FSLFOM,
the SDP resulting from Proposition 5.6 can typically be further simplified,leading to a reduction in its size.
Corollary 5.12. Consider a class of functions FK (E), a performance measure
P and an initialization condition I which are linearly Gram-representable, anda FSLFOM M whose iterations are all linearly independent8 .
(k)

(k)

(k)

In addition, assume there are p points (gi , fi ) such that neither gi nor(k)fi are used in the performance measure P, the initial condition I and themethod M. Then, the performance estimation problem can be written as aconvex SDP using variables FN ∈ Rn(N +2)−p and GN ∈ S(n+1)(N +2)−N −p−1 ,with the possible additional rank constraint rank GN ≤ d.
To see why this corollary holds, note that the variables in the simplified SDP
correspond to the function values and the Gram matrix from which the p unnecessary points were removed, and from which N other variables were substitutedusing the N iteration constraints (FSLFOM)9 .
Under the assumptions of Corollary 5.12, the large-scale assumption becomesd ≥ (n + 1)(N + 2) − N − p − 1. In the cases where only the output from asingle oracle is used at each iteration, we have that p = (n − 1)(N + 1), whichleads to d ≥ N + n + 2.
7 This is due to the fact no necessary and sufficient interpolation conditions for this noisemodel were found — that is, standard conditions are only necessary to guarantee interpolability. Using necessary conditions that are not sufficient still allows obtaining upper bounds
on the worst-case behavior, but those may not be tight.
8 That is, the vectors α used to characterize the iterations are linearly independent —kthis is very reasonable, as every method using new information at each iteration satisfies this.
Remark that it does not imply that the points xi themselves are linearly independent.
9 The additional −1 term appearing in the dimension of the Gram matrix comes from(k)
the fact that one of the g∗ may also be discarded, by substituting it using the optimality
condition of x∗ .

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

122

2Furthermore, for standard performance measures (e.g. FN − F∗ , kxN − x∗ kE ,2

˜ (xN ) ), one arbitrary point xi may be fixed to 0 because solutions to∇FE∗
the SDP are be invariant with respect to translations. This would result inthe large-scale assumption d ≥ N + n + 1. For n = 1, we recover the standardd ≥ N + 2 appearing in the case of a single component in the objective function(see Theorem 4.2 and Corollary 4.5).
The original SDP from Proposition 5.6 may be challenging to solve in practice,because of its potentially large size on the one hand, and because its feasible
region may lack an interior on the other hand. We observe that the simplified performance estimation problem described above typically improves thesituation for both issues, reducing the size of the problem and solving in a
lot of cases the issue of a lack of interior points. Finally, note that as in thecase of unconstrained optimization, structural properties of (SDP-PEP) could
potentially be used for further simplifying the SDP (see Remark 4.6).

5.3

Algorithm analysis

In this section, we analytically and numerically study different algorithms forsolving variants of (CM), and compare our results with standard guarantees
from the literature10 . This section is organized as follows:- We begin with an analytical study of a proximal point algorithm. For
this algorithm, we provide simple analytical and tight convergence results
twice better than the standard theoretical guarantees. We also illustrate
how to incorporate a simple noise model into the performance estimation
framework for this basic method.
- Secondly, we numerically study several simple variants of projected subgradient methods; this illustrates the applicability of (PEP) for very simple and widely studied methods.
- Third, we use performance estimation to compare several standard variants of fast proximal gradient methods. An tentative extension to the
optimized gradient method (OGM) proposed by Kim and Fessler [KF16d]
is proposed using the ideas developed for fast proximal gradient methods.
- Finally, we conclude by illustrating the results of the approach for a
conditional gradient and on two alternate projections schemes. Thosechoices illustrates the applicability of the approach for studying a large
variety of methods and performance measures.
10 Note that most of the literature results are presented for B being the identity operator(and hence E = E∗ ). We will nevertheless compare our slightly more general results withthe standard bounds from the literature (thus even when they are officially valid only for
B being the identity) — we recall that our results are valid for general self-adjoint positive
definite linear operator B : E → E∗ (see Remark 5.9).

123

5.3. ALGORITHM ANALYSIS

5.3.1

A proximal point algorithm

Consider a simple model with only one convex (possibly non-smooth) term in
the objective function,
min F (x),x∈Ewith F ∈ F0,∞ (E). In this first example, we assume that the proximal operation
is easy to compute for F :1
2
xk+1 = pαk+1 F (xk ) = argmin αk+1 F (x) + kxk − xkE .
2
x∈E
That is, the iterations can be written in the form of an implicit method xk+1 =xk − αk+1 B −1 gk+1 , for some gk+1 ∈ ∂F (xk+1 ). For recent overviews andmotivations concerning proximal algorithms, we refer the reader to the work of
Combettes and Pesquet11 [CP11], and to the review works of Bertsekas [Ber10]
and Parikh and Boyd [PB13]. For historical point of view on those methods,
we refer to the pioneer works of Moreau [Mor65], Rockafellar [Roc76] and theanalysis of Guler [Gül91].
Proximal Point Algorithm (PPA)
Input: F ∈ F0,∞ (E), x0 ∈ E. Parameters: {αk }k with αk ≥ 0.
For k = 1 : N
xk = pαk F (xk−1 )Convergence of PPA in function and gradient values
The standard convergence result for the proximal point algorithm is provided
by Guler in [Gül91, Theorem 2.1] :F (xN ) − F∗ ≤

2

R2
PN

k=1 αk,for any initial condition x0 satisfying kx0 − x∗ kE ≤ R. We improve this boundby a factor 2 using the PEP approach.
Theorem 5.13. Let {αk }k be a sequence of positive step sizes and x0 some
initial iterate satisfying kx0 − x∗ kE ≤ R for some optimal point x∗ . Any sequence {xk }k generated by the proximal point algorithm with step sizes {αk }k
11 This work features among others a large list of known proximal operators.

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

124

on a function F ∈ F0,∞ (E) satisfies
F (xN ) − F∗ ≤

4

R2
PN

k=1 αk.

In addition, this bound is tight, and it is attained on√ the l1 -shaped oneRkxk
dimensional function (dim E = dim E∗ = 1) F (x) = 2 PBR|x|= 2 PN EαNα
k=1with Bx20 = R2 .

k

k=1

k

Proof. The proof relies on finding a primal and dual form of (f-PEP) for theproximal point algorithm (details can be found in Appendix 5.A). A dual solution allows us to obtain the upper bound part.
Considering another convergence measure, the exact same idea allows us toobtain strong numerical evidence for the following conjecture.
Conjecture 5.14. Let {αk }k be a sequence of positive step sizes and x0 some
initial iterate satisfying kx0 − x∗ kE ≤ R for some optimal point x∗ . For any
sequence {xk }k generated by the proximal point algorithm with step sizes {αk }kon a function F ∈ F0,∞ (E), there exists a subgradient gN ∈ ∂F (xN ) such thatR
kgN kE∗ ≤ PN

k=1 αk.

−BxN
In particular, the choice gN = BxN −1
is a subgradient satisfying theαNinequality.

Observe that this bound cannot be improved,
as it is attained on the (one√BR|x|dimensional) l1 -shaped function F (x) = PN
with Bx20 = R2 . The particuk=1 αk
lar choice of subgradient suggested in the theorem corresponds to the subgradient appearing in the proximal operation when written as an implicit subgradientstep.
This sort of convergence results in terms of residual (sub)gradient norm isparticularly interesting when considering dual methods. In that case, the dual
residual gradient norm corresponds to the primal distance to feasibility (seee.g., [DGN12]).
PPA with a basic uncertainty model
Assume that one can approximately compute the proximal operation xk =(ε)
xk−1 − αk B −1 (gk − gk ) with the guarantee thatxk − xk−1(ε)+ B −1 gk = gk
≤ ,
αk
E∗
E

(5.6)

125

5.3. ALGORITHM ANALYSIS

with  ≥ 0 a given precision for iteration k and αk ≥ 0 the step size at iteration
k. Even though this model is very basic, it can for example be used when it ispossible to approximate gk+1 ≈ gk with a controlled error kgk − gk+1 kE∗ ≤ ,
in which case the explicit algorithm xk = xk−1 − αk B −1 gk−1 satisfies theassumptions. More practically, it can also be used when the proximal operation
can only approximately be solved — that is, the proximal operator can only
approximately be evaluated, with the guarantee that:˜ (xk ) ∈ ∂F (xk ) s.t.
∃∇Fxk − xk−1
˜ (xk ) ≤ .
+ B −1 ∇F
αkE

The following theorem illustrates the type of results that can be obtained using
the PEP approach for this uncertain PPA.
Theorem 5.15. Let {αk }k be a sequence of positive step sizes, x0 some initial
iterate satisfying kx0 − x∗ kE ≤ R for some optimal point x∗ and  > 0 be apositive constant. There exists a function F ∈ F0,∞ (E) such that the sequence{xi }i generated by the proximal point algorithm with step sizes {αk }k under
the noise model (5.6) satisfies
F (xN ) − F∗ ≥ R,for any x0 satisfying kx0 − x∗ kE ≤ R for some optimal point x∗ .
PN
In addition, the case k=1 αk = R leads to F (xN ) − F∗ ≤ R.
Proof. The proof is presented in Appendix 5.B, and also relies on finding alower bound and a matching dual feasible point.
PN
Note the choice of step sizes {αk }k satisfying k=1 αk = R is optimal for thatsetting, as its worst-case guarantee match the lower boundF (xN ) − F∗ ≥ R,that is valid for any choice of step sizes.
We leave further investigations in this direction as future work.

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

5.3.2

126

Projected subgradient methods

In this section, we consider the constrained non-smooth convex optimization
problemmin F (x),

(5.7)

x∈Q

with F ∈ CM,∞ (E) and Q ⊆ E a closed convex set. For solving this kind ofproblems, we assume here on the one hand that for any x ∈ Q, one can easily
compute at least one element g ∈ ∂F (x), and on the other hand that one caneasily compute projections onto the set Q.
In this setting, one can use the projected subgradient method in order to obtainapproximations to x∗ . That is, one can iterate xk+1 = ΠQ (xk − αk+1 B −1 gk ),with ΠQ (.) the projection operator onto Q, with gk ∈ ∂f (xk ) and with αk ≥ 0some step size parameters.
Projected Subgradient Method (PSM)
Input: F ∈ CM,∞ (E), x0 ∈ Q ⊆ E.
For k = 1 : N
xk = ΠQ xk−1 − αi B −1 gk−1In that setting, it is known that for every step size policy {αi }i , there exists afunction such that both (e.g., [DT16, Theorem A.1]12 ):
!
N
MR
MR
1 X
x i − F∗ ≥ √min F (xi ) − F∗ ≥ √, F, (5.8)0≤i≤N
N + 1 i=0
N +1
N +1
(we refer to the first criterion as the performance of the best iterate, and to thesecond criterion as the performance of the averaged iterate). In fact, the simple
constant step size policy αk = M √RN +1 allows obtaining the corresponding tightworst-case guarantees (see e.g., [Bub15, Nes04]):
!
N
MR
1 X
MR
min F (xi ) − F∗ ≤ √, F
x i − F∗ ≤ √. (5.9)0≤i≤N
N + 1 i=0
N +1
N +1
However, this constant step size policy is very impractical, as its use require theknowledge of the number of iterations in advance. Therefore, we use the performance estimation framework to refine the guarantees that can be obtained
R √1
R 1
with the more practical step size policies αk = M
and αk = M
k+1 .
k+1
12 Note that [DT16, Theorem A.1] does not directly treat those cases, but the function it
uses in the proof remains valid for them.

127

5.3. ALGORITHM ANALYSISAs a reference, we use the following standard guarantee (e.g., [BXM03, Nes04]):
PN
R2 + M 2 k=0 αk2
min F (xi ) − F∗ ≤
.
PN
0≤i≤N
2 k=0 αk(5.10)First, we compare the performances of the different step size policies for thebest iterates13 on Figure 5.1(a). Those results indicates that the step sizes
R 1
R √1
αk = M
k+1 should a priori be preferred over αk = M k+1 for low numberof iterations, which is exactly opposite to what is advised by the standard
guarantee (5.10).
Second, we compare the worst-case performance of the best iterate with theworst-case performance of the averaged iterate14 on Figure 5.1(b). Interestingly
we observe that better performances should be expected from using the best
iterate, and not the averaged one — we are note aware of a generic guarantee
like (5.10) for the averaged iterate.
00Upper bound on function value (log scale)

10

Upper bound on function value (log scale)

10

−1

10

−1

0

10

1

10
Iteration count (log scale)210(a) Comparison between the worst-caseguarantees of PEP with the theoreticalbound (5.10) for the best iterate. Step
R 1
sizes αk = M, theoretical guarank+1
tee (5.10) (dashed, blue), PEP guarantee
R √1(plain, blue); step sizes αk = M,
k+1
theoretical guarantee (5.10) (dashed, red),
PEP guarantee (plain, red) and lower
bound (5.8) (dashed, black).

10

0

1

10

10
Iteration count (log scale)210(b) Comparison between the worst-caseperformances of the last iterate and of theaveraged iterate (using performance estimaR 1tion). Step sizes αk = M, averaged
k+1
iterate (dashed, blue), best iterate (plain,R √1blue); step sizes αk = M, averaged itk+1
erate (dashed, red), best iterate (plain, red)and lower bound (5.8) (dashed, black).

Figure 5.1: Projected subgradient method: comparison between the resultsof PEP with the theoretical bound (5.10) for the best iterate, and comparison
between convergence in the best iterate and convergence in the averaged iterate.

13 That is, the worst-case performance for the criterion min
14 That is, the worst-case performance for the criterion F

F (xi ) 
− F∗ .
 0≤i≤N1 PN
x
i=0 i − F∗ .
N +1

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

5.3.3

128

Fast proximal gradient algorithms

In this section, we consider the two-terms composite objective functionnomin F (x) ≡ F (1) (x) + F (2) (x) ,(5.11)
x∈E

with F (1) ∈ F0,L (E) (smooth convex function) and F ∈ F0,∞ (E) (non-smoothconvex function). We assume that gradients are easy to compute for F (1) , andthat the proximal operation is easy to compute for F (2) :12pαF (2) (x) = argmin αF (y) + kx − ykE .
2y∈E
In order to approximatively solve (5.11), it is common to use different variantsof fast proximal gradient methods (FPGM). We numerically investigate theworst-case guarantees of two variants using different step sizes policies, andpropose new variants with slightly better worst-case behaviors. Also, we illustrate differences in the worst-case performances obtained in the cases whereF (2) = 0 (unconstrained smooth convex minimization), F (2) ∈ I∞ (E) (constrained smooth convex minimization) or F (2) ∈ F0,∞ (E) (regularized smoothconvex minimization).
In the following, we call FPGM1 the standard fast proximal gradient method(FISTA [BT09b]), FPGM2 a variant with slightly better guarantees, and POGM
a proximal version of the optimized gradient method [KF16d]. FPGM2 andPOGM illustrate how performance estimation problems can be used in the development of new optimization algorithms ; their study in this chapter remains
however entirely numerical.
Standard Fast Proximal Gradient Methods (FPMG1)
The first variants of accelerated proximal methods we are considering use astandard proximal step after an explicit gradient step for generating the socalled primary sequence {yk }k .
Fast Proximal Gradient Method (FPGM1)
Input: F (1) ∈ F0,L (E), F (2) ∈ F0,∞ (E) x0 ∈ E, y0 = x0 .
For k = 1 : N


1 −1
(1)
yk = pF (2) /L xk−1 − B ∇F (xk−1 )L
xk = yk + αk (yk − yk−1 )In this algorithm, we refer to αk as inertial parameters. We use two standard(a)variants: αk = k−1
k+2 — among others proposed in [SBC14, Tse08] — and129

5.3. ALGORITHM ANALYSIS√ 2+1
1+ 4θk−1(b)−1
αk = θk−1(withθ=and θ0 = 1) — see [BT09b, Nes83,kθk2
Tse08]. For both variants, the standard convergence result is (see e.g., [BT09b,
SBC14]):F (yN ) − F∗ ≤2LR2,
(N + 1)2(5.12)

2for any initial iterate x0 such that kx0 − x∗ kE ≤ R2 . We numerically compare those two variants of FPGM1 using (f-PEP) on Figure 5.3.3. After 100
iterations, both inertial parameter policies perform about the same way (pa(b)(a)
rameters αk performs only about 2% better than αk in terms of worst-caseperformances). We also observe that the behavior of both variants of FPGM1
is well captured by the standard guarantee (5.12).
0

0

10

Upper bound on fN−f* (log scale)

Upper bound on fN−f* (log scale)

10

−1

10

−2

10

−3

10

−4

10

−1

10

−2

10

−3

10

−4

0

10

1

10
Iteration count (log scale)

2

10

10

0

10

1

10
Iteration count (log scale)2

10Figure 5.2: Comparison of the worst-case convergence speed of the different
variants of FPGM1 (left) and FPGM2 (right) for N ∈ {1, . . . , 100}, L = 1 andR = 1. The curves respectively corresponds to the different inertial coefficient,(a)(b)
namely αk (dashed, black) and αk (red), and the standard guarantee (5.12)(blue).

New Fast Proximal Gradient Methods (FPGM2)
Secondary sequences {xk } are usually converging slightly faster than primary
sequences {yk } in the unconstrained case (F (2) = 0), as observed in [KF16d,
THG16a]. However, some issues may arise with the secondary sequences of
FPGM1 when applied to constrained or proximal problems: iterates may insome cases become infeasible, or the objective may become unbounded (seeTable 5.1 below). We therefore propose new variants of fast proximal gradientmethods that do not suffer from theses drawback, called FPGM2 (with two
different step size policies). Part of the underlying motivation behind FPGM2
is also the ability to generalize it later to the optimized gradient method.

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

130

Remark 5.16. The design of FPGM2 is based on two ideas: on the one hand,
it should be equivalent to the standard fast gradient method in the case ofsmooth unconstrained convex minimization, and on the other hand, it should
not move after two consecutive iterates have reached the same optimal pointfor (5.11) (i.e., xk−1 = xk = x∗ implies xk+1 = x∗ ).
Fast Proximal Gradient Method 2 (FPGM2)
Input: F (1) ∈ F0,L (E), F (2) ∈ F0,∞ (E) x0 ∈ E, z0 = y0 = x0 .
For k = 1 : N
1 −1
B ∇F (1) (xk−1 )Lαkzk = yk + αk (yk − yk−1 ) +(zk−1 − xk−1 )Lγk−1

yk = xk−1 −xk = pγk F (2) (zk )In this algorithm, we use the coefficients γk = αkL+1 . Note that we introduced
two intermediate sequences: on the one hand sequence {γk }k , corresponding
to the step sizes to be taken by the proximal steps, and on the other handsequence {zk }k , which allows keeping track of the subgradient used in theproximal steps (note that γ1k (zk −xk ) corresponds to the subgradient used in theproximal step from zk to xk ). Even if FPGM2 may look more intricate than theclassical FPGM1, it is in fact simpler, as it involves only one sequence on which
both implicit (proximal) and explicit (gradient) steps are being taken. Indeed,
explicit steps are taken using gradient values of F (1) at xk , and subgradientsused in the proximal steps are subgradients of F (2) also at xk . This can beseen by rewriting the iterations of FPGM2 using the secondary sequence {xk }konly
xk+1 =xk + αk+1 (xk − xk−1 )αk+1 −11
αk+1 −1+
B ∇F (1) (xk−1 ) − B −1 ∇F (1) (xk ) −B ∇F (1) (xk )L
LL
αk+1 −1 ˜ (2)1
˜ (2) (xk+1 ) − αk+1 B −1 ∇F˜ (2) (xk+1 ),+B ∇F (xk ) − B −1 ∇FLLL˜ (2) (xk ) the subgradient of F (2) used in the proximal operation generwith ∇F
ating xk .
Comparing the different variants of FPGM2 on Figure 5.3.3 leads to the sameconclusion as for FPGM1: inertial parameters α(b) perform slightly better thanα(a) .
In Table 5.1, we report the different worst-case performances guarantees obtained numerically for FPGM1 (for both sequences) and FPGM2 (for the better secondary sequence only). We consider three situations: F (2) = 0 (unconstrained smooth convex minimization), F (2) ∈ I∞ (E) (constrained smooth

131

5.3. ALGORITHM ANALYSIS

convex minimization with projected methods) and F (2) ∈ F0,∞ (E) (regularizedsmooth convex minimization with proximal methods).

Type

F (yN ) − F∗
(FPGM1)F (xN ) − F∗
(FPGM1)F (xN ) − F∗
(FPGM2)

Unconstrained(F (2) = 0)LR2
4
2 N 2 +5N +6LR2
4
2 N 2 +7N +4LR2
4
2 N 2 +7N +4

Constrained(F (2) ∈ I∞ )4LR2
2 N 2 +5N +2

Infeasible

LR2
4
2 N 2 +7N

Non-smooth(F (2) ∈ F0,∞ )LR2
4
2 N 2 +5N +2

UnboundedLR2
4
2 N 2 +7N

Table 5.1: Worst-case obtained for FPGM1 and FPGM2 with inertial coefficient αk = k−1
k+2 and N ≥ 1.
Convergence results reported in the table correspond to properly identified
functions (i.e. they are rigorous lower bounds). After solving the corresponding
PEPs numerically (for L = R = 1 and 1 ≤ N ≤ 100), we conjecture them tobe equal to the exact worst-case guarantees.
We observe that the worst-case guarantees for FPGM2 are slightly better thanfor FPGM1.Guarantees for the unconstrained case are slightly better than thosefor the constrained and proximal cases, which are equal. Note that the secondary sequence of FPGM1 is not guaranteed to be feasible in the constrained
case, and that the corresponding objective value may be unbounded in theproximal case (for any N ≥ 1).
The worst-case functions identified numerically for the unconstrained case are
Huber-shaped functions [THG16a]. In the constrained case, we identified onedimensional linear optimization problems of the form minx≥0 cx as worst-cases,
where c is a constant defined by√BR
c = PN −1.
2 i=0 [αN ]i
Finally, for the proximal case, our worst-case has function F (1) (x) = cx withthe same c as above, and function F (2) (x) may be chosen equal to zero forx ≥ 0 and to sx for x < 0, for any negative value of the slope s < 0.

5.3.4

A proximal optimized gradient method

In this section, we consider again the regularized smooth convex minimization
problem (5.11). In particular, we are concerned with the possibility of obtaining

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

132

optimized methods for handling this sort of problems (i.e., methods whose
worst-case performances are minimized).
The idea is to extend the optimized gradient method (OGM) developed by
Kim and Fessler in [KF16d], which is originally tailored for smooth unconstrained minimization (F (2) = 0). In the unconstrained smooth minimization
setting, this first-order method was recently shown in [Dro16] to have the best
achievable worst-case guarantee for the criterion FN − F∗ .
The method we propose has been obtained by combining the ideas obtained
from the original OGM [KF16d] and the non-standard placement of the proximal operator used for speeding up the convergence of fast proximal gradientmethods (FPGM2). It was designed using the same two principles as FPGM2(see Remark 5.16): on the one hand, it is equivalent to OGM when applied tosmooth unconstrained convex minimization problems, and on the other hand,
it remains at an optimal point when it reaches one.
Proximal Optimized Gradient Method (POGM)
Input: F (1) ∈ F0,L (E), F (2) ∈ F0,∞ (E), x0 ∈ E, y0 = x0 , θ0 = 1.
For k = 1 : N
1 −1
B ∇F (1) (xk−1 )Lθk−1 − 1
θk−1
θk−1 − 1zk = yk +(yk − yk−1 ) +(yk − xk−1 ) +(zk−1 − xk−1 )θkθkLγk−1 θkyk = xk−1 −xk = pγk F (2) (zk )k −1
and the inertial
In this algorithm, we use the sequence γk = L1 2θk−1θ+θkcoefficients proposed in [KF16d]:√ 2+1
 1+ 4θk−1, i≤N −1
2
√
θk =
2
+1
 1+ 8θk−1
, i=N

2

Simply trying to generalize OGM using the standard proximal step on the primary sequence {yi } (as for FPGM1) does not lead to a converging algorithm.
We have numerical evidence, i.e. worst-case functions showing that this candidate algorithm does not see its worst-case improving after each iteration: inother words its worst-case rate is not converging to zero). Therefore we have to
introduce the same idea used in FPGM2 concerning the place of the proximal
operator.
(b)

We compare POGM to FPGM1 and FPGM2 with inertia αk on Figure 5.3. We
obtain worst-case performances about twice better for POGM when compared
to both FPGM1 and FPGM2. Of course, POGM suffers from the drawback ofrequiring the knowledge of the number of iterations in advance (this is because

133

5.3. ALGORITHM ANALYSIS

the rule to compute the last coefficient θN differs from the rule to computeall the previous ones). This practical disadvantage is not easily solved: if thelast θN is updated with the same rule as all the previous θk , performance isdegraded by a non-negligible factor, rendering it even slower than FPGM (notethat this is already the case for smooth unconstrained minimization [KF16c]).

0Upper bound on fN−f* (log scale)

10

−1

10

−2

10

−3

10

−4

10

0

10

1

10
Iteration count (log scale)2

10Figure 5.3: Comparison between the worst-case performances of FPGM1 (with(b)(b)
inertia αk ) (red), FPGM2 (with inertia αk ) (blue), POGM (black) and OGM(dashed, black) for N ∈ {1, . . . , 100}, L = 1 and R = 1. The worst-caseperformances of POGM are about twice better than the worst-case performanceof FPGM between 1 and 100 iterations. Also, we observe that OGM [KF16d](equivalent to POGM with F (2) = 0) behaves approximately 12% better than
POGM in the worst-case.

5.3.5

A conditional gradient method

Consider the constrained smooth convex optimization problem
min F (x),x∈Qwith F ∈ F0,L (E) and Q ⊂ E a bounded and closed convex set. In that setting,there exists different ways for treating the constraint set Q. In the previous
section, we proposed to use fast gradient methods, which require the abilityof projecting onto the closed convex set Q. In this section, we rather consider
the standard conditional gradient method (also sometimes referred to as theFrank-Wolfe method), which originates from [FW56]. This algorithm has theadvantage of not requiring to perform projections onto Q, but rather to perform

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

134

linear optimization on this set (which is typically easier when Q is a polyhedral
set).
Conditional Gradient Method (CGM)
Input: F ∈ F0,L (E), closed convex Q ⊂ E with kx − ykE ≤ D ∀x, y ∈Q, x0 ∈ Q.
For k = 1 : N
yk = argmin {h∇F (xk−1 ), y − xk−1 i}
y∈Q

λk =

2
1+k

xk = (1 − λk )xk−1 + λk yk

The standard global convergence guarantee for this method (see e.g., [Jag13,
Theorem 1]) is
2LD2
F (xN ) − F∗ ≤,(5.13)
N +2
which we compare with the exact bound provided by PEP on Figure 5.4(a).
As illustrated in Section 5.2.3, this algorithm fits into the (FSLFOM) format.
The numerical guarantees we obtained by solving the performance estimation
problem are between two and three times better than the standard guarantee,
depending on the number of iterations.
00Upper bound on distance to feasibility (log scale)

Upper bound on function value (log scale)

10

−1

10

−2

10

−3

10

0

10

1

10
Iteration count (log scale)210(a) Worst-case performance of CGM (red)and its theoretical guarante (5.13) (blue)
for N ∈ {1, . . . , 100}, L = 1 and D = 1.

10

−1

10

−2

10

0

10

1

10
Iteration count (log scale)210(b) Worst-case performance of APM (red),R
DAPM (blue) and lower bound √M
for
N +1
subgradient methods (dashed, black) forN ∈ {1, . . . , 100} and R = 1 (M = 1 bydefinition of the objective function (5.14)).

Figure 5.4: Numerical analysis of a conditional gradient method (left) and of
two variants of alternate projections algorithms (right).

135

5.3.6

5.3. ALGORITHM ANALYSIS

Alternate projection and Dykstra methods

In this section, we numerically investigate the difference between the worstcase behaviors of the standard alternate projection method (APM) for finding
a point in the intersection of two convex sets, and the Dykstra [BD86] method(DAPM) for finding the closest point in the intersection of two convex sets.
APM is known to converge sublinearly in general, as it is a particular instanceof subgradient-type descent15 applied to the problem
min{f (x) = max kx − ΠQi (x)kE },(5.14)

ix∈E

whose objective function is convex and non-smooth (with Lipschitz constantM = 1). Therefore, its expected global convergence rate is O( √1N ) (see [DT16,
Theorem A.1]). We compare below the convergence of both APM and DAPM
R
with the standard lower bound for subgradient schemes √M
as a reference.
N +1
Alternate Projection Method (APM)
Input:x0 ∈ E, convex sets Q1 , Q2
kx0 − x∗ kE ≤ R, for some x∗ ∈ Q1 ∩ Q2 .

⊆E,

For k = 1 : N
xk = ΠQ2 (ΠQ1 (xk−1 ))Dykstra Alternate Projection Method (DAPM)
Input:x0 ∈ E, convex sets Q1 , Q2 ⊆ E,
kx0 − x∗ kE ≤ R, for some x∗ ∈ Q1 ∩ Q2 . Initialize p0 = q0 = 0.
For k = 0 : N − 1
yk = ΠQ1 (xk + pk )pk+1 = xk + pk − ykxi+1 = ΠQ2 (yk + qk )qk+1 = yk + qk − xk+1

The optimality measure used is minx∈Q1 kx − xN kE = kΠQ1 (xN ) − xN kE (notethat xN ∈ Q2 ). We do not give further details the corresponding performance
estimation problem here, as it is very similar to the previous sections. The
15 It can be shown thatx−ΠQ (x)k
is a subgradient of the function f (x) (at x such that||x−ΠQ (x)||kf (x) = ||x − ΠQk (x)||). Therefore, in the case of two sets Q1 , Q2 , and assuming that x
is feasible for one of the two sets (say, Q1 ), a projection on the other one corresponds to a
subgradient step on f with step size ||x−ΠQ2 (x)||. Hence, APM is an instance of subgradientmethod for k > 1 (when xk is feasible for one of the two sets).

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

136

results for APM and DAPM are shown on Figure 5.4(b), where the (expected)
convergence in O( √1N ) is clearly obtained. Interestingly, DAPM converges
slightly slower than APM (more precisely, DAPM has a worst-case about 18%
higher than APM), which is therefore more advisable in terms of worst-caseperformances for finding a point in the intersection of two convex sets when no
additional structure is assumed. In addition, note that both APM and DAPM
have a worst-case which is about twice better than the standard lower boundfor explicit non-smooth schemes.

5.4

Conclusion

In this chapter, we presented a performance estimation approach for analyzing first-order algorithm for composite optimization problems. The resultsof Chapter 4 (or [THG16a]) were largely extended to handle both larger classesof objective functions (components) and larger classes of first-order algorithms
to possibly be analyzed, all that in a more general setting for handling pairs ofconjugate norms. The contribution is essentially twofold: first, we exploit thestructures of interpolation conditions from Chapter 3 to formulate the exact
worst-case problem for fixed-step linear first-order methods with appropriate
convergence measure and initial conditions. Secondly, we apply the methodology to provide tight analyses for different first-order methods.
Further extensions to the performance estimation framework are proposed in
Chapter 8, including for coping with randomness and monotone operators.
Software. MATLAB implementations of the performance estimation approach fordifferent variants of gradient methods are available online. It can be downloaded fromhttp://perso.uclouvain.be/adrien.taylor.

Appendix
5.A

Proof of Theorem 5.13

We start by proving the lower bound, and then we prove the matching upperbound.
Lower bound.
Let us show that applying PPA to the one-dimensional func√
BR|x|tion F (x) = 2 PN α with x0 = − √RB allows us to achieve:
k=1k

F (xN ) − F (x∗ ) =

4

R2
PN

k=1 αk,which shows that the bound from Theorem 5.13 is tight.
√

First, note that for x 6= 0, we have ∇F (x) = sign(x) 2 PNBRα . Hence,
k=1

xN = x0 + B

−1

k

√

N
X

R
BR
=− √ .
αk PN
2 B
2 k=1 αk
k=1

Therefore, by noting that x∗ = 0 and F (x∗ ) = 0, we have the desired result.
Upper bound. In order to express the corresponding PEP in the simplest
form, we heavily rely on some straightforward simplifications of (SDP-PEP)(see Corollary 5.12 and Remark 5.2.4). Let us denote by PN the matrix containing the information harvested after N iterations (we use the notation gi
for subgradients gi ∈ ∂F (xi )): PN = [g1 g2 . . . gN Bx0 ], and by GN its corresponding Gram matrix (see Section 5.2.1). Also, we introduce the step size
matrices αk for expressing all intermediate iterates xi ’s in terms of x0 and thesubgradient gi ’s, that is: xk = PN αk (k = 0, . . . , N ).
Pk
This results in the following explicit expressions for αk : αk = eN +1 − i=1 αi ei ,along with α0 = eN +1 and α∗ = 0 (i.e., we assume without loss of generality137

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION138

x∗ = 0), where we use the standard notation ei for the unit vector having asingle 1 as its ith component — we also denote e∗ = 0. In order to performthe wort-case analysis for PPA, we now formulate the performance estimation
problem (f-PEP) as the following SDP (simplified version of (SDP-PEP) where
the xk ’s (k = 1, . . . , N ) have been substituted using the form of the algorithm
xk = xk−1 − αk B −1 gk ):max

GN ∈SN +1 ,f1 ,...,fN ,f∗ ∈RN

fN − f∗ ,(PPA-PEP)

s.t. fj − fi + Tr(Aij GN ) ≤ 0,i, j ∈ {1, . . . , N, ∗}2
kx0 − x∗ kE ≤ R2 ,GN  0,with 2Aij = ej (αi − αj )> + (αi − αj )e>
j , the matrices coming from the nonsmooth convex interpolation inequalities (see Condition (3.1)).
In order to obtain an analytical upper bound for PPA, we consider the Lagrangian dual to (PPA-PEP), which is given by the following.
min

λij ≥0,τ ≥0

τ R2(PPA-dPEP)

s.t. eN −

XX
i

XX
i

(λij − λji )ej = 0,j6=iλij Aij + τ α0 α>
0  0,

j6=i(where the constraint corresponding to f∗ can be discarded since it is clear thatletting f∗ = 0 does not change the optimal solution of (PPA-PEP)). Note thatthe set of equality constraints can be assimilated to a set of flow constraints on acomplete directed graph. That is, considering a graph where the optimum andeach iterate correspond to nodes, each 0 ≤ λij ≤ 1 corresponds to the flow on
the edge going from node j to node i (we choose this direction by convention).
This flow constraint imposes that the outgoing flow equals the ingoing flow forevery node, except at iterate N where the outgoing flow should be equal to 1
and at the optimum, where the incoming flow should be equal to 1. We showthat the following choice is a feasible point of the dual (PPA-dPEP).
Pi
λi,i+1 =k=1 αk2PN

k=1 αk −Pi

k=1 αk

,

PN
α
k=1
 Pk
λ∗,i =  P
PiPi−1  ,NN
2 k=1 αk − k=1 αk 2 k=1 αk − k=1 αk
2αi

τ=

4

1
PN

k=1 αk,i ∈ {1, . . . , N − 1}i ∈ {1, . . . , N }139

5.A. PROOF OF THEOREM 5.13

and λij = 0 otherwise. First, we clearly have λij ≥ 0 and some basic computations allow to verify that the equality constraints from (PPA-dPEP) aresatisfied:λ∗,1 − λ1,2 = 0,..
.
λ∗,i + λi−1,i − λi,i+1 = 0,..
.
λ∗,N −1 + λN −2,N −1 − λN −1,N = 0,λ∗,N + λN −1,N = 1.
It remains to show that the corresponding dual matrix S is positive semidefinite.
2S =

N
−1
X

>
2αi+1 λi,i+1 ei+1 e>
i+1 + 2τ eN +1 eN +1

i=1

+

N
X
i=1


λ∗,i ei−eN +1 +

i
X
k=1

!>
αk ek

+

−eN +1 +

i
X

!
αk ek


.
e>
i

k=1

In order to reduce the number of indices to be used, we note shortly λi = λi,i+1
and µi = λ∗,i . Then, using the equality constraints, we arrive at the following
dual matrix:
2α1 λ1
α1 µ2
α1 µ3
...
α1 µN −1
α1 µN−µ1
 α1 µ22α2 λ2α2 µ3...
α2 µN −1
α2 µN
−µ2 


 α1 µ3α2 µ32α3 λ3 . . .
α3 µN −1
α3 µN
−µ3 



..
..
..  .
..
..
2S = 
.
.
.
.
. 


α1 µN −1 α2 µN −1 α3 µN −1 . . . 2αN −1 λN −1 αN −1 µN −µN −1  α1 µN
α2 µN
α3 µN
...
αN −1 µN
2αN
−µN 
−µ1
−µ2
−µ3
...
−µN −1−µN
2τ
In order to prove S  0, we first use a Schur complement and then show thatthe resulting matrix is diagonally dominant with positive diagonal elements.

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

140

After the Schur complement, we obtain the matrix S̃:
2α1 λ1
α1 µ2
α1 µ3
...
α1 µN −1
α1 µN
 α1 µ22α2 λ2α2 µ3...
α2 µN −1
α2 µN  α1 µ3α2 µ32α3 λ3 . . .
α3 µN −1
α3 µN S̃ = 

..
..
..
..


.
.
.
.


α1 µN −1 α2 µN −1 α3 µN −1 . . . 2αN −1 λN −1 αN −1 µN 
α1 µN
α2 µN
α3 µN
...
αN −1 µN2αN   >µ1
µ1
  µ2 µ
2
1   
−  .  .  .
2τ  ..   .. µN

µN

The first step to show the diagonally dominant character of S̃ is to note that
µi µj
every non-diagonal element of S̃ is non-positive: αj µi − 2τ
≤ 0, ∀i 6= j.
Indeed, this is equivalent to write this in the following form (µi > 0):
µj
2τ

 P
2 
PN
Pi
Pi−1   PN
N
αk − k=1 αk − 2 k=1 αk 
 2 k=1αk − k=1 αk 2 k=1
 P= αj 
 ≤ 0,PN
Pi
Pi−1 
N
2 k=1 αk − k=1 αk 2 k=1 αk − k=1 αk

αj −since αk ≥ 0 by assumption. This allows us to discard the absolute values inthe diagonally dominance criteria. Then, using the equality constraints, we
obtain an expression for the sum of all non-diagonal elements of line i of S̃:
µi

i−1
X
j=1

αj + αi

N
X
j=i+1

µi X
µj
2τ
j6=i
(
Pi−1
1
µi j=1 αj + αi (1 − λi ) − 2τµi (1 − µi ),
PN −1
=
1
µN j=1 αj − 2τ µN (1 − µN )µj −if i < N
if i = NUsing the values of µi , λi and τ along with elementary computations allow toverify ∀i ∈ {1, . . . , N }:
(
Pi−1
µ2
1
−(µi j=1 αj + αi (1 − λi ) − 2τµi (1 − µi )) = 2αi λi − 2τi if i = 1, . . . , N − 1,
Pi−1
µ2
1
−(µi j=1 αj − 2τµi (1 − µi ))= 2αi − 2τi
if i = N,
which implies diagonal dominance of S̃ (even more: the sum of the elements ofeach line equals 0).

141

5.B

5.B. PROOF OF THEOREM 5.15

Proof of Theorem 5.15

Again, we denote by PN the matrix containing the information required tomodel the algorithm in the (SDP-PEP) format:
()

()

PN = [g1 g2 . . . gN g1 g2(). . . gN Bx0 ],and by GN its corresponding Gram matrix (see Section 5.2.1). In that case,the step size vectors αk are defined asx k = PN α k(k = 0, . . . , N ),and can be written in the following form:
αk = e2N +1 −kX

αi (ei + eN +i )i=1

along with α∗ = 0 (again, we use the standard notation ei for the unit vector
having a single 1 as its ith component and we denote e∗ = 0). We can now state
the performance estimation problem of the inexact PPA in the (SDP-PEP)
format:max

GN ∈SN +1 ,FN ∈RN

fN − f∗(uncPPA-PEP)

s.t. fj − fi + Tr(Aij GN ) ≤ 0,()2i, j ∈ {1, . . . , N, ∗}≤ 2 ,giE∗2
kx0 − x∗ kE ≤ R2 ,i ∈ {1, . . . , N }GN  0,with 2Aij = ej (αi − αj )> + (αi − αj )e>
j , coming again from the non-smoothinterpolation constraints (see Theorem 3.4).
In order to obtain an analytical upper bounds for PPA, we consider the Lagrangian dual to (uncPPA-PEP), which is given by the following.
min

λij ≥0,γij ≥0,τ ≥0

τ R 2 + 2

N
X

γi

(uncPPA-dPEP)

i=1

s.t. eN −

XX
i

XX
i

j6=i

(λij − λji )ej = 0,j6=iλij Aij +X
i>γi eN +i e>
N +i + τ α0 α0  0.

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

142

Lower bound. We√ start by considering√the following one-dimensional lower
bound: F (x) =  B|x|, and g () = − B. Clearly, whatever the starting
point x0√> 0, the algorithm does not allow it to√ move (similar conclusion forg () =  B and x0 < 0). By choosing x0 = R/ B, we have the desired lower
bound.

Upper bound. Then, we show that the bound is actually tight withR
 by showing that the following choice is dual feasible:

PN

Pi
αk
λi,i+1 = Pk=1,N
k=1 αk

αi
.
2

τ=

1
2

PN

k=1 αk,αi
λ∗,i = PN

k=1 αk,γi =k=1 αk =Those conditions satisfy the flow constraints, as very few computations allow
to verify:
λ1,2 − λ∗,1 = 0,
λ2,3 − λ1,2 − λ∗,2 = 0,..
.
λN −1,N − λN −2,N −1 − λ∗,N −1 = 0,
λN −1,N + λ∗,N = 1.
Let us show that the corresponding dual matrix S is positive semidefinite.
2S =

N
−1
X

>
>
αi+1 λi,i+1 (2ei+1 e>
i+1 + ei+1 eN +i+1 + eN +i+1 ei+1 )

i=1

+

N
X

>
2γi eN +k e>
N +k + 2τ e2N +1 e2N +1

k=1

+

N
X


λ∗,i ei−e2N +1 +

i=1

i
X

!>
αk (ek + eN +k )k=1+−e2N +1 +

i
X

!

#

αk (ek + eN +k ) e>
.
i

k=1

As in the proof of Theorem 5.13, we use a Schur complement followed by aweak diagonal dominance to prove positive semidefinitess of S. The Schur

143

5.B. PROOF OF THEOREM 5.15

complement allows us to write:
2α1 λ1
α1 µ2
α1 µ3
 α1 µ2
2α
λ
α
2
2
2 µ3

 α1 µ3
α
µ
2α
2
3
3 λ3

S̃ = 
..
.
..

.

α1 µN −1 α2 µN −1 α3 µN −1
α1 µN
α2 µN
α3 µN

...
...
...
..
.

α1 µN −1
α2 µN −1
α3 µN −1...
...

2αN −1 λN −1
αN −1 µN

α1 µN
α2 µN
α3 µN
..
.







 − LD−1 L>,
αN −1 µN 
2αNwith the lower triangular matrix L and the diagonal matrix D respectively
defined as


α1 λ1
0
0 ...
0
−µ1
 α1 µ2 α2 λ2
0 ...
0
−µ2 


L = .
..  ,
.
..
 ..
. 
α1 µN α2 µN . . .
αN −1 λN −1 −µN
γ1 00 ... 0 0
 0 γ2 0 . . . 0 0


 ..
..
..  .
..
D =2  .
.
.
.
 0 0 ...
γN 00 0 ...
0 τ
Before going into the details, let us denote the matrix S̃ = A − B with A and
B the matrices arising previously from the Schur complement argument on 2S.
The elements of A are( αi αjPN
if i 6= j
k=1
Piαk
aij =
2αi k=1 αk
PNotherwise.
α
k=1

k

The elements of B arePj−1
22αi αjα α
k=1 αk +αi αj+ PNi jα
P

2
N

α
)
(

k=1 k
k=1 k
Pi−1
P
P
2
α
+
α
+α
α2i ( N)( ik=1 αk )
i
k
k
bij =
k=1
k=1
PN
2


( k=1 αk )bji

if i 6= j, i > j,if i = j,otherwise.

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

144

The following computations allow to guarantee that the diagonal terms of S̃
are indeed non-negative:
P
 P

i
N
2αi
k=1 αkk=1 αk
aii =
,
P
2
N
α
k
k=1
P

P
 P

N
i−1
N
αi2
k=1 αk + αi
k=1 αkk=1 αk
=
,
P
2
N
α
k=1 k
P
2
P
 P

i
i
N
αi
α
+
α
α
α
i
k=1 k
k=1 k
k=i+1 k+P2N
k=1 αk≥ bii ,
using the non-negativity of hk . Some computations In order to discard theabsolute values in the diagonal dominance criterion, remark that all off-diagonalterms of S̃ are non-positive, as for i 6= j, we have
Pj−1
2αi αj k=1 αk + αi αj2S̃ij = aij − bij = −
.
P
2
N
k=1 αkIn order to conclude the proof, we show that S̃ is weakly diagonally dominant,as we have
i−1
N
X
X
S̃ii =(−S̃ij ) +(−S̃ji ).
j=1

j=i+1

This can be written in terms of the elements of S̃, and by using the values ofthe multipliers as
" i
! N
!
#
N
N
i−1
X
X
X
X
X
αi
αk
αk +
αk − αiαk − αi
αk
P
2
N
k=1
k=1
k=i+1
k=1
k=1
k=1 αk! i−1 j−1
!
j
i−1
N
i
i−1
X
X
X
X
X
X
X
αi
= P
αj
αk +
αj αk +αjαk +αk  .
2 
N
j=1
j=1
j=i+1
k=1
k=1
k=1
k=1
α
k=1 k

145

5.B. PROOF OF THEOREM 5.15

Basic simplifications allow to show that proving the last equality is equivalent
to prove the following:


!
j
j−1
i−1
N
i−1
N
X
X
X
X
X
X
αk 
αj − αi  =
αj
αk +
αk +
αk .
k=1

j=1

j=1

k=1

k=1

k=i+1

To prove this equality, we focus on the right-hand term:
!
j
j−1
i−1
N
X
X
X
X
αj
αk +
αk +
αk
j=1

k=1

k=1

k=i+1

=

k
i−1 X
X

αj αk +

k=1 j=1

=

i−2 X
i−1
X

i−1 X
i−1
X

αj αk +=i−1X

αj αk +

i−1 X
N
X

N
i−1 X
X

αj αk ,

k=1 j=i+1

αj αi−1 +j=1

k=1 j=1i−1X

αj αk +

k=1 j=k+1

k=1 j=1

=

i−1
i−2 X
X

i−1 X
N
X

αj αk ,

k=1 j=i+1

αj αk ,

k=1 k=i+1



N
X
αk 
αj − αi  .

k=1

j=1

In order to conclude the proof, observe that the objective value of the dual isτ R 2 + 2NXγi = R,

i=1

using the multiplier values along withPN

R
k=1 αk =  .

CHAPTER 5. PEPS FOR COMPOSITE CONVEX OPTIMIZATION

146

Chapter 6

Steepest Descent with
Exact Line Search
In this chapter, we consider the gradient (or steepest) descent method withexact line search applied to a strongly convex function with Lipschitz continuous gradient. We establish the exact worst-case rate of convergence of thisscheme, and show that this worst-case behavior is exhibited by a certain convex
quadratic function. We also give the tight worst-case complexity bound for anoisy variant of gradient descent method, where exact line search is performed
in a search direction that differs from negative gradient by at most a prescribed
relative tolerance.
The main contributions of the chapter are the following: we provide a tight analysis of the steepest descent algorithm with exact
line search for when the function to be minimized is smooth and strongly
convex.
 We provide a tight analysis of an inexact version of steepest descent withexact line search on the same class of functions.
This chapter is divided into five sections:
 in Section 6.1, we introduce the steepest descent algorithm with exact
line search, claim the main results and compare them with the literature.
 Section 6.2 summarizes properties of steepest descent with exact line
search used prove our main results. Also, the corresponding performance
estimation problem is formulated.
 In Section 6.3, we prove our main result using the performance estimation
approach.
147

CHAPTER 6. EXACT LINE SEARCH

148

 In Section 6.4, we perform a similar analysis for an inexact version. The
methodology used in the analysis relies on performance estimations problems in the same way as in the exact case.
 Finally, Section 6.5 presents our concluding remarks on the algorithms
and on the methodology.
The subsequent text is based on the paper [dKGT16].
Note that for the sake of simplicity, we work in the case E = E∗ = Rd in
the following pages, but all the results also hold for different primal and dual
Euclidean spaces (see Section 2.1 and Chapter 5).

6.1

Introduction

The gradient (or steepest) descent method for unconstrained method was devised by Augustin-Louis Cauchy (1789–1857) in the 19th century, and remains
one of the most iconic algorithms for unconstrained optimization. Indeed, it isusually the first algorithm that is taught during introductory courses on nonlinear optimization. It is therefore somewhat surprising that the worst-caseconvergence rate of the method is not yet precisely understood for smooth
strongly convex functions.
In this chapter we settle the worst-case convergence rate question of the gradient descent method with exact line search for strongly convex, continuously
differentiable functions f with Lipschitz continuous gradient.
The gradient method with exact line search may be described as follows.
Gradient descent method with exact line search
Input: f ∈ Fµ,L (Rd ), x0 ∈ Rd .
for i = 0, 1, . . .
γ = argmin f (xi − γ∇f (xi ))γ∈R

xi+1 = xi − γ∇f (xi )Our main result may now be stated concisely.
Theorem 6.1. Let f ∈ Fµ,L (Rd ), x∗ a global minimizer of f on Rd , andf∗ = f (x∗ ). Each iteration of the gradient method with exact line searchsatisfies2
L−µf (xi+1 ) − f∗ ≤(f (xi ) − f∗ ) i = 0, 1, . . .
(6.1)
L+µ
Note that the result in Theorem 6.1, which establishes a global linear convergence rate on objective function accuracy, is known for the case of quadratic

149

6.1. INTRODUCTION

functions in Fµ,L (Rd ), that is for functions of the formf (x) =

1 >x Qx + c> x2where c ∈ Rd , and the eigenvalues of the n × n symmetric positive definite
matrix Q lie in the interval [µ, L]; see e.g. [Ber99, §1.3], [Pol87, pp. 60–62], or[LY08, pp. 235–238]. Moreover, the bound (6.1) is known to be tight for thefollowing example.
Example 6.2. Consider the following quadratic function from [Ber99, Example on p.69]:
1
f (x) = x> diag(λ1 , λ2 , . . . , λd ) x2
where
0 < µ = λ1 ≤ λ2 ≤ . . . ≤ λd = L,and the starting point11x0 = ( , 0, . . . , 0, )> .
µLOne may readily check that the gradient at x0 is equal to∇f (x0 ) = (1, 0, . . . , 0, 1)>
and that the minimum of the line search from x0 in that direction is attained2
for step γ = L+µ. One therefore obtains

x1 =

L−µ
L+µ



(1/µ, 0, . . . , 0, −1/L)> ,and, for all i = 0, 1, . . .

x2i =

L−µ
L+µ

2i


x0 ,

x2i+1 =

L−µ
L+µ

2i
x1 .

Since f∗ = 0, it is straightforward to verify that equalityf (xi+1 ) − f∗ =

L−µ
L+µ

2
(f (xi ) − f∗ ) i = 0, 1, . . . ,holds as required.
The construction in Example 6.2 is illustrated in Figure 6.1 in the case n = 2,
where the ellipses shown are level curves of the objective function. Each stepfrom xi to xi+1 is orthogonal to the ellipse at xi (since it uses the steepest
descent direction) and tangent to the ellipse at xi+1 (because of the exact line
search direction), hence successive steps are orthogonal to each other.

CHAPTER 6. EXACT LINE SEARCH

150

√1
L

x∗

•

x2
x6 x4
•
• •
• •
x7 x5 x•3

• x0 = [1/µ, 1/L]>
•
x1

√1
µ

Figure 6.1: Illustration of Example 6.2 for the case n = 2 (small arrows indicate
direction of negative gradient).
As an immediate consequence of Theorem 6.1 and Example 6.2, one has the following tight bound on the number of steps needed to obtain -relative accuracy
on the objective function for a given  > 0.
Corollary 6.3. Given  > 0, the gradient method with exact line search yields
a solution
with relative
 for any function f ∈ Fµ,L (Rd ) after at mostl
 accuracym
L+µ
N = 12 log 1 / log L−µiterations, i.e.
f (xN ) − f∗≤ ,f (x0 ) − f∗where x0 is the starting point. Moreover, this iteration bound is tight for thequadratic function defined in Example 6.2.
For non-quadratic functions in Fµ,L (Rd ), only bounds weaker than (6.1) are
known. For example, in [LY08, p. 240], the following bound is shown:µ(f (xi+1 ) − f∗ ) ≤ 1 −(f (xi ) − f∗ ) i = 0, 1, . . .
L

151

6.2. BACKGROUND RESULTS

In [NW06, Theorem 3.4] a stronger result than Theorem 6.1 was claimed, but
retracted in a subsequent erratum1 , which now only claims an asymptotic result.
A result related to Theorem 6.1 is given in [Nem99] where Armijo-rule line
search is used instead of exact line search. An explicit rate in the strongly
convex case is given there in Proposition 3.3.5 on page 53 (definition of themethod is (3.1.2) on page 44). More general upper bounds on the convergence
rates of gradient-type methods for convex functions may be found in the books[NY83, Nes04]. We mention one more particular result by Nesterov [Nes04]
that is similar to our main result in Theorem 6.1, but that uses a fixed step
size and relies on the initial distance to the solution.
Theorem 6.4 (Theorem 2.1.15 in [Nes04]). Given f ∈ Fµ,L (Rd ) and x0 ∈ Rd ,2generate iterates
the gradient descent method with fixed step length γ = µ+L
xi (i = 0, 1,2, . . .) that satisfyLf (xi ) − f∗ ≤
2



L−µ
L+µ

2i
kx0 − x∗ k2

i = 0, 1, . . .

Note that this result does not imply Theorem 6.1.

6.2

Background results

In this section we collect some known results on strongly convex functions andon the gradient method. We will need these results in the proof of our main
result, Theorem 6.1.

6.2.1

Properties of the gradient method with exact line
searchLet xi (i = 1, 2, . . . , N ) be the iterates produced by the gradient method withexact line search started at x0 . Those iterates are defined by the following two
conditions for i = 0, 1, . . . , N − 1
xi+1 − xi + γ∇f (xi ) = 0, for some γ ≥ 0,

(6.2)

h∇f (xi+1 ), xi+1 − xi i = 0(6.3)

where the first condition (6.2) states that we move in the direction of thenegative gradient, and the second condition (6.3) expresses the exact line search
condition.
1 The erratum is available at: http://users.iems.northwestern.edu/ nocedal/book/~2ndprint.pdf.

CHAPTER 6. EXACT LINE SEARCH

152

A consequence of those conditions is that successive gradients are orthogonal,i.e.,h∇f (xi+1 ), ∇f (xi )i = 0 i = 0, 1, . . . , N − 1.
(6.4)
Instead of relying on conditions (6.2)–(6.3) that define the iterates of the gradient method with exact line search, our analysis will be based on the weakerconditions (6.3)–(6.4), which are also satisfied by other sequences of iterates.

6.2.2

Performance estimation of the gradient method withexact line searchConsider the following SDP problem, for fixed parameters N ≥ 1, R > 0, µ > 0
and L > µ:max fN − f∗s.t.
hgi+1 , xi+1 − xi i
hgi+1 , gi i{(xi , fi , gi )}i∈{∗,0,1,...,N }g∗f0 − f∗

=
=
is
=
≤









0 i ∈ {0, 1, . . . , N − 1} 
0 i ∈ {0, 1, . . . , N − 1}
Fµ,L -interpolable




0




R,




(6.5)

where the variables are xi ∈ Rd , fi ∈ Rd and gi ∈ Rd (i ∈ {∗, 0, 1, . . . , N }).
Lemma 6.5. The optimal value of the above SDP problem (6.5) is an upperbound on f (xN )−f∗ , where f is any function from Fµ,L (Rd ), f∗ is its minimum
and xN is the N th iterate of the gradient method with exact line search applied
to f from any starting point x0 that satisfies f (x0 )−f∗ ≤ R.
Proof. Fix any f ∈ Fµ,L (Rd ), and let x0 , . . . , xN be the iterates of the gradientmethod with exact line search applied to f . Now a feasible solution to the SDP
problem is given byxi , fi = f (xi ), gi = ∇f (xi )i ∈ {∗, 0, . . . , N }.

The objective function value at this feasible point is fN = f (xN ), so that the
optimal value of the SDP is an upper bound on f (xN ) − f∗ .
We are now ready to give a proof of our main result. We already mention that
the SDP relaxation (6.5) is not used directly in the proof, but was used todevise the proof, in a sense that will be explained later.

153

6.3

6.3. CONVERGENCE

Proof of Theorem 6.1

We only consider one iteration of the gradient method with exact line search,
as we will see that it is sufficient to prove Theorem 6.1. Thus we consider only
the first iterates, given by x0 and x1 , as well as the minimizer x∗ of f ∈ Fµ,L .
Set fi = f (xi ) and gi = ∇f (xi ) for i ∈ {∗, 0, 1}. Note that g∗ = 0. The
following five inequalities are now satisfied:1
2
kg0 − g1 k
2L
2
µ
1
+
x0 − x1 − (g0 − g1 ) ,2(1 − µ/L)L
1
2f∗ ≥f0 + hg0 , (x∗ − x0 )i +kg0 − g∗ k
2L
2
µ
1
+
x0 − x∗ − (g0 − g∗ ) ,2(1 − µ/L)L
1
2f∗ ≥f1 + hg1 , (x∗ − x1 )i +kg1 − g∗ k
2L
2
µ
1
+
x1 − x∗ − (g1 − g∗ ) ,2(1 − µ/L)L
f0 ≥f1 + hg1 , x0 − x1 i +

1:

2:

3:

4:

0 ≥hg0 , g1 i,

5:0 ≥hg1 , x1 − x0 i.

Indeed, the first three inequalities are the Fµ,L -interpolability conditions, thefourth inequality is a relaxation of (6.4), and the fifth inequality is a relaxationof (6.3).
We aggregate these five inequalities by defining the following positive multipliers,
y1 =

L−µ
,
L+µ

y2 = 2µ(L − µ),(L + µ)2

y3 =

2µ
,
L+µ

y4 =

2
,
L+µ

y5 = 1,(6.6)
and adding the five inequalities together after multiplying each one by thecorresponding multiplier.
The result is the following inequality (as may be verified directly):

f1 − f∗

≤



L−µ
L+µ

2

(f0 − f∗ )

L+µ
2µ
3L+µ
L+µ
− µL(L+3µ)
2(L+µ)2 x0 − L+3µ x1 − L+3µ x∗ − L2 +3µL g0 − L2 +3µL g1
2

2

(L−µ)
2Lµ
L+µ
− L2 +2Lµ−3µ
2 x1 − x∗ − 2µL(L+µ) g0 − 2µL g1

2

2

.
(6.7)

CHAPTER 6. EXACT LINE SEARCH

154

Since the last two right-hand-side terms are nonpositive, we obtain:f1 − f∗ ≤

L−µ
L+µ

2
(f0 − f∗ ).

Since x0 was arbitrary, this completes the proof of Theorem 6.1.

6.3.1

Remarks on the proof of Theorem 6.1.

 First, note that we have proven a bit more than what is stated in Theorem
6.1. Indeed, the result in Theorem 6.1 holds for any iterative method thatsatisfies the five inequalities used in its proof.
 Although the proof of Theorem 6.1 is easy to verify, it is not apparent
how the multipliers y1 , . . . , y5 in (6.6) were obtained. This was in fact
done via preliminary computations, and subsequently guessing the valuesin (6.6), through the following steps:
1. The SDP performance estimation problem (6.5) with N = 1 wassolved numerically for various values of the parameters µ , L and R— actually, the values of L and R can safely be fixed to some positive
constants using appropriate scaling arguments (see e.g., Section 4.2.5
or [THG16a, Section 3.5] for a related discussion).
2. The optimal values of the dual SDP multipliers of the constraints
corresponding to the five inequalities in the proof gave the guesses
for the correct values y1 , . . . , y5 as stated in in (6.6).
3. Finally the correctness of the guess was verified directly (by symbolic
computation and by hand).
 The key inequality (6.7) may be rewritten in another, more symmetric
way
!

2
2
2
1−κ
µ
ks1 k
ks2 k
√ +
√
−
,
(f1 − f∗ ) ≤ (f0 − f∗ )1+κ4 1+ κ 1− κwhere κ = µ/L is the condition number (between 0 and 1) and slack
vectors s1 and s2 are
√
p  
p 
(1 + κ)2 
s1 = −
x0 − x∗ − g0 / Lµ + x1 − x∗ + g1 / Lµ ,
1+κ
√
p  
p 
(1 − κ)2 s2 =x0 − x∗ + g0 / Lµ − x1 − x∗ − g1 / Lµ .
1+κ√Note that the four expressions xi − x∗ ± gi / Lµ expressions are invariant

155

6.3. CONVERGENCE

under dilation of f , and that cases of equality in (6.7) simply correspondto equalities s1 = s2 = 0.
 It is interesting to note that the known proof of Theorem 6.1 for thequadratic case only requires the so-called Kantorovich inequality, that
may be stated as follows.
Theorem 6.6 (Kantorovich inequality; see e.g. Lemma 3.1 in [Ber99]).
Let Q be a symmetric positive definite n × n matrix with smallest andlargest eigenvalues µ > 0 and L ≥ µ respectively. Then, for any unit
vector x ∈ Rd , one has: (µ + L)2
x> Qx x> Q−1 x ≤.
4µL

Thus, the inequality (6.7) replaces the Kantorovich inequality in the proofof Theorem 6.1 for non-quadratic f ∈ Fµ,L (Rd ).
 Finally, we note that this proof can easily be modified to handle the caseof the fixed-step gradient method that was mentioned in Theorem 6.4.
Indeed, observe that the proof aggregates the fourth and fifth inequalities2
with multipliers y4 = L+µand y5 = 1, which leads to the combined
inequality−2
hg0 , g1 i + hg1 , x0 − x1 i ≥ 0
L+µ⇔hg1 , x0 −2
g0 − x1 i ≥ 0 .
L+µ2
Now note that the gradient method with fixed step γ = L+µsatisfies
this combined inequality (since the second factor in the left-hand side
becomes zero), and hence the rest of the proof establishes the same rate
for this method as for the gradient descent with exact line search.

Theorem 6.7. Let f ∈ Fµ,L (Rd ), x∗ a global minimizer of f on Rd , andf∗ = f (x∗ ). Each iteration of the gradient method with fixed step length2γ = µ+L
satisfiesf (xi+1 ) − f∗ ≤

L−µ
L+µ

2
(f (xi ) − f∗ )i = 0, 1, . . .

Note that Example 6.2 also establishes that this rate is tight. Hence wehave the relatively surprising fact that, when looking at the worst-caseconvergence rate of the objective function accuracy, performing exact line
search is not better than using a well-chosen fixed step length.

CHAPTER 6. EXACT LINE SEARCH

6.4

156

Extension to ‘noisy’ gradient descent withexact line search

Theorem 6.1 may be generalized to what we will call noisy gradient descentmethod with exact linear search; see e.g. [Ber99, p.59] where it is called gradientdescent method with (relative) error. Here the search direction at iteration i,say di , satisfiesk − ∇f (xi ) − di k ≤ εk∇f (xi )ki = 0, 1, . . . ,(6.8)

where 0 ≤ ε < 1 is some given relative tolerance on the deviation from thenegative gradient. Note that the algorithm cannot be guaranteed to converge
as soon as ε ≥ 1, since di = 0 then becomes feasible. We recover the normal
gradient descent algorithm when ε = 0. Note that this model differs from theabsolute inaccuracy on the subgradient that we considered on the proximal
point algorithm (see Theorem 5.15).
In the case of more general values of ε, one can for example satisfy the relative
error criterion by imposing a restriction of the type | sin θ| ≤ ε on the angle θ
between search direction di and the current negative gradient −∇f (xi ).
Using a search direction di that satisfies (6.8) corresponds, for example, toan implementation of the gradient descent method where each component of−∇f (xi ) is only calculated to a fixed number of significant digits.
Thus we consider the following algorithm:
Noisy gradient descent method with exact line search
Input: f ∈ Fµ,L (Rd ), x0 ∈ Rd , 0 ≤ ε < 1.
for i = 0, 1, . . .
Select any seach direction di that satisfies (6.8);
γ = argminγ∈R f (xi − γdi )xi+1 = xi − γdi

One may show the following generalization of Theorem 6.1.
Theorem 6.8. Let f ∈ Fµ,L (Rd ), x∗ a global minimizer of f on Rd , andf∗ = f (x∗ ). Given a relative tolerance ε, each iteration of the noisy gradientdescent method with exact line search satisfies
2
1 − κεf (xi+1 ) − f∗ ≤(f (xi ) − f∗ ) i = 0, 1, . . .
(6.9)
1 + κεµ (1−ε)where κε = L(1+ε) .

157

6.4. NOISY SEARCH DIRECTIONS

L−µProof. When ε = 0, the rate becomes 1−κ
1+κ = L+µ , which matches exactly
Theorem 6.1, and the proof of Theorem 6.8 is a straightforward generalizationof the proof of Theorem 6.1. The key is again to consider a wider class ofiterative methods that satisfies certain inequalities. We use the following ones:

1:

2:

3:

1
2
kg0 − g1 k2L21µx0 − x1 − (g0 − g1 ) ,+2(1 − µ/L)L
1
2f∗ ≥f0 + hg0 , x∗ − x0 i +kg0 − g∗ k
2L
2
µ
1
+
x0 − x∗ − (g0 − g∗ ) ,2(1 − µ/L)L
1
2f∗ ≥f1 + hg1 , x∗ − x1 i +kg1 − g∗ k
2L
2
µ
1
+
x1 − x∗ − (g1 − g∗ ) ,2(1 − µ/L)L
f0 ≥f1 + hg1 , x0 − x1 i +4:0 ≥hg1 , x1 − x0 i,

5:0 ≥hg0 , g1 i − εkg0 kkg1 k.

(6.10)

The first four inequalities are the same as before, and the fifth is satisfied by
the iterates of the noisy gradient descent with exact line search. Indeed, in thefirst iteration one has:
0

=
=
≤

hd0 , g1 i
kg1 k
hd0 + g0 , g1 i hg0 , g1 i
−
kg1 k
kg1 k
hg0 , g1 i
εkg0 k −kg1 k(exact line search)(by Cauchy-Schwartz and (6.8)).

We rewrite the fifth inequality as the equivalent linear matrix inequality:
εkg0 k2 hg0 , g1 i 0.
(6.11)
hg0 , g1 i εkg1 k2
We first aggregate the first four inequalities by adding them together after
multiplication by the respective multipliers:
y1 = ρε ,y2 = 2κε

1 − κε,(1 + κε )2

y3 =

2κε
,
1 + κεy4 = 1,CHAPTER 6. EXACT LINE SEARCH

158

µε
ε
where Lε = (1 + ε)L, µε = (1 − ε)µ, κε = Land ρε = 1−κ
1+κε .
εNext we define a positive semidefinite matrix multiplier for the linear matrix
inequality (6.11), namelyaρε −a 0,(6.12)a−a ρε1with a = Lε +µ, and add nonnegativity of the inner product between theεleft-hand-side of (6.11) and the multiplier matrix (6.12) to the aggregated constraints. It can now be checked that the resulting expression is the following
generalization of (6.7)

f1 − f∗ ≤ρ2ε (f0 − f∗ )Lµ(Lε − µε )(Lε + 3µε )2kx0 + α1 x1 − (1 + α1 )x∗ + α2 g0 + α3 g1 k
2(L − µ)(Lε + µε )2
2Lµµε
2
−
kx1 − x∗ + α4 g0 + α5 g1 k ,(L − µ)(Lε + 3µε )−with the appropriate coefficients
α1 = −

4L − Lε + µε
(Lε + µε )(−4L + 3Lε + µε )Lε + µε, α2 = −, α3 =,
Lε + 3µε
L(Lε + 3µε )L(Lε − µε )(Lε + 3µε )(L − µ)(Lε − µε )L+µα4 = −, α5 = −.
2Lµ(Lε + µε )2LµThis completes the proof.

To conclude this section, the following example, based on the same quadratic
function as Example 6.2, shows that our bound (6.9) for the noisy gradientdescent is also tight.
Example 6.9. Consider the same quadratic function as in Example 6.2:

f (x) =

1 >x diag(λ1 , λ2 , . . . , λd ) x, where 0 < µ = λ1 ≤ λ2 ≤ . . . ≤ λd = L.
2

Let θ be an angle satisfying 0 ≤ θ < π2 . Consider the noisy gradient descentmethod where direction d0 is obtained by performing a counter-clockwise 2Drotation with angle θ on the first and last coordinates of the gradient ∇f (x0 ).
As mentioned above, this satisfies our definition with relative tolerance ε =sin θ. Define now the starting point
x0 =

1
1
, 0, . . . , 0,
µ
L

r

1−ε
1+ε

!>
.

159

6.4. NOISY SEARCH DIRECTIONS

√1
L

x∗

x6 x4 x•2
• x0
• •
•
x7 x•5 x•3 x•

•

1

√1
µ

Figure 6.2: Illustration Example 6.9 for n = 2 and ε = 0.3 (small arrows
indicate direction of negative gradient).

Tedious but straightforward computations show thatx1 =1 − κε1 + κε

1
1, 0, . . . , 0, −
µ
L

r

1−ε
1+ε

!>
where κε =µ (1 − ε).
L (1 + ε)Moreover, if one chooses d1 by rotating the second gradient ∇f (x1 ) by thesame angle θ in the clockwise direction, one obtainsx2 =1 − κε1 + κε2

1
1, 0, . . . , 0,
µ
L

r

1−ε
1+ε

!>


=

1 − κε1 + κε2x0 .

A similar reasoning for the next iterates, alternating clockwise and counterclockwise rotations, shows thatx2i =1 − κε1 + κε

2i


x0 ,

x2i+1 =

1 − κε1 + κε2i
x1 for all i = 0, 1, . . .

and hence we have that equalityf (xi+1 ) − f∗ =1 − κε1 + κε2(f (xi ) − f∗ ) i = 0, 1, . . .

holds as announced. Figure 6.2 displays a few iterates, and can be compared
to Figure 6.1.
Before concluding the chapter, let us formulate a more practical version of Theorem 6.8 (which also applies for Theorem 6.1).

CHAPTER 6. EXACT LINE SEARCH

160

Corollary 6.10. Let f ∈ F0,∞ (Rd ), Q ⊂ Rd be the sublevel set Q = {x :f (x) ≤ f (x0 )}, and x∗ ∈ Q be optimal. Assuming that µId  ∇2 f (x)  LId
for any x ∈ Q, each iteration of the noisy gradient method with exact line
search satisfies2
1 − κεf (xi+1 ) − f∗ ≤(f (xi ) − f∗ ) i = 0, 1, . . .
(6.13)
1 + κεµ (1−ε)where κε = L(1+ε) .

Proof. First, note that µId  ∇2 f (x)  LId for any x ∈ Q implies that f
locally satisfies Conditions 2.4 (smoothness) and 2.7 (strong convexity).
Hence, one can apply Corollary 2.60 in order to obtain an extended functionf˜ ∈ Fµ,L (Rd ) such that f (x) = f˜(x) for all x ∈ Q (which is closed by continuityof f , and convex by convexity of f ).
Then, it suffices to apply Theorem 6.1 to f˜(x), which is equal to f for all
iterates (because each iteration improve the value of the objective function,and hence each new iterate belongs to the original sublevel set Q).

6.5

Conclusion

The main results of this chapter are the exact convergence rates of the gradient descent method with exact line search and its noisy variant for strongly
convex functions with Lipschitz continuous gradients. The computer-assisted
technique of proof is also of independent interest, and demonstrates the importance of the SDP performance estimation problems (PEPs) introduced in[DT14].
Indeed, to obtain our proof of Theorem 6.8, the following SDP PEP was solved
numerically for various fixed values of R, µ and L:
max f1 − f∗ subject to (6.10) and f0 − f∗ ≤ R.
It was observed that, for each set of values, the optimal value of the SDP
corresponded exactly to the bound in Theorem 6.8 (actually, for homogeneity
reasons, L and R could be fixed and only µ needed to vary). Based on this,a rigorous proof Theorem 6.8 could be given by guessing the correct values ofthe dual SDP multipliers as functions of µ, L and R, and then verifying theguess through an explicit computation.
We believe this type of computer-assisted proof could prove useful in the analysis of more methods where exact line search is used (see for example Section 5.3.5 where we studied a fixed step conditional gradient method; could we
use line search instead ?).

161

6.5. CONCLUSION

PEPs have been used by now to study worst-case convergence rates of several first-order optimization methods (see Chapters 4, 5 and [DT14, THG16a,
THG16b]). This work differs in an important aspect: the performance estimation problem considered actually characterizes a whole class of methods that
contains the method of interest (gradient descent with exact line search) as wellas many other methods. This relaxation in principle only provides an upperbound on the worst-case of gradient descent, and it is the fact that Example 6.2
matches this bound that allows us to conclude with a tight result.
The reason we could not solve the performance estimation problem for thegradient descent method itself is that equation (6.2), which essentially statesthat the step xi+1 − xi is parallel to the gradient ∇f (xi ), cannot be formulated
as a convex constraint in the SDP formulation. The main obstruction appears
to be that requiring that two vectors are parallel is a nonconvex constraint,even when working with their inner products2 . Instead, our convex formulation
enforces that those two vectors are both orthogonal to a third one, the next
gradient ∇f (xi+1 ).

2 One such nonconvex formulation would be hg , x − xii
i+1 i = kgi kkxi − xi+1 k.

CHAPTER 6. EXACT LINE SEARCH

162

Chapter 7

Proximal Gradient Method
In this chapter, we establish tight convergence rates for the proximal gradientmethod applied to the sum of a smooth strongly convex function and a nonsmooth convex function with a proximal operator available. Those convergenceguarantees are shown to be valid for different standard performance measures:
objective function accuracy, distance to optimality and residual (sub)gradientnorm. The global and exact worst-case guarantees we present for the proximal
gradient method are conceptually very simple, although apparently new.
On the way, our results allow explicitly weakening the assumptions for obtaining the corresponding linear convergence guarantees, establish that the fixed2
step size policy L+µ
is optimal for decreasing the distance to optimality, objective function accuracy and residual gradient norm, and extend a recent resultof Chapter 6 (see also [dKGT16]) on the worst-case behavior of steepest descentwith exact line search to the non-smooth convex composite case.
This chapter is divided into three main parts:
 Section 7.1 presents the context, some particular cases and known convergence results for the proximal gradient method in the strongly convex
case.
 In Section 7.2, we provide simple lower bounds along with matching upper bound for standard convergence measures: objective function accuracy (OFA), distance to optimality (DO) and residual (sub)gradient norm
(RGN).
 In Section 7.3, we present convergence results for mixed initial and final
convergence measures (DO, RGN and OFA).
 Finally, we conclude and propose further research directions in Section 7.4.
The subsequent text is based on work [THG16c].

163

CHAPTER 7. PROXIMAL GRADIENT METHOD

164

In the following, we work in the Euclidean space Rd endowed with the inner2
product h., .i : Rd × Rd → R and the corresponding Euclidean norm kxk =dhx, xi ∀x ∈ R . Nevertheless, the analyses are also directly valid in any finite
dimensional real vector space E and the corresponding dual space E∗ with
Euclidean structures, as introduced in Section 2.1.

7.1

Introduction

We consider the two-term composite strongly convex optimization setting (instance of the general composite optimization problem (CM))
min {F (x) ≡ f (x) + h(x)}

x∈Rd

(7.1)

where f ∈ Fµ,L (Rd ) is a L-smooth µ-strongly convex proper function over Rd ,for some 0 < µ ≤ L and h ∈ F0,∞ (Rd ) is convex, closed and proper overRd (i.e., we only consider strongly convex functions). In addition, we assume
that we can evaluate the gradient of f and the proximal operator of h [PB13,
Section 1.1]:12pγh (x) = argmin γh(y) + kx − yk .
(PROX)
2y∈Rd
Also, we use the proximal gradient method (PGM) with constant step length γ
to solve (7.1).
Proximal gradient method (PGM)
Input: x0 ∈ Rd , f ∈ Fµ,L (Rd ), h ∈ F0,∞ (Rd ), 0 ≤ γ ≤ L2 .
For k = 0 : N − 1
xk+1 = pγh (xk − γ∇f (xk ))For notational convenience, we denote by sk+1 ∈ ∂h(xk+1 ) the subgradient ofh used in the proximal operation (more details in Section 7.2.2), that is
sk+1 =

xk − xk+1− ∇f (xk ).
γ

(7.2)

Note that ∇f (xk+1 )+sk+1 ∈ ∂F (xk+1 ) is a particular subgradient of F at xk+1
that we can actually compute using (7.2) and whose convergence is studied
˜ (x) tohereafter. In the sequel, we will often use the compact notation ∇F˜denote a subgradient of F at x; hence ∇F (x) ∈ ∂F (x).

165

7.1. INTRODUCTION

Example 7.1. The composite convex problem (7.1) has the following very
common particular cases:- the unconstrained minimization problem minx∈Rd f (x) when h(x) = 0
and pγh (x) = x. In this case, PGM is simply the standard unconstrained
gradient method (UGM) xk+1 = xk − γ∇f (xk ).
- The constrained minimization problem minx∈Q f (x) with Q ⊆ Rd a closed
convex set. This corresponds to choosing h(x) = iQ (x) (iQ is the indicator function of Q) for which the proximal operation corresponds to a
projection onto Q: pγh (x) = ΠQ (x). In this case, PGM is simply thestandard projected gradient method (ΠGM) xk+1 = ΠQ (xk − γ∇f (xk )).
- The composite minimization problem minx∈Rd f (x) + h(x) where h(x)has an analytical proximal operator available1 (e.g., the classical `1 regularization term h(x) = kxk1 ).
Convergence rate.
0 < µ ≤ L < ∞)In the sequel, we use the notation (valid for 0 ≤ γ ≤ L2 ,ρ(γ) = max{|1 − Lγ|, |1 − µγ|} = max{(Lγ − 1), (1 − µγ)},(RHO)

so that ρ(γ) ≥ 0 for all values of the step size γ such that 0 ≤ γ ≤ L2 and0 < µ ≤ L < ∞. We prove in Section 7.2.3 that applying PGM to Problem (7.1) produces a sequence of iterates converging with rate ρ2 (γ) in distanceto optimality, residual gradient norm and objective function accuracy.
Note that the term (1 − Lγ)2 in the expression of ρ2 (γ) is minimized by taking
the so-called short step 1/L, whereas the second term (1 − µγ)2 is minimized
by choosing the so-called long step 1/µ. A direct implication is that the best2
possible worst-case convergence rate is achieved by the step size L+µ
for allthree performance measures (this is illustrated on Figure 7.1).
ρ2
1(1 − Lγ)2(1 − µγ)2
0
0

1
L

2
L+µ

2
L

1
µ

γ

Figure 7.1: Rate of convergence ρ2 (γ) as a function of the step size γ.
Our main result can now be stated.
1 A list of useful analytical proximal operators is available in [CP11].

CHAPTER 7. PROXIMAL GRADIENT METHOD

166

Theorem 7.2. Let f ∈ Fµ,L (Rd ), and h ∈ F0,∞ (Rd ) and consider the composite convex optimization problem (7.1) and a feasible starting point x0 ∈ Rd
(i.e., x0 is such that F (x0 ) < ∞). The iterates of PGM with 0 ≤ γ ≤ L2 satisfy
the following ∀k = 0, 1, 2, . . .:)(2kxk − x∗ k
= ρ2k (γ),
max
2kx0 − x∗ k
f ∈ Fµ,L (Rd )h ∈ F0,∞ (Rd )x 0 ∈ Rd)(2k∇f (xk ) + sk k
max
= ρ2k (γ),
2k∇f (x0 ) + s0 k
f ∈ Fµ,L (Rd )h ∈ F0,∞ (Rd )x 0 ∈ Rds0 ∈ ∂h(x0 )F (xk ) − F (x∗ )max= ρ2k (γ),
d
F
(x
)
−
F
(x
)
0
∗
f ∈ Fµ,L (R )h ∈ F0,∞ (Rd )x 0 ∈ Rd
where x∗ ∈ Rd denotes the optimal solution of (7.1), and sk denotes the subgradient used in the proximal operation to generate xk (see Equation 7.2).

Prior works. The UGM and ΠGM are standard methods whose analysis
in the context of smooth strongly convex functions can be found in numerous
references. The convergence in distance to optimality according to2

2

kxk+1 − x∗ k ≤ ρ2 (γ)kxk − x∗ k ,(7.3)

can be found in e.g., [Pol87, Section 1.4: Theorem 3], [RB16, Section 5.1] and[LRP16, Section 4.4] for UGM and ΠGM, with slight variations in the assumptions (depending on whether or not f is required to be twice differentiable). For
the specific step size 1/L, the guarantee (7.3) can be found as a particular caseµ
) forof [SLRB11, Proposition 3]. Also, weaker convergence rates such as (1 − L
the specific step size 1/L can be found in e.g., [Nes04, Theorem 2.2.8] or [Bub15,
Theorem 3.10] for ΠGM. Also note that (7.3) also holds PGM, as it essentiallyfollows the same proof technique as for ΠGM (using the non-expansiveness ofthe proximal operation).
˜ (xN ) are typically not
As far as we know, results in terms of F (xN )−F∗ or ∇F
as emphasized (or known) as compared to convergence in terms of kxN − x∗ k.
˜ (xN )
However, it is standard to convert results in terms of kxN − x∗ k to ∇F

167

7.2. CONVERGENCE

and F (xN ) − F∗ using the smoothness and strong convexity assumptions. In
the particular case of unconstrained minimization (i.e., h = 0), one can use:L2kxk − x∗ k , k∇f (xk )k ≤ Lkxk − x∗ k,
2µ
2
f (xk ) − f (x∗ ) ≥ kxk − x∗ k , k∇f (xk )k ≥ µkxk − x∗ k,
2

f (xk ) − f (x∗ ) ≤in order to adapt the convergence in terms of distance to optimality to convergence in objective function accuracy and residual gradient norm:LL 2k
ρ (γ)(f (x0 ) − f (x∗ )), and k∇f (xk )k ≤ ρk (γ)k∇f (x0 )k.
µ
µ
(7.4)
The bounds (7.4) are not tight because of the leading constant L/µ (see The2k
orem 7.2). In addition, we typically have L> 1 for small values of k,µρand therefore the former inequality (7.4) does not even guarantee an improvement in terms of objective function accuracy or residual gradient norm for few
iterations.

 
2
2
The global convergence rate ρ2 L+µ
in terms of objective func= L−µL+µf (xk ) − f (x∗ ) ≤2tion accuracy was only very recently obtained for UGM with step size L+µ
asa by-product of the convergence guarantee of using exact line search for solving unconstrained smooth convex minimization problems[dKGT16, Theoremµ
1.2], whereas previous results were establishing a 1 − L
global convergence
rate (see e.g., [Ber99, LY08]). Theorem 7.2 further extends this result in thecomposite case (7.1) for the different convergence measures, for embedding a
projection or a proximal step and for all reasonable step sizes. As a by-product,
we generalize Theorem 6.1 (see also [dKGT16, Theorem 1.2]) to proximal gradient methods with line search (see Section 7.4).

7.2

Convergence in distance, gradient and function accuracy

7.2.1

Quadratic lower bounds

First, we focus on the case of a quadratic function f without any nonsmoothterm (h = 0), which provides us with lower complexity bounds for the different
values of the step size γ. Those quadratics correspond to lower bounds for
UGM and therefore also for ΠGM and PGM. We will show that those are tightfor the class of smooth strongly convex functions in the following section.
Consider two constants 0 < µ ≤ L < +∞ and the corresponding quadratic22functions fµ (x) = µ2 kxk and fL (x) = L2 kxk . We clearly have that fµ , fL ∈

CHAPTER 7. PROXIMAL GRADIENT METHOD

168

Fµ,L (Rd ) and that x∗ = 0 and f∗ = 0 for both functions. In addition, one
iteration of UGM on those functions respectively gives:
(µ)

(µ)

(L)

xk+1 = (1 − γµ)xk ,(L)

xk+1 = (1 − γL)xk ,which respectively lead to
(µ)

(µ)

fµ (xk+1 ) = µ2 (1 − γµ)2 xk
(L)

(L)

fL (xk+1 ) = L2 (1 − γL)2 xk

2

,

(µ)

2

,

2

∇fµ (xk+1 )(L)∇fL (xk+1 )2(µ)= (1 − γµ)2 ∇fµ (xk )(L)= (1 − γL)2 ∇fL (xk )

2

,
2

.

Those equalities allow to conclude that the worst-case behaviour for any of22
the criterion f (xk+1 ) − f∗ , kxk+1 − x∗ k and k∇f (xk+1 )k is at least as bad
as in the cases of those two functions. That is, for any γ ∈ R, there exists af ∈ Fµ,L (Rd ) such that one iteration of UGM gives for all k ≥ 0:2

2

kxk − x∗ k = ρ2k (γ)kx0 − x∗ k ,f (xk ) − f∗ = ρ2k (γ)(f (x0 ) − f∗ ),
2

2k

(QLB)

2

k∇f (xk )k = ρ (γ)k∇f (x0 )k .
We will see that that no other function behaves (strictly) worse.

7.2.2

Basic inequalities characterizing one iteration of theproximal gradient method

Now, we make a short inventory of the inequalities available to prove the different global convergence rates. Note that recent works on performance estimation of first-order methods (see [THG16a, THG16b]) guarantee that no otherinequalities are needed in order to obtain the desired convergence results.
In the following, we denote by gi and si the (sub)gradients of respectively thesmooth function f and the non-smooth component h at the iteration k; thatis gk = ∇f (xk ) and sk ∈ ∂h(xk ), and by fk and hk the function values at
those points: fk = f (xk ) and hk = h(xk ). In addition to that, we denote byx∗ the optimal point (unique by strong convexity of F ) and by g∗ = ∇f (x∗ )and s∗ ∈ ∂h(x∗ ) the gradient and some subgradient of respectively f and h at
the optimum. Let us list the (in)equalities that enables us to characterize one
iteration of PGM.
(a) The iteration xk+1 = pγh (xk − γ∇f (xk )) can be rewritten using necessary and sufficient optimality conditions on the definition of the proximal
operation (PROX):
xk+1 = xk − γ(gk + sk+1 )169

7.2. CONVERGENCE

for some sk+1 ∈ ∂h(xk+1 ).
(b) Optimality of x∗ for (7.1) amounts to requiring g∗ + s∗ = 0 for somes∗ ∈ ∂h(x∗ ).
(c) For characterizing smoothness and strong convexity, we use the conditions from [THG16a, Theorem 4]. This should be required between threepoints: xk , xk+1 and x∗ . That is, ∀i 6= j ∈ {k, k + 1, ∗} (i.e., for the six
possible pairs (i, j) within {k, k + 1, ∗}) we have:fi ≥ fj2

1
kgi − gj k+hgj , xi − xj i + 2L2µ+ 2(1−µ/L) xi − xj − L1 (gi − gj ) .

(7.5)(d) Similarly, we require that for all pairs (i, j): i 6= j ∈ {k, k + 1, ∗} (again,
for the six possible combinations):hi − hj − hsj , xi − xj i ≥ 0,

(7.6)

for characterizing the (possibly non-smooth) convex function h.

7.2.3

Tight upper bounds

In this section, we prove the main convergence results of the paper, beginningwith the convergence in terms of distance to optimality.
Distance to optimality. As provided in Section 7.1, the following convergence result in term of distance to optimality is not new. For the sake of clarity
and completeness, we begin by proving it using the same technique that will
be used for the subsequent results (residual gradient norm and objective function accuracy). The proof methodology relies from the performance estimation
methodology (see [DT14, THG16a, THG16b, Dro14]). This technique has theadvantage of being transparent and of explicitly showing minimal assumptions
for obtaining this convergence property (see discussion below)..
Theorem 7.3 (Distance to optimality). Consider the composite convex optimization problem (7.1). Every pair of consecutive iterates of the PGM with0 ≤ γ ≤ L2 satisfies the following inequality:22

kxk+1 − x∗ k ≤ ρ2 (γ)kxk − x∗ k .

Proof. We use the notations and inequalities introduced in the previous section(Section 7.2.2) in order to construct the proof. As proposed in Section 7.2.2, we
use some of the interpolation inequalities (7.5) and (7.6) between the iterates

CHAPTER 7. PROXIMAL GRADIENT METHOD

170

and the optimal point. The proof consists in summing those interpolation
inequalities after multiplying them with their respective coefficients (multipliers
λ’s).

First, we use (7.5) with respectively (i, j) = (∗, k) and (i, j) = (k, ∗):2f∗ ≥ fk

1+hgk , x∗ − xk i + 2L
kgk − g∗ k2µ+ 2(1−µ/L) xk − x∗ − L1 (gk − g∗ )fk ≥ f∗1+hg∗ , xk − x∗ i + 2L
kgk − g∗ k2µ+ 2(1−µ/L) xk − x∗ − L1 (gk − g∗ ): λ0 ,

2: λ1

Then, we use (7.6) with respectively (i, j) = (∗, k + 1) and (i, j) = (k + 1, ∗):h∗ ≥ hk+1 + hsk+1 , x∗ − xk+1 i: λ2 ,

hk+1 ≥ h∗ + hs∗ , xk+1 − x∗ i

: λ3 .

We use the following multipliers
λ0 = λ1 = 2γρ(γ) ≥ 0,

λ2 = λ3 = 2γ ≥ 0.

After appropriate substitutions of xk+1 and s∗ , using xk+1 = xk − γ(gk + sk+1 )(Section 7.2.2, Condition (a)) and s∗ = −g∗ (Section 7.2.2, Condition (b)) , andwith little effort, one can check that the previous weighted sum of inequalities
can be written in one of the following forms. We divide the proof in two cases(corresponding to the two regimes of ρ(γ), see (RHO)).

2(i.e., ρ(γ) = (1 − γµ)), the expression can be written When 0 ≤ γ ≤ L+µ
as
2

2

2

(1 − γµ) kxk − x∗ k ≥kxk+1 − x∗ k + γ 2 kg∗ + sk+1 k+

2γ(2 − γ(L + µ))2kµ(xk − x∗ ) − gk + g∗ k ,L−µ2≥kxk+1 − x∗ k ,where the last inequality follows fromγ 2 ≥ 0, γ(2 − γ(L + µ)) ≥ 0, and L − µ ≥ 0.

2
 Similarly, when L+µ≤ γ ≤ L2 (i.e., ρ(γ) = (Lγ − 1)), the expression is171

7.2. CONVERGENCE

equivalent to22

2(1 − γL) kxk − x∗ k ≥kxk+1 − x∗ k + γ 2 kg∗ + sk+1 k+

2

γ(γ(L + µ) − 2)2kL(xk − x∗ ) − gk + g∗ k ,L−µ2≥kxk+1 − x∗ k ,where the last inequality follows fromγ 2 ≥ 0, γ(γ(L + µ) − 2) ≥ 0, and L − µ ≥ 0.
We note that for any γ such that 0 ≤ γ ≤ L2 , exactly2 one of the two previous
combinations of inequalities is valid (both multipliers and coefficients of thesquared norms are positive). In addition, the valid expression corresponds tothe maximum value between the two possible rates (1 − γµ)2 and (1 − γL)2 ,which concludes the proof.
Corollary 7.4. Let f ∈ Fµ,L (Rd ), and h ∈ F0,∞ (Rd ) and consider the composite convex optimization problem (7.1). The iterates of PGM with 0 ≤ γ ≤ L2
satisfy the following:()
2kxk − x∗ k
= ρ2k (γ).
max
2
d
kx
−
x
k
0
∗
f ∈ Fµ,L (R )h ∈ F0,∞ (Rd )x0 ∈ R d

Proof. Combine Theorem 7.3 with the quadratic lower bounds (QLB).

Before moving to the next convergence result, note that only a subset of theavailable inequalities were used in the previous proof. In fact, any composite
function F for which the f component satisfies ∀x ∈ Rd :1µ12hg∗ − gk , x∗ − xk i ≥ kgk − g∗ k +xk − x∗ − (gk − g∗ )L
1 − µ/LL

2, (7.7)(which is, sum of the two first inequalities used in the proof of Theorem 7.3, as
λ0 = λ1 ) will have a PGM converging with the same rate in terms of distance tooptimality despite being potentially outside of Fµ,L . As an example, consider
the following quadratic function fA (x) = 12 x> Ax with µI  A  LI (hence
2 Actually, both regimes are valid for γ =

2
.
L+µ

CHAPTER 7. PROXIMAL GRADIENT METHOD

172

f ∈ Fµ,L ). Therefore, (7.7) holds and hence:


1+

µ
1
(x∗ − xk )> A(x∗ − xk ) ≥ (xk − x∗ )> A> A(xk − x∗ )LL
+ µ(xk − x∗ )> (xk − x∗ ).

In short, note that this inequality also holds when instead 0  A  LI where x∗
is the projection of xk onto the set of optimal solutions (i.e., xk −x∗ ⊥ Null(A))
and µ > 0 is the smallest nonzero eigenvalue of A.
Also note that only a monotonicity condition on ∂h needs to be satisfied forkeeping the same convergence guarantees, as only the sum of the third andfourth inequalities is required to hold (λ2 = λ3 ):hsk+1 − s∗ , xk+1 − x∗ i ≥ 0,
Note that those sorts of relaxations were further exploited in [ZC15, NNG15](relaxation of the strong convexity requirement, with motivational examples).
We leave further investigations in that direction for future research.
Residual gradient norm. The next theorem is concerned with convergencein terms of residual gradient norm. Note that similar results can be obtained for
k+1
the norm of the (composite) gradient mapping (i.e., xk −x) instead3 , which isγused in some standard references on composite minimization [Nes04, Nes13].
Convergence in terms of residual gradient norm is in fact very natural, as it ismeasurable in practice, as opposed to the distance to optimality which requires
the knowledge of x∗ in order to be evaluated, or in terms of objective function
accuracy which it requires the knowledge (or a least a bound) on the true valueof F (x∗ ).
Theorem 7.5 (Residual gradient norm). Let f ∈ Fµ,L (Rd ), and h ∈ F0,∞ (Rd )and consider the composite convex optimization problem (7.1) and a feasible
starting point x0 ∈ Rd (i.e., x0 is such that F (x0 ) < ∞) such that there exists
s0 ∈ ∂h(x0 ). The iterates of PGM with 0 ≤ γ ≤ L2 satisfy:22k∇f (xk+1 ) + sk+1 k ≤ ρ2 (γ)k∇f (xk ) + sk k ,with sk ∈ ∂h(xk ) (any subgradient of h at xk ) and sk+1 ∈ ∂h(xk+1 ), thesubgradient of h at xk+1 used in the proximal operation (see Equation (7.2)).
Proof. We use the exact same reasoning as for Theorem 7.3: the notations
and inequalities are introduced in the previous section (Section 7.2.2), and
3 The difference between the gradient mapping and the residual gradient norm is simple,
but somewhat subtle. The gradient mapping measures k∇f (xk ) + sk+1 k, whereas RGN
measures k∇f (xk+1 ) + sk+1 k with sk+1 ∈ ∂h(xk+1 ) the subgradient used in the proximal
operation.

173

7.2. CONVERGENCE

the proof consists in summing the following interpolation inequalities after
multiplication with their respective coefficients. The main difference lies in
the choice of the inequalities to be combined; in this proof, we use conditions
between consecutive iterates, instead of using conditions between the current
iterates and the optimum:2

fk ≥ fk+1

1
kgk − gk+1 k+hgk+1 , xk − xk+1 i + 2L2µ+ 2(1−µ/L) xk − xk+1 − L1 (gk − gk+1 )fk+1 ≥ fk

1+hgk , xk+1 − xk i + 2L
kgk − gk+1 k2µ+ 2(1−µ/L) xk − xk+1 − L1 (gk − gk+1 )

: λ0

2

: λ1 ,

hk ≥ hk+1 + hsk+1 , xk − xk+1 i: λ2 ,

hk+1 ≥ hk + hsk , xk+1 − xk i

: λ3 .

We use the following multipliers:
λ0 = λ1 =2ρ(γ) ≥ 0,γ

λ2 = λ3 =2 2
ρ (γ) ≥ 0.
γ

After appropriate substitutions of xk+1 and s∗ , using xk+1 = xk − γ(gk + sk+1 )(Section 7.2.2, Condition (a)) and s∗ = −g∗ (Section 7.2.2, Condition (b)) , wenote that the previous weighted sum corresponds to a sum of squares in thetwo cases of interest (same two regimes as ρ(γ), see (RHO)).
2
 When 0 ≤ γ ≤ L+µ(i.e., when ρ(γ) = (1 − γµ)):
2

2

2

2

(1 − γµ) kgk + sk k ≥kgk+1 + sk+1 k + (1 − γµ) ksk − sk+1 k+

2

2 − γ(L + µ)2
kgk − gk+1 − µγ(gk + sk+1 )k ,γ(L − µ)2

≥kgk+1 + sk+1 k ,where the last inequality follows from(1 − γµ)2 ≥ 0, 2 − γ(L + µ) ≥ 0, and γ(L − µ) ≥ 0.

2
 When L+µ≤ γ ≤ L2 (i.e., when ρ(γ) = (Lγ − 1)):
2

2

2

2

2

(1 − γL) kgk + sk k ≥kgk+1 + sk+1 k + (1 − γL) ksk − sk+1 k+γ(L + µ) − 2
2
kgk − gk+1 − Lγ(gk + sk+1 )k ,γ(L − µ)2

≥kgk+1 + sk+1 k ,CHAPTER 7. PROXIMAL GRADIENT METHOD

174

and the last inequality follows from(1 − γL)2 ≥ 0, γ(L + µ) − 2 ≥ 0, and γ(L − µ) ≥ 0.
We conclude the proof in the same way as for the distance to optimality: sincefor any value of γ such that 0 ≤ γ ≤ L2 , there is always one of the two previous
combinations of inequalities that is valid (both multipliers and coefficients ofthe squared norms are positive), and since the valid one corresponds to the
maximum value between the two possible rates (1 − γµ)2 and (1 − γL)2 , thedesired statement is proved.

Corollary 7.6. Let f ∈ Fµ,L (Rd ), and h ∈ F0,∞ (Rd ) and consider the composite convex optimization problem (7.1) and a feasible starting point x0 ∈ Rd
(i.e., x0 is such that F (x0 ) < ∞). The iterates of PGM with 0 ≤ γ ≤ L2 satisfy:)(2k∇f (xk ) + sk k
= ρ2k (γ).
max
2
d
k∇f
(x
)
+
s
k
0
0
f ∈ Fµ,L (R )h ∈ F0,∞ (Rd )x0 ∈ Rd , s0 ∈ ∂h(x0 )Proof. Combine Theorem 7.5 with the quadratic lower bounds (QLB).
Interestingly, the inequalities used in this proof do not involve the optimal
point, and only use the information available at the consecutive iterates. In
addition, note that as for the convergence in terms of distance to optimality,
λ0 = λ1 tells us that the result hold under the following weaker assumption:1
2
hgk+1 − gk , xk+1 − xk i ≥ kgk − gk+1 k
L
2
µ
1
+
xk − xk+1 − (gk − gk+1 ) .
1 − µ/LL
A consequence of this inequality is that one can benefit from using the locally
better strong convexity and smoothness parameters (i.e., better constants µand L that satisfy this inequality for two consecutive iterates) instead of the
global ones, in order to improve the convergence rate. Also, it is possible toexploit this in order to make online estimations of the strong convexity andsmoothness parameters µ and L (we leave this for further research).
Objective function accuracy. Finally, we consider convergence in terms ofobjective function accuracy. The proof of this convergence rate is much moretedious than the previous ones, and seems to require more assumptions (i.e.,

175

7.2. CONVERGENCE

more inequalities appear to be needed — of course it may be that we just did
not isolate the simplest proof).
Theorem 7.7 (Objective function accuracy). Let f ∈ Fµ,L (Rd ), and h ∈F0,∞ (Rd ) and consider the composite convex optimization problem (7.1) and afeasible starting point x0 ∈ Rd (i.e., x0 is such that F (x0 ) < ∞). The iteratesof PGM with 0 ≤ γ ≤ L2 satisfy the following:F (xk+1 ) − F∗ ≤ max (1 − Lγ)2 , (1 − µγ)2 (F (xk ) − F∗ ) .

Proof. We combine the following interpolation after multiplication with theirrespective coefficients:
fk ≥ fk+1
f∗ ≥ fkf∗ ≥ fk+1

2

1+hgk+1 , xk − xk+1 i + 2L
kgk − gk+1 k2µ+ 2(1−µ/L) xk − xk+1 − L1 (gk − gk+1 ): λ0 ,

2

1+hgk , x∗ − xk i + 2L
kgk − g∗ k2µ+ 2(1−µ/L) xk − x∗ − L1 (gk − g∗ ): λ1 ,
2

1+hgk+1 , x∗ − xk+1 i + 2Lkg∗ − gk+1 k2µ+ 2(1−µ/L) x∗ − xk+1 − L1 (g∗ − gk+1 ): λ2 ,

hk ≥ hk+1 + hsk+1 , xk − xk+1 i: λ3 ,h∗ ≥ hk+1 + hsk+1 , x∗ − xk+1 i: λ4 .

We use the following multipliers:
λ0 = ρ(γ), λ1 = (1 − ρ(γ))ρ(γ), λ2 = 1 − ρ(γ), λ3 = ρ2 (γ), λ4 = 1 − ρ2 (γ).
Appropriate substitutions of xk+1 and s∗ using xk+1 = xk − γ(gk + sk+1 )(Section 7.2.2, Condition (a)) and s∗ = −g∗ (Section 7.2.2, Condition (b)) ,we obtain that the weighted sum of inequalities is equivalent to the following
expressions.
2 When 0 ≤ γ ≤ L+µ(i.e., when ρ(γ) = (1 − γµ)):(1 − γµ)2 (F (xk ) − F∗ )(2 − γµ)βk(1 − γµ)gk − gk+1 + µγg∗ k2
2αγLµ2 (2 − γµ)2L − 2µ + γµ2
gk + gk+1
g∗+(xk − x∗ ) −sk+1 −+
2(L − µ)Lµ(2 − γµ)µ(2 − γµ)L
≥ F (xk+1 ) − F∗ ++2(µγ − 1)Lβ
(L − µ)(2 − γµ)2
αγµ
Lβ
s1 +
gk +
gk+1 +g∗
2L(L − µ)(2 − γµ)
α
α
α

2

,

≥ F (xk+1 ) − F∗ , (because of the signs of the coefficients, see discussion below),CHAPTER 7. PROXIMAL GRADIENT METHOD

176

with α = −(γ 2 L2 µ + 2L(−2 + γµ) + µ(−2 + γµ)2 ) and β = (2 − γ(L + µ)).
2
. Indeed, byNote that α is positive for 0 ≤ µ < L and 0 ≤ γ ≤ µ+L
denoting−α = p(γ) = γ 2 L2 µ + 2L(−2 + γµ) + µ(−2 + γµ)2
(positive definite quadratic function), we have p(0) ≤ 0 and p(γc ) =2(L−µ)− 4L(L+µ)≤ 0.
22≤ γ ≤ L2 (i.e., when ρ(γ) = (Lγ − 1)):
 When L+µ(1 − γL)2 (F (xk ) − F∗ )≥ F (xk+1 ) − F∗ +(2 − γL)βk(1 − γL)gk − gk+1 + γLg∗ k2
2γα
2

+

γL2 µ(2 − γL)
sk+1
1 − γL − γµgk+1g∗(xk − x∗ ) −+gk −+2(L − µ)
µ
γLµ
γLµ
L

+

(2 − γL)(L − µ)µ(γL − 1)Lβ
γα
Lβ
gk +
gk+1 +g∗sk+1 +
2µ(L − µ)
γα
γα
α

2

,

≥ F (xk+1 ) − F∗ , (because of the signs of the coefficients, see discussion below),with α = (−2L2 − 2µ2 + 2Lµ + γL3 + γLµ2 ) and β = (γ(L + µ) − 2).
Again, α is nonnegative as α = p(γ) is an increasing linear function which2
is nonnegative in the region of interest γ ≥ L+µ. Indeed, on can check2
that by evaluating p(.) at γc = L+µ :p(γc ) = 2µ2



L−µ
L+µ


≥ 0.

We conclude the proof in the same way as before: among the two cases, thevalid one corresponds to the maximum value between the two possible rates(1 − γµ)2 and (1 − γL)2 .
Corollary 7.8. Let f ∈ Fµ,L (Rd ), and h ∈ F0,∞ (Rd ) and consider the composite convex optimization problem (7.1) and a feasible starting point x0 ∈ Rd
(i.e., x0 is such that F (x0 ) < ∞). The iterates of PGM with 0 ≤ γ ≤ L2 satisfy
the following:F (xk ) − F (x∗ )max= ρ2k (γ).
F (x0 ) − F (x∗ )f ∈ Fµ,L (Rd )h ∈ F0,∞ (Rd )x 0 ∈ Rd

Proof. Combine Theorem 7.7 with the quadratic lower bounds (QLB).

177

7.3. MIXED PERFORMANCE MEASURESProof of Theorem 7.2. The theorem is the combination of Corollary 7.4,
Corollary 7.6 and Corollary 7.8.

7.3

Mixed performance measures

In this section, we summarize the global convergence results we obtained on
PGM. In order to do that, we first show that Theorem 7.2 can be used to obtaintight bounds on mixed performance measures.
Proposition 7.9. Consider the composite convex optimization problem (7.1)
and a feasible point x ∈ Rd (i.e. x is such F (x) < ∞) such that s ∈ ∂h(x).
The following inequalities are satisfied:22(i) kx − x∗ k ≤ µ12 k∇f (x) + sk ,21(ii) F (x) − F (x∗ ) ≤ 2µk∇f (x) + sk .
2(iii) kx − x∗ k ≤ µ2 (F (x) − F (x∗ )),Proof. (i) By strong convexity of F , we have22k∇f (x) + s − ∇f (x∗ ) − s∗ k ≥ µ2 kx − x∗ k ,with s∗ ∈ ∂h(x∗ ) such that ∇f (x∗ ) + s∗ = 0. Therefore2kx − x∗ k ≤12
k∇f (x) + sk .
µ2(ii) By strong convexity of F (and feasibility of x), we have
F (x) − F∗ ≤11
2
2
k∇f (x) + s − ∇f (x∗ ) − s∗ k =k∇f (x) + sk k .
2µ2µ(iii) Again, by strong convexity of F , we have:
F (x) ≥ F (x∗ ) + h∇f (x∗ ) + s∗ , x − x∗ i +µ2kx − x∗ k
2with s∗ ∈ ∂h(x∗ ) such that ∇f (x∗ ) + s∗ = 0, we obtain the statement.
Theorem 7.10. Consider the composite convex optimization problem (7.1)
and a feasible starting point x0 ∈ Rd (i.e. x0 is such F (x0 ) < ∞) such thats0 ∈ ∂h(x0 ). The iterates of PGM satisfy the following inequalities:
2

2k

2

(i) kxk − x∗ k ≤ ρ µ2(γ) k∇f (x0 ) + s0 k ,2k2(ii) F (xk ) − F (x∗ ) ≤ ρ 2µ(γ) k∇f (x0 ) + s0 k ,CHAPTER 7. PROXIMAL GRADIENT METHOD

2

178

2k

(iii) kxk − x∗ k ≤ 2ρ µ(γ) (F (x0 ) − F (x∗ )).
Proof. Combine results of Theorem 7.2 with those of Proposition 7.9.
Note that the global bounds provided by the previous theorem are exact for the2step sizes 0 ≤ γ ≤ L+µ(thus also γ = L1 ), as provided by the quadratic functionfµ (x) from Section 7.2.1 (i.e., PGM applied on F = fµ satisfies (i),(ii) and (iii)with equalities). Note that the guarantees of Theorem 7.10 do not achieve
exactness for larger step sizes. The exact global convergence guarantees are
summarized in Table 7.1.
Initialization2

kx0 − x∗ k

2

F (x0 ) − F∗k∇f (x0 ) + s0 k

2kxk − x∗ k ≤ρ2k kx0 − x∗ k

22 2k
µ ρ (F (x0 ) − F∗ )F (xk ) − F∗ ≤?

ρ2k (F (x0 ) − F∗ )2
1 2k
µ2 ρ k∇f (x0 ) + s0 k
2
1 2k
2µ ρ k∇f (x0 ) + s0 k

˜ (xk )||2 ≤
||∇f

?

?

ρ2k k∇f (x0 ) + s0 k

2

Table 7.1: Summary of the global convergence guarantees proposed by Theo2
rem 7.2 (exact) and Theorem 7.10 (exact for 0 ≤ γ ≤ L+µ). The corresponding
results for the case of f being quadratic and h = 0 are presented in the workof Nemirovski [Nem92]. The stars denote the combinations of performancemeasures for which no analytical global and exact convergence guarantees wereobtained yet.

7.4

Conclusion

Tight convergence rates for PGM. The main contribution of this work
is to close the gap between lower and upper complexity bounds for PGM insmooth strongly convex optimization. We obtained exact global linear convergence rates for measuring progress in terms of different measures of optimality.
The proof methodology used in order to prove the main results allows a clear
and transparent use of the assumptions of the theorems. As an example, we observed that strong convexity was only required between certain pairs of points.
In addition, Theorem 7.2 may be used to extend the recent results of [dKGT16,
Theorem 1.2] on the exact worst-case complexity of the gradient descent withexact line search. Furthermore, as in the unconstrained case (h(x) = 0), thisresult cannot be improved in general, as it is attained by a two-dimensional
quadratic example [dKGT16, Example 1.3].

179

7.4. CONCLUSION

Proximal gradient method with exact line search
Input: x0 ∈ Rd , f ∈ Fµ,L (Rd ), h ∈ F0,∞ (Rd ), 0 ≤ γ ≤ L2 .
For k = 0 : N − 1
γ = argmin F [pγh (xk − γ∇f (xk ))]γ∈R

xk+1 = pγh (xk − γ∇f (xk ))Corollary 7.11. Let f ∈ Fµ,L (Rd ), and h ∈ F0,∞ (Rd ) and consider the composite convex optimization problem (7.1) and a feasible starting point x0 ∈ Rd
(i.e., x0 is such that F (x0 ) < ∞). The iterates of PGM with exact line searchsatisfy the following inequality:F (xk+1 ) − F∗ ≤

L−µ
L−µ

2
(F (xk ) − F∗ ) .

2
Proof. This is exactly the result of Theorem 7.7 with γ = L+µ. The corresponding result is an upper bound on the worst-case of PGM with exact line
search, which turns out to be tight on the Example 6.2 (quadratic example
in [Ber99, Example on p.69] or [dKGT16, Example 1.3]).

Further work. Tight convergence results are still open for a variety of firstorder methods and different convergence measures, as for example for accelerated schemes [Nes04], for inexact methods [DGN14, SLRB11] and coordinate
descent schemes [Nes12a]. Obtaining such tight convergence results opens thedoor for a better use of gradient-schemes as primitive operations in more complicated algorithms, but also for designing optimized first-order methods (suchresearch directions are carried out among others in [DT14, KF16d] for thesmooth unconstrained convex case and in [LRP16] in the strongly convex casewith the presence of disturbing noise).

CHAPTER 7. PROXIMAL GRADIENT METHOD

180

Part III

Conclusion

Chapter 8

Further Developments in
Performance Estimation
In this chapter, we broaden the applicability range of performance estimation. We do not intend to give a complete overview of the literature for thecorresponding applications, but rather provide a variety of ideas for further
developments. In addition we use those examples to emphasize the difficulties
and limitations inherent to performance estimation and which require further
investigation. Those can be summarized in three points
 first, finding interpolation conditions the problems of interest is mostoften far from being trivial. We provide examples for which no tractableinterpolation conditions were found. Solving the corresponding (relaxed)
performance estimation problems therefore results in upper bounds.
 Second, finding tractable formulations of the algorithms compatible withthe representation of the class of functions may be involved, and also
generally results in relaxations and upper bounds.
 Finally, the computational cost for solving the corresponding semidefinite programs may become prohibitive, even in apparently very simple
situations.
Note that we are only interested in solving the performance estimation problems to global optimality (or, at least, to obtain guaranteed upper bounds),
hence our interest in convex formulations. However, one should note that any
nonconvex formulation that can be solved to global optimality (or for whichmeaningful upper bounds can be obtained) can also be used.
The chapter is organized as follows: in Section 8.1, we illustrate that standard algorithms for solving monotone
inclusions very naturally fit into the performance estimation framework.
183

CHAPTER 8. FURTHER DEVELOPMENTS

184

 Then, we provide other examples of applications of the performance estimation framework for decentralized, randomized, nonconvex and noisy
optimization problems in Section 8.2. In addition, we introduce several
open problems, including the treatment of second-order methods and of
non-Euclidean geometries.
 In Section 8.3, we finally conclude the chapter by emphasizing the main
advantages, difficulties and aspects that require further attention for abroader applicability of the performance estimation framework in various
practical settings.

8.1

Monotone operators and splitting methods

The goal of this section is to convince the reader that the performance estimation framework naturally applies to monotone operators. We illustrate ourpoint on the forward-backward and Douglas-Rachford splitting schemes, after reviewing some of the basic underlying concepts of the field. For a very
nice introduction to monotone operators, we refer the reader to the thesis of
Eckstein [Eck89] and to the seminal references [RW98, BC11].
Monotone operators are a natural extension to subdifferentials (other examples
can be found e.g., in the tutorial [RB16]). They are more and more used in
the mathematical optimization community, in part because of their naturalability for developing distributed algorithms. As an example, the very popular
alternating direction method of multipliers (ADMM) [BPC+ 11] can be seen
as a dual version of the Douglas-Rachford splitting scheme (see e.g., [Gab83,
Eck89]).
d

Notations 8.1. We denote by 2R the set of all subsets of Rd . We use thednotation T : Rd → 2R for meaning that T is a set-valued operator from Rd to
Rd . That is, the operator T maps every x ∈ Rd to a set T x ⊆ Rd . A convenient
way to characterize T is via its graph:graph T = (x, z) ∈ Rd × Rd | z ∈ T x .
dDefinition 8.2. An operator T : Rd → 2R is monotone if ∀(x1 , z1 ), (x2 , z2 ) ∈graph T we havehz1 − z2 , x1 − x2 i ≥ 0.
d

Definition 8.3. A monotone operator T : Rd → 2R is maximally monotone
if there is no monotone operator S such that graph T ( graph S.
Note that it is well known that requiring an operator T to be monotone isonly a necessary condition for the existence of a convex function f satisfying∂f = T . In order to obtain a sufficient condition, one needs to consider cylic

1858.1. MONOTONE OPERATORS AND SPLITTING METHODS

monotonicity conditions1 .

Zeros of maximally monotone operators. The general problem of interest in this section is to solve the following monotone inclusion:find x ∈ Rd such that 0 ∈ T (x),dwith T : Rd → 2R some maximally monotone set-valued operator. The common approach in the monotone operator framework is to reformulate this asan equivalent fixed-point problem. One way to proceed is to associate T with
its resolvent JλT : Rd → Rd
JλT = (I + λT )−1 ,where I is the identity operator. It can be shown (see e.g., [Eck89, Proposition3.9]) that for any maximally monotone operator T , ∀λ > 0 and ∀x ∈ Rd :JλT (x) = {x} ⇔ 0 ∈ T (x).
The resolvent JλT has numerous nice properties due to the maximal monotonicity of T ; among others, it is (firmly) nonexpansive (this motivates using fixedpoint iterations, see e.g., [Eck89, Definition 3.13]) and satisfies dom JλT = Rd
(that is, ∀x ∈ Rd we have JλT (x) 6= ∅, which is very important for practicalcomputations). Also, note that the fixed-point iteration
xk+1 = JλT (xk ) ⇔ xk+1 + λT (xk+1 ) = xk
is usually referred to as the proximal point algorithm; which we already studiedin Section 5.3.1 when T = ∂f for some f ∈ F0,∞ .

Interpolation of maximally monotone operators. In order to use theperformance estimation framework with tightness guarantees, we consider theinterpolation problem for maximally monotone operators. That is, given an
index set and set of couples S = {(xi , ti )}i∈I with xi , ti ∈ Rd ∀i ∈ I, we wantto find conditions guaranteeing the existence of a maximally monotone operator
T such thatti ∈ T (xi ) ∀i ∈ I.
For that purpose, we define the operator TS : Rd → 2Rd

TS (x) = {g ∈ E∗ | (x, g) ∈ S} .
1 See the related convex integration problem from Section 3.4, and the references [RW98,
Theorem 12.15] and [BC11, Theorem 22.14].

CHAPTER 8. FURTHER DEVELOPMENTS

186

Note that if the set S satisfies discrete monotonicity conditions, i.e., if
hti − tj , xi − xj i ≥ 0 ∀i, j ∈ I,

(8.1)

then the operator TS is monotone. In addition, the following theorem implies
the existence of a maximally monotone extension of TS .
d

Theorem 8.4. [BC11, Theorem 20.21] Let T : Rd → 2R be monotone. Then
there exists a maximally monotone extensions of T ; i.e., a maximally monotonedoperator T̃ : Rd → 2R such that graph T ⊆ graph T̃ .
Note that this theorem is proved using Zorn’s lemma (equivalent to the axiomof choice), which is actually a key element in monotone operator theory (seee.g., discussion in [Eck89, Section 2.1]). Therefore, tightness of the results formonotone operators relies on the axiom of choice — however, we suspect there
is a constructive way not relying on Zorn’s lemma as the set S is finite.
Before going into the next section, let us mention the existence of strong monotonicity and cocoercivity conditions. Those conditions are respectively corresponding to strong convexity and smoothness in the case of proper closed convex functions. The definitions are the following: ∀x1 , x2 ∈ Rd , t1 ∈ T x1 , t2 ∈T x2 we have12
kt1 − t2 k ,L2
ht1 − t2 , x1 − x2 i ≥ µkx1 − x2 k ,ht1 − t2 , x1 − x2 i ≥(1/L-cocoercivity),(µ-strong monotonicity).

As in the case of convex functions, one can easily develop interpolation conditions for dealing with 1/L-cocoercivity and µ-strong monotonicity. For doing
that, one possibility is exploit the following elements:
 A maximally monotone operator T (x) is µ-strongly monotone if and only
if (T − µI)(x) is maximally monotone.
 A maximally monotone operator T (x) is 1/L-cocoercive if and only if
T −1 (x) is L-strongly monotone and maximal.
Note that this does not provide a way to interpolate an operator being bothstrongly monotone and cocoercive, which we leave as an open question.
Splitting methods. For the following examples, let us consider the following
monotone inclusionfind x ∈ Rd such that 0 ∈ A(x) + B(x),d(MI)

with A, B : Rd → 2R being a maximally monotone operators. In addition,
we assume A = ∂f for some f ∈ Fµ,L and we use the interpolation conditions
from Theorem 3.8 for the set {(xi , ai , fi )} in the following examples.

187

8.1. MONOTONE OPERATORS AND SPLITTING METHODS

Forward-backward splitting. The forward-backward splitting (FBS) scheme
for solving (MI) performs the following fixed-point iterations:
xk+1 = JλB [(I − λA)xk ].
One can note that the performance estimation problems corresponding to thisscheme have the same forms as those obtained from the proximal gradientmethods from Chapter 7 as long as no function values are involved (as B is a
general monotone operator). Therefore, the results in terms of gradient norms
and distances to the solution (Theorem 7.3 and Theorem 7.5) can be adapted
to FBS:2

2

kxk+1 − x∗ k ≤ max{(1 − λµ)2 , (1 − λL)2 }kxk − x∗ k ,2

2kA(xk+1 ) + bk+1 k ≤ max{(1 − λµ)2 , (1 − λL)2 }kA(xk ) + bk k ,with x∗ the (unique) solution to the monotone inclusion 0 ∈ A(x∗ ) + B(x∗ ),some bk ∈ B(xk ) and bk+1 ∈ B(xk+1 ) the (unique) vector used in the iteration
xk+1 + λbk+1 = xk − λA(xk ).

Douglas-Rachford splitting. For solving (MI), the Douglas-Rachford splitting2 (DRS) uses the alternative fixed-point iterations:
wk+1 ∈ JλA [(2JλB − I)wk ] + [I − JλB ](wk ),which converges to some w∗ such that w∗ = (2JλA − I)(2JλB − I)w∗ . In
order to obtain the corresponding x∗ such that 0 ∈ A(x∗ ) + B(x∗ ), we use theequivalence:
0 ∈ A(x∗ ) + B(x∗ ) ⇔ x∗ = JλB w∗ .
Note that DRS can be written in an expanded form; find xk+1 , B(xk+1 ), yk+1
and A(yk+1 ) such that:xk+1 = wk − λB(xk+1 ),yk+1 = 2xk+1 − wk − λA(yk+1 ),wk+1 = yk+1 − xk+1 + wk .
With little effort, this method can be written using the (FSLFOM) format from
Chapter 5. As an example, the framework easily allows obtaining the following
result (proofs similar to that of Chapter 6 and Chapter 7):no1
λL
kwk+1 − w∗ k ≤ max 1+λµ
, λL+1kwk − w∗ k (Note that this observation reproduces the recent results of [GB16, Theorem 2]).
2 See [LM79, Algorithm II] for the original presentation.

CHAPTER 8. FURTHER DEVELOPMENTS

188

We leave the study of other settings with and without linear convergence forfurther work (as for example tightening the results of [Gis15, Theorem 1] forwhen A is strongly monotone and B is cocoercive).

8.2

Further developments

In this section, we further illustrate potential uses of the performance estimation framework. We do not provide technical details but rather focus onconvincing the reader of the broader applicability of the framework.
Decentralized gradient methods. Consider the following composite optimization problem
( n
)
X
min
fi (xi ) s.t. xi = xj ∀i, j = 1, . . . , n ,xi ∈Rdi=1with fi ∈ Fµi ,Li (Rd ). When the computations have to be performed amongn computers, each one storing information about a single function fi , one
possibility is to use a consensus-based (sub)gradient method (see e.g., [JKJJ08,
NOP10])
 k+1  k˜ 1 (xk )x1 − γ1k ∇f
x1
1
 . 


..
 ..  = W 
,
.
k+1
k
k
k
˜ n (x )xn
x − γ ∇fnnnwith W being some known doubly stochastic matrix. This model is a particularinstance of (FSLFOM), and can therefore be modelled in the performance
estimation framework with tightness guarantees. Note however that the size ofthe problems grows linearly in both the number of components n and in thenumber of iterations N .
Randomized methods. Let us consider two kinds of randomized methods:
stochastic gradient and block-coordinate descent. In those cases, the performance estimation approach can be used to search for functions with the worst
expected convergence results. Indeed, one can apply the PEP framework to the study of stochastic gradient descent on the n-term objective function
min

x∈Rd

n
X

fi (x),

i=1

with fi ∈ Fµi ,Li (see e.g., [Ber10, SSBD14, Kiw04]), with the corresponding update xk+1 = xk − γk,i ∇fi (xk ) with probability pi .

189

8.2. FURTHER DEVELOPMENTS

 The application of randomized block-coordinate descent on the problem
min f (x(1) , . . . , x(n) ),x(i) ∈Rdi

with f ∈ F0,∞ and being Li -smooth separately on every block of coordinates x(i) (see e.g., [RT14, FR15, Nes12a]), and the corresponding
iteration with probability pi :
(
(j)
(j)
xk+1 = xk
for i 6= j,
(i)
(i)
(i)
xk+1 = xk − γk ∇i f (xk ), otherwise,∂f(the standard notation ∇i f (x) is used for meaning ∂x(i) (x)).

In both cases, the algorithm perform updates on one of the components (randomly selected) at each iteration — see Figure 8.1 for n = 2. In order toevaluate the expectation of some convergence measure after N iterations, we(a priori) have to average over all possible sequences of choices, which results
in an exponential number of combinations: (nN +1 − 1)/(n − 1); this renders
the corresponding SDP intractable even for relatively small values of n and N .
Also note that in the case of block coordinate descent methods, it is usual
to assume that the convex function to be minimized satisfies a smoothness
condition separately on every block of coordinates. It is not clear how tointerpolate on this class of convex and block-smooth functions. For example,
the discrete version of the conditions
f (x) ≥ f (y) + h∇f (y), x − yi +12
k∇i f (x) − ∇i f (y)k2Li

are used in [SL16, Lemma 1.1] for studying cyclic block coordinate descentschemes; however, it appears that they do not correspond to interpolation
conditions.

Nonconvexity. As outlined in Chapter 3, it is also possible to use the performance estimation framework for analyzing first-order algorithms for nonconvex
functions. For example, one can use the nonconvex smooth interpolation result
from Theorem 3.21 for studying algorithms tailored formin f (x),x∈Rdwhen f is a L-smooth nonconvex function. However, this is generally difficult
to combine with the expression of the algorithm in consideration. We illustrate
the potential difficulties on the following example.
First, for the steepest descent with exact line-search, we only managed to re-CHAPTER 8. FURTHER DEVELOPMENTS

190

x0
•
s1 = {1}x1s2 = {1, 1}x2•

s1 = {2}••s2 = {1, 2}

s2 = {2, 1}

x2

x2

•

•

x1
s2 = {2, 2}•

x2

Figure 8.1: Possible sequences obtained from a 2-terms objective function (forstochastic gradient descent) or from a 2-blocks coordinate descent scheme. The
list si denotes the ordered list of updated components done so far.
produce the following well known result (see e.g., [Nem99, Proposition 3.3.1]):


2
2
2
min k∇f (xi )kE∗ ≤ (f (x0 ) − f (xN )) ≤ (f (x0 ) − f (x∗ )) .
0≤i≤N
N
N
In that context, we identify two main difficulties:- the intermediate optimality conditions can not be expressed easily (zerogradient on a line is not sufficient to guarantee optimality),- expressing the search direction is a nonconvex constraint (in the smooth
strongly convex case, however it appeared that using a relaxation was
sufficient — see Chapter 6).
Note that this is consistent with the fact exact line search can hardly be used inpractice without additional assumptions on the function f , as we can generally
only have the guarantee of finding critical points.
Nevertheless, we can still consider fixed-step gradient methods (we use step
size 1/L) with tightness guarantees — even if it is much less practical in thegeneral nonconvex setting. In that situation, it is possible to obtain the following corresponding convergence result for minimizing an L-smooth nonconvex
function with no constraint (using smooth nonconvex interpolation conditions
as in Theorem 3.21):


4
4
2
min k∇f (xi )kE∗ ≤(f (x0 ) − f (xN )) ≤(f (x0 ) − f (x∗ )) .
0≤i≤N
3N
3N

191

8.2. FURTHER DEVELOPMENTS

Inexact computations. In Section 5.2.3, we briefly talked about handlinginexact methods using performance estimation. As underlined in that section,
not all inexactness models can be handled directly with tightness guarantees.
As example, the basic noise models as used in Theorem 5.15, or the model proposed by d’Aspremont [d’A08] can easily be used in the framework. However,
inexact (δ, L)-oracles developed by Devolder, Glineur and Nesterov [DGN14]
do not appear to have (as such) corresponding interpolation conditions (so only
upper bounds are usually found).
Note that all the previous results may easily be studied (at least numerically) in
the presence of such noise models (e.g., splitting methods with inexact oracles).
Further ideas. For further development, we believe there are still a lot ofdifferent possibilities for exploring numerical optimization schemes in a tightway using the performance estimation approach. Among others, the previous
chapters did not mention the following possible further directions.
 Structure is crucial for obtaining efficient algorithms (see e.g., [Nes08]).
In this work, we mainly considered generic functions on which it is easy
to perform some operations (oracles); however, considering for example
more specifically linear or quadratic functions as parts of the objectivefunction may provide more appropriate results when dealing with practical problems for which we specifically know the structure.
 As previously underlined, choosing the appropriate problem formulation/structure can play a crucial role in our ability to solve it. In particular, we restricted ourselves to Euclidean norms for the whole work,whereas it may be more appropriate to adapt the choice of the setting(e.g., the norm in which we require smoothness and strong convexity) tothe domain. For example, this is the underlying idea behind the Mirror Descent algorithm [BT03, BTN01] (note that the PEP framework
easily allows dealing with Bregman divergences, but as far as we know,
cannot handle strong convexity and/or smoothness with respect to nonEuclidean norms other than by using their equivalence with Euclideanones).
 In addition to the previous point, we suggested the use of PEP for second
order methods in Chapter 5. However, we did not study them so far— we believe this could be done using the standard idea of measuring
progress in terms of the local norm induced by the Hessian (which isamong the motivations for using generic Euclidean norms in most of thework). In addition, we believe that attention should be given to theanalysis of quasi-Newton-type methods (see e.g., [NW06]) using the PEP
framework.

CHAPTER 8. FURTHER DEVELOPMENTS

8.3

192

Conclusion

Before going into the final concluding chapter, let us summarize what welearned. First, there is still room for using the performance estimation framework to improve, or develop the analyses of a lot of practical optimization
schemes, including currently very popular splitting methods.
Second, the performance estimation framework provide a generic methodology
for analyzing optimization schemes in a tight way. However, the requirementsfor obtaining a tight analysis are generally difficult to satisfy (i.e., obtain appropriate interpolation conditions, formulate the algorithm in a tractable way).
Nevertheless, meaningful relaxations of PEPs may in general still provide interesting convergence results — we can generally at least match the previous
known results.
Finally, desired developments include the possibility of handling second-orderand quasi-Newton methods, and to handle non-Euclidean geometries. Thosepotential developments are apparently of very different natures when comparedto the methods studied using the PEP framework so far, and may therefore
require a significant amount of work.

Chapter 9

Research Outcomes andPerspectives
Research outcomesContributions to worst-case analyses. The main contribution of thiswork was the development of the performance estimation framework, whose
aim is to automate the generation of worst-case guarantees for a family of optimization algorithms. The core underlying idea is to use (convex) optimization
to perform worst-case analyses of optimization schemes. The fundamentals are
summarized in the following points.
 This framework uses optimization software to perform worst-case analyses. The idea is to generate problem instances on which the algorithm
under consideration behaves as bad as possible. The worst-case computation problem is itself an optimization problem whose variables are an
objective function and a domain. Hence, the worst-case computation isan infinite-dimensional optimization problem.
 Under the assumption that the algorithm evaluate the function and its
gradient at a finite set of points (black-box/oracle assumption), the worstcase computation (or performance estimation) problem can be formulated
in a finite way using an appropriate discretization.
In order to render this formulation tractable (more precisely: convex),
we use a standard Gram-matrix trick — a very standard tool for approximating solutions to NP-hard problems (see e.g., the seminal [GW95]).
Note that the Gram-matrix trick is usually used to perform relaxations,which in our case turned out to be very advantageous: it renders thesolutions independent of the dimension of the initial decision space.
193

CHAPTER 9. RESEARCH OUTCOMES AND PERSPECTIVES

194

 Primal solutions to the performance estimation problems correspond tolower bounds (i.e., actual instances of optimization problems on which
the algorithm behaves as badly as possible).
 Dual solutions to the performance estimation problems correspond to
proofs, which can be converted into linear combinations of valid inequalities (i.e., a certification that the primal bound is optimal).

Advantages of the approach.
advantages of the approach:Let us now quickly summarize the main it helps designing analytical proofs (both for lower and upper bounds)
and to develop the underlying intuitions.
 It allows to very easily test new assumptions, by just adding the corresponding constraints to the performance estimation problems.
 It allows designing new methods.
Also, note that the convex interpolation framework has the huge advantage
of providing sufficient requirements for guaranteeing the existence of (tight)
convergence proofs. That is, given an algorithm, a class of functions, a performance measure and an initial condition that can be expressed in the PEP
framework, we know it is possible to derive tight convergence proofs with noother (in)equalities characterizing the functions than the interpolation ones.

Limitations of the approach. However, any user of the performance estimation approach should be aware of the following difficulties and limitations.
 The size of corresponding semidefinite program grows with the numberof iterations to be analyzed. Therefore, performance estimation problems
can only be solved for limited numbers of iterations. As an example, inthe case of first-order methods for smooth unconstrained minimization,
we were not able to perform the numerical worst-case computations formore than around 150 iterations on a simple desktop computer.
The alternative approach of Lessard, Recht and Packard [LRP16] alleviates this difficulty in the case of linearly-converging algorithms, by using
Lyapunov stability theory coupled with relaxations and time-invariant
algorithms, at the cost of losing tightness guarantees (see discussion in
Section 1.3.2).
 The performance estimation approach is intrinsically conservative whenit comes to comparing with practical performances. It is due to the fact
we perform worst-case analysis, and especially to the fact we perform
it on an iteration-per-iteration basis, which means that there may be
no function achieving the worst-case for all number of iterations (so the195

actual convergence rates can only be strictly better). Of course, this is alimitation of most standard approaches to worst-case analyses.
Interestingly, this drawback is also partially alleviated in recent works(see e.g., [AP15]). Roughly speaking, we approached the convergence
rate by trying to answer the question what can we exactly guarantee after N iterations ? (no matter the function, as long as it is within somepredetermined class), whereas an alternative approach may ask the following: given a function, how does the convergence measure improve fromiteration to iteration? This question is approached asymptotically by Attouch and Peypouquet in [AP15]. In the case of the minimization of asmooth convex function f ∈ F0,L , the result of Attouch and Peypouquetin [AP15, Theorem 1] can be stated as follows: the iterates of a variantof the fast gradient method satisfy
lim N 2 (f (xN ) − f (x∗ )) = 0,

N →∞which shows that the asymptotic rate of convergence of the method isstrictly better than the well known O(N −2 ). As far as we know, thiskind of results cannot be obtained by standard worst-case approaches as
the lower bound proposed in [Nes04, Theorem 2.1.6] is O(N −2 ).
Another potential approach for going beyond worst-case analysis is thecelebrated smoothed analysis technique [ST04]. The underlying idea is
to study the stability of the problems on which the algorithm of interest achieves its worst-case. We are not aware of applications of thismethod for worst-case analyses of first-order methods (and we have noidea whether it is relevant to do it), but automated worst-case analyses
may be useful for using those techniques (by carefully choosing appropriate types of perturbations).
 The analyses obtained by the performance estimation approach are very
sensitive to the initial knowledge/assumptions and to choice of the convergence measure. In practice, we generally know more than what is
usually assumed (e.g., on the distance to optimality kx0 − x∗ k), and weshould therefore combine different sorts of initial assumptions in order
to better link theoretical convergence results with practical observations(see e.g., Table 7.1).
Although this point may seem quite theoretical, the choice of the setting
is critical in the development of optimized methods (see [DT14, KF16a,
KF16b, KF16c, KF16d]), which are tailored for very specific set of convergence measure and initial condition.

CHAPTER 9. RESEARCH OUTCOMES AND PERSPECTIVES

196

Convex interpolation. Other than performance estimation, a great deal ofthis work is devoted to convex interpolation (see Chapter 3). Interpolation
turned out to play a crucial role in the development of the performance estimation framework. However, we did not explore further the possibilities for using
it in other contexts.
Regression schemes represents a major part of optimization problems arisingin practice. Under some circumstances, the approximating function is required
to be convex (e.g., in economy, for circuit design; see [HD13] and references
therein). Although the problem of interpolating a function under convexity/concavity constraint is not new (see e.g., [Hil54]), methods for doing it
efficiently were only recently being developed (see e.g., [HD12, HD13, MB09]).
In those recent works, the main idea is to represent convex functions throughsupporting hyperplanes (whereas older works were considering a function structure and imposing convexity of that structure). This kind of representation
corresponds to our discrete representation of convex functions from Chapter 3.
As some interest towards smooth convex interpolation was also recently raised,(see [AFM11]), we expect our interpolation schemes to be easily transposable
to that field.

197

Technical open questions and perspectives
In the previous sections, we focused on perspectives for further developing
the range of applications of performance estimation problems (more classesof functions and algorithms). Let us now consider more general questions
and perspectives concerning the treatment of performance estimation problems
which are currently missing in the literature.
 The structure of the SDP should be exploited: when does there exist lowrank solutions (using for example the geometry of the PSD cone [Pat98,
Bar01]) ? How to use the structure of PEP arising in high-dimensional
contexts (e.g., for randomized methods) ? What can we say a priori on
the worst-case solutions ? We refer to Remark 4.6 for a short discussion
on related topics.
 Exploit interpolation conditions (and semidefinite programming) for designing new methods. The main difficulty arising in those perspectives isthe nonconvexity of the method optimization problem.
For example, new geometric versions of accelerated methods for smooth
strongly convex unconstrained minimization were developed in [DFR16,
BLS15], which we believe could be (at least theoretically) refined using
appropriate interpolation conditions (see e.g., Example 3.16 and Example 3.29).
Note that a performance estimation-like approach was developed in [DT16]
for designing a new optimal method for nonsmooth convex minimization.
 In the very recent work [Dro16], a new lower bound was developed using
a performance estimation-related approach. This new bound allowed toprove optimality of the optimized gradient method in a particular setting(best worst-case in function values, starting from an initially boundeddistance to optimality). We believe that such a methodology could beapplied to much more problem classes.
 Finally, there remains many open questions concerning the worst-casebehavior of widely-used optimization schemes (e.g., for splitting methods). In that direction, further extending the framework to handle bothlarger classes of functions and larger classes of algorithms should clearlybe considered.
However, there may be no way of formulating a PEP in a tractable way forsome classes of practical optimization algorithms. Relaxation strategies
for finding approximate optimal solutions (while keeping upper boundproperties) to nonconvex problems should therefore also be considered inthat perspective.

CHAPTER 9. RESEARCH OUTCOMES AND PERSPECTIVES

198

Bibliography[AFM11] Néstor Aguilera, Liliana Forzani, and Pedro Morin. On uniform consistent estimators for convex regression. Journal of Nonparametric
Statistics, 23(4):897–908, 2011.
[AP15] Hedy Attouch and Juan Peypouquet. The rate of convergence ofNesterov’s accelerated forward-backward method is actually o(k −2 ).
preprint arXiv:1510.08740, 2015.
[APR15] Hedy Attouch, Juan Peypouquet, and Patrick Redont. Fast convergence of an inertial gradient-like system with vanishing viscosity.
preprint arXiv:1507.04782, 2015.
[AZO14] Zeyuan Allen-Zhu and Lorenzo Orecchia. Linear coupling: An
ultimate unification of gradient and mirror descent. preprint
arXiv:1407.1537, 2014.
[Bar01] Alexander Barvinok. A remark on the rank of positive semidefinite
matrices subject to affine constraints. Discrete & Computational
Geometry, 25(1):23–31, 2001.
[BB08] Olivier Bousquet and Léon Bottou. The tradeoffs of large scalelearning. In Advances in neural information processing systems,
pages 161–168, 2008.
[BC11] Heinz H Bauschke and Patrick L Combettes. Convex analysis andmonotone operator theory in Hilbert spaces. Springer, 2011.
[BD86] James P Boyle and Richard L Dykstra. A method for finding
projections onto the intersection of convex sets in hilbert spaces.
In Advances in order restricted statistical inference, pages 28–47.
Springer, 1986.
[Bec07] Amir Beck. Quadratic matrix programming. SIAM Journal on
Optimization, 17(4):1224–1238, 2007.
[Ber99] Dimitri P Bertsekas. Nonlinear Programming. Athena Scientific,
2nd edition, 1999.
199

BIBLIOGRAPHY

200[Ber09] Dimitri P Bertsekas. Convex Optimization Theory. Athena Scientific, 2009.
[Ber10] Dimitri P Bertsekas. Incremental gradient, subgradient, and proximal methods for convex optimization: A survey. Optimization for
Machine Learning, pages 1–38, 2010.
[Ber15] Dimitri P Bertsekas. Convex Optimization Algorithms. Athena Scientific, 2015.
[BL10] Jonathan M Borwein and Adrian S Lewis. Convex analysis andnonlinear optimization: theory and examples. Springer Science &Business Media, 2010.
[BLS15] Sébastien Bubeck, Yin Tat Lee, and Mohit Singh. A geometric alternative to nesterov’s accelerated gradient descent. preprint
arXiv:1506.08187, 2015.
[BM03] Samuel A Burer and Renato D C Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2):329–357, 2003.
[BPC+ 11] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan
Eckstein. Distributed optimization and statistical learning via
the alternating direction method of multipliers. Foundations andTrends R in Machine Learning, 3(1):1–122, 2011.
[BT03] Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations
Research Letters, 31(3):167–175, 2003.
[BT09a] Amir Beck and Marc Teboulle. Fast gradient-based algorithms forconstrained total variation image denoising and deblurring problems. Image Processing, IEEE Transactions on, 18(11):2419–2434,
2009.
[BT09b] Amir Beck and Marc Teboulle.
A fast iterative shrinkagethresholding algorithm for linear inverse problems. SIAM Journal
on Imaging Sciences, 2(1):183–202, 2009.
[BT12] Amir Beck and Marc Teboulle. Smoothing and first order methods:
A unified framework. SIAM Journal on Optimization, 22(2):557–
580, 2012.
[BTN01] Ahron Ben-Tal and Arkadi S Nemirovski. Lectures on modern
convex optimization: analysis, algorithms, and engineering applications, volume 2. SIAM, 2001.

201

BIBLIOGRAPHY[Bub15] Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends R in Machine Learning, 8(3-4):231–
357, 2015.
[BV04] Stephen Boyd and Lieven Vandenberghe. Convex Optimization.
Cambridge University Press, 2004.
[BXM03] Stephen Boyd, Lin Xiao, and Almir Mutapcic. Subgradient methods. lecture notes of EE392o, Stanford University, Autumn Quarter,
2003.
[CP11] Patrick L Combettes and Jean-Christophe Pesquet. Proximal splitting methods in signal processing. In Fixed-point algorithms for inverse problems in science and engineering, pages 185–212. Springer,
2011.
[Cro77] Jean-Pierre Crouzeix. A relationship between the second derivativesof a convex function and of its conjugate. Mathematical Programming, 13(1):364–365, 1977.
[d’A08] Alexandre d’Aspremont. Smooth optimization with approximate
gradient. SIAM Journal on Optimization, 19(3):1171–1183, 2008.
[Dan98] George B Dantzig. Linear programming and extensions. Princeton
university press, 1998.
[DFR16] Dmitriy Drusvyatskiy, Maryam Fazel, and Scott Roy. An optimal
first order method based on optimal quadratic averaging. preprint
arXiv:1604.06543, 2016.
[DGN12] Olivier Devolder, François Glineur, and Yurii Nesterov. Doublesmoothing technique for large-scale linearly constrained convex optimization. SIAM Journal on Optimization, 22(2):702–727, 2012.
[DGN14] Olivier Devolder, François Glineur, and Yurii Nesterov. First-ordermethods of smooth convex optimization with inexact oracle. Mathematical Programming, 146(1-2):37–75, 2014.
[DK10] Etienne De Klerk. Exploiting special structure in semidefinite programming: A survey of theory and applications. European Journalof Operational Research, 201(1):1–10, 2010.
[dKGT16] Etienne de Klerk, François Glineur, and Adrien B Taylor. On theworst-case complexity of the gradient method with exact line searchfor smooth strongly convex functions. preprint arXiv:1606.09365(Accepted in Optimization Letters), 2016.
[Dro14] Yoel Drori. Contributions to the Complexity Analysis of Optimization Algorithms. PhD thesis, Tel-Aviv University, 2014.

BIBLIOGRAPHY

202[Dro16] Yoel Drori. The exact information-based complexity of smooth convex minimization. Journal of Complexity, 2016.
[DT14] Yoel Drori and Marc Teboulle. Performance of first-order methods
for smooth convex minimization: a novel approach. Mathematical
Programming, 145(1-2):451–482, 2014.
[DT16] Yoel Drori and Marc Teboulle. An optimal variant of kelley’s
cutting-plane method. Mathematical Programming, 160(1):321–351,
2016.
[Eck89] Jonathan Eckstein. Splitting methods for monotone operators withapplications to parallel optimization. PhD thesis, Massachusetts Institute of Technology, 1989.
[FB53] Werner Fenchel and Donald W Blackett. Convex cones, sets, andfunctions. Princeton University, Department of Mathematics, 1953.
[Fen49] Werner Fenchel. On conjugate convex functions. Canadian Journalof Mathematics, 1(73-77), 1949.
[FG16] Kimon Fountoulakis and Jacek Gondzio. A second-order method
for strongly convex `1 -regularization problems. Mathematical Programming, 156(1-2):189–219, 2016.
[FR15] Olivier Fercoq and Peter Richtárik. Accelerated, parallel, andproximal coordinate descent. SIAM Journal on Optimization,25(4):1997–2023, 2015.
[FW56] Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-2):95–110, 1956.
[Gab83] Daniel Gabay. Chapter ix applications of the method of multipliers
to variational inequalities. Studies in mathematics and its applications, 15:299–331, 1983.
[GB16] Pontus Giselsson and Stephen Boyd. Linear convergence and metric selection for Douglas-Rachford Splitting and ADMM. preprint
arXiv:1410.8479, 2016.
[Gis15] Pontus Giselsson. Tight global linear convergence rate bounds for
Douglas-Rachford splitting. preprint arXiv:1506.01556, 2015.
[Gon12a] Jacek Gondzio. Interior point methods 25 years later. European
Journal of Operational Research, 218(3):587–601, 2012.
[Gon12b] Jacek Gondzio. Matrix-free interior point method. Computational
Optimization and Applications, 51(2):457–480, 2012.

203

BIBLIOGRAPHY[Gül91] Osman Güler. On the convergence of the proximal point algorithm
for convex minimization. SIAM Journal on Control and Optimization, 29(2):403–419, 1991.
[GW95] Michel X Goemans and David P Williamson. Improved approximation algorithms for maximum cut and satisfiability problems
using semidefinite programming. Journal of the ACM (JACM),
42(6):1115–1145, 1995.
[HD12] Lauren A Hannah and David B Dunson. Ensemble methods for convex regression with applications to geometric programming based
circuit design. In Proceedings of the 29th International Conference
on Machine Learning (ICML-12), pages 369–376, 2012.
[HD13] Lauren A Hannah and David B Dunson. Multivariate convex regression with adaptive partitioning. The Journal of Machine Learning
Research, 14(1):3261–3294, 2013.
[Hil54] Clifford Hildreth. Point estimates of ordinates of concave functions.
Journal of the American Statistical Association, 49(267):598–619,
1954.
[HJL12] V Härter, Christian Jansson, and Marko Lange. VSDP: A matlab toolbox for verified semidefinite-quadratic-linear programming.
Technical report, Technical report, Institute for Reliable Computing, Hamburg University of Technology, 2012, 2012.
[Hub64] Peter J Huber. Robust estimation of a location parameter. The
Annals of Mathematical Statistics, 35(1):73–101, 1964.
[HUL96] Jean-Baptiste Hiriart-Urruty and Claude Lemaréchal. Convex Analysis and Minimization Algorithms. Springer Verlag, Heidelberg,
1996. Two volumes - 2nd printing.
[Jag13] Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex
optimization. In Proceedings of the 30th International Conference
on Machine Learning (ICML-13), pages 427–435, 2013.
[JBAS10] Michel Journée, Francis Bach, Pierre-Antoine Absil, and Rodolphe
Sepulchre. Low-rank optimization on the cone of positive semidefinite matrices. SIAM Journal on Optimization, 20(5):2327–2351,
2010.
[JKJJ08] Bjorn Johansson, Tamás Keviczky, Mikael Johansson, andKarl Henrik Johansson. Subgradient methods and consensus algorithms for solving convex optimization problems. In Proceedingsof the 47th IEEE Conference on Decision and Control (2008), pages
4185–4190, 2008.

BIBLIOGRAPHY

204[KF16a] Donghwan Kim and Jeffrey A Fessler. Another look at the ”Fast
Iterative Shrinkage/Thresholding Algorithm (FISTA)”. preprint
arXiv:1608.03861, 2016.
[KF16b] Donghwan Kim and Jeffrey A Fessler. Generalizing the optimized gradient method for smooth convex minimization. preprint
arXiv:1607.06764, 2016.
[KF16c] Donghwan Kim and Jeffrey A Fessler. On the convergence analysisof the optimized gradient method. Journal of Optimization Theory
and Applications, 2016.
[KF16d] Donghwan Kim and Jeffrey A Fessler. Optimized first-order methods for smooth convex minimization. Mathematical Programming,
159(1):81–107, 2016.
[Kiw04] Krzysztof C Kiwiel. Convergence of approximate and incremental
subgradient methods for convex optimization. SIAM Journal on
Optimization, 14(3):807–840, 2004.
[KSST09] Sham Kakade, Shai Shalev-Shwartz, and Ambuj Tewari. On the
duality of strong convexity and strong smoothness: Learning applications and matrix regularization, 2009.
[L0̈4] Johan Löfberg. YALMIP : A toolbox for modeling and optimizationin MATLAB. In Proceedings of the CACSD Conference, 2004.
[LCNS04] Delphine Lambert, Jean-Pierre Crouzeix, V Hien Nguyen, and JeanJacques Strodiot. Finite convex integration. Journal of Convex
Analysis, 11(1):131–146, 2004.
[LM79] Pierre-Louis Lions and Bertrand Mercier. Splitting algorithms forthe sum of two nonlinear operators. SIAM Journal on Numerical
Analysis, 16(6):964–979, 1979.
[LRP16] Laurent Lessard, Benjamin Recht, and Andrew Packard. Analysis
and design of optimization algorithms via integral quadratic constraints. SIAM Journal on Optimization, 26(1):57–95, 2016.
[LV14] Monique Laurent and Antonios Varvitsiotis. A new graph parameter
related to bounded rank positive semidefinite matrix completions.
Mathematical Programming, 145(1-2):291–325, 2014.
[LY08] David G Luenberger and Yinyu Ye. Linear and nonlinear programming. Springer, 2008.
[MB09] Alessandro Magnani and Stephen Boyd. Convex piecewise-linear
fitting. Optimization and Engineering, 10(1):1–17, 2009.

205

BIBLIOGRAPHY[Mor65] Jean-Jacques Moreau. Proximité et dualité dans un espace hilbertien. Bulletin de la Société mathématique de France, 93:273–299,
1965.
[Mos10] APS Mosek. The MOSEK optimization software.
http://www.mosek.com, 54, 2010.

Online at[Nem92] Arkadi S Nemirovski. Information-based complexity of linear operator equations. Journal of Complexity, 8(2):153–175, 1992.
[Nem99] Arkadi S Nemirovski. Optimization II: Numerical methods for nonlinear continuous optimization. Lecture notes, 1999. Available from:http://www2.isye.gatech.edu/~nemirovs/Lect_OptII.pdf.
[Nes83] Yurii Nesterov. A method of solving a convex programming problem with convergence rate O(1/k 2 )). Soviet Mathematics Doklady,
27:372–376, 1983.
[Nes04] Yurii Nesterov. Introductory lectures on convex optimization: A
basic course, volume 87. Springer Science & Business Media, 2004.
[Nes05] Yurii Nesterov. Smooth minimization of non-smooth functions.
Mathematical programming, 103(1):127–152, 2005.
[Nes08] Yurii Nesterov. How to advance in structural convex optimization.
OPTIMA, MPS Newsletter, 78:2–5, 2008.
[Nes12a] Yurii Nesterov. Efficiency of coordinate descent methods on hugescale optimization problems. SIAM Journal on Optimization,
22(2):341–362, 2012.
[Nes12b] Yurii Nesterov. How to make the gradients small. Optima, 88:10–11,
2012.
[Nes13] Yurii Nesterov. Gradient methods for minimizing composite functions. Mathematical Programming, 140(1):125–161, 2013.
[Nes14] Yurii Nesterov. Subgradient methods for huge-scale optimization
problems. Mathematical Programming, 146(1-2):275–297, 2014.
[NN94] Yurii Nesterov and Arkadi S Nemirovski. Interior-Point Polynomial
Algorithms in Convex Programming. Society for Industrial and Applied Mathematics, 1994.
[NNG15] Ion Necoara, Yurii Nesterov, and François Glineur. Linear convergence of first order methods under weak nondegeneracy assumptions
for convex programming. preprint arXiv:1504.06298, 2015.
[NO09] Angelia Nedic and Asuman Ozdaglar. Distributed subgradientmethods for multi-agent optimization. IEEE Transactions on Automatic Control, 54(1):48–61, 2009.

BIBLIOGRAPHY

206[NOP10] Angelia Nedic, Asuman Ozdaglar, and Pablo A Parrilo. Constrained
consensus and optimization in multi-agent networks. IEEE Transactions on Automatic Control, 55(4):922–938, 2010.
[NW06] Jorge Nocedal and Stephen J Wright. Numerical optimization.
Springer Science & Business Media, 2006.
[NY83] Arkadi S Nemirovski and David B Yudin. Problem complexity andmethod efficiency in optimization. Willey-Interscience, New York,1983.
[OHM06] Robert Orsi, Uwe Helmke, and John B Moore. A newton-like
method for solving rank constrained linear matrix inequalities. Automatica, 42(11):1875–1882, 2006.
[Pat98] Gábor Pataki. On the rank of extreme matrices in semidefinite
programs and the multiplicity of optimal eigenvalues. Mathematics
of Operations Research, 23(2):339–358, 1998.
[PB13] Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations
and Trends in optimization, 1(3):123–231, 2013.
[PE10] Daniel P Palomar and Yonina C Eldar. Convex optimization insignal processing and communications. Cambridge university press,2010.
[Pol64] Boris T Polyak. Some methods of speeding up the convergence of
iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1–17, 1964.
[Pol87] Boris T Polyak. Introduction to Optimization. Optimization Software New York, 1987.
[PV91] Panos M Pardalos and Stephen A Vavasis. Quadratic programmingwith one negative eigenvalue is np-hard. Journal of Global Optimization, 1(1):15–22, 1991.
[RB16] Ernest Ryu and Stephen Boyd. A primer on monotone operator
methods. Applied and Computational Mathematics an International
Journal, 15(1), 2016.
[Ren01] James Renegar. A Mathematical View of Interior-Point Methods inConvex Optimization. SIAM, 2001.
[RM51] Herbert Robbins and Sutton Monro. A stochastic approximation
method. The annals of mathematical statistics, pages 400–407, 1951.
[RM09] James B Rawlings and David Q Mayne. Model predictive control:
Theory and design. Nob Hill Publishing, 2009.

207

BIBLIOGRAPHY[Roc76] R Tyrell Rockafellar. Monotone operators and the proximal point
algorithm. SIAM Journal on Control and Optimization, 14(5):877–
898, 1976.
[Roc96] R Tyrell Rockafellar. Convex Analysis. Princeton University Press,
1996.
[Roc99] R Tyrrell Rockafellar. Second-order convex analysis. Journal of
Nonlinear and Convex Analysis, 1:1–16, 1999.
[RT14] Peter Richtárik and Martin Takáč. Iteration complexity of randomized block-coordinate descent methods for minimizing a composite
function. Mathematical Programming, 144(1-2):1–38, 2014.
[Rus06] Andrzej P Ruszczyński. Nonlinear optimization. Princeton university press, 2006.
[RW98] R Tyrell Rockafellar and Roger J-B Wets. Variational Analysis.
Springer, 1998.
[Sah74] Sartaj Sahni. Computationally related problems. SIAM Journal on
Computing, 3(4):262–279, 1974.
[SBC14] Weijie Su, Stephen Boyd, and Emmanuel Candes. A differential
equation for modeling Nesterov’s accelerated gradient method: Theory and insights. In Advances in Neural Information Processing
Systems, pages 2510–2518, 2014.
[SL16] Ziqiang Shi and Rujie Liu. A better convergence analysis of theblock coordinate descent method for large scale machine learning.
preprint arXiv:1608.04826, 2016.
[SLRB11] Mark Schmidt, Nicolas Le Roux, and Francis Bach. Convergence
rates of inexact proximal-gradient methods for convex optimization.
In Advances in neural information processing systems, pages 1458–
1466, 2011.
[SNW12] Suvrit Sra, Sebastian Nowozin, and Stephen J Wright. Optimization
for machine learning. Mit Press, 2012.
[SSBD14] Shai Shalev-Shwartz and Shai Ben-David. Understanding machinelearning: From theory to algorithms. Cambridge University Press,
2014.
[ST04] Daniel A Spielman and Shang-Hua Teng. Smoothed analysis ofalgorithms: Why the simplex algorithm usually takes polynomial
time. Journal of the ACM (JACM), 51(3):385–463, 2004.

BIBLIOGRAPHY

208[Stu99] Jos F Sturm. Using SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones. Optimization Methods and Software,
11–12:625–653, 1999.
[THG16a] Adrien B Taylor, Julien M Hendrickx, and François Glineur. Smooth
strongly convex interpolation and exact worst-case performance offirst-order methods. Mathematical Programming, 2016.
[THG16b] Adrien B Taylor, Julien M Hendrickx, and François Glineur. Exact
worst-case performance of first-order methods for composite convex
optimization. preprint arXiv:1512.07516 (Accepted in SIAM Journal on Optimization), 2016.
[THG16c] Adrien B Taylor, Julien M Hendrickx, and François Glineur. Exact
and global worst-case convergence of the proximal gradient method
for smooth strongly convex optimization. In preparation, 2016.
[Tse08] Paul Tseng. On accelerated proximal gradient methods for convexconcave optimization. Submitted to SIAM Journal on Optimization,
2008.
[VB94] Lieven Vandenberghe and Stephen Boyd. Semidefinite programming. SIAM Review, 38:49–95, 1994.
[ZC15] Hui Zhang and Lizhi Cheng. Restricted strong convexity and its
applications to convergence analysis of gradient-type methods inconvex optimization. Optimization Letters, 9(5):961–979, 2015.
[ZRM09] Royce KP Zia, Edward F Redish, and Susan R McKay. Making senseof the Legendre transform. American Journal of Physics, 77(7):614–
622, 2009.