Algorithms for Optimization

Algorithms for Optimization
Mykel J. Kochenderfer
Tim A. Wheeler

The MIT Press
Cambridge, Massachusetts
London, EnglandThis work is licensed under a Creative Commons “Attribution-NonCommercialNoDerivatives 4.0 International” license.

© 2019 Massachusetts Institute of Technology
Copyright in this monograph has been licensed exclusively to The MIT Press, http://mitpress.mit.edu.
All inquiries regarding rights should be addressed to The MIT Press, Rights and Permissions Department.
This book was set in TEX Gyre Pagella by the authors in LATEX.

To our families.

Contents
Prefacexvii

Acknowledgments

xix

1

Introduction1

1.1

A History

1.2

Optimization Process

1.3

Basic Optimization Problem

1.4

Constraints

1.5

Critical Points

1.6

Conditions for Local Minima

1.7

Contour Plots

1.8

Overview

11

1.9

Summary17

1.10

Exercises

172

Derivatives and Gradients

2.1

Derivatives

2
4
5

6
7
8

11

19

19

viii c on te n ts

2.2

Derivatives in Multiple Dimensions

2.3

Numerical Differentiation

23

2.4

Automatic Differentiation

27

2.5

Summary32

2.6

Exercises

33

3

Bracketing

35

3.1

Unimodality

35

3.2

Finding an Initial Bracket

3.3

Fibonacci Search

3.4

Golden Section Search

39

3.5

Quadratic Fit Search

43

3.6

Shubert-Piyavskii Method

3.7

Bisection Method

3.8

Summary51

3.9

Exercises

51

4

Local Descent

4.1

Descent Direction Iteration

4.2

Line Search

4.3

Approximate Line Search

4.4

Trust Region Methods

61

4.5

Termination Conditions

63

4.6

Summary66

4.7

Exercises

66

21

35

37

45

49

53
53

54
55

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c on te n ts

5

First-Order Methods

5.1

Gradient Descent5.2

Conjugate Gradient

5.3

Momentum

5.4

Nesterov Momentum

5.5

Adagrad77

5.6

RMSProp78

5.7

Adadelta

78

5.8

Adam

5.9

Hypergradient Descent

5.10

Summary84

5.11

Exercises

84

6

Second-Order Methods

6.1

Newton’s Method

6.2

Secant Method

6.3

Quasi-Newton Methods

6.4

Summary95

6.5

Exercises

95

7

Direct Methods

7.1

Cyclic Coordinate Search

7.2

Powell’s Method

7.3

Hooke-Jeeves

7.4

Generalized Pattern Search

ix

69

69
70

75
76

79
80

87

87
91
91

99
99

100
102
103

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

xc on te n ts

7.5

Nelder-Mead Simplex Method

105

7.6

Divided Rectangles

7.7

Summary120

7.8

Exercises

123

8

Stochastic Methods

8.1

Noisy Descent

8.2

Mesh Adaptive Direct Search

8.3

Simulated Annealing

128

8.4

Cross-Entropy Method

133

8.5

Natural Evolution Strategies

8.6

Covariance Matrix Adaptation

8.7

Summary142

8.8

Exercises

142

9

Population Methods

9.1

Initialization

9.2

Genetic Algorithms

9.3

Differential Evolution

9.4

Particle Swarm Optimization

9.5

Firefly Algorithm

9.6

Cuckoo Search

9.7

Hybrid Methods

9.8

Summary165

9.9

Exercises

165

108

125

125
126

137
138

147

147
148
157
158

159
161
162

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c on te n ts

10

Constraints

167

10.2

Constraint Types

168

10.1

10.3

10.4
10.5

10.6

Constrained Optimization

Duality

177

Penalty Methods

10.9

Interior Point Methods

174

178

Augmented Lagrange Method

10.10 Summary
10.11 Exercises

183

186

183

186

11

Linear Constrained Optimization

11.2

Simplex Algorithm

11.1

11.3

11.4
11.5

Problem Formulation
Dual Certificates
Summary
Exercises

210

195

206

210

Multiobjective Optimization

12.2

Constraint Methods

12.3

12.4
12.5

12.6

12.7

Pareto Optimality

211

Weight Methods

218

Preference Elicitation
Exercises

232

211

216

Multiobjective Population Methods
Summary

189

189

12

12.1

169

171

Inequality Constraints

10.7
10.8

167

Transformations to Remove Constraints
Lagrange Multipliers

xi

228

221

232

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

xii

c on te n ts

13

Sampling Plans

235

13.1

Full Factorial

13.2

Random Sampling

13.3

Uniform Projection Plans

13.4

Stratified Sampling

13.5

Space-Filling Metrics

239

13.6

Space-Filling Subsets

244

13.7

Quasi-Random Sequences

13.8

Summary251

13.9

Exercises251

14

Surrogate Models

14.1

Fitting Surrogate Models

14.2

Linear Models

254

14.3

Basis Functions

255

14.4

Fitting Noisy Objective Functions

14.5

Model Selection

14.6

Summary274

14.7

Exercises274

15

Probabilistic Surrogate Models

15.1

Gaussian Distribution

15.2

Gaussian Processes

15.3

Prediction

15.4

Gradient Measurements

235
236
237

238

245

253
253

263

264

275

275
277

280
282

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c on te n ts

15.5

Noisy Measurements

15.6

Fitting Gaussian Processes

15.7

Summary288

15.8

Exercises288

16

Surrogate Optimization

291

16.1

Prediction-Based Exploration

291

16.2

Error-Based Exploration

16.3

Lower Confidence Bound Exploration

16.4

Probability of Improvement Exploration

16.5

Expected Improvement Exploration

16.6

Safe Optimization

16.7

Summary305

16.8

Exercises

305

17

Optimization under Uncertainty

17.1

Uncertainty

17.2

Set-Based Uncertainty

17.3

Probabilistic Uncertainty

17.4

Summary318

17.5

Exercises318

18

Uncertainty Propagation

18.1

Sampling Methods

18.2

Taylor Approximation

18.3

Polynomial Chaos

xiii

285
287

292
293
293
294

296

307

307
309
312

321

321
322
323

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

xiv

c on te n ts

18.4

Bayesian Monte Carlo

334

18.5

Summary337

18.6

Exercises

337

19

Discrete Optimization

19.1

Integer Programs

19.2

Rounding

19.3

Cutting Planes

19.4

Branch and Bound

19.5

Dynamic Programming

19.6

Ant Colony Optimization

19.7

Summary358

19.8

Exercises

358

20

Expression Optimization

20.1

Grammars

20.2

Genetic Programming

364

20.3

Grammatical Evolution

370

20.4

Probabilistic Grammars

375

20.5

Probabilistic Prototype Trees

20.6

Summary382

20.7

Exercises

384

21

Multidisciplinary Optimization

21.1

Disciplinary Analyses

21.2

Interdisciplinary Compatibility

339

340

341
342
346
351
354

361

361

377

387

387
389

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c on te n ts

21.3

Architectures

21.4

Multidisciplinary Design Feasible

21.5

Sequential Optimization

21.6

Individual Discipline Feasible

21.7

Collaborative Optimization

21.8

Simultaneous Analysis and Design

21.9

Summary407

21.10 Exercises

408

xv

393
393

396

A

Julia

411

A.1

Types

411

A.2

Functions

A.3

Control Flow

A.4

Packages

B

Test Functions

425

B.1

Ackley’s Function

425

B.2

Booth’s Function

426

B.3

Branin Function

427

B.4

Flower Function

428

B.5

Michalewicz Function

B.6

Rosenbrock’s Banana Function

B.7

Wheeler’s Ridge

431

B.8

Circle Function

432

398
403
406

420
422
423

429
430

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

xvi

c on te n ts

C

Mathematical Concepts

C.1

Asymptotic Notation

C.2

Taylor Expansion

C.3

Convexity

C.4

Norms

C.5

Matrix Calculus

C.6

Positive Definiteness

C.7

Gaussian Distribution

442

C.8

Gaussian Quadrature

443

D

SolutionsBibliography
Index

433

433
435

436
439
439
442

447
483

495

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

Preface
This book provides a broad introduction to optimization with a focus on practicalalgorithms for the design of engineering systems. We cover a wide variety of
optimization topics, introducing the underlying mathematical problem formulations and the algorithms for solving them. Figures, examples, and exercises areprovided to convey the intuition behind the various approaches.
This text is intended for advanced undergraduates and graduate students
as well as professionals. The book requires some mathematical maturity andassumes prior exposure to multivariable calculus, linear algebra, and probability
concepts. Some review material is provided in the appendix. Disciplines where
the book would be especially useful include mathematics, statistics, computerscience, aerospace, electrical engineering, and operations research.
Fundamental to this textbook are the algorithms, which are all implemented
in the Julia programming language. We have found the language to be ideal for
specifying algorithms in human readable form. Permission is granted, free ofcharge, to use the code snippets associated with this book, subject to the condition
that the source of the code is acknowledged. We anticipate that others may wantto contribute translations of these algorithms to other programming languages.
As translations become available, we will link to them from the book’s webpage.
M y ke l J. Koc he n d e rfe r
Ti m A. W he e l e r
Stanford, Calif.
October 30, 2018
Ancillary material is available on the book’s webpage:
http://mitpress.mit.edu/algorithms-for-optimization

Acknowledgments
This textbook has grown from a course on engineering design optimization taught
at Stanford. We are grateful to the students and teaching assistants who havehelped shape the course over the past five years. We are also indebted to thefaculty who have taught variations of the course before in our department basedon lecture notes from Joaquim Martins, Juan Alonso, Ilan Kroo, Dev Rajnarayan,
and Jason Hicken. Many of the topics discussed in this textbook were inspired bytheir lecture notes.
The authors wish to thank the many individuals who have provided valuable feedback on early drafts of our manuscript, including Mohamed Abdelaty,
Atish Agarwala, Ross Alexander, Piergiorgio Alotto, Nancy Ammar, Dylan Asmar,
David Ata, Rishi Bedi, Felix Berkenkamp, Raunak Bhattacharyya, Hans Borchers,
Maxime Bouton, Ellis Brown, Abhishek Cauligi, Mo Chen, Zhengyu Chen, Vince
Chiu, Anthony Corso, Holly Dinkel, Jonathan Cox, Katherine Driggs-Campbell,
Thai Duong, Hamza El-Saawy, Sofiane Ennadir, Kaijun Feng, Tamas Gal, Christopher Lazarus Garcia, Wouter Van Gijseghem, Michael Gobble, Robert Goedman,
Jayesh Gupta, Aaron Havens, Richard Hsieh, Zdeněk Hurák, Masha Itkina, Arec
Jamgochian, Bogumił Kamiński, Walker Kehoe, Mindaugas Kepalas, Shogo Kishimoto, Veronika Korneyeva, Erez Krimsky, Petr Krysl, Jessie Lauzon, Ruilin Li, Iblis
Lin, Edward Londner, Charles Lu, Miles Lubin, Marcus Luebke, Robert Lupton,
Jacqueline Machesky, Ashe Magalhaes, Zouhair Mahboubi, Pranav Maheshwari,
Yuki Matsuoka, Travis McGuire, Jeremy Morton, Robert Moss, Trương Minh Nhật,Santiago Padrón, Jimin Park, Harsh Patel, Christian Peel, Derek Phillips, Brad
Rafferty, Sidd Rao, Andreas Reschka, Alex Reynell, Stuart Rogers, Per Rutquist,
Ryan Samuels, Orson Sandoval, Jeffrey Sarnoff, Chelsea Sidrane, Sumeet Singh,
Nathan Stacey, Ethan Strijbosch, Alex Toews, Pamela Toman, Rachael Tompa,
Zacharia Tuten, Alexandros Tzikas, Raman Vilkhu, Yuri Vishnevsky, Julie Walker,

xxa c kn ow l e d g me n ts

Zijian Wang, Patrick Washington, Jacob West, Adam Wiktor, John Wu, Brandon
Yeung, Anil Yildiz, Robert Young, Javier Yu, Andrea Zanette, and Remy Zawislak.
In addition, it has been a pleasure working with Marie Lufkin Lee and Christine
Bridget Savage from the MIT Press in preparing this manuscript for publication.
The style of this book was inspired by Edward Tufte. Among other stylistic
elements, we adopted his wide margins and use of small multiples. In fact, thetypesetting of this book is heavily based on the Tufte-LaTeX package by Kevin
Godby, Bil Kleb, and Bill Wood. We were also inspired by the clarity of the
textbooks by Donald Knuth and Stephen Boyd.
Over the past few years, we have benefited from discussions with the core Julia
developers, including Jeff Bezanson, Stefan Karpinski, and Viral Shah. We havealso benefited from the various open source packages on which this textbook
depends (see appendix A.4). The typesetting of the code is done with the helpof pythontex, which is maintained by Geoffrey Poore. Plotting is handled by
pgfplots, which is maintained by Christian Feuersänger. The book’s color scheme
was adapted from the Monokai theme by Jon Skinner of Sublime Text. For plots,
we use the viridis colormap defined by Stéfan van der Walt and Nathaniel Smith.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

1 Introduction
Many disciplines involve optimization at their core. In physics, systems are driven
to their lowest energy state subject to physical laws. In business, corporations
aim to maximize shareholder value. In biology, fitter organisms are more likely
to survive. This book will focus on optimization from an engineering perspective,
where the objective is to design a system that optimizes a set of metrics subject toconstraints. The system could be a complex physical system like an aircraft, or it
could be a simple structure such as a bicycle frame. The system might not even be
physical; for example, we might be interested in designing a control system foran automated vehicle or a computer vision system that detects whether an imageof a tumor biopsy is cancerous. We want these systems to perform as well as
possible. Depending on the application, relevant metrics might include efficiency,
safety, and accuracy. Constraints on the design might include cost, weight, andstructural soundness.
This book is about the algorithms, or computational processes, for optimization.
Given some representation of the system design, such as a set of numbers encoding
the geometry of an airfoil, these algorithms will tell us how to search the spaceof possible designs with the aim of finding the best one. Depending on theapplication, this search may involve running physical experiments, such as windtunnel tests, or it might involve evaluating an analytical expression or runningcomputer simulations. We will discuss computational approaches for addressinga variety of challenges, such as how to search high-dimensional spaces, handlingproblems where there are multiple competing objectives, and accommodating
uncertainty in the metrics.

2 c ha p te r 1. i n trod u c ti on

1.1

A History

We will begin our discussion of the history of algorithms for optimization1 with theancient Greek philosophers. Pythagoras of Samos (569–475 BCE), the developerof the Pythagorean theorem, claimed that ‘‘the principles of mathematics were theprinciples of all things,’’2 popularizing the idea that mathematics could model theworld. Both Plato (427–347 BCE) and Aristotle (384–322 BCE) used reasoning forthe purpose of societal optimization.3 They contemplated the best style of human
life, which involves the optimization of both individual lifestyle and functioningof the state. Aristotelian logic was an early formal process—an algorithm—by
which deductions can be made.
Optimization of mathematical abstractions also dates back millennia. Euclidof Alexandria (325–265 BCE) solved early optimization problems in geometry,
including how to find the shortest and longest lines from a point to the circumference of a circle. He also showed that a square is the rectangle with the maximum
area for a fixed perimeter.4 The Greek mathematician Zenodorus (200–140 BCE)
studied Dido’s problem, shown in figure 1.1.
Others demonstrated that nature seems to optimize. Heron of Alexandria(10–75 CE) showed that light travels between points through the path of shortest
length. Pappus of Alexandria (290–350 CE), among his many contributions to
optimization, argued that the hexagon repeated in honeycomb is the optimal
regular polygon for storing honey; its hexagonal structure uses the least material
to create a lattice of cells over a plane.5
Central to the study of optimization is the use of algebra, which is the studyof the rules for manipulating mathematical symbols. Algebra is credited to the
Persian mathematician al-Khwārizmī (790–850 CE) with the treatise ‘‘al-Kitāb aljabr wal-muqābala,’’ or ‘‘The Compendious Book on Calculation by Completion
and Balancing.’’ Algebra had the advantage of using Hindu-Arabic numerals,
including the use of zero in base notation. The word al’jabr is Persian for restorationand is the source for the Western word algebra. The term algorithm comes fromalgoritmi, the Latin translation and pronunciation of al-Khwārizmī’s name.
Optimization problems are often posed as a search in a space defined by a setof coordinates. Use of coordinates comes from René Descartes (1596–1650), whoused two numbers to describe a point on a two-dimensional plane. His insightlinked algebra, with its analytic equations, to the descriptive and visual field ofgeometry.6 His work also included a method for finding the tangent to any curveThis discussion is not meant to becomprehensive. A more detailed
history is provided by X.-S. Yang,“A Brief History of Optimization,”in Engineering Optimization. Wiley,2010, pp. 1–13.
1

2
Aristotle, Metaphysics, trans. by
W. D. Ross. 350 BCE, Book I, Part 5.
3
See discussion by S. Kiranyaz, T.
Ince, and M. Gabbouj, Multidimensional Particle Swarm Optimization
for Machine Learning and Pattern
Recognition. Springer, 2014, Section 2.1.

See books III and VI of Euclid, The
Elements, trans. by D. E. Joyce. 300
BCE.
4

sea

Carthage

Figure 1.1. Queen Dido, founderof Carthage, was granted as muchland as she could enclose witha bullhide thong. She made asemicircle with each end of thethong against the MediterraneanSea, thus enclosing the maximum
possible area. This problem is mentioned in Virgil’s Aeneid (19 BCE).

5
T. C. Hales, “The Honeycomb
Conjecture,” Discrete & Computational Geometry, vol. 25, pp. 1–22,
2001.

R. Descartes, “La Géométrie,” in
Discours de la Méthode. 1637.
6© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

1. 1. a hi story 3

whose equation is known. Tangents are useful in identifying the minima andmaxima of functions. Pierre de Fermat (1601–1665) began solving for where thederivative is zero to identify potentially optimal points.
The concept of calculus, or the study of continuous change, plays an important role in our discussion of optimization. Modern calculus stems from thedevelopments of Gottfried Wilhelm Leibniz (1646–1716) and Sir Isaac Newton(1642–1727). Both differential and integral calculus make use of the notion of
convergence of infinite series to a well-defined limit.
The mid-twentieth century saw the rise of the electronic computer, spurringinterest in numerical algorithms for optimization. The ease of calculations allowed
optimization to be applied to much larger problems in a variety of domains. Oneof the major breakthroughs came with the introduction of linear programming,
which is an optimization problem with a linear objective function and linear
constraints. Leonid Kantorovich (1912–1986) presented a formulation for linear
programming and an algorithm to solve it.7 It was applied to optimal resource
allocation problems during World War II. George Dantzig (1914–2005) developed
the simplex algorithm, which represented a significant advance in solving linear
programs efficiently.8 Richard Bellman (1920–1984) developed the notion of dynamic programming, which is a commonly used method for optimally solving
complex problems by breaking them down into simpler problems.9 Dynamic programming has been used extensively for optimal control. This textbook outlines
many of the key algorithms developed for digital computers that have been usedfor various engineering design optimization problems.
Decades of advances in large scale computation have resulted in innovative
physical engineering designs as well as the design of artificially intelligent systems.
The intelligence of these systems has been demonstrated in games such as chess,
Jeopardy!, and Go. IBM’s Deep Blue defeated the world chess champion Garry
Kasparov in 1996 by optimizing moves by evaluating millions of positions. In
2011, IBM’s Watson played Jeopardy! against former winners Brad Futter and Ken
Jennings. Watson won the first place prize of $1 million by optimizing its responsewith respect to probabilistic inferences about 200 million pages of structured
and unstructured data. Since the competition, the system has evolved to assist in
healthcare decisions and weather forecasting. In 2017, Google’s AlphaGo defeated
Ke Jie, the number one ranked Go player in the world. The system used neuralnetworks with millions of parameters that were optimized from self-play and

L. V. Kantorovich, “A New
Method of Solving Some Classesof Extremal Problems,” in Proceedings of the USSR Academy of
Sciences, vol. 28, 1940.
8
The simplex algorithm will becovered in chapter 11.
7

R. Bellman, “On the Theory of Dynamic Programming,” Proceedingsof the National Academy of Sciences of
the United States of America, vol. 38,
no. 8, pp. 716–719, 1952.

9© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

4 c ha p te r 1. i n trod u c ti on

data from human games. The optimization of deep neural networks is fueling amajor revolution in artificial intelligence that will likely continue.10

1.2

Optimization Process

A typical engineering design optimization process is shown in figure 1.2.11 The
role of the designer is to provide a problem specification that details the parameters,
constants, objectives, and constraints that are to be achieved. The designer is
responsible for crafting the problem and quantifying the merits of potential
designs. The designer also typically supplies a baseline design or initial design
point to the optimization algorithm.
ChangeDesigndesignspecifications

InitialDesignEvaluate
Performance

10
I. Goodfellow, Y. Bengio, and
A. Courville, Deep Learning. MIT
Press, 2016.

Further discussion of the design
process in engineering is provided
in J. Arora, Introduction to Optimum
Design, 4th ed. Academic Press,
2016.
11

no

Good?

This book is about automating the process of refining the design to improve
performance. An optimization algorithm is used to incrementally improve thedesign until it can no longer be improved or until the budgeted time or cost has
been reached. The designer is responsible for analyzing the result of the optimization process to ensure its suitability for the final application. Misspecifications in
the problem, poor baseline designs, and improperly implemented or unsuitable
optimization algorithms can all lead to suboptimal or dangerous designs.
There are several advantages of an optimization approach to engineering design. First of all, the optimization process provides a systematic, logical design
procedure. If properly followed, optimization algorithms can help reduce thechance of human error in design. Sometimes intuition in engineering design canbe misleading; it can be much better to optimize with respect to data. Optimization
can speed the process of design, especially when a procedure can be written once
and then be reapplied to other problems. Traditional engineering techniques are

yes

final
designFigure 1.2. The design optimization process. We seek to automate
the optimization procedure highlighted in blue.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

1. 3. ba si c op ti mi z a ti on p robl e m

5often visualized and reasoned about by humans in two or three dimensions. Modern optimization techniques, however, can be applied to problems with millions
of variables and constraints.
There are also challenges associated with using optimization for design. We are
generally limited in our computational resources and time, and so our algorithms
have to be selective in how they explore the design space. Fundamentally, theoptimization algorithms are limited by the designer’s ability to specify the problem. In some cases, the optimization algorithm may exploit modeling errors orprovide a solution that does not adequately solve the intended problem. When an
algorithm results in an apparently optimal design that is counterintuitive, it canbe difficult to interpret. Another limitation is that many optimization algorithms
are not guaranteed to produce optimal designs.

1.3

Basic Optimization Problem

The basic optimization problem is:
minimizexf (x)subject to x ∈ X(1.1)

Here, x is a design point. A design point can be represented as a vector of values
corresponding to different design variables. An n-dimensional design point iswritten:12(1.2)[ x1 , x2 , · · · , x n ]where the ith design variable is denoted xi . The elements in this vector can be
adjusted to minimize the objective function f . Any value of x from among all points
in the feasible set X that minimizes the objective function is called a solution orminimizer. A particular solution is written x∗ . Figure 1.3 shows an example of aone-dimensional optimization problem.
This formulation is general, meaning that any optimization problem can be
rewritten according to equation (1.1). In particular, a problem
maximize f (x) subject to x ∈ X(1.3)

minimize − f (x) subject to x ∈ X

(1.4)

x

can be replaced byx

As in Julia, square brackets withcomma-separated entries are used
to represent column vectors. Design points are column vectors.

12© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c ha p te r 1. i n trod u c ti on

f (x)

6

Figure 1.3. A one-dimensional optimization problem. Note that theminimum is merely the best in thefeasible set—lower points may exist outside the feasible region.

x∗

X
x

The new form is the same problem in that it has the same set of solutions.
Modeling engineering problems within this mathematical formulation can bechallenging. The way in which we formulate an optimization problem can makethe solution process either easy or hard.13 We will focus on the algorithmic aspects
of optimization that arise after the problem has been properly formulated.14
Since this book discusses a wide variety of different optimization algorithms,
one may wonder which algorithm is best. As elaborated by the no free lunch
theorems of Wolpert and Macready, there is no reason to prefer one algorithm overanother unless we make assumptions about the probability distribution over the
space of possible objective functions. If one algorithm performs better than anotheralgorithm on one class of problems, then it will perform worse on another classof problems.15 For many optimization algorithms to work effectively, there needs
to be some regularity in the objective function, such as Lipschitz continuity or
convexity, both topics that we will cover later. As we discuss different algorithms,
we will outline their assumptions, the motivation for their mechanism, and theiradvantages and disadvantages.

See discussion in S. Boyd and
L. Vandenberghe, Convex Optimization. Cambridge University Press,
2004.

13

Many texts provide examples of
how to translate real-world optimization problems into optimization problems. See, for example,
the following: R. K. Arora, Optimization: Algorithms and Applications. Chapman and Hall/CRC,
2015. A. D. Belegundu and T. R.
Chandrupatla, Optimization Concepts and Applications in Engineering, 2nd ed. Cambridge University Press, 2011. A. Keane and P.
Nair, Computational Approaches for
Aerospace Design. Wiley, 2005. P. Y.
Papalambros and D. J. Wilde, Principles of Optimal Design. Cambridge
University Press, 2017.
14

The assumptions and results of
the no free lunch theorems areprovided by D. H. Wolpert and
W. G. Macready, “No Free Lunch
Theorems for Optimization,” IEEE
Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 67–82, 1997.
15

1.4 Constraints
Many problems have constraints. Each constraint limits the set of possible solutions,and together the constraints define the feasible set X . Feasible design points do
not violate any constraints. For example, consider the following optimization© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

1. 5 . c ri ti c a l p oi n ts 7

problem:minimizef ( x1 , x2 )subject tox1 ≥ 0x1 ,x2x2(1.5)(0, 1)

The feasible set is plotted in figure 1.4.
Constraints are typically written with ≤, ≥, or =. If constraints involve < or> (i.e., strict inequalities), then the feasible set does not include the constraint
boundary. A potential issue with not including the boundary is illustrated by thisproblem:minimize xx(1.6)
subject to x > 1X

x2 ≥ 0x1 + x2 ≤ 1

The feasible set is shown in figure 1.5. The point x = 1 produces values smaller
than any x greater than 1, but x = 1 is not feasible. We can pick any x arbitrarilyclose to, but greater than, 1, but no matter what we pick, we can always find an
infinite number of values even closer to 1. We must conclude that the problem
has no solution. To avoid such issues, it is often best to include the constraint
boundary in the feasible set.

1.5

Critical Points

Figure 1.6 shows a univariate function16 f ( x ) with several labeled critical points,
where the derivative is zero, that are of interest when discussing optimization
problems. When minimizing f , we wish to find a global minimizer, a value of x forwhich f ( x ) is minimized. A function may have at most one global minimum, but
it may have multiple global minimizers.
Unfortunately, it is generally difficult to prove that a given candidate point is
at a global minimum. Often, the best we can do is check whether it is at a local
minimum. A point x ∗ is at a local minimum (or is a local minimizer) if there existsa δ > 0 such that f ( x ∗ ) ≤ f ( x ) for all x with | x − x ∗ | < δ. In the multivariate
context, this definition generalizes to there being a δ > 0 such that f (x∗ ) ≤ f (x)whenever kx − x∗ k < δ.

x1

(1, 0)
Figure 1.4. The feasible set X associated with equation (1.5).

f (x)(1, 1)x
Figure 1.5. The problem in equation (1.6) has no solution because
the constraint boundary is not feasible.
A univariate function is a function of a single scalar. The term univariate describes objects involving
one variable.

16© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

8 c ha p te r 1. i n trod u c ti on

Figure 1.6. Examples of criticalpoints of interest to optimization
algorithms (where the derivative
is zero) on a univariate function.

f (x)inflection
weak local min
global min

strong local minx
Figure 1.6 shows two types of local minima: strong local minima and weak local
minima. A strong local minimizer, also known as a strict local minimizer, is a pointthat uniquely minimizes f within a neighborhood. In other words, x ∗ is a strict
local minimizer if there exists a δ > 0 such that f ( x ∗ ) < f ( x ) whenever x ∗ 6= x
and | x − x ∗ | < δ. In the multivariate context, this generalizes to there being a
δ > 0 such that f (x∗ ) < f (x) whenever x∗ 6= x and kx − x∗ k < δ. A weak local
minimizer is a local minimizer that is not a strong local minimizer.
The derivative is zero at all local and global minima of continuous, unbounded
objective functions. While having a zero derivative is a necessary condition for a
local minimum,17 it is not a sufficient condition.
Figure 1.6 also has an inflection point where the derivative is zero but the point
does not locally minimize f . An inflection point is where the sign of the second
derivative of f changes, which corresponds to a local minimum or maximum off ′.

Points with nonzero derivatives
are never minima.

17

1.6 Conditions for Local Minima
Many numerical optimization methods seek local minima. Local minima arelocally optimal, but we do not generally know whether a local minimum is a global
minimum. The conditions we discuss in this section assume that the objectivefunction is differentiable. Derivatives, gradients, and Hessians are reviewed in© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

1. 6 . c on d i ti on s for l oc a l mi n i ma

9the next chapter. We also assume in this section that the problem is unconstrained.
Conditions for optimality in constrained problems are introduced in chapter 10.

1.6.1 Univariate
A design point is guaranteed to be at a strong local minimum if the local derivative
is zero and the second derivative is positive:
1. f ′ ( x ∗ ) = 0
2. f ′′ ( x ∗ ) > 0
A zero derivative ensures that shifting the point by small values does not
significantly affect the function value. A positive second derivative ensures thatthe zero first derivative occurs at the bottom of a bowl.18
A point can also be at a local minimum if it has a zero derivative and the second
derivative is merely nonnegative:

18

1. f ′ ( x ∗ ) = 0, the first-order necessary condition (FONC)1919

A point that satisfies the firstorder necessary condition is sometimes called a stationary point.

2. f ′′ ( x ∗ ) ≥ 0, the second-order necessary condition (SONC)

These conditions are referred to as necessary because all local minima obey
these two rules. Unfortunately, not all points with a zero derivative and a zerosecond derivative are local minima, as demonstrated in figure 1.7.
The first necessary condition can be derived using the Taylor expansion20
about our candidate point x ∗ :f ( x ∗ + h ) = f ( x ∗ ) + h f ′ ( x ∗ ) + O ( h2 )
′

∗

(1.8)

∗

∗

∗

′

∗

(1.9)

∗

∗

′

∗

(1.10)f ( x − h) = f ( x ) − h f ( x ) + O(h )f ( x + h) ≥ f ( x ) =⇒ h f ( x ) ≥ 0

f ( x − h) ≥ f ( x ) =⇒ h f ( x ) ≤ 0=⇒ f ′ ( x ∗ ) = 0

The Taylor expansion is derived
in appendix C.

20

(1.7)

∗

2

If f ′ ( x ) = 0 and f ′′ ( x ) < 0, thenx is a local maximum.

(1.11)

where the asymptotic notation O(h2 ) is covered in appendix C.
The second-order necessary condition can also be obtained from the Taylor
expansion:h2
f ( x ∗ + h) = f ( x ∗ ) + h f ′ ( x ∗ ) + f ′′ ( x ∗ ) + O(h3 )| {z } 2(1.12)=0© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

10c ha p te r 1. i n trod u c ti on

x∗

x∗

x∗
SONC but not FONC

FONC and SONC

FONC and SONC
Figure 1.7. Examples of the necessary but insufficient conditions forstrong local minima.

We know that the first-order necessary condition must apply:f ( x ∗ + h) ≥ f ( x ∗ ) =⇒h2 ′′ ∗f (x ) ≥ 0
2(1.13)

since h > 0. It follows that f ′′ ( x ∗ ) ≥ 0 must hold for x ∗ to be at a local minimum.

1.6.2 Multivariate
The following conditions are necessary for x to be at a local minimum of f :1. ∇ f (x) = 0, the first-order necessary condition (FONC)

2. ∇2 f (x) is positive semidefinite (for a review of this definition, see appendix C.6),
the second-order necessary condition (SONC)The FONC and SONC are generalizations of the univariate case. The FONC tells
us that the function is not changing at x. Figure 1.8 shows examples of multivariate
functions where the FONC is satisfied. The SONC tells us that x is in a bowl.
The FONC and SONC can be obtained from a simple analysis. In order for x∗
to be at a local minimum, it must be smaller than those values around it:f (x∗ ) ≤ f (x∗ + hy)⇔

f (x∗ + hy) − f (x∗ ) ≥ 0(1.14)

If we write the second-order approximation for f (x∗ ), we get:1f (x∗ + hy) = f (x∗ ) + h∇ f (x∗ )⊤ y + h2 y⊤ ∇2 f (x∗ )y + O(h3 )2(1.15)

We know that at a minimum, the first derivative must be zero, and we neglect
the higher order terms. Rearranging, we get:1 2 ⊤ 2h y ∇ f (x∗ )y = f (x∗ + hy) − f (x∗ ) ≥ 0
2(1.16)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

1. 7. c on tou r p l ots

11This is the definition of a positive semidefinite matrix, and we recover the SONC.
Example 1.1 illustrates how these conditions can be applied to the Rosenbrockbanana function.

A local maximum. The gradientat the center is zero, but the
Hessian is negative definite.

A saddle. The gradient at thecenter is zero, but it is not a
local minimum.

A bowl. The gradient at thecenter is zero and the Hessian
is positive definite. It is a local
minimum.

While necessary for optimality, the FONC and SONC are not sufficient for
optimality. For unconstrained optimization of a twice-differentiable function, apoint is guaranteed to be at a strong local minimum if the FONC is satisfiedand ∇2 f (x) is positive definite. These conditions are collectively known as thesecond-order sufficient condition (SOSC).

1.7

Figure 1.8. The three local regions
where the gradient is zero.

Contour PlotsThis book will include problems with a variety of numbers of dimensions, andwill need to display information over one, two, or three dimensions. Functions of
the form f ( x1 , x2 ) = y can be rendered in three-dimensional space, but not allorientations provide a complete view over the domain. A contour plot is a visual
representation of a three-dimensional surface obtained by plotting regions withconstant y values, known as contours, on a two-dimensional plot with axes indexedby x1 and x2 . Example 1.2 illustrates how a contour plot can be interpreted.

1.8 Overview
This section provides a brief overview of the chapters of this book. The conceptual
dependencies between the chapters are outlined in figure 1.9.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12c ha p te r 1. i n trod u c ti on

Example 1.1. Checking the firstand second-order necessary conditions of a point on the Rosenbrock function. The minimizer isindicated by the dot in the figure
below.

Consider the Rosenbrock banana function,f (x) = (1 − x1 )2 + 5( x2 − x12 )2
Does the point (1, 1) satisfy the FONC and SONC?
The gradient is:and the Hessian is:
 2

∂ f
1
∇2 f (x) =  ∂x∂12∂xf∂x2 ∂x1

∂x1
∂f
∂x2

=

"



2

2 10x13 − 10x1 x2 + x1 − 1
10( x2 − x12 )∂2 f∂x1 ∂x2 =∂2 f∂x2 ∂x2"

#x2∇ f (x) =" ∂f #−20( x2 − x12 ) + 40x12 + 2 −20x1
−20x1
10

0

−2
−2

#

0
x1

2

We compute ∇( f )([1, 1]) = 0, so the FONC is satisfied. The Hessian at [1, 1]
is:#"42 −20−20
10which is positive definite, so the SONC is satisfied.

The function f ( x1 , x2 ) = x12 − x22 . This function can be visualized in a threedimensional space based on its two inputs and one output. It can also be
visualized using a contour plot, which shows lines of constant y value. A
three-dimensional visualization and a contour plot are shown below.

Example 1.2. An example threedimensional visualization and theassociated contour plot.

−4

2
0

4

0

4

−2

x2

0
0

0

2

2

5

−5

2
0
x1

0
2

−2

x2

−2

−2

0

−2

−4

−2

0
x1

2

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

1. 8 . ove rvi e w 13

2. Derivatives

4. Descent

3. Bracketing

5. First-Order
6. Second-Order

7. Direct
8. Stochastic

9. Population

20. Expressions

10. Constraints

11. Linear

19. Discrete

Figure 1.9. Dependencies for thechapters in Algorithms for Optimization. Lighter arrows indicate weak
dependencies.

12. Multiobjective
13. SamplingPlans

14. Surrogate
Models

17. Optimization under Uncertainty

15. Probabilistic
Surrogate Models

16. Surrogate
Optimization

18. Uncertainty Propagation

21. Multidisciplinary Design Optimization

Chapter 2 begins by discussing derivatives and their generalization to multiple
dimensions. Derivatives are used in many algorithms to inform the choice of
direction of the search for an optimum. Often derivatives are not known analytically, and so we discuss how to estimate them numerically and using automaticdifferentiation techniques.
Chapter 3 discusses bracketing, which involves identifying an interval in whicha local minimum lies for a univariate function. Different bracketing algorithms use
different schemes for successively shrinking the interval based on function evaluations. One of the approaches we discuss uses knowledge of the Lipschitz constantof the function to guide the bracketing process. These bracketing algorithms areoften used as subroutines within the optimization algorithms discussed later inthe text.
Chapter 4 introduces local descent as a general approach to optimizing multivariate functions. Local descent involves iteratively choosing a descent direction andthen taking a step in that direction and repeating that process until convergenceor some termination condition is met. There are different schemes for choosing
the step length. We will also discuss methods that adaptively restrict the step size
to a region where there is confidence in the local model.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

14c ha p te r 1. i n trod u c ti on

Chapter 5 builds upon the previous chapter, explaining how to use first-orderinformation obtained through the gradient estimate as a local model to informthe descent direction. Simply stepping in the direction of steepest descent is often
not the best strategy for finding a minimum. This chapter discusses a wide variety
of different methods for using the past sequence of gradient estimates to better
inform the search.
Chapter 6 shows how to use local models based on second-order approximations
to inform local descent. These models are based on estimates of the Hessian of
the objective function. The advantage of second-order approximations is that it
can inform both the direction and step size.
Chapter 7 presents a collection of direct methods for finding optima that avoid
using gradient information for informing the direction of search. We begin by
discussing methods that iteratively perform line search along a set of directions.
We then discuss pattern search methods that do not perform line search but rather
perform evaluations some step size away from the current point along a set of
directions. The step size is incrementally adapted as the search proceeds. Anothermethod uses a simplex that adaptively expands and contracts as it traverses thedesign space in the apparent direction of improvement. Finally, we discuss amethod motivated by Lipschitz continuity to increase resolution in areas deemed
likely to contain the global minimum.
Chapter 8 introduces stochastic methods, where randomness is incorporated
into the optimization process. We show how stochasticity can improve some of
the algorithms discussed in earlier chapters, such as steepest descent and patternsearch. Some of the methods involve incrementally traversing the search space, butothers involve learning a probability distribution over the design space, assigning
greater weight to regions that are more likely to contain an optimum.
Chapter 9 discusses population methods, where a collection of points is used to
explore the design space. Having a large number of points distributed through thespace can help reduce the risk of becoming stuck in a local minimum. Population
methods generally rely upon stochasticity to encourage diversity in the population,and they can be combined with local descent methods.
Chapter 10 introduces the notion of constraints in optimization problems. We
begin by discussing the mathematical conditions for optimality with constraints.
We then introduce methods for incorporating constraints into the optimization
algorithms discussed earlier through the use of penalty functions. We also discuss© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

1. 8 . ove rvi e w 15

methods for ensuring that, if we start with a feasible point, the search will remain
feasible.
Chapter 11 makes the assumption that both the objective function and constraints are linear. Although linearity may appear to be a strong assumption, manyengineering problems can be framed as linear constrained optimization problems.
Several methods have been developed for exploiting this linear structure. This
chapter focuses on the simplex algorithm, which is guaranteed to result in a global
minimum.
Chapter 12 shows how to address the problem of multiobjective optimization,
where we have multiple objectives that we are trying to optimize simultaneously.
Engineering often involves a tradeoff between multiple objectives, and it is often
unclear how to prioritize different objectives. We discuss how to transform multiobjective problems into scalar-valued objective functions so that we can use thealgorithms discussed in earlier chapters. We also discuss algorithms for finding
the set of design points that represent the best tradeoff between objectives.
Chapter 13 discusses how to create sampling plans consisting of points thatcover the design space. Random sampling of the design space often does notprovide adequate coverage. We discuss methods for ensuring uniform coveragealong each design dimension and methods for measuring and optimizing thecoverage of the space. In addition, we discuss quasi-random sequences that canalso be used to generate sampling plans.
Chapter 14 explains how to build surrogate models of the objective function.
Surrogate models are often used for problems where evaluating the objectivefunction is very expensive. An optimization algorithm can then use evaluations of
the surrogate model instead of the actual objective function to improve the design.
The evaluations can come from historical data, perhaps obtained through theuse of a sampling plan introduced in the previous chapter. We discuss different
types of surrogate models, how to fit them to data, and how to identify a suitablesurrogate model.
Chapter 15 introduces probabilistic surrogate models that allow us to quantify ourconfidence in the predictions of the models. This chapter focuses on a particular
type of surrogate model called a Gaussian process. We show how to use Gaussian
processes for prediction, how to incorporate gradient measurements and noise,and how to estimate some of the parameters governing the Gaussian processfrom data.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

16

c ha p te r 1. i n trod u c ti on

Chapter 16 shows how to use the probabilistic models from the previous
chapter to guide surrogate optimization. The chapter outlines several techniques
for choosing which design point to evaluate next. We also discuss how surrogate
models can be used to optimize an objective measure in a safe manner.
Chapter 17 explains how to perform optimization under uncertainty, relaxing theassumption made in previous chapters that the objective function is a deterministic
function of the design variables. We discuss different approaches for representing
uncertainty, including set-based and probabilistic approaches, and explain how
to transform the problem to provide robustness to uncertainty.
Chapter 18 outlines approaches to uncertainty propagation, where known input
distributions are used to estimate statistical quantities associated with the output
distribution. Understanding the output distribution of an objective function isimportant to optimization under uncertainty. We discuss a variety of approaches,some based on mathematical concepts such as Monte Carlo, the Taylor series
approximation, orthogonal polynomials, and Gaussian processes. They differ in
the assumptions they make and the quality of their estimates.
Chapter 19 shows how to approach problems where the design variables areconstrained to be discrete. A common approach is to relax the assumption that thevariables are discrete, but this can result in infeasible designs. Another approach
involves incrementally adding linear constraints until the optimal point is discrete.
We also discuss branch and bound along with dynamic programming approaches,both of which guarantee optimality. The chapter also mentions a population-based
method that often scales to large design spaces but does not provide guarantees.
Chapter 20 discusses how to search design spaces consisting of expressions
defined by a grammar. For many problems, the number of variables is unknown,such as in the optimization of graphical structures or computer programs. We
outline several algorithms that account for the grammatical structure of the design
space to make the search more efficient.
Chapter 21 explains how to approach multidisciplinary design optimization. Many
engineering problems involve complicated interactions between several disciplines, and optimizing disciplines individually may not lead to an optimal solution. This chapter discusses a variety of techniques for taking advantage of thestructure of multidisciplinary problems to reduce the effort required for finding
good designs.
The appendices contain supplementary material. Appendix A begins with ashort introduction to the Julia programming language, focusing on the concepts© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

1. 9. su mma ry

17

used to specify the algorithms listed in this book. Appendix B specifies a variety
of test functions used for evaluating the performance of different algorithms.
Appendix C covers mathematical concepts used in the derivation and analysis of
the optimization methods discussed in this text.

1.9 Summary• Optimization in engineering is the process of finding the best system design
subject to a set of constraints.
• Optimization is concerned with finding global minima of a function.
• Minima occur where the gradient is zero, but zero-gradient does not imply
optimality.

1.10 Exercises
Exercise 1.1. Give an example of a function with a local minimum that is not a
global minimum.
Exercise 1.2. What is the minimum of the function f ( x ) = x3 − x?

Exercise 1.3. Does the first-order condition f ′ ( x ) = 0 hold when x is the optimal
solution of a constrained problem?
Exercise 1.4. How many minima does f ( x, y) = x2 + y, subject to x > y ≥ 1,
have?
Exercise 1.5. How many inflection points does f ( x ) = x3 − 10 have?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

2

Derivatives and Gradients

Optimization is concerned with finding the design point that minimizes (ormaximizes) an objective function. Knowing how the value of a function changes
as its input is varied is useful because it tells us in which direction we can move toimprove on previous points. The change in the value of the function is measured
by the derivative in one dimension and the gradient in multiple dimensions. This
chapter briefly reviews some essential elements from calculus.1

2.1 Derivatives

For a more comprehensive review, see S. J. Colley, Vector Calculus, 4th ed. Pearson, 2011.
1

The derivative f ′ ( x ) of a function f of a single variable x is the rate at which thevalue of f changes at x. It is often visualized, as shown in figure 2.1, using thetangent line to the graph of the function at x. The value of the derivative equals
the slope of the tangent line.

f (x)

x

Figure 2.1. The function f is drawn
in black and the tangent line to f ( x )is drawn in blue. The derivative off at x is the slope of the tangentline.

20 c ha p te r 2. d e ri va ti ve s a n d g ra d i e n ts

We can use the derivative to provide a linear approximation of the function
near x:f ( x + ∆x ) ≈ f ( x ) + f ′ ( x )∆x(2.1)

The derivative is the ratio between the change in f and the change in x at the
point x:∆ f (x)(2.2)
f ′ (x) =∆xwhich is the change in f ( x ) divided by the change in x as the step becomes
infinitesimally small as illustrated by figure 2.2.

h

h

h

The notation f ′ ( x ) can be attributed to Lagrange. We also use the notation

created by Leibniz,

Figure 2.2. The tangent line is obtained by joining points with sufficiently small step differences.

d f (x)(2.3)
dx
which emphasizes the fact that the derivative is the ratio of the change in f to thechange in x at the point x.
The limit equation defining the derivative can be presented in three different
ways: the forward difference, the central difference, and the backward difference. Eachmethod uses an infinitely small step size h:f ′ (x) ≡f ′ ( x ) ≡ lim
h →0|

f ( x + h/2) − f ( x − h/2)f ( x ) − f ( x − h)f ( x + h) − f ( x )
= lim
= lim
h
h
h
h →0
h →0
{z
} |
{z
} |
{z
}

forward difference

central difference

backward difference

If f can be represented symbolically, symbolic differentiation can often provide
an exact analytic expression for f ′ by applying derivative rules from calculus. The
analytic expression can then be evaluated at any point x. The process is illustrated
in example 2.1.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com(2.4)

2. 2. d e ri va ti ve s i n mu l ti p l e d i me n si on s 21

The implementation details of symbolic differentiation is outside the scopeof this text. Various software packages, such as SymEngine.jl in Julia and
SymPy in Python, provide implementations. Here we use SymEngine.jl tocompute the derivative of x2 + x/2 − sin( x )/x.

Example 2.1. Symbolic differentiation provides analytical derivatives.

julia> using SymEngine
julia> @vars x; # define x as a symbolic variablejulia> f = x^2 + x/2 - sin(x)/x;
julia> diff(f, x)
1/2 + 2*x + sin(x)/x^2 - cos(x)/x2.2

Derivatives in Multiple Dimensions

The gradient is the generalization of the derivative to multivariate functions. It
captures the local slope of the function, allowing us to predict the effect of taking
a small step from a point in any direction. Recall that the derivative is the slopeof the tangent line. The gradient points in the direction of steepest ascent of thetangent hyperplane as shown in figure 2.3. A hyperplane in an n-dimensional
space is the set of points that satisfies
w1 x 1 + · · · + w n x n = b(2.5)

for some vector w and scalar b. A hyperplane has n − 1 dimensions.
The gradient of f at x is written ∇ f (x) and is a vector. Each component of that
vector is the partial derivative2 of f with respect to that component:
h
i
f (x)
∂ f (x)∂ f (x)∇ f (x) = ∂∂x(2.6), ∂x , . . . , ∂xn
2

1

We use the convention that vectors written with commas are column vectors. For
example, we have [ a, b, c] = [ a b c]⊤ . Example 2.2 shows how to compute thegradient of a function at a particular point.
The Hessian of a multivariate function is a matrix containing all of the second
derivatives with respect to the input.3 The second derivatives capture information
about the local curvature of the function.
 2∂2 f (x)∂2 f (x)∂ f (x)···∂x1 ∂xn  ∂x1 ∂x1 ∂x1 ∂x2
..
2∇ f (x) = 
(2.7)

.

 2
∂ f (x)∂2 f (x)∂2 f (x)· · · ∂xn ∂xn∂xn ∂x∂xn ∂x1

Figure 2.3. Each component of thegradient defines a local tangent
line. These tangent lines define thelocal tangent hyperplane. The gradient vector points in the directionof greatest increase.

The partial derivative of a function with respect to a variable isthe derivative assuming all other
input variables are held constant.
It is denoted ∂ f /∂x.

2

The Hessian is symmetric only if
the second derivatives of f are allcontinuous in a neighborhood of
the point at which it is being evaluated:
3

∂2 f
∂2 f
=
∂x1 ∂x2∂x2 ∂x1

2© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

22 c ha p te r 2. d e ri va ti ve s a n d g ra d i e n ts

Example 2.2. Computing the gradient at a particular point.

Compute the gradient of f (x) = x1 sin( x2 ) + 1 at c = [2, 0].
f (x) = x1 sin( x2 ) + 1


∂f ∂f
∇ f (x) =,= [sin( x2 ), x1 cos( x2 )]∂x1 ∂x2∇ f (c) = [0, 2]

The directional derivative ∇s f (x) of a multivariate function f is the instantaneous
rate of change of f (x) as x is moved with velocity s. The definition is closely related
to the definition of a derivative of a univariate function:4∇s f (x) ≡ lim
h →0|Some texts require that s be aunit vector. See, for example, G. B.
Thomas, Calculus and Analytic Geometry, 9th ed. Addison-Wesley,1968.
4

f (x + hs/2) − f (x − hs/2)
f (x) − f (x − hs)f (x + hs) − f (x)
= lim
= lim
h
h
h
h →0
h →0
{z
} |
{z
} |
{z
}

forward difference

central difference

backward difference

The directional derivative can be computed using the gradient of the function:∇s f (x) = ∇ f (x) ⊤ s(2.9)

Another way to compute the directional derivative ∇s f (x) is to define g(α) ≡f (x + αs) and then compute g′ (0), as illustrated in example 2.3.
The directional derivative is highest in the gradient direction, and it is lowest
in the direction opposite the gradient. This directional dependence arises fromthe dot product in the directional derivative’s definition and from the fact thatthe gradient is a local tangent hyperplane.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com(2.8)

2. 3. n u me ri c a l d i ffe re n ti a ti on 23

We wish to compute the directional derivative of f (x) = x1 x2 at x = [1, 0] inthe direction s = [−1, −1]:
h
i
∂f
∂f
∇ f (x) = ∂x
, ∂x = [ x2 , x1 ]
2
1
" #
h
i −1
= −1
∇s f (x) = ∇ f (x) ⊤ s = 0 1−1

Example 2.3. Computing a directional derivative.

f (α)
α

We can also compute the directional derivative as follows:

1g(α) = f (x + αs) = (1 − α)(−α) = α2 − αg′ (α) = 2α − 1
g ′ (0) = −1

2.3

Numerical Differentiation

The process of estimating derivatives numerically is referred to as numericaldifferentiation. Estimates can be derived in different ways from function evaluations.
This section discusses finite difference methods and the complex step method.5

2.3.1 Finite Difference Methods
As the name implies, finite difference methods compute the difference between twovalues that differ by a finite step size. They approximate the derivative definitions
in equation (2.4) using small differences:f ′ (x) ≈For a more comprehensive treatment of the topics discussed in theremainder of this chapter, see A.
Griewank and A. Walther, Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation,
2nd ed. SIAM, 2008.
5f ( x + h/2) − f ( x − h/2)f ( x ) − f ( x − h)f ( x + h) − f ( x )
≈
≈
h
h
h
|
{z
} |
{z
} |
{z
}
forward difference

central difference(2.10)

backward difference

Mathematically, the smaller the step size h, the better the derivative estimate.
Practically, values for h that are too small can result in numerical cancellation
errors. This effect is shown later in figure 2.4. Algorithm 2.1 provides implementations for these methods.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

24 c ha p te r 2. d e ri va ti ve s a n d g ra d i e n ts

diff_forward(f, x; h=sqrt(eps(Float64))) = (f(x+h) - f(x))/h
diff_central(f, x; h=cbrt(eps(Float64))) = (f(x+h/2) - f(x-h/2))/h
diff_backward(f, x; h=sqrt(eps(Float64))) = (f(x) - f(x-h))/h

The finite difference methods can be derived using the Taylor expansion. We
will derive the forward difference derivative estimate, beginning with the Taylor
expansion of f about x:
f ′ (x)f ′′ ( x ) 2
f ′′′ ( x ) 3h+h +h +···
1!
2!
3!
We can rearrange and solve for the first derivative:f ( x + h) = f ( x ) +(2.11)

Algorithm 2.1. Finite difference
methods for estimating the derivative of a function f at x with finite
difference h. The default step sizes
are the square root or cube root of
the machine precision for floating
point values. These step sizes balance machine round-off error withstep size error.
The eps function provides thestep size between 1.0 and the next
larger representable floating-pointvalue.

f ′′′ ( x ) 3
f ′′ ( x ) 2h −h −···(2.12)
2!
3!
f ( x + h) − f ( x )f ′′ ( x )f ′′′ ( x ) 2
f ′ (x) =−h−h −···(2.13)h2!
3!
f ( x + h) − f ( x )(2.14)f ′ (x) ≈h
The forward difference approximates the true derivative for small h withf ′ ( x )h = f ( x + h) − f ( x ) −f ′′ ( x )f ′′′ ( x )error dependent on 2! h + 3! h2 + · · · . This error term is O(h), meaning theforward difference has linear error as h approaches zero.6
The central difference method has an error term of O(h2 ).7 We can derive
this error term using the Taylor expansion. The Taylor expansions about x forf ( x + h/2) and f ( x − h/2) are:  f ′′ ( x ) h 2
f ′′′ ( x ) h 3
h++···(2.15)f ( x + h/2) = f ( x ) + f ′ ( x ) +
2
2!
2
3!
2
 
 
f ′′′ ( x ) h 3
f ′′ ( x ) h 2
h
′
−
+···
(2.16)f ( x − h/2) = f ( x ) − f ( x ) +
2
2!
2
3!
2

Asymptotic notation is covered inappendix C.

6

J. H. Mathews and K. D. Fink, Numerical Methods Using MATLAB,
4th ed. Pearson, 2004.
7

Subtracting these expansions produces:

 3
h
2 ′′′
h
f ( x + h/2) − f ( x − h/2) ≈ 2 f ( x ) + f ( x )
2 3!
2
′

(2.17)We rearrange to obtain:f ′′′ ( x )h2
f ( x + h/2) − f ( x − h/2)−h24which shows that the approximation has quadratic error.
f ′ (x) ≈(2.18)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

2. 3. n u me ri c a l d i ffe re n ti a ti on 25

2.3.2 Complex Step Method
We often run into the problem of needing to choose a step size h small enoughto provide a good approximation but not too small so as to lead to numericalsubtractive cancellation issues. The complex step method bypasses the effect ofsubtractive cancellation by using a single function evaluation. We evaluate thefunction once after taking a step in the imaginary direction.8
The Taylor expansion for an imaginary step is:f ( x + ih) = f ( x ) + ih f ′ ( x ) − h2

f ′′′ ( x )f ′′ ( x )
− ih3
+···
2!
3!

(2.19)Taking only the imaginary component of each side produces a derivative approximation:
f ′′′ ( x )+···3!
f ′′′ ( x )Im( f ( x + ih))
+ h2
−···
⇒ f ′ (x) =h3!
Im( f ( x + ih))
+ O(h2 ) as h → 0=hIm( f ( x + ih)) = h f ′ ( x ) − h3

8
J. R. R. A. Martins, P. Sturdza, andJ. J. Alonso, “The Complex-Step
Derivative Approximation,” ACM
Transactions on Mathematical Software, vol. 29, no. 3, pp. 245–262,
2003. Special care must be taken to
ensure that the implementation off properly supports complex numbers as input.

(2.20)(2.21)(2.22)

An implementation is provided by algorithm 2.2. The real part approximates f ( x )to within O(h2 ) as h → 0:f ′′ ( x )+...
2!
f ′′ ( x )⇒ f ( x ) = Re( f ( x + ih)) + h2−···
2!
Re( f ( x + ih)) = f ( x ) − h2

(2.23)(2.24)

Thus, we can evaluate both f ( x ) and f ′ ( x ) using a single evaluation of f withcomplex arguments. Example 2.4 shows the calculations involved for estimating
the derivative of a function at a particular point. Algorithm 2.2 implements thecomplex step method. Figure 2.4 compares the numerical error of the complex
step method to the forward and central difference methods as the step size isvaried.
diff_complex(f, x; h=1e-20) = imag(f(x + h*im)) / h

Algorithm 2.2. The complex step
method for estimating the derivative of a function f at x with finite
difference h.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

26

c ha p te r 2. d e ri va ti ve s a n d g ra d i e n ts

Consider f ( x ) = sin( x2 ). The function value at x = π/2 is approximately
0.624266 and the derivative is π cos(π 2 /4) ≈ −2.45425. We can arrive at
this using the complex step method:

Example 2.4. The complex step
method for estimating derivatives.

julia> f = x -> sin(x^2);
julia> v = f(π/2 + 0.001im);julia> real(v) # f(x)0.6242698144866649
julia> imag(v)/0.001 # f'(x)-2.4542516170381785

complexforwardcentralabsolute relative error

103

10−1

10−5

10−9

10−13

Figure 2.4. A comparison of theerror in derivative estimate for thefunction sin( x ) at x = 1/2 as thestep size is varied. The linear errorof the forward difference method
and the quadratic error of the central difference and complex methods can be seen by the constantslopes on the right hand side. The
complex step method avoids thesubtractive cancellation error that
occurs when differencing two function evaluations that are close together.

10−17
10−18

10−15

10−12

10−9

10−6

10−3

100

step size h© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

2. 4 . a u toma ti c d i ffe re n ti a ti on 27

2.4

Automatic Differentiation

This section introduces algorithms for the numeric evaluation of derivatives offunctions specified by a computer program. Key to these automatic differentiation
techniques is the application of the chain rule:dd f dgdf ( g( x )) =( f ◦ g)( x ) =
dx
dx
dg dx

(2.25)A program is composed of elementary operations like addition, subtraction,
multiplication, and division.
Consider the function f ( a, b) = ln( ab + max( a, 2)). If we want to compute thepartial derivative with respect to a at a point, we need to apply the chain rule
several times:9∂f∂=ln( ab + max( a, 2))
∂a
∂a
∂
1
=
( ab + max( a, 2))
ab + max( a, 2) ∂a∂( ab) ∂ max( a, 2)
1+=
ab + max( a, 2) ∂a
∂a

 

1
∂a
∂2
∂b
∂a
=
b +a
+ (2 > a ) + (2 < a )ab + max( a, 2)
∂a
∂a
∂a
∂a
1
=
[b + (2 < a)]ab + max( a, 2)(2.26)

We adopt the convention that
Boolean expressions like (2 < α)
are 1 if true and 0 if false.

9(2.27)(2.28)(2.29)(2.30)

This process can be automated through the use of a computational graph. A
computational graph represents a function where the nodes are operations andthe edges are input-output relations. The leaf nodes of a computational graphare input variables or constants, and terminal nodes are values output by thefunction. A computational graph is shown in figure 2.5.
There are two methods for automatically differentiating f using its computational graph. The forward accumulation method used by dual numbers traverses
the tree from inputs to outputs, whereas reverse accumulation requires a backwards
pass through the graph.

2.4.1 Forward Accumulation
Forward accumulation will automatically differentiate a function using a single forward pass through the function’s computational graph. The method is equivalent© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

28

c ha p te r 2. d e ri va ti ve s a n d g ra d i e n ts

Figure 2.5. The computationalgraph for ln( ab + max( a, 2)).

b

×

c1

a

+
max

c3

ln

c4

c2

2
to iteratively expanding the chain rule of the inner operation:





df
d f dc4d f dc4 dc3
dc dc
d f dc4 dc3 dc2
=
=
+ 3 1
=
dx
dc4 dxdc4 dc3 dx
dc4 dc3 dc2 dx
dc1 dx(2.31)

To illustrate forward accumulation, we apply it to the example function f ( a, b) =ln( ab + max( a, 2)) to calculate the partial derivative at a = 3, b = 2 with respect
to a.
1. The procedure starts at the graph’s source nodes consisting of the function
inputs and any constant values. For each of these nodes, we note both thevalue and the partial derivative with respect to our target variable, as shownin figure 2.6.10
2. Next we proceed down the tree, one node at a time, choosing as our next node
one whose inputs have already been computed. We can compute the value by
passing through the previous nodes’ values, and we can compute the local
partial derivative with respect to a using both the previous nodes’ values andtheir partial derivatives. The calculations are shown in figure 2.7.

10
For compactness in this figure,
we use dot notation or Newton’s notation for derivatives. For example,
if it is clear that we are taking thederivative with respect to a, we canwrite ∂b/∂a as ḃ.

We end up with the correct result, f (3, 2) = ln 9 and ∂ f /∂a = 1/3. This wasdone using one pass through the computational graph.
This process can be conveniently automated by a computer using a programming language which has overridden each operation to produce both the value
and its derivative. Such pairs are called dual numbers.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

2. 4 . a u toma ti c d i ffe re n ti a ti on 29b=2ḃ = 0a=3ȧ = 1Figure 2.6. The computationalgraph for ln( ab + max( a, 2)) being
set up for forward accumulation
to calculate ∂ f /∂a with a = 3 andb = 2.

b
c1

×
a

c3

+

c4

ln

c2

max
2

b=2
ḃ = 0a=3ȧ = 1c1 = a × b = 6ċ1 = b ȧ + aḃ = 2

b

×

c1

a

+
max

2

c2

c3

c4 = ln c3 = ln 9ċ4 = ċ3 /c3 = 31
ln

Figure 2.7. The computationalgraph for ln( ab + max( a, 2)) after
forward accumulation is applied
to calculate ∂ f /∂a with a = 3 andb = 2.

c4

c3 = c1 + c2 = 9ċ3 = ċ1 + ċ2 = 3

c2 = max ( a, 2) = 3
0 if 2 > aċ2 ==1 ȧ if 2 < a© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

30 c ha p te r 2. d e ri va ti ve s a n d g ra d i e n ts

Dual numbers can be expressed mathematically by including the abstract
quantity ǫ, where ǫ2 is defined to be 0. Like a complex number, a dual number iswritten a + bǫ where a and b are both real values. We have:( a + bǫ) + (c + dǫ) = ( a + c) + (b + d)ǫ

(2.32)( a + bǫ) × (c + dǫ) = ( ac) + ( ad + bc)ǫ(2.33)

In fact, by passing a dual number into any smooth function f , we get the evaluationand its derivative. We can show this using the Taylor series:∞

f (k) ( a )( x − a)kk!
k =0

(2.34)

f (x) = ∑∞f (k) ( a )( a + bǫ − a)kk!
k =0f ( a + bǫ) = ∑∞f (k) ( a ) bk ǫkk!
k =0(2.35)(2.36)= ∑∞f ( k ) ( a ) b k ( k −2)
ǫ
k!
k =2

= f ( a ) + b f ′ ( a ) ǫ + ǫ2 ∑(2.37)= f ( a) + b f ′ ( a)ǫ

(2.38)

Example 2.5 shows an implementation.

2.4.2 Reverse Accumulation
Forward accumulation requires n passes in order to compute an n-dimensional
gradient. Reverse accumulation11 requires only a single run in order to compute acomplete gradient but requires two passes through the graph: a forward pass during
which necessary intermediate values are computed and a backward pass which
computes the gradient. Reverse accumulation is often preferred over forward
accumulation when gradients are needed, though care must be taken on memoryconstrained systems when the computational graph is very large.12
Like forward accumulation, reverse accumulation will compute the partial
derivative with respect to the chosen target variable but iteratively substitutes theouter function instead:





df
d f dc1 dc3 dc4d f dc4d f dc3 dc4d f dc2+===(2.39)dx
dc4 dxdc3 dc4 dx
dc2 dc3
dc1 dc3 dc4 dx

S. Linnainmaa, “The Representation of the Cumulative Rounding
Error of an Algorithm as a Taylor
Expansion of the Local Rounding
Errors,” M.S. thesis, University of
Helsinki, 1970.

11

Reverse accumulation is central
to the backpropagation algorithm
used to train neural networks. D. E.
Rumelhart, G. E. Hinton, and R. J.
Williams, “Learning Representations by Back-Propagating Errors,”
Nature, vol. 323, pp. 533–536, 1986.
12© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

2. 4 . a u toma ti c d i ffe re n ti a ti on 31

Dual numbers can be implemented by defining a struct Dual that contains
two fields, the value v and the derivative ∂.

Example 2.5. An implementationof dual numbers allows for automatic forward accumulation.

struct Dualv∂
end

We must then implement methods for each of the base operations required.
These methods take in dual numbers and produce new dual numbers usingthat operation’s chain rule logic.
Base.:+(a::Dual, b::Dual) = Dual(a.v + b.v, a.∂ + b.∂)Base.:*(a::Dual, b::Dual) = Dual(a.v * b.v, a.v*b.∂ + b.v*a.∂)Base.log(a::Dual) = Dual(log(a.v), a.∂/a.v)
function Base.max(a::Dual, b::Dual)v = max(a.v, b.v)
∂ = a.v > b.v ? a.∂ : a.v < b.v ? b.∂ : NaN
return Dual(v, ∂)endfunction Base.max(a::Dual, b::Int)v = max(a.v, b)∂ = a.v > b ? a.∂ : a.v < b ? 0 : NaN
return Dual(v, ∂)end

The ForwardDiff.jl package supports an extensive set of mathematicaloperations and additionally provides gradients and Hessians.
julia> using ForwardDiff
julia> a = ForwardDiff.Dual(3,1);
julia> b = ForwardDiff.Dual(2,0);
julia> log(a*b + max(a,2))
Dual{Nothing}(2.1972245773362196,0.3333333333333333)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

32 c ha p te r 2. d e ri va ti ve s a n d g ra d i e n ts

This process is the reverse pass, the evaluation of which requires intermediate
values that are obtained during a forward pass.
Reverse accumulation can be implemented through operation overloading13 in a 13 Operation overloading refers
similar manner to the way dual numbers are used to implement forward accumu- to providing implementations
for common operations such as
lation. Two functions must be implemented for each fundamental operation: a +, −, or = for custom variableforward operation that overloads the operation to store local gradient information types. Overloading is discussed inappendix A.2.5.
during the forward pass and a backward operation that uses the information to 14
Tensorflow is an open source softpropagate the gradient backwards. Packages like Tensorflow14 or Zygote.jl can ware library for numerical compuautomatically construct the computational graph and the associated forward and tation using data flow graphs andis often used for deep learning apbackwards pass operations. Example 2.6 shows how Zygote.jl can be used.
plications. It may be obtained fromThe Zygote.jl package provides automatic differentiation in the form ofreverse-accumulation. Here the gradient function is used to automaticallygenerate the backwards pass through the source code of f to obtain thegradient.

tensorflow.org.

Example 2.6. Automatic differentiation using the Zygote.jl package.
We find that the gradient at [3, 2] is[1/3, 1/3].

julia> import Zygote: gradientjulia> f(a, b) = log(a*b + max(a,2));
julia> gradient(f, 3.0, 2.0)
(0.3333333333333333, 0.3333333333333333)

2.5 Summary• Derivatives are useful in optimization because they provide information about
how to change a given point in order to improve the objective function.
• For multivariate functions, various derivative-based concepts are useful for
directing the search for an optimum, including the gradient, the Hessian, andthe directional derivative.
• One approach to numerical differentiation includes finite difference approximations.
• The complex step method can eliminate the effect of subtractive cancellation
error when taking small steps, resulting in high quality gradient estimates.
• Analytic differentiation methods include forward and reverse accumulationon computational graphs.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

2. 6 . e x e rc i se s 33

2.6 Exercises
Exercise 2.1. Adopt the forward difference method to approximate the Hessianof f (x) using its gradient, ∇ f (x).

Exercise 2.2. What is a drawback of the central difference method over otherfinite difference methods if we already know f (x)?
Exercise 2.3. Compute the gradient of f ( x ) = ln x + e x + 1x for a point x close to
zero. What term dominates in the expression?
Exercise 2.4. Suppose f ( x ) is a real-valued function that is also defined forcomplex inputs. If f (3 + ih) = 2 + 4ih, what is f ′ (3)?

Exercise 2.5. Draw the computational graph for f ( x, y) = sin( x + y2 ). Use thecomputational graph with forward accumulation to compute ∂ f /∂y at ( x, y) =(1, 1). Label the intermediate values and partial derivatives as they are propagatedthrough the graph.
Exercise 2.6. Combine the forward and backward difference methods to obtain adifference method for estimating the second-order derivative of a function f at x
using three function evaluations.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

3 Bracketing
This chapter presents a variety of bracketing methods for univariate functions,or functions involving a single variable. Bracketing is the process of identifying
an interval in which a local minimum lies and then successively shrinking theinterval. For many functions, derivative information can be helpful in directing
the search for an optimum, but, for some functions, this information may not be
available or might not exist. This chapter outlines a wide variety of approaches
that leverage different assumptions. Later chapters that consider multivariate
optimization will build upon the concepts introduced here.

3.1 Unimodality
Several of the algorithms presented in this chapter assume unimodality of theobjective function. A unimodal function f is one where there is a unique x ∗ , suchthat f is monotonically decreasing for x ≤ x ∗ and monotonically increasing forx ≥ x ∗ . It follows from this definition that the unique global minimum is at x ∗ ,and there are no other local minima.1
Given a unimodal function, we can bracket an interval [ a, c] containing the global
minimum if we can find three points a < b < c, such that f ( a) > f (b) < f (c).
Figure 3.1 shows an example.

3.2 Finding an Initial Bracket
When optimizing a function, we often start by first bracketing an interval containing a local minimum. We then successively reduce the size of the bracketedinterval to converge on the local minimum. A simple procedure (algorithm 3.1)
can be used to find an initial bracket. Starting at a given point, we take a step

It is perhaps more conventional todefine unimodal functions in the
opposite sense, such that there isa unique global maximum rather
than a minimum. However, in thistext, we try to minimize functions,and so we use the definition in thisparagraph.

1

36 c ha p te r 3. bra c ke ti n g

Figure 3.1. Three points shownbracketing a minimum.

ab cin the positive direction. The distance we take is a hyperparameter to this algorithm,2 but the algorithm provided defaults it to 1 × 10−2 . We then search in thedownhill direction to find a new point that exceeds the lowest point. With each
step, we expand the step size by some factor, which is another hyperparameterto this algorithm that is often set to 2. An example is shown in figure 3.2. Functions without local minima, such as exp( x ), cannot be bracketed and will cause
bracket_minimum to fail.
function bracket_minimum(f, x=0; s=1e-2, k=2.0)
a, ya = x, f(x)b, yb = a + s, f(a + s)
if yb > yaa, b = b, aya, yb = yb, ya
s = -send
while truec, yc = b + s, f(b + s)
if yc > yb
return a < c ? (a, c) : (c, a)
end
a, ya, b, yb = b, yb, c, yc
s *= kend
end

A hyperparameter is a parameterthat governs the function of an algorithm. It can be set by an expert
or tuned using an optimization algorithm. Many of the algorithms inthis text have hyperparameters. We
often provide default values suggested in the literature. The successof an algorithm can be sensitive tothe choice of hyperparameter.

2

Algorithm 3.1. An algorithm forbracketing an interval in which a
local minimum must exist. It takes
as input a univariate function f andstarting position x, which defaults
to 0. The starting step size s andthe expansion factor k can be specified. It returns a tuple containing
the new interval [ a, b].

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

y

3. 3. fi bon a c c i se a rc h 37

ab

ba
x

x

b a
x

3.3 Fibonacci Search
Suppose we have a unimodal f bracketed by the interval [ a, b]. Given a limiton the number of times we can query the objective function, Fibonacci search(algorithm 3.2) is guaranteed to maximally shrink the bracketed interval.
Suppose we can query f only twice. If we query f on the one-third and twothird points on the interval, then we are guaranteed to remove one-third of ourinterval, regardless of f , as shown in figure 3.3.
2

1

new interval if y1 < y2bax

Figure 3.2. An example of running bracket_minimum on a function. The method reverses direction between the first and second
iteration and then expands until
a minimum is bracketed in the
fourth iteration.

Figure 3.3. Our initial guess for two
queries will remove one-third of
the initial interval.

new interval if y1 > y2
We can guarantee a tighter bracket by moving our guesses toward the center.
In the limit as ǫ → 0, we are guaranteed to shrink our interval by a factor of two
as shown in figure 3.4.
Figure 3.4. The most we can guarantee to shrink our interval is byjust under a factor of two.

ǫ
new interval if y1 < y2
new interval if y1 > y2

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

38 c ha p te r 3. bra c ke ti n g

With three queries, we can shrink the interval by a factor of three. We firstquery f on the one-third and two-third points on the interval, discard one-third of
the interval, and then sample just next to the better sample as shown in figure 3.5.
Figure 3.5. With three queries wecan shrink the domain by a factorof three. The third query is madebased on the result of the first two
queries.

2

1
3

For n queries, the interval lengths are related to the Fibonacci sequence: 1, 1,
2, 3, 5, 8, and so forth. The first two terms are one, and the following terms arealways the sum of the previous two:1
if n ≤ 2
Fn =(3.1)
 Fn−1 + Fn−2 otherwise
Figure 3.6 shows the relationship between the intervals. Example 3.1 walks
through an application to a univariate function.

Figure 3.6. For n queries we are
guaranteed to shrink our interval
by a factor of Fn+1 . The length of every interval constructed during Fibonacci search can be expressed interms of the final interval times a Fibonacci number. If the final, smallest interval has length In , then thesecond smallest interval has length
In−1 = F3 In , the third smallest interval has length In−2 = F4 In , andso forth.

I1 = I2 + I3 = 8I5
I2 = I3 + I4 = 5I5
I3 = I4 + I5 = 3I5
I4 = 2I5
I5The Fibonacci sequence can be determined analytically using Binet’s formula:ϕ n − (1 − ϕ ) n
√
,
5

(3.2)

Fn
1 − s n +1
=ϕ
Fn−1
1 − sn

(3.3)

Fn =

√
where ϕ = (1 + 5)/2 ≈ 1.61803 is the golden ratio.
The ratio between successive values in the Fibonacci sequence is:where s = (1 −√5) / (1 +√5) ≈ −0.382.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

3. 4 . g ol d e n se c ti on se a rc h 39

Algorithm 3.2. Fibonacci search
to be run on univariate function f, with bracketing interval[ a, b], for n > 1 function evaluations. It returns the new interval(a, b). The optional parameter ϵcontrols the lowest-level interval.
The golden ratio φ is defined inBase.MathConstants.jl.

function fibonacci_search(f, a, b, n; ϵ=0.01)
s = (1-√5)/(1+√5)
ρ = 1 / (φ*(1-s^(n+1))/(1-s^n))d = ρ*b + (1-ρ)*ayd = f(d)for i in 1 : n-1
if i == n-1c = ϵ*a + (1-ϵ)*delse
c = ρ*a + (1-ρ)*b
end
yc = f(c)if yc < ydb, d, yd = d, c, yc
else
a, b = b, cend
ρ = 1 / (φ*(1-s^(n-i+1))/(1-s^(n-i)))end
return a < b ? (a, b) : (b, a)end

3.4 Golden Section Search
If we take the limit for large n, we see that the ratio between successive values of
the Fibonacci sequence approaches the golden ratio:
Fn
= ϕ.
lim
n→∞ Fn−1(3.4)

Golden section search (algorithm 3.3) uses the golden ratio to approximate Fibonacci
search. Figure 3.7 shows the relationship between the intervals. Figures 3.8 and 3.9
compare Fibonacci search with golden section search on unimodal and nonunimodal functions, respectively.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

40 c ha p te r 3. bra c ke ti n g

Consider using Fibonacci search with five function evaluations to minimizef ( x ) = exp( x − 2) − x over the interval [ a, b] = [−2, 6]. The first two function
evaluations are made at FF56 and 1 − FF65 , along the length of the initial bracketinginterval:
F5f ( x (1) ) = f a + ( b − a ) 1 −= f (1) = −0.632
F6


F5
= f (3) = −0.282
f ( x (2) ) = f a + ( b − a )F6

Example 3.1. Using Fibonacci
search with five function evaluations to optimize a univariate function.

The evaluation at x (1) is lower, yielding the new interval [ a, b] = [−2, 3].
Two evaluations are needed for the next interval split:F
xleft = a + (b − a) 1 − 4 = 0
F5F4xright = a + (b − a) = 1
F5
A third function evaluation is thus made at xleft , as xright has already been
evaluated:f ( x (3) ) = f (0) = 0.135
The evaluation at x (1) is lower, yielding the new interval [ a, b] = [0, 3].
Two evaluations are needed for the next interval split:
F3
xleft = a + (b − a) 1 −=1
F4
F3
xright = a + (b − a) = 2
F4
A fourth functional evaluation is thus made at xright , as xleft has alreadybeen evaluated:f ( x (4) ) = f ( 2 ) = − 1

The new interval is [ a, b] = [1, 3]. A final evaluation is made just next tothe center of the interval at 2 + ǫ, and it is found to have a slightly higher
value than f (2). The final interval is [1, 2 + ǫ].

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

3. 4 . g ol d e n se c ti on se a rc h 41

I1
I2 = I1 ϕ−1Figure 3.7. For n queries of a univariate function we are guaranteed
to shrink a bracketing interval by afactor of ϕn−1 .

I3 = I1 ϕ−2
I4 = I1 ϕ−3
I5 = I1 ϕ−4function golden_section_search(f, a, b, n)
ρ = φ-1
d = ρ * b + (1 - ρ)*ayd = f(d)for i = 1 : n-1
c = ρ*a + (1 - ρ)*b
yc = f(c)if yc < ydb, d, yd = d, c, yc
else
a, b = b, c
end
end
return a < b ? (a, b) : (b, a)
end

Algorithm 3.3. Golden section
search to be run on a univariate
function f, with bracketing interval [ a, b], for n > 1 function evaluations. It returns the new interval (a, b). Julia already has thegolden ratio φ defined. Guaranteeing convergence to within ǫ requires n = (b − a)/(ǫ ln ϕ) iterations.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

42 c ha p te r 3. bra c ke ti n gyFibonacci Search

x

x

x

x

x

y

Golden Section Search

x

x

x

x

x
Figure 3.8. Fibonacci and golden
section search on a unimodal function.

yFibonacci Search

x

x

x

x

x

x

x

y

Golden Section Search

x

x

x

Figure 3.9. Fibonacci and golden
section search on a nonunimodal
function. Search is not guaranteed
to find a global minimum.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

3. 5 . q u a d ra ti c fi t se a rc h 43

3.5 Quadratic Fit Search
Quadratic fit search leverages our ability to analytically solve for the minimumof a quadratic function. Many local minima look quadratic when we zoom inclose enough. Quadratic fit search iteratively fits a quadratic function to threebracketing points, solves for the minimum, chooses a new set of bracketing points,
and repeats as shown in figure 3.10.
Given bracketing points a < b < c, we wish to find the coefficients p1 , p2 , andp3 for the quadratic function q that goes through ( a, y a ), (b, yb ), and (c, yc ):q ( x ) = p1 + p2 x + p3 x 2(3.5)

y a = p1 + p2 a + p3 a2(3.6)

y b = p1 + p2 b + p3 b2

(3.7)

2

(3.8)

y c = p1 + p2 c + p3 cyx
Figure 3.10. Quadratic fit search fitsa quadratic function to three bracketing points (black dots) and uses
the analytic minimum (blue dot)
to determine the next set of bracketing points.

In matrix form, we have
  
1
ya
  
 y b  = 1
yc
1

a
b
c

 
a2
p1
 
2
b   p2 c2
p3

We can solve for the coefficients through matrix inversion: −1    
1 a a2
ya
p1
  
  
2
 p2  = 1 b b   y b p3
1 c c2yc(3.9)(3.10)

Our quadratic function is then( x − b)( x − c)( x − a)( x − c)( x − a)( x − b)
+ yb
+ yc
( a − b)( a − c)(b − a)(b − c)(c − a)(c − b)
We can solve for the unique minimum by finding where the derivative is zero:q( x ) = y ax∗ =1 y a ( b2 − c2 ) + y b ( c2 − a2 ) + y c ( a2 − b2 )2y a (b − c) + yb (c − a) + yc ( a − b)(3.11)(3.12)

Quadratic fit search is typically faster than golden section search. It may needsafeguards for cases where the next point is very close to other points. A basic
implementation is provided in algorithm 3.4. Figure 3.11 shows several iterationsof the algorithm.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

44 c ha p te r 3. bra c ke ti n g

Algorithm 3.4. Quadratic fit search
to be run on univariate function f,with bracketing interval [ a, c] witha < b < c. The method will run for
n function evaluations. It returns
the new bracketing values as a tuple, (a, b, c).

yfunction quadratic_fit_search(f, a, b, c, n)ya, yb, yc = f(a), f(b), f(c)for i in 1:n-3
x = 0.5*(ya*(b^2-c^2)+yb*(c^2-a^2)+yc*(a^2-b^2)) /(ya*(b-c)+yb*(c-a)+yc*(a-b))yx = f(x)if x > b
if yx > yb
c, yc = x, yx
else
a, ya, b, yb = b, yb, x, yx
endelseif x < b
if yx > yb
a, ya = x, yx
else
c, yc, b, yb = b, yb, x, yx
end
end
end
return (a, b, c)
end

a

b

c
x

ab
x

c

a b
x

c

ab

c

x
Figure 3.11. Four iterations of thequadratic fit method.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

3. 6 . shu be rt-p i y a vski i me thod45

3.6 Shubert-Piyavskii Method
In contrast with previous methods in this chapter, the Shubert-Piyavskii method3
is a global optimization method over a domain [ a, b], meaning it is guaranteed toconverge on the global minimum of a function irrespective of any local minima
or whether the function is unimodal. A basic implementation is provided byalgorithm 3.5.
The Shubert-Piyavskii method requires that the function be Lipschitz continuous,
meaning that it is continuous and there is an upper bound on the magnitude ofits derivative. A function f is Lipschitz continuous on [ a, b] if there exists an ℓ > 0such that:4| f ( x ) − f (y)| ≤ ℓ| x − y| for all x, y ∈ [ a, b](3.13)

Intuitively, ℓ is as large as the largest unsigned instantaneous rate of change
the function attains on [ a, b]. Given a point ( x0 , f ( x0 )), we know that the lines
f ( x0 ) − ℓ( x − x0 ) for x > x0 and f ( x0 ) + ℓ( x − x0 ) for x < x0 form a lower boundof f .
The Shubert-Piyavskii method iteratively builds a tighter and tighter lower
bound on the function. Given a valid Lipschitz constant ℓ, the algorithm begins bysampling the midpoint, x (1) = ( a + b)/2. A sawtooth lower bound is constructedusing lines of slope ±ℓ from this point. These lines will always lie below f if ℓ isa valid Lipschitz constant as shown in figure 3.12.

y

f (x)
lower bound

S. Piyavskii, “An Algorithm for
Finding the Absolute Extremum
of a Function,” USSR Computational Mathematics and Mathematical Physics, vol. 12, no. 4, pp. 57–67,
1972. B. O. Shubert, “A Sequential
Method Seeking the Global Maximum of a Function,” SIAM Journal
on Numerical Analysis, vol. 9, no. 3,
pp. 379–388, 1972.
3

We can extend the definition of
Lipschitz continuity to multivariate functions, where x and y arevectors and the absolute value isreplaced by any vector norm.
4

Figure 3.12. The first iteration of the
Shubert-Piyavskii method.

ℓ
1

a

x (1)
x

b

Upper vertices in the sawtooth correspond to sampled points. Lower vertices
correspond to intersections between the Lipschitz lines originating from each© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

46 c ha p te r 3. bra c ke ti n gsampled point. Further iterations find the minimum point in the sawtooth, evaluate the function at that x value, and then use the result to update the sawtooth.
Figure 3.13 illustrates this process.
Figure 3.13. Updating the lower
bound involves sampling a new
point and intersecting the newlines with the existing sawtooth.

y

f (x)
lower bounda = x (2)

x (1)
x

x (4)

b = x (3)The algorithm is typically stopped when the difference in height between theminimum sawtooth value and the function evaluation at that point is less thana given tolerance ǫ. For the minimum peak ( x (n) , y(n) ) and function evaluationf ( x (n) ), we thus terminate if y(n) − f ( x (n) ) < ǫ.
The regions in which the minimum could lie can be computed using this update
information. For every peak, an uncertainty region can be computed according
to:


1
1
(3.14)
x (i) − (ymin − y(i) ), x (i) + (ymin − y(i) )ℓℓfor each sawtooth lower vertex ( x (i) , y(i) ) and the minimum sawtooth upper vertex( xmin , ymin ). A point will contribute an uncertainty region only if y(i) < ymin . The
minimum is located in one of these peak uncertainty regions.
The main drawback of the Shubert-Piyavskii method is that it requires knowing
a valid Lipschitz constant. Large Lipschitz constants will result in poor lower
bounds. Figure 3.14 shows several iterations of the Shubert-Piyavskii method.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

3. 6 . shu be rt-p i y a vski i me thod 47

struct Ptxyend
function _get_sp_intersection(A, B, l)t = ((A.y - B.y) - l*(A.x - B.x)) / 2l
return Pt(A.x + t, A.y - t*l)end
function shubert_piyavskii(f, a, b, l, ϵ, δ=0.01)
m = (a+b)/2
A, M, B = Pt(a, f(a)), Pt(m, f(m)), Pt(b, f(b))
pts = [A, _get_sp_intersection(A, M, l),M, _get_sp_intersection(M, B, l), B]
Δ = Infwhile Δ > ϵi = argmin([P.y for P in pts])
P = Pt(pts[i].x, f(pts[i].x))
Δ = P.y - pts[i].yAlgorithm 3.5. The ShubertPiyavskii method to be run on
univariate function f, with bracketing interval a < b and Lipschitz
constant l. The algorithm runs
until the update is less than thetolerance ϵ. Both the best point
and the set of uncertainty intervals
are returned. The uncertainty
intervals are returned as an arrayof (a,b) tuples. The parameter δ
is a tolerance used to merge theuncertainty intervals.

P_prev = _get_sp_intersection(pts[i-1], P, l)
P_next = _get_sp_intersection(P, pts[i+1], l)

end

end

deleteat!(pts, i)insert!(pts, i, P_next)
insert!(pts, i, P)
insert!(pts, i, P_prev)

intervals = []P_min = pts[2*(argmin([P.y for P in pts[1:2:end]])) - 1]
y_min = P_min.y
for i in 2:2:length(pts)if pts[i].y < y_min
dy = y_min - pts[i].yx_lo = max(a, pts[i].x - dy/l)x_hi = min(b, pts[i].x + dy/l)
if !isempty(intervals) && intervals[end][2] + δ ≥ x_lo
intervals[end] = (intervals[end][1], x_hi)
else
push!(intervals, (x_lo, x_hi))
end
end
end
return (P_min, intervals)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

48 c ha p te r 3. bra c ke ti n g

y

y

y

Figure 3.14. Nine iterations of the
Shubert-Piyavskii method proceeding left to right and top to bottom. The blue lines are uncertainty
regions in which the global minimum could lie.

x

x

x

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

3. 7. bi se c ti on me thod 49

3.7

Bisection Method

The bisection method (algorithm 3.6) can be used to find roots of a function, or points
where the function is zero. Such root-finding methods can be used for optimization
by applying them to the derivative of the objective, locating where f ′ ( x ) = 0. In
general, we must ensure that the resulting points are indeed local minima.
The bisection method maintains a bracket [ a, b] in which at least one root is
known to exist. If f is continuous on [ a, b], and there is some y ∈ [ f ( a), f (b)], thenthe intermediate value theorem stipulates that there exists at least one x ∈ [ a, b], suchthat f ( x ) = y as shown in figure 3.15. It follows that a bracket [ a, b] is guaranteed
to contain a zero if f ( a) and f (b) have opposite signs.
The bisection method cuts the bracketed region in half with every iteration.
The midpoint ( a + b)/2 is evaluated, and the new bracket is formed from the
midpoint and whichever side that continues to bracket a zero. We can terminate
immediately if the midpoint evaluates to zero. Otherwise we can terminate after
a fixed number of iterations. Figure 3.16 shows four iterations of the bisection
method. This method is guaranteed to converge within ǫ of x ∗ within lg
iterations, where lg denotes the base 2 logarithm.

f (b)yf ( a)a xb

Figure 3.15. A horizontal line
drawn from any y ∈ [ f ( a), f (b)]must intersect the graph at leastonce.

|b− a|
ǫf′Figure 3.16. Four iterations of thebisection method. The horizontal
line corresponds to f ′ ( x ) = 0. Notethat multiple roots exist within theinitial bracket.
x

x

x

x

Root-finding algorithms like the bisection method require starting intervals[ a, b] on opposite sides of a zero. That is, sign( f ′ ( a)) 6= sign( f ′ (b)), or equivalently, f ′ ( a) f ′ (b) ≤ 0. Algorithm 3.7 provides a method for automatically determining such an interval. It starts with a guess interval [ a, b]. So long as theinterval is invalid, its width is increased by a constant factor. Doubling the interval size is a common choice. This method will not always succeed as shownin figure 3.17. Functions that have two nearby roots can be missed, causing theinterval to infinitely increase without termination.
The Brent-Dekker method is an extension of the bisection method. It is a rootfinding algorithm that combines elements of the secant method (section 6.2) andinverse quadratic interpolation. It has reliable and fast convergence properties, andf′xFigure 3.17. A bracketing method
initialized such that it straddles thetwo roots in this figure will expand
forever, never to find a sign change.
Also, if the initial interval is between the two roots, doubling theinterval can cause both ends of theinterval to simultaneously pass thetwo roots.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

50 c ha p te r 3. bra c ke ti n gfunction bisection(f′, a, b, ϵ)
if a > b; a,b = b,a; end # ensure a < bya, yb = f′(a), f′(b)if ya == 0; b = a; end
if yb == 0; a = b; end
while b - a > ϵx = (a+b)/2
y = f′(x)if y == 0a, b = x, x
elseif sign(y) == sign(ya)
a = x
else
b = x
end
end
end

return (a,b)

function bracket_sign_change(f′, a, b; k=2)
if a > b; a,b = b,a; end # ensure a < bcenter, half_width = (b+a)/2, (b-a)/2
while f′(a)*f′(b) > 0
half_width *= ka = center - half_widthb = center + half_width
end
end

Algorithm 3.6. The bisection algorithm, where f′ is the derivative of the univariate function weseek to optimize. We have a < bthat bracket a zero of f′. The interval width tolerance is ϵ. Callingbisection returns the new bracketed interval [ a, b] as a tuple.
The prime character ′ is not anapostrophe. Thus, f′ is a variablename rather than a transposed vector f. The symbol can be created bytyping \prime and hitting tab.

Algorithm 3.7. An algorithm forfinding an interval in which a signchange occurs. The inputs are thereal-valued function f′ defined on
the real numbers, and starting interval [ a, b]. It returns the new interval as a tuple by expanding theinterval width until there is a signchange between the function evaluated at the interval bounds. The
expansion factor k defaults to 2.

return (a,b)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

3. 8 . su mma ry 51

it is the univariate optimization algorithm of choice in many popular numerical
optimization packages.5

3.8 Summary• Many optimization methods shrink a bracketing interval, including Fibonacci
search, golden section search, and quadratic fit search.
• The Shubert-Piyavskii method outputs a set of bracketed intervals containing
the global minima, given the Lipschitz constant.

The details of this algorithm canbe found in R. P. Brent, Algorithms
for Minimization Without Derivatives.
Prentice Hall, 1973. The algorithm
is an extension of the work by
T. J. Dekker, “Finding a Zero by
Means of Successive Linear Interpolation,” in Constructive Aspects of
the Fundamental Theorem of Algebra,
B. Dejon and P. Henrici, eds., Interscience, 1969.
5• Root-finding methods like the bisection method can be used to find where thederivative of a function is zero.

3.9 Exercises
Exercise 3.1. Give an example of a problem when Fibonacci search is preferredover the bisection method.
Exercise 3.2. What is a drawback of the Shubert-Piyavskii method?
Exercise 3.3. Give an example of a nontrivial function where quadratic fit searchwould identify the minimum correctly once the function values at three distinct
points are available.
Exercise 3.4. Suppose we have f ( x ) = x2 /2 − x. Apply the bisection method tofind an interval containing the minimizer of f starting with the interval [0, 1000].
Execute three steps of the algorithm.
Exercise 3.5. Suppose we have a function f ( x ) = ( x + 2)2 on the interval [0, 1].
Is 2 a valid Lipschitz constant for f on that interval?
Exercise 3.6. Suppose we have a unimodal function defined on the interval [1, 32].
After three function evaluations of our choice, will we be able to narrow the
optimum to an interval of at most length 10? Why or why not?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

4 Local Descent
Up to this point, we have focused on optimization involving a single design
variable. This chapter introduces a general approach to optimization involving
multivariate functions, or functions with more than one variable. The focus of thischapter is on how to use local models to incrementally improve a design point
until some convergence criterion is met. We begin by discussing methods that,at each iteration, choose a descent direction based on a local model and thenchoose a step size. We then discuss methods that restrict the step to be within aregion where the local model is believed to be valid. This chapter concludes witha discussion of convergence conditions. The next two chapters will discuss how
to use first- and second-order models built from gradient or Hessian information.

4.1

Descent Direction Iteration

A common approach to optimization is to incrementally improve a design point
x by taking a step that minimizes the objective value based on a local model. The
local model may be obtained, for example, from a first- or second-order Taylor
approximation. Optimization algorithms that follow this general approach arereferred to as descent direction methods. They start with a design point x(1) andthen generate a sequence of points, sometimes called iterates, to converge to a
local minimum.1
The iterative descent direction procedure involves the following steps:
1. Check whether x(k) satisfies the termination conditions. If it does, terminate;
otherwise proceed to the next step.
2. Determine the descent direction d(k) using local information such as the gradientor Hessian. Some algorithms assume kd(k) k = 1, but others do not.

The choice of x(1) can affect thesuccess of the algorithm in finding a minimum. Domain knowledge is often used to choose a reasonable value. When that is notavailable, we can search over thedesign space using the techniquesthat will be covered in chapter 13.

1

54 c ha p te r 4 . l oc a l d e sc e n t

3. Determine the step size or learning rate α(k) . Some algorithms attempt to
optimize the step size so that the step maximally decreases f .2
4. Compute the next design point according to:x( k +1) ← x( k ) + α ( k ) d( k )(4.1)

There are many different optimization methods, each with their own ways ofdetermining α and d.

We use step size to refer to themagnitude of the overall step. Obtaining a new iterate using equation (4.1) with a step size α(k) implies that the descent direction d(k)has unit length. We use learning
rate to refer to a scalar multiple
used on a descent direction vector which does not necessarily haveunit length.

2

4.2 Line Search
For the moment, assume we have chosen a descent direction d, perhaps using oneof the methods discussed in one of the subsequent chapters. We need to choose
the step factor α to obtain our next design point. One approach is to use line search,which selects the step factor that minimizes the one-dimensional function:
minimize f (x + αd)
α

(4.2)

Line search is a univariate optimization problem, which was covered in chapter 3.
We can apply the univariate optimization method of our choice.3 To inform thesearch, we can use the derivative of the line search objective, which is simply the
directional derivative along d at x + αd. Line search is demonstrated in example 4.1
and implemented in algorithm 4.1.
function line_search(f, x, d)objective = α -> f(x + α*d)a, b = bracket_minimum(objective)
α = minimize(objective, a, b)
return x + α*d
end

One disadvantage of conducting a line search at each step is the computational
cost of optimizing α to a high degree of precision. Instead, it is common to quickly
find a reasonable value and then move on, selecting x(k+1) , and then picking a
new direction d(k+1) .

3
The Brent-Dekker method, mentioned in the previous chapter, isa commonly used univariate optimization method. It combines
the robustness of the bisection
method with the speed of the secant method.

Algorithm 4.1. A method for conducting a line search, which finds
the optimal step factor along adescent direction d from design
point x to minimize function f.
The minimize function can be implemented using a univariate optimization algorithm such as the
Brent-Dekker method.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

4 . 3. a p p rox i ma te l i n e se a rc h

55

Some algorithms use a fixed step factor. Large steps will tend to result in fasterconvergence but risk overshooting the minimum. Smaller steps tend to be morestable but can result in slower convergence. A fixed step factor α is sometimes
referred to as a learning rate.
Another method is to use a decaying step factor:α ( k ) = α (1) γ k −1for γ ∈ (0, 1](4.3)

Decaying step factors are especially popular when minimizing noisy objectivefunctions,4 and are commonly used in machine learning applications.

Example 4.1. Line search used tominimize a function along a descent direction.
line search objective

Consider conducting a line search on f ( x1 , x2 , x3 ) = sin( x1 x2 ) + exp( x2 +x3 ) − x3 from x = [1, 2, 3] in the direction d = [0, −1, −1]. The corresponding
optimization problem is:

We will discuss optimization in
the presence of noise and otherforms of uncertainty in chapter 17.

4minimize sin((1 + 0α)(2 − α)) + exp((2 − α) + (3 − α)) − (3 − α)αwhich simplifies to:minimize sin(2 − α) + exp(5 − 2α) + α − 3
α

150
100
50
α∗

0
0

The minimum is at α ≈ 3.127 with x ≈ [1, −1.126, −0.126].

24α

4.3 Approximate Line Search
It is often more computationally efficient to perform more iterations of a descentmethod than to do exact line search at each iteration, especially if the function
and derivative calculations are expensive. Many of the methods discussed so far
can benefit from using approximate line search to find a suitable step size with asmall number of evaluations. Since descent methods must descend, a step size α
may be suitable if it causes a decrease in the objective function value. However, avariety of other conditions may be enforced to encourage faster convergence.
The condition for sufficient decrease5 requires that the step size cause a sufficientdecrease in the objective function value:f (x(k+1) ) ≤ f (x(k) ) + βα∇d(k) f (x(k) )This condition is sometimes referred to as the Armijo condition.
5

(4.4)

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

56 c ha p te r 4 . l oc a l d e sc e n t

with β ∈ [0, 1] often set to β = 1 × 10−4 . Figure 4.1 illustrates this condition. If
β = 0, then any decrease is acceptable. If β = 1, then the decrease has to be atleast as much as what would be predicted by a first-order approximation.
f (x) + β α ∇yf (x) +d f (x)α∇d f (x)

Figure 4.1. The sufficient decrease
condition, the first Wolfe condition,
can always be satisfied by a sufficiently small step size along a descent direction.

f (x + αd)
sufficient decrease0α

If d is a valid descent direction, then there must exist a sufficiently small step
size that satisfies the sufficient decrease condition. We can thus start with a large
step size and decrease it by a constant reduction factor until the sufficient decrease
condition is satisfied. This algorithm is known as backtracking line search6 becauseof how it backtracks along the descent direction. Backtracking line search is shownin figure 4.2 and implemented in algorithm 4.2. We walk through the procedure
in example 4.2.
function backtracking_line_search(f, ∇f, x, d, α; p=0.5, β=1e-4)
y, g = f(x), ∇f(x)while f(x + α*d) > y + β*α*(g⋅d)α *= pend
return α
end

The first condition is insufficient to guarantee convergence to a local minimum.
Very small step sizes will satisfy the first condition but can prematurely converge.
Backtracking line search avoids premature convergence by accepting the largest
satisfactory step size obtained by sequential downscaling and is guaranteed toconverge to a local minimum.
Another condition, called the curvature condition, requires the directional derivative at the next iterate to be shallower:∇ d( k ) f ( x( k + 1) ) ≥ σ ∇ d( k ) f ( x( k ) )Also known as Armijo line search,
L. Armijo, “Minimization of Functions Having Lipschitz Continuous First Partial Derivatives,” Pacific Journal of Mathematics, vol. 16,
no. 1, pp. 1–3, 1966.

6

Algorithm 4.2. The backtracking
line search algorithm, which takes
objective function f, its gradient∇f, the current design point x, a
descent direction d, and the maximum step size α. We can optionallyspecify the reduction factor p andthe first Wolfe condition parameterβ.
Note that the cdot character ⋅aliases to the dot function such thata⋅b is equivalent to dot(a,b). The
symbol can be created by typing\cdot and hitting tab.

(4.5)

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

4 . 3. a p p rox i ma te l i n e se a rc h 57

4

3

5

Figure 4.2. Backtracking line
search used on the Rosenbrockfunction (appendix B.6). The blacklines show the seven iterations
taken by the descent method andthe red lines show the points considered during each line search.

7

6
x2

8

2

1
x1

where σ controls how shallow the next directional derivative must be. Figures 4.3
and 4.4 illustrate this condition. It is common to set β < σ < 1 with σ = 0.1 whenapproximate linear search is used with the conjugate gradient method and to 0.9
when used with Newton’s method.7
An alternative to the curvature condition is the strong curvature condition, which
is a more restrictive criterion in that the slope is also required not to be too positive:|∇d(k) f (x(k+1) )| ≤ −σ∇d(k) f (x(k) )The conjugate gradient method is
introduced in section 5.2, and Newton’s method is introduced in section 6.1.
7

(4.6)

Figure 4.5 illustrates this condition.
Together, the sufficient decrease condition and the curvature condition form
the Wolfe conditions. The sufficient decrease condition is often called the first Wolfe
condition and the curvature condition is called the second Wolfe condition. The
sufficient decrease condition with the strong curvature condition form the strong
Wolfe conditions.
Satisfying the strong Wolfe conditions requires a more complicated algorithm,
strong backtracking line search (algorithm 4.3).8 The method operates in two phases.
The first phase, the bracketing phase, tests successively larger step sizes to bracket
an interval [α(k−1) , α(k) ] guaranteed to contain step lengths satisfying the Wolfe
conditions.

J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed. Springer,
2006.
8© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

58 c ha p te r 4 . l oc a l d e sc e n t

Consider approximate line search on f ( x1 , x2 ) = x12 + x1 x2 + x22 fromx = [1, 2] in the direction d = [−1, −1], using a maximum step size of 10,
a reduction factor of 0.5, a first Wolfe condition parameter β = 1 × 10−4 , anda second Wolfe condition parameter σ = 0.9.
We check whether the maximum step size satisfies the first Wolfe condition,
where the gradient at x is g = [4, 5]:

Example 4.2. An example of backtracking line search, an approximate line search method.

f (x + αd) ≤ f (x) + βα(g⊤ d)f ([1, 2] + 10 · [−1, −1]) ≤ 7 + 1 × 10−4 · 10 · [4, 5]⊤ [−1, −1]
217 ≤ 6.991

It is not satisfied.
The step size is multiplied by 0.5 to obtain 5, and the first Wolfe condition
is checked again:f ([1, 2] + 5 · [−1, −1]) ≤ 7 + 1 × 10−4 · 5 · [4, 5]⊤ [−1, −1]
37 ≤ 6.996

It is not satisfied.
The step size is multiplied by 0.5 to obtain 2.5, and the first Wolfe condition
is checked again:f ([1, 2] + 2.5 · [−1, −1]) ≤ 7 + 1 × 10−4 · 2.5 · [4, 5]⊤ [−1, −1]
3.25 ≤ 6.998

The first Wolfe condition is satisfied.
The candidate design point x′ = x + αd = [−1.5, −0.5] is checked against
the second Wolfe condition:∇ d f (x′ ) ≥ σ ∇ d f (x )[−3.5, −2.5]⊤ [−1, −1] ≥ σ [4, 5]⊤ [−1, −1]
6 ≥ −8.1

The second Wolfe condition is satisfied.
Approximate line search terminates with x = [−1.5, −0.5].

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

4 . 3. a p p rox i ma te l i n e se a rc h 59yFigure 4.3. The curvature condition, the second Wolfe condition, is necessary to ensurethat second-order function approximations have positive curvature,
thereby having a unique global
minimum.

minimum of second-order approximation0αf (x) +
yf (x) +

Figure 4.4. Regions where the curvature condition is satisfied.

ασ∇

α∇

d f (x)d f (x)reduced curvature0αf (x) +
yf (x) +

Figure 4.5. Regions where thestrong curvature condition is satisfied.

ασ∇

α∇

d f (x)d f (x)0α© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

60c ha p te r 4 . l oc a l d e sc e n t

An interval guaranteed to contain step lengths satisfying the Wolfe conditions
is found when one of the following conditions hold:f (x + αd) ≥ f (x)f (x + αd) > f (x) + βα∇d f (x)∇ f (x + αd) ≥ 0

y = f (x)yf (x) +βα∇d f (x)

(4.7)
(4.8)
(4.9)

f (x + αd) ≥ f (x)f (x + αd) > f (x) + βα∇d f (x)∇ f (x + αd) ≥ 0
Wolfe conditions satisfied

0αFigure 4.6. The interval [0, α] isguaranteed to bracket an interval
containing a step length satisfying
the strong Wolfe conditions when
any of these three conditions is
true.

Satisfying equation (4.8) is equivalent to violating the first Wolfe condition,
thereby ensuring that shrinking the step length will guarantee a satisfactory step
length. Similarly, equation (4.7) and equation (4.9) guarantee that the descent
step has overshot a local minimum, and the region between must therefore containa satisfactory step length.
Figure 4.6 shows where each bracketing condition is true for an example line
search. The figure shows bracket intervals [0, α], whereas advanced backtracking
line search successively increases the step length to obtain a bracketing interval[ α ( k −1) , α ( k ) ] .
In the zoom phase, we shrink the interval to find a step size satisfying thestrong Wolfe conditions. The shrinking can be done using the bisection method(section 3.7), updating the interval boundaries according to the same interval
conditions. This process is shown in figure 4.7.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

4 . 4 . tru st re g i on me thod s 61

f (x + αd) ≥ f (x)f (x + αd) > f (x) + βα∇d f (x)∇ f (x + αd) ≥ 0

α (1)
α (2)
y

α (4)
α (3)

Figure 4.7. The first phase of strong
backtracking line search, indicatedwith black open circles, is used tobracket an interval. In this case, thecondition ∇ f (x + α(4) d) ≥ 0 is
triggered, causing the bracketinginterval to be [α(3) , α(4) ]. Then thezoom phase, indicated by the red
open circle, shrinks the bracketedregion until a suitable step length
is found.

α (5)α4.4 Trust Region Methods
Descent methods can place too much trust in their first- or second-order information, which can result in excessively large steps or premature convergence. A
trust region9 is the local area of the design space where the local model is believed
to be reliable. A trust region method, or restricted step method, maintains a local
model of the trust region that both limits the step taken by traditional line searchand predicts the improvement associated with taking the step. If the improvement closely matches the predicted value, the trust region is expanded. If the
improvement deviates from the predicted value, the trust region is contracted.10
Figure 4.8 shows a design point centered within a circular trust region.
Trust region methods first choose the maximum step size and then the step
direction, which is in contrast with line search methods that first choose a step
direction and then optimize the step size. A trust region approach finds thenext step by minimizing a model of the objective function fˆ over a trust region
centered on the current design point x. An example of fˆ is a second-order Taylor
approximation (see appendix C.2). The radius of the trust region, δ, is expanded
and contracted based on how well the model predicts function evaluations. The
next design point x′ is obtained by solving:
minimizefˆ(x′ )

subject tox − x′ ≤ δx′(4.10)

where the trust region is defined by the positive radius δ and a vector norm.11
The equation above is a constrained optimization problem, which is covered inchapter 10.

K. Levenberg, “A Method for
the Solution of Certain Non-Linear
Problems in Least Squares,” Quarterly of Applied Mathematics, vol. 2,
no. 2, pp. 164–168, 1944.

9

A recent review of trust region methods is provided by Y. X.
Yuan, “Recent Advances in Trust
Region Algorithms,” Mathematical Programming, vol. 151, no. 1,
pp. 249–281, 2015.
10

There are a variety of efficientmethods for solving equation (4.10) efficiently. For an
overview of the trust region
method applied to quadraticmodels, see D. C. Sorensen,“Newton’s Method with a Model
Trust Region Modification,” SIAM
Journal on Numerical Analysis,
vol. 19, no. 2, pp. 409–426, 1982.
11© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

62c ha p te r 4 . l oc a l d e sc e n tfunction strong_backtracking(f, ∇, x, d; α=1, β=1e-4, σ=0.1)
y0, g0, y_prev, α_prev = f(x), ∇(x)⋅d, NaN, 0αlo, αhi = NaN, NaN# bracket phase
while true
y = f(x + α*d)if y > y0 + β*α*g0 || (!isnan(y_prev) && y ≥ y_prev)
αlo, αhi = α_prev, α
break
end
g = ∇(x + α*d)⋅d
if abs(g) ≤ -σ*g0
return αelseif g ≥ 0αlo, αhi = α, α_prev
break
end
y_prev, α_prev, α = y, α, 2α
end

end

Algorithm 4.3. Strong backtracking approximate line search forsatisfying the strong Wolfe conditions. It takes as input the objectivefunction f, the gradient function∇, the design point x and direction
d from which line search is conducted, an initial step size α, and
the Wolfe condition parameters
β and σ. The algorithm’s bracket
phase first brackets an interval containing a step size that satisfies thestrong Wolfe conditions. It then reduces this bracketed interval in thezoom phase until a suitable step
size is found. We interpolate withbisection, but other schemes can beused.

# zoom phase
ylo = f(x + αlo*d)while true
α = (αlo + αhi)/2
y = f(x + α*d)if y > y0 + β*α*g0 || y ≥ ylo
αhi = αelse
g = ∇(x + α*d)⋅d
if abs(g) ≤ -σ*g0
return αelseif g*(αhi - αlo) ≥ 0
αhi = αloend
αlo = αend
end

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

4 . 5 . te rmi n a ti on c on d i ti on s 63

Figure 4.8. Trust region methods
constrain the next step to lie within
a local region. The trusted region
is expanded and contracted basedon the predictive performance ofmodels of the objective function.

The trust region radius δ is expanded or contracted based on the local model’s
predictive performance. Trust region methods compare the predicted improvement ∆ypred = f (x) − fˆ(x′ ) to the actual improvement ∆yact = f (x) − f (x′ ):η=f (x ) − f (x′ )actual improvement=
predicted improvement
f (x) − fˆ(x′ )(4.11)The ratio η is close to 1 when the predicted step size matches the actual step
size. If the ratio is too small, such as below a threshold η1 , then the improvement
is considered sufficiently less than expected, and the trust region radius is scaled
down by a factor γ1 < 1. If the ratio is sufficiently large, such as above a thresholdη2 , then our prediction is considered accurate, and the trust region radius isscaled up by a factor γ2 > 1. Algorithm 4.4 provides an implementation andfigure 4.9 visualizes the optimization procedure. Example 4.3 shows how toconstruct noncircular trust regions.

4.5 Termination Conditions
There are four common termination conditions for descent direction methods:• Maximum iterations. We may want to terminate when the number of iterations
k exceeds some threshold kmax . Alternatively, we might want to terminate oncea maximum amount of elapsed time is exceeded.
k > kmax(4.12)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

64

c ha p te r 4 . l oc a l d e sc e n tfunction trust_region_descent(f, ∇f, H, x, k_max;
η1=0.25, η2=0.5, γ1=0.5, γ2=2.0, δ=1.0)
y = f(x)for k in 1 : k_max
x′, y′ = solve_trust_region_subproblem(∇f, H, x, δ)η = (y - f(x′)) / (y - y′) # improvement ratio
if η < η1
δ *= γ1 # scale down trust region
else
x, y = x′, y′ # accept new point
if η > η2
δ *= γ2 # scale up trust region
end
end
end
return x
end

Algorithm 4.4. The trust region
descent method, where f is theobjective function, ∇f produces
the derivative, H produces the
Hessian, x is an initial design
point, and k_max is the number of
iterations. The optional parameters
η1 and η2 determine when thetrust region radius δ is increased
or decreased, and γ1 and γ2
control the magnitude of thechange. An implementation forsolve_trust_region_subproblem

must be provided that solves
equation (4.10). We have provided
an example implementationthat uses a second-order Taylor
approximation about x0 with acircular trust region.

using Convex, SCS
function solve_trust_region_subproblem(∇f, H, x0, δ)
x = Variable(length(x0))
p = minimize(∇f(x0)⋅(x-x0) + quadform(x-x0, H(x0))/2)
p.constraints += norm(x-x0) <= δsolve!(p, SCS.Optimizer)
return (x.value, p.optval)end© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

4 . 5 . te rmi n a ti on c on d i ti on s 65

x2

x(10)

Figure 4.9. Trust region optimization used on the Rosenbrock function (appendix B.6).

x(1)
x1

Trust regions need not be circular. In some cases, certain directions may havehigher trust than others.
A norm can be constructed to produce elliptical regions:kx − x0 kE = (x − x0 )⊤ E(x − x0 )Example 4.3. Trust region optimization need not use circulartrust regions. Additional detail isprovided by J. Nocedal and S. J.
Wright, “Trust-Region Methods,”
in Numerical Optimization. Springer,
2006, pp. 66–100.

with kx − x0 kE ≤ 1 where E is a symmetric matrix that defines the ellipse.

The ellipse matrix E can be updated with each descent iteration, which
can involve more complicated adjustments than scaling the trusted region.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

66

c ha p te r 4 . l oc a l d e sc e n t• Absolute improvement. This termination condition looks at the change in thefunction value over subsequent steps. If the change is smaller than a given
threshold, it will terminate:f (x( k ) ) − f (x( k +1) ) < ǫ a(4.13)• Relative improvement. This termination condition also looks at the change infunction value but uses the step factor relative to the current function value:f (x(k) ) − f (x(k+1) ) < ǫr | f (x(k) )|(4.14)• Gradient magnitude. We can also terminate based on the magnitude of thegradient:k∇ f (x(k+1) )k < ǫg
(4.15)
In cases where multiple local minima are likely to exist, it can be beneficial toincorporate random restarts after our termination conditions are met where we
restart our local descent method from randomly selected initial points.

4.6 Summary• Descent direction methods incrementally descend toward a local optimum.
• Univariate optimization can be applied during line search.
• Approximate line search can be used to identify appropriate descent step sizes.
• Trust region methods constrain the step to lie within a local region that expands
or contracts based on predictive accuracy.
• Termination conditions for descent methods can be based on criteria such as
the change in the objective function value or magnitude of the gradient.

4.7 Exercises
Exercise 4.1. Why is it important to have more than one termination condition?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

4 . 7. e x e rc i se s 67

Exercise 4.2. The first Wolfe condition requiresf (x(k) + αd(k) ) ≤ f (x(k) ) + βα∇d(k) f (x(k) )(4.16)

What is the maximum step length α that satisfies this condition, given that f (x) =5 + x12 + x22 , x(k) = [−1, −1], d = [1, 0], and β = 10−4 ?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

5 First-Order Methods
The previous chapter introduced the general concept of descent direction methods.
This chapter discusses a variety of algorithms that use first-order methods toselect the appropriate descent direction. First-order methods rely on gradient
information to help direct the search for a minimum, which can be obtained using
methods outlined in chapter 2.

5.1

Gradient Descent

An intuitive choice for descent direction d is the direction of steepest descent.
Following the direction of steepest descent is guaranteed to lead to improvement,
provided that the objective function is smooth, the step size is sufficiently small,and we are not already at a point where the gradient is zero.1 The direction ofsteepest descent is the direction opposite the gradient ∇ f , hence the name gradientdescent. For convenience in this chapter, we defineg( k ) = ∇ f (x( k ) )(5.1)

where x(k) is our design point at descent iteration k.
In gradient descent, we typically normalize the direction of steepest descent(see example 5.1):g( k )d( k ) = − ( k )
(5.2)
kg k
Jagged search paths result if we choose a step size that leads to the maximal
decrease in f . In fact, the next direction will always be orthogonal to the current
direction. We can show this as follows:

A point where the gradient is zerois called a stationary point.

1

70 c ha p te r 5 . fi rst-ord e r me thod s

Suppose we have f (x) = x1 x22 . The gradient is ∇ f = [ x22 , 2x1 x2 ]. For x(k) =[1, 2] we get an unnormalized direction of steepest descent d = [−4, −4],
which is normalized to d = [− √1 , − √1 ].
2

Example 5.1. Computing the gradient descent direction.

2

If we optimize the step size at each step, we haveα(k) = arg min f (x(k) + αd(k) )
α

(5.3)

The optimization above implies that the directional derivative equals zero. Using
equation (2.9), we have∇ f (x(k) + αd(k) )⊤ d(k) = 0
We knowd( k +1) = −Hence,∇ f (x(k) + αd(k) )k∇ f (x(k) + αd(k) )kd(k+1)⊤ d(k) = 0

(5.4)

(5.5)
(5.6)

which means that d(k+1) and d(k) are orthogonal.
Narrow valleys aligned with a descent direction are not an issue. When thedescent directions cross over the valley, many steps must be taken in order tomake progress along the valley’s floor as shown in figure 5.1. An implementationof gradient descent is provided by algorithm 5.1.

5.2 Conjugate Gradient
Gradient descent can perform poorly in narrow valleys. The conjugate gradientmethod overcomes this issue by borrowing inspiration from methods for optimizing quadratic functions:minimize f (x) =x

1 ⊤x Ax + b⊤ x + c
2

(5.7)

where A is symmetric and positive definite, and thus f has a unique local minimum (section 1.6.2).
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

5 . 2. c on j u g a te g ra d i e n t 71

x2

Figure 5.1. Gradient descent can
result in zig-zagging in narrowcanyons. Here we see the effecton the Rosenbrock function (appendix B.6).

x1

abstract type DescentMethod end
struct GradientDescent <: DescentMethod
α
end
init!(M::GradientDescent, f, ∇f, x) = M
function step!(M::GradientDescent, f, ∇f, x)
α, g = M.α, ∇f(x)return x - α*gend

Algorithm 5.1. The gradient descent method, which follows the
direction of gradient descent witha fixed learning rate. The step!
function produces the next iterate
whereas the init function doesnothing.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

72 c ha p te r 5 . fi rst-ord e r me thod s

The mutually conjugate vectors are the basis vectors of A. They are generally notorthogonal to one another.
The successive conjugate directions are computed using gradient informationand the previous descent direction. The algorithm starts with the direction ofsteepest descent:d(1) = − g(1)
(5.9)

x2

The conjugate gradient method can optimize n-dimensional quadratic functions in n steps as shown in figure 5.2. Its directions are mutually conjugate withrespect to A:d(i)⊤ A d( j) = 0 for all i 6= j
(5.8)

3
2
1

x1

Figure 5.2. Conjugate gradientdescent converges in n steps
when applied to an n-dimensional
quadratic function.

We then use line search to find the next design point. For quadratic functions, thestep factor α can be computed exactly (example 5.2). The update is then:x(2) = x(1) + α (1) d(1)(5.10)

Suppose we want to derive the optimal step factor for a line search on aquadratic function:
minimize f (x + αd)

Example 5.2. The optimal step factor for a line search on a quadratic
function.

α

We can compute the derivative with respect to α:∂ 1∂ f (x + αd)=(x + αd)⊤ A(x + αd) + b⊤ (x + αd) + c∂α∂α 2= d⊤ A(x + αd) + d⊤ b

= d⊤ (Ax + b) + αd⊤ Ad
Setting∂ f (x+αd)= 0 results in:∂αα=−d⊤ (Ax + b)d⊤ Ad

Subsequent iterations choose d(k+1) based on the current gradient and a contribution from the previous descent direction:d( k ) = − g( k ) + β ( k ) d( k −1)(5.11)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

5 . 2. c on j u g a te g ra d i e n t 73

for scalar parameter β. Larger values of β indicate that the previous descentdirection contributes more strongly.
We can derive the best value for β for a known A, using the fact that d(k) is
conjugate to d(k−1) :d(k)⊤ Ad(k−1) = 0(5.12)=0(5.13)⇒ −g(k) Ad(k−1) + β(k) d(k−1)⊤ Ad(k−1) = 0(5.14)⇒ β(k) =(5.15)

⇒ (−g

(k)

+β

( k ) ( k −1) ⊤

) Ad

d

( k −1)⊤(k )⊤Ad(k−1)gd(k−1)⊤ Ad(k−1)

The conjugate gradient method can be applied to nonquadratic functions as
well. Smooth, continuous functions behave like quadratic functions close to a
local minimum, and the conjugate gradient method will converge very quickly insuch regions.
Unfortunately, we do not know the value of A that best approximates f around(k)
x . Instead, several choices for β(k) tend to work well:
Fletcher-Reeves:2g(k)⊤ g(k)β(k) = (k−1)⊤ (k−1)(5.16)
gg
Polak-Ribière:3β(k) =g(k)⊤g( k ) − g( k −1)

g(k−1)⊤ g(k−1)



E. Polak and G. Ribière, “Notesur la Convergence de MéthodesdeDirections
Conjuguées,”
Revue Française d’informatique etde Recherche Opérationnelle, Série
Rouge, vol. 3, no. 1, pp. 35–43, 1969.
3(5.17)

Convergence for the Polak-Ribière method (algorithm 5.2) can be guaranteed
if we modify it to allow for automatic resets:β ← max( β, 0)

R. Fletcher and C. M. Reeves,“Function Minimization by Conjugate Gradients,” The Computer Journal, vol. 7, no. 2, pp. 149–154, 1964.
2(5.18)

Figure 5.3 shows an example search using this method.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

74 c ha p te r 5 . fi rst-ord e r me thod s

x2

Figure 5.3. The conjugate gradientmethod with the Polak-Ribière update. Gradient descent is shown in
gray.

x1

mutable struct ConjugateGradientDescent <: DescentMethoddgend
function init!(M::ConjugateGradientDescent, f, ∇f, x)
M.g = ∇f(x)M.d = -M.g
return M
endfunction step!(M::ConjugateGradientDescent, f, ∇f, x)
d, g = M.d, M.gg′ = ∇f(x)β = max(0, dot(g′, g′-g)/(g⋅g))d′ = -g′ + β*dx′ = line_search(f, x, d′)M.d, M.g = d′, g′return x′
end

Algorithm 5.2. The conjugate gradient method with the Polak-Ribière
update, where d is the previous
search direction and g is the previous gradient.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

5 . 3. mome n tu m 75

Gradient descent will take a long time to traverse a nearly flat surface as shownin figure 5.4. Allowing momentum to accumulate is one way to speed progress.
We can modify gradient descent to incorporate momentum.
The momentum update equations are:(k)

(5.19)x( k +1) = x( k ) + v( k +1)(5.20)v( k +1)

= βv

(k)

− αg

For β = 0, we recover gradient descent. Momentum can be interpreted as a ballrolling down a nearly horizontal incline. The ball naturally gathers momentum
as gravity causes it to accelerate, just as the gradient causes momentum to accumulate in this descent method. An implementation is provided in algorithm 5.3.
Momentum descent is compared to gradient descent in figure 5.5.
mutable struct Momentum <: DescentMethod
α # learning rate
β # momentum decayv # momentum
end
function init!(M::Momentum, f, ∇f, x)
M.v = zeros(length(x))
return M
endfunction step!(M::Momentum, f, ∇f, x)α, β, v, g = M.α, M.β, M.v, ∇f(x)v[:] = β*v - α*greturn x + v
end− exp(− x2 )5.3 Momentum
0

−0.5
−1
−4 −2

0
x

2

4

Figure 5.4. Regions that are nearlyflat have gradients with small magnitudes and can thus require many
iterations of gradient descent to traverse.

Algorithm 5.3. The momentum
method for accelerated descent.
The first line in step! makes
copies of the scalars α and β,
but creates a reference to the vector v. Thus, the following linev[:] = β*v - α*g modifies theoriginal momentum vector in the
struct M.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

76 c ha p te r 5 . fi rst-ord e r me thod s

gradient descent
momentumx2

Figure 5.5. Gradient descent andthe momentum method comparedon the Rosenbrock function withb = 100; see appendix B.6.

x1

5.4 Nesterov Momentum
One issue of momentum is that the steps do not slow down enough at the bottomof a valley and tend to overshoot the valley floor. Nesterov momentum4 modifies
the momentum algorithm to use the gradient at the projected future position:v(k+1) = βv(k) − α∇ f (x(k) + βv(k) )x( k +1) = x( k ) + v( k +1)(5.21)(5.22)

Y. Nesterov, “A Method of Solving
a Convex Programming Problem
with Convergence Rate O(1/k2 ),”Soviet Mathematics Doklady, vol. 27,
no. 2, pp. 543–547, 1983.
4

An implementation is provided by algorithm 5.4. The Nesterov momentum andmomentum descent methods are compared in figure 5.6.
mutable struct NesterovMomentum <: DescentMethod
α # learning rate
β # momentum decayv # momentum
end
function init!(M::NesterovMomentum, f, ∇f, x)
M.v = zeros(length(x))
return M
endfunction step!(M::NesterovMomentum, f, ∇f, x)α, β, v = M.α, M.β, M.v
v[:] = β*v - α*∇f(x + β*v)
return x + v
end

Algorithm 5.4. Nesterov’s momentum method of accelerated descent.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

5 . 5 . a d a g ra d 77

momentum
Nesterov momentum

x2

Figure 5.6. The momentum and
Nesterov momentum methods
compared on the Rosenbrockfunction with b = 100; seeappendix B.6.

x1

5.5 Adagrad
Momentum and Nesterov momentum update all components of x with the samelearning rate. The adaptive subgradient method, or Adagrad,5 adapts a learning
rate for each component of x. Adagrad dulls the influence of parameters withconsistently high gradients, thereby increasing the influence of parameters with
infrequent updates.6
The Adagrad update step is:( k +1)

xi

(k)

= xi

−

ǫ+

α
q

(k)
g
(k) i
si

(5.23)

where s(k) is a vector whose ith entry is the sum of the squares of the partials,
with respect to xi , up to time step k,
(k)

si


k 
( j) 2= ∑ gi(5.24)

5
J. Duchi, E. Hazan, and Y. Singer,“Adaptive Subgradient Methods
for Online Learning and Stochastic
Optimization,” Journal of Machine
Learning Research, vol. 12, pp. 2121–
2159, 2011.
6
Adagrad excels when the gradient is sparse. The original paper
was motivated by stochastic gradient descent, which picks a random
batch of training data for each iteration from which to compute a gradient. Many deep learning datasetsfor real-world problems produce
sparse gradients where some features occur far less frequently thanothers.

j =1

and ǫ is a small value, on the order of 1 × 10−8 , to prevent division by zero.
Adagrad is far less sensitive to the learning rate parameter α. The learning rate
parameter is typically set to a default value of 0.01. Adagrad’s primary weakness
is that the components of s are each strictly nondecreasing. The accumulated sum
causes the effective learning rate to decrease during training, often becoming
infinitesimally small before convergence. An implementation is provided byalgorithm 5.5.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

78 c ha p te r 5 . fi rst-ord e r me thod s

Algorithm 5.5. The Adagrad accelerated descent method.

mutable struct Adagrad <: DescentMethod
α # learning rateϵ # small values # sum of squared gradientendfunction init!(M::Adagrad, f, ∇f, x)
M.s = zeros(length(x))
return M
endfunction step!(M::Adagrad, f, ∇f, x)α, ϵ, s, g = M.α, M.ϵ, M.s, ∇f(x)s[:] += g.*g
return x - α*g ./ (sqrt.(s) .+ ϵ)
end

5.6

RMSProp

RMSProp7 extends Adagrad to avoid the effect of a monotonically decreasing
learning rate. RMSProp maintains a decaying average of squared gradients. Thisaverage is updated according to:8ŝ(k+1) = γŝ(k) + (1 − γ) g(k) ⊙ g(k)(5.25)

where the decay γ ∈ [0, 1] is typically close to 0.9.
The decaying average of past squared gradients can be substituted into RMSProp’s update equation:9( k +1)

xi

α
(k)
q
g
( k +1) i
ǫ + ŝiα(k)(k)= xi −gǫ + RMS( gi ) i
(k)

= xi

−

(5.26)(5.27)

RMSProp is unpublished andcomes from Lecture 6e of Geoff
Hinton’s Coursera class.
8
The operation a ⊙ b is theelement-wise product between
vectors a and b.
7

The denominator is similar to theroot mean square (RMS) of thegradient component. In this chapter we use RMS( x ) to refer to the
decaying root mean square of thetime series of x.
9

An implementation is provided by algorithm 5.6.

5.7 Adadelta
Adadelta10 is another method for overcoming Adagrad’s monotonically decreasing
learning rate. After independently deriving the RMSProp update, the authors
noticed that the units in the update equations for gradient descent, momentum,

10
M. D. Zeiler, “ADADELTA: An
Adaptive Learning Rate Method,”
ArXiv, no. 1212.5701, 2012.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

5.8. adam

79

Algorithm 5.6. The RMSProp accelerated descent method.

mutable struct RMSProp <: DescentMethod
α # learning rateγ # decay
ϵ # small values # sum of squared gradientendfunction init!(M::RMSProp, f, ∇f, x)
M.s = zeros(length(x))
return M
endfunction step!(M::RMSProp, f, ∇f, x)α, γ, ϵ, s, g = M.α, M.γ, M.ϵ, M.s, ∇f(x)s[:] = γ*s + (1-γ)*(g.*g)
return x - α*g ./ (sqrt.(s) .+ ϵ)
end

and Adagrad do not match. To fix this, they use an exponentially decaying averageof the square updates:( k +1)

xi

(k)

= xi

−

RMS(∆xi ) (k)gǫ + RMS( gi ) i(5.28)

which eliminates the learning rate parameter entirely. An implementation isprovided by algorithm 5.7.

5.8

Adam

The adaptive moment estimation method, or Adam,11 also adapts learning rates toeach parameter (algorithm 5.8). It stores both an exponentially decaying squared
gradient like RMSProp and Adadelta, but also an exponentially decaying gradient
like momentum.
Initializing the gradient and squared gradient to zero introduces a bias. A bias
correction step helps alleviate the issue.12 The equations applied during each

D. Kingma and J. Ba, “Adam: A
Method for Stochastic Optimization,” in International Conference
on Learning Representations (ICLR),
2015.

11

According to the original
paper, good default settings are
α = 0.001, γv = 0.9, γs = 0.999,
and ǫ = 1 × 10−8 .

12© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

80 c ha p te r 5 . fi rst-ord e r me thod s

Algorithm 5.7. The Adadelta accelerated descent method. The small
constant ϵ is added to the numerator as well to prevent progressfrom entirely decaying to zero andto start off the first iteration where∆x = 0.

mutable struct Adadelta <: DescentMethod
γs # gradient decay
γx # update decay
ϵ # small values # sum of squared gradientsu # sum of squared updates
endfunction init!(M::Adadelta, f, ∇f, x)
M.s = zeros(length(x))
M.u = zeros(length(x))
return M
endfunction step!(M::Adadelta, f, ∇f, x)
γs, γx, ϵ, s, u, g = M.γs, M.γx, M.ϵ, M.s, M.u, ∇f(x)s[:] = γs*s + (1-γs)*g.*g
Δx = - (sqrt.(u) .+ ϵ) ./ (sqrt.(s) .+ ϵ) .* gu[:] = γx*u + (1-γx)*Δx.*Δx
return x + Δx
end

iteration for Adam are:biased decaying momentum: v(k+1) = γv v(k) + (1 − γv )g(k)(5.29)
biased decaying sq. gradient: s(k+1) = γs s(k) + (1 − γs ) g(k) ⊙ g(k) (5.30)(5.31)

corrected decaying momentum: v̂(k+1) = v(k+1) /(1 − γvk )corrected decaying sq. gradient: ŝ( k +1)=s( k +1)/(1 − γsk )next iterate: x(k+1) = x(k) − αv̂(k+1) / ǫ +(5.32)pŝ(k+1)(5.33)

5.9 Hypergradient Descent
The accelerated descent methods are either extremely sensitive to the learning rateor go to great lengths to adapt the learning rate during execution. The learning
rate dictates how sensitive the method is to the gradient signal. A rate that is toohigh or too low often drastically affects performance.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

5 . 9. hy p e rg ra d i e n t d e sc e n t 81

mutable struct Adam <: DescentMethod
α # learning rate
γv # decay
γs # decay
ϵ # small value
k # step counterv # 1st moment estimates # 2nd moment estimate
endfunction init!(M::Adam, f, ∇f, x)
M.k = 0M.v = zeros(length(x))
M.s = zeros(length(x))
return M
endfunction step!(M::Adam, f, ∇f, x)
α, γv, γs, ϵ, k = M.α, M.γv, M.γs, M.ϵ, M.ks, v, g = M.s, M.v, ∇f(x)v[:] = γv*v + (1-γv)*gs[:] = γs*s + (1-γs)*g.*g
M.k = k += 1
v_hat = v ./ (1 - γv^k)s_hat = s ./ (1 - γs^k)
return x - α*v_hat ./ (sqrt.(s_hat) .+ ϵ)end

Algorithm 5.8. The Adam accelerated descent method.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

82 c ha p te r 5 . fi rst-ord e r me thod sHypergradient descent13 was developed with the understanding that the derivative of the learning rate should be useful for improving optimizer performance. A
hypergradient is a derivative taken with respect to a hyperparameter. Hypergradient algorithms reduce the sensitivity to the hyperparameter, allowing it to adapt
more quickly.
Hypergradient descent applies gradient descent to the learning rate of anunderlying descent method. The method requires the partial derivative of theobjective function with respect to the learning rate. For gradient descent, thispartial derivative is:∂ f (x( k +1) )( k +1) ⊤ ∂(k)(k) (k)
=
(
g
)
x
−
α
g
(k)
∂α(k)
∂α


= ( g( k +1) ) ⊤ − g( k )A. G. Baydin, R. Cornish, D. M.
Rubio, M. Schmidt, and F. Wood,“Online Learning Rate Adaptation
with Hypergradient Descent,” inInternational Conference on Learning
Representations (ICLR), 2018.
13(5.34)(5.35)

Computing the hypergradient thus requires keeping track of the last gradient.
The resulting update rule is:∂ f (x( k ) )∂α(k−1)( k −1)=α+ µ (g( k ) ) ⊤ g( k −1)

α ( k ) = α ( k −1) − µ(5.36)(5.37)

where µ is the hypergradient learning rate.
This derivation can be applied to any gradient-based descent method that follows equation (4.1). Implementations are provided for the hypergradient versionsof gradient descent (algorithm 5.9) and Nesterov momentum (algorithm 5.10).
These methods are visualized in figure 5.7.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

5 . 9. hy p e rg ra d i e n t d e sc e n t 83

mutable struct HyperGradientDescent <: DescentMethod
α0 # initial learning rateμ # learning rate of the learning rate
α # current learning rate
g_prev # previous gradientendfunction init!(M::HyperGradientDescent, f, ∇f, x)
M.α = M.α0
M.g_prev = zeros(length(x))
return M
endfunction step!(M::HyperGradientDescent, f, ∇f, x)
α, μ, g, g_prev = M.α, M.μ, ∇f(x), M.g_prev
α = α + μ*(g⋅g_prev)
M.g_prev, M.α = g, α
return x - α*gend

Algorithm 5.9. The hypergradient
form of gradient descent.

mutable struct HyperNesterovMomentum <: DescentMethod
α0 # initial learning rateμ # learning rate of the learning rate
β # momentum decayv # momentum
α # current learning rate
h_prev # previous gradientendfunction init!(M::HyperNesterovMomentum, f, ∇f, x)
M.α = M.α0
M.v = zeros(length(x))
M.h_prev = zeros(length(x))
return M
endfunction step!(M::HyperNesterovMomentum, f, ∇f, x)α, β, μ, v = M.α, M.β, M.μ, M.v
h, h_prev = ∇f(x + β*v), M.h_prev
α = α + μ*(h⋅h_prev)v[:] = β*v - α*hM.h_prev, M.α = h, αreturn x + v
end

Algorithm 5.10. The hypergradient
form of the Nesterov momentum
descent method.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

84 c ha p te r 5 . fi rst-ord e r me thod s

hypergradienthyper-Nesterov

x2

Figure 5.7. Hypergradient versions of gradient descent and Nesterov momentum compared on the
Rosenbrock function with b = 100;
see appendix B.6.

x1

5.10

Summary• Gradient descent follows the direction of steepest descent.
• The conjugate gradient method can automatically adjust to local valleys.
• Descent methods with momentum build up progress in favorable directions.
• A wide variety of accelerated descent methods use special techniques to speed
up descent.
• Hypergradient descent applies gradient descent to the learning rate of anunderlying descent method.

5.11 Exercises
Exercise 5.1. Compute the gradient of x⊤ Ax + b⊤ x when A is symmetric.
Exercise 5.2. Apply gradient descent with a unit step factor to f ( x ) = x4 from astarting point of your choice. Compute two iterations.
Exercise 5.3. Apply one step of gradient descent to f ( x ) = e x + e− x from x (1) = 10with both a unit step factor and with exact line search.
Exercise 5.4. The conjugate gradient method can also be used to find a search
direction d when a local quadratic model of a function is available at the current
point. With d as search direction, let the model beq(d) = d⊤ Hd + b⊤ d + c© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

5 . 11. e x e rc i se s 85

for a symmetric matrix H. What is the Hessian in this case? What is the gradientof q when d = 0? What can go wrong if the conjugate gradient method is applied
to the quadratic model to get the search direction d?
Exercise 5.5. How is Nesterov momentum an improvement over momentum?
Exercise 5.6. In what way is the conjugate gradient method an improvement oversteepest descent?
Exercise 5.7. In conjugate gradient descent, what is the descent direction at
the first iteration for the function f ( x, y) = x2 + xy + y2 + 5 when initializedat ( x, y) = (1, 1)? What is the resulting point after two steps of the conjugate
gradient method?
Exercise 5.8. We have a polynomial function f such that f (x) > 2 for all x inthree-dimensional Euclidean space. Suppose we are using steepest descent withstep lengths optimized at each step, and we want to find a local minimum of f .
If our unnormalized descent direction is [1, 2, 3] at step k, is it possible for ourunnormalized descent direction at step k + 1 to be [0, 0, −3]? Why or why not?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

6 Second-Order Methods
The previous chapter focused on optimization methods that involve first-orderapproximations of the objective function using the gradient. This chapter focuseson leveraging second-order approximations that use the second derivative in univariate optimization or the Hessian in multivariate optimization to direct the
search. This additional information can help improve the local model used for
informing the selection of directions and step lengths in descent algorithms.

6.1 Newton’s Method
Knowing the function value and gradient for a design point can help determine
the direction to travel, but this first-order information does not directly help
determine how far to step to reach a local minimum. Second-order information,on the other hand, allows us to make a quadratic approximation of the objectivefunction and approximate the right step size to reach a local minimum as shownin figure 6.1. As we have seen with quadratic fit search in chapter 3, we cananalytically obtain the location where a quadratic approximation has a zero
gradient. We can then use that location as the next iteration to approach a local
minimum.
In univariate optimization, the quadratic approximation about a point x (k)comes from the second-order Taylor expansion:q ( x ) = f ( x (k) ) + ( x − x (k) ) f ′ ( x (k) ) +( x − x (k) )2 ′′ (k)f (x )
2(6.1)

Figure 6.1. A comparison of firstorder and second-order approximations. Bowl-shaped quadratic
approximations have unique locations where the derivative is zero.

f

f

88 c ha p te r 6 . se c on d -ord e r me thod sxx

Setting the derivative to zero and solving for the root yields the update equation
for Newton’s method:∂q( x ) = f ′ ( x (k) ) + ( x − x (k) ) f ′′ ( x (k) ) = 0∂xf ′ ( x (k) )x ( k +1) = x ( k ) −f ′′ ( x (k) )

(6.2)
(6.3)

This update is shown in figure 6.2.

f

f′

Figure 6.2. Newton’s method canbe interpreted as a root-finding
method applied to f ′ that iteratively improves a univariate design
point by taking the tangent line at( x, f ′ ( x )), finding the intersection
with the x-axis, and using that x
value as the next design point.

0x ( k +1)xx (k)x ( k +1)xx (k)The update rule in Newton’s method involves dividing by the second derivative.
The update is undefined if the second derivative is zero, which occurs whenthe quadratic approximation is a line. Instability also occurs when the second
derivative is very close to zero, in which case the next iterate will lie very far fromthe current design point, far from where the local quadratic approximation isvalid. Poor local approximations can lead to poor performance with Newton’s
method. Figure 6.3 shows three kinds of failure cases.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

6 . 1. n e w ton ’s me thod

Oscillation

89

Negative f ′′fOvershoot

x

x (k)

x ( k +1)x ( k +1)xx

Newton’s method does tend to converge quickly when in a bowl-like region
that is sufficiently close to a local minimum. It has quadratic convergence, meaning
the difference between the minimum and the iterate is approximately squared
with every iteration. This rate of convergence holds for Newton’s method startingfrom x (1) within a distance δ of a root x ∗ if1• f ′′ ( x ) 6= 0 for all points in I,• f ′′′ ( x ) is continuous on I, and
•

x (k)

′′′ (1)f ′′′ ( x ∗ )1 f (x )2 | f ′′ ( x (1) ) | < c | f ′′ ( x ∗ ) | for some c < ∞Figure 6.3. Examples of failurecases with Newton’s method.

The final condition enforces sufficient closeness, ensuring that thefunction is sufficiently approximated by the Taylor expansion.
J. Stoer and R. Bulirsch, Introduction to Numerical Analysis, 3rd ed.
Springer, 2002.

1

for an interval I = [ x ∗ − δ, x ∗ + δ]. The final condition guards against overshoot.
Newton’s method can be extended to multivariate optimization (algorithm 6.1).
The multivariate second-order Taylor expansion at x(k) is:1
f ( x ) ≈ q (x ) = f ( x( k ) ) + ( g( k ) ) ⊤ (x − x( k ) ) + (x − x( k ) ) ⊤ H( k ) (x − x( k ) )
2

(6.4)

where g(k) and H(k) are the gradient and Hessian at x(k) , respectively.
We evaluate the gradient and set it to zero:∇ q (x ) = g( k ) + H( k ) (x − x( k ) ) = 0(6.5)

We then solve for the next iterate, thereby obtaining Newton’s method in multivariate form:x( k +1) = x( k ) − (H( k ) ) −1 g( k )(6.6)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

90

c hap te r 6 . se c on d -ord e r me thod s

If f is quadratic and its Hessian is positive definite, then the update converges
to the global minimum in one step. For general functions, Newton’s method
is often terminated once x ceases to change by more than a given tolerance.2
Example 6.1 shows how Newton’s method can be used to minimize a function.
With x(1) = [9, 8], we will use Newton’s method to minimize Booth’s function:f (x) = ( x1 + 2x2 − 7)2 + (2x1 + x2 − 5)2

Termination conditions for descent methods are given in chapter 4.

2

Example 6.1. Newton’s method
used to minimize Booth’s function;
see appendix B.2.

The gradient of Booth’s function is:∇ f (x) = [10x1 + 8x2 − 34, 8x1 + 10x2 − 38]
The Hessian of Booth’s function is:"10
H(x) =
8

8
10#

The first iteration of Newton’s method yields:
x

(2)

=x

(1)



− H

(1)

" # "
10
9
−
=
8
8

" # "
10
9
−
g =
8
8
# " #
# −1 "
1
120
8
=
3
114
10

 −1

(1)

8
10

# −1 "

10 · 9 + 8 · 8 − 34
8 · 9 + 10 · 8 − 38#

The gradient at x(2) is zero, so we have converged after a single iteration. The
Hessian is positive definite everywhere, so x(2) is the global minimum.
Newton’s method can also be used to supply a descent direction to line searchor can be modified to use a step factor.3 Smaller steps toward the minimum orline searches along the descent direction can increase the method’s robustness.
The descent direction is:4d(k) = −(H(k) )−1 g(k)

(6.7)

3

See chapter 5.

The descent direction given by
Newton’s method is similar to thenatural gradient or covariant gradient. S. Amari, “Natural Gradient Works Efficiently in Learning,”
Neural Computation, vol. 10, no. 2,
pp. 251–276, 1998.

4© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

6 . 2. se c a n t me thod 91

Algorithm 6.1. Newton’s method,which takes the gradient of thefunction ∇f, the Hessian of the objective function H, an initial point x,
a step size tolerance ϵ, and a maximum number of iterations k_max.

function newtons_method(∇f, H, x, ϵ, k_max)k, Δ = 1, fill(Inf, length(x))
while norm(Δ) > ϵ && k ≤ k_max
Δ = H(x) \ ∇f(x)x -= Δk += 1end
return x
end

6.2 Secant Method
Newton’s method for univariate function minimization requires the first andsecond derivatives f ′ and f ′′ . In many cases, f ′ is known but the second derivative
is not. The secant method (algorithm 6.2) applies Newton’s method using estimatesof the second derivative and thus only requires f ′ . This property makes the secant
method more convenient to use in practice.
The secant method uses the last two iterates to approximate the second derivative:f ′ ( x ( k ) ) − f ′ ( x ( k −1) )f ′′ ( x (k) ) ≈(6.8)
x ( k ) − x ( k −1)
This estimate is substituted into Newton’s method:x ( k +1) ← x ( k ) −x ( k ) − x ( k −1)
f ′ ( x (k) )f ′ ( x ( k ) ) − f ′ ( x ( k −1) )(6.9)

The secant method requires an additional initial design point. It suffers fromthe same problems as Newton’s method and may take more iterations to converge
due to approximating the second derivative.

6.3 Quasi-Newton Methods
Just as the secant method approximates f ′′ in the univariate case, quasi-Newton
methods approximate the inverse Hessian. Quasi-Newton method updates havethe form:x( k +1) ← x( k ) − α ( k ) Q( k ) g( k )(6.10)

where α(k) is a scalar step factor and Q(k) approximates the inverse of the Hessianat x(k) .

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

92

c ha p te r 6 . se c on d -ord e r me thod s

Algorithm 6.2. The secant method
for univariate function minimization. The inputs are the first derivative f′ of the target function, twoinitial points x0 and x1, and thedesired tolerance ϵ. The final xcoordinate is returned.

function secant_method(f′, x0, x1, ϵ)
g0 = f′(x0)Δ = Infwhile abs(Δ) > ϵ
g1 = f′(x1)Δ = (x1 - x0)/(g1 - g0)*g1
x0, x1, g0 = x1, x1 - Δ, g1
end
return x1
end

These methods typically set Q(1) to the identity matrix, and they then apply updates to reflect information learned with each iteration. To simplify the equations
for the various quasi-Newton methods, we define the following:γ( k +1) ≡ g( k +1) − g( k )δ( k +1)≡x( k +1)−x(k)(6.11)(6.12)The Davidon-Fletcher-Powell (DFP) method (algorithm 6.3) uses:5
Q ← Q−Qγγ⊤ Q δδ⊤
+ ⊤
γ⊤ Qγ
δ γ

(6.13)

where all terms on the right hand side are evaluated at the same iteration.
The update for Q in the DFP method has three properties:
1. Q remains symmetric and positive definite.
2. If f (x) = 21 x⊤ Ax + b⊤ x + c, then Q = A−1 . Thus the DFP has the same
convergence properties as the conjugate gradient method.
3. For high-dimensional problems, storing and updating Q can be significant
compared to other methods like the conjugate gradient method.
An alternative to DFP, the Broyden-Fletcher-Goldfarb-Shanno (BFGS) method(algorithm 6.4), uses:6!
!
γ⊤ Qγ δδ⊤δγ⊤ Q + Qγδ⊤+ 1+ ⊤(6.14)Q ← Q−
δ⊤ γ
δ γ
δ⊤ γ

5
The original concept was presented in a technical report, W. C.
Davidon, “Variable Metric Method
for Minimization,” Argonne National Laboratory, Tech. Rep. ANL5990, 1959. It was later published:
W. C. Davidon, “Variable Metric
Method for Minimization,” SIAM
Journal on Optimization, vol. 1, no. 1,
pp. 1–17, 1991. The method was
modified by R. Fletcher and M. J. D.
Powell, “A Rapidly Convergent Descent Method for Minimization,”
The Computer Journal, vol. 6, no. 2,
pp. 163–168, 1963.

R. Fletcher, Practical Methods of Optimization, 2nd ed. Wiley, 1987.

6© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

6 . 3. q u a si -n e w ton me thod s 93

mutable struct DFP <: DescentMethodQ
endfunction init!(M::DFP, f, ∇f, x)
m = length(x)M.Q = Matrix(1.0I, m, m)
return M
endfunction step!(M::DFP, f, ∇f, x)
Q, g = M.Q, ∇f(x)x′ = line_search(f, x, -Q*g)g′ = ∇f(x′)δ = x′ - x
γ = g′ - g
Q[:] = Q - Q*γ*γ'*Q/(γ'*Q*γ) + δ*δ'/(δ'*γ)
return x′
end

Algorithm 6.3. The DavidonFletcher-Powell descent method.

mutable struct BFGS <: DescentMethodQ
endfunction init!(M::BFGS, f, ∇f, x)
m = length(x)M.Q = Matrix(1.0I, m, m)
return M
endfunction step!(M::BFGS, f, ∇f, x)
Q, g = M.Q, ∇f(x)x′ = line_search(f, x, -Q*g)g′ = ∇f(x′)δ = x′ - x
γ = g′ - g
Q[:] = Q - (δ*γ'*Q + Q*γ*δ')/(δ'*γ) +(1 + (γ'*Q*γ)/(δ'*γ))[1]*(δ*δ')/(δ'*γ)
return x′
end

Algorithm 6.4. The BroydenFletcher-Goldfarb-Shanno descentmethod.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

94

c ha p te r 6 . se c on d -ord e r me thod s

BFGS does better than DFP with approximate line search but still uses ann × n dense matrix. For very large problems where space is a concern, the Limitedmemory BFGS method (algorithm 6.5), or L-BFGS, can be used to approximate
BFGS.7 L-BFGS stores the last m values for δ and γ rather than the full inverse
Hessian, where i = 1 indexes the oldest value and i = m indexes the most recent.
The process for computing the descent direction d at x begins by computing
q(m) = ∇ f (x). The remaining vectors q(i) for i from m − 1 down to 1 are computed
using

⊤
δ( i +1) q( i +1)γ( i +1)(6.15)q( i ) = q( i +1) −⊤γ( i +1) δ( i +1)

7
J. Nocedal, “Updating QuasiNewton Matrices with Limited
Storage,” Mathematics of Computation, vol. 35, no. 151, pp. 773–782,
1980.

These vectors are used to compute another m + 1 vectors, starting withz(0) =γ( m ) ⊙ δ( m ) ⊙ q( m )⊤γ( m ) γ( m )(6.16)and proceeding with z(i) for i from 1 to m according to



⊤

⊤
( i −1)( i −1)( i −1)( i −1)δqzγ
z( i ) = z( i −1) + δ( i −1)  ⊤ ( i −1) − ⊤ ( i −1) γ( i −1) δγ( i −1) δ(6.17)The descent direction is d = −z(m) .
For minimization, the inverse Hessian Q must remain positive definite. The
initial Hessian is often set to the diagonal of⊤γ(1) δ(1)Q(1) =
⊤γ(1) γ(1)(6.18)

Computing the diagonal for the above expression and substituting the result intoz(1) = Q(1) q(1) results in the equation for z(1) .
The quasi-Newton methods discussed in this section are compared in figure 6.4.
They often perform quite similarly.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

6 . 4 . su mma ry 95

Figure 6.4. Several quasi-Newton
methods compared on the Rosenbrock function; see appendix B.6.
All methods have nearly identical
updates, with L-BFGS noticeably
deviating only when its history, m,
is 1.

x2

DFP
BFGS
L-BFGS (m = 3)
L-BFGS (m = 2)
L-BFGS (m = 1)

x1

6.4 Summary• Incorporating second-order information in descent methods often speeds convergence.
• Newton’s method is a root-finding method that leverages second-order information to quickly descend to a local minimum.
• The secant method and quasi-Newton methods approximate Newton’s method
when the second-order information is not directly available.

6.5

Exercises

Exercise 6.1. What advantage does second-order information provide about
convergence that first-order information lacks?
Exercise 6.2. When finding roots in one dimension, when would we use Newton’s
method instead of the bisection method?
Exercise 6.3. Apply Newton’s method to f ( x ) = x2 from a starting point of yourchoice. How many steps do we need to converge?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

96 c ha p te r 6 . se c on d -ord e r me thod smutable struct LimitedMemoryBFGS <: DescentMethod
m
δs
γs
qs
end
function init!(M::LimitedMemoryBFGS, f, ∇f, x)
M.δs = []M.γs = []M.qs = []return M
endfunction step!(M::LimitedMemoryBFGS, f, ∇f, x)
δs, γs, qs, g = M.δs, M.γs, M.qs, ∇f(x)m = length(δs)
if m > 0q = g
for i in m : -1 : 1
qs[i] = copy(q)q -= (δs[i]⋅q)/(γs[i]⋅δs[i])*γs[i]endz = (γs[m] .* δs[m] .* q) / (γs[m]⋅γs[m])for i in 1 : m
z += δs[i]*(δs[i]⋅qs[i] - γs[i]⋅z)/(γs[i]⋅δs[i])endx′ = line_search(f, x, -z)else
x′ = line_search(f, x, -g)
end
g′ = ∇f(x′)
push!(δs, x′ - x); push!(γs, g′ - g)push!(qs, zeros(length(x)))
while length(δs) > M.m
popfirst!(δs); popfirst!(γs); popfirst!(qs)end
return x′
end

Algorithm 6.5.
The Limitedmemory BFGS descent method,which avoids storing the approximate inverse Hessian. The
parameter m determines the history
size. The LimitedMemoryBFGS typealso stores the step differences
δs, the gradient changes γs, andstorage vectors qs.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

6 . 5 . e x e rc i se s 97

Exercise 6.4. Apply Newton’s method to f ( x ) = 21 x⊤ Hx starting from x(1) =[1, 1]. What have you observed? Use H as follows:#"
10(6.19)
H=
0 1000
Next, apply gradient descent to the same optimization problem by steppingwith the unnormalized gradient. Do two steps of the algorithm. What have you
observed? Finally, apply the conjugate gradient method. How many steps do you
need to converge?
Exercise 6.5. Compare Newton’s method and the secant method on f ( x ) = x2 + x4 ,with x (1) = −3 and x (0) = −4. Run each method for 10 iterations. Make twoplots:
1. Plot f vs. the iteration for each method.
2. Plot f ′ vs. x. Overlay the progression of each method, drawing lines from( x (i) , f ′ ( x (i) )) to ( x (i+1) , 0) to ( x (i+1) , f ′ ( x (i+1) )) for each transition.
What can we conclude about this comparison?
Exercise 6.6. Give an example of a sequence of points x (1) , x (2) , . . . and a function
f such that f ( x (1) ) > f ( x (2) ) > · · · and yet the sequence does not converge to a
local minimum. Assume f is bounded from below.
Exercise 6.7. What is the advantage of a Quasi-Newton method over Newton’s
method?
Exercise 6.8. Give an example where the BFGS update does not exist. What would
you do in this case?
Exercise 6.9. Suppose we have a function f (x) = ( x1 + 1)2 + ( x2 + 3)2 + 4. If westart at the origin, what is the resulting point after one step of Newton’s method?
Exercise 6.10. In this problem we will derive the optimization problem fromwhich the Davidon-Fletcher-Powell update is obtained. Start with a quadratic
approximation at x(k) :
⊤



⊤ 
 1
x − x( k ) H( k ) x − x( k )f ( k ) (x ) = y ( k ) + g( k )x − x( k ) +2© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

98 c ha p te r 6 . se c on d -ord e r me thod s

where y(k) , g(k) , and H(k) are the objective function value, the true gradient, anda positive definite Hessian approximation at x(k) .
The next iterate is chosen using line search to obtain: −1
x( k +1) ← x( k ) − α ( k ) H( k )g( k )

We can construct a new quadratic approximation f (k+1) at x(k+1) . The approximation should enforce that the local function evaluation is correct:f ( k +1) ( x( k +1) ) = y ( k +1)
and that the local gradient is correct:∇ f ( k +1) (x( k +1) ) = g( k +1)
and that the previous gradient is correct:∇ f ( k +1) (x( k ) ) = g( k )Show that updating the Hessian approximation to obtain H(k+1) requires:8
H( k +1) δ( k +1) = γ( k +1)
Then, show that in order for H(k+1) to be positive definite, we require:9δ( k +1)⊤γ( k +1) > 0

Finally, assuming that the curvature condition is enforced, explain why one
then solves the following optimization problem to obtain H(k+1) :10
minimize

H − H( k )subject to

H = H⊤

H

Hδ(k+1) = γ(k+1)

8This condition is called the secant
equation. The vectors δ and γ are
defined in equation (6.11).

This condition is called the curvature condition. It can be enforced
using the Wolfe conditions during
line search.

9

The Davidon-Fletcher-Powell update is obtained by solving suchan optimization problem to obtainan analytical solution and thenfinding the corresponding update
equation for the inverse Hessian
approximation.

10

where H − H(k) is a matrix norm that defines a distance between H and H(k) .

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

Direct methods rely solely on the objective function f . These methods are also
called zero-order, black box, pattern search, or derivative-free methods. Direct methods
do not rely on derivative information to guide them toward a local minimum
or identify when they have reached a local minimum. They use other criteria tochoose the next search direction and to judge when they have converged.

7.1 Cyclic Coordinate Searchx2

7 Direct Methods

x1

(1)

(2)

(1)

(1)

(7.1)

(2)

(7.2)

x1 = arg min f ( x1 , x2 , x3 , . . . , xn )x1

x2

Cyclic coordinate search, also known as coordinate descent or taxicab search, simply
alternates between coordinate directions for its line search. The search starts froman initial x(1) and optimizes the first input:

Figure 7.1. Cyclic coordinate descent alternates between coordinate directions.

Having solved this, it optimizes the next coordinate:
(3)

(2)

(2)

x2 = arg min f ( x1 , x2 , x3 , . . . , xn )x2This process is equivalent to doing a sequence of line searches along the set of nbasis vectors, where the ith basis vector is all zero except for the ith component,
which has value 1 (algorithm 7.1). For example, the third basis function, denotede(3) , in a four-dimensional space is:e(3) = [0, 0, 1, 0]

(7.3)

x1
Figure 7.2. Above is an example of
how cyclic coordinate search can
get stuck. Moving in either of thecoordinate directions will result
only in increasing f , but moving
diagonally, which is not allowed incyclic coordinate search, can resultin lowering f .

c ha p te r 7. d i re c t me thod sbasis(i, n) = [k == i ? 1.0 : 0.0 for k in 1 : n]Figure 7.1 shows an example of a search through a two-dimensional space.
Like steepest descent, cyclic coordinate search is guaranteed either to improveor to remain the same with each iteration. No significant improvement after a fullcycle over all coordinates indicates that the method has converged. Algorithm 7.2
provides an implementation. As figure 7.2 shows, cyclic coordinate search canfail to find even a local minimum.
function cyclic_coordinate_descent(f, x, ϵ)Δ, n = Inf, length(x)while abs(Δ) > ϵx′ = copy(x)for i in 1 : n
d = basis(i, n)x = line_search(f, x, d)end
Δ = norm(x - x′)
end
return x
end

The method can be augmented with an acceleration step to help traverse
diagonal valleys. For every full cycle starting with optimizing x(1) along e(1)and ending with x(n+1) after optimizing along e(n) , an additional line search is
conducted along the direction x(n+1) − x(1) . An implementation is provided inalgorithm 7.3 and an example search trajectory is shown in figure 7.3.

7.2 Powell’s Method

Algorithm 7.1. A function for constructing the ith basis vector oflength n.

Algorithm 7.2. The cyclic coordinate descent method takes as input the objective function f and astarting point x, and it runs until
the step size over a full cycle is lessthan a given tolerance ϵ.

x2

100

x1
original
accelerated
Figure 7.3. Adding the acceleration step to cyclic coordinate descent helps traverse valleys. Six
steps are shown for both the original and accelerated versions.

Powell’s method was first introduced by M. J. D. Powell, “An Efficient Method for Finding the
Powell’s method1 can search in directions that are not orthogonal to each other. The Minimum of a Function of Sevmethod can automatically adjust for long, narrow valleys that might otherwise eral Variables Without Calculating Derivatives,” Computer Journal,
require a large number of iterations for cyclic coordinate descent or other methods vol. 7, no. 2, pp. 155–162, 1964.
An overview is presented by W. H.
that search in axis-aligned directions.
Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical
Recipes in C: The Art of Scientific
Computing. Cambridge University
Press, 1982, vol. 2.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com
1

7. 2. p ow e l l ’s me thod 101

The algorithm maintains a list of search directions u(1) , . . . , u(n) , which are
initially the coordinate basis vectors, u(i) = e(i) for all i. Starting at x(1) , Powell’smethod conducts a line search for each search direction in succession, updating
the design point each time:x(i+1) ← line_search( f , x(i) , u(i) ) for all i in {1, . . . , n}Algorithm 7.3. The cyclic coordinate descent method with an acceleration step takes as input theobjective function f and a starting
point x, and it runs until the step
size over a full cycle is less than agiven tolerance ϵ.

x2function cyclic_coordinate_descent_with_acceleration_step(f, x, ϵ)Δ, n = Inf, length(x)while abs(Δ) > ϵx′ = copy(x)for i in 1 : n
d = basis(i, n)x = line_search(f, x, d)end
x = line_search(f, x, x - x′) # acceleration stepΔ = norm(x - x′)
end
return x
end

(7.4)

Next, all search directions are shifted down by one index, dropping the oldest
search direction, u(1) :u(i) ← u(i+1) for all i in {1, . . . , n − 1}

(7.5)

u( n ) ← x( n +1) − x(1)

(7.6)

x1
Figure 7.4. Powell’s method starts
the same as cyclic coordinate descent but iteratively learns conjugate directions.

The last search direction is replaced with the direction from x(1) to x(n+1) , whichis the overall direction of progress over the last cycle:

and another line search is conducted along the new direction to obtain a new x(1) .
This process is repeated until convergence. Algorithm 7.4 provides an implementation. Figure 7.4 shows an example search trajectory.
Powell showed that for quadratic functions, after k full iterations the last k
directions will be mutually conjugate. Recall that n line searches along mutuallyconjugate directions will optimize an n-dimensional quadratic function. Thus, nfull iterations of Powell’s method, totaling n(n + 1) line searches, will minimize aquadratic function.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

102

c ha p te r 7. d i re c t me thod sfunction powell(f, x, ϵ)n = length(x)U = [basis(i,n) for i in 1 : n]
Δ = Infwhile Δ > ϵx′ = xfor i in 1 : n
d = U[i]x′ = line_search(f, x′, d)end
for i in 1 : n-1
U[i] = U[i+1]end
U[n] = d = x′ - x
x′ = line_search(f, x′, d)Δ = norm(x′ - x)
x = x′
end
return x
end

Algorithm 7.4. Powell’s method,which takes the objective functionf, a starting point x, and a toleranceϵ.

The procedure of dropping the oldest search direction in favor of the overall
direction of progress can lead the search directions to become linearly dependent.
Without search vectors that are linearly independent, the search directions canno longer cover the full design space, and the method may not be able to find theminimum. This weakness can be mitigated by periodically resetting the search
directions to the basis vectors. One recommendation is to reset every n or n + 1
iterations.

7.3 Hooke-Jeeves
The Hooke-Jeeves method (algorithm 7.5) traverses the search space based on evaluations at small steps in each coordinate direction.2 At every iteration, the HookeJeeves method evaluates f (x) and f (x ± αe(i) ) for a given step size α in every
coordinate direction from an anchoring point x. It accepts any improvement it may
find. If no improvements are found, it will decrease the step size. The processrepeats until the step size is sufficiently small. Figure 7.5 shows a few iterationsof the algorithm.

R. Hooke and T. A. Jeeves, “Direct
Search Solution of Numerical andStatistical Problems,” Journal of theACM (JACM), vol. 8, no. 2, pp. 212–
229, 1961.
2© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

x1

x1

x2

x2

x2

x2

7. 4 . g e n e ra l i z e d p a tte rn se a rc h 103x1x1

One step of the Hooke-Jeeves method requires 2n function evaluations foran n-dimensional problem, which can be expensive for problems with many
dimensions. The Hooke-Jeeves method is susceptible to local minima. The method
has been proven to converge on certain classes of functions.3

7.4 Generalized Pattern Search

Figure 7.5. The Hooke-Jeeves
method, proceeding left to right.
It begins with a large step size
but then reduces it once it cannot
improve by taking a step in any
coordinate direction.
3
E. D. Dolan, R. M. Lewis, and V.
Torczon, “On the Local Convergence of Pattern Search,” SIAM
Journal on Optimization, vol. 14,
no. 2, pp. 567–583, 2003.

In contrast with the Hooke-Jeeves method, which searches in the coordinate
directions, generalized pattern search can search in arbitrary directions. A pattern P
can be constructed from a set of directions D about an anchoring point x with astep size α according to:

P = {x + αd for each d in D}(7.7)

The Hooke-Jeeves method uses 2n directions for problems in n dimensions, but
generalized pattern search can use as few as n + 1.
For generalized pattern search to converge to a local minimum, certain conditions must be met. The set of directions must be a positive spanning set, whichmeans that we can construct any point in R n using a nonnegative linear combination of the directions in D . A positive spanning set ensures that at least one of thedirections is a descent direction from a location with a nonzero gradient.4
We can determine whether a given set of directions D = {d(1) , d(2) , . . . , d(m) }in R n is a positive spanning set. First, we construct the matrix D whose columns
are the directions in D (see figure 7.6). The set of directions D is a positive spanning set if D has full row rank and if Dx = −D1 with x ≥ 0 has a solution.5 This
optimization problem is identical to the initialization phase of a linear program,which is covered in chapter 11.

Convergence guarantees for generalized pattern search require thatall sampled points fall on a scaled
lattice. Each direction must thus bea product d( j) = Gz( j) for a fixednonsingular n × n matrix G andinteger vector z. V. Torczon, “On
the Convergence of Pattern Search
Algorithms,” SIAM Journal of Optimization, vol. 7, no. 1, pp. 1–25,
1997.
4

R. G. Regis, “On the Properties of
Positive Spanning Sets and Positive
Bases,” Optimization and Engineering, vol. 17, no. 1, pp. 229–262, 2016.
5© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

104 c ha p te r 7. d i re c t me thod sfunction hooke_jeeves(f, x, α, ϵ, γ=0.5)
y, n = f(x), length(x)while α > ϵimproved = false
x_best, y_best = x, y
for i in 1 : n
for sgn in (-1,1)x′ = x + sgn*α*basis(i, n)y′ = f(x′)
if y′ < y_best
x_best, y_best, improved = x′, y′, true
end
end
end
x, y = x_best, y_best

Algorithm 7.5. The Hooke-Jeeves
method, which takes the targetfunction f, a starting point x, a starting step size α, a tolerance ϵ, anda step decay γ. The method runs
until the step size is less than ϵ andthe points sampled along the coordinate directions do not provide an
improvement. Based on the implementation from A. F. Kaupe Jr, “Algorithm 178: Direct Search,” Communications of the ACM, vol. 6, no. 6,
pp. 313–314, 1963.

if !improved
α *= γ
end

end

end
return x

Figure 7.6. A valid pattern for generalized pattern search requires apositive spanning set. These directions are stored in the set D .

only positively spans theconeonly positively spans 1d
space

positively spans R2© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

7. 5 . n e l d e r-me a d si mp l e x me thod 105

Figure 7.7. All previous points ingeneralized pattern search lie on ascaled lattice, or mesh. The lattice is
not explicitly constructed and neednot be axis-aligned.

x (1)x (2) = x (3)x (4) = x (5)The implementation of generalized pattern search in algorithm 7.6 contains
additional enhancements over the original Hooke-Jeeves method.6 First, the implementation is opportunistic—as soon as an evaluation improves the current best
design, it is accepted as the anchoring design point for the next iteration. Second,
the implementation uses dynamic ordering to accelerate convergence—a directionthat leads to an improvement is promoted to the beginning of the list of directions.
Figure 7.7 shows a few iterations of the algorithm.

These enhancements are presented in C. Audet and J. E. Dennis
Jr., “Mesh Adaptive Direct Search
Algorithms for Constrained Optimization,” SIAM Journal on Optimization, vol. 17, no. 1, pp. 188–217,
2006.
6

7.5 Nelder-Mead Simplex Method
The Nelder-Mead simplex method7 uses a simplex to traverse the space in searchof a minimum. A simplex is a generalization of a tetrahedron to n-dimensional
space. A simplex in one dimension is a line, and in two dimensions it is a triangle(see figure 7.8). The simplex derives its name from the fact that it is the simplest
possible polytope in any given space.

7
The original simplex method
is covered in J. A. Nelder and
R. Mead, “A Simplex Method
for Function Minimization,” The
Computer Journal, vol. 7, no. 4,
pp. 308–313, 1965. We incorporate the improvement in J. C. Lagarias, J. A. Reeds, M. H. Wright,
and P. E. Wright, “Convergence
Properties of the Nelder–Mead
Simplex Method in Low Dimensions,” SIAM Journal on Optimization, vol. 9, no. 1, pp. 112–147, 1998.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

106 c ha p te r 7. d i re c t me thod sfunction generalized_pattern_search(f, x, α, D, ϵ, γ=0.5)
y, n = f(x), length(x)while α > ϵimproved = falsefor (i,d) in enumerate(D)x′ = x + α*dy′ = f(x′)
if y′ < yx, y, improved = x′, y′, true
D = pushfirst!(deleteat!(D, i), d)
break
end
end
if !improved
α *= γend
end
return x
end

The Nelder-Mead method uses a series of rules that dictate how the simplex is
updated based on evaluations of the objective function at its vertices. A flowchart
outlines the procedure in figure 7.9, and algorithm 7.7 provides an implementation. Like the Hooke-Jeeves method, the simplex can move around while roughly
maintaining its size, and it can shrink down as it approaches an optimum.
The simplex consist of the points x(1) , . . . , x(n+1) . Let xh be the vertex with thehighest function value, let xs be the vertex with the second highest function value,and let xℓ be the vertex with the lowest function value. Let x̄ be the mean of allvertices except the highest point xh . Finally, for any design point xθ , let yθ = f (xθ )be its objective function value. A single iteration then evaluates four simplex
operations:

Algorithm 7.6. Generalized pattern search, which takes the target function f, a starting point x,
a starting step size α, a set of search
directions D, a tolerance ϵ, and astep decay γ. The method runs until the step size is less than ϵ andthe points sampled along the coordinate directions do not provide an
improvement.

Figure 7.8. A simplex in two dimensions is a triangle. In order forthe simplex to be valid, it musthave a nonzero area.

Reflection. xr = x̄ + α(x̄ − xh ), reflects the highest-valued point over the centroid.
This typically moves the simplex from high regions toward lower regions. Here,α > 0 and is typically set to 1.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

7. 5 . n e l d e r-me a d si mp l e x me thod

107

Initial simplex
Sort the simplex entries
Compute x̄
Compute the reflection point:
xr = x̄ + α(x̄ − xh )yr < y l ?

no

yes

yr ≥ y s ?

yes

no

yr ≥ y h ?

no

yes

Replace xh with xr

Compute the expansion point:
xe = x̄ + β(xr − x̄)Compute the contraction point:
xc = x̄ + γ(xh − x̄)y e < yr ?

yc > yh ?

no

no

yes
Replace xh with xe

Shrink by replacing allx(i) with (x(i) + xℓ )/2

Replace xh with xc

Replace xh with xr

Converged?

yes

yes

Return best point

noFigure 7.9. Flowchart for the
Nelder-Mead algorithm.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

108 c ha p te r 7. d i re c t me thod s

Expansion. xe = x̄ + β(xr − x̄), like reflection, but the reflected point is sent evenfurther. This is done when the reflected point has an objective function valueless than all points in the simplex. Here, β > max(1, α) and is typically set to 2.
Contraction. xc = x̄ + γ(xh − x̄), the simplex is shrunk down by moving awayfrom the worst point. It is parameterized by γ ∈ (0, 1) which is typically set to
0.5.
Shrinkage. All points are moved toward the best point, typically halving theseparation distance.
Figure 7.10 shows the four simplex operations. Figure 7.11 shows several iterationsof the algorithm.
xs

xs

xs
xe

xr
xh

x̄
xℓ
reflection

xh

x̄
xℓ
expansion

xr

xh

xs

xc

xh

x̄
xℓ
contraction

The convergence criterion for the Nelder-Mead simplex method is unlike
Powell’s method in that it considers the variation in the function values rather
than the changes to the points in the design space. It compares the standard
deviation of the sample8 y(1) , . . . , y(n+1) to a tolerance ǫ. This value is high for a
simplex over a highly curved region, and it is low for a simplex over a flat region.
A highly curved region indicates that there is still further optimization possible.

7.6 Divided Rectangles
The divided rectangles algorithm, or DIRECT for DIvided RECTangles, is a Lipschitzian optimization approach, similar in some ways to the Shubert-Piyavskii
method described in section 3.6.9 However, it eliminates the need for specifying a
Lipschitz constant, and it can be more efficiently extended to multiple dimensions.

xℓshrinkage
Figure 7.10. The Nelder-Mead simplex operations visualized in twodimensions.

The standard deviation of the sample
is also called the uncorrected sample
standard
deviation. In our case, it isq2
n +1 ( i )1− ȳ , where ȳ isn +1 ∑ i =1 y
8

the mean of y(1) , . . . , y(n+1) .

D. R. Jones, C. D. Perttunen, andB. E. Stuckman, “Lipschitzian Optimization Without the Lipschitz
Constant,” Journal of Optimization
Theory and Application, vol. 79, no. 1,
pp. 157–181, 1993.
9© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

7. 6 . d i vi d e d re c ta n g l e s 109

Figure 7.11. The Nelder-Meadmethod, proceeding left to rightand top to bottom.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

110 c ha p te r 7. d i re c t me thod sfunction nelder_mead(f, S, ϵ; α=1.0, β=2.0, γ=0.5)
Δ, y_arr = Inf, f.(S)while Δ > ϵp = sortperm(y_arr) # sort lowest to highest
S, y_arr = S[p], y_arr[p]xl, yl = S[1], y_arr[1] # lowest
xh, yh = S[end], y_arr[end] # highest
xs, ys = S[end-1], y_arr[end-1] # second-highest
xm = mean(S[1:end-1]) # centroid
xr = xm + α*(xm - xh) # reflection point
yr = f(xr)Algorithm 7.7. The Nelder-Meadsimplex method, which takes theobjective function f, a starting simplex S consisting of a list of vectors, and a tolerance ϵ. The NelderMead parameters can be specified as well and default to recommended values.

if yr < ylxe = xm + β*(xr-xm) # expansion pointye = f(xe)S[end],y_arr[end] = ye < yr ? (xe, ye) : (xr, yr)
elseif yr ≥ ys
if yr < yh
xh, yh, S[end], y_arr[end] = xr, yr, xr, yr
end
xc = xm + γ*(xh - xm) # contraction point
yc = f(xc)
if yc > yh
for i in 2 : length(y_arr)
S[i] = (S[i] + xl)/2
y_arr[i] = f(S[i])endelse
S[end], y_arr[end] = xc, yc
end
elseS[end], y_arr[end] = xr, yr
end

end

Δ = std(y_arr, corrected=false)
end
return S[argmin(y_arr)]© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

7. 6 . d i vi d e d re c ta n g l e s 111

The notion of Lipschitz continuity can be extended to multiple dimensions. If
f is Lipschitz continuous over a domain X with Lipschitz constant ℓ > 0, thenfor a given design x(1) and y = f (x(1) ), the circular cone
(7.8)

f (x(1) ) − ℓkx − x(1) k2

forms a lower bound of f . Given m function evaluations with design points{x(1) , . . . , x(m) }, we can construct a superposition of these lower bounds by taking
their maximum:
maximize f (x(i) ) − ℓkx − x(i) k2(7.9)
i

divided rectangles lower bound

Lipschitz lower bound
2

x2

1

0

Figure 7.12. The Lipschitz lower
bound is an intersection of cones,
which creates complicated surfaces
in multidimensional space. The divided rectangle lower bound isolates each lower-bound cone toits own hyper-rectangular region,
making it trivial to compute theminimum value in each region
given a Lipschitz constant.

−1
−2
−2

−1

0
x1

1

2

−2

−1

0
x1

1

2

The Shubert-Piyavskii method samples at a point where the bound derived
from a known Lipschitz constant is lowest. Unfortunately, a Lipschitz lower boundhas intricate geometry whose complexity increases with the dimensionality ofthe design space. The left contour plot in figure 7.12 shows such a lower boundusing five function evaluations. The right contour plot shows the approximation
made by DIRECT, which divides the region into hyper-rectangles—one centeredabout each design point. Making this assumption allows for the rapid calculationof the minimum of the lower bound.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

112 c ha p te r 7. d i re c t me thod s

The DIRECT method does not assume a Lipschitz constant is known. Figures 7.13 and 7.14 show lower bounds constructed using Lipschitz continuity
and the DIRECT approximation, respectively, for several different Lipschitz constants. Notice that the location of the minimum changes as the Lipschitz constantchanges, with small values of ℓ leading to designs near the lowest function evaluations and larger values of ℓ leading to designs the farthest from previous function
evaluations.

y

4

3

−4

−2

0
x

2

4

y

4

3

−4

−2

0
x

2

4

Figure 7.13. The Lipschitz lowerf (x)bound for different Lipschitz conℓ = 0.1
stants ℓ. Not only does the estiℓ = 0.2
mated minimum change locally asℓ = 0.3
the Lipschitz constant is varied, theℓ = 0.4
region in which the minimum lies
ℓ = 0.5
can vary as well.
ℓ = 0.6ℓ = 0.7ℓ = 0.8ℓ = 0.9ℓ = 1.0
lower-bound minima
evaluation points

Figure 7.14. The DIRECT lowerf (x)bound for different Lipschitz conℓ = 0.1
stants ℓ. The lower bound is notℓ = 0.2
continuous. The minimum doesℓ = 0.3
not change locally but can changeℓ = 0.4
regionally as the Lipschitz constantℓ = 0.5changes.
ℓ = 0.6ℓ = 0.7ℓ = 0.8ℓ = 0.9ℓ = 1.0
lower-bound minima
evaluation points© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

7. 6 . d i vi d e d re c ta n g l e s 113

7.6.1 Univariate DIRECT
In one dimension, DIRECT recursively divides intervals into thirds and thensamples the objective function at the center of the intervals as shown in figure 7.15.
This scheme is in contrast with the Shubert-Piyavskii method, where sampling
occurs at a point where the bound derived from a known Lipschitz constant islowest.
For an interval [ a, b] with center c = ( a + b)/2, the lower bound based on f (c)is:f ( x ) ≥ f (c) − ℓ| x − c|(7.10)

before division:
after division:

Figure 7.15. Center-point sampling,using the DIRECT scheme, divides
intervals into thirds.

where ℓ is a Lipschitz constant that is unknown to us. The lowest value that thebound obtains on that interval is f (c) − ℓ(b − a)/2, and it occurs at the edges of
the interval.
Even though we do not know ℓ, we can deduce that the lower bound of someintervals are lower than others. If we have two intervals of the same length, forexample, and the first one has a lower evaluation at the center point than thesecond one, then the lower bound of the first interval is lower than that of thesecond. Although this does not entail that the first interval contains a minimizer,
it is an indication that we may want to focus our search in that interval.
During our search, we will have many intervals [ a1 , b1 ], . . . , [ an , bn ] of different
widths. We can plot our intervals according to their center value and interval
width, as we do in figure 7.16. The lower bound for each interval is the verticalintercept of a line of slope ℓ through its center point. The center of the intervalwith the lowest lower bound will be the first point intersected as we shift a linewith slope ℓ upwards from below.
DIRECT splits all intervals for which a Lipschitz constant exists such thatthey have the lowest lower bound, as shown in figure 7.17. We refer to theseselected intervals as potentially optimal. Any interval may technically containthe optimum, though the selected points heuristically have the best chance ofcontaining the optimum.
One iteration of the one-dimensional DIRECT algorithm consists of identifying
the set of potentially optimal intervals and then dividing each interval into thirds.
Example 7.1 demonstrates the univariate DIRECT method.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c ha p te r 7. d i re c t me thod sinterval center value f (c)114(bi − ai )/2

slope = ℓFigure 7.16. Interval selection for
a particular Lipschitz constant ℓ.
Black dots represent DIRECT intervals and the corresponding function evaluations at their centers.
A black line of slope ℓ is drawn
through the dot belonging to the selected interval. All other dots mustlie on or above this line.

f ( ci )f (ci ) − ℓ(bi − ai )/2

interval center value f (c)interval half-width (b − a)/2

Figure 7.17. The potentially optimal intervals for the DIRECT
method form a piecewise boundary that encloses all intervals along
the lower-right. Each dot corresponds to an interval.

suboptimalpotentially optimal

interval half-width (b − a)/2

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

7. 6 . d i vi d e d re c ta n g l e s 115

Consider the function f ( x ) = sin( x ) + sin(2x ) + sin(4x ) + sin(8x ) on theinterval [−2, 2] with a global minimizer near −0.272. Optimization is difficult
because of multiple local minima.
The figure below shows the progression of the univariate DIRECT method,with intervals chosen for splitting rendered in blue. The left side shows theintervals overlaid on the objective function. The right side shows the intervals
as scatter points in the interval half-width versus center value space.

Example 7.1. The DIRECT method
applied to a univariate function.

y

2
0

−2

y

2
0

−2

y

2
0

−2

y

2
0

−2

y

2
0

−2

y

2
0

−2
−2

−1

0
x

1

20

0.5

1

1.5

2

interval half-width© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

116

c ha p te r 7. d i re c t me thod s

7.6.2 Multivariate DIRECT
In multiple dimensions, we divide rectangles (or hyper-rectangles in more than
two dimensions) instead of intervals. Similar to the univariate case, we divide therectangles into thirds along the axis directions. Before commencing the divisionof the rectangles, DIRECT normalizes the search space to be the unit hypercube.
As illustrated in figure 7.18, the choice of ordering of the directions whensplitting the unit hypercube matters. DIRECT prioritizes the assignment of larger
rectangles for the points with lower function evaluations. Larger rectangles areprioritized for additional splitting.
horizontal split

vertical split

vertical split

horizontal split

sampling

When splitting a region without equal side lengths, only the longest dimensions
are split (figure 7.19). Splitting then proceeds on these dimensions in the samemanner as with a hypercube.
The set of potentially optimal intervals is obtained as it is with one dimension.
The lower bound for each hyper-rectangle can be computed based on the longest
center-to-vertex distance and center value. We can construct a diagram similar tofigure 7.16 to identify the potentially optimal rectangles.10

7.6.3 Implementation
DIRECT (algorithm 7.8) is best understood when it is broken down into subroutines. We present these subroutines below.
Normalization to the unit hypercube is done by algorithm 7.9.

Figure 7.18. Interval splitting in
multiple dimensions (using DIRECT) requires choosing an ordering for the split dimensions.

Figure 7.19. DIRECT will split only
the longest dimensions of hyperrectangles.
As an additional requirement for
selection, DIRECT also requiresthat the lower bound for the interval improve the current best value
by a nontrivial amount.
10© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

7. 6 . d i vi d e d re c ta n g l e s 117

function direct(f, a, b, ϵ, k_max)
g = reparameterize_to_unit_hypercube(f, a, b)intervals = Intervals()n = length(a)c = fill(0.5, n)interval = Interval(c, g(c), fill(0, n))add_interval!(intervals, interval)c_best, y_best = copy(interval.c), interval.y

Algorithm 7.8. DIRECT, whichtakes the multidimensional objective function f, vector of lower
bounds a, vector of upper bounds
b, tolerance parameter ϵ, and number of iterations k_max. It returns
the best coordinate.

for k in 1 : k_max
S = get_opt_intervals(intervals, ϵ, y_best)
to_add = Interval[]
for interval in S
append!(to_add, divide(g, interval))dequeue!(intervals[vertex_dist(interval)])end
for interval in to_add
add_interval!(intervals, interval)
if interval.y < y_best
c_best, y_best = copy(interval.c), interval.y
end
end
end
end

return rev_unit_hypercube_parameterization(c_best, a, b)

rev_unit_hypercube_parameterization(x, a, b) = x.*(b-a) + afunction reparameterize_to_unit_hypercube(f, a, b)
Δ = b-areturn x->f(x.*Δ + a)end

Algorithm 7.9. A function that creates a function defined over theunit hypercube that is a reparameterized version of the function fdefined over the hypercube withlower and upper bounds a and b.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

118

c ha p te r 7. d i re c t me thod s

Computing the set of potentially optimal rectangles can be done efficiently(algorithm 7.10). We use the fact that the interval width can only take on powersof one-third, and thus many of the points will share the same x coordinate. For
any given x coordinate, only the one with the lowest y value can be a potentially
optimal interval. We store the rectangular intervals according to their depth andthen in a priority queue according to their centerpoint value.
using DataStructures
struct Intervalcydepthsend
min_depth(interval) = minimum(interval.depths)
vertex_dist(interval) = norm(0.5*3.0.^(-interval.depths), 2)
const Intervals =OrderedDict{Float64,PriorityQueue{Interval, Float64}}function add_interval!(intervals, interval)
d = vertex_dist(interval)
if !haskey(intervals, d)intervals[d] = PriorityQueue{Interval, Float64}()
end
return enqueue!(intervals[d], interval, interval.y)end

Algorithm 7.10. The data structureused in DIRECT. Here, Interval
has three fields: the interval centerc, the center point value y = f(c),and the number of divisions ineach dimension depths. The function add_interval! inserts a new
Interval into the data structure.

We can use this data structure to obtain all potentially optimal intervals (algorithm 7.11). The algorithm proceeds from lowest interval width to highest interval
width. For each point, we first determine whether it is above or below the line
joining the previous two points. If it is below, we skip it. The same determinationis then made for the next point.
Finally, we need a method for dividing the intervals. This is implemented by
the divide method (algorithm 7.12).
The intervals obtained from running DIRECT in two dimensions are visualized
in figure 7.20. Two iterations of DIRECT in two dimensions are worked out inexample 7.2.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

7. 6 . d i vi d e d re c ta n g l e s 119

function get_opt_intervals(intervals, ϵ, y_best)stack = Interval[]for (x, pq) in intervals
if !isempty(pq)interval = DataStructures.peek(pq)[1]y = interval.y

Algorithm 7.11. A routine for obtaining the potentially optimal intervals, where intervals is of typeIntervals, ϵ is a tolerance parameter, and y_best is the best function
evaluation.

while length(stack) > 1
interval1 = stack[end]interval2 = stack[end-1]
x1, y1 = vertex_dist(interval1), interval1.y
x2, y2 = vertex_dist(interval2), interval2.yℓ = (y2 - y) / (x2 - x)
if y1 <= ℓ*(x1-x) + y + ϵ
break
end
# remove previous intervalpop!(stack)end
if !isempty(stack) && interval.y > stack[end].y + ϵ# skip new interval
continue
end
end

end

push!(stack, interval) # add new interval

end
return stack© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

120c ha p te r 7. d i re c t me thod sfunction divide(f, interval)c, d, n = interval.c, min_depth(interval), length(interval.c)dirs = findall(interval.depths .== d)cs = [(c + 3.0^(-d-1)*basis(i,n),c - 3.0^(-d-1)*basis(i,n)) for i in dirs]
vs = [(f(C[1]), f(C[2])) for C in cs]
minvals = [min(V[1], V[2]) for V in vs]

end

7.7

Algorithm 7.12. The divide routine for dividing an interval, where
f is the objective function andinterval is the interval to be divided. It returns a list of the resulting smaller intervals.

intervals = Interval[]
depths = copy(interval.depths)
for j in sortperm(minvals)
depths[dirs[j]] += 1
C, V = cs[j], vs[j]push!(intervals, Interval(C[1], V[1], copy(depths)))
push!(intervals, Interval(C[2], V[2], copy(depths)))
end
push!(intervals, Interval(c, interval.y, copy(depths)))
return intervals

Summary• Direct methods rely solely on the objective function and do not use derivative
information.
• Cyclic coordinate search optimizes one coordinate direction at a time.
• Powell’s method adapts the set of search directions based on the direction ofprogress.
• Hooke-Jeeves searches in each coordinate direction from the current point
using a step size that is adapted over time.
• Generalized pattern search is similar to Hooke-Jeeves, but it uses fewer search
directions that positively span the design space.
• The Nelder-Mead simplex method uses a simplex to search the design space,
adaptively expanding and contracting the size of the simplex in response toevaluations of the objective function.
• The divided rectangles algorithm extends the Shubert-Piyavskii approach tomultiple dimensions and does not require specifying a valid Lipschitz constant.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

7. 7. su mma ry

1

300

250

x2

200

121

Figure 7.20. The DIRECT method
after 16 iterations on the Braninfunction, appendix B.3. Each cell isbordered by white lines. The cells
are much denser around the minima of the Branin function, as the
DIRECT method procedurally increases its resolution in those regions.

150

100

50

0

0

1
x1

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

122

c ha p te r 7. d i re c t me thod s

Consider using DIRECT to optimize the flower function (appendix B.4)
over x1 ∈ [−1, 3], x2 ∈ [−2, 1]. The function is first normalized to the unit
hypercube such that we optimize x1′ , x2′ ∈ [0, 1]:

Example 7.2. The first two iterations of DIRECT worked out in detail.

f ( x1′ , x2′ ) = flower(4x1′ − 1, 3x2′ − 2)
The objective function is sampled at [0.5, 0.5] to obtain 0.158. We have asingle interval with center [0.5, 0.5] and side lengths [1, 1]. The interval is
divided twice, first into thirds in x1′ and then the center interval is divided
into thirds in x2′ .
1
x2′
0

0

x1′

1

We now have five intervals:
centerside lengths

vertex distancecenter value

1
2
3
4
5

[1/6, 3/6]
[5/6, 3/6]
[3/6, 3/6]
[3/6, 1/6]
[3/6, 5/6]

[1/3, 1]
[1/3, 1]
[1/3, 1/3]
[1/3, 1/3]
[1/3, 1/3]

0.527
0.527
0.236
0.236
0.236

0.500
1.231
0.158
2.029
1.861

interval center valueinterval

4
5

2

2
1

1

suboptimalpotentially optimal

3

0
0

1/6

1/2

interval half-width

We next split on the two intervals centered at [1/6, 3/6] and [3/6, 3/6].
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

7. 8 . e x e rc i se s 123

7.8 Exercises
Exercise 7.1. Previous chapters covered methods that leverage the derivative
to descend toward a minimum. Direct methods are able to use only zero-orderinformation—evaluations of f . How many evaluations are needed to approximate
the derivative and the Hessian of an n-dimensional objective function using finite
difference methods? Why do you think it is important to have zero-order methods?
Exercise 7.2. Design an objective function and a starting point x0 such that HookeJeeves will fail to attain a reduction in the objective function. You need to choose
x0 such that it is not a local minimizer.
Exercise 7.3. Is the design point obtained using the Hooke-Jeeves method guaranteed to be within ǫ of a local minimum?
Exercise 7.4. Give an example of a concrete engineering problem where you may
not be able to compute analytical derivatives.
Exercise 7.5. State a difference between the divided rectangles algorithm in one
dimension and the Shubert-Piyavskii method.
Exercise 7.6. Suppose our search algorithm has us transition from x(k) = [1, 2, 3, 4]
to x(k+1) = [2, 2, 2, 4]. Could our search algorithm be (a) cyclic coordinate search,(b) Powell’s method, (c) both a and b, or (d) neither a nor b? Why?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

8Stochastic Methods

This chapter presents a variety of stochastic methods that use randomization strategically to help explore the design space for an optimum. Randomness can help escape local optima and increase the chances of finding a global optimum. Stochastic
methods typically use pseudo-random number generators to ensure repeatability.1
A large amount of randomness is generally ineffective because it prevents us from
effectively using previous evaluation points to help guide the search. This chapter
discusses a variety of ways to control the degree of randomness in our search.

8.1 Noisy Descent
Adding stochasticity to gradient descent can be beneficial in large nonlinear
optimization problems. Saddle points, where the gradient is very close to zero,can cause descent methods to select step sizes that are too small to be useful. One
approach is to add Gaussian noise at each descent step:2
x(k+1) ← x(k) + αg(k) + ǫ(k)(8.1)

where ǫ(k) is zero-mean Gaussian noise with standard deviation σ. The amountof noise is typically reduced over time. The standard deviation of the noise istypically a decreasing sequence σ(k) such as 1/k.3 Algorithm 8.1 provides animplementation of this method. Figure 8.1 compares descent with and without
noise on a saddle function.
A common approach for training neural networks is stochastic gradient descent,which uses a noisy gradient approximation. In addition to helping traverse pastsaddle points, evaluating noisy gradients using randomly chosen subsets of the
training data4 is significantly less expensive computationally than calculating thetrue gradient at every iteration.

Although pseudo-random number generators produce numbersthat appear random, they are actually a result of a deterministic process. Pseudo-random numbers can be produced through calls
to the rand function. The process
can be reset to an initial state using the seed! function from theRandom.jl package.

1

G. Hinton and S. Roweis,“Stochastic Neighbor Embedding,”
in Advances in Neural Information
Processing Systems (NIPS), 2003.
2

The Hinton and Roweis paper
used a fixed standard deviation for
the first 3,500 iterations and set thestandard deviation to zero thereafter.
3

4

These subsets are called batches.

126 c ha p te r 8 . stoc ha sti c me thod s

Algorithm 8.1. A noisy descentmethod, which augments anotherdescent method with additive
Gaussian noise. The method
takes another DescentMethod
submethod, a noise sequence σ,
and stores the iteration count k.

mutable struct NoisyDescent <: DescentMethod
submethodσkendfunction init!(M::NoisyDescent, f, ∇f, x)
init!(M.submethod, f, ∇f, x)
M.k = 1
return M
endfunction step!(M::NoisyDescent, f, ∇f, x)
x = step!(M.submethod, f, ∇f, x)
σ = M.σ(M.k)
x += σ.*randn(length(x))
M.k += 1
return x
end

∞

∞ 

∑ α(k) = ∞∑ α(k)

k =1

k =1

2

<∞

x2

Convergence guarantees for stochastic gradient descent require that the positive
step sizes be chosen such that:(8.2)

These conditions ensure that the step sizes decrease and allow the method toconverge, but not too quickly so as to become stuck away from a local minimum.

8.2 Mesh Adaptive Direct Search
The generalized pattern search methods covered in section 7.4 restricted local
exploration to a fixed set of directions. In contrast, mesh adaptive direct search uses
random positive spanning directions.5
The procedure used to sample positive spanning sets (see example 8.1) begins
by constructing an initial linearly spanning set in the form
√ of a lower triangularmatrix L. The diagonal terms in L are sampled from ±1/ α(k) , where α(k) is thestep size at iteration k. The lower components of L are sampled from
n
o
p
p
p
−1/ α(k) + 1, −1/ α(k) + 2, . . . , 1/ α(k) − 1
(8.3)

x1
stochastic gradient descentsteepest descentFigure 8.1. Adding stochasticity
to a descent method helps withtraversing saddle points such as
f (x) = x12 − x22 shown here. Dueto the initialization, the steepest
descent method converges to thesaddle point where the gradient is
zero.
5This section follows the lower
triangular mesh adaptive direct
search given by C. Audet and J. E.
Dennis Jr., “Mesh Adaptive Direct
Search Algorithms for Constrained
Optimization,” SIAM Journal on Optimization, vol. 17, no. 1, pp. 188–
217, 2006.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

8 . 2. me sh a d a p ti ve d i re c t se a rc h 127

Consider positive spanning sets constructed from the nonzero directions
d1 , d2 ∈ {−1, 0, 1}. There are 8 positive spanning sets with 3 elements thatcan be constructed from these directions:

Example 8.1. Positive spanningsets for R2 . Note that the lower
triangular generation strategy can
only generate the first two columnsof spanning sets.

The rows and columns of L are then randomly permuted to obtain a matrix Dnwhose columns correspond to n directions√ that linearly span R . The maximum
magnitude among these directions is 1/ α(k) .
Two common methods for obtaining a positive spanning set from the linearlyspanning set are to add one additional direction d(n+1) = − ∑in=1 d(i) and to add
n additional directions d(n+ j) = −d( j) for j in {1, . . . , n}. We use the first method
in algorithm 8.2.
The step size α starts at 1, is always a power of 4, and never exceeds 1. Using a
power of 4 causes the maximum possible step size taken in each iteration to be√√scaled by a factor of 2, as the maximum step size α/ α has length 4m / 4m = 2m
for integer m < 1. The step size is updated according to:α(k) /4
if no improvement was found in this iteration
α ( k +1) ←min(1, 4α(k) ) otherwise(8.4)
Mesh adaptive direct search is opportunistic but cannot support dynamicordering6 since, after a successful iteration, the step size is increased, and anotherstep in the successful direction would lie outside of the mesh. The algorithm
queries a new design point along the accepted descent direction. If f (x(k) =x(k−1) + αd) < f (x(k−1) ), then the queried point is x(k−1) + 4αd = x(k) + 3αd. The

6

See section 7.4.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

128

c ha p te r 8 . stoc ha sti c me thod sfunction rand_positive_spanning_set(α, n)δ = round(Int, 1/sqrt(α))
L = Matrix(Diagonal(δ*rand([1,-1], n)))for i in 1 : n-1
for j in 1:i-1
L[i,j] = rand(-δ+1:δ-1)
end
end
D = L[randperm(n),:]
D = D[:,randperm(n)]D = hcat(D, -sum(D,dims=2))
return [D[:,i] for i in 1 : n+1]
end

Algorithm 8.2. Randomly sampling a positive spanning set ofn + 1 directions according to mesh
adaptive direct search with step
size α and number of dimensions
n.

procedure is outlined in algorithm 8.3. Figure 8.2 illustrates how this algorithm
explores the search space.

8.3 Simulated Annealing
Simulated annealing7 borrows inspiration from metallurgy.8 Temperature is used
to control the degree of stochasticity during the randomized search. The temperature starts high, allowing the process to freely move about the search space,with the hope that in this phase the process will find a good region with thebest local minimum. The temperature is then slowly brought down, reducing
the stochasticity and forcing the search to converge to a minimum. Simulated
annealing is often used on functions with many local minima due to its ability toescape local minima.
At every iteration, a candidate transition from x to x′ is sampled from a transition distribution T and is accepted with probability1
if ∆y ≤ 0(8.5)
e−∆y/t if ∆y > 0

7
S. Kirkpatrick, C. D. Gelatt Jr.,and M. P. Vecchi, “Optimization
by Simulated Annealing,” Science,
vol. 220, no. 4598, pp. 671–680,
1983.
8
Annealing is a process in which amaterial is heated and then cooled,making it more workable. Whenhot, the atoms in the material are
more free to move around, and,through random motion, tend tosettle into better positions. A slow
cooling brings the material to anordered, crystalline state. A fast,
abrupt quenching causes defects
because the material is forced tosettle in its current condition.

where ∆y = f (x′ ) − f (x) is the difference in the objective and t is the temperature.
It is this acceptance probability, known as the Metropolis criterion, that allows thealgorithm to escape from local minima when the temperature is high.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

8 . 3. si mu l a te d a n n e a l i n g 129

function mesh_adaptive_direct_search(f, x, ϵ)
α, y, n = 1, f(x), length(x)while α > ϵimproved = falsefor (i,d) in enumerate(rand_positive_spanning_set(α, n))x′ = x + α*dy′ = f(x′)
if y′ < yx, y, improved = x′, y′, truex′ = x + 3α*dy′ = f(x′)
if y′ < yx, y = x′, y′
end
break
end
end
α = improved ? min(4α, 1) : α/4
end
return x
end

Algorithm 8.3. Mesh adaptive direct search for an objective function
f , an initial design x, and a tolerance ϵ.

Figure 8.2. Mesh adaptive direct
search proceeding left to right and
top to bottom.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

130 c ha p te r 8 . stoc ha sti c me thod st(k+1) = γt(k)10temperature

The temperature parameter t controls the acceptance probability. An annealingschedule is used to slowly bring down the temperature as the algorithm progresses, as illustrated by figure 8.3. The temperature must be brought down toensure convergence. If it is brought down too quickly, the search method may notcover the portion of the search space containing the global minimum.
It can be shown that a logarithmic annealing schedule of t(k) = t(1) ln(2)/ ln(k + 1)
for the kth iteration is guaranteed to asymptotically reach the global optimum
under certain conditions,9 but it can be very slow in practice. The exponential
annealing schedule, which is more common, uses a simple decay factor:

0100function simulated_annealing(f, x, T, t, k_max)
y = f(x)x_best, y_best = x, y
for k in 1 : k_max
x′ = x + rand(T)y′ = f(x′)Δy = y′ - y
if Δy ≤ 0 || rand() < exp(-Δy/t(k))
x, y = x′, y′end
if y′ < y_best
x_best, y_best = x′, y′end
end
return x_bestend

A more sophisticated algorithm was introduced by Corana et al. in 1987 thatallows for the step size to change during the search.11 Rather than using a fixed102
iteration

104

logarithmic
exponential, γ = 1/4
exponential, γ = 1/2
exponential, γ = 3/4
fast

(8.6)

for some γ ∈ (0, 1). Another common annealing schedule, fast annealing,10 uses atemperature of
t (1)
(8.7)
t(k) =k
A basic implementation of simulated annealing is provided by algorithm 8.4.
Example 8.2 shows the effect different transition distributions and annealingschedules have on the optimization process.

5

Figure 8.3. Several annealing schedules commonly used in simulated
annealing. The schedules have an
initial temperature of 10.
9
B. Hajek, “Cooling Schedules for
Optimal Annealing,” Mathematics
of Operations Research, vol. 13, no. 2,
pp. 311–329, 1988.

Algorithm 8.4. Simulated annealing, which takes as input an objective function f, an initial point x, atransition distribution T, an annealing schedule t, and the number of
iterations k_max.
H. Szu and R. Hartley, “Fast Simulated Annealing,” Physics Letters
A, vol. 122, no. 3-4, pp. 157–162,
1987.
10

A. Corana, M. Marchesi, C. Martini, and S. Ridella, “Minimizing
Multimodal Functions of Continuous Variables with the ‘Simulated Annealing’ Algorithm,” ACM
Transactions on Mathematical Software, vol. 13, no. 3, pp. 262–280,
1987.
11

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

8 . 3. si mu l a te d a n n e a l i n g 131

We can use simulated annealing to optimize Ackley’s function, appendix B.1.
Ackley’s function has many local minima, making it easy for gradient-based
methods to get stuck.
Suppose we start at x(1) = [15, 15] and run 100 iterations. Below we show
the distribution over iterations for multiple runs with different combinationsof three zero-mean, diagonal covariance (σI) Gaussian transition distributions, and three different temperature schedules t(k) = t(1) /k.
σ = 1, t(1) = 1

σ = 1, t(1) = 10

σ = 1, t(1) = 25

σ = 5, t(1) = 1

σ = 5, t(1) = 10

σ = 5, t(1) = 25

σ = 25, t(1) = 1

σ = 25, t(1) = 10

σ = 25, t(1) = 25

Example 8.2. Exploring the effectof distribution variance and temperature on the performance of
simulated annealing. The blue regions indicate the 5% to 95% and
25% to 75% empirical Gaussianquantiles of the objective function
value.

30

y

20
10
0

30

y

20
10
0

30

y

20
10
0
20 40 60 80 100
iteration

20 40 60 80 100
iteration

20 40 60 80 100
iteration

In this case, the spread of the transition distribution has the greatest impact
on performance.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

132 c ha p te r 8 . stoc ha sti c me thod stransition distribution, this adaptive simulated annealing method keeps track ofa separate step size v for each coordinate direction. For a given point x, a cycle ofrandom moves is performed in each coordinate direction i according to:(8.8)

where r is drawn uniformly at random from [−1, 1] and vi is the maximum step
size in the ith coordinate direction. Each new point is accepted according to theMetropolis criterion. The number of accepted points in each coordinate direction
is stored in a vector a.
After ns cycles, the step sizes are adjusted with the aim to maintain an approximately equal number of accepted and rejected designs with an average acceptance
rate near one-half. Rejecting too many moves is a waste of computational effort,
while accepting too many moves indicates that the configuration is evolving too
slowly because candidate points are too similar to the current location. The update
formula used by Corana et al. is: 
ai /ns −0.6

if ai > 0.6ns
v
1
+
c

i
0.4

 i
 −1
vi = vi 1 + ci 0.4− ai /ns(8.9)
if ai < 0.4ns
0.4



v
otherwisei
The ci parameter controls the step variation along each direction and is typically
set to 2 as shown in figure 8.4. An implementation of the update is shown inalgorithm 8.5.
function corana_update!(v, a, c, ns)
for i in 1 : length(v)ai, ci = a[i], c[i]if ai > 0.6nsv[i] *= (1 + ci*(ai/ns - 0.6)/0.4)
elseif ai < 0.4nsv[i] /= (1 + ci*(0.4-ai/ns)/0.4)
end
end
return v
end

Step Multiplication Factor

x′ = x + rvi ei

1+c

1
1
1+ c

0

0.4 0.6
1
Acceptance Ratio ai /ns

Figure 8.4. The step multiplication
factor as a function of acceptance
rate for c = 2.
Algorithm 8.5. The update formula used by Corana et al. in adaptive simulated annealing, where v
is a vector of coordinate step sizes,
a is a vector of the number of accepted steps in each coordinate direction, c is a vector of step scaling
factors for each coordinate direction, and ns is the number of cycles
before running the step size adjustment.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

8 . 4 . c ross-e n trop y me thod

133

Temperature reduction occurs every nt step adjustments, which is every ns · nt
cycles. The original implementation simply multiplies the temperature by a reduction factor.
The process is terminated when the temperature sinks low enough such that
improvement can no longer be expected. Termination occurs when the most recent
function value is no farther than ǫ from the previous nǫ iterations and the best
function value obtained over the course of execution. Algorithm 8.6 provides animplementation and the algorithm is visualized in figure 8.5.

8.4 Cross-Entropy Method
The cross-entropy method,12 in contrast with the methods we have discussed so
far in this chapter, maintains an explicit probability distribution over the design
space.13 This probability distribution, often called a proposal distribution, is used topropose new samples for the next iteration. At each iteration, we sample from the
proposal distribution and then update the proposal distribution to fit a collectionof the best samples. The aim at convergence is for the proposal distribution to
focus on the global optima. Algorithm 8.7 provides an implementation.
The cross-entropy method requires choosing a family of distributions parameterized by θ. One common choice is the family of multivariate normal distributions
parameterized by a mean vector and a covariance matrix. The algorithm also
requires us to specify the number of elite samples, melite , to use when fitting theparameters for the next iteration.
Depending on the choice of distribution family, the process of fitting the distribution to the elite samples can be done analytically. In the case of the multivariate
normal distribution, the parameters are updated according to the maximum
likelihood estimate:µ( k +1) =Σ ( k +1) =1

melite1melitemelite i∑=1melite i∑=1x( i )(8.10)(x(i) − µ(k+1) )(x(i) − µ(k+1) )⊤(8.11)

12
R. Y. Rubinstein and D. P. Kroese,
The Cross-Entropy Method: A Unified
Approach to Combinatorial Optimization, Monte-Carlo Simulation, andMachine Learning. Springer, 2004.

The name of this method comesfrom the fact that the process offitting the distribution involves
minimizing cross-entropy, which isalso called the Kullback–Leibler divergence. Under certain conditions,minimizing the cross-entropy corresponds to finding the maximum
likelihood estimate of the parameters of the distribution.

13

Example 8.3 applies the cross-entropy method to a simple function. Figure 8.6
shows several iterations on a more complex function. Example 8.4 shows thepotential limitation of using a multivariate normal distribution for fitting elite
samples.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

134 c ha p te r 8 . stoc ha sti c me thod sfunction adaptive_simulated_annealing(f, x, v, t, ϵ;
ns=20, nϵ=4, nt=max(100,5length(x)),γ=0.85, c=fill(2,length(x)) )y = f(x)x_best, y_best = x, yy_arr, n, U = [], length(x), Uniform(-1.0,1.0)a,counts_cycles,counts_resets = zeros(n), 0, 0
while true
for i in 1:nx′ = x + basis(i,n)*rand(U)*v[i]y′ = f(x′)Δy = y′ - y
if Δy < 0 || rand() < exp(-Δy/t)x, y = x′, y′a[i] += 1
if y′ < y_best; x_best, y_best = x′, y′; end
end
end
counts_cycles += 1
counts_cycles ≥ ns || continue
counts_cycles = 0
corana_update!(v, a, c, ns)
fill!(a, 0)counts_resets += 1
counts_resets ≥ nt || continue

Algorithm 8.6. The adaptive simulated annealing algorithm, where
f is the multivariate objective function, x is the starting point, v isstarting step vector, t is the starting temperature, and ϵ is the termination criterion parameter. The optional parameters are the numberof cycles before running the step
size adjustment ns, the number ofcycles before reducing the temperature nt, the number of successive temperature reductions to test
for termination nϵ, the temperature reduction coefficient γ, and the
direction-wise varying criterion c.
Below is a flowchart for theadaptive simulated annealing algorithm as presented in the original
paper.
Initialize Parameters
Perform a cycle of random moves,each along one coordinate direction. Accept or reject each pointaccording to the Metropolis criterion. Record the optimum point
reached so far.

t *= γcounts_resets = 0
push!(y_arr, y)
if !(length(y_arr) > nϵ && y_arr[end] - y_best ≤ ϵ &&all(abs(y_arr[end]-y_arr[end-u]) ≤ ϵ for u in 1:nϵ))x, y = x_best, y_best
else
break
end

end

end
return x_best© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

No. cycles ≥ ns ?
yesnoAdjust step vector v.
Reset no. cycles to 0.
No. step adjustments ≥ nt ?
yesnoReduce temperature.
Reset no. adjustments to 0.
Add point to optimum list.
Stopping criterion satisfied?
yes
End

no

135

y

y

8 . 4 . c ross-e n trop y me thod

x

x

x

x
Figure 8.5. Simulated annealing
with an exponentially decaying
temperature, where the histograms
indicate the probability of simulated annealing being at a particular position at that iteration.

using Distributions
function cross_entropy_method(f, P, k_max, m=100, m_elite=10)
for k in 1 : k_max
samples = rand(P, m)order = sortperm([f(samples[:,i]) for i in 1:m])
P = fit(typeof(P), samples[:,order[1:m_elite]])end
return P
end

Algorithm 8.7. The cross-entropy
method, which takes an objectivefunction f to be minimized, a proposal distribution P, an iteration
count k_max, a sample size m, andthe number of samples to use when
refitting the distribution m_elite.
It returns the updated distribution
over where the global minimum islikely to exist.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

136

c ha p te r 8 . stoc ha sti c me thod s

We can use Distributions.jl to represent, sample from, and fit proposal
distributions. The parameter vector θ is replaced by a distribution P. Callingrand(P,m) will produce an n × m matrix corresponding to m samples ofn-dimensional samples from P, and calling fit will fit a new distribution of
the given input type.

Example 8.3. An example of using
the cross-entropy method.

import Random: seed!
import LinearAlgebra: norm
seed!(0) # set random seed for reproducible results
f = x->norm(x)μ = [0.5, 1.5]
Σ = [1.0 0.2; 0.2 2.0]
P = MvNormal(μ, Σ)
k_max = 10
P = cross_entropy_method(f, P, k_max)@show P.μx2

P.μ = [-6.136226751273704e-7, -1.372160422208573e-6]

x1

x1

x1

x1

Figure 8.6. The cross-entropy
method with m = 40 applied to
the Branin function (appendix B.3)
using a multivariate Gaussian proposal distribution. The 10 elite samples in each iteration are in red.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

8 . 5 . n a tu ra l e vol u ti on stra te g i e s 137

The distribution family should be flexible enough to capture the relevant
features of the objective function. Here we show the limitations of using
a normal distribution on a multimodal objective function, which assigns
greater density in between the two minima. A mixture model is able to center
itself over each minimum.
0.3

Example 8.4. The normal distribution is unable to capture multiple
local minima, in contrast to mixture models which can maintainseveral.

f
mixture model fit
normal fit
samples

0.2

0.1

0

8.5

−5

0
x

5

Natural Evolution Strategies

Like the cross-entropy method, natural evolution strategies14 optimize a proposal
distribution parameterized by θ. We have to specify the proposal distribution
family and the number of samples. The aim is to minimize the expectation
Ex∼ p(·|θ) [ f (x)]. Instead of fitting elite samples, evolution strategies apply gradientdescent. The gradient is estimated from the samples:15∇θ Ex∼ p(·|θ) [ f (x)] ===Z∇θ p(x | θ) f (x) dx(8.12)(8.13)Zp (x | θ)∇ p(x | θ) f (x) dx
p (x | θ) θp(x | θ)∇θ log p(x | θ) f (x) dx

Z(8.14)= Ex∼ p(·|θ) [ f (x)∇θ log p(x | θ)](8.15)≈(8.16)m1
f (x(i) )∇θ log p(x(i) | θ)
m i∑
=1

14
I. Rechenberg, Evolutionsstrategie Optimierung technischer Systeme
nach Prinzipien der biologischen Evolution. Frommann-Holzboog, 1973.

This gradient estimation has recently been successfully applied
to proposal distributions represented by deep neural networks.
T. Salimans, J. Ho, X. Chen, andI. Sutskever, “Evolution Strategies as a Scalable Alternative toReinforcement Learning,” ArXiv,
no. 1703.03864, 2017.
15

Although we do not need the gradient of the objective function, we do need thegradient of the log likelihood, log p(x | θ). Example 8.5 shows how to compute© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

138

c ha p te r 8 . stoc ha sti c me thod s

the gradient of the log likelihood for the multivariate normal distribution. The
estimated gradient can be used along with any of the descent methods discussed
in previous chapters to improve θ. Algorithm 8.8 uses gradient descent with afixed step size. Figure 8.7 shows a few iterations of the algorithm.
using Distributions
function natural_evolution_strategies(f, θ, k_max; m=100, α=0.01)
for k in 1 : k_max
samples = [rand(θ) for i in 1 : m]θ -= α*sum(f(x)*∇logp(x, θ) for x in samples)/m
end
return θ
end

Algorithm 8.8. The natural evolution strategies method, whichtakes an objective function f to
be minimized, an initial distribution parameter vector θ, an iteration count k_max, a sample size m,
and a step factor α. An optimized
parameter vector is returned. The
method rand(θ) should sample
from the distribution parameterized by θ, and ∇logp(x, θ) should
return the log likelihood gradient.

8.6 Covariance Matrix Adaptation
Another popular method is covariance matrix adaptation,16 which is also referred toas CMA-ES for covariance matrix adaptation evolutionary strategy. It has similarities
with natural evolution strategies from section 8.5, but the two should not be
confused. This method maintains a covariance matrix and is robust and sample
efficient. Like the cross-entropy method and natural evolution strategies, a distribution is improved over time based on samples. Covariance matrix adaptation
uses multivariate Gaussian distributions.17
Covariance matrix adaptation maintains a mean vector µ, a covariance matrix
Σ, and an additional step-size scalar σ. The covariance matrix only increases ordecreases in a single direction with every iteration, whereas the step-size scalaris adapted to control the overall spread of the distribution. At every iteration, mdesigns are sampled from the multivariate Gaussian:18
x ∼ N (µ, σ2 Σ)(8.17)The designs are then sorted according to their objective function values suchthat f (x(1) ) ≤ f (x(2) ) ≤ · · · ≤ f (x(m) ). A new mean vector µ(k+1) is formed
using a weighted average of the first melite sampled designs:
meliteµ( k +1) ← ∑ wi x( i )It is common to use the phrase
evolution strategies to refer specifically to covariance matrix adaptation.

16

N. Hansen, “The CMA Evolution Strategy: A Tutorial,” ArXiv,
no. 1604.00772, 2016.

17

For optimization in R n , it is recommended to use at least m =4 + ⌊3 ln n⌋ samples per iteration,and melite = ⌊m/2⌋ elite samples.

18(8.18)

i =1© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

8 . 6 . c ova ri a n c e ma tri x a d a p ta ti on

The multivariate normal distribution N (µ, Σ) with mean µ and covarianceΣ is a popular distribution family due to having analytic solutions. The
likelihood in d dimensions has the form


1
− 12
⊤ −1
− 2d
p(x | µ, Σ) = (2π ) |Σ| exp − (x − µ) Σ (x − µ)2where |Σ| is the determinant of Σ. The log likelihood isd1
1
log p(x | µ, Σ) = − log(2π ) − log |Σ| − (x − µ)⊤ Σ−1 (x − µ)
2
2
2

139

Example 8.5. A derivation of thelog likelihood gradient equations
for the multivariate Gaussian distribution. For the original derivation and several more sophisticatedsolutions for handling the positive
definite covariance matrix, see D.
Wierstra, T. Schaul, T. Glasmachers,
Y. Sun, and J. Schmidhuber, “Natural Evolution Strategies,” ArXiv,
no. 1106.4487, 2011.

The parameters can be updated using their log likelihood gradients:∇(µ) log p(x | µ, Σ) = Σ−1 (x − µ)11∇(Σ) log p(x | µ, Σ) = Σ−1 (x − µ)(x − µ)⊤ Σ−1 − Σ−122

Figure 8.7. Natural evolution strategies using multivariate Gaussian
distributions applied to Wheeler’s
Ridge, appendix B.7.

x2

The term ∇(Σ) contains the partial derivative of each entry of Σ with respect
to the log likelihood.
Directly updating Σ may not result in a positive definite matrix, as is required for covariance matrices. One solution is to represent Σ as a productA⊤ A, which guarantees that Σ remains positive semidefinite, and then update A instead. Replacing Σ by A⊤ A and taking the gradient with respect toA yields:hi∇(A) log p(x | µ, A) = A ∇(Σ) log p(x | µ, Σ) + ∇(Σ) log p(x | µ, Σ)⊤

x1

x1

x1

x1

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

140 c ha p te r 8 . stoc ha sti c me thod s

where the first melite weights sum to 1, and all the weights approximately sum to
0 and are ordered largest to smallest:19
melitem∑ wi ≈ 0∑ wi = 1

w1 ≥ w2 ≥ · · · ≥ w m

i =1

i =1

(8.19)

We can approximate the mean update in the cross-entropy method by setting
the first melite weights to 1/melite , and setting the remaining weights to zero. Covariance matrix adaptation instead distributes decreasing weight to all m designs,
including some negative weights. The recommended weighting is obtained bynormalizing
m+1
wi′ = ln
− ln i for i ∈ {1, . . . , m}
(8.20)
2
to obtain w. The positive and negative weights are normalized separately. Figure 8.8 compares the mean updates for covariance matrix adaptation and thecross-entropy method.
The step size is updated using a cumulative variable pσ that tracks steps over
time:
(1)
pσ = 0q(8.21)( k +1)← (1 − cσ )pσ + cσ (2 − cσ )µeff (Σ(k) )−1/2 δw
pσ
where cσ < 1 controls the rate of decay and the right hand term determines
whether the step size should be increased or decreased based on the observed
samples with respect to the present scale of the distribution. The variance effective
selection mass µeff has the form
µeff =

1

In the recommended weighting,the first melite samples are positive, and the remaining samples
are non-positive.

19

Figure 8.8. Shown is an initial
proposal distribution (white contours), six samples (white dots),and the new updated means forboth covariance matrix adaptation(blue dot) and the cross-entropy
method (red dot) using three elite
samples. Covariance matrix adaptation tends to update the mean
more aggressively than the crossentropy method (red dot), as it assigns higher weight to better sampled designs, and negative weight
to worse sampled designs.

(8.22)m2∑i=elite
1 wiand δw is computed from the sampled deviations:meliteδw = ∑ wi δ(i) for δ(i) =i =1x( i ) − µ( k )σ (k)The new step size is obtained according to
 

cσ
k pσ kσ(k+1) ← σ(k) exp−1dσ E kN (0, I)k

where

√ Γ
E kN (0, I)k = 2



n +1
2

n
Γ 2



≈

√



1
1
n 1−
+
4n 21n2(8.23)(8.24)(8.25)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

8 . 6 . c ova ri a n c e ma tri x a d a p ta ti on 141

is the expected length of a vector drawn from a Gaussian distribution. Comparing
the length of pσ to its expected length under random selection provides themechanism by which σ is increased or decreased. The constants cσ and dσ haverecommended values:cσ = (µeff + 2)/(n + µeff + 5) q
dσ = 1 + 2 max 0, (µeff − 1)/(n + 1) − 1 + cσ

(8.26)

The covariance matrix is also updated using a cumulative vector:(1)pΣ = 0( k +1)

pΣ
where
hσ =

(k)

← (1 − c Σ )pΣ + h σ1 if qkpσ k1−(1−cσ0 otherwise)2( k +1)q

(8.27)cΣ (2 − cΣ )µeff δw

2< 1.4 + n+
1 E kN (0, I)k(8.28)The hσ stalls the update of pΣ if kpΣ k is too large, thereby preventing excessive
increases in Σ when the step size is too small.
The update requires the adjusted weights w◦ :wi
if wi ≥ 0◦wi =(8.29)nwi Σ−1/2 δ(i) 2 otherwise
The covariance update is then






Σ(k+1) ← 1 + c1 cΣ (1 − hσ )(2 − cΣ ) − c1 − cµ Σ(k) +|{z}typically zero

The constants cΣ , c1 and cµ have recommended values

4 + µeff /n
n + 4 + 2µeff /n2
c1 =(n + 1.3)2 + µeff
µ − 2 + 1/µeff
cµ = min 1 − c1 , 2 eff(n + 2)2 + µeff

c 1 pΣ p⊤| {z Σ}rank-one updateµ⊤+ cµ ∑ wi◦ δ(i) δ(i)

|

i =1

{z

rank-µ update(8.30)}cΣ =(8.31)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

142 c ha p te r 8 . stoc ha sti c me thod s

The covariance update consists of three components: the previous covariancematrix Σ(k) , a rank-one update, and a rank-µ update. The rank-one update gets
its name from the fact that pΣ p⊤
Σ has rank one; it has only one eigenvector alongpΣ . Rank-one updates using the cumulation vector allow for correlations between
consecutive steps to be exploited, permitting the covariance matrix to elongate
itself more quickly along a favorable axis.
⊤µThe rank-µ update gets its name from the fact that ∑i=1 wi◦ δ(i) δ(i)has rank
min(µ, n). One important difference between the empirical covariance matrix
update used by the cross-entropy method and the rank-µ update is that theformer estimates the covariance about the new mean µ(k+1) , whereas the latter
estimates the covariance about the original mean µ(k) . The δ(i) values thus help
estimate the variances of the sampled steps rather than the variance within thesampled designs.
Covariance matrix adaptation is depicted in figure 8.9.

8.7 Summary• Stochastic methods employ random numbers during the optimization process.
• Simulated annealing uses a temperature that controls random exploration andwhich is reduced over time to converge on a local minimum.
• The cross-entropy method and evolution strategies maintain proposal distributions from which they sample in order to inform updates.
• Natural evolution strategies uses gradient descent with respect to the log
likelihood to update its proposal distribution.
• Covariance matrix adaptation is a robust and sample-efficient optimizer thatmaintains a multivariate Gaussian proposal distribution with a full covariancematrix.

8.8 Exercises
Exercise 8.1. We have shown that mixture proposal distributions can better
capture multiple minima. Why might their use in the cross-entropy method belimited?
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

8 . 8 . e x e rc i se s 143

function covariance_matrix_adaptation(f, x, k_max;
σ = 1.0,m = 4 + floor(Int, 3*log(length(x))),m_elite = div(m,2))μ, n = copy(x), length(x)ws = log((m+1)/2) .- log.(1:m)
ws[1:m_elite] ./= sum(ws[1:m_elite])
μ_eff = 1 / sum(ws[1:m_elite].^2)
cσ = (μ_eff + 2)/(n + μ_eff + 5)
dσ = 1 + 2max(0, sqrt((μ_eff-1)/(n+1))-1) + cσcΣ = (4 + μ_eff/n)/(n + 4 + 2μ_eff/n)
c1 = 2/((n+1.3)^2 + μ_eff)cμ = min(1-c1, 2*(μ_eff-2+1/μ_eff)/((n+2)^2 + μ_eff))
ws[m_elite+1:end] .*= -(1 + c1/cμ)/sum(ws[m_elite+1:end])
E = n^0.5*(1-1/(4n)+1/(21*n^2))pσ, pΣ, Σ = zeros(n), zeros(n), Matrix(1.0I, n, n)
for k in 1 : k_max
P = MvNormal(μ, σ^2*Σ)
xs = [rand(P) for i in 1 : m]
ys = [f(x) for x in xs]
is = sortperm(ys) # best to worst

Algorithm 8.9. Covariance matrix
adaptation, which takes an objective function f to be minimized, an
initial design point x, and an iteration count k_max. One can optionally specify the step-size scalar σ,
the sample size m, and the numberof elite samples m_elite.
The best candidate design pointis returned, which is the mean of
the final sample distribution.
The covariance matrix undergoes an additional operation to ensure that it remains symmetric; otherwise small numerical inconsistencies can cause the matrix nolonger to be positive definite.
This implementation uses a
simplified normalization strategy
for the negative weights. The original can be found in Equations
50–53 of N. Hansen, “The CMA
Evolution Strategy: A Tutorial,”
ArXiv, no. 1604.00772, 2016.

# selection and mean update
δs = [(x - μ)/σ for x in xs]
δw = sum(ws[i]*δs[is[i]] for i in 1 : m_elite)
μ += σ*δw# step-size control
C = Σ^-0.5
pσ = (1-cσ)*pσ + sqrt(cσ*(2-cσ)*μ_eff)*C*δw
σ *= exp(cσ/dσ * (norm(pσ)/E - 1))# covariance adaptation
hσ = Int(norm(pσ)/sqrt(1-(1-cσ)^(2k)) < (1.4+2/(n+1))*E)pΣ = (1-cΣ)*pΣ + hσ*sqrt(cΣ*(2-cΣ)*μ_eff)*δw
w0 = [ws[i]≥0 ? ws[i] : n*ws[i]/norm(C*δs[is[i]])^2
for i in 1:m]
Σ = (1-c1-cμ) * Σ +c1*(pΣ*pΣ' + (1-hσ) * cΣ*(2-cΣ) * Σ) +cμ*sum(w0[i]*δs[is[i]]*δs[is[i]]' for i in 1 : m)
Σ = triu(Σ)+triu(Σ,1)' # enforce symmetry

end

end
return μ© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c ha p te r 8 . stoc ha sti c me thod s

x2

x2

x2

x2

144

x1

x1

x1

x1
Figure 8.9. Covariance matrix adaptation using multivariate Gaussian
distributions applied to the flower
function, appendix B.4.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

8 . 8 . e x e rc i se s 145

Exercise 8.2. In the cross-entropy method, what is a potential effect of using anelite sample size that is very close to the total sample size?
Exercise 8.3. The log-likelihood of a value sampled from a Gaussian distribution
with mean µ and variance ν is:1
1( x − µ )2ℓ( x | µ, ν) = − ln 2π − ln ν −222ν
Show why evolution strategies using Gaussian distributions may encounter difficulties while applying a descent update on the variance when the mean is on the
optimum, µ = x ∗ .
Exercise 8.4. Derive the maximum likelihood estimate for the cross-entropy
method using multivariate normal distributions:µ( k +1) =1 m (i )xm i∑=1

Σ ( k +1) =1 m (i )(x − µ(k+1) )(x(i) − µ(k+1) )⊤m i∑=1where the maximum likelihood estimates aren the parameterovalues that maximize
the likelihood of sampling the individuals x(1) , · · · , x(m) .

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

9 Population Methods
Previous chapters have focused on methods where a single design point is moved
incrementally toward a minimum. This chapter presents a variety of population
methods that involve optimization using a collection of design points, called individuals. Having a large number of individuals distributed throughout the design
space can help the algorithm avoid becoming stuck in a local minimum. Information at different points in the design space can be shared between individuals to
globally optimize the objective function. Most population methods are stochastic
in nature, and it is generally easy to parallelize the computation.

9.1 Initialization
Population methods begin with an initial population, just as descent methods
require an initial design point. The initial population should be spread over thedesign space to increase the chances that the samples are close to the best regions.
This section presents several initialization methods, but more advanced sampling
methods are discussed in detail in chapter 13.
We can often constrain the design variables to a region of interest consisting ofa hyperrectangle defined by lower and upper bounds a and b. Initial populations
can be sampled from a uniform distribution for each coordinate:1
( j)

xi

∼ U ( a i , bi )(9.1)

where x( j) is the jth individual in the population as seen in algorithm 9.1.
Another common approach is to use a multivariate normal distribution centered over a region of interest. The covariance matrix is typically diagonal, withdiagonal entries scaled to adequately cover the search space. Algorithm 9.2 provides an implementation.

1
Some population methods require additional information to
be associated with the individual,such as velocity in the case of particle swarm optimization, discussed
later. Velocity is often initialized according to a uniform or normal distribution.

148

c ha p te r 9. p op u l a ti on me thod sfunction rand_population_uniform(m, a, b)
d = length(a)return [a+rand(d).*(b-a) for i in 1:m]
end

Algorithm 9.1. A method for sampling an initial population of m design points over a uniform hyperrectangle with lower-bound vector
a and upper-bound vector b.

using Distributions
function rand_population_normal(m, μ, Σ)
D = MvNormal(μ,Σ)
return [rand(D) for i in 1:m]
end

Algorithm 9.2. A method for sampling an initial population of m design points using a multivariate
normal distribution with mean μand covariance Σ.

Uniform and normal distributions limit the covered design space to a concentrated region. The Cauchy distribution (figure 9.1) has an unbounded varianceand can cover a much broader space. Algorithm 9.3 provides an implementation.
Figure 9.2, on the next page, compares example initial populations generatedusing different methods.
NormalCauchyp( x )

0.4

0.2

0

−5

0
x

5

9.2 Genetic Algorithms
Genetic algorithms (algorithm 9.4) borrow inspiration from biological evolution,
where fitter individuals are more likely to pass on their genes to the next generation.2 An individual’s fitness for reproduction is inversely related to the value ofthe objective function at that point. The design point associated with an individual
is represented as a chromosome. At each generation, the chromosomes of the fitter
individuals are passed on to the next generation after undergoing the genetic
operations of crossover and mutation.

Figure 9.1. A comparison of thenormal distribution with standard
deviation 1 and the Cauchy distribution with scale 1. Although
σ is sometimes used for the scaleparameter in the Cauchy distribution, this should not be confused with the standard deviation
since the standard deviation of the
Cauchy distribution is undefined.
The Cauchy distribution is heavytailed, allowing it to cover the design space more broadly.

D. E. Goldberg, Genetic Algorithms
in Search, Optimization, and Machine
Learning. Addison-Wesley, 1989.
2© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

9. 2. g e n e ti c a l g ori thms

using Distributions
function rand_population_cauchy(m, μ, σ)n = length(μ)
return [[rand(Cauchy(μ[j],σ[j])) for j in 1:n] for i in 1:m]
end

Uniform42
x2

Algorithm 9.3. A method for sampling an initial population of m design points using a Cauchy distribution with location μ and scale
σ for each dimension. The location and scale are analogous to themean and standard deviation used
in a normal distribution.

Figure 9.2. Initial populations ofsize 1,000 sampled using a uniform
hyperrectangle with a = [−2, −2],b = [2, 2], a zero-mean normal distribution with diagonal covarianceΣ = I, and Cauchy distributions
centered at the origin with scale
σ = 1.

CauchyNormal

149

0

−2
−4
−4

−2

0
x1

2

4 −4

−2

0
x1

2

4 −4

−2

0
x1

2

4

function genetic_algorithm(f, population, k_max, S, C, M)
for k in 1 : k_max
parents = select(S, f.(population))children = [crossover(C,population[p[1]],population[p[2]])for p in parents]
population .= mutate.(Ref(M), children)end
population[argmin(f.(population))]end

Algorithm 9.4. The genetic
algorithm, which takes anobjective function f, an initial
population, number of iterations
k_max, a SelectionMethod S,
a CrossoverMethod C, and a
MutationMethod M.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

150

c ha p te r 9. p op u l a ti on me thod s

9.2.1 Chromosomes
There are several ways to represent chromosomes. The simplest is the binary
string chromosome, a representation that is similar to the way DNA is encoded.3 A
random binary string of length d can be generated using bitrand(d). A binary
string chromosome is depicted in figure 9.3.

Instead of a binary representation,
DNA contains four nucleobases:
adenine, thymine, cytosine, andguanine, which are often abbreviated A, T, C, and G.
3

Figure 9.3. A chromosome represented as a binary string.

Binary strings are often used due to the ease of expressing crossover andmutation. Unfortunately, the process of decoding a binary string and producing adesign point is not always straightforward. Sometimes the binary string might not
represent a valid point in the design space. It is often more natural to represent achromosome using a list of real values. Such real-valued chromosomes are vectors
in R d that directly correspond to points in the design space.

9.2.2 Initialization
Genetic algorithms start with a random initial population. Binary string chromosomes are typically initialized using random bit strings as seen in algorithm 9.5.
Real-valued chromosomes are typically initialized using the methods from theprevious section.
rand_population_binary(m, n) = [bitrand(n) for i in 1:m]

Algorithm 9.5. A method for sampling random starting populationsof m bit-string chromosomes oflength n.

9.2.3 Selection
Selection is the process of choosing chromosomes to use as parents for the next
generation. For a population with m chromosomes, a selection method will produce a list of m parental pairs4 for the m children of the next generation. The
selected pairs may contain duplicates.

In some cases, one might use
groups, should one wish to combine more than two parents to forma child.

4© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

9. 2. g e n e ti c a l g ori thms

151

Figure 9.4. Truncation selection
with a population size m = 7 andsample size k = 3. The height of abar indicates its objective function
value whereas its color indicateswhat individual it corresponds to.

yyThere are several approaches for biasing the selection toward the fittest (algorithm 9.6). In truncation selection (figure 9.4), we sample parents from among thebest k chromosomes in the population. In tournament selection (figure 9.5), eachparent is the fittest out of k randomly chosen chromosomes of the population. In
roulette wheel selection (figure 9.6), also known as fitness proportionate selection, eachparent is chosen with a probability proportional to its performance relative to the
population. Since we are interested in minimizing an objective function f , the fitness of the ith individual x (i) is inversely related to y(i) = f ( x (i) ). There are different ways to transform a collection y(1) , . . . , y(m) into fitnesses. A simple approachis to assign the fitness of individual i according to max{y(1) , . . . , y(m) } − y(i) .

individualyyindividual

individualylikelihood

individual

individual

individual

Figure 9.5. Tournament selection
with a population size m = 7
and a sample size k = 3, which
is run separately for each parent.
The height of a bar indicates its objective function value whereas its
color indicates what individual itcorresponds to.

Figure 9.6. Roulette wheel selection with a population size m = 7,
which is run separately for eachparent. The approach used causes
the individual with the worst objective function value to have a zero
likelihood of being selected. The
height of a bar indicates its objective function value (left), or its likelihood (right), whereas its color
indicates what individual it corresponds to.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

152 c ha p te r 9. p op u l a ti on me thod s

abstract type SelectionMethod end# Pick pairs randomly from top k parents
struct TruncationSelection <: SelectionMethod
k # top k to keependfunction select(t::TruncationSelection, y)p = sortperm(y)return [p[rand(1:t.k, 2)] for i in y]
end

Algorithm 9.6. Several selection
methods for genetic algorithms. Calling select with a
SelectionMethod and a list of
objective function values y willproduce a list of parental pairs.

# Pick parents by choosing best among random subsets
struct TournamentSelection <: SelectionMethodkendfunction select(t::TournamentSelection, y)
getparent() = begin
p = randperm(length(y))
p[argmin(y[p[1:t.k]])]end
return [[getparent(), getparent()] for i in y]end# Sample parents proportionately to fitness
struct RouletteWheelSelection <: SelectionMethod end
function select(::RouletteWheelSelection, y)y = maximum(y) .- ycat = Categorical(normalize(y, 1))
return [rand(cat, 2) for i in y]end© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

9. 2. g e n e ti c a l g ori thms

153

9.2.4 Crossover
Crossover combines the chromosomes of parents to form children. As with selection, there are several crossover schemes (algorithm 9.7).
• In single-point crossover (figure 9.7), the first portion of parent A’s chromosome
forms the first portion of the child chromosome, and the latter portion of parent
B’s chromosome forms the latter part of the child chromosome. The crossover
point where the transition occurs is determined uniformly at random.
parent A
parent B

Figure 9.7. Single-point crossover.

childcrossover point• In two-point crossover (figure 9.8), we use two random crossover points.
parent A
parent B
child

Figure 9.8. Two-point crossover.

crossover point 1

crossover point 2• In uniform crossover (figure 9.9), each bit has a fifty percent chance of comingfrom either one of the two parents. This scheme is equivalent to each point
having a fifty percent chance of being a crossover point.
parent A
parent B

Figure 9.9. Uniform crossover.

child

The previous crossover methods also work for real-valued chromosomes. We
can, however, define an additional crossover routine that interpolates between
real values (algorithm 9.8). Here, the real values are linearly interpolated between
the parents’ values xa and xb :x ← (1 − λ)xa + λxb(9.2)

where λ is a scalar parameter typically set to one-half.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

154 c ha p te r 9. p op u l a ti on me thod s

abstract type CrossoverMethod end
struct SinglePointCrossover <: CrossoverMethod end
function crossover(::SinglePointCrossover, a, b)
i = rand(1:length(a))
return vcat(a[1:i], b[i+1:end])
end
struct TwoPointCrossover <: CrossoverMethod end
function crossover(::TwoPointCrossover, a, b)
n = length(a)i, j = rand(1:n, 2)
if i > j(i,j) = (j,i)
endreturn vcat(a[1:i], b[i+1:j], a[j+1:n])end

Algorithm 9.7. Several crossover
methods for genetic algorithms. Calling crossover witha CrossoverMethod and twoparents a and b will produce achild chromosome that contains
a mixture of the parents’ genetic
codes. These methods work forboth binary string and real-valued
chromosomes.

struct UniformCrossover <: CrossoverMethod end
function crossover(::UniformCrossover, a, b)
child = copy(a)for i in 1 : length(a)if rand() < 0.5
child[i] = b[i]endendreturn childend

struct InterpolationCrossover <: CrossoverMethodλendcrossover(C::InterpolationCrossover, a, b) = (1-C.λ)*a + C.λ*b

Algorithm 9.8. A crossover
method for real-valued chromosomes which performs linear
interpolation between the parents.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

9. 2. g e n e ti c a l g ori thms 155

9.2.5 Mutation
If new chromosomes were produced only through crossover, many traits that were
not present in the initial random population could never occur, and the most-fit
genes could saturate the population. Mutation allows new traits to spontaneously
appear, allowing the genetic algorithm to explore more of the state space. Childchromosomes undergo mutation after crossover.
Each bit in a binary-valued chromosome typically has a small probability of
being flipped (figure 9.10). For a chromosome with m bits, this mutation rate is
typically set to 1/m, yielding an average of one mutation per child chromosome.
Mutation for real-valued chromosomes can be implemented using bitwise flips,
but it is more common to add zero-mean Gaussian noise. Algorithm 9.9 provides
implementations.
before mutation
after mutation

Figure 9.10. Mutation for binary
string chromosomes gives each bita small probability of flipping.

abstract type MutationMethod end
struct BitwiseMutation <: MutationMethodλendfunction mutate(M::BitwiseMutation, child)return [rand() < M.λ ? !v : v for v in child]end

Algorithm 9.9. The bitwise mutation method for binary string chromosomes and the Gaussian mutation method for real-valued chromosomes. Here, λ is the mutation
rate, and σ is the standard deviation.

struct GaussianMutation <: MutationMethodσendfunction mutate(M::GaussianMutation, child)return child + randn(length(child))*M.σ
end

Figure 9.11 illustrates several generations of a genetic algorithm. Example 9.1
shows how to combine selection, crossover, and mutation strategies discussed inthis section.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c ha p te r 9. p op u l a ti on me thod s

x2

156

x1

x1

x1

x1
Figure 9.11. A genetic algorithmwith truncation selection, single
point crossover, and Gaussian mutation with σ = 0.1 applied to theMichalewicz function defined inappendix B.5.

We will demonstrate using genetic algorithms to optimize a simple function.
import Random: seed!
import LinearAlgebra: norm
seed!(0) # set random seed for reproducible results
f = x->norm(x)m = 100 # population size
k_max = 10 # number of iterations
population = rand_population_uniform(m, [-3, 3], [3,3])
S = TruncationSelection(10) # select top 10
C = SinglePointCrossover()M = GaussianMutation(0.5) # small mutation rate
x = genetic_algorithm(f, population, k_max, S, C, M)@show xExample 9.1. Demonstration of using a genetic algorithm for optimizing a simple function.

x = [-0.00994906141228906, -0.05198433759424115]© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

9. 3. d i ffe re n ti a l e vol u ti on

157

9.3 Differential Evolution
Differential evolution (algorithm 9.10) attempts to improve each individual in the
population by recombining other individuals in the population according to asimple formula.5 It is parameterized by a crossover probability p and a differential
weight w. Typically, w is between 0.4 and 1. For each individual x:
1. Choose three random distinct individuals a, b, and c.

S. Das and P. N. Suganthan, “Differential Evolution: A Survey of theState-of-the-Art,” IEEE Transactions
on Evolutionary Computation, vol. 15,
no. 1, pp. 4–31, 2011.
5

2. Construct an interim design z = a + w · (b − c) as shown in figure 9.12.

3. Choose a random dimension j ∈ [1, . . . , n] for optimization in n dimensions.

4. Construct the candidate individual x′ using binary crossover.
z if i = j or with probability pixi′ = xi otherwise

z

(9.3)

5. Insert the better design between x and x′ into the next generation.
The algorithm is demonstrated in figure 9.13.
using StatsBase
function differential_evolution(f, population, k_max; p=0.5, w=1)
n, m = length(population[1]), length(population)
for k in 1 : k_max
for (k,x) in enumerate(population)a, b, c = sample(population,
Weights([j!=k for j in 1:m]), 3, replace=false)z = a + w*(b-c)
j = rand(1:n)
x′ = [i == j || rand() < p ? z[i] : x[i] for i in 1:n]
if f(x′) < f(x)x[:] = x′
end
end
end
return population[argmin(f.(population))]endbw (b − c)ab−ccFigure 9.12. Differential evolution
takes three individuals a, b, and c
and combines them to form the candidate individual z.
Algorithm 9.10. Differential evolution, which takes an objective function f, a population population,
a number of iterations k_max, acrossover probability p, and a differential weight w. The best individual is returned.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c ha p te r 9. p op u l a ti on me thod s

x2

x2

158

x1

x1

x1

9.4 Particle Swarm Optimization
Particle swarm optimization introduces momentum to accelerate convergence toward minima.6 Each individual, or particle, in the population keeps track of its
current position, velocity, and the best position it has seen so far (algorithm 9.11).
Momentum allows an individual to accumulate speed in a favorable direction,
independent of local perturbations.
mutable struct Particle
xv
x_bestend

x1
Figure 9.13. Differential evolution
with p = 0.5 and w = 0.2 applied
to Ackley’s function, defined in appendix B.1.
J. Kennedy, R. C. Eberhart, andY. Shi, Swarm Intelligence. Morgan
Kaufmann, 2001.
6

Algorithm 9.11. Each particle inparticle swarm optimization has aposition x and velocity v in design
space and keeps track of the best
design point found so far, x_best.

At each iteration, each individual is accelerated toward both the best position it
has seen and the best position found thus far by any individual. The acceleration
is weighted by a random term, with separate random numbers being generated© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

9. 5 . fi re fl y a l g ori thm 159

for each acceleration. The update equations are:x (i ) ← x (i ) + v (i )

(9.4)


(i )





v(i) ← wv(i) + c1 r1 xbest − x (i) + c2 r2 xbest − x (i)



(9.5)

where xbest is the best location found so far over all particles; w, c1 , and c2 are parameters; and r1 and r2 are random numbers drawn from U (0, 1).7 Algorithm 9.12
provides an implementation. Figure 9.14 shows several iterations of the algorithm.

Algorithm 9.12. Particle swarm optimization, which takes an objective function f, a list of particles
population, a number of iterations
k_max, an inertia w, and momentum coefficients c1 and c2. The default values are those used by R.
Eberhart and J. Kennedy, “A New
Optimizer Using Particle Swarm
Theory,” in International Symposiumon Micro Machine and Human Science, 1995.

function particle_swarm_optimization(f, population, k_max;
w=1, c1=1, c2=1)
n = length(population[1].x)
x_best, y_best = copy(population[1].x_best), Inf
for P in populationy = f(P.x)
if y < y_best; x_best[:], y_best = P.x, y; end
end
for k in 1 : k_max
for P in population
r1, r2 = rand(n), rand(n)P.x += P.v
P.v = w*P.v + c1*r1.*(P.x_best - P.x) +c2*r2.*(x_best - P.x)
y = f(P.x)
if y < y_best; x_best[:], y_best = P.x, y; end
if y < f(P.x_best); P.x_best[:] = P.x; end
end
end
return population
end

9.5

A common strategy is to allow theinertia w to decay over time.
7

X.-S. Yang, Nature-Inspired Metaheuristic Algorithms. Luniver Press,
2008. Interestingly, male fireflies
flash to attract members of the
opposite sex, but females sometimes flash to attract males of otherspecies, which they then eat.
8

Firefly Algorithm

The firefly algorithm (algorithm 9.13) was inspired by the manner in which fireflies
flash their lights to attract mates.8 In the firefly algorithm, each individual in the
population is a firefly and can flash to attract other fireflies. At each iteration, allfireflies are moved toward all more attractive fireflies. A firefly a is moved toward
a firefly b with greater attraction according toa ← a + βI (kb − ak)(b − a) + αǫ

(9.6)

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

160 c ha p te r 9. p op u l a ti on me thod s

where I is the intensity of the attraction and β is the source intensity. A random
walk component is included as well, where ǫ is drawn from a zero-mean, unit
covariance multivariate Gaussian, and α scales the step size. The resulting update
is a random walk biased toward brighter fireflies.9
The intensity I decreases as the distance r between the two fireflies increases
and is defined to be 1 when r = 0. One approach is to model the intensity as apoint source radiating into space, in which case the intensity decreases according
to the inverse square law
1
I (r ) = 2(9.7)
rAlternatively, if a source is suspended in a medium that absorbs light, the intensity
will decrease according to an exponential decay
I (r ) = e−γr2Yang recommends β = 1 and α ∈[0, 1]. If β = 0, the behavior is arandom walk.

9

(9.8)

where γ is a the light absorption coefficient.10
We generally want to avoid equation (9.7) in practice due to the singularity at r = 0. A combination of the inverse square law and absorption can beapproximated with a Gaussian brightness drop-off:
I (r ) = e−γr

Figure 9.14. The particle swarm
method with w = 0.1, c1 = 0.25,
and c2 = 2 applied to Wheeler’s
Ridge, appendix B.7.

The distance between fireflies
ceases to matter as γ approaches
zero.

10

(9.9)

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

9. 6 . c u c koo se a rc h 161

A firefly’s attraction is proportional to its performance. Attraction affects only
whether one fly is attracted to another fly, whereas intensity affects how muchthe less attractive fly moves. Figure 9.15 shows a few iterations of the algorithm.
using Distributions
function firefly(f, population, k_max;
β=1, α=0.1, brightness=r->exp(-r^2))

end

9.6

m = length(population[1])N = MvNormal(Matrix(1.0I, m, m))
for k in 1 : k_max
for a in population, b in population
if f(b) < f(a)r = norm(b-a)
a[:] += β*brightness(r)*(b-a) + α*rand(N)
end
end
end
return population[argmin([f(x) for x in population])]

Algorithm 9.13. The firefly algorithm, which takes an objectivefunction f, a population flies consisting of design points, a numberof iterations k_max, a source intensity β, a random walk step size α,
and an intensity function I. The
best design point is returned.

Figure 9.15. Firefly search with
α = 0.5, β = 1, and γ = 0.1 applied to the Branin function (appendix B.3).

Cuckoo Search

Cuckoo search (algorithm 9.14) is another nature-inspired algorithm named after
the cuckoo bird, which engages in a form of brood parasitism.11 Cuckoos lay theireggs in the nests of other birds, often birds of other species. When this occurs,
the host bird may detect the invasive egg and then destroy it or establish a new

X.-S. Yang and S. Deb, “Cuckoo
Search via Lévy Flights,” in World
Congress on Nature & Biologically Inspired Computing (NaBIC), 2009.
11© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

162 c ha p te r 9. p op u l a ti on me thod snest somewhere else. However, there is also a chance that the egg is accepted andraised by the host bird.12
In cuckoo search, each nest represents a design point. New design points can beproduced using Lévy flights from nests, which are random walks with step-lengths
from a heavy-tailed distribution. A new design point can replace a nest if it has abetter objective function value, which is analogous to cuckoo eggs replacing theeggs of birds of other species.
The core rules are:

Interestingly, an instinct of newly
hatched cuckoos is to knock othereggs or hatchlings (those belonging to the host bird) out of the nest.

12

1. A cuckoo will lay an egg in a randomly chosen nest.
2. The best nests with the best eggs will survive to the next generation.
3. Cuckoo eggs have a chance of being discovered by the host bird, in which casethe eggs are destroyed.
Cuckoo search relies on random flights to establish new nest locations. Theseflights start from an existing nest and then move randomly to a new location.
While we might be tempted to use a uniform or Gaussian distribution for the walk,these restrict the search to a relatively concentrated region. Instead, cuckoo searchuses a Cauchy distribution, which has a heavier tail. In addition, the Cauchy
distribution has been shown to be more representative of the movements of otheranimals in the wild.13 Figure 9.16 shows a few iterations of cuckoo search.
Other nature-inspired algorithms include the artificial bee colony, the gray wolf
optimizer, the bat algorithm, glowworm swarm optimization, intelligent water
drops, and harmony search.14 There has been some criticism of the proliferationof methods that make analogies to nature without fundamentally contributing
novel methods and understanding.15

9.7 Hybrid Methods
Many population methods perform well in global search, being able to avoid local
minima and finding the best regions of the design space. Unfortunately, thesemethods do not perform as well in local search in comparison to descent methods.
Several hybrid methods16 have been developed to extend population methods withdescent-based features to improve their performance in local search. There are
two general approaches to combining population methods with local searchtechniques:17

For example, a certain speciesof fruit fly explores its surroundings using Cauchy-like steps separated by 90° turns. A. M. Reynolds
and M. A. Frye, “Free-Flight Odor
Tracking in Drosophila is Consistent with an Optimal Intermittent Scale-Free Search,” PLoS ONE,vol. 2, no. 4, e354, 2007.
13

See, for example, D. Simon, Evolutionary Optimization Algorithms. Wiley, 2013.

14

This viewpoint is expressed byK. Sörensen, “Metaheuristics—the
Metaphor Exposed,” International
Transactions in Operational Research,
vol. 22, no. 1, pp. 3–18, 2015.
15

In the literature, these kinds oftechniques are also referred to as
memetic algorithms or genetic local
search.

16

K. W. C. Ku and M.-W. Mak, “Exploring the Effects of Lamarckian
and Baldwinian Learning in Evolving Recurrent Neural Networks,”
in IEEE Congress on Evolutionary
Computation(CEC), 1997.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

17

9. 7. hy bri d me thod susing Distributions
mutable struct Nest
x # positiony # value, f(x)endfunction cuckoo_search(f, population, k_max;
p_a=0.1, C=Cauchy(0,1))m, n = length(population), length(population[1].x)
a = round(Int, m*p_a)for k in 1 : k_max
i, j = rand(1:m), rand(1:m)
x = population[j].x + [rand(C) for k in 1 : n]y = f(x)if y < population[i].y
population[i].x[:] = x
population[i].y = y
end

163

Algorithm 9.14. Cuckoo search,which takes an objective functionf, an initial set of nests population,
a number of iterations k_max, percent of nests to abandon p_a, andflight distribution C. The flight distribution is typically a centered
Cauchy distribution.

p = sortperm(population, by=nest->nest.y, rev=true)
for i in 1 : a
j = rand(1:m-a)+a
population[p[i]] = Nest(population[p[j]].x +[rand(C) for k in 1 : n],f(population[p[i]].x)
)
end

end

end
return population

Figure 9.16. Cuckoo search applied to the Branin function (appendix B.3).
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

164

c ha p te r 9. p op u l a ti on me thod s• In Lamarckian learning, the population method is extended with a local searchmethod that locally improves each individual. The original individual and its
objective function value are replaced by the individual’s optimized counterpart
and its objective function value.
• In Baldwinian learning, the same local search method is applied to each individual, but the results are used only to update the individual’s perceived objectivefunction value. Individuals are not replaced but are merely associated with
optimized objective function values, which are not the same as their actual
objective function value. Baldwinian learning can help prevent premature
convergence.
The difference between these approaches is illustrated in example 9.2.
Consider optimizing f ( x ) = −e− x − 2e−( x−3) using a population of individuals initialized near x = 0.
22yLamarckian

Baldwinian

0

0

−1

−1

−2

−2

−2

0

2
x

Example 9.2. A comparison of theLamarckian and Baldwinian hybrid methods.

4

−2

0

2

4

x

A Lamarckian local search update applied to this population would movethe individuals toward the local minimum, reducing the chance that futureindividuals escape and find the global optimum near x = 3. A Baldwinian
approach will compute the same update but leaves the original designs
unchanged. The selection step will value each design according to its value
from a local search.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

9. 8 . su mma ry

165

9.8 Summary• Population methods use a collection of individuals in the design space to guideprogression toward an optimum.
• Genetic algorithms leverage selection, crossover, and mutations to produce
better subsequent generations.
• Differential evolution, particle swarm optimization, the firefly algorithm, andcuckoo search include rules and mechanisms for attracting design points tothe best individuals in the population while maintaining suitable state space
exploration.
• Population methods can be extended with local search approaches to improve
convergence.

9.9 Exercises
Exercise 9.1. What is the motivation behind the selection operation in genetic
algorithms?
Exercise 9.2. Why does mutation play such a fundamental role in genetic algorithms? How would we choose the mutation rate if we suspect there is a better
optimal solution?
Exercise 9.3. If we observe that particle swarm optimization results in fast convergence to a nonglobal minimum, how might we change the parameters of thealgorithm?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

10 Constraints
Previous chapters have focused on unconstrained problems where the domain ofeach design variable is the space of real numbers. Many problems are constrained,
which forces design points to satisfy certain conditions. This chapter presents avariety of approaches for transforming problems with constraints into problems
without constraints, thereby permitting the use of the optimization algorithms
we have already discussed. Analytical methods are also discussed, including theconcepts of duality and the necessary conditions for optimality under constrained
optimization.

10.1 Constrained Optimization
Recall the core optimization problem equation (1.1):
minimizexf (x)subject to x ∈ X(10.1)In unconstrained problems, the feasible set X is R n . In constrained problems, thefeasible set is some subset thereof.
Some constraints are simply upper or lower bounds on the design variables,
as we have seen in bracketed line search, in which x must lie between a and b.
A bracketing constraint x ∈ [ a, b] can be replaced by two inequality constraints:a ≤ x and x ≤ b as shown in figure 10.1. In multivariate problems, bracketing the
input variables forces them to lie within a hyperrectangle as shown in figure 10.2.

168

c ha p te r 10 . c on stra i n ts

Figure 10.1. A simple optimization
problem constrained by upper andlower bounds.

f (x)minimizef (x)subject tox ∈ [ a, b]xx
aConstraints arise naturally when formulating real problems. A hedge fundmanager cannot sell more stock than they have, an airplane cannot have wings
with zero thickness, and the number of hours you spend per week on yourhomework cannot exceed 168. We include constraints in such problems to preventthe optimization algorithm from suggesting an infeasible solution.
Applying constraints to a problem can affect the solution, but this need not be
the case as shown in figure 10.3.
Constrained, Same Solution

Unconstrained

x2

b

x1
Figure 10.2. Bracketing constraints
force the solution to lie within a
hyperrectangle.

Constrained, New Solution

f (x)

x∗

x∗

x∗
x

x
a

Constraints are not typically specified directly through a known feasible set X .
Instead, the feasible set is typically formed from two types of constraints:1
2. inequality constraints, g(x) ≤ 0ab10.2 Constraint Types

1. equality constraints, h(x) = 0xbFigure 10.3. Constraints can change
the solution to a problem, but do
not have to.

We have g representing a lessthan inequality constraint. Greaterthan equality constraints can betranslated into less-than inequality
constraints by introducing a negative sign.

1© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

10 . 3. tra n sforma ti on s to re move c on stra i n ts

169

Any optimization problem can be rewritten using these constraints:
minimizex

subject tof (x)hi (x) = 0 for all i in {1, . . . , ℓ}(10.2)g j (x) ≤ 0 for all j in {1, . . . , m}

Of course, constraints can be constructed from a feasible set X :h (x) = (x ∈/ X)

(10.3)

where Boolean expressions evaluate to 0 or 1.
We often use equality and inequality functions (h(x) = 0, g(x) ≤ 0) to define
constraints rather than set membership (x ∈ X ) because the functions can provide
information about how far a given point is from being feasible. This information
helps drive solution methods toward feasibility.
Equality constraints are sometimes decomposed into two inequality constraints: h (x) ≤ 0h(x) = 0 ⇐⇒(10.4)
 h (x) ≥ 0

However, sometimes we want to handle equality constraints separately, as we
will discuss later in this chapter.

10.3

Transformations to Remove Constraints

In some cases, it may be possible to transform a problem so that constraints can beremoved. For example, bound constraints a ≤ x ≤ b can be removed by passing
x through a transform (figure 10.4):
b+a b−a2x̂
x = t a,b ( x̂ ) =+(10.5)
2
2
1 + x̂2x

Example 10.1 demonstrates this process.
Some equality constraints can be used to solve for xn given x1 , . . . , xn−1 . In
other words, if we know the first n − 1 components of x, we can use the constraint
equation to obtain xn . In such cases, the optimization problem can be reformulated over x1 , . . . , xn−1 instead, removing the constraint and removing one design
variable. Example 10.2 demonstrates this process.

b

a

−1 1
x̂

Figure 10.4. This transform ensuresthat x lies between a and b.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

170c ha p te r 10 . c on stra i n ts

Example 10.1. Removing bounds
constraints using a transform on
the input variable.

Consider the optimization problem
minimizex

x sin( x )subject to 2 ≤ x ≤ 6
We can transform the problem to remove the constraints:
minimizex̂minimize
x̂

4+2



2x̂
1 + x̂2t2,6 ( x̂ ) sin(t2,6 ( x̂ ))



2x̂
sin 4 + 2
1 + x̂2

We can use the optimization method of our choice to solve the unconstrained problem. In doing so, we find two minima: x̂ ≈ 0.242 and x̂ ≈ 4.139,
both of which have a function value of approximately −4.814.

( f ◦ T2,6 )( x̂ )f ( x ) = x sin( x )
2
10
0
0

−2
−4

−10
0

5

10
x

15

−5

0

5

x̂

The solution for the original problem is obtained by passing x̂ throughthe transform. Both values of x̂ produce x = t2,6 ( x̂ ) ≈ 4.914.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

10 . 4 . l a g ra n g e mu l ti p l i e rs 171

Example 10.2. Using constraint
equations to eliminate design variables.

Consider the constraint:h (x) = c1 x1 + c2 x2 + · · · + c n x n = 0
We can solve for xn using the first n − 1 variables:xn =We can transform1(−c1 x1 − c2 x2 − · · · − cn−1 xn−1 )cnminimizef (x)subject toh (x) = 0x

intominimize
x1 ,...,xn−1

f



1
x1 , . . . , xn−1 , (−c1 x1 − c2 x2 − · · · − cn−1 xn−1 )cn

10.4 Lagrange Multipliers
The method of Lagrange multipliers is used to optimize a function subject to equality
constraints. Consider an optimization problem with a single equality constraint:
minimizef (x)subject toh (x) = 0x(10.6)

where f and h have continuous partial derivatives. Example 10.3 discusses such aproblem.
The method of Lagrange multipliers is used to compute where a contour lineof f is aligned with the contour line of h(x) = 0. Since the gradient of a function
at a point is perpendicular to the contour line of that function through that point,
we know the gradient of h will be perpendicular to the contour line h(x) = 0.
Hence, we need to find where the gradient of f and the gradient of h are aligned.
We seek the best x such that the constraint
h (x) = 0

(10.7)

is satisfied and the gradients are aligned∇ f (x) = λ ∇ h (x)(10.8)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c ha p te r 10 . c on stra i n ts

Consider the minimization problem:
minimizex

subject to

3− exp − x1 x2 −
2

2



3
− x2 −22 !

x1 − x22 = 0

We substitute the constraint x1 = x22 into the objective function to obtain
an unconstrained objective:



 !
3 2
3 2
3
f unc = − exp − x2 −− x2 −
2
2

Example 10.3. A motivating example of the method of Lagrange multipliers.

h (x) = 0x2

172whose derivative is:



 !

3 2
1
3 2
3 2 1
∂
3
5
f unc = 6 exp − x2 −− x2 −
x2 − x2 + x2 −
∂x2
2
2
2
3
2
Setting the derivative to zero and solving for x2 yields x2 ≈ 1.165. The
solution to the original optimization problem is thus x∗ ≈ [1.358, 1.165]. The
optimum lies where the contour line of f is aligned with h.
If the point x ∗ optimizes f along h, then its directional derivative at x ∗along h must be zero. That is, small shifts of x ∗ along h cannot result in an
improvement.
The contour lines of f are lines of constant f . Thus, if a contour line of f
is tangent to h, then the directional derivative of h at that point, along thedirection of the contour h(x) = 0, must be zero.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

x∗x1

10 . 4 . l a g ra n g e mu l ti p l i e rs 173

for some Lagrange multiplier λ. We need the scalar λ because the magnitudes of
the gradients may not be the same.2
We can formulate the Lagrangian, which is a function of the design variables
and the multiplierL(x, λ) = f (x) − λh(x)(10.9)
Solving ∇L(x, λ) = 0 solves equations (10.7) and (10.8). Specifically, ∇x L = 0
gives us the condition ∇ f = λ∇h, and ∇λ L = 0 gives us h(x) = 0. Any solution
is considered a critical point. Critical points can be local minima, global minima,or saddle points.3 Example 10.4 demonstrates this approach.
We can use the method of Lagrange multipliers to solve the problem inexample 10.3. We form the Lagrangian


 !

3 2
3 2
− x2 −− λ( x1 − x22 )L( x1 , x2 , λ) = − exp − x1 x2 −22When ∇ f is zero, the Lagrange
multiplier λ equals zero, irrespective of ∇h.

2

The method of Lagrange multipliers gives us a first-order necessary
condition to test for optimality. We
will extend this method to include
inequalities.
3

Example 10.4. Using the method of
Lagrange multipliers to solve theproblem in example 10.3.

and compute the gradient∂L3= 2x2 f (x)− x1 x2 − λ
∂x1
2


∂L
3
3
= 2λx2 + f (x) −2x1 ( x1 x2 − ) − 2( x2 − )
∂x2
2
2
∂L
= x22 − x1
∂λSetting these derivatives to zero and solving yields x1 ≈ 1.358, x2 ≈ 1.165,
and λ ≈ 0.170.
The method of Lagrange multipliers can be extended to multiple equality
constraints. Consider a problem with two equality constraints:
minimizef (x)subject toh1 (x) = 0

x(10.10)h2 (x) = 0© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

174

c ha p te r 10 . c on stra i n ts

We can collapse these constraints into a single constraint. The new constraint
is satisfied by exactly the same points as before, so the solution is unchanged.
minimizef (x)subject tohcomb (x) = h1 (x)2 + h2 (x)2 = 0

x(10.11)

We can now apply the method of Lagrange multipliers as we did before. In
particular, we compute the gradient condition∇ f − λ∇hcomb = 0∇ f − 2λ(h1 ∇h1 + h2 ∇h2 ) = 0(10.12)(10.13)

Our choice for hcomb was somewhat arbitrary. We could have usedhcomb (x) = h1 (x)2 + c · h2 (x)2

(10.14)for some constant c > 0.
With this more general formulation, we get(10.15)0 = ∇ f − λ∇hcomb= ∇ f − 2λh1 ∇h1 − 2cλh2 ∇h2= ∇ f − λ1 ∇ h1 − λ2 ∇ h2

(10.16)(10.17)

We can thus define a Lagrangian with ℓ Lagrange multipliers for problems
with ℓ equality constraintsℓL(x, λ) = f (x) − ∑ λi hi (x) = f (x) − λ⊤ h(x)(10.18)

i =1

10.5

Inequality Constraints

Consider a problem with a single inequality constraint:
minimizef (x)subject tog (x) ≤ 0x

(10.19)We know that if the solution lies at the constraint boundary, then the Lagrange
condition holds∇ f + µ∇ g = 0(10.20)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

10 . 5 . i n e q u a l i ty c on stra i n ts 175

for some non-negative constant µ. When this occurs, the constraint is considered
active, and the gradient of the objective function is limited exactly as it was withequality constraints. Figure 10.5 shows an example.
If the solution to the problem does not lie at the constraint boundary, then theconstraint is considered inactive. Solutions of f will simply lie where the gradientof f is zero, as with unconstrained optimization. In this case, equation (10.20)
will hold by setting µ to zero. Figure 10.6 shows an example.
We could optimize a problem with an inequality constraint by introducing an
infinite step penalty for infeasible points: f (x) if g(x) ≤ 0(10.21)f ∞-step (x) =∞otherwise= f (x) + ∞ · ( g (x) > 0)

We can recover f ∞-step by maximizing with respect to µµ ≥0(10.24)xµ ≥0This reformulation is known as the primal problem.

x∗

Figure 10.6. An inactive inequality
constraint.
Such a problem formulation canbe optimized using direct methods
such as the mesh adaptive direct
search (section 8.2).

4

1
0

For any infeasible x we get infinity and for any feasible x we get f (x).
The new optimization problem is thus
minimize maximize L(x, µ)

Figure 10.5. An active inequality
constraint. The corresponding contour line is shown in red.

(10.22)

Unfortunately, f ∞-step is inconvenient to optimize.4 It is discontinuous andnondifferentiable. Search routines obtain no directional information to steer themselves toward feasibility.
We can instead use a linear penalty µg(x), which forms a lower bound on∞ · ( g(x) > 0) and penalizes the objective toward feasibility as long as µ > 0.
This linear penalty is visualized in figure 10.7.
We can use this linear penalty to construct a Lagrangian for inequality constraints
L(x, µ) = f (x) + µg(x)(10.23)f ∞-step = maximize L(x, µ)x∗−1(10.25)

−1

0

1

g (x)
∞ · ( g (x) > 0)µg(x)Figure 10.7. The linear functionµg(x) is a lower bound to the infinite step penalty so long as µ ≥ 0.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

176 c ha p te r 10 . c on stra i n ts

Optimizing the primal problem requires finding critical points x∗ such that:
1. g(x∗ ) ≤ 0
The point is feasible.
2. µ ≥ 0
The penalty must point in the right direction. This requirement is sometimes
called dual feasibility.
3. µg(x∗ ) = 0
A feasible point on the boundary will have g(x) = 0, whereas a feasible pointwith g(x) < 0 will have µ = 0 to recover f (x∗ ) from the Lagrangian.
4. ∇ f (x∗ ) + µ∇ g(x∗ ) = 0
When the constraint is active, we require that the contour lines of f and g bealigned, which is equivalent to saying that their gradients be aligned. Whenthe constraint is inactive, our optimum will have ∇ f (x∗ ) = 0 and µ = 0.
These four requirements can be generalized to optimization problems withany number of equality and inequality constraints:5
minimizef (x)subject tog(x) ≤ 0x(10.26)

If u and v are vectors of the samelength, then we say u ≤ v whenui ≤ vi for all i. We define ≥, <,and > similarly for vectors.
5h(x) = 0
where each component of g is an inequality constraint and each component of his an equality constraint. The four conditions are called the KKT conditions.6
1. Feasibility: The constraints are all satisfied.
g (x∗ ) ≤ 0∗

h(x ) = 0

(10.27)(10.28)2. Dual feasibility: Penalization is toward feasibility.
µ≥0(10.29)3. Complementary slackness: The Lagrange multipliers takes up the slack,
where µi is zero or gi (x∗ ) is zero.7µ⊙g = 0(10.30)

Named after Harold W. Kuhn
and Albert W. Tucker who published the conditions in 1951. It
was later discovered that William
Karush studied these conditionsin an unpublished master’s thesis
in 1939. A historical prospective
is provided by T. H. Kjeldsen, “A
Contextualized Historical Analysis of the Kuhn-Tucker Theorem
in Nonlinear Programming: The
Impact of World War II,” Historia
Mathematica, vol. 27, no. 4, pp. 331–
361, 2000.
6

7
The operation a ⊙ b indicates theelement-wise product between vectors a and b.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

10 . 6 . d u a l i ty 177

4. Stationarity: The objective function contour is tangent to each active constraint.8∇ f ( x ∗ ) + ∑ µ i ∇ gi ( x ∗ ) + ∑ λ j ∇ h j ( x ∗ ) = 0(10.31)ijThese four conditions are first-order necessary conditions for optimality andare thus FONCs for problems with smooth constraints. Just as with the FONCs forunconstrained optimization, special care must be taken to ensure that identified
critical points are actually local minima.

10.6

8
Since the sign of λ is not restricted, we can reverse the sign for
the equality constraints from themethod of Lagrange multipliers.

Duality

In deriving the FONCs for constrained optimization, we also find a more generalform for the Lagrangian. This generalized Lagrangian isL(x, µ, λ) = f (x) + ∑ µi gi (x) + ∑ λ j h j (x)i(10.32)j

The primal form of the optimization problem is the original optimization problem formulated using the generalized Lagrangianminimize maximize L(x, µ, λ)
x

µ≥0,λ

(10.33)The primal problem is identical to the original problem and is just as difficult
to optimize.
The dual form of the optimization problem reverses the order of the minimization and maximization in equation (10.33):
maximize minimize L(x, µ, λ)
µ≥0,λ

x

(10.34)The max-min inequality states that for any function f (a, b):
maximize minimize f (a, b) ≤ minimize maximize f (a, b)
a

b

b

a

(10.35)The solution to the dual problem is thus a lower bound to the solution of the
primal problem. That is, d∗ ≤ p∗ , where d∗ is the dual value and p∗ is the primalvalue.
The inner minimization in the dual problem is often folded into a dual function,D(µ ≥ 0, λ) = minimize L(x, µ, λ)x(10.36)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

178 c ha p te r 10 . c on stra i n ts

for notational convenience. The dual function is concave.9 Gradient ascent on aconcave function always converges to the global maximum. Optimizing the dual
problem is easy whenever minimizing the Lagrangian with respect to x is easy.
We know that maximizeµ≥0,λ D(µ, λ) ≤ p∗ . It follows that the dual functionis always a lower bound on the primal problem (see example 10.5). For any µ ≥ 0
and any λ, we haveD(µ ≥ 0, λ) ≤ p∗
(10.37)The difference p∗ − d∗ between the dual and primal values is called the duality
gap. In some cases, the dual problem is guaranteed to have the same solution as
the original problem—the duality gap is zero.10 In such cases, duality provides
an alternative approach for optimizing our problem. Example 10.6 demonstrates
this approach.

10.7

For a detailed overview, see S.
Nash and A. Sofer, Linear and Nonlinear Programming. McGraw-Hill,
1996.
9

Conditions that guarantee a zero
duality gap are discussed in S.
Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2004.
10

Penalty Methods

We can use penalty methods to convert constrained optimization problems into
unconstrained optimization problems by adding penalty terms to the objectivefunction, allowing us to use the methods developed in previous chapters.
Consider a general optimization problem:
minimizef (x)subject tog(x) ≤ 0x(10.38)h(x) = 0
A simple penalty method counts the number of constraint equations that are
violated:
pcount (x) = ∑( gi (x) > 0) + ∑ h j (x) 6= 0
(10.39)ijwhich results in the unconstrained optimization problem that penalizes infeasibility
minimize f (x) + ρ · pcount (x)(10.40)x

where ρ > 0 adjusts the penalty magnitude. Figure 10.8 shows an example.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

10 . 7. p e n a l ty me thod s 179

Example 10.5. The dual function isa lower bound of the primal problem.

Consider the optimization problem:
minimizesin( x )subject tox2 ≤ 1x

The generalized Lagrangian is L( x, µ) = sin( x ) + µ( x2 − 1), making the
primal problem:
minimize maximize sin( x ) + µ( x2 − 1)xµ ≥0

and the dual problem:
maximize minimize sin( x ) + µ( x2 − 1)
x

µ ≥0

10

µ = 0.00µ = 0.25µ = 0.50µ = 0.75µ = 1.00µ = 1.25µ = 1.50µ = 1.75µ = 2.00

y

5

x∗

0

−4

−2

0
x

2

4

The objective function is plotted in black, with the feasible region traced
over in blue. The minimum is at x ∗ = −1 with p∗ ≈ −0.841. The purple
lines are the Lagrangian L( x, µ) for different values of µ, each of which has
a minimum lower than p∗ .

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

180 c ha p te r 10 . c on stra i n ts

Consider the problem:
minimizex1 + x2 + x1 x2

subject tox12 + x22 = 1x

The Lagrangian is L( x1 , x2 , λ) = x1 + x2 + x1 x2 + λ( x12 + x22 − 1).
We apply the method of Lagrange multipliers:

Example 10.6. An example of Lagrangian duality applied to a problem with an equality constraint.
The top figure shows the objectivefunction contour and the constraint
with the four critical points marked
by scatter points. The bottom figure shows the dual function.

x2∂L= 1 + x2 + 2λx1 = 0∂x1∂L= 1 + x1 + 2λx2 = 0∂x2∂L= x12 + x22 − 1 = 0
∂λ

2

0

−2
−2

Solving yields four potential solutions, and thus four critical points:x2

λx1 + x2 + x1 x2

−1
0
√

0
−1
√

1/2
 1/2 √ 12  −1 − 2√1
2 −1 + 2

−1
−1

√2+1
√2+2
√2−1
2−2

√2+1
√2+2
√2−1
2−2

1
2 +
1
2 −

√

√

2 ≈ 1.914

2 ≈ −0.914

We find that the two optimal solutions are [−1, 0] and [0, −1].
The dual function has the form

D(λ) = minimize x1 + x2 + x1 x2 + λ( x12 + x22 − 1)x1 ,x2The dual function is unbounded below when λ is less than 1/2 (consider
x1 → ∞ and x2 → −∞). Setting the gradient to 0 and solving yields x2 =−1 − 2λx1 and x1 = (2λ − 1)/(1 − 4λ2 ) for λ 6= ±1/2. When λ = 1/2,
x1 = −1 − x2 and D(1/2) = −1. Substituting these into the dual function
yields:
−λ − 11
2λ+1 λ ≥ 2D(λ) =
−∞otherwise
The dual problem maximizeλ D(λ) is maximized at λ = 1/2.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

2

0

−2

D(λ)

x1

0
x1

−4

λ∗

−6
0

2

4
λ

10 . 7. p e n a l ty me thod s 181

Penalty methods start with an initial point x and a small value for ρ. The
unconstrained optimization problem equation (10.40) is then solved. The resulting
design point is then used as the starting point for another optimization with anincreased penalty. We continue with this procedure until the resulting point isfeasible, or a maximum number of iterations has been reached. Algorithm 10.1
provides an implementation.
function penalty_method(f, p, x, k_max; ρ=1, γ=2)
for k in 1 : k_max
x = minimize(x -> f(x) + ρ*p(x), x)ρ *= γif p(x) == 0
return x
end
end
return x
endj

Quadratic penalties close to the constraint boundary are very small and may
require ρ to approach infinity before the solution ceases to violate the constraints.
It is also possible to mix a count and a quadratic penalty function (figure 10.10):
pmixed (x) = ρ1 pcount (x) + ρ2 pquadratic (x)

a

b
f (x)
f ( x ) + ρpcount ( x )Figure 10.8. The original and countpenalized objective functions forminimizing f subject to x ∈ [ a, b].
Algorithm 10.1. The penalty
method for objective function f,
penalty function p, initial point
x, number of iterations k_max,
initial penalty ρ > 0, and penalty
multiplier γ > 1. The methodminimize should be replacedwith a suitable unconstrained
minimization method.

This penalty will preserve the problem solution for large values of ρ, but it
introduces a sharp discontinuity. Points not inside the feasible set lack gradient
information to guide the search towards feasibility.
We can use quadratic penalties to produce a smooth objective function (figure 10.9):
pquadratic (x) = ∑ max( gi (x), 0)2 + ∑ h j (x)2
(10.41)
i

x

x
a

b

f (x)
f ( x ) + ρpquadratic ( x )Figure 10.9. Using a quadratic
penalty function for minimizing f
subject to x ∈ [ a, b].

(10.42)

Such a penalty mixture provides a clear boundary between the feasible region
and the infeasible region while providing gradient information to the solver.
Figure 10.11 shows the progress of the penalty function as ρ is increased.
Quadratic penalty functions cannot ensure feasibility as discussed in example 10.7.

x
a

b
f (x)
f ( x ) + pmixed ( x )Figure 10.10. Using both aquadratic and discrete penalty
function for minimizing f subject
to x ∈ [ a, b].
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

182

c ha p te r 10 . c on stra i n ts

ρ=0

ρ=1

ρ=2

ρ=3

x

x

x

x2

2

x

0

−2
−2

0
x1

−2

2

0
x1

2

−2

0
x1

−2

2

0
x1

2

Figure 10.11. The penalty method
applied to the flower function, appendix B.4, and the circular constraint
x12 + x22 ≥ 2

Consider the problem
minimizex

subject tox≥5x

Example 10.7. An example showing how quadratic penalties cannot
ensure feasibility.

using a quadratic penalty function.
The unconstrained objective function isf ( x ) = x + ρ max(5 − x, 0)2
The minimum of the unconstrained objective function isx∗ = 5 −1
2ρ

While the minimum of the constrained optimization problem is clearly x =5, the minimum of the penalized optimization problem merely approaches
x = 5, requiring an infinite penalty to achieve feasibility.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

10 . 8 . a u g me n te d l a g ra n g e me thod 183

10.8 Augmented Lagrange Method
The augmented Lagrange method11 is an adaptation of the penalty method specifically for equality constraints. Unlike the penalty method, where ρ must sometimes
approach infinity before a feasible solution is found, the augmented Lagrange
method will work with smaller values of ρ. It uses both a quadratic and a linear
penalty for each constraint.
For an optimization problem with equality constraints h(x) = 0, the penalty
function is:1
pLagrange (x) = ρ ∑(hi (x))2 − ∑ λi hi (x)(10.43)2 ii

Not to be confused with themethod of Lagrange multipliers.

11

where λ converges toward the Lagrange multiplier.
In addition to increasing ρ with each iteration, the linear penalty vector is
updated according to:λ(k+1) = λ(k) − ρh(x)(10.44)
Algorithm 10.2 provides an implementation.

function augmented_lagrange_method(f, h, x, k_max; ρ=1, γ=2)λ = zeros(length(h(x)))
for k in 1 : k_max
p = x -> ρ/2*sum(h(x).^2) - λ⋅h(x)x = minimize(x -> f(x) + p(x), x)λ -= ρ*h(x)ρ *= γend
return x
end

10.9

Algorithm 10.2. The augmented Lagrange method for objective function f, equality constraint function h, initial point x, numberof iterations k_max, initial penalty
scalar ρ > 0, and penalty multiplier γ > 1. The function minimize
should be replaced with the minimization method of your choice.

Interior Point Methods

Interior point methods (algorithm 10.3), sometimes referred to as barrier methods, areoptimization methods that ensure that the search points always remain feasible.12
Interior point methods use a barrier function that approaches infinity as one
approaches a constraint boundary. This barrier function, pbarrier (x), must satisfy
several properties:

Interior point methods that are
stopped early produce nearly optimal, though feasible, design points.
Methods may be stopped early due
to time or processing constraints.

12

1. pbarrier (x) is continuous© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

184

c ha p te r 10 . c on stra i n ts

2. pbarrier (x) is nonnegative (pbarrier (x) ≥ 0) in the feasible region

3. pbarrier (x) approaches infinity as x approaches any constraint boundarySome examples of barrier functions are:Inverse Barrier:pbarrier (x) = − ∑i1gi ( x )(10.45)Log Barrier:pbarrier (x) = − ∑log(− g (x))

0
i

i

if gi (x) ≥ −1otherwise(10.46)

A problem with inequality constraints can be transformed into an unconstrained optimization problem
minimize f (x) +
x

1
p
(x)
ρ barrier(10.47)

When ρ is increased, the penalty for approaching the boundary decreases (figure 10.12).
Special care must be taken such that line searches do not leave the feasible
region. Line searches f (x + αd) are constrained to the interval α = [0, αu ], where
αu is the step to the nearest boundary. In practice, αu is chosen such that x + αd isjust inside the boundary to avoid the boundary singularity.
Like the penalty method, the interior point method begins with a low value forρ and slowly increases it until convergence. The interior point method is typicallyterminated when the difference between subsequent points is less than a certain
threshold. Figure 10.13 shows an example of the effect of incrementally increasingρ.
The interior point method requires a feasible point from which to start the
search. One convenient method for finding a feasible point is to optimize thequadratic penalty function
minimize pquadratic (x)
x

x
a

b
f (x)
f ( x ) + pbarrier ( x )f ( x ) + 12 pbarrier ( x )1f ( x ) + 10
pbarrier ( x )Figure 10.12. Applying the interior
point method with an inverse barrier for minimizing f subject tox ∈ [ a, b].

(10.48)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

10 . 9. i n te ri or p oi n t me thod s

Algorithm 10.3. The interior point
method for objective function f,barrier function p, initial point x,
initial penalty ρ > 0, penalty multiplier γ > 1, and stopping tolerance ϵ > 0.

function interior_point_method(f, p, x; ρ=1, γ=2, ϵ=0.001)
delta = Inf
while delta > ϵx′ = minimize(x -> f(x) + p(x)/ρ, x)delta = norm(x′ - x)
x = x′
ρ *= γend
return xendρ = 0.5x2x2

ρ = 1.0

ρ = 10.0

ρ = 100.0

x

x

x

185

0

−2
−2

0
x1

2

−2

0
x1

2

−2

0
x1

2

−2

0
x1

2

Figure 10.13. The interior point
method with the inverse barrier
applied to the flower function, appendix B.4, and the constraint
x12 + x22 ≥ 2© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

186

c ha p te r 10 . c on stra i n ts

10.10 Summary• Constraints are requirements on the design points that a solution must satisfy.
• Some constraints can be transformed or substituted into the problem to resultin an unconstrained optimization problem.
• Analytical methods using Lagrange multipliers yield the generalized Lagrangian
and the necessary conditions for optimality under constraints.
• A constrained optimization problem has a dual problem formulation that iseasier to solve and whose solution is a lower bound of the solution to theoriginal problem.
• Penalty methods penalize infeasible solutions and often provide gradient
information to the optimizer to guide infeasible points toward feasibility.
• Interior point methods maintain feasibility but use barrier functions to avoid
leaving the feasible set.

10.11 Exercises
Exercise 10.1. Solveminimizex

subject tox≥0x(10.49)using the quadratic penalty method with ρ > 0. Solve the problem in closed form.
Exercise 10.2. Solve the problem above using the count penalty method with
ρ > 1 and compare it to the quadratic penalty method.
Exercise 10.3. Suppose that you are solving a constrained problem with thepenalty method. You notice that the iterates remain infeasible and you decide tostop the algorithm. What can you do to be more successful in your next attempt?
Exercise 10.4. Consider a simple univariate minimization problem where you
minimize a function f ( x ) subject to x ≥ 0. Assume that you know that theconstraint is active, that is, x ∗ = 0 where x ∗ is the minimizer and f ′ ( x ∗ ) > 0 fromthe optimality conditions. Show that solving the same problem with the penalty
method(10.50)f ( x ) + (min( x, 0))2
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

10 . 11. e x e rc i se s 187

yields an infeasible solution with respect to the original problem.
Exercise 10.5. What is the advantage of the augmented Lagrange method compared to the quadratic penalty method?
Exercise 10.6. When would you use the barrier method in place of the penalty
method?
Exercise 10.7. Give an example of a smooth optimization problem, such that,for any penalty parameter ρ > 0, there exists a starting point x (1) for which thesteepest descent method diverges.
Exercise 10.8. Suppose you have an optimization problem
minimizexf (x)subject to h(x) = 0(10.51)g(x) ≤ 0
but do not have an initial feasible design. How would you find a feasible pointwith respect to the constraints, provided that one exists?
Exercise 10.9. Solve the constrained optimization problem 
4minimizesinxx
subject to(10.52)x ∈ [1, 10]

using both the transform x = t a,b ( x̂ ) and a sigmoid transform for constraint
bounds x ∈ [ a, b]:(b − a)x = s( x̂ ) = a +(10.53)
1 + e− x̂
Why is the t transform better than the sigmoid transform?
Exercise 10.10. Give an example of a quadratic objective function involving two
design variables where the addition of a linear constraint results in a different
optimum.
Exercise 10.11. Suppose we want to minimize x13 + x22 + x3 subject to the constraintthat x1 + 2x2 + 3x3 = 6. How might we transform this into an unconstrained
problem with the same minimizer?
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

188c ha p te r 10 . c on stra i n ts

Exercise 10.12. Suppose we want to minimize − x1 − 2x2 subject to the constraints
ax1 + x2 ≤ 5 and x1 , x2 ≥ 0. If a is a bounded constant, what range of values of awill result in an infinite number of optimal solutions?
Exercise 10.13. Consider using a penalty method to optimize
minimize

1 − x2

subject to|x| ≤ 2x(10.54)

Optimization with the penalty method typically involves running several optimizations with increasing penalty weights. Impatient engineers may wish to
optimize once using a very large penalty weight. Explain what issues are encountered for both the count penalty method and the quadratic penalty method.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

11

Linear Constrained Optimization

Linear programming involves solving optimization problems with linear objective functions and linear constraints. Many problems are naturally described bylinear programs, including problems from fields as diverse as transportation,communication networks, manufacturing, economics, and operations research.
Many problems that are not naturally linear can often be approximated by linear
programs. Several methods have been developed for exploiting the linear structure. Modern techniques and hardware can globally minimize problems withmillions of variables and millions of constraints.1

11.1 Problem Formulation
A linear programming problem, called a linear program,2 can be expressed inseveral forms. Each linear program consists of a linear objective function and aset of linear constraints:c⊤ x

minimizex

subject towLE x ≤ bi for i ∈ {1, 2, . . .}(i )⊤wGE x ≥ b j for j ∈ {1, 2, . . .}( j)⊤

(11.1)

wEQ x = bk for k ∈ {1, 2, . . .}(k)⊤where i, j, and k vary over finite sets of constraints. Such an optimization problem
is given in example 11.1. Transforming real problems into this mathematical form
is often nontrivial. This text focuses on the algorithms for obtaining solutions, butother texts discuss how to go about modeling real problems.3 Several interesting
conversions are given in example 11.2.

This chapter is a short introduction to linear programs andone variation of the simplex algorithm used to solve them. Several textbooks are dedicated entirely to linear programs, including R. J. Vanderbei, Linear Programming: Foundations and Extensions,
4th ed. Springer, 2014. There are avariety of packages for solving linear programs, such as Convex.jl
and JuMP.jl, both of which include interfaces to open-source
and commercial solvers.
1

A quadratic program is a generalization of a linear program,where the objective function isquadratic and the constraints arelinear. Common approaches forsolving such problems include
some of the algorithms discussed
in earlier chapters, including theinterior point method, augmented
Lagrange method, and conjugate
gradient. The simplex method, covered in this chapter, has also been
adapted for optimizing quadratic
programs. J. Nocedal and S. J.
Wright, Numerical Optimization,
2nd ed. Springer, 2006.
2

See, for example, H. P. Williams,
Model Building in Mathematical Programming, 5th ed. Wiley, 2013.
3

190 c ha p te r 11. l i n e a r c on stra i n e d op ti mi z a ti on

The following problem has a linear objective and linear constraints, making
it a linear program.

Example 11.1. An example linear
program.

2x1 − 3x2 + 7x3

minimize
x1 ,x2 ,x3

subject to 2x1 + 3x2 − 8x3 ≤5

4x1 + x2 + 3x3 ≤9x1 + x2 + 2x3 =1x1 − 5x2 − 3x3 ≥ −4

Many problems can be converted into linear programs that have the same
solution. Two examples are L1 and L∞ minimization problems:
minimizekAx − bk1

The first problem is equivalent to solving

Example 11.2. Common norm minimization problems that can be converted into linear programs.

minimizekAx − bk∞1⊤ sminimizex,ssubject to Ax − b ≤sAx − b ≥ −swith the additional variables s.
The second problem is equivalent to solving
minimizetsubject to Ax − b ≤

t1

x,t

Ax − b ≥ −t1

with the additional variable t.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

11. 1. p robl e m formu l a ti on

191

11.1.1 General Form
We can write linear programs more compactly using matrices and arrive at thegeneral form:4
minimizec⊤ x
x

ALE x ≤ bLE

subject to

AGE x ≥ bGE(11.2)

4
Here, each constraint is elementwise. For example, in writinga ≤ b,

we mean ai ≤ bi for all i.

AEQ x = bEQ

11.1.2 Standard Form
The general linear program given in equation (11.2) can be converted into standard
form where all constraints are less-than inequalities and the design variables arenonnegative
minimizec⊤ xx

subject to Ax ≤ b(11.3)

x≥0

Greater-than inequalities are inverted, and equality constraints are split in two
AGE x ≥ bGE→

AEQ x = bEQ→−AGE x ≤ −bGE
 A x≤ b
EQ
EQ
−AEQ x ≤ −bEQ

(11.4)

We must ensure that all x entries are nonnegative as well. Suppose we startwith a linear program where x is not constrained to be nonnegative:
minimizexc⊤ x

subject to Ax ≤ b(11.5)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

192 c ha p te r 11. l i n e a r c on stra i n e d op ti mi z a ti on

We replace x with x+ − x− and constrain x+ ≥ 0 and x− ≥ 0:
" #
i x+
h
⊤
⊤
minimizec−cx−
x+ ,x−" #hi x+
subject to A −A
≤b
x−
" #
x+
≥0
x−

(11.6)The linear objective function c⊤ x forms a flat ramp. The function increases inthe direction of c, and, as a result, all contour lines are perpendicular to c andparallel to one another as shown in figure 11.1.
A single inequality constraint w⊤ x ≤ b forms a half-space, or a region on one
side of a hyperplane. The hyperplane is perpendicular to w and is defined byw⊤ x = b as shown in figure 11.2. The region w⊤ x > b is on the +w side of thehyperplane, whereas w⊤ x < b is on the −w side of the hyperplane.
w⊤
x=

c⊤
x=

co

ns
t

c

Figure 11.1. The contours of a linear objective function c⊤ x, which
increase in the direction of c.

Figure 11.2. A linear constraint.

bw

w⊤ x > bw⊤ x < bHalf-spaces are convex sets (see appendix C.3), and the intersection of convex
sets is convex, as shown in figure 11.3. Thus, the feasible set of a linear program
will always form a convex set. Convexity of the feasible set, along with convexityof the objective function, implies that if we find a local feasible minimum, it isalso a global feasible minimum.
The feasible set is a convex region enclosed by flat faces. Depending on theregion’s configuration, the solution can lie at a vertex, on an edge, or on an entire
face. If the problem is not properly constrained, the solution can be unbounded,X

Figure 11.3. The intersection of linear constraints is a convex set.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

11. 1. p robl e m formu l a ti on

193

and, if the system is over-constrained, there is no feasible solution. Several suchcases are shown in figure 11.4.
Unbounded Solution

Infinite Solutions

No Solution

−c

−c

−c

−c

x2

One Solution

x∗
x∗
x1

11.1.3x1

x1

x1
Figure 11.4. Several different linear problem forms with different
solutions.

Equality Form

Linear programs are often solved in a third form, the equality form
minimizexc⊤ x

subject to Ax = b(11.7)

x≥0
where x and c each have n components, A is an m × n matrix, and b has mcomponents. In other words, we have n nonnegative design variables and asystem of m equations defining equality constraints.
The equality form has constraints in two parts. The first, Ax = b, forces thesolution to lie in an affine subspace.5 Such a constraint is convenient because
search techniques can constrain themselves to the constrained affine subspace toremain feasible. The second part of the constraints requires x ≥ 0, which forces
the solution to lie in the positive quadrant. The feasible set is thus the nonnegative
portion of an affine subspace. Example 11.3 provides a visualization for a simple
linear program.
Any linear program in standard form can be transformed to equality form. The
constraints are converted to:Ax ≤ b→Ax + s = b, s ≥ 0

5Informally, an affine subspace is avector space that has been translated such that its origin in a higherdimensional space is not necessarily 0.

(11.8)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

194c ha p te r 11. l i n e a r c on stra i n e d op ti mi z a ti on

Example 11.3. Feasible sets forthe equality form are hyperplanes.
sConsider the standard-form linear program:
minimizex

subject to

x≥1

x

2

When we convert this to equality form, we getx2minimizex

subject to

x−s = 1x,sx, s ≥ 0

The equality constraint requires that feasible points fall on the line x − s =1. That line is a one-dimensional affine subspace of the two-dimensional
Euclidean space.

by introducing slack variables s. These variables take up the extra slack to enforce
equality.
Starting with a linear program:
minimizexc⊤ x

subject to Ax ≤ b(11.9)x≥0

We introduce the slack variables:
minimizex,ssubject to

" #
i x
c⊤ 0⊤
s
" #
h
i x
=b
A I
s
" #
x
≥0
s
h

(11.10)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

411. 2. si mp l e x a l g ori thm 195

Example 11.4 demonstrates converting from standard to equality form.
Example 11.4. Converting a linear
program to equality form.

Consider the linear programminimizex

5x1 + 4x2

subject to 2x1 + 3x2 ≤ 5
4x1 + x2 ≤ 11

To convert to equality form, we first introduce two slack variables:
minimizex,s

5x1 + 4x2

subject to 2x1 + 3x2 + s1 = 5
4x1 + x2 + s2 = 11
s1 , s2 ≥ 0
We then split x:5( x1+ − x1− ) + 4( x2+ − x2− )minimize
x+ ,x− ,ssubject to 2( x1+ − x1− ) + 3( x2+ − x2− ) + s1 = 5

4( x1+ − x1− ) + ( x2+ − x2− ) + s2 = 11
x1+ , x1− , x2+ , x2− , s1 , s2 ≥ 0

11.2 Simplex Algorithm
The simplex algorithm solves linear programs by moving from vertex to vertex of
the feasible set.6 The method is guaranteed to arrive at an optimal solution solong as the linear program is feasible and bounded.
The simplex algorithm operates on equality-form linear programs (Ax = b,x ≥ 0). We assume that the rows of A are linearly independent.7 We also assume
that the problem has no more equality constraints than it has design variables(m ≤ n), which ensures that the problem is not over constrained. A preprocessing
phase guarantees that A satisfies these conditions.

The simplex algorithm was originally developed in the 1940s by
George Dantzig. A history of thedevelopment can be found here:
G. B. Dantzig, “Origins of the Simplex Method,” in A History of Scientific Computing, S. G. Nash, ed.,ACM, 1990, pp. 141–151.
6

A matrix whose rows are linearly
independent is said to have fullrow rank. Linear independence isachieved by removing redundantequality constraints.
7© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

196

c ha p te r 11. l i n e a r c on stra i n e d op ti mi z a ti on

11.2.1 Vertices
Linear programs in equality form have feasible sets in the form of convex polytopes,
which are geometric objects with flat faces. These polytopes are formed by theintersection of the equality constraints with the positive quadrant. Associatedwith a polytope are vertices, which are points in the feasible set that do not lie
between any other points in the feasible set.
The feasible set consists of several different types of design points. Points on
the interior are never optimal because they can be improved by moving along−c. Points on faces can be optimal only if the face is perpendicular to c. Points
on faces not perpendicular to c can be improved by sliding along the face in the
direction of the projection of −c onto the face. Similarly, points on edges can be
optimal only if the edge is perpendicular to c, and can otherwise be improvedby sliding along the projection of −c onto the edge. Finally, vertices can also be
optimal.
The simplex algorithm produces an optimal vertex. If a linear program hasa bounded solution, then it also contains at least one vertex. Furthermore, atleast one solution must lie at a vertex. In the case where an entire edge or face is
optimal, a vertex-solution is just as good as any other.
Every vertex for a linear program in equality form can be uniquely defined byn − m components of x that equal zero. These components are actively constrainedby xi ≥ 0. Figure 11.5 visualizes the active constraints needed to identify a vertex.
The equality constraint Ax = b has a unique solution when A is square.
We have assumed that m ≤ n, so choosing m design variables and setting theremaining variables to zero effectively removes n − m columns of A, yielding an
m × m constraint matrix (see example 11.5).
For a problem with 5 design variables and 3 constraints, setting 2 variables
to zero uniquely defines a point.


a11

 a21
a31

a12
a22
a32

a13
a23
a33

a14
a24
a34

 
 x1


a15 
a11
0
  
a25   x3  =  a21 
a35  x4 
a31
0

a13
a23
a33

X
vertexFigure 11.5. In two dimensions, anyvertex will have at least two active
constraints.

Example 11.5. Setting n − m components of x to zero can uniquely
define a point.

   b1
x1
a14
   
a24   x3  = b2 
b3
x4
a34

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

11. 2. si mp l e x a l g ori thm 197

The indices into the components {1, . . . , n} of any vertex can be partitioned
into two sets, B and V , such that:• The design values associated with indices in V are zero:i ∈ V =⇒ xi = 0(11.11)• The design values associated with indices in B may or may not be zero:i ∈ B =⇒ xi ≥ 0(11.12)• B has exactly m elements and V has exactly n − m elements.

We use xB to refer to the vector consisting of the components of x that are in B
and xV to refer to the vector consisting of the components of x that are in V . Notethat xV = 0.
The vertex associated with a partition (B , V ) can be obtained using the m × m
matrix AB formed by the m columns of A selected by B :8
Ax = AB xB = b→

1xB = A−
B b(11.13)Knowing xB is sufficient to construct x; the remaining design variables are zero. Algorithm 11.1 implements this procedure, and example 11.6 demonstrates verifyingthat a given design point is a vertex.
mutable struct LinearProgram
A
b
c
end
function get_vertex(B, LP)A, b, c = LP.A, LP.b, LP.cb_inds = sort!(collect(B))
AB = A[:,b_inds]
xB = AB\bx = zeros(length(c))
x[b_inds] = xB
return x
end

If B and V identify a vertex, thenthe columns of AB must be linearly
independent because Ax = b musthave exactly one solution. Hence,
AB xB = b must have exactly one
solution. This linear independence
guarantees that AB is invertible.
8

Algorithm 11.1. A method for extracting the vertex associated witha partition B and a linear program
LP in equality form. We introduce
the special type LinearProgram forlinear programs.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

198

c ha p te r 11. l i n e a r c on stra i n e d op ti mi z a ti on

While every vertex has an associated partition (B , V ), not every partition corresponds to a vertex. A partition corresponds to a vertex only if AB is nonsingularand the design obtained by applying equation (11.13) is feasible.9 Identifying partitions that correspond to vertices is nontrivial, and we show in section 11.2.4 thatfinding such a partition involves solving a linear program! The simplex algorithm
operates in two phases—an initialization phase that identifies a vertex partition andan optimization phase that transitions between vertex partitions toward a partition
corresponding to an optimal vertex. We will discuss both of these phases later inthis section.
Consider the constraints:

1
1

0
−
1

2
1

1
2
2

 

2
1
 

3 x =  −1 ,3−1For example, B = {1, 2} for theconstraints
 

 x
 
1 2 0  1
1
x2 =
1 2 1
1
x3

9

corresponds to   
1 2 x11=
1 2 x2
1which does not produce an invertible AB and does not have a uniquesolution.
Example 11.6. Verifying that a design point is a vertex for constraints
in equality form.

x≥0

Consider the design point x = [1, 1, 0, 0]. We can verify that x is feasible
and that it has no more than three nonzero components. We can choose either
B = {1, 2, 3} or B = {1, 2, 4}. Both


1
1 1


A{1,2,3} = 0 −1 2
2
1 2

and



1

A{1,2,4} = 0
2

1
−1
1


1

3
−1

are invertible. Thus, x is a vertex of the feasible set polytope.

11.2.2 First-Order Necessary Conditions
The first-order necessary conditions (FONCs) for optimality are used to determine
when a vertex is optimal and to inform how to transition to a more favorable
vertex. We construct a Lagrangian for the equality form of the linear program:10

L(x, µ, λ) = c⊤ x − µ⊤ x − λ⊤ (Ax − b)(11.14)Note that in x ≥ 0 the polarityof the inequality must be inverted
by multiplying both sides by −1,
yielding the negative sign in frontof µ. The Lagrangian can be defined with either positive or negative λ.
10© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

11. 2. si mp l e x a l g ori thm

199with the following FONCs:
1. feasibility: Ax = b, x ≥ 0

2. dual feasibility: µ ≥ 0

3. complementary slackness: µ ⊙ x = 0

4. stationarity: A⊤ λ + µ = c

The FONCs are sufficient conditions for optimality for linear programs. Thus,
if µ and λ can be computed for a given vertex and all four FONC equations are
satisfied, then the vertex is optimal.
We can decompose the stationarity condition into B and V components: A⊤ λ + µ = c
B
BBA⊤ λ + µ = c →(11.15) A⊤ λ + µ = cVVV

We can choose µB = 0 to satisfy complementary slackness. The value of λ canbe computed from B :11
A⊤µB = cBB λ + |{z}(11.16)=0
We can use this to obtain

λ = A−⊤B cB

A⊤
V λ + µV = cV

µV = cV − A⊤
Vλ

⊤
1
µV = cV − A−cB
B AV

We use A−⊤ to refer to the transpose of the inverse of A: ⊤   −1
= A⊤
A−⊤ = A−1

11

(11.17)Knowing µV allows us to assess the optimality of the vertices. If µV contains
negative components, then dual feasibility is not satisfied and the vertex is suboptimal.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

200 c ha p te r 11. l i n e a r c on stra i n e d op ti mi z a ti on

11.2.3 Optimization Phase
The simplex algorithm maintains a partition (B , V ), which corresponds to a vertexof the feasible set polytope. The partition can be updated by swapping indicesbetween B and V . Such a swap equates to moving from one vertex along an edgeof the feasible set polytope to another vertex. If the initial partition corresponds
to a vertex and the problem is bounded, the simplex algorithm is guaranteed toconverge to an optimum.
A transition x → x′ between vertices must satisfy Ax′ = b. Starting with apartition defined by B , we choose an entering index q ∈ V that is to enter B using
one of the heuristics described near the end of this section. The new vertex x′
must satisfy:
Ax′ = AB x′B + A{q} xq′ = AB xB = Ax = b(11.18)One leaving index p ∈ B in x′B becomes zero during the transition and is replacedby the column of A corresponding to index q. This action is referred to as pivoting.
We can solve for the new design point(11.19)1′x′B = xB − A−
B A{ q } x q

A particular leaving index p ∈ B becomes active when:
1x′B p = 0 = (xB ) p − A−
A
xq′
q
{
}
B
p

(11.20)and is thus obtained by increasing xq = 0 to xq′ with:xq′ = (xB ) p

1
A−
B A{ q } p(11.21)The leaving index is obtained using the minimum ratio test, which computes foreach potential leaving index and selects the one with minimum xq′ . We then swap
p and q between B and V . The edge transition is implemented in algorithm 11.2.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

11. 2. si mp l e x a l g ori thm 201

Algorithm 11.2. A method for computing the index p and the newcoordinate value xq′ obtained byincreasing index q of the vertex
defined by the partition B in theequality-form linear program LP.

function edge_transition(LP, B, q)
A, b, c = LP.A, LP.b, LP.cn = size(A, 2)
b_inds = sort(B)n_inds = sort!(setdiff(1:n, B))
AB = A[:,b_inds]
d, xB = AB\A[:,n_inds[q]], AB\bend

p, xq′ = 0, Inf
for i in 1 : length(d)if d[i] > 0v = xB[i] / d[i]if v < xq′
p, xq′ = i, v
end
end
end
return (p, xq′)

The effect that a transition has on the objective function can be computed using
xq′ . The objective function value at the new vertex is12′′c⊤ x′ = c⊤
B xB + c q x q


−1
′
′
= c⊤
B xB − AB A{ q } x q + c q x q
′
′
⊤ −1
= c⊤
B xB − cB AB A{ q } x q + c q x q

′
′
= c⊤
B xB − c q − µ q x q + c q x q= c⊤ x + µq xq′(11.22)

12Here, we used the fact that λ =⊤A−⊤
B cB and that A{q} λ = cq − µq .

(11.23)(11.24)(11.25)(11.26)

Choosing an entering index q decreases the objective function value byc⊤ x′ − c⊤ x = µq xq′(11.27)The objective function decreases only if µq is negative. In order to progress toward optimality, we must choose an index q in V such that µq is negative. If allcomponents of µV are non-negative, we have found a global optimum.
Since there can be multiple negative entries in µV , different heuristics can beused to select an entering index:13

Modern implementations use
more sophisticated rules. For example, see J. J. Forrest and D. Goldfarb, “Steepest-Edge Simplex Algorithms for Linear Programming,”
Mathematical Programming, vol. 57,
no. 1, pp. 341–374, 1992.

13© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

202 c ha p te r 11. l i n e a r c on stra i n e d op ti mi z a ti on• Greedy heuristic, which chooses a q that maximally reduces c⊤ x.
• Dantzig’s rule, which chooses the q with the most negative entry in µ. This ruleis easy to calculate, but it does not guarantee the maximum reduction in c⊤ x.
It is also sensitive to scaling of the constraints.14• Bland’s rule, which chooses the first q with a negative entry in µ. When used on
its own, Bland’s rule tends to result in poor performance in practice. However,this rule can help us prevent cycles, which occur when we return to a vertex
we have visited before without decreasing the objective function. This rule isusually used only after no improvements have been made for several iterationsof a different rule to break out of a cycle and ensure convergence.

For constraint A⊤ x = b →αA⊤ x = αb, α > 0, we do
not change the solution but theLagrange multipliers are scaled,
λ → α−1 λ.

14

One iteration of the simplex method’s optimization phase moves a vertex
partition to a neighboring vertex based on a heuristic for the entering index. Algorithm 11.3 implements such an iteration with the greedy heuristic. Example 11.7
demonstrates using the simplex algorithm starting from a known vertex partition
to solve a linear program.

11.2.4 Initialization Phase
The optimization phase of the simplex algorithm is implemented in algorithm 11.4.
Unfortunately, algorithm 11.4 requires an initial partition that corresponds to avertex. If we do not have such a partition, we must solve an auxiliary linear program
to obtain this partition as part of an initialization phase.
The auxiliary linear program to be solved in the initialization phase includes
extra variables z ∈ R m , which we seek to zero out:15
" #
i x
h
⊤
⊤
minimize
0
1
x,z
z
" #
h
i x
(11.28)=bsubject to A Z
z
" #
x
≥0
z

The values for z represent theamount by which Ax = b is violated. By zeroing out z, we finda feasible point. If, in solving theauxiliary problem, we do not finda vertex with a zeroed-out z, thenwe can conclude that the problem is infeasible. Furthermore, it
is not always necessary to add
all m extra variables, especiallywhen slack variables are included
when transforming between standard and equality form.
15© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

11. 2. si mp l e x a l g ori thm 203

Example 11.7. Solving a linear program with the simplex algorithm.

Consider the equality-form linear program with

A=

"

1
−4

1
2

1
0

#

" #
9
b=
,
2

0
,
1




3
 −1
 
c= 
 0 
0

and the initial vertex defined by B = {3, 4}. After verifying that B defines afeasible vertex, we can begin one iteration of the simplex algorithm.
We extract xB :1
xB = A−
B b=

"

1
0

0
1

# −1 " # " #
9
9
=
2
2

We then compute λ:λ = A−⊤
B cB =

"

1
0

0
1

#−⊤ " #
0=00and µV :

1µV = cV − A−
B AV

⊤

# "
1
3
−
cB =
0
−1
"

0
1

# −1 "

1−4#⊤ " # " #
3
1  0
=
−1
0
2

We find that µV contains negative elements, so our current B is suboptimal.
We will pivot on the index of the only negative element, q = 2. An edge
1transition is run from xB in the direction −A−
B A{q} = [1, 2].
Using equation (11.19), we increase xq′ until a new constraint becomes
active. In this case, xq′ = 1 causes x4 to become zero. We update our set ofbasic indices to B = {2, 3}.
In the second iteration, we find:
#
"
#
"
" #
1
0
1
.
, µV =
, λ=
xB =
1/2
−1/2
8
The vertex is optimal because µV has no negative entries. Our algorithm
thus terminates with B = {2, 3}, for which the design point is x∗ = [0, 1, 8, 0].
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

204

c ha p te r 11. l i n e a r c on stra i n e d op ti mi z a ti on

function step_lp!(B, LP)A, b, c = LP.A, LP.b, LP.cn = size(A, 2)
b_inds = sort!(B)n_inds = sort!(setdiff(1:n, B))
AB, AV = A[:,b_inds], A[:,n_inds]
xB = AB\bcB = c[b_inds]
λ = AB' \ cB
cV = c[n_inds]
μV = cV - AV'*λAlgorithm 11.3. A single iterationof the simplex algorithm in which
the set B is moved from one vertex to a neighbor while maximallydecreasing the objective function.
Here, step_lp! takes a partition
defined by B and a linear programLP.

q, p, xq′, Δ = 0, 0, Inf, Inf
for i in 1 : length(μV)
if μV[i] < 0pi, xi′ = edge_transition(LP, B, i)
if μV[i]*xi′ < Δ
q, p, xq′, Δ = i, pi, xi′, μV[i]*xi′
end
end
end
if q == 0
return (B, true) # optimal point foundend
if isinf(xq′)
error("unbounded")
end

end

j = findfirst(isequal(b_inds[p]), B)
B[j] = n_inds[q] # swap indices
return (B, false) # new vertex but not optimalfunction minimize_lp!(B, LP)done = false
while !done
B, done = step_lp!(B, LP)
end
return B
end

Algorithm 11.4. Minimizing a linear program given a vertex partition defined by B and a linear program LP.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

11. 2. si mp l e x a l g ori thm

where Z is a diagonal matrix whose diagonal entries are+1 if b ≥ 0iZii =−1 otherwise.

205(11.29)The auxiliary linear program is solved with a partition defined by B , which
selects only the z values. The corresponding vertex has x = 0, and each z-element
is the absolute value of the corresponding b-value: z j = |b j |. This initial vertex
can easily be shown to be feasible.
Example 11.8 demonstrates using an auxiliary linear program to obtain afeasible vertex.
Example 11.8. Using an auxiliary
linear program to obtain a feasible
vertex.

Consider the equality-form linear program:
minimize
x1 ,x2 ,x3

c1 x1 + c2 x2 + c3 x3

subject to 2x1 − 1x2 + 2x3 =

1x1 , x2 , x3 ≥0

5x1 + 1x2 − 3x3 = −2

We can identify a feasible vertex by solving:
minimize

x1 ,x2 ,x3 ,z1 ,z2

z1 + z2

subject to 2x1 − 1x2 + 2x3 + z1 =1x1 , x2 , x3 , z1 , z2 ≥0

5x1 + 1x2 − 3x3 − z2 = −2

with an initial vertex defined by B = {4, 5}.
The initial vertex has:# −1 " # " #
"
1
0
1
1
(1)
−1
xB = AB bB =
=
0 −1
−2
2
and is thus x(1) = [0, 0, 0, 1, 2]. Solving the auxiliary problem yields x∗ ≈[0.045, 1.713, 1.312, 0, 0]. Thus [0.045, 1.713, 1.312] is a feasible vertex in theoriginal problem.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

206

c ha p te r 11. l i n e a r c on stra i n e d op ti mi z a ti on

The partition obtained by solving the auxiliary linear program will produce a
feasible design point, Ax = b because z will have been zeroed out. If z is nonzero,then the original linear program is infeasible. If z is zero, then the resulting
partition can be used as the initial partition for the optimization phase of thesimplex algorithm. The original problem must be slightly modified to incorporate
the new z variables:
" #
i x
h
⊤
⊤
minimize
c
0
x,z
z
"
#" # " #A I xbsubject to(11.30)
=
0 I z
0
" #
x
≥0
z
The z values must be included. Despite their vector counterparts being zero, it
is possible that some indices in the components of z are included in the initial
partition B . One can inspect the initial partition and include only the specificcomponents that are needed.
The solution (x∗ , z∗ ) obtained by solving the second LP will have z∗ = 0. Thus,∗x will be a solution to the original linear problem.
An implementation for the complete simplex algorithm is given in algorithm 11.5.

11.3

Dual Certificates

Suppose we have a candidate solution and we want to verify that it is optimal.
Verifying optimality using dual certificates (algorithm 11.6) is useful in many cases,
such as debugging our linear program code.
We know from the FONCs for constrained optimization that the optimal valueof the dual problem d∗ is a lower bound of the optimal value of the primal problemp∗ . Linear programs are linear and convex, and one can show that the optimalvalue of the dual problem is also the optimal value of the primal problem, d∗ = p∗ .

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

11. 3. d u a l c e rti fi c a te sfunction minimize_lp(LP)A, b, c = LP.A, LP.b, LP.cm, n = size(A)z = ones(m)Z = Matrix(Diagonal([j ≥ 0 ? 1 : -1 for j in b]))207

Algorithm 11.5. The simplex algorithm for solving linear programsin equality form when an initial
partition is not known.

A′ = hcat(A, Z)
b′ = bc′ = vcat(zeros(n), z)
LP_init = LinearProgram(A′, b′, c′)
B = collect(1:m).+nminimize_lp!(B, LP_init)
if any(i-> i > n, B)
error("infeasible")
end

end

A′′ = [AMatrix(1.0I, m, m);zeros(m,n) Matrix(1.0I, m, m)]b′′ = vcat(b, zeros(m))
c′′ = c′
LP_opt = LinearProgram(A′′, b′′, c′′)minimize_lp!(B, LP_opt)
return get_vertex(B, LP_opt)[1:n]© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

208

c ha p te r 11. l i n e a r c on stra i n e d op ti mi z a ti on

The primal linear program can be converted to its dual form:
max min L(x, µ, λ) = max min c⊤ x − µ⊤ x − λ⊤ (Ax − b)(11.31)= max min(c − µ − A⊤ λ)⊤ x + λ⊤ b

(11.32)

x

µ≥0,λ

µ≥0,λ

x

µ≥0,λ

x

From the FONC, we know c − µ − A⊤ λ = 0, which allows us to drop the firstterm in the objective above. In addition, we know µ = c − A⊤ λ ≥ 0, which
implies A⊤ λ ≤ c. In summary, we have:Primal Form (equality)
minimizex

Dual Formc⊤ xsubject to Ax = b
x≥0

maximize
λb⊤ λsubject to A⊤ λ ≤ c

If the primal problem has n variables and m equality constraints, then the dual
problem has m variables and n constraints.16 Furthermore, the dual of the dual is
the primal problem.
Optimality can be assessed by verifying three properties. If someone claims∗(x , λ∗ ) is optimal, we can quickly verify the claim by checking whether all threeof the following conditions are satisfied:
1. x∗ is feasible in the primal problem.
2. λ∗ is feasible in the dual problem.

An alternative to the simplex algorithm, the self-dual simplex algorithm, tends to be faster in practice. It does not require that the ma1
trix AB satisfy xB = A−B b ≥ 0.
The self-dual simplex algorithm isa modification of the simplex algorithm for the dual of the linear
programming problem in standard
form.
16

3. p∗ = c⊤ x∗ = b⊤ λ∗ = d∗ .
Dual certificates are used in example 11.9 to verify the solution to a linear
program.
function dual_certificate(LP, x, λ, ϵ=1e-6)
A, b, c = LP.A, LP.b, LP.c
primal_feasible = all(x .≥ 0) && A*x ≈ b
dual_feasible = all(A'*λ .≤ c)return primal_feasible && dual_feasible &&isapprox(c⋅x, b⋅λ, atol=ϵ)end

Algorithm 11.6. A method forchecking whether a candidate solution given by design point x anddual point λ for the linear program
LP in equality form is optimal. The
parameter ϵ controls the tolerance
for the equality constraint.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

11. 3. d u a l c e rti fi c a te s 209

Consider the standard-form linear program with

 

1
1
1 −1

 

A =  −120 , b =  −2 ,5
1
2
3

Example 11.9. Verifying a solution
using dual certificates.


1 
c =  1−1


We would like to determine whether x∗ = [2, 0, 1] and λ∗ = [1, 0, 0] are an
optimal solution pair. We first verify that x∗ is feasible:
Ax∗ = [1, −2, 5] = b,x∗ ≥ 0We then verify that λ∗ is dual-feasible:A⊤ λ∗ = [1, 1, −1] ≤ cFinally, we verify that p∗ and d∗ are the same:p ∗ = c⊤ x∗ = 1 = b⊤ λ∗ = d ∗We conclude that (x∗ , λ∗ ) are optimal.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

210

11.4

c ha p te r 11. l i n e a r c on stra i n e d op ti mi z a ti on

Summary• Linear programs are problems consisting of a linear objective function andlinear constraints.
• The simplex algorithm can optimize linear programs globally in an efficientmanner.
• Dual certificates allow us to verify that a candidate primal-dual solution pairis optimal.

11.5 Exercises
Exercise 11.1. Suppose you do not know any optimization algorithm for solving
a linear program. You decide to evaluate all the vertices and determine, by inspection, which one minimizes the objective function. Give a loose upper boundon the number of possible minimizers you will examine. Furthermore, does thismethod properly handle all linear constrained optimization problems?
Exercise 11.2. If the program in example 11.1 is bounded below, argue that thesimplex method must converge.
Exercise 11.3. Suppose we want to minimize 6x1 + 5x2 subject to the constraint
3x1 − 2x2 ≥ 5. How would we translate this problem into a linear program inequality form with the same minimizer?
Exercise 11.4. Suppose your optimization algorithm has found a search directiond and you want to conduct a line search. However, you know that there is a linear
constraint w⊤ x ≥ 0. How would you modify the line search to take this constraint
into account? You can assume that your current design point is feasible.
Exercise 11.5. Reformulate the linear programminimizexc⊤ x

subject to Ax ≥ 0(11.33)

into an unconstrained optimization problem with a log barrier penalty.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12Multiobjective Optimization

Previous chapters have developed methods for optimizing single-objective functions, but this chapter is concerned with multiobjective optimization, or vector optimization, where we must optimize with respect to several objectives simultaneously. Engineering is often a tradeoff between cost, performance, and timeto-market, and it is often unclear how to prioritize different objectives. We will
discuss various methods for transforming vector-valued objective functions toscalar-valued objective functions so that we can use the algorithms discussed inprevious chapters to arrive at an optimum. In addition, we will discuss algorithms
for identifying the set of design points that represent the best tradeoff between
objectives, without having to commit to a particular prioritization of objectives.
These design points can then be presented to experts who can then identify themost desirable design.1

12.1 Pareto Optimality
The notion of Pareto optimality is useful when discussing problems where there
are multiple objectives. A design is Pareto optimal if it is impossible to improvein one objective without worsening at least one other objective. In multiobjective
design optimization, we can generally focus our efforts on designs that are Pareto
optimal without having to commit to a particular tradeoff between objectives.
This section introduces some definitions and concepts that are helpful when
discussing approaches to identifying Pareto-optimal designs.

Additional methods are surveyed
in R. T. Marler and J. S. Arora, “Survey of Multi-Objective Optimization Methods for Engineering,”
Structural and Multidisciplinary Optimization, vol. 26, no. 6, pp. 369–
395, 2004. For a textbook dedicatedentirely to multiobjective optimization, see K. Miettinen, Nonlinear
Multiobjective Optimization. Kluwer
Academic Publishers, 1999.
1

212

c ha p te r 12. mu l ti obj e c ti ve op ti mi z a ti on

12.1.1 DominanceIn single-objective optimization, two design points x and x′ can be ranked objectively based on their scalar function values. The point x′ is better whenever f (x′ )is less than f (x).
In multiobjective optimization, our objective function f returns an m-dimensional
vector of values y when evaluated at a design point x. The different dimensionsof y correspond to different objectives, sometimes also referred to as metrics orcriteria. We can objectively rank two design points x and x′ only when one isbetter in at least one objective and no worse in any other. That is, x dominates x′ if
and only if
f i (x) ≤ f i (x′ ) for all i in {1, . . . , m}
(12.1)
and f i (x) < f i (x′ ) for some ias compactly implemented in algorithm 12.1.
dominates(y, y′) = all(y .≤ y′) && any(y .< y′)Figure 12.1 shows that in multiple dimensions there are regions with dominance ambiguity. This ambiguity arises whenever x is better in some objectives
and x′ is better in others. Several methods exist for resolving these ambiguities.
Single Objective

Multiple Objectivesf (x)yf (x′ ) < f (x )x′ is betterf (x′ ) > f (x )x′ is worse

y2

f(x)

f (x′ ) > f (x )x′ is worse

Algorithm 12.1. A method forchecking whether x dominates x′ ,where y is the vector of objective
values for f(x) and y′ is the vectorof objective values for f(x′ ).

Figure 12.1. Design points canbe objectively ranked in singleobjective optimization but canbe objectively ranked in multiobjective optimization only insome cases.

f (x′ ) < f (x )x′ is bettery1© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12. 1. p a re to op ti ma l i ty 213

12.1.2

Pareto Frontier

In mathematics, an image of an input set through some function is the set of
all possible outputs of that function when evaluated on elements of that input
set. We will denote the image of X through f as Y , and we will refer to Y as
the criterion space. Figure 12.2 shows examples of criterion space for problems
with single and multiple objectives. As illustrated, the criterion space in singleobjective optimization is one dimensional. All of the global optima share a single
objective function value, y∗ . In multiobjective optimization, the criterion space ism-dimensional, where m is the number of objectives. There is typically no globally
best objective function value because there may be ambiguity when tradeoffs
between objectives are not specified.
Single Objective

Multiple Objectives

y∗

y2

Y

y

Y

Figure 12.2. The criterion space isthe set of all objective values obtained by feasible design points.
Well-posed problems have criterion spaces that are bounded frombelow, but they do not have to bebounded from above. The Pareto
frontier is highlighted in dark blue.

y1

In multiobjective optimization, we can define the notion of Pareto optimality.
A design point x is Pareto-optimal when no point dominates it. That is, x ∈ X
is Pareto-optimal if there does not exist an x′ ∈ X such that x′ dominates x.
The set of Pareto-optimal points forms the Pareto frontier. The Pareto frontier isvaluable for helping decision-makers make design trade decisions as discussed inexample 12.1. In two dimensions, the Pareto frontier is also referred to as a Pareto
curve.
All Pareto-optimal points lie on the boundary of the criterion space. Some multiobjective optimization methods also find weakly Pareto-optimal points. Whereas
Pareto-optimal points are those such that no other point improves at least one ob© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

214

c ha p te r 12. mu l ti obj e c ti ve op ti mi z a ti on

jective, weakly Pareto-optimal points are those such that no other point improves
all of the objectives (figure 12.3). That is, x ∈ X is weakly Pareto-optimal if there
does not exist an x′ ∈ X such that f(x′ ) < f(x). Pareto-optimal points are also
weakly Pareto optimal. Weakly Pareto-optimal points are not necessarily Pareto
optimal.
Several methods discussed below use another special point. We define theutopia point to be the point in the objective space consisting of the component-wise
optima:utopia= minimize f i (x)(12.2)yix∈X

The utopia point is often not attainable; optimizing one component typicallyrequires a tradeoff in another component.

y2

y1
Figure 12.3.
Weakly Paretooptimal points, shown in red,cannot be improved simultaneously in all objectives.

12.1.3 Pareto Frontier Generation
There are several methods for generating Pareto frontiers. A naive approachis to sample design points throughout the design space and then to identify
the nondominated points (algorithm 12.2). This approach is typically wasteful,leading to many dominated design points as shown in figure 12.4. In addition, thisapproach does not guarantee a smooth or correct Pareto frontier. The remainderof this chapter discusses more effective ways to generate Pareto frontiers.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12. 1. p a re to op ti ma l i ty 215

When constructing a collision avoidance system for aircraft, one must minimize both the collision rate and the alert rate. Although more alerts can resultin preventing more collisions if the alerts are followed, too many alerts canresult in pilots losing trust in the system and lead to decreased compliancewith the system. Hence, the designers of the system must carefully tradealerts and collision risk.

Example 12.1. An approximate
Pareto frontier obtained from
evaluating many different design
points for an aircraft collision
avoidance system.

Pareto dominated points(suboptimal)
collision ratecriterion space(denoted Y )IdealPareto Frontieralert rate

By varying the collision avoidance system’s design parameters, we canobtain many different collision avoidance systems, but, as the figure shows,some of these will be better than others. A Pareto frontier can be extracted
to help domain experts and regulators understand the effects that objective
tradeoffs will have on the optimized system.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

216 c ha p te r 12. mu l ti obj e c ti ve op ti mi z a ti on

Algorithm 12.2. A method forgenerating a Pareto frontier using
randomly sampled design points
xs and their multiobjective values
ys. Both the Pareto-optimal design
points and their objective values
are returned.

function naive_pareto(xs, ys)pareto_xs, pareto_ys = similar(xs, 0), similar(ys, 0)for (x,y) in zip(xs,ys)
if !any(dominates(y′,y) for y′ in ys)
push!(pareto_xs, x)
push!(pareto_ys, y)
end
end
return (pareto_xs, pareto_ys)
end

y2

12.2

Constraint Methods

Constraints can be used to cut out sections of the Pareto frontier and obtain asingle optimal point in the criterion space. Constraints can be supplied either
by the problem designer or automatically obtained based on an ordering of theobjectives.

12.2.1 Constraint Methody1Figure 12.4. Generating Pareto
frontiers with naively scatteredpoints is straightforward but inefficient and approximate.

The constraint method constrains all but one of the objectives. Here we choose f 1
without loss of generality:
minimizef 1 (x)subject tof 2 (x) ≤ c2x

f 3 (x) ≤ c3..
.

(12.3)f m (x) ≤ c mx∈XGiven the vector c, the constraint method produces a unique optimal pointin the criterion space, provided that the constraints are feasible. The constraint
method can be used to generate Pareto frontiers by varying c as shown in figure 12.5.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12. 2. c on stra i n t me thod s217

Figure 12.5. The constraint method
for generating a Pareto frontier.
This method can identify points inthe concave region of the Pareto
frontier.

y2c2 largeYc2 smally1

12.2.2 Lexicographic Method
The lexicographic method ranks the objectives in order of importance. A seriesof single-objective optimizations are performed on the objectives in order ofimportance. Each optimization problem includes constraints to preserve theoptimality with respect to previously optimized objectives as shown in figure 12.6.

minimizef 1 (x)

minimizef 2 (x)

minimizef 3 (x)subject tox∈Xsubject tof 1 (x) ≤ y1∗

subject tof 1 (x) ≤ y1∗

x

y1∗

x

x

f 2 (x) ≤ y2∗

x∈X

y2∗

Iterations are always feasible because the minimum point from the previous
optimization is always feasible. The constraints could also be replaced with equalities, but inequalities are often easier for optimizers to enforce. In addition, if the
optimization method used is not optimal, then subsequent optimizations may
encounter better solutions that would otherwise be rejected. The lexicographic
method is sensitive to the ordering of the objective functions.

x∈X

y3∗
Figure 12.6. The lexicographic
method for an optimization problem with three objectives.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

218 c ha p te r 12. mu l ti obj e c ti ve op ti mi z a ti on

12.3 Weight Methods
A designer can sometimes identify preferences between the objectives and encode
these preferences as a vector of weights. In cases where the choice of weights is not
obvious, we can generate a Pareto frontier by sweeping over the space of weights.
This section also discusses a variety of alternative methods for transforming
multiobjective functions into single-objective functions.

12.3.1 Weighted Sum Method
The weighted sum method (algorithm 12.3) uses a vector of weights w to convert f
to a single objective f :2
f (x ) = w⊤ f (x )(12.4)
where the weights are nonnegative and sum to 1. The weights can be interpreted
as costs associated with each objective. The Pareto frontier can be extracted byvarying w and solving the associated optimization problem with the objective inequation (12.4). In two dimensions, we vary w1 from 0 to 1, setting w2 = 1 − w1 .
This approach is illustrated in figure 12.7.

L. Zadeh, “Optimality and NonScalar-Valued Performance Criteria,” IEEE Transactions on AutomaticControl, vol. 8, no. 1, pp. 59–60,
1963.
2

Figure 12.7. The weighted sum
method used to generate a Pareto
frontier. Varying the weights allows us to trace the Pareto frontier.

Y
y2
w1
w1 = 0w2

y2

w1 = 1y1

In contrast with the constraint method, the weighted sum method cannot
obtain points in nonconvex regions of the Pareto frontier as shown in figure 12.8.

y1
Figure 12.8. The points in red are
Pareto optimal but cannot be obtained using the weighted sum
method.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12. 3. w e i g ht me thod s

219

A given set of weights forms a linear objective function with parallel contour
lines marching away from the origin. If the feasible set bends away from theorigin, it will have other Pareto optimal points on the boundary that cannot berecovered by minimizing equation (12.4).
Algorithm 12.3. The weighted sum
method for generating a Pareto
frontier, which takes objective functions f1 and f2 and number of
Pareto points npts.

function weight_pareto(f1, f2, npts)
return [optimize(x->w1*f1(x) + (1-w1)*f2(x))
for w1 in range(0,stop=1,length=npts)]
end

12.3.2 Goal Programming
Goal programming3 is a method for converting a multiobjective function to a singleobjective function by minimizing an L p norm between f(x) and a goal point:
minimize f(x) − ygoalx∈Xp(12.5)

Goal programming generally
refers to using p = 1. An overview
is presented in D. Jones and M.
Tamiz, Practical Goal Programming.
Springer, 2010.
3

where the goal point is typically the utopia point. The equation above does not
involve a vector of weights, but the other methods discussed in this chapter can bethought of as generalizations of goal programming. This approach is illustrated
in figure 12.9.

12.3.3Weighted Exponential Sum

The weighted exponential sum combines goal programming and the weighted sum
method4mgoal pf (x) = ∑ wi f i (x) − y i(12.6)
i =1

where w is a vector of positive weights that sum to 1 and p ≥ 1 is an exponent
similar to that used in L p norms. As before, zero-valued weights can result in
weakly Pareto-optimal points.

P. L. Yu, “Cone Convexity, Cone
Extreme Points, and Nondominated Solutions in Decision Problems with Multiobjectives,” Journalof Optimization Theory and Applications, vol. 14, no. 3, pp. 319–377,
1974.
4© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

220 c ha p te r 12. mu l ti obj e c ti ve op ti mi z a ti on

Figure 12.9. Solutions to goal programming as the value for p ischanged.

y2

p=1
p=2
p=∞

ygoal
y1

The weighted exponential sum weighs each component of the distance between
the solution point and the goal point in the criterion space. Increasing p increases
the relative penalty of the largest coordinate deviation between f(x) and the goal
point. While portions of the Pareto-optimal set can be obtained by continuously
varying p, we are not guaranteed to obtain the complete Pareto frontier, and it isgenerally preferable to vary w using a constant p.

12.3.4 Weighted Min-Max Method
Using higher values of p with the weighted exponential sum objective tends
to produce better coverage of the Pareto frontier because the distance contours
are able to enter nonconvex regions of the Pareto frontier. The weighted min-max
method, also called the weighted Tchebycheff method, is the limit as p approaches
infinity:5h 
if (x) = max wiigoal
f i (x) − y i(12.7)The maximization can be removed by including an additionalparameter λ:

5The weighted min-max method can provide the complete Pareto-optimal set
by scanning over the weights but will also produce weakly Pareto-optimal points.
The method can be augmented to produce only the Pareto frontier
h 
i
goal
f (x) = max wi f i (x) − yi
+ ρ f(x)⊤ ygoal(12.8)i© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

minimizex,λλsubject to x ∈ X
w ⊙ f(x) − ygoal − λ1 ≤ 012. 4 . mu l ti obj e c ti ve p op u l a ti on me thod s

221where ρ is a small positive scalar with values typically between 0.0001 and 0.01.
The added term requires that all terms in ygoal be positive, which can be accomplished by shifting the objective function. By definition, f(x) ≥ ygoal for all
x. Any weakly Pareto-optimal point will have f(x)⊤ ygoal larger than a strongly
Pareto-optimal point closer to ygoal .

12.3.5 Exponential Weighted Criterion
The exponential weighted criterion6 was motivated by the inability of the weighted
sum method to obtain points on nonconvex portions of the Pareto frontier. It
constructs a scalar objective function according tomf (x) = ∑ (e pwi − 1)e p f i (x)(12.9)i =1

T. W. Athan and P. Y. Papalambros, “A Note on Weighted Criteria Methods for Compromise Solutions in Multi-Objective Optimization,” Engineering Optimization, vol. 27, no. 2, pp. 155–176,
1996.
6Each objective is individually transformed and reweighted. High values of p canlead to numerical overflow.

12.4

Multiobjective Population Methods

Population methods have also been applied to multiobjective optimization.7 We
can adapt the standard algorithms to encourage populations to spread over the
Pareto frontier.

Population methods are coveredin chapter 9.
712.4.1 Subpopulations
Population methods can divide their attention over several potentially competing
objectives. The population can be partitioned into subpopulations, where each
subpopulation is optimized with respect to different objectives. A traditional
genetic algorithm, for example, can be modified to bias the selection of individuals
for recombination toward the fittest individuals within each subpopulation. Those
selected can form offspring with individuals from different subpopulations.
One of the first adaptations of population methods to multiobjective optimization is the vector evaluated genetic algorithm8 (algorithm 12.4). Figure 12.10 shows
how subpopulations are used in a vector evaluated genetic algorithm to maintain
diversity over multiple objectives. The progression of a vector evaluated genetic
algorithm is shown in figure 12.11.

8
J. D. Schaffer, “Multiple Objective
Optimization with Vector Evaluated Genetic Algorithms,” in International Conference on Genetic Algorithms and Their Applications, 1985.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

222

c ha p te r 12. mu l ti obj e c ti ve op ti mi z a ti on

population

parent pairs

subpopulations

new population

1

Figure 12.10. Using subpopulations in a vector evaluated genetic
algorithm.

2
select m subpopulations..
.

shufflecrossover andmutationmfunction vector_evaluated_genetic_algorithm(f, population,
k_max, S, C, M)
m = length(f(population[1]))m_pop = length(population)m_subpop = m_pop ÷ m
for k in 1 : k_max
ys = f.(population)
parents = select(S, [y[1] for y in ys])[1:m_subpop]
for i in 2 : m
subpop = select(S,[y[i] for y in ys])[1:m_subpop]
append!(parents, subpop)
end

Algorithm 12.4.
The vector
evaluated genetic algorithm,
which takes a vector-valued
objective function f, an initial
population, number of iterations
k_max, a SelectionMethod S,
a CrossoverMethod C, and a
MutationMethod M. The resulting
population is returned.

p = randperm(2m_pop)
p_ind=i->parents[mod(p[i]-1,m_pop)+1][(p[i]-1)÷m_pop + 1]
parents = [[p_ind(i), p_ind(i+1)] for i in 1 : 2 : 2m_pop]
children = [crossover(C,population[p[1]],population[p[2]])for p in parents]
population = [mutate(M, c) for c in children]

end

end
return population© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12. 4 . mu l ti obj e c ti ve p op u l a ti on me thod s

Gen 3

Gen 2

Gen 1

223

Gen 4

2

y2

1.5
1
0.5
0

0

0.5

1
y1

1.5

2

0

0.5

1
y1

1.5

2

0

0.5

1
y1

1.5

12.4.2 Nondomination Ranking
One can compute naive Pareto frontiers using the individuals in a population.
A design point that lies on the approximate Pareto frontier is considered better
than a value deep within the criterion space. We can use nondomination ranking(algorithm 12.5) to rank individuals according to the following levels:9
Level 1. Nondominated individuals in the population.
Level 2. Nondominated individuals except those in Level 1.

2

0

0.5

1
y1

1.5

2

Figure 12.11. A vector evaluated genetic algorithm applied tothe circle function defined in appendix B.8. The Pareto frontier isshown in blue.

9
K. Deb, A. Pratap, S. Agarwal, andT. Meyarivan, “A Fast and Elitist
Multiobjective Genetic Algorithm:
NSGA-II,” IEEE Transactions on Evolutionary Computation, vol. 6, no. 2,
pp. 182–197, 2002.

Level 3. Nondominated individuals except those in Levels 1 or 2.
..
.

Level k. Nondominated individuals except those in Levels 1 to k − 1.

Level 1 is obtained by applying algorithm 12.2 to the population. Subsequentlevels are generated by removing all previous levels from the population andthen applying algorithm 12.2 again. This process is repeated until all individuals
have been ranked. An individual’s objective function value is proportional to its
rank.
The nondomination levels for an example population are shown in figure 12.12.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

224

c ha p te r 12. mu l ti obj e c ti ve op ti mi z a ti on

function get_non_domination_levels(ys)L, m = 0, length(ys)levels = zeros(Int, m)
while minimum(levels) == 0
L += 1for (i,y) in enumerate(ys)if levels[i] == 0 &&!any((levels[i] == 0 || levels[i] == L) &&dominates(ys[i],y) for i in 1 : m)levels[i] = L
end
end
end
return levels
end

Algorithm 12.5. A function for getting the nondomination levels ofan array of multiobjective function
evaluations, ys.

Figure 12.12. The nondominationlevels for a population. Darker levels have lower (better) rankings.

y2

Level 10

Level 1

Level 2y1

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12. 4 . mu l ti obj e c ti ve p op u l a ti on me thod s 225

12.4.3

Pareto Filters

Population methods can be augmented with a Pareto filter, which is a populationthat approximates the Pareto frontier.10 The filter is typically updated with every
generation (algorithm 12.7). Individuals in the population that are not dominatedby any individuals in the filter are added. Any dominated points in the filter are
removed. Individuals from the Pareto filter can be injected into the population,
thereby reducing the chance that portions of the Pareto frontier are lost between
generations.
The filter often has a maximum capacity.11 Filters that are overcapacity can be
reduced by finding the closest pair of design points in the criterion space andremoving one individual from that pair. This pruning method is implemented
in algorithm 12.6. A Pareto filter obtained using a genetic algorithm is shown infigure 12.13.
function discard_closest_pair!(xs, ys)
index, min_dist = 0, Inf
for (i,y) in enumerate(ys)for (j, y′) in enumerate(ys[i+1:end])
dist = norm(y - y′)
if dist < min_dist
index, min_dist = rand([i,j]), dist
end
end
end
deleteat!(xs, index)
deleteat!(ys, index)
return (xs, ys)
end

10
H. Ishibuchi and T. Murata,“A Multi-Objective Genetic Local
Search Algorithm and Its Application to Flowshop Scheduling,”
IEEE Transactions on Systems, Man,
and Cybernetics, vol. 28, no. 3,
pp. 392–403, 1998.

Typically the size of the population.

11

Algorithm 12.6.

The method

discard_closest_pair! is used toremove one individual from a filter that is above capacity. The
method takes the filter’s list of design points xs and associated objective function values ys.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

226 c ha p te r 12. mu l ti obj e c ti ve op ti mi z a ti on

function update_pareto_filter!(filter_xs, filter_ys, xs, ys;
capacity=length(xs),)
for (x,y) in zip(xs, ys)
if !any(dominates(y′,y) for y′ in filter_ys)
push!(filter_xs, x)
push!(filter_ys, y)
end
end
filter_xs, filter_ys = naive_pareto(filter_xs, filter_ys)
while length(filter_xs) > capacity
discard_closest_pair!(filter_xs, filter_ys)
end
return (filter_xs, filter_ys)
end

Algorithm 12.7. A method for
updating a Pareto filter withdesign points filter_xs, corresponding objective function values
filter_ys, a population withdesign points xs and objectivefunction values ys, and filter
capacity capacity which defaults
to the population size.

Figure 12.13. A Pareto filter usedon the genetic algorithm in figure 12.11 to approximate the Pareto
frontier.

1.5

y2

1

0.5

0

0

0.5

1

1.5

y1

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12. 4 . mu l ti obj e c ti ve p op u l a ti on me thod s

227

12.4.4 Niche Techniques
The term niche refers to a focused cluster of points, typically in the criterion space,as shown in figure 12.14. Population methods can converge on a few niches, which
limits their spread over the Pareto frontier. Niche techniques help encourage aneven spread of points.
In fitness sharing,12 shown in figure 12.15, an individual’s objective values are
penalized by a factor equal to the number of other points within a specified
distance in the criterion space.13 This scheme causes all points in a local region toshare the fitness of the other points within the local region. Fitness sharing canbe used together with nondomination ranking and subpopulation evaluation.
Equivalence class sharing can be applied to nondomination ranking. When comparing two individuals, the fitter individual is first determined based on the
nondomination ranking. If they are equal, the better individual is the one withthe fewest number of individuals within a specified distance in the criterion space.
Another niche technique has been proposed for genetic algorithms in which
parents selected for crossover cannot be too close together in the criterion space.
Selecting only nondominated individuals is also recommended.14
1.5

y2

y1
Figure 12.14. Two clear niches fora population in a two-dimensional
criterion space.
Fitness is inversely related to theobjective being minimized.

1213
D. E. Goldberg and J. Richardson,“Genetic Algorithms with Sharing
for Multimodal Function Optimization,” in International Conference on
Genetic Algorithms, 1987.

S. Narayanan and S. Azarm,“On Improving Multiobjective Genetic Algorithms for Design Optimization,” Structural Optimization,
vol. 18, no. 2-3, pp. 146–155, 1999.
14

Figure 12.15. The results of applying fitness sharing to the Pareto filter in figure 12.13, thereby significantly improving its coverage.

y2

1

0.5

0

0

0.5

1

1.5

y1

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

228 c ha p te r 12. mu l ti obj e c ti ve op ti mi z a ti on

12.5 Preference Elicitation
Preference elicitation involves inferring a scalar-valued objective function frompreferences of experts about the tradeoffs between objectives.15 There are manydifferent ways to represent the scalar-valued objective function, but this section
will focus on the weighted sum model where f (x) = w⊤ f(x). Once we identify
a suitable w, we can use this scalar-valued objective function to find an optimal
design.

12.5.1 Model Identification
A common approach for identifying the weight vector w in our preference model
involves asking experts to state their preference between two points a and b in thecriterion space Y (figure 12.16). Each of these points is the result of optimizing
for a point on the Pareto frontier using an associated weight vector wa and wb .
The expert’s response is either a preference for a or a preference for b. There areother schemes for eliciting preference information, such as ranking points in thecriterion space, but this binary preference query has been shown to pose minimal
cognitive burden on the expert.16

This chapter overviews nonBayesian approaches to preference
elicitation. For Bayesian approaches, see: S. Guo and S.
Sanner, “Real-Time Multiattribute
Bayesian Preference Elicitation
with
PairwiseComparison
Queries,” in International Conference on Artificial Intelligenceand Statistics (AISTATS), 2010.
J. R. Lepird, M. P. Owen, and
M. J. Kochenderfer, “Bayesian
Preference Elicitation for Multiobjective Engineering Design
Optimization,” Journal of Aerospace
Information Systems, vol. 12, no. 10,
pp. 634–645, 2015.
15

V. Conitzer, “Eliciting SinglePeaked Preferences Using Comparison Queries,” Journal of Artificial Intelligence Research, vol. 35, pp. 161–
191, 2009.
16

Figure 12.16. Preference elicitation schemes often involve asking
experts their preferences between
two points in the criterion space.

y1

y2

y3

y4

y5

y1

y2

a

y3

y4

y5

b

Suppose the outcomes of the expert queries have resulted in a set of criterion
pairs
n
o

( a(1) , b(1) ) , . . . , ( a( n ) , b( n ) )(12.10)where a(i) is preferable to b(i) in each pair. For each of these preferences, theweight vector must satisfyw⊤ a(i) < w⊤ b(i) =⇒ (a(i) − b(i) )⊤ w < 0(12.11)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12. 5 . p re fe re n c e e l i c i ta ti on 229

In order to be consistent with the data, the weight vector must satisfy(i )(i ) ⊤(a − b ) w < 0 for all i in {1, . . . , n}
1⊤ w = 1(12.12)w≥0

Many different weight vectors could potentially satisfy the above equation.
One approach is to choose a w that best separates w⊤ a(i) from w⊤ b(i)nminimizew

subject to∑ ( a( i ) − b( i ) ) ⊤ w

i =1
(i )

(a

− b(i) )⊤ w < 0 for i ∈ {1, . . . , n}

1⊤ w = 1(12.13)w≥0

It is often desirable to choose the next weight vector such that it minimizes the
distance from the previous weight vector. We can replace the objective function inequation (12.13) with kw − w(n) k1 , thereby ensuring that our new weight vectorw(n+1) is as close as possible to our current one.17

12.5.2Paired Query Selection

We generally want to choose the two points in the criterion space so that theoutcome of the query is as informative as possible. There are many different
approaches for such paired query selection, but we will focus on methods that
attempt to reduce the space of weights consistent with expert responses, preference
information supplied by a domain expert.
We will denote the set of weights consistent with expert responses as W ,which is defined by the linear constraints in equation (12.12). Because weights
are bounded between 0 and 1, the feasible set is an enclosed region forming a
convex polytope with finite volume. We generally want to reduce the volume of
W in as few queries as possible.
Q-Eval18 , shown in figure 12.17, is a greedy elicitation strategy that heuristicallyseeks to reduce the volume of W as quickly as possible with each iteration. It
chooses the query that comes closest to bisecting W into two equal parts. The
method operates on a finite sampling of Pareto-optimal design points. The procedure for choosing a query pair is:

The previous weight vector
may or may not be consistent
with the added constraint(a(n) − b(n) )⊤ w < 0.
17

V. S. Iyengar, J. Lee, and M.
Campbell, “Q-EVAL: Evaluating
Multiple Attribute Items Using
Queries,” in ACM Conference on
Electronic Commerce, 2001.
18© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

230

c ha p te r 12. mu l ti obj e c ti ve op ti mi z a ti on

1. Compute the prime analytic center c of W , which is the point that maximizes
the sum of the logarithms of the distances between itself and the closest pointon each nonredundant constraint in W :nc = arg max ∑ ln (b(i) − a(i) )⊤ w
w∈W

(12.14)i =1

2. Compute the normal distance from the bisecting hyperplane between eachpair of points and the center.
3. Sort the design-point pairs in order of increasing distance.
4. For each of the k hyperplanes closest to c, compute the volume ratio of the two
polytopes formed by splitting W along the hyperplane.
5. Choose the design-point pair with split ratio closest to 1.
Set of Consistent Weights W

w2

Analytic Center and Bisecting Hyperplane

w2

w2

wa c

W

w1

Two Regions under Query a vs. b

wb

w1

The polyhedral method19 works by approximating W with a bounding ellipsoid

centered at the analytic center of W as shown in figure 12.18. Queries are designed
to partition the bounding ellipsoid into approximately equal parts and to favor
cuts that are perpendicular to the longest axis of the ellipsoid to reduce both
uncertainty and to balance the breadth in each dimension.

12.5.3 Design Selection
The previous section discussed query methods that select query pairs for efficiently reducing the search space. After query selection is complete, one muststill select a final design. This process is known as design selection.

w1
Figure 12.17. Visualizations of the
Q-Eval greedy elicitation strategy.
The figure progression shows theinitial set of weights W consistent
with previous preferences, a pairof weight vectors and their corresponding bisecting hyperplane,
and the two polytopes formed by
splitting along the bisecting hyperplane. The algorithm considers all possible pairs from a finite
sampling of Pareto-optimal design
points and chooses the query thatmost evenly splits W .

19
D. Braziunas and C. Boutilier,“Elicitation of Factored Utilities,”
AI Magazine, vol. 29, no. 4, pp. 79–
92, 2009.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.

2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12. 5 . p re fe re n c e e l i c i ta ti on 231

w2
wa
Figure 12.18.
The polyhedral
method uses a bounding ellipsoidfor W .

wb

w1
One such method, decision quality improvement,20 is based on the idea that if
we have to commit to a particular weight, we should commit to the one for which
the worst-case objective value is lowest:x∗ = arg min max w⊤ f(x)
x∈X

w∈W

(12.15)This minimax decision is robust because it provides an upper bound on the objective
value.
The minimax regret21 instead minimizes the maximum amount of regret theuser can have when selecting a particular design:x∗ = arg min max max w⊤ f(x) − w⊤ f(x′ )w∈W x′ ∈X
x∈X
{z
}
|

(12.16)maximum regret

20
D. Braziunas and C. Boutilier,“Minimax Regret-Based Elicitationof Generalized Additive Utilities,”in Conference on Uncertainty in Artificial Intelligence (UAI), 2007.

C. Boutilier, R. Patrascu, P.
Poupart, and D. Schuurmans,“Constraint-Based Optimizationand Utility Elicitation Using the
Minimax Decision Criterion,”
Artificial Intelligence, vol. 170,
no. 8-9, pp. 686–713, 2006.
21

where w⊤ f(x) − w⊤ f(x′ ) is the regret associated with choosing design x insteadof design x′ under the preference weight vector w. Minimax regret can be viewed
as accounting for the decision system’s uncertainty with respect to the designer’strue utility function.
The minimax regret can be used as stopping criteria for preference elicitation
strategies. We can terminate the preference elicitation procedure once the minimax
regret drops below a certain threshold.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

232

c ha p te r 12. mu l ti obj e c ti ve op ti mi z a ti on

12.6 Summary• Design problems with multiple objectives often involve trading performance
between different objectives.
• The Pareto frontier represents the set of potentially optimal solutions.
• Vector-valued objective functions can be converted to scalar-valued objectivefunctions using constraint-based or weight-based methods.
• Population methods can be extended to produce individuals that span the
Pareto frontier.
• Knowing the preferences of experts between pairs of points in the criterion
space can help guide the inference of a scalar-valued objective function.

12.7

Exercises

Exercise 12.1. The weighted sum method is a very simple approach, and it is indeed used by engineers for multiobjective optimization. What is one disadvantageof the procedure when it is used to compute the Pareto frontier?
Exercise 12.2. Why are population methods well-suited for multiobjective optimization?
Exercise 12.3. Suppose you have the points {[1, 2], [2, 1], [2, 2], [1, 1]} in the criterion space and you wish to approximate a Pareto frontier. Which points are Pareto
optimal with respect to the rest of the points? Are any weakly Pareto-optimal
points?
Exercise 12.4. Multiobjective optimization is not easily done with second-ordermethods. Why is this the case?
Exercise 12.5. Consider a square criterion space Y with y1 ∈ [0, 1] and y2 ∈ [0, 1].
Plot the criterion space, indicate the Pareto-optimal points, and indicate the weakly
Pareto optimal points.
Exercise 12.6. Enforcing w ≥ 0 and kwk1 = 1 in the weighted sum method
is not sufficient for Pareto optimality. Give an example where coordinates withzero-valued weights find weakly Pareto-optimal points.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

12. 7. e x e rc i se s 233

Exercise 12.7. Provide an example where goal programming does not produce a
Pareto-optimal point.
Exercise 12.8. Use the constraint method to obtain the Pareto curve for the optimization problem:
minimize [ x2 , ( x − 2)2 ](12.17)x

Exercise 12.9. Suppose we have a multiobjective optimization problem where
the two objectives are as follows:f 1 ( x ) = −( x − 2) sin( x )2f 2 ( x ) = −( x + 3) sin( x )(12.18)(12.19)With x ∈ {−5, −3, −1, 1, 3, 5}, plot the points in the criterion space. How many
points are on the Pareto frontier?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

13 Sampling Plans
For many optimization problems, function evaluations can be quite expensive.
For example, evaluating a hardware design may require a lengthy fabrication
process, an aircraft design may require a wind tunnel test, and new deep learning
hyperparameters may require a week of GPU training. A common approach for
optimizing in contexts where evaluating design points is expensive is to build asurrogate model, which is a model of the optimization problem that can be efficiently
optimized in lieu of the true objective function. Further evaluations of the true
objective function can be used to improve the model. Fitting such models requires
an initial set of points, ideally points that are space-filling; that is, points that cover
the region as well as possible. This chapter covers different sampling plans forcovering the search space when we have limited resources.1

13.1 Full Factorial
The full factorial sampling plan (algorithm 13.1) places a grid of evenly spaced
points over the search space. This approach is easy to implement, does not rely
on randomness, and covers the space, but it uses a large number of points. A gridof evenly spaced points is spread over the search space as shown in figure 13.1.
Optimization over the points in a full factorial sampling plan is referred to as gridsearch.
The sampling grid is defined by a lower-bound vector a and an upper-boundvector b such that ai ≤ xi ≤ bi for each component i. For a grid with misamples in the ith dimension, the nearest points are separated by a distance( bi − a i ) / ( m i − 1 ) .
The full factorial method requires a sample count exponential in the numberof dimensions.2 For n dimensions with m samples per dimension, we have mn

There are other references that
discuss the topics in this chapter
in greater detail. See, for example:
G. E. P. Box, W. G. Hunter, and J. S.
Hunter, Statistics for Experimenters:
An Introduction to Design, Data Analysis, and Model Building, 2nd ed. Wiley, 2005. A. Dean, D. Voss, and
D. Draguljić, Design and Analysis
of Experiments, 2nd ed. Springer,
2017. D. C. Montgomery, Design
and Analysis of Experiments. Wiley,2017.
1

The full factorial method gets its
name not from a factorial sample
count (it is exponential) but fromdesigning with two or more discrete
factors. Here the factors are the mdiscretized levels associated witheach variable.
2236 c ha p te r 13. sa mp l i n g p l a n s

Figure 13.1. Full factorial searchcovers the search space in a grid ofpoints.

b1 − a1
m1 −1

b2

x2

b2 − a2
m2 −1

a2
a1

x1

b1

total samples. This exponential growth is far too high to be of practical use when
there are more than a few variables. Even when full factorial sampling is able tobe used, the grid points are generally forced to be quite coarse and therefore caneasily miss small, local features of the optimization landscape.
function samples_full_factorial(a, b, m)
ranges = [range(a[i], stop=b[i], length=m[i])for i in 1 : length(a)]collect.(collect(Iterators.product(ranges...)))
end

Algorithm 13.1. A function for obtaining all sample locations for thefull factorial grid. Here, a is a vector of variable lower bounds, b isa vector of variable upper bounds,
and m is a vector of sample counts
for each dimension.

13.2 Random SamplingA straightforward alternative to full factorial sampling is random sampling, whichsimply draws m random samples over the design space using a pseudorandom
number generator. To generate a random sample x, we can sample each variableindependently from a distribution. If we have bounds on the variables, such asai ≤ xi ≤ bi , a common approach is to use a uniform distribution over [ ai , bi ],although other distributions may be used. For some variables, it may make sense
to use a log-uniform distribution.3 The samples of design points are uncorrelatedwith each other. The hope is that the randomness, on its own, will result in an
adequate cover of the design space.

Some parameters, such as thelearning rate for deep neural networks, are best searched in logspace.
3© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

13. 3. u n i form p roj e c ti on p l a n s 237

13.3

Uniform Projection Plans

Suppose we have a two-dimensional optimization problem discretized into anm × m sampling grid as with the full factorial method, but, instead of taking all
m2 samples, we want to sample only m positions. We could choose the samples
at random, but not all arrangements are equally useful. We want the samples tobe spread across the space, and we want the samples to be spread across eachindividual component.
no variation in
one component

too clustereduniform projection

A uniform projection plan is a sampling plan over a discrete grid where the distribution over each dimension is uniform. For example, in the rightmost sampling
plan in figure 13.2, each row has exactly one entry and each column has exactly
one entry.
A uniform projection plan with m samples on an m × m grid can be constructedusing an m-element permutation as shown in figure 13.3. There are therefore m!
possible uniform projection plans.44= 4 2 1 3 5

For m = 5 this is already 5! = 120
possible plans. For m = 10 there
are 3,628,800 plans.

4

Figure 13.3. Constructing a uniform projection plan using a permutation.

5p

Figure 13.2. Several ways to choosem samples from a two-dimensional
grid. We generally prefer sampling
plans that cover the space and varyacross each component.

3
2
1
1

2

3

4

5

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

238 c ha p te r 13. sa mp l i n g p l a n s

Sampling with uniform projection plans is sometimes called Latin-hypercube
sampling because of the connection to Latin squares (figure 13.4). A Latin squareis an m × m grid where each row contains each integer 1 through m and eachcolumn contains each integer 1 through m. Latin-hypercubes are a generalization
to any number of dimensions.
Uniform projection plans for n dimensions can be constructed using a permutation for each dimension (algorithm 13.2).
function uniform_projection_plan(m, n)perms = [randperm(m) for i in 1 : n][[perms[i][j] for i in 1 : n] for j in 1 : m]end

13.4 Stratified Sampling

4

1

3

2

1

4

2

3

3

2

1

4

2

3

4

1

Figure 13.4. A 4 × 4 Latin square.
A uniform projection plan can beconstructed by choosing a valuei ∈ {1, 2, 3, 4} and sampling allcells with that value.
Algorithm 13.2. A function for constructing a uniform projection plan
for an n-dimensional hypercube
with m samples per dimension. It
returns a vector of index vectors.

yMany sampling plans, including uniform projection and full factorial plans, arebased on an m × m grid. Such a grid, even if fully sampled, could miss important
information due to systematic regularities as shown in figure 13.5. One method
for providing an opportunity to hit every point is to use stratified sampling.
Stratified sampling modifies any grid-based sampling plan, including full
factorial and uniform projection plans. Cells are sampled at a point chosen uniformly at random from within the cell rather than at the cell’s center as shown infigure 13.6.

xf (x)sampling on gridstratified sampling
Figure 13.5. Using an evenlyspaced grid on a function with systematic regularities can miss important information.

Figure 13.6. Stratified sampling applied to a uniform projection plan.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

13. 5 . sp a c e -fi l l i n g me tri c s 239

13.5 Space-Filling Metrics
A good sampling plan fills the design space since the ability for a surrogate model
to generalize from samples decays with the distance from those samples. Not
all plans, even uniform projection plans, are equally good at covering the search
space. For example, a grid diagonal (figure 13.7) is a uniform projection plan but
only covers a narrow strip. This section discusses different space-filling metrics formeasuring the degree to which a sampling plan X ⊆ X fills the design space.

13.5.1 Discrepancy
The ability of the sampling plan to fill a hyper-rectangular design space can bemeasured by its discrepancy.5 If X has low discrepancy, then a randomly chosen
subset of the design space should contain a fraction of samples proportional to thesubset’s volume.6 The discrepancy associated with X is the maximum difference
between the fraction of samples in a hyper-rectangular subset H and that subset’s
volume:T#( X H)(13.1)− λ(H)d( X ) = supremum#X
H
where #X and #( X H) are the number of points in X and the number of points
in X that lie in H, respectively. The value λ(H) is the n-dimensional volumeof H, the product of the side lengths of H. The term supremum is very similar
to maximization but allows a solution to exist for problems where H merely
approaches a particular rectangular subset, as seen in example 13.1.7
Computing the discrepancy of a sampling plan over the unit hyper-rectangle
is often difficult, and it is not always clear how we can compute the discrepancy
for nonrectangular feasible sets.
T

13.5.2 Pairwise Distances

Figure 13.7. A uniform projection
plan that is not space-filling.
5
L. Kuipers and H. Niederreiter,
Uniform Distribution of Sequences.
Dover, 2012.
6
In arbitrary dimensions, we canuse the Lebesgue measure, which
is a generalization of volume toany subset of n-dimensional Euclidean space. It is length in onedimensional space, area in twodimensional space, and volume inthree-dimensional space.

7
The definition of discrepancy requires hyper-rectangles and typically assumes that X is a finite subset of a unit hypercube. The notionof discrepancy can be extended toallow H to include other sets, suchas convex polytopes.

An alternative method determining which of two m-point sampling plans is more
space-filling is to compare the pairwise distances between all points within eachsampling plan. Sampling plans that are more spread out will tend to have larger
pairwise distances.
The comparison is typically done by sorting each set’s pairwise distances inascending order. The plan with the first pairwise distance that exceeds the otheris considered more space-filling.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

240c ha p te r 13. sa mp l i n g p l a n s

Consider the set:

 
 
 
 
 

1 1
2 1
1 3
9 3
1 1
3 4
X=
, , , ,
, ,
,
,
,
, ,
5 5
5 5
10 5
10 10
50 50
5 5
The discrepancy of X with respect to the unit square is determined by arectangular subset H that either has very small area but contains very many
points or has very large area and contains very few points.

Example 13.1. Computing the discrepancy for a sampling plan overthe unit square. The sizes of therectangles are slightly exaggerated
to clearly show which points they
contain.

x2

1

0

0

x1

1

i
i
h
h
1 2
, 5 , x2 ∈ 51 , 35 , has a volume of 0.12 andThe blue rectangle, x1 ∈ 10
contains 3 points. Its corresponding
discrepancy
measureis thusi 0.38.
i
h
h

9
1
+ ǫ, 10− ǫ , x2 ∈ 15 + ǫ, 54 − ǫ , produces
The purple rectangle, x1 ∈ 10
an even higher discrepancy. As ǫ approaches zero, the volume and the discrepancy approach 0.48 because the rectangle contains no points. Note thatthe limit was required, reflecting the need to use a supremum in the definitionof discrepancy.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

13. 5 . sp a c e -fi l l i n g me tri c s 241

Algorithm 13.3 computes all pairwise distances between points in a sampling
plan. Algorithm 13.4 compares how well two sampling plans fill space using theirrespective pairwise distances.
import LinearAlgebra: norm
function pairwise_distances(X, p=2)m = length(X)[norm(X[i]-X[j], p) for i in 1:(m-1) for j in (i+1):m]
end

Algorithm 13.3. A function for
obtaining the list of pairwise distances between points in sampling
plan X using the L p norm specifiedby p.

function compare_sampling_plans(A, B, p=2)
pA = sort(pairwise_distances(A, p))pB = sort(pairwise_distances(B, p))for (dA, dB) in zip(pA, pB)
if dA < dB
return 1
elseif dA > dB
return -1
end
end
return 0
end

Algorithm 13.4. A function forcomparing the degree to which two
sampling plans A and B are spacefilling using the L p norm specified
by p. The function returns -1 if A is
more space-filling than B. It returns
1 if B is more space-filling than A. It
returns 0 if they are equivalent.

One method for generating a space-filling uniform projection plan is to generate
several candidates at random and then use the one that is most space-filling.
We can search for a space-filling uniform projection plan by repeatedly mutating a uniform projection plan in a way that preserves the uniform projection
property (algorithm 13.5). Simulated annealing, for example, could be used tosearch the space for a sampling plan with good coverage.
function mutate!(X)m, n = length(X), length(X[1])j = rand(1:n)
i = randperm(m)[1:2]X[i[1]][j], X[i[2]][j] = X[i[2]][j], X[i[1]][j]return X
end

Algorithm 13.5. A function for mutating uniform projection plan X,
while maintaining its uniform projection property.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

242

c ha p te r 13. sa mp l i n g p l a n s

13.5.3 Morris-Mitchell Criterion
The comparison scheme in section 13.5.2 typically results in a challenging optimization problem with many local minima. An alternative is to optimize withrespect to the Morris-Mitchell criterion (algorithm 13.6):8
Φq ( X ) =

∑
i

−q
di

!1/q

(13.2)

where di is the ith pairwise distance between points in X and q > 0 is a tunable
parameter.9 Morris and Mitchell recommend optimizing:
minimizeXmaximizeq∈{1,2,5,10,20,50,100}

Φq ( X )(13.3)

function phiq(X, q=1, p=2)
dists = pairwise_distances(X, p)
return sum(dists.^(-q))^(1/q)
end

8
M. D. Morris and T. J. Mitchell,“Exploratory Designs for Computational Experiments,” Journal of
Statistical Planning and Inference,
vol. 43, no. 3, pp. 381–402, 1995.

Larger values of q will give higher
penalties to large distances.

9

Algorithm 13.6. An implementation of the Morris-Mitchell criterion which takes a list of design points X, the criterion parameter q > 0, and a norm parameterp ≥ 1.

Figure 13.8 shows the Morris-Mitchell criterion evaluated for several randomly
generated uniform projection plans.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

13. 5 . sp a c e -fi l l i n g me tri c s 243

Φ1 = 23.7

Φ1 = 24.2

Φ1 = 24.2

Φ1 = 24.3

Φ1 = 24.3

Φ1 = 24.3

Φ1 = 24.3

Φ1 = 24.5

Φ1 = 24.8

Φ1 = 24.9

Φ1 = 25.0

Φ1 = 25.0

Φ1 = 25.3

Φ1 = 25.5

Φ1 = 26.2

Φ1 = 27.2

Φ1 = 33.0

Φ1 = 36.7

Figure 13.8. Uniform projectionplans sorted from best to worst according to Φ1 .

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

244

c ha p te r 13. sa mp l i n g p l a n s

13.6 Space-Filling Subsets
In some cases, we have a set of points X and want to find a subset of points
S ⊂ X that still maximally fills X. The need for identifying space-filling subsets ofX arises in the context of multifidelity models.10 For example, suppose we used asampling plan X to identify a variety of aircraft wing designs to evaluate using
computational fluid dynamic models in simulation. We can choose only a subsetof these design points S to build and test in a wind tunnel. We still want S to be
space filling.
The degree to which S fills the design space can be quantified using the maximum distance between a point in X and the closest point in S. This metric
generalizes to any two finite sets A and B (algorithm 13.7). We can use any L p
norm, but we typically use L2 , the Euclidean distance:
dmax ( X, S) = maximize minimizeks − xk p
x∈ X

s∈ S

(13.4)

min_dist(a, B, p) = minimum(norm(a-b, p) for b in B)
d_max(A, B, p=2) = maximum(min_dist(a, B, p) for a in A)

A space-filling sampling plan is one that minimizes this metric.11 Finding aspace-filling sampling plan with m elements is an optimization problem
minimizedmax ( X, S)

subject to

S⊆X

S

10
A. I. J. Forrester, A. Sóbester, andA. J. Keane, “Multi-Fidelity Optimization via Surrogate Modelling,” Proceedings of the Royal Society of London A: Mathematical,
Physical and Engineering Sciences,vol. 463, no. 2088, pp. 3251–3269,
2007.

Algorithm 13.7. The set L p distancemetrics between two discrete sets,
where A and B are lists of design
points and p is the L p norm parameter.
We can also minimize the MorrisMitchell criterion for S.

11(13.5)#S = m
Optimizing equation (13.5) is typically computationally intractable. A brute
force approach would try all d!/m!(d − m)! size-m subsets for a dataset of d design
points. Both greedy local search (algorithm 13.8) and the exchange algorithm (algorithm 13.9) are heuristic strategies for overcoming this difficulty. They typicallyfind acceptable space-filling subsets of X.
Greedy local search starts with a point selected randomly from X and incrementally adds the next best point that minimizes the distance metric. Points areadded until the desired number of points is reached. Because the points are initialized randomly, the best results are obtained by running greedy local searchseveral times and keeping the best sampling plan (algorithm 13.10).
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

13. 7. q u a si -ra n d om se q u e n c e s 245

Algorithm 13.8. Greedy local
search, for finding m-element sampling plans that minimize a distance metric d for discrete set X.

function greedy_local_search(X, m, d=d_max)
S = [sample(X)]for i in 2 : m
j = argmin([x ∈ S ? Inf : d(X, push!(copy(S), x))
for x in X])
push!(S, X[j])endreturn S
end

The exchange algorithm initializes S to a random subset of X and repeatedly
replaces points that are in S with a different point in X that is not already in S
to improve on the distance metric. The exchange algorithm is also typically run
multiple times.
Figure 13.9 compares space-filling subsets obtained using greedy local searchand the exchange algorithm.

13.7

Quasi-Random Sequences

Quasi-random sequences,12 also called low-discrepancy sequences, are often used in
the context of trying to approximate an integral over a multidimensional space:ZX

f (x) dx ≈v mf (x( i ) )
m i∑
=1

12
C. Lemieux, Monte Carlo andQuasi-Monte
Carlo
Sampling.
Springer, 2009.

(13.6)

where each x(i) is sampled uniformly at random over the domain X and v is thevolume of X . This approximation is known as Monte Carlo integration.
Rather than relying on random or pseudorandom numbers to generate integration points, quasi-random sequences are deterministic sequences that fill the
space in a systematic manner so that the integral converges as fast as possible in
the number of points m.13 These quasi-Monte Carlo methods have an error conver√gence of O(1/m) as opposed to O(1/ m) for typical Monte Carlo integration, as
shown in figure 13.10.

13
Pseudorandom number sequences, such as those produced
by a sequence of calls to rand, aredeterministic given a particularseed, but they appear random.
Quasi-random numbers are also
deterministic but do not appear
random.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

246

c ha p te r 13. sa mp l i n g p l a n sfunction exchange_algorithm(X, m, d=d_max)
S = sample(X, m, replace=false)δ, done = d(X, S), false
while !done
best_pair = (0,0)
for i in 1 : m
s = S[i]for (j,x) in enumerate(X)
if !in(x, S)
S[i] = xδ′ = d(X, S)
if δ′ < δ
δ = δ′
best_pair = (i,j)
end
end
end
S[i] = senddone = best_pair == (0,0)
if !donei,j = best_pair
S[i] = X[j]endendreturn S
endfunction multistart_local_search(X, m, alg, k_max, d=d_max)
sets = [alg(X, m, d) for i in 1 : k_max]
return sets[argmin([d(X, S) for S in sets])]
end

Algorithm 13.9. The exchange algorithm for finding m-element sampling plans that minimize a distance metric d for discrete set X.

Algorithm 13.10. Multistart local
search runs a particular searchalgorithm multiple times and returns the best result. Here, X isthe list of points, m is the sizeof the desired sampling plan,alg is either exchange_algorithm
or greedy_local_search, k_max isthe number of iterations to run,and d is the distance metric.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

13. 7. q u a si -ra n d om se q u e n c e s 247

exchange algorithm

x1

x1

Figure 13.9. Space-filling subsets obtained with both greedy local search and the exchange algorithm.

Quasi-random sequences are typically constructed for the unit n-dimensional
hypercube, [0, 1]n . Any multidimensional function with bounds on each variable
can be transformed into such a hypercube. This transformation is implemented
in algorithm 7.9.
Various methods exist for generating quasi-random sequences. Several suchmethods are compared to random sampling in figure 13.12.

100
10−2
10−4
100
102
104
number of samples
rand

13.7.1 Additive Recurrence
Simple recurrence relations of the form:x ( k +1) = x ( k ) + c

relative error

x2

greedy local search(mod 1)(13.7)

produce space-filling sets provided that c is irrational. The value of c leading to
the smallest discrepancy is√
5−1
c = 1− ϕ =≈ 0.618034(13.8)
2
where ϕ is the golden ratio.14
We can construct a space-filling set over n dimensions using an additive recurrence sequence for each coordinate, each with its own value of c. The square rootsof the primes are known to be irrational, and can thus be used to obtain different
sequences for each coordinate:
√
√
√
√
√
c1 = 2, c2 = 3, c3 = 5, c4 = 7, c5 = 11, . . .
(13.9)

Sobol

Figure 13.10. The error from
R1
estimating 0 sin(10x ) dx using
Monte Carlo integration with random numbers from U (0, 1) anda Sobol sequence. The Sobol sequence, covered in section 13.7.3,
converges faster.

C. Schretter, L. Kobbelt, and P.-O.
Dehaye, “Golden Ratio Sequencesfor Low-Discrepancy Sampling,”
Journal of Graphics Tools, vol. 16,
no. 2, pp. 95–104, 2016.

14© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

248 c ha p te r 13. sa mp l i n g p l a n s

Methods for additive recurrence are implemented in algorithm 13.11.
using Primes
function get_filling_set_additive_recurrence(m; c=φ-1)
X = [rand()]for i in 2 : m
push!(X, mod(X[end] + c, 1))
end
return X
endfunction get_filling_set_additive_recurrence(m, n)
ps = primes(max(ceil(Int, n*(log(n) + log(log(n)))), 13))seqs = [get_filling_set_additive_recurrence(m, c=sqrt(p))
for p in ps[1:n]]
return [collect(x) for x in zip(seqs...)]
end

Algorithm 13.11.
Additiverecurrenceforconstructing
m-element filling sequences overn-dimensional unit hypercubes.
The Primes package is used togenerate the first n prime numbers,
where the kth prime number isbounded byk (log k + log log k)
for k > 6, and primes(a) returns
all primes up to a. Note that 13 is
the sixth prime number.

13.7.2 Halton Sequence
The Halton sequence is a multidimensional quasi-random space-filling set.15 The
single-dimensional version, called van der Corput sequences, generates sequenceswhere the unit interval is divided into powers of base b. For example, b = 2
produces:
1 1 3 1 5 3 7 1
X=, , , , , , ,,...
(13.10)2 4 4 8 8 8 8 16J. H. Halton, “Algorithm 247:
Radical-Inverse Quasi-Random
Point Sequence,” Communicationsof the ACM, vol. 7, no. 12, pp. 701–
702, 1964.

15

whereas b = 5 produces:X=

1 2 3 4 1 6 11
, , , ,,,,...
5 5 5 5 25 25 25

(13.11)

Multi-dimensional space-filling sequences use one van der Corput sequence foreach dimension, each with its own base b. The bases, however, must be coprime16
in order to be uncorrelated. Methods for constructing Halton sequences are
implemented in algorithm 13.12.

Two integers are coprime if the
only positive integer that divides
them both is 1.

16© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

13. 7. q u a si -ra n d om se q u e n c e s 249

using Primes
function halton(i, b)
result, f = 0.0, 1.0
while i > 0
f = f / b;
result = result + f * mod(i, b)
i = floor(Int, i / b)
end
return result
end
get_filling_set_halton(m; b=2) = [halton(i,b) for i in 1: m]function get_filling_set_halton(m, n)
bs = primes(max(ceil(Int, n*(log(n) + log(log(n)))), 6))
seqs = [get_filling_set_halton(m, b=b) for b in bs[1:n]]
return [collect(x) for x in zip(seqs...)]
end

For large primes, we can get correlation in the first few numbers. Such a correlation is shown in figure 13.11. Correlation can be avoided by the leaped Haltonmethod,17 which takes every pth point, where p is a prime different from allcoordinate bases.

13.7.3 Sobol Sequences
Sobol sequences are quasi-random space-filling sequences for n-dimensional hypercubes.18 They are generated by xor-ing the previous Sobol number with a setof direction numbers:19(k)( i −1)(i )(13.12)⊻ vj
Xj = Xj
where v j is the jth bit of the kth direction number. Tables of good direction
numbers have been provided by various authors.20
A comparison of these and previous approaches is shown in figure 13.12. For
high values several methods exhibit a clear underlying structure.
(k)

Algorithm
13.12.
Haltonquasi-random m-element filling
sequences over n-dimensional unit
hypercubes, where b is the base.
The bases bs must be coprime.

Figure 13.11. The Halton sequencewith b = [19, 23] for which thefirst 18 samples are perfectly linearly correlated.
17
L. Kocis and W. J. Whiten,“Computational
Investigationsof Low-Discrepancy Sequences,”
ACM Transactions on Mathematical
Software, vol. 23, no. 2, pp. 266–294,
1997.

I. M. Sobol, “On the Distributionof Points in a Cube and the Approximate Evaluation of Integrals,”
USSR Computational Mathematics
and Mathematical Physics, vol. 7,
no. 4, pp. 86–112, 1967.
18

The symbol ⊻ denotes the xor operation, which returns true if andonly if both inputs are different.

19

The Sobol.jl package provides
an implementation for up to 1,111
dimensions.
20© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c ha p te r 13. sa mp l i n g p l a n s

m = 1000

m = 100

m = 10250

RandomAdditive Recurrence

HaltonUniform Projection

Sobol

Figure 13.12. A comparison of
space-filling sampling plans in two
dimensions. Samples are coloredaccording to the order in which
they are sampled. The uniform
projection plan was generated randomly and is not optimized.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

13. 8 . su mma ry 251

13.8 Summary• Sampling plans are used to cover search spaces with a limited number ofpoints.
• Full factorial sampling, which involves sampling at the vertices of a uniformly
discretized grid, requires a number of points exponential in the number ofdimensions.
• Uniform projection plans, which project uniformly over each dimension, canbe efficiently generated and can be optimized to be space filling.
• Greedy local search and the exchange algorithm can be used to find a subset
of points that maximally fill a space.

13.9 Exercises
Exercise 13.1. Filling a multidimensional space requires exponentially morepoints as the number of dimensions increases. To help build this intuition, determine the side lengths of an n-dimensional hypercube such that it fills half of thevolume of the n-dimensional unit hypercube.
Exercise 13.2. Suppose that you sample randomly inside a unit sphere in ndimensions. Compute the probability that a randomly sampled point is within ǫ
distance from the surface of the sphere as n → ∞. Hint: The volume of a sphere
is C (n)r n , where r is the radius and C (n) is a function of the dimension n only.

hypercube side length ℓ• Quasi-random sequences are deterministic procedures by which space-fillingsampling plans can be generated.

1
0.8
0.6
5
10
number of dimensions, nExercise 13.3. Suppose we have a sampling plan X = {x(1) , . . . , x(10) }, wherex(i) = [cos(2πi/10), sin(2πi/10)](13.13)

Compute the Morris-Mitchell criterion for X using an L2 norm when the parameter q is set to 2. In other words, evaluate Φ2 ( X ). If we add [2, 3] to each x(i) , will
Φ2 ( X ) change? Why or why not?
Exercise 13.4. Additive recurrence requires that the multiplicative factor c inequation (13.7) be irrational. Why can c not be rational?
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

14

Surrogate Models

The previous chapter discussed methods for producing a sampling plan. This
chapter shows how to use these samples to construct models of the objectivefunction that can be used in place of the real objective function. Such surrogate
models are designed to be smooth and inexpensive to evaluate so that they can be
efficiently optimized. The surrogate model can then be used to help direct the
search for the optimum of the real objective function.

Fitting Surrogate Models

y

14.1

A surrogate model fˆ parameterized by θ is designed to mimic the true objectivefunction f . The parameters θ can be adjusted to fit the model based on samples
collected from f . An example surrogate model is shown in figure 14.1.
Suppose we have m design pointsnoX = x(1) , x(2) , . . . , x( m )(14.1)
and associated function evaluationsnoy = y (1) , y (2) , . . . , y ( m )For a particular set of parameters, the model will predictonŷ = fˆθ (x(1) ), fˆθ (x(2) ), . . . , fˆθ (x(m) )(14.2)

θky − ŷk p

design points
surrogate model
true objective function
Figure 14.1. Surrogate models approximate the true objective function. The model is fitted to the evaluated design points but deviates
farther away from them.

(14.3)

Fitting a model to a set of points requires tuning the parameters to minimize
the difference between the true evaluations and those predicted by the model,
typically according to an L p norm:1
minimizex(14.4)

It is common to use the L2 norm.
Minimizing this equation with an
L2 norm is equivalent to minimizing the mean squared error at thosedata points.

1

254

c ha p te r 14 . su rrog a te mod e l s

Equation (14.4) penalizes the deviation of the model only at the data points.
There is no guarantee that the model will continue to fit well away from observed
data, and model accuracy typically decreases the farther we go from the sampled
points.
This form of model fitting is called regression. A large body of work exists forsolving regression problems, and it is extensively studied in machine learning.2
The rest of this chapter covers several popular surrogate models and algorithms
for fitting surrogate models to data, and concludes with methods for choosing
between types of models.

2
K. P. Murphy, Machine Learning: A
Probabilistic Perspective. MIT Press,
2012.

14.2 Linear Models
A simple surrogate model is the linear model, which has the form3fˆ = w0 + w⊤ x

θ = { w0 , w }This equation may seem familiar.
It is the equation for a hyperplane.

3(14.5)For an n-dimensional design space, the linear model has n + 1 parameters, andthus requires at least n + 1 samples to fit unambiguously.
Instead of having both w and w0 as parameters, it is common to construct asingle vector of parameters θ = [w0 , w] and prepend 1 to the vector x to getfˆ = θ⊤ x(14.6)

Finding an optimal θ requires solving a linear regression problem:
minimizeθky − ŷk22(14.7)

ky − Xθk22(14.8)

which is equivalent to solving
minimizeθ

where X is a design matrix formed from m data points( x(1) ) ⊤ (2) ⊤ 
 (x ) 

X=
..




.

(14.9)( x( m ) ) ⊤© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

14 . 3. ba si s fu n c ti on s 255

Algorithm 14.1. A method for constructing a design matrix from a listof design points X and a method
for fitting a surrogate model using
linear regression to a list of design
points X and a vector of objectivefunction values y.

function design_matrix(X)n, m = length(X[1]), length(X)return [j==0 ? 1.0 : X[i][j] for i in 1:m, j in 0:n]end
function linear_regression(X, y)θ = pinv(design_matrix(X))*yreturn x -> θ⋅[1; x]
end

Algorithm 14.1 implements methods for computing a design matrix and forsolving a linear regression problem. Several cases for linear regression are shownin figure 14.2.
Linear regression has an analytic solution(14.10)θ = X+ ywhere X+ is the Moore-Penrose pseudoinverse of X.
If X⊤ X is invertible, the pseudoinverse can be computed as

 −1
X⊤
X+ = X⊤ X(14.11) −1
X+ = X⊤ XX⊤(14.12)

If XX⊤ is invertible, the pseudoinverse can be computed as

The function pinv computes the pseudoinverse of a given matrix.4

14.3

The function pinv uses the singular value decomposition, X =UΣV∗ , to compute the pseudoinverse:
4Basis Functions

X+ = VΣ+ U∗

The linear model is a linear combination of the components of x:nfˆ(x) = θ1 x1 + · · · + θn xn = ∑ θi xi = θ⊤ x

(14.13)i =1

which is a specific example of a more general linear combination of basis functionsqfˆ(x) = θ1 b1 (x) + · · · + θq bq (x) = ∑ θi bi (x) = θ⊤ b(x)where the pseudoinverse of the diagonal matrix Σ is obtained by taking the reciprocal of each nonzero
element of the diagonal and thentransposing the result.

(14.14)i =1© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

256 c ha p te r 14 . su rrog a te mod e l s

m = n+1yym < n+1x

nonindependent points

m > n+1yyxFigure 14.2. Models resulting fromlinear regression, which minimizes
the square vertical distance of themodel from each point. The pseudoinverse produces a unique solution for any nonempty point configuration.
The bottom-left subfigure
shows the model obtained for
two repeated points, in this case,m = n + 1. Because the two entries
are repeated, the matrix X is
singular. Although X does nothave an inverse in this case, thepseudoinverse produces a uniquesolution that passes between thetwo points.

xx© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

14 . 3. ba si s fu n c ti on s

257

In the case of linear regression, the basis functions simply extract each component,bi ( x ) = x i .
Any surrogate model represented as a linear combination of basis functions
can be fit using regression:
minimizeθky − Bθk22

where B is the basis matrix formed from m data points:b ( x(1) ) ⊤ b ( x(2) ) ⊤ 


B=
..



.
(
m
)
⊤
b(x )

(14.15)(14.16)

The weighting parameters can be obtained using the pseudoinverse
θ = B+ y(14.17)

Algorithm 14.2 implements this more general regression procedure.
using LinearAlgebra
function regression(X, y, bases)
B = [b(x) for x in X, b in bases]θ = pinv(B)*yreturn x -> sum(θ[i] * bases[i](x) for i in 1 : length(θ))
end

Algorithm 14.2. A method for fitting a surrogate model to a list ofdesign points X and corresponding
objective function values y using regression with basis functions contained in the bases array.

Linear models cannot capture nonlinear relations. There are a variety of otherfamilies of basis functions that can represent more expressive surrogate models.
The remainder of this section discusses a few common families.

14.3.1 Polynomial Basis Functions
Polynomial basis functions consist of a product of design vector components, eachraised to a power. Linear basis functions are a special case of polynomial basis
functions.
From the Taylor series expansion5 we know that any infinitely differentiable
function can be closely approximated by a polynomial of sufficient degree. We
can construct these bases using algorithm 14.3.

5

Covered in appendix C.2.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

258 c ha p te r 14 . su rrog a te mod e l s

In one dimension, a polynomial model of degree k has the formkfˆ( x ) = θ0 + θ1 x + θ2 x2 + θ3 x3 + · · · + θk x k = ∑ θi xi(14.18)i =0Hence, we have a set of basis functions bi ( x ) = xi for i ranging from 0 to k.
In two dimensions, a polynomial model of degree k has basis functions of theformjbij (x) = x1i x2 for i, j ∈ {0, . . . , k}, i + j ≤ k(14.19)Fitting a polynomial surrogate model is a regression problem, so a polynomial
model is linear in higher dimensional space (figure 14.3). Any linear combinationof basis functions can be viewed as linear regression in a higher dimensional
space.

yyf (x)fˆ( x )xx

Figure 14.3. A polynomial model
is linear in higher dimensions. The
function exists in the plane formed
from its bases, but it does not occupy the entire plane because theterms are not independent.

x2

polynomial_bases_1d(i, k) = [x->x[i]^p for p in 0:k]function polynomial_bases(n, k)bases = [polynomial_bases_1d(i, k) for i in 1 : n]terms = Function[]for ks in Iterators.product([0:k for i in 1:n]...)
if sum(ks) ≤ k
push!(terms,
x->prod(b[j+1](x) for (j,b) in zip(ks,bases)))end
end
return terms
end

Algorithm 14.3. A method for constructing an array of polynomial
basis functions up to a degree k
for the ith component of a design
point, and a method for constructing a list of n-dimensional polynomial bases for terms up to degreek.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

14 . 3. ba si s fu n c ti on s 259

14.3.2 Sinusoidal Basis Functions
Any continuous function over a finite domain can be represented using an infinite set of sinusoidal basis functions.6 A Fourier series can be constructed for any
integrable univariate function f on an interval [ a, b]
f (x) =





∞
2πix
2πix
θ0
(sin)
(cos)
sin
cos
+ θi
+ ∑ θi
2
b−a
b−a
i =1

The Fourier series is also exact forfunctions defined over the entire
real line if the function is periodic.

6(14.20)

where
Z

b
2
f ( x ) dx
b−a a


Z b
2
2πix
(sin)
dx
f ( x ) sin
=
θi
b−a a
b−a


Z b
2
2πix
(cos)
θi
=
f ( x ) cos
dx
b−a a
b−a

θ0

=

(14.21)(14.22)(14.23)

Just as the first few terms of a Taylor series are used in polynomial models, so
too are the first few terms of the Fourier series used in sinusoidal models. The
bases for a single component over the domain x ∈ [ a, b] are:



b (x)
= 1/2



 0
(sin)
bi ( x ) = sin 2πix
(14.24)
b
−
a




(
cos
)

2πix
b
( x ) = cos b−ai

We can combine the terms for multidimensional sinusoidal models in the same
way we combine terms in polynomial models. Algorithm 14.4 can be used toconstruct sinusoidal basis functions. Several cases for sinusoidal regression are
shown in figure 14.4.

14.3.3 Radial Basis Functions
A radial function ψ is one which depends only on the distance of a point fromsome center point c, such that it can be written ψ(x, c) = ψ(kx − ck) = ψ(r ).
Radial functions are convenient basis functions because placing a radial function
contributes a hill or valley to the function landscape. Some common radial basis
functions are shown in figure 14.5.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

260

c ha p te r 14 . su rrog a te mod e l sfunction sinusoidal_bases_1d(j, k, a, b)
T = b[j] - a[j]bases = Function[x->1/2]
for i in 1 : k
push!(bases, x->sin(2π*i*x[j]/T))
push!(bases, x->cos(2π*i*x[j]/T))
end
return bases
endfunction sinusoidal_bases(k, a, b)
n = length(a)bases = [sinusoidal_bases_1d(i, k, a, b) for i in 1 : n]terms = Function[]for ks in Iterators.product([0:2k for i in 1:n]...)
powers = [div(k+1,2) for k in ks]
if sum(powers) ≤ k
push!(terms,
x->prod(b[j+1](x) for (j,b) in zip(ks,bases)))end
end
return terms
end

Algorithm 14.4.

The method

sinusoidal_bases_1d produces alist of basis functions up to degree
k for the ith component of the design vector given lower bound a
and upper bound b. The method
sinusoidal_bases produces allbase function combinations up todegree k for lower-bound vector aand upper-bound vector b.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

14 . 3. ba si s fu n c ti on s

m = n+1

Figure 14.4. Fitting sinusoidal
models to noisy points.

yym < n+1

261

x

nonindependent points

m > n+1

y

y

x

x

x

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

262 c ha p te r 14 . su rrog a te mod e l sthin plate spline: r2 log r

cubic: r3

linear: r

3

8

2

ψ

6
1

2

4

1

2
0

0

0

Gaussian: e−r /2σ22

multiquadratic: (r2 + σ2 ) 2

1

inverse multiquadratic: (r2 + σ2 )− 2
1

1

1
2
ψ

0.8
0.5

1.5
0.6
1

0
0

1
r

2

0

1
r

2

0.4

0

1
r

2

Figure 14.5. Several radial basis
functions.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

14 . 4 . fi tti n g n oi sy obj e c ti ve fu n c ti on s 263

Radial basis functions require specifying the center points. One approach when
fitting radial basis functions to a set of data points is to use the data points as thecenters. For a set of m points, one thus constructs m radial basis functionsbi (x) = ψ(kx − x(i) k)

for i ∈ {1, . . . , m}

(14.25)The corresponding m × m basis matrix is always semidefinite. Algorithm 14.5 canbe used to construct radial basis functions with known center points. Surrogate
models with different radial basis functions are shown in figure 14.6.
radial_bases(ψ, C, p=2) = [x->ψ(norm(x - c, p)) for c in C]

0.5

x sin(5x )ψ = exp(−2r2 )ψ = exp(−5r2 )y0

Algorithm 14.5. A method for obtaining a list of basis functions
given a radial basis function ψ, alist of centers C, and an L p norm
parameter p.
Figure 14.6. Several different Gaussian radial basis functions used tofit x sin(5x ) based on 4 noise-freesamples.

ψ = exp(−10r2 )

−0.5

−1

−1.5

0

0.2

0.6

0.4

0.8

1

x

14.4

Fitting Noisy Objective Functions

Models fit using regression will pass as close as possible to every design point.
When the objective function evaluations are noisy, complex models are likely to
excessively contort themselves to pass through every point. However, smootherfits are often better predictors of the true underlying objective function.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

264

c ha p te r 14 . su rrog a te mod e l s

The basis regression problem specified in equation (14.15) can be augmented
to prefer smoother solutions. A regularization term is added in addition to theprediction error in order to give preference to solutions with lower weights. The
resulting basis regression problem with L2 regularization7 is:minimizeθky − Bθk22 + λkθk22(14.26)

where λ ≥ 0 is a smoothing parameter, with λ = 0 resulting in no smoothing.
The optimal parameter vector is given by:8

 −1
B⊤ y
θ = B⊤ B + λI

(14.27)

where I is the identity matrix.
Algorithm 14.6 implements regression with L2 regularization. Surrogate models with different radial basis functions fit to noisy samples are shown in figure 14.7.
function regression(X, y, bases, λ)B = [b(x) for x in X, b in bases]θ = (B'B + λ*I)\B'y
return x -> sum(θ[i] * bases[i](x) for i in 1 : length(θ))
end

Other L p -norms, covered in appendix C.4, can be used as well.
Using the L1 norm will encouragesparse solutions with less influential component weights set to zero,
which can be useful in identifying
important basis functions.

8
The matrix B⊤ B + λI is not always invertible if λ = 0. However,
we can always produce an invertible matrix with a positive λ.
7

Algorithm 14.6. A method for regression in the presence of noise,
where λ is a smoothing term. It returns a surrogate model fitted toa list of design points X and corresponding objective function values
y using regression with basis functions bases.

14.5 Model Selection
So far, we have discussed how to fit a particular model to data. This section
explains how to select which model to use. We generally want to minimize generalization error, which is a measure of the error of the model on the full design
space, including points that may not be included in the data used to train themodel. One way to measure generalization error is to use the expected squared
error of its predictions:2 ǫgen = Ex∼X f (x) − fˆ(x)(14.28)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

14 . 5 . mod e l se l e c ti on

2yFigure 14.7. Several different Gaussian radial basis functions used tofit x sin(5x ) with zero mean, standard deviation 0.1 error based on
10 noisy samples and radial basis
function ψ = exp(−5r2 ).

x sin(5x )λ=0
λ = 0.1
λ = 0.5

1

265

0

−1

−2

0

0.2

0.6

0.4

0.8

1

x

Of course, we cannot calculate this generalization error exactly because it requires
knowing the function we are trying to approximate. It may be tempting to estimate
the generalization error of a model from the training error. One way to measuretraining error is to use the mean squared error (MSE) of the model evaluated on
the m samples used for training:ǫtrain =2
1 m
f (x(i) ) − fˆ(x(i) )∑m i =1(14.29)However, performing well on the training data does not necessarily correspondto low generalization error. Complex models may reduce the error on the training
set, but they may not provide good predictions in other points in the design space
as illustrated in example 14.1.9
This section discusses several methods for estimating generalization error.
These methods train and test on subsets of the data. We introduce the TrainTest
type (algorithm 14.7), which contains a list of training indices and a list of test
indices. The method fit takes in a training set and produces a model. The method
metric takes a model and a test set and produces a metric, such as the meansquared error. The method train_and_validate (algorithm 14.7) is a utility
function for training and then evaluating a model. Although we train on subsetsof the data when estimating the generalization error, once we have decided whichmodel to use, we can train on the full dataset.

A major theme in machine learning is balancing model complexity to avoid overfitting the training
data. K. P. Murphy, Machine Learning: A Probabilistic Perspective. MIT
Press, 2012.
9© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

266 c ha p te r 14 . su rrog a te mod e l s

Consider fitting polynomials of varying degrees to evaluations of the objective functionf ( x ) = x/10 + sin( x )/4 + exp − x2

Example 14.1. A comparison oftraining and generalization error
as the degree of a polynomial surrogate model is varied.

Below we plot polynomial surrogate models of varying degrees using thesame nine evaluations evenly spaced over [−4, 4]. The training and generalization error are shown as well, where generalization is calculated over[−5, 5].

y

1

k=0

k=1

k=2

0
ǫtrain = 0.199
ǫgen = 0.187

ǫtrain = 0.089
ǫgen = 0.1

ǫtrain = 0.132ǫgen = 0.127

−1

y

1

k=3

k=5

k=4

0
ǫtrain = 0.062ǫgen = 0.09

ǫtrain = 0.029
ǫgen = 0.129ǫtrain = 0.027
ǫgen = 0.112

−1

y

1

k=6

k=8

k=7

0
ǫtrain = 0.008
ǫgen = 1.162

−1

−4 −2

0
x

2

4

ǫtrain = 0.008
ǫgen = 1.157

−4 −2

0
x

2

4

ǫtrain = 0.0ǫgen = 87.633

−4 −2

0
x

2

4

The plot shows that the generalization error is high for both very low andhigh values of k, and that training error decreases as we increase the polynomial degree. The high-degree polynomials are particularly poor predictors
for designs outside [−4, 4].

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

14 . 5 . mod e l se l e c ti on 267

struct TrainTest
train
test
end
function train_and_validate(X, y, tt, fit, metric)model = fit(X[tt.train], y[tt.train])return metric(model, X[tt.test], y[tt.test])
end

14.5.1 Holdout
test

train

Figure 14.8. The holdout method(left) partitions the data into train
and test sets.

generalization error estimate

test( fˆ, •)A simple approach to estimating the generalization error is the holdout method,which partitions the available data into a test set Dh with h samples and a training
set Dt consisting of all remaining m − h samples as shown in figure 14.8. The
training set is used to fit model parameters. The held out test set is not usedduring model fitting, and can thus be used to estimate the generalization error.
Different split ratios are used, typically ranging from 50% train, 50% test to 90%
train, 10% test, depending on the size and nature of the dataset. Using too few
samples for training can result in poor fits (figure 14.9), whereas using too many
will result in poor generalization estimates.
The holdout error for a model fˆ fit to the training set is2
1(14.30)ǫholdout =y − fˆ(x)∑h (x,y)∈D

ytrain(•)

Algorithm 14.7. A utility type andmethod for training a model andthen validating it on a metric. Here,train and test are lists of indicesinto the training data, X is a list ofdesign points, y is the vector of corresponding function evaluations,tt is a train-test partition, fit is amodel fitting function, and metricevaluates a model on a test set to
produce an estimate of generalization error.

xffˆ
training samples
holdout samples
Figure 14.9. Poor train-test splits
can result in poor model performance.

hEven if the partition ratio is fixed, the holdout error will depend on the particular train-test partition chosen. Choosing a partition at random (algorithm 14.8)
will only give a point estimate. In random subsampling (algorithm 14.9), we apply
the holdout method multiple times with randomly selected train-test partitions.
The estimated generalization error is the mean over all runs.10 Because the validation sets are chosen randomly, this method does not guarantee that we validateon all of the data points.

The standard deviation over all
runs can be used to estimate thestandard deviation of the estimated generalization error.

10© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

268 c ha p te r 14 . su rrog a te mod e l s

Algorithm 14.8. A method for
randomly partitioning m data samples into training and holdout sets,
where h samples are assigned tothe holdout set.

function holdout_partition(m, h=div(m,2))p = randperm(m)train = p[(h+1):m]
holdout = p[1:h]return TrainTest(train, holdout)
endfunction random_subsampling(X, y, fit, metric;
h=div(length(X),2), k_max=10)m = length(X)mean(train_and_validate(X, y, holdout_partition(m, h),fit, metric) for k in 1 : k_max)
end

Algorithm 14.9. The random subsampling method used to obtainmean and standard deviation estimates for model generalization error using k_max runs of the holdout
method.

14.5.2 Cross Validation

D1

D2

train(••••)
train(••••)
train(••••)
train(••••)
train(••••)

D3
test( fˆ, •)test( fˆ, •)test( fˆ, •)test( fˆ, •)test( fˆ, •)D4

D5

generalization error estimate
generalization error estimate
generalization error estimate
generalization error estimate
generalization error estimate

Figure 14.10. Cross-validation partitions the data into equally sized
sets. Each set is the holdout set
once. Here we show 5-fold crossvalidation.

generalization error µ and σ
Using a train-test partition can be wasteful because our model tuning can takeadvantage only of a segment of our data. Better results are often obtained by using
k-fold cross validation.11 Here, the original dataset D is randomly partitioned into ksets D1 , D2 , . . . , Dk of equal, or approximately equal, size, as shown in figure 14.10
and implemented in algorithm 14.10. We then train k models, one on each subsetof k − 1 sets, and we use the withheld set to estimate the generalization error. The
cross-validation estimate of generalization error is the mean generalization error
over all folds:12

11

Also known as rotation estimation.

As with random subsampling,an estimate of variance can be obtained from the standard deviation
over folds.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com
1214 . 5 . mod e l se l e c ti on

1 k (i )ǫcross-validation
k i∑
=1

2
1
(i )
ˆ(i) (x)y−fǫcross-validation =∑(i )|Dtest | (x,y)∈D (i)269ǫcross-validation =(14.31)

test

(i )
(i )
where ǫcross-validation and Dtest are the cross-validation estimate and the withheld

test set, respectively, for the ith fold.

function k_fold_cross_validation_sets(m, k)
perm = randperm(m)sets = TrainTest[]
for i = 1:kvalidate = perm[i:k:m];train = perm[setdiff(1:m, i:k:m)]
push!(sets, TrainTest(train, validate))end
return sets
endfunction cross_validation_estimate(X, y, sets, fit, metric)mean(train_and_validate(X, y, tt, fit, metric)
for tt in sets)
end

Cross-validation also depends on the particular data partition. An exception isleave-one-out cross-validation with k = m, which has a deterministic partition. It
trains on as much data as possible, but it requires training m models.13 Averagming over all (m/k) possible partitions, known as complete cross-validation, is often
too expensive. While one can average multiple cross-validation runs, it is morecommon to average the models from a single cross-validation partition.
Cross-validation is demonstrated in example 14.2.

Algorithm 14.10.

The methodk_fold_cross_validation_sets

constructs the sets needed for
k-fold cross validation on m
samples, with k ≤ m. The method
cross_validation_estimate

computes the mean of the generalization error estimate bytraining and validating on the listof train-validate sets contained
in sets. The other variables are
the list of design points X, thecorresponding objective function
values y, a function fit that trains
a surrogate model, and a function
metric that evaluates a model ona data set.

M. Stone, “Cross-Validatory
Choice and Assessment of Statistical Predictions,” Journal of the Royal
Statistical Society, vol. 36, no. 2,
pp. 111–147, 1974.
13© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

270 c ha p te r 14 . su rrog a te mod e l s

14.5.3 The Bootstrap
The bootstrap method14 uses multiple bootstrap samples, which consist of m indicesinto a dataset of size m independently chosen uniformly at random. The indices
are chosen with replacement, so some indices may be chosen multiple times andsome indices may not be chosen at all as shown in figure 14.11. The bootstrap
sample is used to fit a model that is then evaluated on the original training set. A
method for obtaining bootstrap samples is given in algorithm 14.11.
If b bootstrap samples are made, then the bootstrap estimate of the generalization error is the mean of the corresponding generalization error estimates

14
B. Efron, “Bootstrap Methods:
Another Look at the Jackknife,” The
Annals of Statistics, vol. 7, pp. 1–26,
1979.

ǫtest , . . . , ǫtest :
(1)

(b)

ǫboot ==

1 b (i )
ǫtest
b i∑
=1

(14.32)

2
1 m 1 b  ( j)(i ) ( j )
ˆ
y
−
f
(
x
)
m j∑
b∑
=1 i =1(14.33)where fˆ(i) is the model fit to the ith bootstrap sample. The bootstrap method isimplemented in algorithm 14.12.
The bootstrap error in equation (14.32) tests models on data points to which
they were fit. The leave-one-out bootstrap estimate removes this source of bias byonly evaluating fitted models to withheld data:2
1 m 1 b  y( j) − fˆ(i) (x( j) )ǫleave-one-out-boot =
m j∑
c ∑
=1 − j i =1 0

if jth index was not in the ith bootstrap sample
otherwise

where c− j is the number of bootstrap samples that do not contain index j. The
leave-one-out bootstrap method is implemented in algorithm 14.13.
The probability of a particular index not being in a bootstrap sample is:1 m≈ e−1 ≈ 0.368(14.35)1−m
so a bootstrap sample is expected to have on average 0.632m distinct indices fromthe original dataset.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com(14.34)

14 . 5 . mod e l se l e c ti on 271

f ( x ) = sin(2x ) cos(10x ) + ǫ/10
where x ∈ [0, 1] and ǫ is random noise with zero mean and unit variance,ǫ ∼ N (0, 1).

Example 14.2.
Cross validation used to fit a hyperparameter.
mean cross validated MSE

Suppose we want to fit a noisy objective function using radial basis functions
with the noise hyperparameter λ (section 14.4). We can use cross validation
to determine λ. We are given ten samples from our noisy objective function.
In practice, the objective function will be unknown, but this example uses

0.1
5 · 10−2
0

10−4

10−1
λ

102

Random.seed!(0)f = x->sin(2x)*cos(10x)X = rand(10)y = f.(X) + randn(length(X))/10

We will use three folds assigned randomly:
sets = k_fold_cross_validation_sets(length(X), 3)

Next, we implement our metric. We use the mean squared error:
metric = (f, X, y)->begin
m = length(X)return sum((f(X[i]) - y[i])^2 for i in m)/mend

We now loop through different values of λ and fit different radial basis
functions. We will use the Gaussian radial basis. Cross validation is used toobtain the MSE for each value:
λs = 10 .^ range(-4, stop=2, length=101)
es = []
basis = r->exp(-5r^2)
for λ in λs
fit = (X, y)->regression(X, y, radial_bases(basis, X), λ)push!(es,cross_validation_estimate(X, y, sets, fit, metric)[1])end

The resulting curve has a minimum at λ ≈ 0.2.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

272 c ha p te r 14 . su rrog a te mod e l s

train(•)

test( fˆ, •)generalization error estimate

bootstrap_sets(m, b) = [TrainTest(rand(1:m, m), 1:m) for i in 1:b]function bootstrap_estimate(X, y, sets, fit, metric)mean(train_and_validate(X, y, tt, fit, metric) for tt in sets)
endfunction leave_one_out_bootstrap_estimate(X, y, sets, fit, metric)m, b = length(X), length(sets)
ε = 0.0
models = [fit(X[tt.train], y[tt.train]) for tt in sets]
for j in 1 : m
c = 0
δ = 0.0
for i in 1 : b
if j ∉ sets[i].train
c += 1δ += metric(models[i], [X[j]], [y[j]])endend
ε += δ/cend
return ε/mend

Figure 14.11. A single bootstrap
sample consists of m indices into
the dataset sampled with replacement. A bootstrap sample is used
to train a model, which is evaluatedon the full dataset to obtain an estimate of the generalization error.

Algorithm 14.11. A method for obtaining b bootstrap samples, eachfor a data set of size m.
Algorithm 14.12. A method forcomputing the bootstrap generalization error estimate by training
and validating on the list of trainvalidate sets contained in sets. The
other variables are the list of design
points X, the corresponding objective function values y, a function
fit that trains a surrogate model,
and a function metric that evaluates a model on a data set.
Algorithm 14.13. A method forcomputing the leave-one-out bootstrap generalization error estimate
using the train-validate sets sets.
The other variables are the list ofdesign points X, the corresponding objective function values y, afunction fit that trains a surrogate
model, and a function metric that
evaluates a model on a data set.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

14 . 5 . mod e l se l e c ti on 273

Unfortunately, the leave-one-out bootstrap estimate introduces a new bias
due to the varying test set sizes. The 0.632 bootstrap estimate15 (algorithm 14.14)
alleviates this bias:(14.36)

ǫ0.632-boot = 0.632ǫleave-one-out-boot + 0.368ǫboot

function bootstrap_632_estimate(X, y, sets, fit, metric)models = [fit(X[tt.train], y[tt.train]) for tt in sets]
ϵ_loob = leave_one_out_bootstrap_estimate(X,y,sets,fit,metric)
ϵ_boot = bootstrap_estimate(X,y,sets,fit,metric)
return 0.632ϵ_loob + 0.368ϵ_bootend

Several generalization estimation methods are compared in example 14.3.
Consider ten evenly spread samples of f ( x ) = x2 + ǫ/2 over x ∈ [−3, 3],
where ǫ is zero-mean, unit-variance Gaussian noise. We would like to test
several different generalization error estimation methods when fitting a linear
model to this data. Our metric is the root mean squared error, which is thesquare root of the mean squared error.
The methods used are the holdout method with eight training samples,five-fold cross validation, and the bootstrap methods each with ten bootstrap
samples. Each method was fitted 100 times and the resulting statistics areshown below.

The 0.632 bootstrap estimate wasintroduced in B. Efron, “Estimating the Error Rate of a Prediction Rule: Improvement on CrossValidation,” Journal of the American Statistical Association, vol. 78,
no. 382, pp. 316–331, 1983. A variant, the 0.632+ bootstrap estimate,
was introduced in B. Efron and
R. Tibshirani, “Improvements on
Cross-Validation: The .632+ Bootstrap Method,” Journal of the American Statistical Association, vol. 92,
no. 438, pp. 548–560, 1997.
15

Algorithm 14.14. A method for
obtaining the 0.632 bootstrap estimate for data points X, objectivefunction values y, number of bootstrap samples b, fitting function
fit, and metric function metric.
Example 14.3. A comparisonof generalization error estimation
methods. The vertical lines in thebox and whisker plots indicate theminimum, maximum, first andthird quartiles, and median of every generalization error estimation
method among 50 trials.

Holdout
5-Fold Cross Validation
Bootstrap

Leave-One-Out Bootstrap
0.632 Bootstrap

0

2

4

6

8

10

12

root mean squared error© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

274 c ha p te r 14 . su rrog a te mod e l s

14.6 Summary• Surrogate models are function approximations that can be optimized insteadof the true, potentially expensive objective function.
• Many surrogate models can be represented using a linear combination of basis
functions.
• Model selection involves a bias-variance tradeoff between models with low
complexity that cannot capture important trends and models with high complexity that overfit to noise.
• Generalization error can be estimated using techniques such as holdout, k-foldcross validation, and the bootstrap.

14.7 Exercises
Exercise 14.1. Derive an expression satisfied by the optimum of the regression
problem equation (14.8) by setting the gradient to zero. Do not invert any matrices.
The resulting relation is called the normal equation.
Exercise 14.2. When would we use a more descriptive model, for example, withpolynomial features, versus a simpler model like linear regression?
Exercise 14.3. A linear regression problem of the form in equation (14.8) is notalways solved analytically, and optimization techniques are used instead. Why isthis the case?
Exercise 14.4. Suppose we evaluate our objective function at four points: 1, 2,
3, and 4, and we get back 0, 5, 4, and 6. We want to fit a polynomial modelf ( x ) = ∑ik=0 θi xi . Compute the leave-one-out cross validation estimate of themean squared error as k varies between 0 and 4. According to this metric, what isthe best value for k, and what are the best values for the elements of θ?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

15 Probabilistic Surrogate Models
The previous chapter discussed how to construct surrogate models from evaluateddesign points. When using surrogate models for the purpose of optimization, it
is often useful to quantify our confidence in the predictions of these models. One
way to quantify our confidence is by taking a probabilistic approach to surrogate
modeling. A common probabilistic surrogate model is the Gaussian process, which
represents a probability distribution over functions. This chapter will explain
how to use Gaussian processes to infer a distribution over the values of different
design points given the values of previously evaluated design points. We will
discuss how to incorporate gradient information as well as noisy measurementsof the objective function. Since the predictions made by a Gaussian process aregoverned by a set of parameters, we will discuss how to infer these parameters
directly from data.

15.1

Gaussian Distribution

Before introducing Gaussian processes, we will first review some relevant properties of the multivariate Gaussian distribution, often also referred to as the multivariate normal distribution.1 An n-dimensional Gaussian distribution is parameterized
by its mean µ and its covariance matrix Σ. The probability density at x is


1
⊤ −1
−n/2
−1/2
(15.1)N (x | µ, Σ) = (2π )|Σ|exp − (x − µ) Σ (x − µ)2
Figure 15.1 shows contour plots of the density functions with different covariancematrices. Covariance matrices are always positive semidefinite.
A value sampled from a Gaussian is writtenx ∼ N (µ, Σ)(15.2)The univariate Gaussian distribution is discussed in appendix C.7.

1

276

c ha p te r 15 . p roba bi l i sti c su rrog a te mod e l s

"

1
Σ=
0

0
1

#

"

3
Σ=
0

0
0.5

#

"

1
Σ=
0.9

0.9
1

#

x2

2

0

−2
−2

0
x1

2

−2

0
x1

Two jointly Gaussian random variables a and b can be written
#!
" # "
" #
A C
µa
a
,
∼N
C⊤ B
µb
b

−2

2

b ∼ N ( µb , B )(15.3)(15.4)

The conditional distribution for a multivariate Gaussian also has a convenient
closed-form solution:a | b ∼ N µa | b , Σ a | b(15.5)µa|b = µa + CB−1 (b − µb )

Σa|b = A − CB−1 C⊤

2Figure 15.1. Multivariate Gaussians with different covariance matrices.

The marginal distribution2 for a vector of random variables is given by its corresponding mean and covariancea ∼ N ( µa , A )0
x1

2
The marginal distribution is the
distribution of a subset of the variables when the rest are integrated,or marginalized, out. For a distribution over two variables a and b
the marginal distribution over a is:(15.6)(15.7)

Example 15.1 illustrates how to extract the marginal and conditional distributions
from a multivariate Gaussian.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

p( a) =Zp( a, b) db

15 . 2. g a u ssi a n p roc e sse s 277

For example, consider
" # "
" #
3
0
x1
∼N
,
x2
1
1

1
2

#!

Example 15.1. Marginal and conditional distributions for a multivariate Gaussian.

The marginal distribution for x1 is N (0, 3), and the marginal distribution
for x2 is N (1, 2).
The conditional distribution for x1 given x2 = 2 is
µx1 | x2 =2 = 0 + 1 · 2−1 · (2 − 1) = 0.5

Σ x1 | x2 =2 = 3 − 1 · 2−1 · 1 = 2.5
x1 | ( x2 = 2) ∼ N (0.5, 2.5)

15.2

Gaussian Processes

In the previous chapter, we approximated the objective function f using a surrogate model function fˆ fitted to previously evaluated design points. A special
type of surrogate model known as a Gaussian process allows us not only to predict f but also to quantify our uncertainty in that prediction using a probability
distribution.3
A Gaussian process is a distribution over functions. For any finite set of points({x 1) , . . . , x(m) }, the associated function evaluations {y1 , . . . , ym } are distributed
according to:

 

 
k ( x(1) , x(1) ) · · · k ( x(1) , x( m ) )m ( x(1) )
y1

 

 . 
..
..
..
..

,
 .  ∼ N 
(15.8)
.
.
.
.

 

 . 
k ( x( m ) , x(1) ) · · · k ( x( m ) , x( m ) )m (x( m ) )ym

where m(x) is a mean function and k(x, x′ ) is the covariance function, or kernel.4
The mean function can represent prior knowledge about the function. The kernel
controls the smoothness of the functions. Methods for constructing the mean
vector and covariance matrix using mean and covariance functions are given inalgorithm 15.1.

3
A more extensive introduction
to Gaussian processes is provided
by C. E. Rasmussen and C. K. I.
Williams, Gaussian Processes for Machine Learning. MIT Press, 2006.

The mean function produces the
expectation:

4m(x) = E [ f (x)]and the covariance function produces the covariance:k(x, x′ ) =E ( f (x) − m(x))( f (x′ ) − m(x′ ))© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

278

c ha p te r 15 . p roba bi l i sti c su rrog a te mod e l s

Algorithm 15.1. The function μ forconstructing a mean vector given
a list of design points and a meanfunction m, and the function Σ forconstructing a covariance matrix
given one or two lists of design
points and a covariance functionk.

μ(X, m) = [m(x) for x in X]
Σ(X, k) = [k(x,x′) for x in X, x′ in X]
K(X, X′, k) = [k(x,x′) for x in X, x′ in X′]

A common kernel function is the squared exponential kernel, where( x − x ′ )2k( x, x ) = exp −
2ℓ2
′



(15.9)The parameter ℓ corresponds to what is called the characteristic length-scale, which
can be thought of as the distance we have to travel in design space until theobjective function value changes significantly.5 Hence, larger values of ℓ resultin smoother functions. Figure 15.2 shows functions sampled from a Gaussian
process with a zero-mean function and a squared exponential kernel with different
characteristic length-scales.

ℓ = 1/2

A mathematical definition of characteristic length-scale is provided
by C. E. Rasmussen and C. K. I.
Williams, Gaussian Processes for Machine Learning. MIT Press, 2006.
5

ℓ=1

ℓ=2

y

2

0

−2
0

2

6

4
x

8

10

0

2

6

4

8

10

0

2

x

There are many other kernel functions besides the squared exponential. Several
are shown in figure 15.3. Many kernel functions use r, which is the distance
between x and x′ . Usually the Euclidean distance is used. The Matérn kernel uses
the gamma function Γ, implemented by gamma from the SpecialFunctions.jl
package, and Kν ( x ) is the modified Bessel function of the second kind, implemented
by besselk(ν,x). The neural network kernel augments each design vector with a 1
for ease of notation: x̄ = [1, x1 , x2 , . . .] and x̄′ = [1, x1′ , x2′ , . . .].

6

4

8

10

x
Figure 15.2. Functions sampledfrom Gaussian processes withsquared exponential kernels.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

15 . 2. g a u ssi a n p roc e sse s 279nConstant: σ02

Linear: ∑ σd2 xd xd′

Exponential: exp(− ℓr )γ-Exponential: exp(−( ℓr )γ )Polynomial: ( x ⊤ x ′ + σ02 ) p

d =1

y

2
0

−2

Squared Exponential: exp(− 2rℓ2 )
2

y

2
0

−2

1− ν
Matérn: 2Γ(ν)

√

2ν ℓr

ν

Kν

√

2ν ℓr



Rational Quadratic:2
1 + 2αr ℓ2−αNeural Network: sin

4

−4

−1

q

2x̄⊤ Σx̄′(1+2x̄⊤ Σx̄)(1+2x̄′⊤ Σx̄′ )

!

y

2
0

−2
−4

−2

0
x

2

4

−4

−2

0
x

2

−2

0
x

2

4

Figure 15.3. Functions sampledfrom Gaussian processes with different kernel functions. Shownfunctions are for σ02 = σd2 = ℓ = 1,
p = 2, γ = ν = α = 0.5, and Σ = I.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

280c ha p te r 15 . p roba bi l i sti c su rrog a te mod e l s

This chapter will focus on examples of Gaussian processes with single-dimensional
design spaces for ease of plotting. However, Gaussian processes can be definedover multidimensional design spaces, as illustrated in figure 15.4.

ℓ=1

x1

x1

ℓ=2

x2

ℓ = 1/2

As we will see in section 15.5, Gaussian processes can also incorporate priorindependent noise variance, denoted ν. A Gaussian process is thus defined bymean and covariance functions, prior design points and their function evaluations,and a noise variance. The associated type is given in algorithm 15.2.
mutable struct GaussianProcessm # meank # covariance function
X # design points
y # objective values
ν # noise variance
end

15.3

x1
Figure 15.4. Functions sampledfrom a Gaussian process with zeromean and squared-exponential
kernels over a two-dimensional design space.

Algorithm 15.2. A Gaussian process is defined by a mean function
m, a covariance function k, sampled
design vectors X and their corresponding values y, and a noise variance ν.

Prediction

Gaussian processes are able to represent distributions over functions using conditional probabilities. Suppose we already have a set of points X and the corresponding y, but we wish to predict the values ŷ at points X ∗ . The joint distribution© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

15 . 3. p re d i c ti on

is

" #
"
# "
ŷ
m( X ∗ )K( X ∗ , X ∗ )∼N,ym( X )K( X , X ∗ )K( X ∗ , X )K( X , X )

#!

281

(15.10)In the equation above, we use the functions m and K, which are defined as follows:m( X ) = [m(x(1) ), . . . , m(x(n) )]
k(x(1) , x′(1) ) · · · k(x(1) , x′(m) )


..
..
..

K( X, X ′ ) = 
.
.
.


k(x(n) , x′(1) ) · · · k(x(n) , x′(m) )(15.11)(15.12)The conditional distribution is given byIn the language of Bayesian statistics, the posterior distribution isthe distribution of possible unobserved values conditioned on observed values.

6ŷ | y ∼ N m( X ∗ ) + K( X ∗ , X )K( X, X )−1 (y − m( X )), K( X ∗ , X ∗ ) − K( X ∗ , X )K( X, X )−1 K( X, X ∗ ) (15.13)
{z
}
{z
} |
|
mean

covariance

Note that the covariance does not depend on y. This distribution is often referred
to as the posterior distribution.6 A method for computing and sampling from theposterior distribution defined by a Gaussian process is given in algorithm 15.3.

Algorithm 15.3. The function
mvnrand samples from a multivariate Gaussian with an added inflation factor to prevent numerical issues. The method rand samples a
Gaussian process GP at the given
design points in matrix X.

function mvnrand(μ, Σ, inflation=1e-6)
N = MvNormal(μ, Σ + inflation*I)
return rand(N)endBase.rand(GP, X) = mvnrand(μ(X, GP.m), Σ(X, GP.k))

The predicted mean can be written as a function of x:µ̂(x) = m(x) + K(x, X )K( X, X )−1 (y − m( X ))= m(x) + θ⊤ K( X, x)(15.14)(15.15)where θ = K( X, X )−1 (y − m( X )) can be computed once and reused for different
values of x. Notice the similarity to the surrogate models in the previous chapter. The value of the Gaussian process beyond the surrogate models discussed
previously is that it also quantifies our uncertainty in our predictions.
The variance of the predicted mean can also be obtained as a function of x:ν̂(x) = K(x, x) − K(x, X )K( X, X )−1 K( X, x)(15.16)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

282 c ha p te r 15 . p roba bi l i sti c su rrog a te mod e l s

In some cases, it is more convenient to formulate equations in terms of thestandard deviation, which is the square root of the variance:qσ̂(x) = ν̂(x)(15.17)The standard deviation has the same units as the mean. From the standard deviation, we can compute the 95% confidence region, which is an interval containing
95% of the probability mass associated with the distribution over y given x. For
a particular x, the 95% confidence region is given by µ̂(x) ± 1.96σ̂ (x). One may
want to use a confidence level different from 95%, but we will use 95% for theplots in this chapter. Figure 15.5 shows a plot of a confidence region associatedwith a Gaussian process fit to four function evaluations.

yconfidence intervaltrue objective function
predicted function mean
fit points

Figure 15.5. A Gaussian processusing the squared exponential kernel and its 95% confidence interval.
Uncertainty increases the farther
we are from a data point. The expected function value approaches
zero as we move far away from thedata point.

x

15.4 Gradient Measurements
Gradient observations can be incorporated into Gaussian processes in a mannerconsistent with the existing Gaussian process machinery.7 The Gaussian process
is extended to include both the function value and its gradient:
"
# "
#!
" #
mf
Kf f Kf∇y∼N,(15.18)m∇
K∇ f K∇∇∇y7
For an overview, see for example
A. O’Hagan, “Some Bayesian Numerical Analysis,” Bayesian Statistics, vol. 4, J. M. Bernardo, J. O.
Berger, A. P. Dawid, and A. F. M.
Smith, eds., pp. 345–363, 1992.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

15 . 4 . g ra d i e n t me a su re me n ts 283


where y ∼ N m f , K f f is a traditional Gaussian process, m∇ is a mean function

for the gradient,8 K f ∇ is the covariance matrix between function values andgradients, K∇ f is the covariance matrix between function gradients and values,
and K∇∇ is the covariance matrix between function gradients.
These covariance matrices are constructed using covariance functions. The
linearity of Gaussians causes these covariance functions to be related:(15.19)k f f (x, x′ ) = k(x, x′ )′(15.20)′k ∇ f (x, x ) = ∇x k(x, x )′(15.21)′k f ∇ (x, x ) = ∇x′ k(x, x )′8Like the mean of the function
value, m∇ is often zero.

′k ∇∇ (x, x ) = ∇x ∇x′ k (x, x )(15.22)

Example 15.2 uses these relations to derive the higher-order covariance functions
for a particular kernel.
Example 15.2. Deriving covariancefunctions for a Gaussian processwith gradient observations.

Consider the squared exponential covariance function


1
′
′ 2
k f f (x, x ) = exp − x − x2
We can use equations (15.19) to (15.22) to obtain the other covariance functions necessary for using Gaussian processes with gradient information:


1
′
′
′ 2
k ∇ f (x, x )i = −(xi − xi ) exp − x − x
2




1
′ 2
′
′
′
k ∇∇ (x, x )ij = − (i = j) − (xi − xi )(x j − x j ) exp − x − x2As a reminder, Boolean expressions, such as (i = j), return 1 if true and 0 if
false.

Prediction can be accomplished in the same manner as with a traditional
Gaussian process. We first construct the joint distribution




 
ŷ
m f (X∗ )K f f (X∗ , X∗ )
 

 
 y  ∼ N  m f ( X )  ,  K f f ( X , X ∗ )∇ym∇ ( X )K∇ f ( X , X ∗ )K f f (X∗ , X )K f f (X , X )K∇ f ( X , X )
K f ∇ (X∗ , X )
K f ∇ ( X , X ) K∇∇ ( X , X )(15.23)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

284 c ha p te r 15 . p roba bi l i sti c su rrog a te mod e l s

For a Gaussian process over n-dimensional design vectors given m pairs offunction and gradient evaluations and ℓ query points, the covariance blocks havethe following dimensions:ℓ×ℓℓ×mℓ × nmm×ℓ
m×m
m × nmnm × ℓ nm × m nm × nm

(15.24)

Example 15.3 constructs such a covariance matrix.
Suppose we have evaluated a function and its gradient at two locations, x(1)and x(2) , and we wish to predict the function value at x̂. We can infer thejoint distribution over ŷ, y, and ∇y using a Gaussian process. The covariancematrix is:

k f f (x̂, x̂) k (x(1) , x̂) ff
 k (x(2) , x̂)
 ff

(1)
k
 ∇ f (x , x̂)1
(1)
k
 ∇ f (x , x̂)2(2) , x̂)
k
(
x
1
 ∇f
k ∇ f (x(2) , x̂)2

k f f (x̂, x(1) )k f f (x(1) , x(1) )k f f (x(2) , x(1) )k ∇ f (x(1) , x(1) )1
k ∇ f (x(1) , x(1) )2
k ∇ f (x(2) , x(1) )1
k ∇ f (x(2) , x(1) )2

k f f (x̂, x(2) )k f f (x(1) , x(2) )k f f (x(2) , x(2) )k ∇ f (x(1) , x(2) )1
k ∇ f (x(1) , x(2) )2k ∇ f (x(2) , x(2) )1
k ∇ f (x(2) , x(2) )2

k f ∇ (x̂, x(1) )1
k f ∇ (x(1) , x(1) )1
k f ∇ (x(2) , x(1) )1k ∇∇ (x(1) , x(1) )11
k ∇∇ (x(1) , x(1) )21
k ∇∇ (x(2) , x(1) )11
k ∇∇ (x(2) , x(1) )21

k f ∇ (x̂, x(1) )2
k f ∇ (x(1) , x(1) )2
k f ∇ (x(2) , x(1) )2k ∇∇ (x(1) , x(1) )12
k ∇∇ (x(1) , x(1) )22
k ∇∇ (x(2) , x(1) )12
k ∇∇ (x(2) , x(1) )22

k f ∇ (x̂, x(2) )1
k f ∇ (x(1) , x(2) )1
k f ∇ (x(2) , x(2) )1
k ∇∇ (x(1) , x(2) )11
k ∇∇ (x(1) , x(2) )21
k ∇∇ (x(2) , x(2) )11
k ∇∇ (x(2) , x(2) )21

Example 15.3. Constructing the covariance matrix for a Gaussian process with gradient observations.

k f ∇ (x̂, x(2) )2
k f ∇ (x(1) , x(2) )2 
k f ∇ (x(2) , x(2) )2 


(
1
)
(
2
)
k ∇∇ (x , x )12 


(
1
)
(
2
)
k ∇∇ (x , x )22 k ∇∇ (x(2) , x(2) )12 k ∇∇ (x(2) , x(2) )22

The conditional distribution follows the same Gaussian relations as in equation (15.13):(15.25)ŷ | y, ∇y ∼ N (µ∇ , Σ∇ )where:## −1 "#⊤ "y − m f (X)K f f (X , X ) K f ∇ (X , X )K f f (X , X∗ )µ∇ = m f ( X ) +∇ y − m∇ ( X )K∇ f ( X , X ) K∇∇ ( X , X )K∇ f ( X , X ∗ )"#⊤ "# −1 "#
K f f (X , X∗ )K f f (X , X ) K f ∇ (X , X )K f f (X , X∗ )∗∗
Σ∇ = K f f ( X , X ) −K∇ f ( X , X ∗ )K∇ f ( X , X ) K∇∇ ( X , X )K∇ f ( X , X ∗ )∗"The regions obtained when including gradient observations are compared tothose without gradient observations in figure 15.6.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com(15.26)(15.27)

15 . 5 . n oi sy me a su re me n ts 285

yconfidence interval without gradientconfidence interval with gradientpredicted mean without gradientpredicted mean with gradientfit points
true function

Figure 15.6. Gaussian processes
with and without gradient information using squared exponential
kernels. Incorporating gradient information can significantly reducethe confidence intervals.

x

15.5 Noisy Measurements
So far we have assumed that the objective function f is deterministic. In practice,
however, evaluations of f may include measurement noise, experimental error,or numerical roundoff.
We can model noisy evaluations as y = f (x) + z, where f is deterministic but
z is zero-mean Gaussian noise, z ∼ N (0, ν). The variance of the noise ν can be
adjusted to control the uncertainty.9
The new joint distribution is:
#!
# "
"
" #
K( X ∗ , X ∗ ) K( X ∗ , X )m( X ∗ )ŷ(15.28),∼N
K( X , X ∗ ) K( X , X ) + νIm( X )y

The techniques covered in section 14.5 can be used to tune thevariance of the noise.
9with conditional distribution:ŷ | y, ν ∼ N (µ∗ , Σ∗ )∗∗(15.29)∗µ = m( X ) + K( X , X )(K( X, X ) + νI)−1(y − m( X ))Σ∗ = K( X ∗ , X ∗ ) − K( X ∗ , X )(K( X, X ) + νI)−1 K( X, X ∗ )(15.30)(15.31)As the equations above show, accounting for Gaussian noise is straightforward
and the posterior distribution can be computed analytically. Figure 15.7 showsa noisy Gaussian process. Algorithm 15.4 implements prediction for Gaussian
processes with noisy measurements.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

286 c ha p te r 15 . p roba bi l i sti c su rrog a te mod e l s

Figure 15.7. A noisy Gaussian process using a squared exponential
kernel.

yconfidence regiontrue objective function
predicted function mean
fit pointsxfunction predict(GP, X_pred)m, k, ν = GP.m, GP.k, GP.νtmp = K(X_pred, GP.X, k) / (K(GP.X, GP.X, k) + ν*I)
μₚ = μ(X_pred, m) + tmp*(GP.y - μ(GP.X, m))
S = K(X_pred, X_pred, k) - tmp*K(GP.X, X_pred, k)
νₚ = diag(S) .+ eps() # eps prevents numerical issues
return (μₚ, νₚ)
end

Algorithm 15.4. A method for obtaining the predicted means andstandard deviations in f undera Gaussian process. The method
takes a Gaussian process GP anda list of points X_pred at which toevaluate the prediction. It returns
the mean and variance at each evaluation point.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

15 . 6 . fi tti n g g a u ssi a n p roc e sse s 287

15.6

Fitting Gaussian Processes

The choice of kernel and parameters has a large effect on the form of the Gaussian
process between evaluated design points. Kernels and their parameters can bechosen using cross validation introduced in the previous chapter. Instead ofminimizing the squared error on the test data, we maximize the likelihood of thedata.10 That is, we seek the parameters θ that maximize the probability of thefunction values, p(y | X, θ). The likelihood of the data is the probability that theobserved points were drawn from the model. Equivalently, we can maximize thelog likelihood, which is generally preferable because multiplying small probabilities
in the likelihood calculation can produce extremely small values. Given a dataset
D with n entries, the log likelihood is given by

Alternatively, we could maximize the pseudolikelihood as discussed by C. E. Rasmussen and
C. K. I. Williams, Gaussian Processes
for Machine Learning. MIT Press,
2006.

10

n
1
1
log 2π − log |Kθ ( X, X ) + νI| − (y − mθ ( X ))⊤ (Kθ ( X, X ) + νI)−1 (y − mθ ( X ))2
2
2
(15.32)where the mean and covariance functions are parameterized by θ.
Let us assume a zero mean such that mθ ( X ) = 0 and θ refers only to theparameters for the Gaussian process covariance function. We can arrive at a
maximum likelihood estimate by gradient ascent. The gradient is then given by!
1 ⊤ −1 ∂K −1
1
∂
−1 ∂K
(15.33)log p(y | X, θ) = y K
K y − tr Σθ
∂θ j
2
∂θ j
2
∂θ j
log p(y | X, ν, θ) = −where Σθ = Kθ ( X, X ) + νI. Above, we use the matrix derivative relations
∂K −1
∂K−1
= − K−1
K
∂θ j
∂θ j
!
∂ log |K|
−1 ∂K
= tr K∂θ j∂θ j(15.34)(15.35)where tr(A) denotes the trace of a matrix A, defined to be the sum of the elementson the main diagonal.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

288

c ha p te r 15 . p roba bi l i sti c su rrog a te mod e l s

15.7 Summary• Gaussian processes are probability distributions over functions.
• The choice of kernel affects the smoothness of the functions sampled from a
Gaussian process.
• The multivariate normal distribution has analytic conditional and marginal
distributions.
• We can compute the mean and standard deviation of our prediction of an
objective function at a particular design point given a set of past evaluations.
• We can incorporate gradient observations to improve our predictions of theobjective value and its gradient.
• We can incorporate measurement noise into a Gaussian process.
• We can fit the parameters of a Gaussian process using maximum likelihood.

15.8 Exercises
Exercise 15.1. Gaussian processes will grow in complexity during the optimization process as more samples accumulate. How can this be an advantage overmodels based on regression?
Exercise 15.2. How does the computational complexity of prediction with aGaussian process increase with the number of data points m?

Exercise 15.3. Consider the function f ( x ) = sin( x )/ x2 + 1 over [−5, 5]. Plotthe 95% confidence bounds for a Gaussian process with derivative information
fitted to the evaluations at {−5, −2.5, 0, 2.5, 5}. What is the maximum standard
deviation in the predicted distribution within [−5, 5]? How many function evaluations, evenly-spaced over the domain, are needed such that a Gaussian process
without derivative information achieves the same maximum predictive standard
deviation?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

15 . 8 . e x e rc i se s 289

Assume zero-mean functions and noise-free observations, and use the covariance functions:1
2
k f f ( x, x ′ ) = exp − x − x ′ 2
2


1
2
k ∇ f ( x, x ′ ) = ( x ′ − x ) exp − x − x ′ 2212
k ∇∇ ( x, x ′ ) = (( x − x ′ )2 − 1) exp(− x − x ′ 2 )2
Exercise 15.4. Derive the relation k f ∇ (x, x′ )i = cov f (x), ∂x∂ ′ f (x′ ) = ∂x∂ ′ k f f (x, x′ ).
iiExercise 15.5. Suppose we have a multivariate Gaussian distribution over two
variables a and b. Show that the variance of the conditional distribution over agiven b is no greater than the variance of the marginal distribution over a. Does
this make intuitive sense?

Exercise 15.6. Suppose we observe many outliers while sampling, that is, weobserve samples that do not fall within the confidence interval given by theGaussian process. This means the probabilistic model we chose is not appropriate.
What can we do?
Exercise 15.7. Consider model selection for the function evaluation pairs ( x, y):{(1, 0), (2, −1), (3, −2), (4, 1), (5, 0)}Use leave-one-out cross-validation to select the kernel that maximizes thelikelihood of predicting the withheld pair given a Gaussian process over the otherpairs in the fold. Assume zero mean with no noise. Select from the kernels:exp(−k x − x ′ k)exp(−k x − x ′ k2 )(1 + k x − x ′ k)−1

(1 + k x − x ′ k2 ) −1

(1 + k x − x ′ k)−2

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

16 Surrogate Optimization
The previous chapter explained how to use a probabilistic surrogate model, inparticular a Gaussian process, to infer probability distributions over the true
objective function. These distributions can be used to guide an optimization
process toward better design points.1 This chapter outlines several common
techniques for choosing which design point to evaluate next. The techniques
we discuss here greedily optimize various metrics.2 We will also discuss how
surrogate models can be used to optimize an objective measure in a safe manner.

16.1

Prediction-Based Exploration

In prediction-based exploration, we select the minimizer of the surrogate function.
An example of this approach is the quadratic fit search that we discussed earlier
in section 3.5. With quadratic fit search, we use a quadratic surrogate model to fit
the last three bracketing points and then select the point at the minimum of thequadratic function.
If we use a Gaussian process surrogate model, prediction-based optimization
has us select the minimizer of the mean function
x(m+1) = arg min µ̂(x)x∈X(16.1)

where µ̂(x) is the predicted mean of a Gaussian process at a design point x basedon the previous m design points. The process is illustrated in figure 16.1.
Prediction-based optimization does not take uncertainty into account, and new
samples can be generated very close to existing samples. Sampling at locations
where we are already confident in the objective value is a waste of function
evaluations.

A. Forrester, A. Sobester, and A.
Keane, Engineering Design via Surrogate Modelling: A Practical Guide.
Wiley, 2008.
1

An alternative to greedy optimization is to frame the problem
as a partially observable Markov decision process and plan ahead somenumber of steps as outlined by M.
Toussaint, “The Bayesian SearchGame,” in Theory and Principled
Methods for the Design of Metaheuristics, Y. Borenstein and A. Moraglio,
eds. Springer, 2014, pp. 129–144.
See also R. Lam, K. Willcox, andD. H. Wolpert, “Bayesian Optimization with a Finite Budget: An Approximate Dynamic Programming
Approach,” in Advances in Neural Information Processing Systems(NIPS), 2016.
2

c ha p te r 16 . su rrog a te op ti mi z a ti on

true
predicted
fit points
next sample

y

292

x

x

x

x
Figure 16.1. Prediction-based optimization selects the point that minimizes the mean of the objectivefunction.

16.2 Error-Based Exploration
Error-based exploration seeks to increase confidence in the true function. A Gaussian
process can tell us both the mean and standard deviation at every point. A large
standard deviation indicates low confidence, so error-based exploration samples
at design points with maximum uncertainty.
The next sample point is:(16.2)x (m+1) = arg max σ̂(x)x∈X

where σ̂ (x) is the standard deviation of a Gaussian process at a design point x
based on the previous m design points. The process is illustrated in figure 16.2.

yyconfidence regiontrue
predicted
fit points
sampled

x

x

x

x

Gaussian processes are often defined over all of R n . Optimization problems

with unbounded feasible sets will always have high uncertainty far away fromsampled points, making it impossible to become confident in the true underlying
function over the entire domain. Error-based exploration must thus be constrained
to a closed region.

Figure 16.2. Error-based exploration selects a point with maximal
uncertainty.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

16 . 3. l ow e r c on fi d e n c e bou n d e x p l ora ti on 293

16.3 Lower Confidence Bound Exploration
While error-based exploration reduces the uncertainty in the objective function
overall, its samples are often in regions that are unlikely to contain a global minimum. Lower confidence bound exploration trades off between greedy minimization
employed by prediction-based optimization and uncertainty reduction employedby error-based exploration. The next sample minimizes the lower confidence boundof the objective function
LB(x) = µ̂(x) − ασ̂(x)(16.3)

where α ≥ 0 is a constant that controls the trade-off between exploration and exploitation. Exploration involves minimizing uncertainty, and exploitation involves
minimizing the predicted mean. We have prediction-based optimization with
α = 0, and we have error-based exploration as α approaches ∞. The process is
illustrated in figure 16.3.

yyconfidence regiontruelower boundpredicted
fit points
sampled

x

x

x

x

16.4 Probability of Improvement Exploration
We can sometimes obtain faster convergence by selecting the design point thatmaximizes the chance that the new point will be better than the samples we haveseen so far. The improvement for a function sampled at x producing y = f (x) isymin − y if y < ymin
(16.4)
I (y) =
0otherwise

Figure 16.3. Lower confidencebound exploration trades off between minimizing uncertainty andminimizing the predicted function.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

294 c ha p te r 16 . su rrog a te op ti mi z a ti on

where ymin is the minimum value sampled so far.
The probability of improvement at points where σ̂ > 0 is
Z ymin(16.5)N (y | µ̂, σ̂2 )dyymin − µ̂=Φσ̂P(y < ymin ) =−∞(16.6)

where Φ is the standard normal cumulative distribution function (see appendix C.7).
This calculation (algorithm 16.1) is shown in figure 16.4. Figure 16.5 illustrates
this process. When σ̂ = 0, which occurs at points where we have noiseless measurements, the probability of improvement is zero.
true
predicted
fit points

Figure 16.4. The probability of improvement is the probability that
evaluating a particular point will
yield a better result than the best
so far. This figure shows the probability density function predicted
at a query point, with the shaded
region below ymin corresponding
to the probability of improvement.

prob_of_improvement(y_min, μ, σ) = cdf(Normal(μ, σ), y_min)

Algorithm 16.1. Computing theprobability of improvement for agiven best y value y_min, mean μ,
and variance ν.

ybest so far

probability of improvement
query pointx

16.5

Expected Improvement Exploration

Optimization is concerned with finding the minimum of the objective function.
While maximizing the probability of improvement will tend to decrease theobjective function over time, it does not improve very much with each iteration.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

16 . 5 . e x p e c te d i mp rove me n t e x p l ora ti on 295

yconfidence regiontrue
predicted
fit points
sampledP[ f (x) < ymin ]

1

0

x

x

x

x

We can focus our exploration of points that maximize our expected improvement
over the current best function value.
Through a substitutionz=y − µ̂
σ̂

′
ymin
=

ymin − µ̂σ̂Figure 16.5. Maximizing the probability of improvement selects samples most likely to produce lower
objective point values.

(16.7)

we can write the improvement in equation (16.4) asσ̂(y′ − z) if z < y′ and σ̂ > 0minmin
I (y) =
0otherwise(16.8)

where µ̂ and σ̂ are the predicted mean and standard deviation at the sample pointx.
We can calculate the expected improvement using the distribution predictedby the Gaussian process:E [ I (y)] = σ̂

Z y′

min

−∞

(16.9)

′ymin− z N (z | 0, 1) dz



Z y′
Z y′
min
min
′
= σ̂ ymin
N (z | 0, 1) dz −z N (z | 0, 1) dz
−∞
−∞



′
′
′
P(z ≤ ymin
) + N (ymin| 0, 1) − N (−∞ | 0, 1)
= σ̂ymin
|
{z
}
=0

= (ymin − µ̂) P(y ≤ ymin ) + σ̂2 N (ymin | µ̂, σ̂2 )(16.10)(16.11)(16.12)

Figure 16.6 illustrates this process using algorithm 16.2.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

296 c ha p te r 16 . su rrog a te op ti mi z a ti on

Algorithm 16.2. Computing the expected improvement for a given
best y value y_min, mean μ, andstandard deviation σ.

function expected_improvement(y_min, μ, σ)p_imp = prob_of_improvement(y_min, μ, σ)p_ymin = pdf(Normal(μ, σ), y_min)
return (y_min - μ)*p_imp + σ^2*p_ymin
endyconfidence regiontrue
predicted
fit points
sampledE [ I (y)]

1

x

x

x

x

16.6 Safe Optimization
In some contexts, it may be costly to evaluate points that are deemed unsafe,
which may correspond to low performing or infeasible points. Problems such as
the in-flight tuning of the controller of a drone or safe movie recommendations
require safe exploration—searching for an optimal design point while carefully
avoiding sampling an unsafe design.
This section outlines the SafeOpt algorithm,3 which addresses a class of safe
exploration problems. We sample a series of design points x(1) , . . . , x(m) in pursuitof a minimum but without f (x(i) ) exceeding a critical safety threshold ymax . In
addition, we receive only noisy measurements of the objective function, where thenoise is zero-mean with variance ν. Such an objective function and its associatedsafe regions are shown in figure 16.7.
The SafeOpt algorithm uses Gaussian process surrogate models for prediction.
At each iteration, we fit a Gaussian process to the noisy samples from f . After the

Figure 16.6. Maximizing the expected improvement selects samples which are likely to improve thelower bound by as much as possible.
ymax

y

0

x∗
x
Figure 16.7. SafeOpt solves safe exploration problems that minimize
f while remaining within safe regions defined by maximum objective function values.
3
Y. Sui, A. Gotovos, J. Burdick,and A. Krause, “Safe Exploration
for Optimization with GaussianProcesses,” in International Conference on Machine Learning (ICML),
vol. 37, 2015.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

16 . 6 . sa fe op ti mi z a ti on 297

ith sample, SafeOpt calculates the upper and lower confidence bounds:q(16.13)ui (x) = µ̂i−1 (x) + βν̂i−1 (x)qℓi (x) = µ̂i−1 (x) − βν̂i−1 (x)(16.14)

where larger values of β yield wider confidence regions. Such bounds are shownin figure 16.8.
confidence regiontrue
predictedℓ( x )u( x )
fit points

Figure 16.8. An illustration of functions based on the predictions of a
Gaussian process used by SafeOpt.

x

The Gaussian process predicts a distribution over f (x) for any design point.
Being Gaussian, these predictions can provide only a probabilistic guarantee ofsafety up to an arbitrary factor:4!
ymax − µ̂(x)pP( f (x) ≤ ymax ) = Φ≥ Psafe(16.15)ν̂(x)The predicted safe region S consists of the design points that provide a probabilityof safety greater than the required level Psafe , as illustrated in figure 16.9. The safe
region can also be defined in terms of Lipschitz upper bounds constructed fromupper bounds evaluated at previously sampled points.
SafeOpt chooses a safe sample point that balances the desire to localize areachable minimizer of f and to expand the safe region. The set of potential
minimizers of f is denoted M (figure 16.10), and the set of points that will
potentially lead to the expansion of the safe regions is denoted E (figure 16.11).
To trade off exploration and exploitation, we choose the design point x with thelargest predictive variance among both sets M and E .5
The set of potential minimizers consists of the safe points whose lower confidence bound is lower than the lowest upper bound:′
Mi = x ∈ Si | ℓi (x) ≤ min ui (x )(16.16)x′ ∈Si

Note the similarity to the probability of improvement.

4

5
For a variation of this algorithm,
see F. Berkenkamp, A. P. Schoellig,and A. Krause, “Safe Controller
Optimization for Quadrotors with
Gaussian Processes,” in IEEE International Conference on Robotics andAutomation (ICRA), 2016.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

y298 c ha p te r 16 . su rrog a te op ti mi z a ti on

confidence interval
objective function ffit points
safety threshold
estimated safe region S

Figure 16.9. The safety regions(green) predicted by a Gaussian
process.

estimated safe region S

Figure 16.10. The potential minimizers are the safe points whose
lower bounds are lower than thebest, safe upper bound.

safety probability

x
1

Psafe

0

x

potential minimizers Myconfidence region
objective function
fit points
safety threshold
best upper boundx© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

16 . 6 . sa fe op ti mi z a ti on 299

At step i, the set of potential expanders Ei consists of the safe points that, if
added to the Gaussian process, optimistically assuming the lower bound, produce
a posterior distribution with a larger safe set. The potential expanders naturally
lie near the boundary of the safe region.
estimated safe region S
expected expanders E

Figure 16.11. The set of potential
expanders.

yconfidence region
objective function
fit points
safety thresholdx

Given an initial safe point6 x(1) , SafeOpt chooses the design point among sets
M and E with the greatest uncertainty, as quantified by the width wi ( x ) =u( x ) − ℓ( x ):wi (x)(16.17)x (i) = arg max
Sx∈Mi

SafeOpt cannot guarantee safety
if it is not initialized with at least
one point that it knows is safe.

6

Ei

SafeOpt proceeds until a termination condition is met. It is common to run thealgorithm for a fixed number of iterations or until the maximum width is lessthan a set threshold.
Maintaining sets in multidimensional spaces can be computationally challenging. SafeOpt assumes a finite design space X that can be obtained with a sampling
method applied over the continuous search domain. Increasing the density of thefinite design space leads to more accurate results with respect to the continuous
space, but it takes longer per iteration.
SafeOpt is implemented in algorithm 16.3, and calls algorithm 16.4 to update
the predicted confidence intervals; algorithm 16.5 to compute the safe, minimizer,
and expander regions; and algorithm 16.6 to select a query point. The progressionof SafeOpt is shown for one dimension in figure 16.12, and for two dimensions infigure 16.13.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

300 c ha p te r 16 . su rrog a te op ti mi z a ti on

function safe_opt(GP, X, i, f, y_max; β=3.0, k_max=10)
push!(GP, X[i], f(X[i])) # make first observationm = length(X)u, l = fill(Inf, m), fill(-Inf, m)
S, M, E = falses(m), falses(m), falses(m)for k in 1 : k_max
update_confidence_intervals!(GP, X, u, l, β)
compute_sets!(GP, S, M, E, X, u, l, y_max, β)
i = get_new_query_point(M, E, u, l)
i != 0 || break
push!(GP, X[i], f(X[i]))
end

end

Algorithm 16.3. The SafeOpt algorithm applied to an empty Gaussian process GP, a finite design
space X, index of initial safe pointi, objective function f, and safety
threshold y_max. The optional parameters are the confidence scalar
β and the number of iterations
k_max. A tuple containing the bestsafe upper bound and its index in
X is returned.

# return the best point
update_confidence_intervals!(GP, X, u, l, β)S[:] = u .≤ y_max
if any(S)u_best, i_best = findmin(u[S])i_best = findfirst(isequal(i_best), cumsum(S))
return (u_best, i_best)
else
return (NaN,0)
end

function update_confidence_intervals!(GP, X, u, l, β)
μₚ, νₚ = predict(GP, X)
u[:] = μₚ + sqrt.(β*νₚ)l[:] = μₚ - sqrt.(β*νₚ)return (u, l)end

Algorithm 16.4. A method for
updating the lower and upperbounds used in SafeOpt, whichtakes the Gaussian process GP, thefinite search space X, the upper andlower-bound vectors u and l, andthe confidence scalar β.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

16 . 6 . sa fe op ti mi z a ti on 301

function compute_sets!(GP, S, M, E, X, u, l, y_max, β)
fill!(M, false)
fill!(E, false)# safe set
S[:] = u .≤ y_max
if any(S)# potential minimizers
M[S] = l[S] .< minimum(u[S])Algorithm 16.5. A method for updating the safe S, minimizer M, andexpander E sets used in SafeOpt.
The sets are all Boolean vectors indicating whether the corresponding design point in X is in the set.
The method also takes the Gaussian process GP, the upper andlower bounds u and l, respectively,
the safety threshold y_max, and theconfidence scalar β.

# maximum width (in M)
w_max = maximum(u[M] - l[M])

end
end

# expanders - skip values in M or those with w ≤ w_max
E[:] = S .& .~M # skip points in M
if any(E)E[E] .= maximum(u[E] - l[E]) .> w_max
for (i,e) in enumerate(E)if e && u[i] - l[i] > w_max
push!(GP, X[i], l[i])μₚ, νₚ = predict(GP, X[.~S])
pop!(GP)E[i] = any(μₚ + sqrt.(β*νₚ) .≥ y_max)
if E[i]; w_max = u[i] - l[i]; end
end
end
end

return (S,M,E)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

302 c ha p te r 16 . su rrog a te op ti mi z a ti on

function get_new_query_point(M, E, u, l)ME = M .| E
if any(ME)v = argmax(u[ME] - l[ME])return findfirst(isequal(v), cumsum(ME))
else
return 0
end
end

Algorithm 16.6. A method for obtaining the next query point in
SafeOpt. The index of the point in X
with the greatest width is returned.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

16 . 6 . sa fe op ti mi z a ti on 303

Figure 16.12. The first eight iterations of SafeOpt on a univariate
function. SafeOpt can never reachthe global optimum on the righthand side because it requires crossing an unsafe region. We can only
hope to find the global minima in
our locally reachable safe region.

y

y

y

y

y

y

y

y

y

estimated safe region
potential minimizers
potential expandersx© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

304 c ha p te r 16 . su rrog a te op ti mi z a ti on

Figure 16.13. SafeOpt applied to
the flower function (appendix B.4)
with ymax = 2, a Gaussian process mean of µ(x) = 2.5, varianceν = 0.01, β = 10, a 51 × 51 uniform
grid over the search space, and an
initial point x (1) = [−2.04, 0.96].
The color indicates the value of theupper bound, the cross indicatesthe safe point with the lowest upper bound, and the white contour
line is the estimated safe region.
The objective function with thetrue safe region outlined in white:

x2

2
0

−2

x2

2
0

−2

2
x2

x2

2
0

0

−2

−2

−2

x2

2
0

−2
−2

0
x1

2

−2

0
x1

2

−2

0
x1

2

−2

0
x1

2

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

0
x1

2

16 . 7. su mma ry

305

16.7 Summary• Gaussian processes can be used to guide the optimization process using avariety of strategies that use estimates of quantities such as the lower confidencebound, probability of improvement, and expected improvement.
• Some problems do not allow for the evaluation of unsafe designs, in whichcase we can use safe exploration strategies that rely on Gaussian processes.

16.8

Exercises

Exercise 16.1. Give an example in which prediction-based optimization fails.
Exercise 16.2. What is the main difference between lower confidence boundexploration and error-based exploration in the context of optimization?
Exercise 16.3. We have a function f ( x ) = ( x − 2)2 /40 − 0.5 with x ∈ [−5, 5],
and we have evaluation points at −1 and 1. Assume we use a Gaussian processsurrogate model with a zero-mean function, and a squared exponential kernelexp(−r2 /2), where r is the Euclidean distance between two points. Which value
for x would we evaluate next if we were maximizing probability of improvement?
Which value for x would we evaluate next if we were maximizing expected
improvement?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

17 Optimization under Uncertainty
Previous chapters assumed that the optimization objective is to minimize a deterministic function of our design points. In many engineering tasks, however, there
may be uncertainty in the objective function or the constraints. Uncertainty may
arise due to a number of factors, such as model approximations, imprecision, andfluctuations of parameters over time. This chapter covers a variety of methods foraccounting for uncertainty in our optimization to enhance robustness.1

17.1 Uncertainty
Uncertainty in the optimization process can arise for a variety of reasons. Theremay be irreducible uncertainty,2 which is inherent to the system, such as background noise, varying material properties, and quantum effects. These uncertainties cannot be avoided and our design should accommodate them. There may
also be epistemic uncertainty,3 which is uncertainty caused by a subjective lack ofknowledge by the designer. This uncertainty can arise from approximations in
the model4 used when formulating the design problem and errors introduced bynumerical solution methods.
Accounting for these various forms of uncertainty is critical to ensuring robust
designs. In this chapter, we will use z ∈ Z to represent a vector of random
values. We want to minimize f (x, z), but we do not have control over z. Feasibility
depends on both the design vector x and the uncertain vector z. This chapter
introduces the feasible set over x and z pairs as F . We have feasibility if and only
if (x, z) ∈ F . We will use X as the design space, which may include potentiallyinfeasible designs depending on the value of z.
Optimization with uncertainty was briefly introduced in section 15.5 in thecontext of using a Gaussian process to represent an objective function inferred

Additional references include:
H.-G. Beyer and B. Sendhoff,“Robust Optimization—A Comprehensive Survey,” Computer
Methods in Applied Mechanics
and Engineering, vol. 196, no. 33,
pp. 3190–3218, 2007. G.-J. Park,
T.-H. Lee, K. H. Lee, and K.-H.
Hwang, “Robust Design: An
Overview,” AIAA Journal, vol. 44,
no. 1, pp. 181–191, 2006.
1

This form of uncertainty is sometimes called aleatory uncertainty or
random uncertainty.

2

Epistemic uncertainty is also
called reducible uncertainty.
3

The statistician George Box famously wrote: All models are wrong;some models are useful. G. E. P. Box,
W. G. Hunter, and J. S. Hunter,
Statistics for Experimenters: An Introduction to Design, Data Analysis, andModel Building, 2nd ed. Wiley, 2005.
p. 440.
4

308 c ha p te r 17. op ti mi z a ti on u n d e r u n c e rta i n ty

from noisy measurements. We had f (x, z) = f (x) + z with the additional assumption that z comes from a zero-mean Gaussian distribution.5 Uncertainty
may be incorporated into the evaluation of a design point in other ways. For
example, if we had noise in the input to the objective function,6 we might havef (x, z) = f (x + z). In general, f (x, z) can be a complex, nonlinear function of x
and z. In addition, z may not come from a Gaussian distribution; in fact, it may
come from a distribution that is not known.
Figure 17.1 demonstrates how the degree of uncertainty can affect our choiceof design. For simplicity, x is a scalar and z is selected from a zero-mean Gaussian
distribution. We assume that z corresponds to noise in the input to f , and sof ( x, z) = f ( x + z). The figure shows the expected value of the objective function
for different levels of noise. The global minimum without noise is a. However,aiming for a design near a can be risky since it lies within a steep valley, making it
rather sensitive to noise. Even with low noise, it may be better to choose a designnear b. Designs near c can provide even greater robustness to larger amounts ofnoise. If the noise is very high, the best design might even fall between b and c,which corresponds to a local maximum in the absence of noise.
There are a variety of different ways to account for uncertainty in optimization.
We will discuss both set-based uncertainty and probabilistic uncertainty.7

Here, the two-argument versionof f takes as input the design point
and random vector, but the singleargument version of f represents
a deterministic function of the design point without noise.
5

For example, there may be variability in the manufacturing of ourdesign.

6

Other approaches for representing uncertainty include DempsterShafer theory, fuzzy-set theory, andpossibility theory, which are beyond
the scope of this book.
7

expected value

Figure 17.1. The global minimum
at a in the noiseless case is sensitive
to noise. Other design points may
be more robust depending on the
anticipated level of noise.

a

b

c
x

noise-freevery low noise

low noisehigh noisevery high noise© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

17. 2. se t-ba se d u n c e rta i n ty 309

17.2 Set-Based Uncertainty
Set-based uncertainty approaches assume that z belongs to a set Z , but theseapproaches make no assumptions about the relative likelihood of different points
within that set. The set Z can be defined in different ways. One way is to defineintervals for each component of Z . Another way is to define Z by a set of inequality
constraints, g(x, z) ≤ 0, similar to what was done for the design space X inchapter 10.

17.2.1Minimax

In problems with set-based uncertainty, we often want to minimize the maximum
possible value of the objective function. Such a minimax approach8 solves the
optimization problem
minimize maximize f (x, z)
x∈X

z∈Z

Also called the robust counterpart
approach or robust regularization.
8(17.1)

In other words, we want to find an x that minimizes f , assuming the worst-casevalue for z.
This optimization is equivalent to defining a modified objective function
f mod (x) = maximize f (x, z)(17.2)

minimize f mod (x)(17.3)z∈Z

and then solvingx∈X

Example 17.1 shows this optimization on a univariate problem and illustrates theeffect of different levels of uncertainty.
In problems where we have feasibility constraints, our optimization problem
becomesminimize maximize f (x, z) subject to (x, z) ∈ F and (x, z′ ) ∈ F for all z′
x∈X

z∈Z

(17.4)

Example 17.2 shows the effect of applying minimax on the space of feasible design
points when there are constraints.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

310 c ha p te r 17. op ti mi z a ti on u n d e r u n c e rta i n ty

Example 17.1. Example of a minimax approach to optimization under set-based uncertainty.

Consider the objective functionf ( x, z) = f ( x + z) = f ( x̃ ) =


− x̃


x̃2

if x̃ ≤ 0otherwise

where x̃ = x + z, with a set-based uncertainty region z ∈ [−ǫ, ǫ]. The minimax approach is a minimization problem over the modified objective function
f mod ( x ) = maximizez∈[−ǫ,ǫ] f ( x, z).
4trueǫ = 0.2
ǫ = 0.4
ǫ = 0.6
ǫ = 0.8
ǫ = 1.0
ǫ = 1.2
ǫ = 1.4
ǫ = 1.6
ǫ = 1.8
ǫ = 2.0

y

3

2

1

0

−1

−0.5

0
x

0.5

1

The figure above shows f mod ( x ) for several different values of ǫ. The
minimum for ǫ = 0 coincides with the minimum of f ( x, 0). As ǫ is increased,
the minimum first shifts right as x increases faster than x2 and then shifts
left as x2 increases faster than x. The robust minimizer does not generally
coincide with the minimum of f ( x, 0).

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

17. 2. se t-ba se d u n c e rta i n ty

Consider an uncertain feasible set in the form of a rotated ellipse, where(x, z) ∈ F if and only if z ∈ [0, π/2] and311

Example 17.2. The minimax approach applied to uncertainty inthe feasible set.

( x1 cos z + x2 sin z)2 + ( x1 sin z − x2 cos z)2 /16 ≤ 1
When z = 0, the major axis of the ellipse is vertical. Increasing values of z
slowly rotates it counter clockwise to horizontal at z = π/2. The figure below
shows the vertical and horizontal ellipses and the set of all points that arefeasible for at least one z in blue.
A minimax approach to optimization should consider only design points
that are feasible under all values of z. The set of designs that are always
feasible are given by the intersection of all ellipses formed by varying z. Thisset is outlined in red.

x2

x1

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

312 c ha p te r 17. op ti mi z a ti on u n d e r u n c e rta i n ty

17.2.2 Information-Gap Decision Theory
Instead of assuming the uncertainty set Z is fixed, an alternative approach known
as information-gap decision theory9 parameterizes the uncertainty set by a nonnegative scalar gap parameter ǫ. The gap controls the volume of the parameterized set
Z (ǫ) centered at some nominal value z̄ = Z (0). One way to define Z (ǫ) is as a
hypersphere of radius ǫ centered at a nominal point z̄:Z (ǫ) = {z | kz − z̄k2 ≤ ǫ}

9F. M. Hemez and Y. Ben-Haim,“Info-Gap Robustness for the Correlation of Tests and Simulations ofa Non-Linear Transient,” Mechanical Systems and Signal Processing,vol. 18, no. 6, pp. 1443–1467, 2004.

(17.5)

Figure 17.2 illustrates this definition in two dimensions.
By parameterizing the uncertainty set, we avoid committing to a particular
uncertainty set. Uncertainty sets that are too large sacrifice the quality of thesolution, and uncertainty sets that are too small sacrifice robustness. Design
points that remain feasible for larger gaps are considered more robust.
In information-gap decision theory, we try to find the design point that allows
for the largest gap while preserving feasibility. This design point can be obtainedby solving the following optimization problem:ǫ if (x, z) ∈ F for all z ∈ Z (ǫ)(17.6)x∗ = arg max maximizeǫ∈[0,∞) 0 otherwisex∈X
This optimization focuses on finding designs that ensure feasibility in thepresence of uncertainty. In fact, equation (17.6) does not explicitly include theobjective function f . However, we can incorporate the constraint that f (x, z) be no
greater than some threshold ymax . Such performance constraints can help us avoid
excessive risk aversion. Figure 17.3 and example 17.3 illustrate the application of
information-gap decision theory.

17.3 Probabilistic Uncertainty
Models of probabilistic uncertainty use distributions over a set Z . Probabilistic
uncertainty models provide more information than set-based uncertainty models,
allowing the designer to account for the probability of different outcomes of adesign. These distributions can be defined using expert knowledge or learned
from data. Given a distribution p over Z , we can infer a distribution over the
output of f using methods that will be discussed in chapter 18. This section will

z2

ǫ

z̄

z1
Figure 17.2. A parametrized uncertainty set Z (ǫ) in the form of a
hypersphere.

x̃2

x∗

F
x̃1
Figure 17.3. Information-gap decision theory applied to an objectivefunction with additive noise f (x̃)
with x̃ = x + z and a circular uncertainty set

Z ( ǫ ) = {z | k z k2 ≤ ǫ }The design x∗ is optimal underinformation-gap decision theory as
it allows for the largest possible ǫsuch that all x∗ + z are feasible.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

17. 3. p roba bi l i sti c u n c e rta i n ty 313

Consider the robust optimization of f ( x, z) = x̃2 + 6e− x̃ with x̃ = x + z
subject to the constraint x̃ ∈ [−2, 2] with the uncertainty set Z (ǫ) = [−ǫ, ǫ].
2

10

y

noise-freeǫ = 0.3
ǫ = 0.6
ǫ = 0.9
ǫ = 1.2
ǫ = 1.5
ǫ = 1.8
ǫ = 2.1
ǫ = 2.4
ǫ = 2.7
ǫ = 3.0

Z

8
6

x∗

4
2
10

Z

y

8
6
4
2
−3

Example 17.3. We can mitigate excessive risk aversion by applying a
constraint on the maximum acceptable objective function value when
applying information-gap decision
theory.

x∗
−2

−1

0
x

1

2

3

Applying information-gap decision theory to this problem results in a
maximally-sized uncertainty set and a design centered in the suboptimal
region of the objective function. Applying an additional constraint on the
maximum objective function value, f ( x, z) ≤ 5, allows the same approach tofind a design with better noise-free performance. The blue lines indicate theworst-case objective function value for a given uncertainty parameter ǫ.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

314 c ha p te r 17. op ti mi z a ti on u n d e r u n c e rta i n ty

outline five different metrics for converting this distribution into a scalar value
given a particular design x. We can then optimize with respect to these metrics.10

17.3.1 Expected Value
One way to convert the distribution output by f into a scalar value is to use theexpected value or mean. The expected value is the average output that we can expect
when considering all outputs of f (x, z) for all z ∈ Z and their corresponding
probabilities. The expected value as a function of the design point x is
Ez∼ p [ f (x, z)] =ZZf (x, z) p(z) dz

Further discussion of various
metrics can be found in A. Shapiro,
D. Dentcheva, and A. Ruszczyński,Lectures on Stochastic Programming:
Modeling and Theory, 2nd ed. SIAM,
2014.
10

(17.7)The expected value does not necessarily correspond to the objective function
without noise, as illustrated in example 17.4.
Computing the integral in equation (17.7) analytically may not be possible.
One may approximate that value using sampling or a variety of other moresophisticated techniques discussed in chapter 18.

17.3.2 Variance
Besides optimizing with respect to the expected value of the function, we may
also be interested in choosing design points whose value is not overly sensitive to
uncertainty.11 Such regions can be quantified using the variance of f :h2 iVar[ f (x, z)] = Ez∼ p f (x, z) − Ez∼ p [ f (x, z)](17.8)=ZZ

f (x, z)2 p(z) dz − Ez∼ p [ f (x, z)]2

(17.9)

Sometimes
designers
seekplateau-like regions where theoutput of the objective function
is relatively constant, such as
producing materials with consistent performance or schedulingtrains such that they arrive at aconsistent time.

11© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

17. 3. p roba bi l i sti c u n c e rta i n ty 315

One common model is to apply zero-mean Gaussian noise to the function output, f (x, z) = f (x) + z, as was the case with Gaussian processes in chapter 16.
The expected value is equivalent to the noise-free case:Example 17.4. The expected valueof an uncertain objective function
depends on how the uncertainty
is incorporated into the objectivefunction.

Ez∼N (0,Σ) [ f (x) + z] = Ez∼N (0,Σ) [ f (x)] + Ez∼N (0,Σ) [z] = f (x)It is also common to add noise directly to the design vector,
f (x, z) = f (x + z) = f (x̃). In such cases the expected value is affected by
the variance of zero-mean Gaussian noise.
Consider minimizing the expected value of f ( x̃ ) = sin(2x̃ )/ x̃ withx̃ = x + z for z drawn from a zero-mean Gaussian distribution N (0, ν). Increasing the variance increases the effect that the local function landscapehas on a design.

y

2

ν=0
ν = 0.5
ν = 1.0
ν = 1.5
ν = 2.0

1
0

−10

−8

−6

−4

−2

0
x

2

4

6

8

10

The plot above shows that changing the variance affects the location of
the optima.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

316 c ha p te r 17. op ti mi z a ti on u n d e r u n c e rta i n ty

sensitivey

We call design points with large variance sensitive and design points withsmall variance robust. Examples of sensitive and robust points are shown infigure 17.4. We are typically interested in good points as measured by their
expected value that are also robust. Managing the trade-off between the expected
objective function value and the variance is a multiobjective optimization problem(see example 17.5), and we can use techniques discussed in chapter 12.

robustx17.3.3 Statistical Feasibility
An alternative metric against which to optimize is statistical feasibility. Given p(z),we can compute the probability a design point x is feasible:
P (x ∈ F ) =Z

Z((x, z) ∈ F ) p(z) dz(17.10)This probability can be estimated through sampling. If we are also interested
in ensuring that the objective value does not exceed a certain threshold, we canincorporate a constraint f (x, z) ≤ ymax as is done with information-gap decision
theory. Unlike the expected value and variance metrics, we want to maximize
this metric.

Figure 17.4.
Probabilistic approaches produce probability distributions over the model output.
Design points can be sensitive or
robust to uncertainty. The blue regions show how the distribution
over a normally distributed design
is affected by the objective function.

17.3.4 Value at Risk
The value at risk (VaR) is the best objective value that can be guaranteed withprobability α. We can write this definition mathematically in terms of the cumulative distribution function, denoted Φ(y), over the random output of the objectivefunction.The probability that the outcome is less than or equal to y is given by
Φ(y). VaR with confidence α is the minimum value of y such that Φ(y) ≥ α. Thisdefinition is equivalent to the α quantile of a probability distribution. An α close to
1 is sensitive to unfavorable outliers, whereas an α close to 0 is overly optimistic
and close to the best possible outcome.

17.3.5 Conditional Value at Risk
The conditional value at risk (CVaR) is related to the value at risk.12 CVaR is the
expected value of the top 1 − α quantile of the probability distribution over the
output. This quantity is illustrated in figure 17.5.

The conditional value at risk
is also known as the mean excess loss, mean shortfall, and tailvalue at risk. R. T. Rockafellar and
S. Uryasev, “Optimization of Conditional Value-at-Risk,” Journal of
Risk, vol. 2, pp. 21–42, 2000.
12© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

17. 3. p roba bi l i sti c u n c e rta i n ty 317

Consider the objective function f ( x, z) = x2 + z, with z drawn from a Gamma
distribution that depends on x. We can construct a function dist(x) that
returns a Gamma distribution from the Distributions.jl package:

Example 17.5. Considering boththe expected value and the variance in optimization under uncertainty.

dist(x) = Gamma(2/(1+abs(x)),2)

This distribution has mean 4/(1 + | x |) and variance 8/(1 + | x |).

y

10

5

0
−3

−2

−1

0
x

1

2

3

expected value±σ

We can find a robust optimizer that minimizes both the expected value
and the variance. Minimizing with respect to the expected value, ignoring
the variance, produces two minima at x ≈ ±0.695. Incorporating a penalty
for the variance shifts these minima away from the origin. The figure below
pshows objective functions of the form α E [y | x ] + (1 − α) Var[y | x ] forα ∈ [0, 1] along with their associated minima.

αµ + (1 − α)σ

10
8
6
4
2

−3

−2

−1

0
x

1

2

3

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

318 c ha p te r 17. op ti mi z a ti on u n d e r u n c e rta i n ty

0.4

p(y)

0.3
0.2

top 1 − α quantile

0.1
0

VaR

expected value

Figure 17.5. CVaR and VaR for aparticular level α. CVaR is the expected value of the top 1 − α quantile, whereas the VaR is the lowest
objective function value over thesame quantile.

CVaR

y

CVaR has some theoretical and computational advantages over VaR. CVaR isless sensitive to estimation errors in the distribution over the objective output.
For example, if the cumulative distribution function is flat in some intervals, thenVaR can jump with small changes in α. In addition, VaR does not account for costs
beyond the α quantile, which is undesirable if there are rare outliers with very
poor objective values.13

17.4 Summary• Uncertainty in the optimization process can arise due to errors in the data, themodels, or the optimization method itself.
• Accounting for these sources of uncertainty is important in ensuring robust
designs.
• Optimization with respect to set-based uncertainty includes the minimax approach that assumes the worst-case and information-gap decision theory thatfinds a design robust to a maximally sized uncertainty set.

For an overview of properties,
see G. C. Pflug, “Some Remarkson the Value-at-Risk and theConditional Value-at-Risk,” inProbabilistic Constrained Optimization: Methodology and Applications,S. P. Uryasev, ed. Springer, 2000,
pp. 272–281. and R. T. Rockafellar
and S. Uryasev, “Conditional
Value-at-Risk for General Loss
Distributions,” Journal of Banking
and Finance, vol. 26, pp. 1443–1471,
2002.
13

• Probabilistic approaches typically minimize the expected value, the variance,risk of infeasibility, value at risk, conditional value at risk, or a combination ofthese.

17.5 Exercises
Exercise 17.1. Suppose we have zero-mean Gaussian noise in the input such thatf ( x, z) = f ( x + z). Consider the three points a, b, and c in the figure below:© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

319

y

17. 5 . e x e rc i se sabg( x ) ≤ 0

c
x

X

y

Which design point is best if we are minimizing the expected value minus thestandard deviation?
Exercise 17.2. Optima, such as the one depicted in figure 17.6, often lie on aconstraint boundary and are thus sensitive to uncertainties that could cause them
to become infeasible. One approach to overcome uncertainty with respect tofeasibility is to make the constraints more stringent, reducing the size of thefeasible region as shown in figure 17.7.

x1

xFigure 17.6. Optima with active
constraints are often sensitive to
uncertainty.

Figure 17.7. Applying more stringent constraints during optimization prevents designs from being
too close to the true feasibility
boundary.

x2

x2

x∗

x1

It is common to rewrite constraints of the form g(x) ≤ gmax to γg(x) ≤ gmax ,where γ > 1 is a factor of safety. Optimizing such that the constraint values stay
below gmax /γ provides an additional safety buffer.
Consider a beam with a square cross section thought to fail when the stresses
exceed σmax = 1. We wish to minimize the cross section f ( x ) = x2 , where x is the
cross section length. The stress in the beam is also a function of the cross section
length g( x ) = x −2 . Plot the probability that the optimized design does not fail as
the factor of safety varies from 1 to 2:• Uncertainty in maximum stress, g( x, z) = x −2 + z• Uncertainty in construction tolerance, g( x, z) = ( x + z)−2
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

320 c ha p te r 17. op ti mi z a ti on u n d e r u n c e rta i n ty

• Uncertainty in material properties, g( x, z) = (1 + z) x −2
where z is zero-mean noise with variance 0.01.
Exercise 17.3. The six-sigma method is a special case of statistical feasibility inwhich a production or industrial process is improved until its assumed Gaussian
output violates design requirements only with outliers that exceed six standard
deviations. This requirement is fairly demanding, as is illustrated in figure 17.8.
Figure 17.8. Statistical feasibility
can be met either by shifting theobjective function mean away fromthe feasibility boundary or by reducing the variance of the objectivefunction.

N ( x | µ, σ)

0.4
0.3

feasibility boundary

0.2
0.1
0

−6

−2

−4

0

2

4

6

number of standard deviations from the mean

Consider the optimization problem
minimize x1xsubject to e x1 ≤ x2 + z ≤ 2e x1
with z ∼ N (0, 1). Find the optimal design x ∗ such that ( x, z) is feasible for all|z| ≤ 6.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

18 Uncertainty Propagation
As discussed in the previous chapter, probabilistic approaches to optimization under uncertainty model some of the inputs to the objective function as a probability
distribution. This chapter discusses how to propagate known input distributions
to estimate quantities associated with the output distribution, such as the mean
and variance of the objective function. There are a variety of approaches to uncertainty propagation, some based on mathematical concepts such as Monte Carlo, theTaylor series approximation, orthogonal polynomials, and Gaussian processes.
These approaches differ in the assumptions they make and the quality of theirestimates.

18.1 Sampling Methods
The mean and variance of the objective function at a particular design point canbe approximated using Monte Carlo integration,1 which approximates the integral
using m samples z(1) , . . . , z(m) , from the distribution p over Z . These estimates
are also called the sample mean and sample variance:Ez∼ p [ f (z)] ≈ µ̂ =Varz∼ p [ f (z)] ≈ ν̂ =1 m
f (z( i ) )
m i∑
=1
1 m
f (z( i ) )2
m i∑
=1

(18.1)!

− µ̂2(18.2)In the equation above, and for the rest of this chapter, we drop x from f (x, z) for
notational convenience, but the dependency on x still exists. For each new design
point x in our optimization process, we recompute the mean and variance.

Alternatively, quasi Monte Carlointegration can be used to produce
estimates with faster convergence
as discussed in chapter 13.

1

322 c ha p te r 18 . u n c e rta i n ty p rop a g a ti on

A desirable property of this sampling-based approach is that p does not need
to be known exactly. We can obtain samples directly from simulation or real-world
experiments. A potential limitation of this approach is that many samples may
be required before there is convergence to a suitable estimate. The variance ofthe sample mean for a normally distributed f is Var[µ̂] = ν/m, where ν is thetrue variance of f . Thus, doubling the number of samples m tends to halve thevariance of the sample mean.

18.2 Taylor Approximation
Another way to estimate µ̂ and ν̂ is to use the Taylor series approximation for f at
a fixed design point x.2 For the moment, we will assume that the n componentsof z are uncorrelated and have finite variance. We will denote the mean of the
distribution over z as µ and the variances of the individual components of z as
ν.3 The following is the second-order Taylor series approximation of f (z) at the
point z = µ:n
1 n n ∂2 f
∂f
( zi − µi ) + ∑ ∑(z − µi )(z j − µ j )fˆ(z) = f (µ) + ∑∂zi
2 i=1 j=1 ∂zi ∂z j ii =1(18.3)For a derivation of the mean andvariance of a general function of n
random variables, see H. Benaroya
and S. M. Han, Probability Models
in Engineering and Science. Taylor &Francis, 2005.
3
If the components of z are uncorrelated, then the covariance matrix
is diagonal and ν would be the vector composed of the diagonal elements.
2

From this approximation, we can analytically compute estimates of the mean
and variance of f :1 n ∂2 f
ν
2 i
2 i∑
z=µ
=1 ∂zi
!2
2
n n
n 
1
∂2 f
∂f
νi + ∑ ∑νi νjν̂ = ∑∂zi
2 i=1 j=1 ∂zi ∂z j
z=µ
i =1

µ̂ = f (µ) +(18.4)(18.5)The higher-order terms can be neglected to obtain a first-order approximation:µ̂ = f (µ)

n ν̂ = ∑

i =1

∂f
∂zi

2

(18.6)

νiz=µWe can relax the assumption that the components of z are uncorrelated, but it
makes the mathematics more complex. In practice, it can be easier to transformthe random variables so that they are uncorrelated. We can transform a vector ofn correlated random variables c with covariance matrix C into m uncorrelated© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

18 . 3. p ol y n omi a l c ha os 323

random variables z by multiplying by an orthogonal m × n matrix T containing
the eigenvectors corresponding to the m largest eigenvalues of C. We have z = Tc.4
The Taylor approximation method is implemented in algorithm 18.1. First- andsecond-order approximations are compared in example 18.1.
using ForwardDiff
function taylor_approx(f, μ, ν, secondorder=false)
μhat = f(μ)∇ = (z -> ForwardDiff.gradient(f, z))(μ)νhat = ∇.^2⋅ν
if secondorderH = (z -> ForwardDiff.hessian(f, z))(μ)μhat += (diag(H)⋅ν)/2
νhat += ν⋅(H.^2*ν)/2
end
return (μhat, νhat)
end

18.3

It is also common to scale the outputs such that the covariance matrix becomes the identity matrix.
This process is known as whitening.
J. H. Friedman, “Exploratory Projection Pursuit,” Journal of the American Statistical Association, vol. 82,
no. 397, pp. 249–266, 1987.
4

Algorithm 18.1. A method forautomatically computing the Taylor approximation of the mean
and variance of objective function f at design point x withnoise mean vector μ and variancevector ν. The Boolean parametersecondorder controls whether the
first- or second-order approximation is computed.

Polynomial Chaos

Polynomial chaos is a method for fitting a polynomial to evaluations of f (z) andusing the resulting surrogate model to estimate the mean and variance. We will
begin this section by discussing how polynomial chaos is used in the univariatecase. We will then generalize the concept to multivariate functions and show
how to obtain estimates of the mean and variance by integrating the function
represented by our surrogate model.

18.3.1 Univariate
In one dimension, we approximate f (z) with a surrogate model consisting of kpolynomial basis functions, b1 , . . . , bk :kf (z) ≈ fˆ(z) = ∑ θi bi (z)(18.7)

i =1© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

324 c ha p te r 18 . u n c e rta i n ty p rop a g a ti on

Consider the objective function f ( x, z) = sin( x + z1 ) cos( x + z2 ), where z1
and z2 are zero-mean Gaussian noise with variances 0.1 and 0.2, respectively.
The first and second partial derivatives of f with respect to the zs are
∂f
=
∂z1

∂2 f
= − sin( x + z1 ) cos( x + z2 )∂z22cos( x + z1 ) cos( x + z2 )∂f
= − sin( x + z1 ) sin( x + z2 )∂z2∂2 f
= − sin( x + z1 ) cos( x + z2 )∂z21

Example 18.1. The Taylor approximation applied to a univariate design problem with twodimensional Gaussian noise.

∂2 f= − cos( x + z1 ) sin( x + z2 )∂z1 ∂z2

which allow us to construct the Taylor approximation:µ̂( x ) = 0.85 sin( x ) cos( x )ν̂( x ) = 0.1 cos4 ( x ) + 0.2 sin4 ( x ) + 0.045 sin2 ( x ) cos2 ( x )We can use taylor_approx for a given x using:taylor_approx(z->sin(x+z[1])*cos(x+z[2]), [0,0], [0.1,0.2])

mean

0.5
0

variance

−0.5
0.2
0.15
0.1
0.05
0

−3

true

−2

−1

0
x

first-order Taylor approx.

1

2

3

second-order Taylor approx.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

18 . 3. p ol y n omi a l c ha os 325

In contrast with the Monte Carlo methods discussed in section 18.1, our samplesof z do not have to be randomly drawn from p. In fact, it may be desirable toobtain samples using one of the sampling plans discussed in chapter 13. We will
discuss how to obtain the basis coefficients in section 18.3.2.
We can use the surrogate model fˆ to estimate the mean:h i(18.8)µ̂ = E fˆ

=

=

Z

Z

(18.9)

fˆ(z) p(z) dz

Zk(18.10)∑ θi bi (z) p(z) dz
Z
i =1

k

Z

= ∑ θi
i =1

= θ1

Z

(18.11)bi (z) p(z) dz

Z

b1 (z) p(z) dz + . . . + θkZ

We can also estimate the variance:h i2 ν̂ = E fˆ − E fˆZZ

bk (z) p(z) dz(18.13)h i2 
h i= E fˆ2 − 2 fˆ E fˆ + E fˆ

(18.14)h i
h i2= E fˆ2 − E fˆ

=

Z

=

Z

=

Z

Z

(18.15)(18.16)fˆ(z)2 p(z) dz − µ2kk(18.17)∑ ∑ θi θ j bi (z)bj (z) p(z) dz − µ2

Z i =1 j =1
k

∑

Z

i =1

k

Z

= ∑ θi2i =1

θi2 bi (z)2 + 2

k i −1!

(18.18)bi (z)b j (z) p(z) dz − µ2(18.19)∑ ∑ θi θ j bi (z)bj (z) p(z) dz − µ2

i =2 j =1

k i −1Z(18.12)bi (z)2 p(z) dz + 2 ∑ ∑ θi θ j
i =2 j =1

Z

Z

The mean and variance can be efficiently computed if the basis functions arechosen to be orthogonal under p. Two basis functions bi and b j are orthogonal withrespect to a probability density p(z) ifZZbi (z)b j (z) p(z) dz = 0

if i 6= j(18.20)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

326 c ha p te r 18 . u n c e rta i n ty p rop a g a ti on

If the chosen basis functions are all orthogonal to one another and the firstbasis function is b1 (z) = 1, the mean is:µ̂ = θ1

= θ1
= θ1
= θ1

Z

ZZ
Z

Z
Z

b1 (z) p(z) dz + θ2Z

b1 (z)2 p(z) dz + θ2ZZ

b2 (z) p(z) dz + · · · + θk

Z

Z

Z

bk (z) p(z) dz

b1 (z)b2 (z) p(z) dz + · · · + θkZZ

b1 (z)bk (z) p(z) dz(18.21)(18.22)(18.23)p(z) dz + 0 + · · · + 0(18.24)

Similarly, the variance is:kν̂ = ∑ θi2
i =1k= ∑ θi2
i =1

= θ12

Z

k

Z

Z

Z

bi (z)2 p(z) dz + 2 ∑ ∑ θi θ j
i =2 j =1

Z

b1 (z)2 p(z) dz + ∑ θi2
i =2

Z

Z

Z

Z

bi (z)b j (z) p(z) dz − µ2bi (z)2 p(z) dz(18.25)(18.26)bi (z)2 p(z) dz − µ2k= ∑ θi2i =2

k i −1ZZ

Z

bi (z)2 p(z) dz − θ12(18.27)(18.28)The mean thus falls immediately from fitting a surrogate model to the observed data, and the variance can be very efficiently computed given the valuesR2
5
Z bi ( z ) p ( z ) dz for a choice of basis functions and probability distribution. Example 18.2 uses these procedures to estimate the mean and variance with different
sample sizes.
Polynomial chaos approximates the function using kth degree orthogonal
polynomial basis functions with i ∈ {1, . . . , k + 1} and b1 = 1. All orthogonal
polynomials satisfy the recurrence relation:(z − α )b (z)for i = 1i i(18.29)bi + 1 ( z ) =(z − αi )bi (z) − β i bi−1 (z) for i > 1

5Integrals of this form can be efficiently computed using Gaussian quadrature, covered in appendix C.8.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

18 . 3. p ol y n omi a l c ha os 327

Example 18.2. Estimating the expected value of an unknown objective function using polynomialchaos.

Consider optimizing the (unknown) objective function2f ( x, z) = 1 − e−( x+z−1) − 2e−( x+z−3)

2

E[ f | x]

E[ f | x]

E[ f | x]with z known to be drawn from a zero-mean unit-Gaussian distribution.
The objective function, its true expected value, and estimated expectedvalues with different sample counts are plotted below. The estimated expectedvalue is computed using third-order Hermite polynomials.
noise-free
exact
95% interval
mean

1
0

−1

10 samples

1
0

−1

30 samples

1
0

−1

−2

50 samples

−1

0

1

2
x

3

4

5

6

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

328 c ha p te r 18 . u n c e rta i n ty p rop a g a ti on

with b1 (z) = 1 and weightsRz bi (z)2 p(z) dz
αi = RZbi (z)2 p(z) dz
RZbi (z)2 p(z) dz
βi = R Z
2
Z bi −1 ( z ) p ( z ) dz

(18.30)

The recurrence relation can be used to generate the basis functions. Each basis
function bi is a polynomial of degree i − 1. The basis functions for several common probability distributions are given in table 18.1, can be generated using themethods in algorithm 18.2, and are plotted in figure 18.1. Example 18.3 illustrates
the effect the polynomial order has on the estimates of the mean and variance.
Distribution

DomainDensity

Name

Recursive Formhk

Uniform[−1, 1]

1
2

LegendredLek ( x ) = 2k1k! dx
kExponential[0, ∞)e− x

Laguerreddx Lak ( x ) =Unit Gaussian(−∞, ∞)

2√1 e− x /2
2π

Hermitex2 − 1ddx − 1



 i
k

Lak−1

d
Hk −1
Hk ( x ) = xHk−1 − dx

Closed Formi −1 
1−x j−i − 2i−1bi ( x ) = ∑
j
j
2
j =0
i −1 

bi ( x ) = ∑j =0bi ( x ) =
i − 1 (−1) j jxj!
j⌊(i −1)/2⌋

Basis functions for arbitrary probability density functions and domains canbe constructed both analytically and numerically.6 The Stieltjes algorithm7 (algorithm 18.3) generates orthogonal polynomials using the recurrence relationin equation (18.29). Example 18.4 shows how the polynomial order affects theestimates of the mean and variance.

18.3.2 Coefficients
The coefficients θ1 , . . . , θk in equation (18.7) can be inferred in two different ways.
The first way is to fit the values of the samples from Z using the linear regression
method discussed in section 14.3. The second way is to exploit the orthogonalityof the basis functions, producing an integration term amenable to Gaussian
quadrature.

∑

j =0

( i − 1) !

i −1(−1) 2 − j(2x )2j(2j)!( i−2 1 − j)!

Table 18.1. Orthogonal polynomial
basis functions for several common
probability distributions.
The polynomials can be scaled bya nonzero factor. It is convention to
set b1 ( x ) = 1.

6

T. J. Stieltjes, “Quelques Recherches sur la Théorie des Quadratures Dites Mécaniques,” Annales Scientifiques de l’École Normale
Supérieure, vol. 1, pp. 409–426, 1884.
in French. An overview in English
is provided by W. Gautschi, Orthogonal Polynomials: Computation andApproximation. Oxford University
Press, 2004.
7© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

18 . 3. p ol y n omi a l c ha os 329

Algorithm 18.2. Methods for constructing polynomial orthogonalbasis functions, where i indicatesthe construction of bi .

using Polynomials
function legendre(i)n = i-1
p = Polynomial([-1,0,1])^n
for i in 1 : n
p = derivative(p)endreturn p / (2^n * factorial(n))
endfunction laguerre(i)p = Polynomial([1])
for j in 2 : i
p = integrate(derivative(p) - p) + 1
end
return pendfunction hermite(i)p = Polynomial([1])
x = Polynomial([0,1])
for j in 2 : ip = x*p - derivative(p)endreturn p
end

Laguerre

Legendre

Hermite

20

b1
b2
b3
b4
b5
b6

1
10
10
0

0

0

−10
−1
−1

−0.5

0
z

0.5

1

−20

−10
0

5

10
z

15

−4

−2

0
z

2

4

Figure 18.1. Orthogonal basis functions for uniform, exponential, andunit Gaussian distributions.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

330 c ha p te r 18 . u n c e rta i n ty p rop a g a ti on

Consider the function f (z) = sin(πz) with input z drawn from a uniform
distribution over the domain [−1, 1]. The true mean and variance can becomputed analytically:
Z b

Z 1

1
sin(πz) dz = 0
2
a
−1
Z b
Z 1
1
1
ν=
f (z)2 p(z) dz − µ2 =sin2 (πz) dz − 0 =
2
2
a
−1

µ=

f (z) p(z) dz =Example 18.3. Legendre polynomials used to estimate the mean andvariance of a function with a uniformly distributed input.

(18.31)(18.32)

Suppose we have five samples of f at z = {−1, −0.2, 0.3, 0.7, 0.9}. We
can fit a Legendre polynomial to the data to obtain our surrogate model fˆ.
Polynomials of different degrees yield:
true
i=1
i=2
i=3
i=4
i=5

1

0

µ = +0.268µ = +0.207µ = +0.170µ = −0.056µ = −0.136

ν = 0.000
ν = 0.063
ν = 0.075
ν = 0.573
ν = 0.757

−1

−1

−0.5

0
z

0.5

1

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

18 . 3. p ol y n omi a l c ha os 331

Algorithm 18.3. The Stieltjes algorithm for constructing the next
polynomial basis function bi+1 according to the orthogonal recurrence relation, where bs contains{b1 , . . . , bi }, p is the probability distribution, and dom is a tuple containing a lower and upper boundfor z. The optional parameter ϵ controls the absolute tolerance of thenumerical integration. We makeuse of the Polynomials.jl package.

using Polynomials
function orthogonal_recurrence(bs, p, dom, ϵ=1e-6)
i = length(bs)c1 = quadgk(z->z*bs[i](z)^2*p(z), dom..., atol=ϵ)[1]c2 = quadgk(z-> bs[i](z)^2*p(z), dom..., atol=ϵ)[1]α = c1 / c2
if i > 1
c3 = quadgk(z->bs[i-1](z)^2*p(z), dom..., atol=ϵ)[1]β = c2 / c3
return Polynomial([-α, 1])*bs[i] - β*bs[i-1]else
return Polynomial([-α, 1])*bs[i]endend

We multiply each side of equation (18.7) by the jth basis and our probability
density function and integrate:k(18.33)f ( z ) ≈ ∑ θ i bi ( z )
i =1

Z

Z

f (z)b j (z) p(z) dz ≈

Z

k

i =1

k

Z

= ∑ θii =1

Z(18.34)bi (z)b j (z) p(z) dz(18.35)∑ θi bi (z) bj (z) p(z) dz

Z

= θj

!

Z

Z

b j (z)2 p(z) dz(18.36)

where we made use of the orthogonality property from equation (18.20).
It follows that the jth coefficient is:R
f (z)b j (z) p(z) dz
θ j = ZR
(18.37)2
Z b j ( z ) p ( z ) dz
The denominator of equation (18.37) typically has a known analytic solution
or can be inexpensively precomputed. Calculating the coefficient thus primarily
requires solving the integral in the numerator, which can be done numericallyusing Gaussian quadrature.8

Gaussian quadrature is implemented in QuadGK.jl via thequadgk function, and is covered inappendix C.8. Quadrature rules
can also be obtained using the
eigenvalues and eigenvectors of atri-diagonal matrix formed using
the coefficients αi and β i fromequation (18.30). G. H. Golub andJ. H. Welsch, “Calculation of Gauss
Quadrature Rules,” Mathematics
of Computation, vol. 23, no. 106,
pp. 221–230, 1969.
8© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

332 c ha p te r 18 . u n c e rta i n ty p rop a g a ti on

Consider the function f (z) = sin(πz) with input z drawn from a truncatedGaussian distribution with mean 3 and variance 1 over the domain [2, 5]. The
true mean and variance are:
µ=
ν=

Z b
a

Z b
a

f (z) p(z) dz =Z 5
2

Example 18.4. Legendre polynomials constructed using the Stieltjes
method to estimate the mean andvariance of a function with a random variable input.

sin(πz) p(z) dz ≈ 0.104

f (z)2 p(z) dz − µ2 =Z 52

sin2 (πz) p(z) dz − 0.1042 ≈ 0.495

where the probability density of the truncated Gaussian is: R 5 N (z|3,1)
if z ∈ [2, 5]
2 N ( τ |3,1) dτ
p(z) =
0otherwise

Suppose we have five samples of f at z = {2.1, 2.5, 3.3, 3.9, 4.7}. We canfit orthogonal polynomials to the data to obtain our surrogate model fˆ.
Polynomials of different degrees yield:
true
i=1
i=2
i=3
i=4
i=5

2

1

µ = +0.200µ = +0.204µ = −0.036µ = +0.037µ = +0.055

ν = 0.000
ν = 0.001
ν = 0.159
ν = 0.273
ν = 0.478

0

−1
2

3

4

5

z

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

18 . 3. p ol y n omi a l c ha os 333

18.3.3 Multivariate
Polynomial chaos can be applied to functions with multiple random inputs. Multivariate basis functions over m variables are constructed as a product over univariate orthogonal polynomials:mbi ( z ) = ∏ b a j ( z j )(18.38)j =1

where a is an assignment vector that assigns the a j th basis function to the jthrandom component. This basis function construction is demonstrated in example 18.5.
Consider a three-dimensional polynomial chaos model of which one of themultidimensional basis functions is b(z) = b3 (z1 )b1 (z2 )b3 (z3 ). The corresponding assignment vector is a = [3, 1, 3].
A common method for constructing multivariate basis functions is to generateunivariate orthogonal polynomials for each random variable and then to constructa multivariate basis function for every possible combination.9 This procedure
is implemented in algorithm 18.4. Constructing basis functions in this mannerassumes that the variables are independent. Interdependence can be resolved
using the same transformation discussed in section 18.2.

Example 18.5. Constructing a multivariate polynomial chaos basis
function using equation (18.38).

Here the number of multivariate
exponential basis functions grows
exponentially in the number ofvariables.
9

Algorithm 18.4. A method for constructing multivariate basis functions where bases1d contains listsof univariate orthogonal basis functions for each random variable.

function polynomial_chaos_bases(bases1d)bases = []for a in Iterators.product(bases1d...)push!(bases,z -> prod(b(z[i]) for (i,b) in enumerate(a)))
endreturn bases
end

A multivariate polynomial chaos approximation with k basis functions is stilla linear combination of termskf (z) ≈ fˆ(z) = ∑ θi bi (z)(18.39)i =1© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

334 c ha p te r 18 . u n c e rta i n ty p rop a g a ti on

where the mean and variance can be computed using the equations in section 18.3.1, provided that b1 (z) = 1.

18.4

Bayesian Monte Carlo

Gaussian processes, covered in chapter 16, are probability distributions overfunctions. They can be used as surrogates for stochastic objective functions. We canincorporate prior information, such as the expected smoothness of the objectivefunction, in a process known as Bayesian Monte Carlo or Bayes-Hermite Quadrature.
Consider a Gaussian process fit to several points with the same value forthe design point x but different values for the uncertain point z. The Gaussian
process obtained is a distribution over functions based on the observed data.
When obtaining the expected value through integration, we must consider the
expected value of the functions in the probability distribution represented by theGaussian process p( fˆ):hh ii
Ez∼ p [ f ] ≈ E fˆ∼ p( fˆ) Ez∼ p fˆ(18.40)Z Z=
fˆ(z) p(z) dz p( fˆ) d fˆ(18.41)
F̂
Z


Z
Z
=
fˆ(z) p( fˆ) d fˆ p(z) dz(18.42)

=

Z

Z

Z

F̂

µ̂(z) p(z) dz(18.43)where µ̂(z) is the predicted mean under the Gaussian process and F̂ is the spaceof functions. The variance of the estimate ishh ii
Varz∼ p [ f ] ≈ Var fˆ∼ p( fˆ) Ez∼ p fˆ
2
Z
Z Z
fˆ(z) p(z) dz −E fˆ∼ p( fˆ) [ fˆ(z′ )] p(z′ ) dz′ p( fˆ) d fˆ=ZZF̂Z Z Z hiihfˆ(z) − E fˆ∼ p( fˆ) [ fˆ(z)] fˆ(z′ ) − E fˆ∼ p( fˆ) [ fˆ(z′ )] p( fˆ) d fˆ p(z) p(z′ ) dz dz′==

ZZ ZZ F̂Z

Z

Cov( fˆ(z), fˆ(z′ )) p(z) p(z′ ) dz dz′

where Cov is the posterior covariance under the Gaussian process:Cov( fˆ(z), fˆ(z′ )) = k(z, z′ ) − k (z, Z )K( Z, Z )−1 k ( Z, z′ )(18.48)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com(18.44)(18.45)(18.46)(18.47)

18 . 4 . ba y e si a n mon te c a rl o 335

where Z contains the observed inputs.
Analytic expressions exist for the mean and variance for the special case wherez is Gaussian.10 Under a Gaussian kernel,
2 !
1 n xi − xi′′(18.49)k(x, x ) = exp − ∑2 i =1wi2
the mean for Gaussian uncertainty z ∼ N (µz , Σz ) ishh ii
Ez∼ p [ f ] ≈ E fˆ∼ p( fˆ) Ez∼ p fˆ = q⊤ K−1 ynk(x, x′ ) = ∏ r ( xi − xi′ )i =1(18.50)with

qi = |W−1

Σz + I|

−1/2


⊤

1
−1
(i )
(i )
µz − z(18.51)
exp − µz − z( Σz + W)2

where W = diag[w12 , . . . , w2n ], and we have constructed our Gaussian processusing samples (z(i) , yi ) for i ∈ {1, . . . , m}.11
The variance isVarz∼ p [ f ] = |2W−1 Σz + I|−1/2 − q⊤ K−1 q(18.52)

Even when the analytic expressions are not available, there are many problems
for which numerically evaluating the expectation is sufficiently inexpensive that
the Gaussian process approach is better than a Monte Carlo estimation.
Bayesian Monte Carlo is implemented in algorithm 18.5 and is worked out inexample 18.6.
function bayesian_monte_carlo(GP, w, μz, Σz)
W = Matrix(Diagonal(w.^2))
invK = inv(K(GP.X, GP.X, GP.k))q = [exp(-((z-μz)⋅(inv(W+Σz)*(z-μz)))/2) for z in GP.X]
q .*= (det(W\Σz + I))^(-0.5)
μ = q'*invK*GP.yν = (det(2W\Σz + I))^(-0.5) - (q'*invK*q)[1]
return (μ, ν)
endIt is also required that the covariance function obey the product correlation rule, that is, it can be written as the product of a univariate
positive-definite function r:

10

Analytic results exist for polynomial kernels and mixtures ofGaussians. C. E. Rasmussen and
Z. Ghahramani, “Bayesian Monte
Carlo,” in Advances in Neural Information Processing Systems (NIPS),
2003.

See A. Girard, C. E. Rasmussen, J. Q. Candela, and R.
Murray-Smith, “Gaussian Process
Priors with Uncertain Inputs—
Application to Multiple-Step
Ahead Time Series Forecasting,”
in Advances in Neural Information
Processing Systems (NIPS), 2003.
11

Algorithm 18.5. A method for obtaining the Bayesian Monte Carloestimate for the expected value ofa function under a Gaussian process GP with a Gaussian kernel withweights w, where the variables are
drawn from a normal distribution
with mean μz and covariance Σz.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

336 c ha p te r 18 . u n c e rta i n ty p rop a g a ti on

Consider again estimating the expected value and variance of f ( x, z) =sin( x + z1 ) cos( x + z2 ), where z1 and z2 are zero-mean Gaussian noise withvariances 1 and 1/2, respectively: µz = [0, 0] and Σz = diag([1, 1/2]).
We use Bayesian Monte Carlo with a Gaussian kernel with unit weights
w = [1, 1] for x = 0 with samples Z = {[0, 0], [1, 0], [−1, 0], [0, 1], [0, −1]}.
We compute:
#
"
1 0
W=
0 1


1
0.607 0.607 0.607 0.607
0.60710.135 0.368 0.368K = 0.607 0.135
10.368 0.368

0.607 0.368 0.368
10.135
0.607 0.368 0.368 0.135

Example 18.6. An example of using
Bayesian Monte Carlo to estimate
the expected value and variance ofa function.
The figure compares theBayesian Monte Carlo method tothe sample mean for estimating the
expected value of a function. The
same randomly sampled z values
generated for each evaluated x
were input to each method.

1q = [0.577, 0.450, 0.450, 0.417, 0.417]

Ez∼ p [ f ] = 0.0

Varz∼ p [ f ] = 0.327
Below we plot the expected value as a function of x using the same approach with ten random samples of z at each point.
true
sample mean
Bayesian MCE[ f | x]

0.5

0

−0.5
−2

−1

0
x

1

2

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

18 . 5 . su mma ry 337

18.5

Summary• The expected value and variance of the objective function are useful when
optimizing problems involving uncertainty, but computing these quantities
reliably can be challenging.
• One of the simplest approaches is to estimate the moments using sampling ina process known as Monte Carlo integration.
• Other approaches, such as the Taylor approximation, use knowledge of theobjective function’s partial derivatives.
• Polynomial chaos is a powerful uncertainty propagation technique based onorthogonal polynomials.
• Bayesian Monte Carlo uses Gaussian processes to efficiently arrive at the moments with analytic results for Gaussian kernels.

18.6

Exercises

Exercise 18.1. Suppose we draw a sample from a univariate Gaussian distribution.
What is the probability that our sample falls within one standard deviation of themean (x ∈ [µ − σ, µ + σ])? What is the probability that our sample is less than
one standard deviation above the mean (x < µ + σ)?
Exercise 18.2. Let x (1) , x (2) , . . . , x (m) be a random sample of independent, identically distributed values of size m from a distribution with mean µ and variance ν.
Show that the variance of the sample mean Var(µ̂) is ν/m.
Exercise 18.3. Derive the recurrence relation equation (18.29) that is satisfied by
all orthogonal polynomials.
Exercise 18.4. Suppose we have fitted a polynomial chaos model of an objective function f (x, z) for a particular design point x using m evaluations withz(1) , . . . , z(m) . Derive an expression for estimating the partial derivative of thepolynomial chaos coefficients with respect to a design component xi .

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

338 c ha p te r 18 . u n c e rta i n ty p rop a g a ti on

Exercise 18.5. Consider an objective function f (x, z) with design variables x
and random variables z. As discussed in chapter 17, optimization under uncertainty often involves minimizing a linear combination of the estimated mean andvariance:f mod (x, z) = αµ̂(x) + (1 − α)ν̂(x)How can one use polynomial chaos to estimate the gradient of f mod withrespect to a design variable x?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

19 Discrete Optimization
Previous chapters have focused on optimizing problems involving design variables that are continuous. Many problems, however, have design variables thatare naturally discrete, such as manufacturing problems involving mechanicalcomponents that come in fixed sizes or navigation problems involving choices between discrete paths. A discrete optimization problem has constraints such that thedesign variables must be chosen from a discrete set. Some discrete optimization
problems have infinite design spaces, and others are finite.1 Even for finite problems, where we could in theory enumerate every possible solution, it is generally
not computationally feasible to do so in practice. This chapter discusses both
exact and approximate approaches to solving discrete optimization problems
that avoid enumeration. Many of the methods covered earlier, such as simulated
annealing and genetic programming, can easily be adapted for discrete optimization problems, but we will focus this chapter on categories of techniques we have
not yet discussed.
Discrete optimization constrains the design to be integral. Consider the
problem:minimizex1 + x2
xExample 19.1. Discrete versionsof problems constrain the solution, often resulting in worse solutions than their continuous counterparts.

kxk ≤ 2x is integral√√
The optimum in the continuous case is x∗ = [− 2, − 2] with a value of√y = −2 2 ≈ −2.828. If x1 and x2 are constrained to be integer-valued, thenthe best we can do is to have y = −2 with x∗ ∈ {[−2, 0], [−1, −1], [0, −2]}.

2x2

subject toDiscrete optimization with a finite
design space is sometimes referred
to as combinatorial optimization. Fora review, see B. Korte and J. Vygen,
Combinatorial Optimization: Theory
and Algorithms, 5th ed. Springer,
2012.
1

0

−2
−2

0
x1

2

340 c ha p te r 19. d i sc re te op ti mi z a ti on

19.1 Integer Programs
An integer program is a linear program2 with integral constraints. By integral
constraints, we mean that the design variables must come from the set of integers.3
Integer programs are sometimes referred to as integer linear programs to emphasize
the assumption that the objective function and constraints are linear.
An integer program in standard form is expressed as:
minimizexc⊤ x

subject to Ax ≤ b
x≥0(19.1)

2

See chapter 11.

Integer programming is a very
mature field, with applicationsin operations research, communications networks, task scheduling, and other disciplines. Modern solvers, such as Gurobi and
CPLEX, can routinely handle problems with millions of variables.
There are packages for Julia thatprovide access to Gurobi, CPLEX,
and a variety of other solvers.
3x ∈ Znwhere Z n is the set of n-dimensional integral vectors.
Like linear programs, integer programs are often solved in equality form. Transforming an integer program to equality form often requires adding additionalslack variables s that do not need to be integral. Thus, the equality form for
integral programs is:
minimizec⊤ xx

subject toAx + s = bx≥0(19.2)s≥0x ∈ ZnMore generally, a mixed integer program (algorithm 19.1) includes both continuous and discrete design components. Such a problem, in equality form, isexpressed as:
minimizec⊤ xx

subject to

Ax = bx≥0(19.3)xD ∈ Z kDk

where D is a set of indices into the design variables that are constrained to bediscrete. Here, x = [xD , xC ], where xD represents the vector of discrete design
variables and xC the vector of continuous design variables.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

19. 2. rou n d i n g 341

mutable struct MixedIntegerProgram
A
b
c
D
end

Algorithm 19.1. A mixed integer
linear program type that reflects
equation (19.3). Here, D is the setof design indices constrained to bediscrete.

19.2 Rounding

relax(MIP) = LinearProgram(MIP.A, MIP.b, MIP.c)function round_ip(MIP)x = minimize_lp(relax(MIP))
for i in MIP.D
x[i] = round(Int, x[i])endreturn xend

We can show that rounding the continuous solution for a constraint Ax ≤ bwhen A is integral is never too far from the optimal integral solution.4 If x∗c isan optimal solution of the LP with m × n matrix A, then there exists an optimal

Figure 19.1. Rounding can produce
an infeasible design point.

y

A common approach to discrete optimization is to relax the constraint that thedesign points must come from a discrete set. The advantage of this relaxation isthat we can use techniques, like gradient descent or linear programming, thattake advantage of the continuous nature of the objective function to direct the
search. After a continuous solution is found, the design variables are rounded to
the nearest feasible discrete design.
There are potential issues with rounding. Rounding might result in an infeasible design point, as shown in figure 19.1. Even if rounding results in a feasible
point, it may be far from optimal, as shown in figure 19.2. The addition of thediscrete constraint will typically worsen the objective value as illustrated in example 19.1. However, for some problems, we can show the relaxed solution is close
to the optimal discrete solution.
We can solve integer programs using rounding by removing the integer constraint, solving the corresponding linear program, or LP, and then rounding thesolution to the nearest integer. This method is implemented in algorithm 19.2.

x∗
x
Figure 19.2. The nearest feasible discrete design may be significantly worse than the best feasible
discrete design.
Algorithm 19.2. Methods for relaxing a mixed integer linear program
into a linear program and solving
a mixed integer linear program byrounding. Both methods accept amixed integer linear program MIP.
The solution obtained by rounding
may be suboptimal or infeasible.
4
W. Cook, A. M. Gerards, A. Schrijver, and É. Tardos, “Sensitivity Theorems in Integer Linear Programming,” Mathematical Programming,
vol. 34, no. 3, pp. 251–264, 1986.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

342 c ha p te r 19. d i sc re te op ti mi z a ti on

discrete solution x∗d with x∗c − x∗d ∞ less than or equal to n times the maximum
absolute value of the determinants of the submatrices of A.
The vector c need not be integral for an LP to have an optimal integral solution
because the feasible region is purely determined by A and b. Some approaches
use the dual formulation for the LP, which has a feasible region dependent on c,in which case having an integral c is also required.
In the special case of totally unimodular integer programs, where A, b, and c haveall integer entries and A is totally unimodular, the simplex algorithm is guaranteed
to return an integer solution. A matrix is totally unimodular if the determinant of
every submatrix5 is 0, 1, or −1, and the inverse of a totally unimodular matrix isalso integral. In fact, every vertex solution of a totally unimodular integer programis integral. Thus, every Ax = b for unimodular A and integral b has an integral
solution.
Several matrices and their total unimodularity are discussed in example 19.2.
Methods for determining whether a matrix or an integer linear program are totally
unimodular are given in algorithm 19.3.
Consider the following matrices:



1
0
1
0
1



0
0
0
 0
 0
1
0
1
0 −1

Example 19.2. Examples of totally
unimodular matrices.


1

0
0

The left matrix is not totally unimodular, as
1
1


−1 −1
0
0
0


0 −1 −1
0
 1
0
1
1
0 −1


1
= −2
−1

The other two matrices are totally unimodular.

19.3

A submatrix is a matrix obtainedby deleting rows and/or columnsof another matrix.
5

Cutting Planes

The cutting plane method is an exact method for solving mixed integer programs
when A is not totally unimodular.6 Modern practical methods for solving integer
programs use branch and cut algorithms7 that combine the cutting plane method

R. E. Gomory, “An Algorithm
for Integer Solutions to Linear
Programs,” Recent Advances inMathematical Programming, vol. 64,
pp. 269–302, 1963.

6

M. Padberg and G. Rinaldi, “A
Branch-and-Cut Algorithm for the
Resolution of Large-Scale Symmetric Traveling Salesman Problems,”
SIAM Review, vol. 33, no. 1, pp. 60–
100, 1991.
7© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

19. 3. c u tti n g p l a n e s 343

isint(x, ϵ=1e-10) = abs(round(x) - x) ≤ ϵfunction is_totally_unimodular(A::Matrix)# all entries must be in [0,1,-1]
if any(a ∉ (0,-1,1) for a in A)
return falseend# brute force check every subdeterminantr,c = size(A)for i in 1 : min(r,c)
for a in subsets(1:r, i)for b in subsets(1:c, i)
B = A[a,b]if det(B) ∉ (0,-1,1)
return false
end
end
end
end
return true
end
function is_totally_unimodular(MIP)
return is_totally_unimodular(MIP.A) &&all(isint, MIP.b) && all(isint, MIP.c)
end

Algorithm 19.3. Methods for determining whether matrices A or
mixed integer programs MIP are
totally unimodular. The method
isint returns true if the given
value is integral.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

344 c ha p te r 19. d i sc re te op ti mi z a ti on

with the branch and bound method, discussed in the next section. The cutting
plane method works by solving the relaxed LP and then adding linear constraints
that result in an optimal solution.
We begin the cutting method with a solution x∗c to the relaxed LP, which must
be a vertex of Ax = b. If the D components in x∗c are integral, then it is also an
optimal solution to the original mixed integer program, and we are done. Aslong as the D components in x∗c are not integral, we find a hyperplane with x∗c on
one side and all feasible discrete solutions on the other. This cutting plane is anadditional linear constraint to exclude x∗c . The augmented LP is then solved for a
new x∗c .
Each iteration of algorithm 19.4 introduces cutting planes that make nonintegral components of x∗c infeasible while preserving the feasibility of the nearest
integral solutions and the rest of the feasible set. The integer program modified with these cutting plane constraints is solved for a new relaxed solution.
Figure 19.3 illustrates this process.

x2

Figure 19.3. The cutting plane
method introduces constraints until the solution to the LP is integral. The cutting plane is shown asa red line on the left. The feasible
region of the augmented LP is on
the right.

x2
x∗c

x∗c
x1

x1

We wish to add constraints that cut out nonintegral components of x∗c . For an
LP in equality form with constraint Ax = b, recall from section 11.2.1 that we canpartition a vertex solution x∗c to arrive at
AB x∗B + AV x∗V = b(19.4)

where x∗V = 0. The nonintegral components of x∗c will thus occur only in x∗B .
We can introduce an additional inequality constraint for each b ∈ B such thatxb∗ is nonintegral:8xb∗ − ⌊ xb∗ ⌋ − ∑ Ābv − Ābv xv ≤ 0(19.5)

Note that ⌊ x ⌋, or floor of x, rounds
x down to the nearest integer.

8v∈V© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

19. 3. c u tti n g p l a n e s 345

frac(x) = modf(x)[1]function cutting_plane(MIP)
LP = relax(MIP)x, b_inds, v_inds = minimize_lp(LP)n_orig = length(x)D = copy(MIP.D)
while !all(isint(x[i]) for i in D)
AB, AV = LP.A[:,b_inds], LP.A[:,v_inds]
Abar = AB\AVb = 0
for i in D
if !isint(x[i])b += 1A2 = [LP.A zeros(size(LP.A,1));zeros(1,size(LP.A,2)+1)]A2[end,end] = 1
A2[end,v_inds] = (x->floor(x) - x).(Abar[b,:])b2 = vcat(LP.b, -frac(x[i]))c2 = vcat(LP.c, 0)
LP = LinearProgram(A2,b2,c2)
end
end
x, b_inds, v_inds = minimize_lp(LP)endreturn x[1:n_orig]
end

Algorithm 19.4. The cutting plane
method solves a given mixed integer program MIP and returns an
optimal design vector. An error isthrown if no feasible solution exists. The helper function frac returns the fractional part of a number, and the implementation forminimize_lp, algorithm 11.5, hasbeen adjusted to return the basic
and nonbasic indices b_inds andv_inds along with an optimal design x.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

346 c ha p te r 19. d i sc re te op ti mi z a ti on−1
where Ā = AB
AV . These cutting planes use only the V -components to cut out
the nonintegral components of x∗c .
Introducing a cutting plane constraint cuts out the relaxed solution x∗c , because
all xv are zero:xb∗ − ⌊ xb∗ ⌋ − ∑ Ābv − Ābv xv > 0(19.6)| {z } v∈V
>0
|
{z
}
0

A cutting plane is written in equality form using an additional integral slack
variable xk :
(19.7)
xk + ∑ Ābv − Ābv xv = ⌊ xb∗ ⌋ − xb∗v∈V

Each iteration of algorithm 19.4 thus increases the number of constraints andthe number of variables until solving the LP produces an integral solution. Only
the components corresponding to the original design variables are returned.
The cutting plane method is used to solve a simple integer linear program inexample 19.3.

19.4 Branch and Bound
One method for finding the global optimum of a discrete problem, such as aninteger program, is to enumerate all possible solutions. The branch and bound9
method guarantees that an optimal solution is found without having to evaluate
all possible solutions. Many commercial integer program solvers use ideas fromboth the cutting plane method and branch and bound. The method gets its namefrom the branch operation that partitions the solution space10 and the bound operationthat computes a lower bound for a partition.
Branch and bound is a general method that can be applied to many kinds of
discrete optimization problems, but we will focus here on how it can be used for
integer programming. Algorithm 19.5 provides an implementation that uses apriority queue, which is a data structure that associates priorities with elements
in a collection. We can add an element and its priority value to a priority queue
using the enqueue! operation. We can remove the element with the minimum
priority value using the dequeue! operation.

9
A. H. Land and A. G. Doig, “AnAutomatic Method of Solving Discrete Programming Problems,”
Econometrica, vol. 28, no. 3, pp. 497–
520, 1960.

The subsets are typically disjoint,
but this is not required. For branch
and bound to work, at least one
subset must have an optimal solution. D. A. Bader, W. E. Hart, and
C. A. Phillips, “Parallel Algorithm
Design for Branch and Bound,” inTutorials on Emerging Methodologies
and Applications in Operations Research, H. J. Greenberg, ed., Kluwer
Academic Press, 2004.
10© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

19. 4 . bra n c h a n d bou n d 347

Example 19.3. The cutting plane
method used to solve an integer
program.

Consider the integer program:
2x1 + x2 + 3x3
#
#
"
2.5
−0.5
1.0
x=
−1.5
0.5 −1.5

minimizex

subject to

"

0.5
2.0

x ∈ Z3x≥0

The relaxed solution is x∗ ≈ [0.818, 0, 2.091], yielding:
"
#
"
#
"
#
0.5
1
−0.5
−0.091
AB =
AV =
Ā =
2 −1.50.5−0.455
From equation (19.7), the constraint for x1 with slack variable x4 is:
x4 + (⌊−0.091⌋ − (−0.091)) x2 = ⌊0.818⌋ − 0.818
x4 − 0.909x2 = −0.818

The constraint for x3 with slack variable x5 is:
x5 + (⌊−0.455⌋ − (−0.455)) x2 = ⌊2.091⌋ − 2.091
x5 − 0.545x2 = −0.091

The modified integer program has:


0.5
2

A=
0
0

−0.5
1
0.5
−1.5
−0.9090−0.545
0



0 0
0 0


1 0
0 1





2.5
−1.5 b=−0.818
−0.091

 
2
1
 
 
c = 3 00Solving the modified LP, we get x∗ ≈ [0.9, 0.9, 2.5, 0.0, 0.4]. Since this point
is not integral, we repeat the procedure with constraints:
x6 − 0.9x4 = −0.9

x8 − 0.5x4 = −0.5

x7 − 0.9x4 = −0.9

x9 − 0.4x4 = −0.4

and solve a third LP to obtain: x∗ = [1, 2, 3, 1, 1, 0, 0, 0, 0] with a final solutionof xi∗ = [1, 2, 3].
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

348 c ha p te r 19. d i sc re te op ti mi z a ti on

The algorithm begins with a priority queue containing a single LP relaxationof the original mixed integer program. Associated with that LP is a solution x∗cand objective value yc = c⊤ x∗c . The objective value serves as a lower bound on
the solution and thus serves as the LP’s priority in the priority queue. At each
iteration of the algorithm, we check whether the priority queue is empty. If it
is not empty, we dequeue the LP with the lowest priority value. If the solution
associated with that element has the necessary integral components, then wekeep track of whether it is the best integral solution found so far.
If the dequeued solution has one or more components in D that are nonintegral, we choose from x∗c such a component that is farthest from an integer value.
Suppose this component corresponds to the ith design variable. We branch by considering two new LPs, each one created by adding one of the following constraints
to the dequeued LP:11
 ∗
 ∗
(19.8)
orxi ≥ xi,cxi ≤ xi,cNote that ⌈ x ⌉, or ceiling of x,
rounds x up to the nearest integer.

11

as shown in figure 19.4. Example 19.4 demonstrates this process.
We compute the solution associated with these two LPs, which provide lower
bounds on the value of the original mixed integer program. If either solution
lowers the objective value when compared to the best integral solution seen sofar, it is placed into the priority queue. Not placing solutions already known to
be inferior to the best integral solution seen thus far allows branch and bound toprune the search space. The process continues until the priority queue is empty,
and we return the best feasible integral solution. Example 19.5 shows how branch
and bound can be applied to a small integer program.
Consider a relaxed solution x∗c = [3, 2.4, 1.2, 5.8] for an integer program withc = [−1, −2, −3, −4]. The lower bound is

Example 19.4. An example of asingle application of the branching
step in branch and bound.

y ≥ c⊤ x∗c = −34.6
We branch on a nonintegral coordinate of x∗c , typically the one farthest from∗ ,an integer value. In this case, we choose the first nonintegral coordinate, x2,cwhich is 0.4 from the nearest integer value. We then consider two new LPs,
one with x2 ≤ 2 as an additional constraint and the other with x2 ≥ 3 as anadditional constraint.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

19. 4 . bra n c h a n d bou n d 349

x∗cFigure 19.4. Branching splits thefeasible set into subsets with anadditional integral inequality constraint.

x2

x1

x1 ≥ 2x1 ≤ 1x∗c,left
x2

x2

x1

x∗c,right

x1© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

350 c ha p te r 19. d i sc re te op ti mi z a ti on

function minimize_lp_and_y(LP)tryx = minimize_lp(LP)return (x, x⋅LP.c)
catch
return (fill(NaN, length(LP.c)), Inf)
end
end
function branch_and_bound(MIP)
LP = relax(MIP)x, y = minimize_lp_and_y(LP)n = length(x)x_best, y_best, Q = deepcopy(x), Inf, PriorityQueue()enqueue!(Q, (LP,x,y), y)while !isempty(Q)LP, x, y = dequeue!(Q)if any(isnan.(x)) || all(isint(x[i]) for i in MIP.D)
if y < y_best
x_best, y_best = x[1:n], yendelse
i = argmax([abs(x[i] - round(x[i])) for i in MIP.D])# x_i ≤ floor(x_i)
A, b, c = LP.A, LP.b, LP.c
A2 = [A zeros(size(A,1));[j==i for j in 1:size(A,2)]' 1]
b2, c2 = vcat(b, floor(x[i])), vcat(c, 0)
LP2 = LinearProgram(A2,b2,c2)
x2, y2 = minimize_lp_and_y(LP2)
if y2 ≤ y_best
enqueue!(Q, (LP2,x2,y2), y2)
end# x_i ≥ ceil(x_i)
A2 = [A zeros(size(A,1));[j==i for j in 1:size(A,2)]' -1]
b2, c2 = vcat(b, ceil(x[i])), vcat(c, 0)
LP2 = LinearProgram(A2,b2,c2)
x2, y2 = minimize_lp_and_y(LP2)
if y2 ≤ y_best
enqueue!(Q, (LP2,x2,y2), y2)
end
end
end
return x_best
end

Algorithm 19.5.
The branch
and
bound
algorithm
forsolving a mixted integer program MIP. The helper methodminimize_lp_and_y solves an LP
and returns both the solution
and its value. An infeasible LP
produces a NaN solution and an
Inf value. More sophisticatedimplementations will drop variables whose solutions are known
in order to speed computation.
The PriorityQueue type is provided by the DataStructures.jl
package.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

19. 5 . d y n a mi c p rog ra mmi n g 351

We can use branch and bound to solve the integer program in example 19.3.
As before, the relaxed solution is x∗c = [0.818, 0, 2.09], with a value of 7.909.
We branch on the first component, resulting in two integer programs, onewith x1 ≤ 0 and one with x1 ≥ 1:
 




2
2.5
0.5 −0.5
1
0
1

 



cleft =  bleft = −1.5
Aleft =  2
0.5 −1.5 0
3
0
1
0
0
1
0


0.5

Aright =  2
1





−0.5
1
0

0.5 −1.5 0 
0
0
−1



2.5


bright = −1.5
1

 
2
1
 
cright =  30The left LP with x1 ≤ 0 is infeasible. The right LP with x1 ≥ 1 has arelaxed solution, x∗c = [1, 2, 3, 0], and a value of 13. We have thus obtained
our integral solution, xi∗ = [1, 2, 3].

19.5

Dynamic Programming

Dynamic programming12 is a technique that can be applied to problems with optimal
substructure and overlapping subproblems. A problem has optimal substructure if
an optimal solution can be constructed from optimal solutions of its subproblems.
Figure 19.5 shows an example.

ab

Example 19.5. Appying branch andbound to solve an integer programming problem.

c

12
The term dynamic programming
was chosen by Richard Bellman toreflect the time-varying aspect ofthe problems he applied it to andto avoid the sometimes negative
connotations words like research
and mathematics had. He wrote,‘‘I thought dynamic programming
was a good name. It was something
not even a Congressman could object to. So I used it as an umbrella
for my activities.’’ R. Bellman, Eyeof the Hurricane: An Autobiography.
World Scientific, 1984. p. 159.

Figure 19.5. Shortest path problems have optimal substructure because if the shortest path from any
a to c passes through b, then the
subpaths a → b and b → c are both
shortest paths.

A problem with overlapping subproblems solved recursively will encounterthe same subproblem many times. Instead of enumerating exponentially many potential solutions, dynamic programming either stores subproblem solutions, andthereby avoids having to recompute them, or recursively builds the optimal solu© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

352 c ha p te r 19. d i sc re te op ti mi z a ti ontion in a single pass. Problems with recurrence relations often have overlapping
subproblems. Figure 19.6 shows an example.
P9
P9

P4

P5
P3
P1

P2 P2

P7

P6

P7
P4
P1 P2

P4

P5

P3
P1 P1

P6

P0

P3

P2

Figure 19.6. We can computethe nth term of the Padovan
sequence, Pn = Pn−2 + Pn−3 , with
P0 = P1 = P2 = 1 by recursing
through all subterms (left). A
more efficient approach is tocompute subterms once and reuse
their values in subsequent calculations by exploiting the problem’s
overlapping substructure (right).

P0
P0

P1

Dynamic programming can be implemented either top-down or bottom-up, as
demonstrated in algorithm 19.6. The top-down approach begins with the desired
problem and recurses down to smaller and smaller subproblems. Subproblem
solutions are stored so that when we are given a new subproblem, we can either
retrieve the computed solution or solve and store it for future use.13 The bottomup approach starts by solving the smaller subproblems and uses their solutions
to obtain solutions to larger problems.
function padovan_topdown(n, P=Dict())
if !haskey(P, n)P[n] = n < 3 ? 1 :padovan_topdown(n-2,P) + padovan_topdown(n-3,P)
end
return P[n]endfunction padovan_bottomup(n)P = Dict(0=>1,1=>1,2=>1)
for i in 3 : n
P[i] = P[i-2] + P[i-3]end
return P[n]end

Storing subproblem solutions inthis manner is called memoization.

13

Algorithm 19.6. Computing the
Padovan sequence using dynamic
programming, with both the topdown and bottom-up approaches.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

19. 5 . d y n a mi c p rog ra mmi n g 353

The knapsack problem is a well-known combinatorial optimization problem that
often arises in resource allocation.14 Suppose we are packing our knapsack for atrip, but we have limited space and want to pack the most valuable items. There
are several variations of the knapsack problem. In the 0-1 knapsack problem, we
have the following optimization problem:

The knapsack problem is an integer program with a single constraint, but it can be efficiently
solved using dynamic programming.

14n− ∑ vi xi

minimize
x

i =1

n

subject to∑ wi xi ≤ wmax

(19.9)

i =1

xi ∈ {0, 1} for all i in {1, . . . , n}

We have n items, with the ith item having integral weight wi > 0 and value vi .
The design vector x consists of binary values that indicate whether an item ispacked. The total weight cannot exceed our integral capacity wmax , and we seek
to maximize the total value of packed items.
There are 2n possible design vectors, which makes direct enumeration for large
n intractable. However, we can use dynamic programming. The 0-1 knapsackproblem has optimal substructure and overlapping subproblems. Consider having
solved knapsack problems with n items and several capacities up to wmax . The
solution to a larger knapsack problem with one additional item with weight wn+1
and capacity wmax will either include or not include the new item:• If it is not worth including the new item, the solution will have the same value
as a knapsack with n − 1 items and capacity wmax .
• If it is worth including the new item, the solution will have the value of aknapsack with n − 1 items and capacity wmax − wn+1 plus the value of the
new item.
The recurrence relation is:


0




knapsack(i − 1, wmax )knapsack(i, wmax ) =knapsack(i − 1, w


max )


max


knapsack(i − 1, wmax − wi ) + vi

if i = 0
if wi > wmax
(discard new item)(include new item)(19.10)otherwise© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

354 c ha p te r 19. d i sc re te op ti mi z a ti on

The 0-1 knapsack problem can be solved using the implementation in algorithm 19.7.
function knapsack(v, w, w_max)
n = length(v)y = Dict((0,j) => 0.0 for j in 0:w_max)
for i in 1 : n
for j in 0 : w_max
y[i,j] = w[i] > j ? y[i-1,j] :max(y[i-1,j],y[i-1,j-w[i]] + v[i])
end
end

end

Algorithm 19.7. A method for solving the 0-1 knapsack problem withitem values v, integral item weights
w, and integral capacity w_max. Recovering the design vector fromthe cached solutions requires additional iteration.

# recover solution
x, j = falses(n), w_max
for i in n: -1 : 1
if w[i] ≤ j && y[i,j] - y[i-1, j-w[i]] == v[i]# the ith element is in the knapsackx[i] = true
j -= w[i]endendreturn x19.6 Ant Colony Optimization
Ant colony optimization15 is a stochastic method for optimizing paths throughgraphs. This method was inspired by some ant species that wander randomly
in search of food, leaving pheromone trails as they go. Other ants that stumble
upon a pheromone trail are likely to start following it, thereby reinforcing thetrail’s scent. Pheromones slowly evaporate over time, causing unused trails to
fade. Short paths, with stronger pheromones, are traveled more often and thus
attract more ants. Thus, short paths create positive feedback that lead other ants
to follow and further reinforce the shorter path.
Basic shortest path problems, such as the shortest paths found by ants between
the ant hill and sources of food, can be efficiently solved using dynamic pro-

M. Dorigo, V. Maniezzo, and A.
Colorni, “Ant System: Optimization by a Colony of Cooperating
Agents,” IEEE Transactions on Systems, Man, and Cybernetics, Part B(Cybernetics), vol. 26, no. 1, pp. 29–
41, 1996.
15© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

19. 6 . a n t c ol on y op ti mi z a ti on 355

gramming. Ant colony optimization has been used to find near-optimal solutions
to the traveling salesman problem, a much more difficult problem in which wewant to find the shortest path that passes through each node exactly once. Ant
colony optimization has also been used to route multiple vehicles, find optimal
locations for factories, and fold proteins.16 The algorithm is stochastic in natureand is thus resistant to changes to the problem over time, such as traffic delays
changing effective edge lengths in the graph or networking issues that remove
edges entirely.
Ants move stochastically based on the attractiveness of the edges available tothem. The attractiveness of transition i → j depends on the pheromone level andan optional prior factor:
A (i → j ) = τ (i → j )α η (i → j ) β(19.11)

where α and β are exponents for the pheromone level τ and prior factor η, respectively.17 For problems involving shortest paths, we can set the prior factor
to the inverse edge length ℓ(i → j) to encourage the traversal of shorter paths:
η (i → j) = 1/ℓ(i → j). A method for computing the edge attractiveness is given
in algorithm 19.8.
Suppose an ant is at node i and can transition to any of the nodes j ∈ J . The setof successor nodes J contains all valid outgoing neighbors.18 Sometimes edges
are excluded, such as in the traveling salesman problem where ants are prevented
from visiting the same node twice. It follows that J is dependent on both i andthe ant’s history.
function edge_attractiveness(graph, τ, η; α=1, β=5)A = Dict()
for i in 1 : nv(graph)neighbors = outneighbors(graph, i)
for j in neighborsv = τ[(i,j)]^α * η[(i,j)]^β
A[(i,j)] = v
end
end
return A
end

These and other applications
are discussed in these references:M. Manfrin, “Ant Colony Optimization for the Vehicle Routing
Problem,” Ph.D. dissertation,
Université Libre de Bruxelles,
2004. T. Stützle, “MAX-MIN Ant
System for Quadratic Assignment
Problems,” Technical University
Darmstadt, Tech. Rep., 1997.
and A. Shmygelska, R. AguirreHernández, and H. H. Hoos, “AnAnt Colony Algorithm for the 2D
HP Protein Folding Problem,”in International Workshop on Ant
Algorithms (ANTS), 2002.
16

Dorigo, Maniezzo, and Colorni
recommend α = 1 and β = 5.

17

The outgoing neighbors of a node
i are all nodes j such that i → j
is in the graph. In an undirected
graph, the neighbors and the outgoing neighbors are identical.

18

Algorithm 19.8. A method for computing the edge attractiveness table given graph graph, pheromone
levels τ, prior edge weights η,pheromone exponent α, and priorexponent β.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

356 c ha p te r 19. d i sc re te op ti mi z a ti on

The probability of edge transition i → j is:
P (i → j ) =A (i → j )∑ j′ ∈J A(i → j′ )(19.12)

Ants affect subsequent generations of ants by depositing pheromones. There
are several methods for modeling pheromone deposition. A common approach is
to deposit pheromones establishing after a complete path.19 Ants that do not finda path do not deposit pheromones. For shortest path problems, a successful ant
that has established a path of length ℓ deposits 1/ℓ pheromones on each edge it
traversed.
import StatsBase: Weights, sample
function run_ant(G, lengths, τ, A, x_best, y_best)
x = [1]
while length(x) < nv(G)i = x[end]neighbors = setdiff(outneighbors(G, i), x)
if isempty(neighbors) # ant got stuck
return (x_best, y_best)
end

end

end

as = [A[(i,j)] for j in neighbors]
push!(x, neighbors[sample(Weights(as))])M. Dorigo, G. Di Caro, and L. M.
Gambardella, “Ant Algorithms for
Discrete Optimization,” Artificial
Life, vol. 5, no. 2, pp. 137–172, 1999.
19

Algorithm 19.9. A method for simulating a single ant on a travelingsalesman problem in which the ant
starts at the first node and attempts
to visit each node exactly once.
Pheromone levels are increased at
the end of a successful tour. The
parameters are the graph G, edge
lengths lengths, pheromone levels
τ, edge attractiveness A, the best solution found thus far x_best, andits value y_best.

l = sum(lengths[(x[i-1],x[i])] for i in 2:length(x))
for i in 2 : length(x)τ[(x[i-1],x[i])] += 1/l
end
if l < y_best
return (x, l)
else
return (x_best, y_best)end

Ant colony optimization also models pheromone evaporation, which naturally
occurs in the real world. Modeling evaporation helps prevent the algorithm fromprematurely converging to a single, potentially suboptimal, solution. Pheromone
evaporation is executed at the end of each iteration after all ant simulations have© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

19. 6 . a n t c ol on y op ti mi z a ti on 357

been completed. Evaporation decreases the pheromone level of each transition
by a factor of 1 − ρ, with ρ ∈ [0, 1].20
For m ants at iteration k, the effective pheromone update ismτ ( i → j ) ( k +1) = (1 − ρ ) τ ( i → j ) ( k ) + ∑a =1 ℓ1 ( a)(i → j ) ∈ P ( a)20

It is common to use ρ = 1/2.

(19.13)where ℓ( a) is the path length and P ( a) is the set of edges traversed by ant a.
Ant colony optimization is implemented in algorithm 19.10, with individual ant
simulations using algorithm 19.9. Figure 19.7 visualizes ant colony optimization
used to solve a traveling salesman problem.
function ant_colony_optimization(G, lengths;
m = 1000, k_max=100, α=1.0, β=5.0, ρ=0.5,
η = Dict((e.src,e.dst)=>1/lengths[(e.src,e.dst)]
for e in edges(G)))τ = Dict((e.src,e.dst)=>1.0 for e in edges(G))
x_best, y_best = [], Inf
for k in 1 : k_max
A = edge_attractiveness(G, τ, η, α=α, β=β)for (e,v) in ττ[e] = (1-ρ)*v
end
for ant in 1 : m
x_best,y_best = run_ant(G,lengths,τ,A,x_best,y_best)
end
end
return x_best
end

Algorithm 19.10.
Ant colony
optimization, which takes a directed or undirected graph G from
LightGraphs.jl and a dictionaryof edge tuples to path lengths
lengths. Ants start at the first nodein the graph. Optional parameters
include the number of ants per iteration m, the number of iterations
k_max, the pheromone exponent α,
the prior exponent β, the evaporation scalar ρ, and a dictionary ofprior edge weights η.

Figure 19.7. Ant colony optimization used to solve a travelingsalesman problem on a directed
graph using 50 ants per iteration.
Path lengths are the Euclidean distances. Color opacity corresponds
to pheromone level.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

358 c ha p te r 19. d i sc re te op ti mi z a ti on

19.7 Summary• Discrete optimization problems require that the design variables be chosenfrom discrete sets.
• Relaxation, in which the continuous version of the discrete problem is solved,
is by itself an unreliable technique for finding an optimal discrete solution but
is central to more sophisticated algorithms.
• Many combinatorial optimization problems can be framed as an integer program, which is a linear program with integer constraints.
• Both the cutting plane and branch and bound methods can be used to solve
integer programs efficiently and exactly. The branch and bound method isquite general and can be applied to a wide variety of discrete optimization
problems.
• Dynamic programming is a powerful technique that exploits optimal overlapping substructure in some problems.
• Ant colony optimization is a nature-inspired algorithm that can be used for
optimizing paths in graphs.

19.8 Exercises
Exercise 19.1. A Boolean satisfiability problem, often abbreviated SAT, requires determining whether a Boolean design exists that causes a Boolean-valued objectivefunction to output true. SAT problems were the first to be proven to belong to
the difficult class of NP-complete problems.21 This means that SAT is at least as
difficult as all other problems whose solutions can be verified in polynomial time.
Consider the Boolean objective function:

S. Cook, “The Complexity ofTheorem-Proving Procedures,” in
ACM Symposium on Theory of Computing, 1971.
21f (x) = x1 ∧ ( x2 ∨ ¬ x3 ) ∧ (¬ x1 ∨ ¬ x2 )Find an optimal design using enumeration. How many designs must be considered for an n-dimensional design vector in the worst case?
Exercise 19.2. Formulate the problem in exercise 19.1 as an integer linear program. Can any Boolean satisfiability problem be formulated as an integer linear
program?
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

19. 8 . e x e rc i se s 359

Exercise 19.3. Why are we interested in totally unimodular matrices? Furthermore,
why does every totally unimodular matrix contain only entries that are 0, 1, or−1?
Exercise 19.4. This chapter solved the 0-1 knapsack problem using dynamic
programming. Show how to apply branch and bound to the 0-1 knapsack problem, and use your approach to solve the knapsack problem with values v =[9, 4, 2, 3, 5, 3], and weights w = [7, 8, 4, 5, 9, 4] with capacity wmax = 20.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 Expression Optimization
Previous chapters discussed optimization over a fixed set of design variables. For
many problems, the number of variables is unknown, such as in the optimizationof graphical structures or computer programs. Designs in these contexts can be
represented by expressions that belong to a grammar. This chapter discusses
ways to make the search of optimal designs more efficient by accounting for thegrammatical structure of the design space.

20.1 Grammars
An expression can be represented by a tree of symbols. For example, the mathematical expression x + ln 2 can be represented using the tree in figure 20.1 consisting of
the symbols +, x, ln, and 2. Grammars specify constraints on the space of possible
expressions.
A grammar is represented by a set of production rules. These rules involvesymbols as well as types. A type can be interpreted as a set of expression trees. A
production rule represents a possible expansion of type to an expression involving
symbols or types. If a rule expands only to symbols, then it is called terminal
because it cannot be expanded further. An example of a nonterminal rule is
R 7→ R + R, which means that the type R can consist of elements of the set R
added to elements in the set R.1
We can generate an expression from a grammar by starting with a start type
and then recursively applying different production rules. We stop when thetree contains only symbols. Figure 20.2 illustrates this process for the expression
x + ln 2. An application to natural language expressions is shown in example 20.1.
The number of possible expressions allowed by a grammar can be infinite.
Example 20.2 shows a grammar that allows for infinitely many valid expressions.

+
x

ln

2
Figure 20.1. The expression x +ln 2 represented as a tree.

This chapter focuses on contextfree grammars, but other forms exist.
See L. Kallmeyer, Parsing Beyond
Context-Free Grammars. Springer,2010.
1362 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

+

R
R

+
x

R

Figure 20.2. Using the production
rules
R 7→ R + R

+
ln

x

R 7→ xln

R 7→ ln(R )R

2Consider a grammar that allows for the generation of simple English statements:
S 7→ N V
V 7→ V A

A 7→ rapidly | efficiently

N 7→ Alice | Bob | Mykel | Tim
V 7→ runs | reads | writes

R 7→ 2

to generate x + ln 2. Blue nodes are
unexpanded types.

Example 20.1. A grammar for producing simple English statements.
Using | on the right-hand side ofan expression is shorthand for or.
Thus, the rule
A 7→ rapidly | efficiently

is equivalent to having two rules,
A 7→ rapidly and A 7→ efficiently.

The types S, N, V, and A correspond to statements, nouns, verbs, andadverbs, respectively. An expression is generated by starting with the type S
and iteratively replacing types:
S
NV
Mykel V A
Mykel writes rapidly
Not all terminal symbol categories must be used. For instance, the statement‘‘Alice runs’’ can also be generated.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 1. g ra mma rs

363

Expression optimization often constrains expressions to a maximum depth or
penalizes expressions based on their depth or node count. Even if the grammar
allows a finite number of expressions, the space is often too vast to search exhaustively. Hence, there is a need for algorithms that efficiently search the space ofpossible expressions for one that optimizes an objective function.
Consider a four-function calculator grammar that applies addition, subtraction, multiplication, and division to the ten digits:
R 7→ R + R

Example 20.2. Some of the challenges associated with grammars,
as illustrated with a four-function
calculator grammar.

R 7→ R − R
R 7→ R × R
R 7→ R / R

R 7→ 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
An infinite number of expressions can be generated because the nonterminal
R can always be expanded into one of the calculator operations.
Many expressions will produce the same value. Addition and multiplication operators are commutative, meaning that the order does not matter.
For example, a + b is the same as b + a. These operations are also associative,
meaning the order in which multiple operations of the same type occur do
not matter. For example, a × b × c is the same as c × b × a. Other operations
preserve values, like adding zero or multiplying by one.
Not all expressions under this grammar are mathematically valid. For
example, division by zero is undefined. Removing zero as a terminal symbol
is insufficient to prevent this error because zero can be constructed using
other operations, such as 1 − 1. Such exceptions are often handled by theobjective function, which can catch exceptions and penalize them.
The expression optimization routines covered in the chapter use ExprRules.jl.
Grammars can be defined using the grammar macro by listing the production
rules, as shown in example 20.3.
Many of the expression optimization algorithms involve manipulating components of an expression tree in a way that preserves the way the types wereexpanded. A RuleNode object tracks which production rules were applied when
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

364 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

We may define a grammar using the grammar macro. The nonterminals areon the left of the equal sign, and the expressions with terminals and nonterminals are the on the right. The package includes some syntax to represent
grammars more compactly.

Example 20.3. Example ofdefining a grammar using the
ExprRules.jl package.

using ExprRules
grammar = @grammar begin
R = x# reference a variable
R = R * A# multiple children
R = f(R)# call a function
R = _(randn()) # random variable generated on node creation
R = 1 | 2 | 3 # equivalent to R = 1, R = 2, and R = 3
R = |(4:6)# equivalent to R = 4, R = 5, and R = 6
A = 7# rules for different return types
end;doing an expansion. Calling rand with a specified starting type will generate
a random expression represented by a RuleNode. Calling sample will select arandom RuleNode from an existing RuleNode tree. Nodes are evaluated usingeval.
The method return_type returns the node’s return type as a symbol, isterminal
returns whether the symbol is terminal, child_types returns the list of nonterminal symbols associated with the node’s production rule, and nchildren returns
the number of children. These four methods each take as input the grammar
and the node. The number of nodes in an expression tree is obtained using
length(node), and the depth is obtained using depth(node).
A third type, NodeLoc, is used to refer to a node’s location in the expression
tree. Subtrees manipulation often requires NodeLocs.
loc = sample(NodeLoc, node); # uniformly sample a node loc
loc = sample(NodeLoc, node, :R, grammar); # sample a node loc of type R
subtree = get(node, loc);

20.2 Genetic Programming
Genetic algorithms (see chapter 9) use chromosomes that encode design points
in a sequential format. Genetic programming2 represents individuals using trees

2
J. R. Koza, Genetic Programming:
On the Programming of Computers
by Means of Natural Selection. MIT
Press, 1992.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 2. g e n e ti c p rog ra mmi n g 365

instead (figure 20.3), which are better at representing mathematical functions,programs, decision trees, and other hierarchical structures.
Figure 20.3. A tree representationof the Julia method:

if

×

≤
x

2

x ≤ 2 ? x*(y+3) : 0

0

+

x
y

3

Similar to genetic algorithms, genetic programs are initialized randomly andsupport crossover and mutation. In tree crossover (figure 20.4), two parent trees
are mixed to form a child tree. A random node is chosen in each parent, and thesubtree at the chosen node in the first parent is replaced with the subtree at thechosen node of the second parent. Tree crossover works on parents with different
sizes and shapes, allowing arbitrary trees to mix. In some cases one must ensurethat replacement nodes have certain types, such as Boolean values input into thecondition of an if statement.3 Tree crossover is implemented in algorithm 20.1.

This book focuses only on genetic operations that adhere to
the constraints of the grammar.
Sometimes genetic programmingwith this restriction is referred toas strongly typed genetic programming, as discussed in D. J. Montana, “Strongly Typed Genetic Programming,” Evolutionary Computation, vol. 3, no. 2, pp. 199–230, 1995.
3

Figure 20.4. Tree crossover is used
to combine two parent trees to produce a child tree.

Parent A

Parent B

Child

Tree crossover tends to produce trees with greater depth than the parent trees.
Each generation tends to increase in complexity, which often results in overly
complicated solutions and slower runtimes. We encourage parsimony, or simplicity,in the solution, by introducing a small bias in the objective function value basedon a tree’s depth or node count.
Applying tree mutation (figure 20.5) starts by choosing a random node in thetree. The subtree rooted at that node is deleted, and a new random subtree isgenerated to replace the old subtree. In contrast to mutation in binary chromo© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

366 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

struct TreeCrossover <: CrossoverMethod
grammar
max_depthendfunction crossover(C::TreeCrossover, a, b)
child = deepcopy(a)crosspoint = sample(b)typ = return_type(C.grammar, crosspoint.ind)
d_subtree = depth(crosspoint)
d_max = C.max_depth + 1 - d_subtree
if d_max > 0 && contains_returntype(child,C.grammar,typ,d_max)
loc = sample(NodeLoc, child, typ, C.grammar, d_max)
insert!(child, loc, deepcopy(crosspoint))
end
child
end

Algorithm 20.1. Tree crossover
implemented for a and b of type
RuleNode from ExprRules.jl. The
TreeCrossover struct contains arule set grammar and a maximum
depth max_depth.

somes, tree mutation can typically occur at most once, often with a low probability
around 1%. Tree mutation is implemented in algorithm 20.2.
Figure 20.5. Tree mutation deletes
a random subtree and generates a
new one to replace it.

BeforeAfter© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 2. g e n e ti c p rog ra mmi n g 367

struct TreeMutation <: MutationMethod
grammarpendfunction mutate(M::TreeMutation, a)
child = deepcopy(a)if rand() < M.p
loc = sample(NodeLoc, child)
typ = return_type(M.grammar, get(child, loc).ind)
subtree = rand(RuleNode, M.grammar, typ)insert!(child, loc, subtree)
end
return child
end

Algorithm 20.2. Tree mutation implemented for an individual a oftype RuleNode from ExprRules.jl.
The TreeMutation struct contains
a rule set grammar and a mutation
probability p.

Tree permutation (figure 20.6) is a second form of genetic mutation. Here, thechildren of a randomly chosen node are randomly permuted. Tree permutation
alone is typically not sufficient to introduce new genetic material into a populationand is often combined with tree mutation. Tree permutation is implemented inalgorithm 20.3.
Figure 20.6. Tree permutation permutes the children of a randomly
chosen node.

BeforeAfter

The implementation of genetic programming is otherwise identical to thatof genetic algorithms. More care must typically be taken in implementing thecrossover and mutation routines, particularly when determining what sorts ofnodes can be generated and that only syntactically correct trees are produced.
Genetic programming is used to generate an expression that approximates π in
example 20.4.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

368 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

struct TreePermutation <: MutationMethod
grammarpendfunction mutate(M::TreePermutation, a)
child = deepcopy(a)if rand() < M.p
node = sample(child)n = length(node.children)
types = child_types(M.grammar, node)
for i in 1 : n-1
c = 1
for k in i+1 : n
if types[k] == types[i] &&rand() < 1/(c+=1)

end

end

end

Algorithm 20.3. Tree permutation
implemented for an individual a oftype RuleNode from ExprRules.jl,
where p is the mutation probability.

node.children[i], node.children[k] =node.children[k], node.children[i]endend
return child© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 2. g e n e ti c p rog ra mmi n g 369

Consider approximating π using only operations on a four-function calculator. We can solve this problem using genetic programming where nodes canbe any of the elementary operations: add, subtract, multiply, and divide, andthe digits 1 − 9.
We use ExprRules.jl to specify our grammar:
grammar = @grammar begin
R = |(1:9)
R = R + R
R = R - R
R = R / R
R = R * Rend

We construct an objective function and penalize large trees:
function f(node)
value = eval(node, grammar)
if isinf(value) || isnan(value)
return Inf
end
Δ = abs(value - π)return log(Δ) + length(node)/1e3
end

Example 20.4. Using genetic programming to estimate π using only
digits and the four principle arithmetic operations.
/
+

4
+

3
/

9
+

7

+

7
/

5
+

3
/
3

8
9

We finally run our genetic program using a call to the genetic_algorithm
function from section 9.2:
population = [rand(RuleNode, grammar, :R) for i in 1:1000]
best_tree = genetic_algorithm(f, population, 30,
TruncationSelection(50),
TreeCrossover(grammar, 10),
TreeMutation(grammar, 0.25))

The best performing tree is shown on the right. It evaluates to 3.141586, whichmatches π to four decimal places.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

370 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

20.3

Grammatical Evolution

Grammatical evolution4 operates on an integer array instead of a tree, allowing thesame techniques employed in genetic algorithms to be applied. Unlike genetic
algorithms, the chromosomes in grammatical evolution encode expressions basedon a grammar. Grammatical evolution was inspired by genetic material, which isinherently serial like the chromosomes used in genetic algorithms. 5
In grammatical evolution, designs are integer arrays much like the chromosomes used in genetic algorithms. Each integer is unbounded because indexing
is performed using modular arithmetic. The integer array can be translated into
an expression tree by parsing it from left to right.
We begin with a starting symbol and a grammar. Suppose n rules in thegrammar can be applied to the starting symbol. The jth rule is applied, wherej = i mod1 n and i is the first integer in the integer array.6
We then consider the rules applicable to the resulting expression and use
similar modular arithmetic based on the second integer in the array to selectwhich rule to apply. This process is repeated until no rules can be applied and thephenotype is complete.7 The decoding process is implemented in algorithm 20.4
and is worked through in example 20.5.
It is possible for the integer array to be too short, thereby causing the translation
process to run past the length of the array. Rather than producing an invalid
individual and penalizing it in the objective function, the process wraps aroundto the beginning of the array instead. This wrap-around effect means that the samedecision can be read several times during the transcription process. Transcription
can result in infinite loops, which can be prevented by a maximum depth.
Genetic operations work directly on the integer design array. We can adopt
the operations used on real-valued chromosomes and apply them to the integervalued chromosomes. The only change is that mutation must preserve real values.
A mutation method for integer-valued chromosomes using zero-mean Gaussian
perturbations is implemented in algorithm 20.5.
Grammatical evolution uses two additional genetic operators. The first, gene
duplication, occurs naturally as an error in DNA replication and repair. Geneduplication can allow new genetic material to be generated and can store a second
copy of a useful gene to reduce the chance of a lethal mutation removing the genefrom the gene pool. Gene duplication chooses a random interval of genes in the

C. Ryan, J. J. Collins, and M. O.
Neill, “Grammatical Evolution:
Evolving Programs for an Arbitrary Language,” in European
Conference on Genetic Programming,
1998.
5
Our serial DNA is read and used
to construct complicated protein
structures. DNA is often referred
to as the genotype—the object
on which genetic operations are
performed. The protein structureis the phenotype—the object encoded by the genotype whose performance is evaluated. The grammatical evolution literature often
refers to the integer design vector
as the genotype and the resulting
expression as the phenotype.
4

We use x mod1 n to refer to the 1index modulus:

6(( x − 1) mod n) + 1

This type of modulus is useful with
1-based indexing. The corresponding Julia function is mod1.
7
No genetic information is readwhen there is only a single applicable rule.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 3. g ra mma ti c a l e vol u ti on

struct DecodedExpression
node
n_rules_applied
end
function decode(x, grammar, sym, c_max=1000, c=0)
node, c = _decode(x, grammar, sym, c_max, c)
DecodedExpression(node, c)endfunction _decode(x, grammar, typ, c_max, c)
types = grammar[typ]if length(types) > 1
g = x[mod1(c+=1, length(x))]
rule = types[mod1(g, length(types))]else
rule = types[1]endnode = RuleNode(rule)childtypes = child_types(grammar, node)
if !isempty(childtypes) && c < c_max
for ctyp in childtypes
cnode, c = _decode(x, grammar, ctyp, c_max, c)
push!(node.children, cnode)end
end
return (node, c)
end

371

Algorithm 20.4. A method for decoding an integer design vector
to produce an expression, where
x is a vector of integers, grammar
is a Grammar, and sym is the root
symbol. The counter c is usedduring the recursion process andthe parameter c_max is an upper
limit on the maximum number ofrule applications, to prevent an infinite loop. The method returns
a DecodedExpression, which contains the expression tree and thenumber of rules applied during thedecoding process.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

372 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

Consider a grammar for real-valued strings:
R 7→ D D ′ P E

D ′ 7→ D D ′ | ǫ

P 7→ . D D ′ | ǫ

E 7→ E S D D ′ | ǫ
S 7→ + | − | ǫ

Example 20.5. The process bywhich an integer design vector ingrammatical evolution is decoded
into an expression.
Our implementation is depthfirst. If 52 were instead 51, the ruleD ′ 7→ D D ′ would be applied,followed by selecting a rule forthe new D, eventually resulting in
43950.950E+8.

D 7→ 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
where R is a real value, D is a terminal decimal, D ′ is a nonterminal decimal,
P is the decimal part, E is the exponent, and S is the sign. Any ǫ values
produce empty strings.
Suppose our design is [205, 52, 4, 27, 10, 59, 6] and we have the starting
symbol R. There is only one applicable rule, so we do not use any genetic
information and we replace R with DD ′ PE.
Next we must replace D. There are 10 options. We select 205 mod1 10 = 5,
and thus obtain 4D ′ PE
Next we replace D ′ , which has two options. We select index 52 mod1 2 = 2,
which corresponds to ǫ.
Continuing in this manner we produce the string 4E+8.
The above grammar can be implemented in ExprRules using:grammar = @grammar begin
R = D * De * P * E
De = D * De | ""P = "." * D * De | ""E = "E" * S * D * De | ""S = "+"|"-"|""
D = "0"|"1"|"2"|"3"|"4"|"5"|"6"|"7"|"8"|"9"end

and can be evaluated using:x = [205, 52, 4, 27, 10, 59, 6]
str = eval(decode(x, grammar, :R).node, grammar)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 3. g ra mma ti c a l e vol u ti on

struct IntegerGaussianMutation <: MutationMethodσendfunction mutate(M::IntegerGaussianMutation, child)return child + round.(Int, randn(length(child)).*M.σ)
end

373

Algorithm 20.5. The Gaussian mutation method modified to preserve integer values for integervalued chromosomes. Each value
is perturbed by a zero-mean Gaussian random value with standard
deviation σ and then rounded to
the nearest integer.

chromosome to duplicate. A copy of the selected interval is appended to the backof the chromosome. Duplication is implemented in algorithm 20.6.
struct GeneDuplication <: MutationMethod
endfunction mutate(M::GeneDuplication, child)n = length(child)i, j = rand(1:n), rand(1:n)
interval = min(i,j) : max(i,j)
return vcat(child, deepcopy(child[interval]))end

The second genetic operation, pruning, tackles a problem encountered duringcrossover. As illustrated in figure 20.7, crossover will select a crossover point at
random in each chromosome and construct a new chromosome using the left
side of the first and the right side of the second chromosome. Unlike genetic
algorithms, the trailing entries in chromosomes of grammatical evolution may
not be used; during parsing, once the tree is complete, the remaining entries
are ignored. The more unused entries, the more likely it is that the crossover
point lies in the inactive region, thus not providing new beneficial material. An
individual is pruned with a specified probability, and, if pruned, its chromosome
is truncated such that only active genes are retained. Pruning is implemented inalgorithm 20.7 and is visualized in figure 20.8.
Like genetic programming, grammatical evolution can use the genetic algorithm method.8 We can construct a MutationMethod that applies multiple mutation methods in order to use pruning, duplication, and standard mutation
approaches in tandem. Such a method is implemented in algorithm 20.8.
Grammatical evolution suffers from two primary drawbacks. First, it is difficult
to tell whether the chromosome is feasible without decoding it into an expression.

Algorithm 20.6. The gene duplication method used in grammaticalevolution.

8
Genotype to phenotype mapping
would occur in the objective function.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

374 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

parent aparent b
child

struct GenePruning <: MutationMethodpgrammartypendfunction mutate(M::GenePruning, child)
if rand() < M.p
c = decode(child, M.grammar, M.typ).n_rules_applied
if c < length(child)child = child[1:c]end
end
return childend

beforeafter

struct MultiMutate <: MutationMethodMs
endfunction mutate(M::MultiMutate, child)for m in M.Ms
child = mutate(m, child)end
return childend

Figure 20.7. Crossover applied tochromosomes in grammatical evolution may not affect the active
genes in the front of the chromosome. The child shown here inherits all of the active shaded genes
from parent a so it will effectively
act as an identical expression. Pruning was developed to overcome
this issue.
Algorithm 20.7. The gene pruning
method used in grammatical evolution.

Figure 20.8. Pruning truncates thechromosome such that only active
genes remain.

Algorithm

20.8.
A
for applying
all mutation methods stored in thevector Ms.
MutationMethod© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 4 . p roba bi l i sti c g ra mma rs 375

Second, a small change in the chromosome may produce a large change in thecorresponding expression.

20.4 Probabilistic Grammars
A probabilistic grammar9 adds a weight to each rule in the genetic program’s grammar. When sampling from all applicable rules for a given node, we select a rulestochastically according to the relative weights. The probability of an expression is
the product of the probabilities of sampling each rule. Algorithm 20.9 implements
the probability calculation. Example 20.6 demonstrates sampling an expression
from a probabilistic grammar and computes its likelihood.
struct ProbabilisticGrammar
grammarws
endfunction probability(probgram, node)
typ = return_type(probgram.grammar, node)i = findfirst(isequal(node.ind), probgram.grammar[typ])prob = probgram.ws[typ][i] / sum(probgram.ws[typ])for (i,c) in enumerate(node.children)
prob *= probability(probgram, c)
end
return prob
end

Optimization using a probabilistic grammar improves its weights with each
iteration using elite samples from a population. At each iteration, a population of
expressions is sampled and their objective function values are computed. Some
number of the best expressions are considered the elite samples and can be used
to update the weights. A new set of weights is generated for the probabilistic
grammar, where the weight wiT for the ith production rule applicable to return
type T is set to the number of times the production rule was used in generating
the elite samples. This update procedure is implemented in algorithm 20.10.
The probabilistic grammars above can be extended to more complicated probability distributions that consider other factors, such as the depth in the expression
or local dependencies among siblings in subtrees. One approach is to use Bayesian
networks.10

T. L. Booth and R. A. Thompson,“Applying Probability Measures to
Abstract Languages,” IEEE Transactions on Computers, vol. C-22, no. 5,
pp. 442–450, 1973.
9

Algorithm 20.9. A method for computing the probability of an expression based on a probabilistic grammar, where probgram is a probabilistic grammar consisting of agrammar grammar and a mappingof types to weights for all applicable rules ws, and node is a RuleNode
expression.

P. K. Wong, L. Y. Lo, M. L. Wong,and K. S. Leung, “GrammarBased Genetic Programmingwith Bayesian Network,” inIEEE Congress on Evolutionary
Computation (CEC), 2014.
10© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

376 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

Consider a probabilistic grammar for strings composed entirely of ‘‘a’’s:
A 7→ a A

w1A = 17→ ǫ

w3A = 27→ ǫw2B = 17→ a B a A

B 7→ a B

Example 20.6. Sampling an expression from a probabilistic grammar
and computing the expression’s
likelihood.

w2A = 3
w1B = 4

where we have a set of weights w for each parent type and ǫ is an empty
string.
Suppose we generate an expression starting with the type A. The probability distribution over the three possible rules is:P(A 7→ a A ) = 1/(1 + 3 + 2)

P(A 7→ a B a A ) = 3/(1 + 3 + 2)

P(A 7→ ǫ) = 2/(1 + 3 + 2)

= 1/6
= 1/2
= 1/3

Suppose we sample the second rule and obtain a B a A.
Next we sample a rule to apply to B. The probability distribution over thetwo possible rules is:P(B 7→ a B ) = 4/(4 + 1)

P(B 7→ ǫ) = 1/(4 + 1)

= 4/5
= 1/5

Suppose we sample the second rule and obtain a ǫ a A.
Next we sample a rule to apply to A. Suppose we sample A 7→ ǫ to obtaina ǫ a ǫ, which produces the ‘‘a’’-string ‘‘aa’’. The probability of the sequenceof rules applied to produce ‘‘aa’’ under the probabilistic grammar is:
P (A 7 → a B a A ) P (B 7 → ǫ ) P (A 7 → ǫ ) =

1
1 1 1
· · =
2 5 3
30

Note that this is not the same as the probability of obtaining ‘‘aa’’, as othersequences of production rules could also have produced it.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 5 . p roba bi l i sti c p rototy p e tre e sfunction _update!(probgram, x)
grammar = probgram.grammar
typ = return_type(grammar, x)
i = findfirst(isequal(x.ind), grammar[typ])probgram.ws[typ][i] += 1
for c in x.children_update!(probgram, c)
end
return probgram
end
function update!(probgram, Xs)
for w in values(probgram.ws)
fill!(w,0)
end
for x in Xs_update!(probgram, x)
end
return probgram
end

20.5

377

Algorithm 20.10. A method for applying a learning update to a probabilistic grammar probgram basedon an elite sample of expressions
Xs.

Probabilistic Prototype Trees

The probabilistic prototype tree11 is a different approach that learns a distribution
for every node in the expression tree. Each node in a probabilistic prototype
tree contains a probability vector representing a categorical distribution overthe grammar’s production rules. The probability vectors are updated to reflectknowledge gained from successive generations of expressions. The maximum
number of children for a node is the maximum number of nonterminals among
rules in the grammar.12
Probability vectors are randomly initialized when a node is created. Random
probability vectors can be drawn from a Dirichlet distribution.13 The original implementation initializes terminals to a scalar value of 0.6 and nonterminals to 0.4.
In order to handle strongly-typed grammars we maintain a probability vector for
applicable rules to each parent type. Algorithm 20.11 defines a node type andimplements this initialization method.
Expressions are sampled using the probability vectors in the probabilistic
prototype tree. A rule in a node is drawn from the categorical distribution definedby the node’s probability vector for the required return type, normalizing theR. Salustowicz and J. Schmidhuber, “Probabilistic Incremental Program Evolution,” Evolutionary Computation, vol. 5, no. 2, pp. 123–141,
1997.
11

The arity of a function is thenumber of arguments. The arityof a grammar rule, which can beviewed as a function, is the number of nonterminals in the rule.
13
The Dirichlet distribution is often used to represent a distribution over discrete distributions. D.
Barber, Bayesian Reasoning and Machine Learning. Cambridge University Press, 2012.
12© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

378 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

Algorithm 20.11. A probabilistic
prototype tree node type and associated initialization function where
ps is a dictionary mapping a symbol corresponding to a return typeto a probability vector over applicable rules, and children is a list of
PPTNodes. The method get_childwill automatically expand the tree
when attempting to access a nonexistent child.

struct PPTNodepschildrenendfunction PPTNode(grammar;
w_terminal = 0.6,
w_nonterm = 1-w_terminal,)
ps = Dict(typ => normalize!([isterminal(grammar, i) ?
w_terminal : w_nonterm
for i in grammar[typ]], 1)
for typ in nonterminals(grammar))
PPTNode(ps, PPTNode[])end
function get_child(ppt::PPTNode, grammar, i)
if i > length(ppt.children)
push!(ppt.children, PPTNode(grammar))end
return ppt.children[i]endassociated probability vector values to obtain a valid probability distribution. The
tree is traversed in depth-first order. This sampling procedure is implemented inalgorithm 20.12 and visualized in figure 20.9.
Learning can use information either from an entire sampled population orfrom elite samples. Let the best expression in the current generation be xbest andthe best expression found so far be xelite . The node probabilities are updated toincrease the likelihood of generating xbest .14
The probability of generating xbest is the product of the probabilities of choosing
each rule in xbest when traversing through the probabilistic prototype tree. We
compute a target probability for P( xbest ):Ptarget = P( xbest ) + (1 − P( xbest )) · α ·ǫ − yeliteǫ − ybest

The original probabilistic prototype tree implementation will periodically increase the likelihood ofgenerating xelite .

14(20.1)

where α and ǫ are positive constants. The fraction on the right-hand side produces
larger steps toward expressions with better objective function values. The target
probability can be calculated using algorithm 20.13.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 5 . p roba bi l i sti c p rototy p e tre e s 379

Algorithm 20.12. A method forsampling an expression from aprobabilistic prototype tree. The
tree is expanded as needed.

function rand(ppt, grammar, typ)rules = grammar[typ]rule_index = sample(rules, Weights(ppt.ps[typ]))
ctypes = child_types(grammar, rule_index)
arr = Vector{RuleNode}(undef, length(ctypes))
node = iseval(grammar, rule_index) ?
RuleNode(rule_index, eval(grammar, rule_index), arr) :RuleNode(rule_index, arr)endfor (i,typ) in enumerate(ctypes)
node.children[i] =rand(get_child(ppt, grammar, i), grammar, typ)end
return node

sample 1st expression
×
ln
p1

x

2
p11
p111

Figure 20.9. A probabilistic prototype tree initially contains only
the root node but expands as additional nodes are needed during
expression generation.

sample 2nd expression
×
p1

−
p12

3

p1

+
x

2

7

p11
p111

p12
p112

function probability(ppt, grammar, expr)
typ = return_type(grammar, expr)
i = findfirst(isequal(expr.ind), grammar[typ])p = ppt.ps[typ][i]for (i,c) in enumerate(expr.children)p *= probability(get_child(ppt, grammar, i), grammar, c)
end
return pendfunction p_target(ppt, grammar, x_best, y_best, y_elite, α, ϵ)p_best = probability(ppt, grammar, x_best)
return p_best + (1-p_best)*α*(ϵ - y_elite)/(ϵ - y_best)
end

p121

p122

Algorithm 20.13. Methods for computing the probability of an expression and the target probability, where ppt is the root nodeof the probabilistic prototype tree,grammar is the grammar, expr andx_best are RuleNode expressions,y_best and y_elite are scalar objective function values, and α andϵ are scalar parameters.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

380 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

The target probability is used to adjust the probability vectors in the probabilistic prototype tree. The probabilities associated with the chosen nodes are
increased iteratively until the target probability is exceeded:







(i )
(i )
(i )
for all i in {1, 2, . . .}(20.2)
P xbest ← P xbest + c · α · 1 − P xbest

where xbest is the ith rule applied in expression xbest and c is a scalar.15
The adapted probability vectors are then renormalized to 1 by downscaling thevalues of all nonincreased vector components proportionally to their current value.
The probability vector p, where the ith component was increased, is adjusted
according to:
1 − pipj ← pj
for j 6= i
(20.3)
kpk1 − p i
(i )

15

A recommended value is c = 0.1.

The learning update is implemented in algorithm 20.14.

function _update!(ppt, grammar, x, c, α)
typ = return_type(grammar, x)
i = findfirst(isequal(x.ind), grammar[typ])p = ppt.ps[typ]p[i] += c*α*(1-p[i])psum = sum(p)for j in 1 : length(p)if j != ip[j] *= (1- p[i])/(psum - p[i])endendfor (pptchild,xchild) in zip(ppt.children, x.children)_update!(pptchild, grammar, xchild, c, α)
end
return ppt
endfunction update!(ppt, grammar, x_best, y_best, y_elite, α, c, ϵ)
p_targ = p_target(ppt, grammar, x_best, y_best, y_elite, α, ϵ)while probability(ppt, grammar, x_best) < p_targ_update!(ppt, grammar, x_best, c, α)
end
return ppt
end

Algorithm 20.14. A method for
applying a learning update to aprobabilistic prototype tree withroot ppt, grammar grammar, best
expression x_best with objectivefunction value y_best, elite objective function value y_elite, learning rate α, learning rate multiplierc, and parameter ϵ.

In addition to population-based learning, probabilistic prototype trees can also
explore the design space via mutations. The tree is mutated to explore the region© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 5 . p roba bi l i sti c p rototy p e tre e s 381

around xbest . Let p be a probability vector in a node that was accessed whengenerating xbest . Each component in p is mutated with a probability proportional
to the problem size:
pmutation√(20.4)#p #xbestwhere pmutation is a mutation parameter, #p is the number of components in p,
and #xbest is the number of rules applied in xbest . A component i, selected formutation, is adjusted according to:p i ← p i + β · (1 − p i )(20.5)

where β controls the amount of mutation. Small probabilities undergo larger
mutations than do larger probabilities. All mutated probability vectors must be
renormalized. Mutation is implemented in algorithm 20.15 and visualized infigure 20.10.
function mutate!(ppt, grammar, x_best, p_mutation, β;
sqrtlen = sqrt(length(x_best)),)typ = return_type(grammar, x_best)p = ppt.ps[typ]prob = p_mutation/(length(p)*sqrtlen)
for i in 1 : length(p)if rand() < prob
p[i] += β*(1-p[i])endendnormalize!(p, 1)
for (pptchild,xchild) in zip(ppt.children, x_best.children)
mutate!(pptchild, grammar, xchild, p_mutation, β,
sqrtlen=sqrtlen)
end
return ppt
end

Algorithm 20.15. A method for mutating a probabilistic prototype treewith root ppt, grammar grammar,
best expression x_best, mutation
parameter p_mutation, and mutation rate β.

Finally, subtrees in the probabilistic prototype tree are pruned in order toremove stale portions of the tree. A child node is removed if its parent contains
a probability component above a specified threshold such that, when chosen,
causes the child to be irrelevant. This is always the case for a terminal and may© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

382 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

probability

10increaseprobability vector entries

normalize

probability vector entries

be the case for a nonterminal. Pruning is implemented in algorithm 20.16 anddemonstrated in example 20.7.

20.6 Summary

probability vector entries
Figure 20.10. Mutating a probability vector in a probabilistic
prototype tree with β = 0.5.
The mutated components wereincreased according to equation (20.5) and the resulting
probability vector was renormalized. Notice how smaller
probabilities
receivegreaterincreases.

• Expression optimization allows for optimizing tree structures that, under agrammar, can express sophisticated programs, structures, and other designs
lacking a fixed size.
• Grammars define the rules used to construct expressions.
• Genetic programming adapts genetic algorithms to perform mutation andcrossover on expression trees.
• Grammatical evolution operates on an integer array that can be decoded into
an expression tree.
• Probabilistic grammars learn which rules are best to generate, and probabilistic
prototype trees learn probabilities for every iteration of the expression rule
generation process.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 6 . su mma ry 383

function prune!(ppt, grammar; p_threshold=0.99)
kmax, pmax = :None, 0.0
for (k, p) in ppt.ps
pmax′ = maximum(p)if pmax′ > pmax
kmax, pmax = k, pmax′
endend
if pmax > p_thresholdi = argmax(ppt.ps[kmax])
if isterminal(grammar, i)empty!(ppt.children[kmax])else
max_arity_for_rule = maximum(nchildren(grammar, r) forr in grammar[kmax])
while length(ppt.children) > max_arity_for_rule
pop!(ppt.children)
end
end
end
for child in ppt.children
prune!(child, grammar, p_threshold=threshold)end
return ppt
end

Algorithm 20.16. A method forpruning a probabilistic prototype
tree with root ppt, grammar
grammar, and pruning probability
threshold p_treshold.

Consider a node with a probability vector over the rule set:

Example 20.7. An example ofwhen pruning for probabilistic prototype trees should be applied.

R 7→ R + R
R 7→ ln(R )R 7→ 2 | x

R 7→ S

If the probability of selecting 2 or x grows large, then any children in theprobabilistic prototype tree are unlikely to be needed and can be pruned.
Similarly, if the probability of choosing S grows large, any children withreturn type R are unneeded and can be pruned.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

384 c ha p te r 20 . e x p re ssi on op ti mi z a ti on

20.7

Exercises

Exercise 20.1. How many expression trees can be generated using the following
grammar and the starting set {R, I, F }?
R 7→ I | F
I 7→ 1 | 2

F 7→ πExercise 20.2. The number of expression trees up to height h that can be generated
under a grammar grows super-exponentially. As a reference, calculate the numberof expressions of height h can be generated using the grammar:16
N 7→ {N, N } | {N, } | {, N } | {}(20.6)

Let an empty expression haveheight 0, the expression {} haveheight 1, and so on.

16

Exercise 20.3. Define a grammar which can generate any nonnegative integer.
Exercise 20.4. How do expression optimization methods handle divide-by-zerovalues or other exceptions encountered when generating random subtrees?
Exercise 20.5. Consider an arithmetic grammar such as:
R 7→ x | y | z | R + R | R − R | R × R | R/R | ln R | sin R | cos R
Suppose the variables x, y, and z each have units, and the output is expected to
be in particular units. How might such a grammar be modified to respect units?
Exercise 20.6. Consider the grammar
S 7→ NP VP

NP 7→ ADJ NP | ADJ N
VP 7→ V ADVADJ 7→ a | the | big | little | blue | red
N 7→ mouse | cat | dog | pony
V 7→ ran | sat | slept | ate

ADV 7→ quietly | quickly | soundly | happily

What is the phenotype corresponding to the genotype [2, 10, 19, 0, 6] and thestarting symbol S?
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

20 . 7. e x e rc i se s 385
rpExercise 20.7. Use genetic programming to evolve the gear ratios for a clock. Assume all gears are restricted to have radii selected from R = {10, 25, 30, 50, 60, 100}.
Each gear can either be attached to its parent’s axle, thereby sharing the samerotation period, or be interlocked on its parent’s rim, thereby having a rotation
period depending on the parent’s rotation period and on the gear ratio as shownin figure 20.11.
The clock can also contain hands, which are mounted on the axle of a parent
gear. Assume the root gear turns with a period of troot = 0.1 s and has a radius of
25. The objective is to produce a clock with a second, minute, and hour hand.
Score each individual according to:minimize (1 − thand )
hands

2





+ minimize (60 − thand )
hands

2





rc

tc = t p rrpc
Figure 20.11. The rotation period
tc of a child gear attached to therim of a parent gear depends on
the rotation period of the parent
gear, t p and the ratio of the gears’
radii.

+ minimize (3600 − thand )
hands

2



+ #nodes · 10−3

where thand is the rotation period of a particular hand in seconds and #nodes isthe number of nodes in the expression tree. Ignore rotation direction.
Exercise 20.8. The four 4s puzzle17 is a mathematical challenge in which we use
four 4 digits and mathematical operations to generate expressions for each of theintegers from 0 to 100. For example, the first two integers can be produced by
4 + 4 − 4 − 4 and 44/44, respectively. Complete the four 4s puzzle.

W. W. R. Ball, Mathematical Recreations and Essays. Macmillan, 1892.

17

Exercise 20.9. Consider the probabilistic grammar
R 7→ R + R | R × R | F | I

wR = [1, 1, 5, 5]

I 7→ 1 | 2 | 3

wI = [1, 1, 1]

F 7→ 1.5 | ∞wF = [4, 3]

What is the generation probability of the expression 1.5 + 2?
Exercise 20.10. What is the probabilistic grammar from the previous question
after clearing the counts and applying a learning update on 1.5 + 2?

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21

Multidisciplinary Optimization

Multidisciplinary design optimization (MDO) involves solving optimization problems spanning across disciplines. Many real-world problems involve complicatedinteractions between several disciplines, and optimizing disciplines individually
may not lead to an optimal solution. This chapter discusses a variety of techniques for taking advantage of the structure of MDO problems to reduce theeffort required for finding good designs.1

21.1 Disciplinary Analyses
There are many different disciplinary analyses that might factor into a design. For
example, the design of a rocket might involve analysis from disciplines suchas structures, aerodynamics, and controls. The different disciplines have theirown analytical tools, such as finite element analysis. Often these disciplinary
analyses tend to be quite sophisticated and computationally expensive. In addition, disciplinary analyses are often tightly coupled with each other, where one
discipline may require the output of another’s disciplinary analysis. Resolving
these interdependencies can be nontrivial.
In an MDO setting, we still have a set of design variables as before, but we also
keep track of the outputs, or response variables, of each disciplinary analysis.2 We
write the response variables of the ith disciplinary analysis as y(i) . In general, theith disciplinary analysis Fi can depend on the design variables or the response
variables from any other discipline:y(i) ← Fi x, y(1) , . . . , y(i−1) , y(i+1) , . . . , y(m)(21.1)
where m is the total number of disciplines. The inputs to a computational fluid
dynamics analysis for an aircraft may include the deflections of the wing, which

An extensive survey is provided
by J. R. R. A. Martins and A. B.
Lambe, “Multidisciplinary Design
Optimization: A Survey of Architectures,” AIAA Journal, vol. 51,
no. 9, pp. 2049–2075, 2013. Further discussion can be found in J.
Sobieszczanski-Sobieski, A. Morris, and M. van Tooren, Multidisciplinary Design Optimization Supported by Knowledge Based Engineering. Wiley, 2015. See also N. M.
Alexandrov and M. Y. Hussaini,eds., Multidisciplinary Design Optimization: State of the Art. SIAM,
1997.
1

2
A disciplinary analysis can provide inputs for other disciplines,
the objective function, or the constraints. In addition, it can also provide gradient information for the
optimizer.

388 c ha p te r 21. mu l ti d i sc i p l i n a ry op ti mi z a ti oncome from a structural analysis that requires the forces from computational fluid
dynamics. An important part of formulating MDO problems is taking into accountsuch dependencies between analyses.
In order to make reasoning about disciplinary analyses easier, we introduce
the concept of an assignment. An assignment A is a set of variable names andtheir corresponding values relevant to a multidisciplinary design optimization
problem. To access a variable v, we write A[v].
A disciplinary analysis is a function that takes an assignment and uses thedesign point and response variables from other analyses to overwrite the response
variable for its discipline:
A′ ← Fi (A)(21.2)
where Fi (A) updates y(i) in A to produce A′ .
Assignments can be represented in code using dictionaries.3 Each variable
is assigned a name of type String. Variables are not restricted to floating-point
numbers but can include other objects, such as vectors. Example 21.1 shows animplementation using a dictionary.
Consider an optimization with one design variable x and two disciplines.
Suppose the first disciplinary analysis F1 computes a response variabley(1) = f 1 ( x, y(2) ) and the second disciplinary analysis F2 computes a response variable y(2) = f 2 ( x, y(1) ).
This problem can be implemented as:

3
A dictionary, also called an associative array, is a common data
structure that allows indexing bykeys rather than integers. See appendix A.1.7.

Example 21.1. Basic code syntax forthe assignment-based representation of multidisciplinary design optimization problems.

function F1(A)A["y1"] = f1(A["x"], A["y2"])
return A
endfunction F2(A)A["y2"] = f2(A["x"], A["y1"])
return A
end

The assignment may be initialized with guesses for y(1) and y(2) , and aknown input for x. For example:
A = Dict("x"=>1, "y1"=>2, "y2"=>3)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21. 2. i n te rd i sc i p l i n a ry c omp a ti bi l i ty 389

21.2

Interdisciplinary Compatibility

Evaluating the objective function value and feasibility of a design point x requires obtaining values for the response variables that satisfy interdisciplinary
compatibility, which means that the response variables must be consistent withthe disciplinary analyses. Interdisciplinary compatibility holds for a particularassignment if the assignment is unchanged under all disciplinary analyses:
Fi (A) = A for i ∈ {1, . . . , m}

(21.3)

Running any analysis will produce the same values. Finding an assignment thatsatisfies interdisciplinary compatibility is called a multidisciplinary analysis.
System optimization for a single discipline requires an optimizer to selectdesign variables and query the disciplinary analysis in order to evaluate theconstraints and the objective function, as shown in figure 21.1. Single-discipline
optimization does not require that we consider disciplinary coupling.
Optimizer
minimizef (x, y)x

subject to[x, y] ∈ X

yxDisciplinary
Analysis

Figure 21.1. Optimization diagram
for a single discipline. Gradients
may or may not be computed.

System optimization for multiple disciplines can introduce dependencies, inwhich case coupling becomes an issue. A diagram for two coupled disciplines isgiven in figure 21.2. Applying conventional optimization to this problem is lessstraightforward because interdisciplinary compatibility must be established.

Optimizer
minimizex

subject to

Disciplinary
Analysis 1xf (x, y(1) , y(2) )y(1)[x, y(1) , y(2) ] ∈ X

x
y(2)

y(1)

Figure 21.2. Optimization diagram
for a two-discipline analysis withinterdisciplinary coupling.

y(2)Disciplinary
Analysis 2© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

390 c ha p te r 21. mu l ti d i sc i p l i n a ry op ti mi z a ti on

If a multidisciplinary analysis does not have a dependency cycle,4 then solving
for interdisciplinary compatibility is straightforward. We say discipline i depends
on discipline j if i requires any of j’s outputs. This dependency relation can beused to form a dependency graph, where each node corresponds to a discipline andan edge j → i is included if discipline i depends on j. Figure 21.3 shows examples
of dependency graphs involving two disciplines with and without cycles.
Structural Analysis

Structural Analysis

Cost Forecast

Aerodynamics Analysis

An acyclic dependency graph. An evaluation
ordering can be specified such that the required
inputs for each discipline are available frompreviously evaluated disciplines.

A cyclic dependency graph. The structural
analysis depends on the aerodynamics analysis
and vice versa.

If the dependency graph has no cycles, then there always exists an order of
evaluation that, if followed, ensures that the necessary disciplinary analyses areevaluated before the disciplinary analyses that depend on them. Such an ordering
is called a topological ordering and can be found using a topological sorting method
such as Kahn’s algorithm.5 The reordering of analyses is illustrated in figure 21.4.
If the dependency graph has cycles, then no topological ordering exists. To
address cycles, we can use the Gauss-Seidel method (algorithm 21.1), which attempts to resolve the multidisciplinary analysis by iterating until convergence.6
The Gauss-Seidel algorithm is sensitive to the ordering of the disciplines as illustrated by example 21.2. A poor ordering can prevent or slow convergence. The
best orderings are those with minimal feedback connections.7
It can be advantageous to merge disciplines into a new disciplinary analysis—
to group conceptually related analyses, simultaneously evaluate tightly coupledanalyses, or more efficiently apply some of the architectures discussed in thischapter. Disciplinary analyses can be merged to form a new analysis whose
response variables consist of the response variables of the merged disciplines.
The form of the new analysis depends on the disciplinary interdependencies. If

A dependency cycle arises when
disciplines depend on each other.

4

Figure 21.3. Cyclic and acyclic dependency graphs.

5A. B. Kahn, “Topological Sortingof Large Networks,” Communications of the ACM, vol. 5, no. 11,
pp. 558–562, 1962.

The Gauss-Seidel algorithm canalso be written to execute analyses
in parallel.

67
In some cases, disciplines canbe separated into different clusters that are independent of eachother. Each connected cluster canbe solved using its own, smaller
multidisciplinary analysis.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21. 2. i n te rd i sc i p l i n a ry c omp a ti bi l i ty 391

Figure 21.4. A topological sort canbe used to reorder the disciplinary
analyses to remove feedback connections.

function gauss_seidel!(Fs, A; k_max=100, ϵ=1e-4)
k, converged = 0, false
while !converged && k ≤ k_max
k += 1
A_old = deepcopy(A)for F in Fs
F(A)end
converged = all(isapprox(A[v], A_old[v], rtol=ϵ)for v in keys(A))
endreturn (A, converged)
end

Algorithm 21.1. The Gauss-Seidel
algorithm for conducting a multidisciplinary analysis. Here, Fs is avector of disciplinary analysis functions that take and modify an assignment, A. There are two optional
arguments: the maximum numberof iterations k_max and the relative
error tolerance ϵ. The method returns the modified assignment andwhether it converged.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

392 c ha p te r 21. mu l ti d i sc i p l i n a ry op ti mi z a ti on

Consider a multidisciplinary design optimization problem with one design
variable x and three disciplines, each with one response variable:y(1) ← F1 ( x, y(2) , y(3) ) = y(2) − x

Example 21.2. An example that illustrates the importance of choosing an appropriate ordering whenrunning a multidisciplinary analysis.

y(2) ← F2 ( x, y(1) , y(3) ) = sin(y(1) + y(3) )y(3) ← F3 ( x, y(1) , y(2) ) = cos( x + y(1) + y(2) )The disciplinary analyses can be implemented as:function F1(A)A["y1"] = A["y2"] - A["x"]return A
endfunction F2(A)A["y2"] = sin(A["y1"] + A["y3"])
return A
endfunction F3(A)A["y3"] = cos(A["x"] + A["y2"] + A["y1"])
return A
end

Consider running a multidisciplinary analysis for x = 1, having initialized
our assignment with all 1’s:
A = Dict("x"=>1, "y1"=>1, "y2"=>1, "y3"=>1)

Running the Gauss-Seidel algorithm with the ordering F1 , F2 , F3 converges,
but running with F1 , F3 , F2 does not.
1

y (1)

0

y (2)
y (3)

−1
−2
1
0

−1
−2

0

5

10

15

20

iteration© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21. 3. a rc hi te c tu re s 393

the merged disciplines are acyclic then an ordering exists in which the analyses
can be serially executed. If the merged disciplines are cyclic, then the new analysis
must internally run a multidisciplinary analysis to achieve compatibility.

21.3

Architectures

Multidisciplinary design optimization problems can be written:
minimizef (A)subject toA∈Xx(21.4)Fi (A) = A for each discipline i ∈ {1, . . . , m}where the objective function f and feasible set X depend on both the design andresponse variables. The design variables in the assignment are specified by the
optimizer. The condition Fi (A) = A ensures that the ith discipline is consistent
with the values in A. This last condition enforces interdisciplinary compatibility.
There are several challenges associated with optimizing multidisciplinary
problems. The interdependence of disciplinary analyses causes the orderingof analyses to matter and often makes parallelization difficult or impossible.
There is a trade-off between an optimizer that directly controls all variables
and incorporating suboptimizers8 that leverage discipline-specific expertise to
optimize values locally. In addition, there is a trade-off between the expense of
running disciplinary analyses and the expense of globally optimizing too many
variables. Finally, every architecture must enforce interdisciplinary compatibility
in the final solution.
The remainder of this chapter discusses a variety of different optimization architectures for addressing these challenges. These architectures are demonstratedusing the hypothetical ride-sharing problem introduced in example 21.3.

21.4

A suboptimizer is an optimization
routine called within another optimization routine.
8

Multidisciplinary Design Feasible

The multidisciplinary design feasible architecture structures the MDO problem suchthat standard optimization algorithms can be directly applied to optimize thedesign variables. A multidisciplinary design analysis is run for any given design
point to obtain compatible response values.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

394 c ha p te r 21. mu l ti d i sc i p l i n a ry op ti mi z a ti on

Consider a ride-sharing company developing a self-driving fleet. This hypothetical company is simultaneously designing the vehicle, its sensor package,a routing strategy, and a pricing scheme. These portions of the design arereferred to as v, s, r, and p, respectively, each of which contains numerous
design variables. The vehicle, for instance, may include parameters governing the structural geometry, engine and drive train, battery capacity, andpassenger capacity.
The objective of the ride-sharing company is to maximize profit. The
profit depends on a large-scale simulation of the routing algorithm andpassenger demand, which, in turn, depends on response variables from anautonomy analysis of the vehicle and its sensor package. Several analyses
extract additional information. The performance of the routing algorithm
depends on the demand generated by the pricing scheme and the demand
generated by the pricing scheme depends on performance of the routing
algorithm. The vehicle range and fuel efficiency depends on the weight, drag,and power consumption of the sensor package. The sensor package requires
vehicle geometry and performance information to meet the necessary safety
requirements. A dependency diagram is presented below.

Example 21.3. A ride-sharing problem used throughout this chapter
to demonstrate optimization architectures.

Vehicle Analysis
Sensor Analysis
Autonomy Analysis
Routing Analysis
Demand Analysis
Profit Analysis© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21. 4 . mu l ti d i sc i p l i n a ry d e si g n fe a si bl e 395

Optimizer
minimizef (A)update x

subject toA∈Xupdate y(1) , . . . , y(m)xminimizex

subject to



(1)

(m)



f x, y , . . . , yhix, y(1) , . . . , y(m) ∈ X

Multidisciplinary
Analysisminimizef (MDA(x))

subject to

MDA(x) ∈ X

xFigure 21.5. The multidisciplinary
design feasible architecture. The
optimizer chooses design points x,
and the multidisciplinary analysis
computes a consistent assignment
A. The structure is similar to thatof single-discipline optimization.
Figure 21.6.
Formulating an
MDO problem into a typicaloptimization problem using
multidisciplinary design analyses, where MDA(x) returns amultidisciplinary
compatible
assignment.

An architecture diagram is given in figure 21.5. It consists of two blocks, theoptimizer and the multidisciplinary analysis. The optimizer is the method used
for selecting design points with the goal of minimizing the objective function. The
optimizer calls the multidisciplinary analysis block by passing it a design point
x and receives a compatible assignment A. If interdisciplinary compatibility is
not possible, the multidisciplinary analysis block informs the optimizer and suchdesign points are treated as infeasible. Figure 21.6 shows how an MDO problem
can be transformed into a typical optimization problem using multidisciplinary
design analyses.
The primary advantages of the multidisciplinary design feasible architecture
are its conceptual simplicity and that it is guaranteed to maintain interdisciplinary
compatibility at each step in the optimization. Its name reflects the fact thatmultidisciplinary design analyses are run at every design evaluation, ensuringthat the system-level optimizer only considers feasible designs.
The primary disadvantage is that multidisciplinary design analyses are expensive to run, typically requiring several iterations over all analyses. Iterative
Gauss-Seidel methods may be slow to converge or may not converge at all, depending on the initialization of the response variables and the ordering of thedisciplinary analyses.
Lumping the analyses together makes it necessary for all local variables—
typically only relevant to a particular discipline—to be considered by the analysis
as a whole. Many practical problems have a very large number of local design
variables, such as mesh control points in aerodynamics, element dimensions in
structures, component placements in electrical engineering, and neural network© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

396 c ha p te r 21. mu l ti d i sc i p l i n a ry op ti mi z a ti on

weights in machine learning. Multidisciplinary design feasible optimization requires that the system optimizer specify all of these values across all disciplines
while satisfying all constraints.
The multidisciplinary design feasible architecture is applied to the ride-sharing
problem in example 21.4.
The multidisciplinary design feasible architecture can be applied to the ridesharing problem. The architectural diagram is shown below.
Optimizerminimize f v, s, r, p, y(v) , y(s) , y( a) , y(r) , y(d) , y( p)v,s,r,phisubject to v, s, r, p, y(v) , y(s) , y( a) , y(r) , y(d) , y( p) ∈ X
update y(v) , y(s) , y( a) , y(r) , y(d) , y( p)Example 21.4. The multidisciplinary design feasible architecture
applied to the ride-sharing problem. A multidisciplinary analysis
over all response variables must becompleted for every candidate design point. This tends to be very
computationally intensive.

update v, s, r, p

Multidisciplinary Analysis

21.5 Sequential Optimization
The sequential optimization architecture (figure 21.7) is an architecture that canleverage discipline-specific tools and experience to optimize subproblems but
can lead to suboptimal solutions. This architecture is included to demonstrate
the limitations of a naive approach and to serve as a baseline against which otherarchitectures can be compared.
A subproblem is an optimization procedure conducted at every iteration of an
overarching optimization process. Sometimes design variables can be removed
from the outer optimization procedure, the system-level optimizer, and can be more
efficiently optimized in subproblems.
The design variables for the ith discipline can be partitioned according tox(i) = [xg , xℓ ], where xg are global design variables shared with other disciplines
(i )

(i )

(i )

and xℓ are local design variables used only by the associated disciplinary subproblem.9 The response variables can be similarly partitioned into the global response(i )variables yg and the local response variables yℓ . Disciplinary autonomy is
(i )

(i )

The vehicle subproblem in the
ride-sharing problem may include
global design variables such as thevehicle capacity and range that affect other disciplines but may also
include local design variables suchas the seating configuration that do
not impact other disciplines.
9© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21. 5 . se q u e n ti a l op ti mi z a ti on 397

Figure 21.7. The sequential optimization architecture. Each subproblem is represented by a blue
block and optimizes a particulardiscipline on a local objective function.

update xg

minimize f (A) subject to A ∈ X
xg

update xℓ(1)minimize f 1 (A) subject to [x

(1)

(1)
xℓ

,y

(1)

] ∈ X1

update y(1)update xℓ(2)minimize f 2 (A) subject to [x
(2)
xℓ

(2)

,y

(2)

] ∈ X2..

update y(2)update xℓ(m)minimize f m (A) subject to [x(m) , y(m) ] ∈ Xm(m)xℓ

update y(m)Disciplinary
Analysis 1

Disciplinary
Analysis 2

Disciplinary
Analysis m© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

398 c ha p te r 21. mu l ti d i sc i p l i n a ry op ti mi z a ti onachieved by optimizing the local variables in their own disciplinary optimizers. A
local objective function f i must be chosen such that optimizing it also benefits theglobal objective. A top-level optimizer is responsible for optimizing the global
design variables xg with respect to the original objective function. An instantiation of xg is evaluated through sequential optimizations; each subproblem is
optimized one after the other, passing its results to the next subproblem until all
have been evaluated.
Sequential optimization takes advantage of the locality of disciplines; thatmany variables are unique to a particular discipline and do not need to be sharedacross discipline boundaries. Sequential optimization harnesses each discipline’s
proficiency at solving its discipline-specific problem. The subproblem optimizers
have complete control over their discipline-specific design variables to meet local
design objectives and constraints.
Except in special cases, sequential optimization does not lead to an optimal
solution of the original problem for the same reason that Gauss-Seidel is notguaranteed to converge. The solution is sensitive to the local objective functions,and finding suitable local objective functions is often a challenge. Sequential optimization does not support parallel execution, and interdisciplinary compatibility
is enforced through iteration and does not always converge.
Example 21.5 applies sequential optimization to the ride-sharing problem.

21.6 Individual Discipline Feasible
The individual discipline feasible (IDF) architecture removes the need to run expensive multidisciplinary design analyses and allows disciplinary analyses tobe executed in parallel. It loses the guarantee that interdisciplinary compatibility is maintained throughout its execution, with eventual agreement enforced
through equality constraints in the optimizer. Compatibility is not enforced in
multidisciplinary analyses but rather by the optimizer itself.
IDF introduces coupling variables to the design space. For each discipline, anadditional vector c(i) is added to the optimization problem to act as aliases forthe response variables y(i) . The response variables are unknown until they arecomputed by their respective domain analyses; inclusion of the coupling variables allows the optimizer to provide these estimates to multiple disciplines
simultaneously when running analyses in parallel. Equality between the cou© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21. 6 . i n d i vi d u a l d i sc i p l i n e fe a si bl e 399

The sequential optimization architecture can optimize some variables locally.
Figure 21.8 shows the result of applying sequential optimization to the ridesharing problem.
The design variables for the vehicle, sensor system, routing algorithm, andpricing scheme are split into local discipline-specific variables and top-levelglobal variables. For example, the vehicle subproblem can optimize local
vehicle parameters vℓ such as drive train components, whereas parameterslike vehicle capacity that are used by other analyses are controlled globallyin vg .
The tight coupling between the vehicle and sensor systems is poorly handled by the sequential optimization architecture. While changes made by
the vehicle subproblem are immediately addressed by the sensor subproblem, the effect of the sensor subproblem on the vehicle subproblem is notaddressed until the next iteration.
Not all analyses require their own subproblems. The profit analysis is
assumed not to have any local design variables and can thus be executedwithout needing a subproblem block.

Example 21.5. Sequential optimization for the ride-sharing problem.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

400 c ha p te r 21. mu l ti d i sc i p l i n a ry op ti mi z a ti on

update vg , sg , rg , pg

minimize f (A) subject to A ∈ X
vg ,sg ,rg ,pg

minimize f v (A) subject to [vℓ , y(v) ] ∈ Xvvℓminimize f s (A) subject to [sℓ , y] ∈ Xs( a)aℓminimize f r (A) subject to [rℓ , y
rℓ

(r )

update y(v)update sℓ(s)sℓminimize f a (A) subject to [aℓ , yupdate vℓupdate y(s)update aℓ] ∈ Xaupdate y( a)update rℓ] ∈ Xrminimize f d (A) subject to [dℓ , y(d) ] ∈ Xddℓupdate y(r)update dℓupdate y(d)VehicleAnalysis

SensorAnalysisAutonomy
Analysis

Routing
Analysis

DemandAnalysis

ProfitAnalysis

Figure 21.8. The sequential optimization architecture applied to
the ride-sharing problem.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21. 6 . i n d i vi d u a l d i sc i p l i n e fe a si bl e 401

pling and response variables is typically reached through iteration. Equality is an
optimization constraint, c(i) = y(i) , for each discipline.
Figure 21.9 shows the general IDF architecture. The system-level optimizer
operates on the coupling variables and uses these to populate an assignment thatis copied to the disciplinary analysis in each iteration:

A[x, y(1) , . . . , y(m) ] ← [x, c(1) , . . . , c(m) ]minimizef (x, c(1) , . . . , c(m) )subject to[x, c(1) , . . . , c(m) ] ∈ X

x,c(1) ,...,c(m)(21.5)

c(i) = y(i) for i ∈ {1, . . . , m}copy of A

Disciplinary
Analysis 1

update y(1)copy of A

Disciplinary
Analysis 2

update y(2)copy of A

Disciplinary
Analysis m

update y(m)..

Despite the added freedom to execute analyses in parallel, IDF suffers fromthe shortcoming that it cannot leverage domain-specific optimization proceduresin the same way as sequential optimization as optimization is top-level only. Furthermore, the optimizer must satisfy additional equality constraints and has morevariables to optimize. IDF can have difficulties with gradient-based optimization
since the chosen search direction must take constraints into account as shown infigure 21.10. Changes in the design variables must not cause the coupling variables to become infeasible with respect to the disciplinary analyses. Evaluating
the gradients of the objective and constraint function is very costly when thedisciplinary analyses are expensive.
The individual discipline feasible architecture is applied to the ride-sharing
problem in figure 21.11.

Figure 21.9. The individual discipline feasible architecture allows
disciplinary analyses to be run inparallel. This chapter assumes thatdisciplinary analyses mutate their
inputs, so copies of the system level
optimizer’s assignment are passed
to each disciplinary analysis.

x2

x1
Figure 21.10. The search direction
for a point on a constraint boundary must lead into the feasible set.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

402 c ha p te r 21. mu l ti d i sc i p l i n a ry op ti mi z a ti onf (v, s, r, p, c(v) , c(s) , c( a) , c(r) , c(d) , c( p) )minimizev,s,r,p,c(v) ,c(s) ,c( a) ,c(r) ,c(d) ,c( p)subject tohiv, s, r, p, c(v) , c(s) , c( a) , c(r) , c(d) , c( p) ∈ X
hic( v ) , c( s ) , c( a ) , c(r ) , c( d ) , c( p ) = [ y( v ) , y( s ) , y( a ) , y(r ) , y( d ) , y( p ) ]copy of A

Vehicle Analysis

copy of A

Sensor Analysis

copy of A

Autonomy Analysis

copy of A

Routing Analysis

copy of A

Demand Analysis

copy of A

Profit Analysis

update y(v)update y(s)update y( a)update y(r)update y(d)update y( p)Figure 21.11. The individual discipline feasible architecture applied
to the ride-sharing problem. The
individual design feasible architecture allows for parallel execution ofanalyses, but the system-level optimizer must optimize a large number of variables.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21. 7. c ol l a bora ti ve op ti mi z a ti on

21.7

403

Collaborative Optimization

The collaborative optimization architecture (figure 21.12) breaks a problem into
disciplinary subproblems that have full control over their local design variables
and discipline-specific constraints. Subproblems can be solved using disciplinespecific tools and can be optimized in parallel.
The ith subproblem has the form:
minimizef i ( x( i ) , y( i ) )subject to[ x( i ) , y( i ) ] ∈ X i

x( i )(21.6)

with x(i) containing a subset of the design variables x and response variables y(i) .
The constraint ensures that the solution satisfies discipline-specific constraints.
Interdisciplinary compatibility requires that the global variables xg and yg
agree between all disciplines. We define a set of coupling variables A g that includes variables corresponding to all design and response variables that are global
(i )

(i )

in at least one subproblem. Agreement is enforced by constraining each xg and(i )yg to match its corresponding coupling variables:
(i )

(i )

and

(i )

x g = A g [x g ]

(i )

(i )

y g = A g [y g ](21.7)

where A g [xg ] and A g [yg ] are the coupling variables corresponding to the global
design and response variables in the ith discipline. This constraint is enforced
using the subproblem objective function:
(i )

(i )

(i )

(i )

f i = x g − A g [x g ]

2
2

(i )

(i )

+ y g − A g [y g ]22(21.8)

Each subproblem thus seeks feasible solutions that minimally deviate from thecoupling variables.
The subproblems are managed by a system-level optimizer that is responsible
for optimizing the coupling variables A g to minimize the objective function. Evaluating an instance of the coupling variables requires running each disciplinary
subproblem, typically in parallel.
Disciplinary subproblems may deviate from the coupling variables during
the optimization process. This discrepancy occurs when two or more disciplines
disagree on a variable or when subproblem constraints prevent matching thetarget values set by the system-level optimizer. The top-level constraint that f i = 0
for each discipline ensures that coupling is eventually attained.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

404 c ha p te r 21. mu l ti d i sc i p l i n a ry op ti mi z a ti on

minimize f (A g ) subject to f i = 0 for i ∈ {1, . . . , m}Ag

f1

minimize
x(1)

(1)

(1)

f 1 = x g − A g [x g ]Agsubject to[xf2

minimizef 2 = x g − A g [x g ]

x(2)

(1)

,y

(1)

(2)

..

Ag

subject to[xfm

minimizef m = xg

subject to[ x( m ) , y( m ) ] ∈ X m

x( m )

Ag

,y

(2)

2

(1)

(1)

+ y g − A g [y g ]22] ∈ X1

(2)

(2)

2

] ∈ X2(m)(m)− A g [x g ]

2
2

2
2

(2)

(2)

+ y g − A g [y g ]

(m)

+ yg

The primary advantages of collaborative optimization stem from its ability
to isolate some design variables into disciplinary subproblems. Collaborative
optimization is readily applicable to real-world multidisciplinary problem solving,as each discipline is typically well segregated, and thus largely unaffected bysmall decisions made in other disciplines. The decentralized formulation allows
traditional discipline optimization methods to be applied, allowing problem
designers to leverage existing tools and methodologies.
Collaborative optimization requires optimizing over the coupling variables,
which includes both design and response variables. Collaborative optimization
does not perform well in problems with high coupling because the additional
coupling variables can outweigh the benefits of local optimization.
Collaborative optimization is a distributed architecture that decomposes a single
optimization problem into a smaller set of optimization problems that have thesame solution when their solutions are combined. Distributed architectures havethe advantage of reduced solving times, as subproblems can be optimized inparallel.
Collaborative optimization is applied to the ride-sharing problem in example 21.6.

(m)− A g [y g ]

2
2

2
2

Figure 21.12. Design architecture
for collaborative optimization.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21. 7. c ol l a bora ti ve op ti mi z a ti on

The collaborative optimization architecture can be applied to the vehicle
routing problem by producing six different disciplinary subproblems. Unfortunately, having six different subproblems requires any variables sharedacross disciplines to be optimized at the global level.
Figure 21.13 shows two disciplinary subproblems obtained by grouping
the vehicle, sensor, and autonomy disciplines into a transport subproblem
and the routing, demand, and profit disciplines into a network subproblem. The disciplines grouped into each subproblem are tightly coupled.
Having only two subproblems significantly reduces the number of global
variables considered by the system-level optimizer because presumably very
few design variables are directly used by both the transport and network
subproblems.
The subproblems are each multidisciplinary optimization problems, themselves amenable to optimization using the techniques covered in this chapter.
We can, for example, use sequential optimization within the transport subproblem. We can also add another instance of collaborative optimization
within the network subproblem.

405

Example 21.6. Applying collaborative optimization to the ridesharing problem.

minimize f (A g ) subject to f transport = f network = 0
Agminimizex(transport)

subject tominimizex(network)

subject to(transport)

f transport = xg

[x(transport),y(transport)(network)

[x

,y

]

2
2

(transport)+ yg(transport)− A g [y g]22] ∈ Xtransport

f network = xg
(network)(transport)− A g [x g(network)(network)− A g [x g] ∈ Xnetwork]

2
2(network)+ yg(network)− A g [y g]22Figure 21.13. The collaborative optimization architecture applied to
the ride-sharing problem. The colored circles correspond to the disciplinary analyses contained withineach subproblem.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

406 c ha p te r 21. mu l ti d i sc i p l i n a ry op ti mi z a ti on

21.8 Simultaneous Analysis and Design
The simultaneous analysis and design (SAND) architecture avoids the central challenge of coordinating between multiple disciplinary analyses by having the optimizer conduct the analyses. Instead of running an analysis Fi (A) to obtain aresidual, SAND optimizes both the design and response variables subject to a
constraint Fi (A) = A. The optimizer is responsible for simultaneously optimizing
the design variables and finding the corresponding response variables.
Any disciplinary analysis can be transformed into residual form. The residualri (A) is used to indicate whether an assignment A is compatible with the ith
discipline. If Fi (A) = A, then ri (A) = 0; otherwise, ri (A) 6= 0. We can obtain aresidual form using the disciplinary analysis:ri (A) = Fi (A) − A[y(i) ](21.9)

though this is typically inefficient, as demonstrated in example 21.7.
Consider a disciplinary analysis that solves the equation Ay = x. The analysis
is F (x) = A−1 x, which requires an expensive matrix inversion. We canconstruct a residual form using equation (21.9):
r1 (x, y) = k F (x) − yk = A−1 x − yAlternatively, we can use the original constraint to construct a more efficient
residual form:
r2 (x, y) = kAy − xk

Example 21.7. Evaluating a disciplinary analysis in a residual istypically counter-productive. The
analysis must typically perform additional work to solve the problem
whereas a cleaner residual form
can more efficiently verify whether
the inputs are compatible.

In aerodynamics, these may include the Navier-Stokes equations.
In structural engineering, these
may include the elasticity equations. In electrical engineering,these may include the differential
equations for current flow.

10

The residual form of a discipline consists of the set of disciplinary equations
that are solved by the disciplinary analysis.10 It is often much easier to evaluate aresidual than to run a disciplinary analysis. In SAND, figure 21.14, the analysis
effort is the responsibility of the optimizer.
minimize f (A) subject to A ∈ X , ri (A) = 0 for each disciplineA

SAND can explore regions of the design space that are infeasible with respect
to the residual equations, as shown in figure 21.15. Exploring infeasible regions

Figure 21.14. Simultaneous analysis and design places the entire
burden on the optimizer. It uses
disciplinary residuals rather thandisciplinary analyses.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21. 9. su mma ry

can allow us to traverse the design space more easily and find solutions in feasible
regions disconnected from the feasible region of the starting design point. SAND
suffers from having to simultaneously optimize a very large number of variables
for which derivatives and other discipline-specific expertise are not available.
Furthermore, SAND gains much of its value from residuals that can be computed
more efficiently than can disciplinary analyses. Use of SAND in real-world applications is often limited by the inability to modify existing disciplinary analysis
code to produce an efficient residual form.
SAND is applied to the ride-sharing problem in example 21.8.
Applying the simultaneous analysis and design architecture to the ridesharing problem requires disciplinary residuals. These can potentially depend on all design and response variables. The architecture requires thatthe optimizer optimize all of the design variables and all of the response
variables.
minimizev,s,r,p,y(v) ,y(s) ,y( a) ,y(r) ,y(d) ,y( p)subject to

407

x2
x(1)

x1
Figure 21.15. SAND can explore regions of the design space that are
infeasible and potentially bridge
the gap between feasible subsets.
Example 21.8. The simultaneous
analysis and design architecture
applied to the ride-sharing problem.


f v, s, r, p, y(v) , y(s) , y( a) , y(r) , y(d) , y( p)hiv, s, r, p, y(v) , y(s) , y( a) , y(r) , y(d) , y( p) ∈ X

rv (v, s, r, p, y(v) , y(s) , y( a) , y(r) , y(d) , y( p) ) = 0
rs (v, s, r, p, y(v) , y(s) , y( a) , y(r) , y(d) , y( p) ) = 0
r a (v, s, r, p, y(v) , y(s) , y( a) , y(r) , y(d) , y( p) ) = 0
rr (v, s, r, p, y(v) , y(s) , y( a) , y(r) , y(d) , y( p) ) = 0rd (v, s, r, p, y(v) , y(s) , y( a) , y(r) , y(d) , y( p) ) = 0
r p (v, s, r, p, y(v) , y(s) , y( a) , y(r) , y(d) , y( p) ) = 0

21.9 Summary• Multidisciplinary design optimization requires reasoning about multiple disciplines and achieving agreement between coupled variables.
• Disciplinary analyses can often be ordered to minimize dependency cycles.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

408 c ha p te r 21. mu l ti d i sc i p l i n a ry op ti mi z a ti on• Multidisciplinary design problems can be structured in different architecturesthat take advantage of problem features to improve the optimization process.
• The multidisciplinary design feasible architecture maintains feasibility andcompatibility through the use of slow and potentially nonconvergent multidisciplinary design analyses.
• Sequential optimization allows each discipline to optimize its discipline-specificvariables but does not always yield optimal designs.
• The individual discipline feasible architecture allows parallel execution ofanalyses at the expense of adding coupling variables to the global optimizer.
• Collaborative optimization incorporates suboptimizers that can leverage domain specialization to optimize some variables locally.
• The simultaneous analysis and design architecture replaces design analyses
with residuals, allowing the optimizer to find compatible solutions but cannot
directly use disciplinary solution techniques.

21.10 Exercises
Exercise 21.1. Provide an example of a practical engineering problem that is
multidisciplinary.
Exercise 21.2. Provide an abstract example of a multidisciplinary problem where
the order of the analyses is important.
Exercise 21.3. What is one advantage of the individual discipline feasible architecture over the multidisciplinary design feasible and sequential optimizationarchitectures?
Exercise 21.4. Consider applying multidisciplinary design analysis to minimizing
the weight of a wing whose deformation and loading are computed by separate
disciplines. We will use a simplified version of the problem, representing thewing as a horizontally-mounted pendulum supported by a torsional spring.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

21. 10 . e x e rc i se s 409

k

kθ

θ

ℓ
mgℓ cos(θ )mg

The objective is to minimize the spring stiffness, k, such that the pendulum’s displacement does not exceed a target threshold. The pendulum length ℓ, pendulum
point mass m, and gravitational constant g are fixed.
We use two simplified analyses in place of more sophisticated analyses used tocompute deformations and loadings of aircraft wings. Assuming the pendulum
is rigid, the loading moment M is equal to mgℓ cos(θ ). The torsional spring resists
deformation such that the pendulum’s angular displacement θ is M/k.
Formulate the spring-pendulum problem under the multidisciplinary design
feasible architecture, and then solve it according to that architecture for:m = 1 kg, ℓ = 1 m, g = 9.81 m s−2 , θmax = 10◦
Exercise 21.5. Formulate the spring-pendulum problem under the individual
design feasible architecture.
Exercise 21.6. Formulate the spring-pendulum under the collaborative optimization architecture. Present the two disciplinary optimization problems and thesystem-level optimization problem.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

A

Julia

Julia is a scientific programming language that is free and open source.1 It is arelatively new language that borrows inspiration from languages like Python,
MATLAB, and R. It was selected for use in this book because it is sufficiently
high level2 so that the algorithms can be compactly expressed and readable while
also being fast. This book is compatible with Julia version 1.0. This appendix
introduces the necessary concepts for understanding the code included in the
text.

A.1

Types

Julia has a variety of basic types that can represent data such as truth values,
numbers, strings, arrays, tuples, and dictionaries. Users can also define their own
types. This section explains how to use some of the basic types and define new
types.

A.1.1 Booleans
The Boolean type in Julia, written Bool, includes the values true and false. We
can assign these values to variables. Variable names can be any string of characters,
including Unicode, with a few restrictions.
done = false
α = false

The left-hand side of the equal sign is the variable name, and the right hand sideis the value.
We can make assignments in the Julia console. The console will return a response to the expression being evaulated.

1

Julia may be obtained fromhttp://julialang.org.

In contrast with languages like
C++, Julia does not require programmers to worry about memory
management and other lower-level
details.
2

412 a p p e n d i x a. j u l i ajulia> x = true
true
julia> y = false
false
julia> typeof(x)BoolThe standard Boolean operators are supported.
julia> !x
# not
false
julia> x && y # andfalse
julia> x || y # ortrue

The # symbol indicates that the rest of the line is a comment and should not be
evaluated.

A.1.2

Numbers

Julia supports integer and floating point numbers as shown here
julia> typeof(42)Int64julia> typeof(42.0)
Float64

Here, Int64 denotes a 64-bit integer, and Float64 denotes a 64-bit floating pointvalue.3 We can also perform the standard mathematical operations:julia> x = 4
4julia> y = 22julia> x + y
6julia> x - y
2julia> x * y
8julia> x / y
2.0
julia> x ^ y
16

3
On 32-bit machines, an integer
literal like 42 is interpreted as an
Int32.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a. 1. ty p e s 413

julia> x % y # x modulo y0Note that the result of x / y is a Float64, even when x and y are integers. We
can also perform these operations at the same time as an assignment. For example,
x += 1 is shorthand for x = x + 1.
We can also make comparisons:julia> 3 > 4
false
julia> 3 >= 4
false
julia> 3 ≥ 4# unicode also works
false
julia> 3 < 4
truejulia> 3 <= 4
truejulia> 3 ≤ 4# unicode also works
true
julia> 3 == 4
false
julia> 3 < 4 < 5
true

A.1.3

Strings

A string is an array of characters. Strings are not used very much in this textbook
except for reporting certain errors. An object of type String can be constructedusing " characters. For example:
julia> x = "optimal""optimal"julia> typeof(x)StringA.1.4

Vectors

A vector is a one-dimensional array that stores a sequence of values. We can construct a vector using square brackets, separating elements by commas. Semicolons
in these examples suppress the output.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

414 a p p e n d i x a. j u l i ajulia> x = [];# empty vector
julia> x = trues(3);# Boolean vector containing three trues
julia> x = ones(3);# vector of three ones
julia> x = zeros(3);# vector of three zeros
julia> x = rand(3);# vector of three random numbers between 0 and 1
julia> x = [3, 1, 4];# vector of integers
julia> x = [3.1415, 1.618, 2.7182]; # vector of floats

An array comprehension can be used to create vectors. Below, we use the printfunction so that the output is printed horizontally.4
julia> print([sin(x) for x = 1:5])[0.841471, 0.909297, 0.14112, -0.756802, -0.958924]

Print statements were used forcompactness and are not needed.

4

We can inspect the type of vectors:
julia> typeof([3, 1, 4])# 1-dimensional array of Int64s
Array{Int64,1}
julia> typeof([3.1415, 1.618, 2.7182]) # 1-dimensional array of Float64s
Array{Float64,1}

We index into vectors using square brackets.
julia> x[1]# first element is indexed by 1
3.1415julia> x[3]# third element
2.7182julia> x[end]# use end to reference the end of the array
2.7182julia> x[end - 1] # this returns the second to last element
1.618

We can pull out a range of elements from an array. Ranges are specified using
a colon notation.
julia> x = [1, 1, 2, 3, 5, 8, 13];julia> print(x[1:3])# pull out the first three elements[1, 1, 2]julia> print(x[1:2:end]) # pull out every other element[1, 2, 5, 13]julia> print(x[end:-1:1]) # pull out all the elements in reverse order[13, 8, 5, 3, 2, 1, 1]

We can perform a variety of different operations on arrays. The exclamation
mark at the end of function names is often used to indicate that the function
mutates (i.e., changes) the input.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a. 1. ty p e s 415

julia> print([x, x])# concatenation[[1, 1, 2, 3, 5, 8, 13], [1, 1, 2, 3, 5, 8, 13]]julia> length(x)7julia> print(push!(x, -1))
# add an element to the end[1, 1, 2, 3, 5, 8, 13, -1]
julia> pop!(x)# remove an element from the end-1julia> print(append!(x, [2, 3])) # append y to the end of x
[1, 1, 2, 3, 5, 8, 13, 2, 3]
julia> print(sort!(x))# sort the elements in the vector[1, 1, 2, 2, 3, 3, 5, 8, 13]julia> x[1] = 2; print(x)# change the first element to 2
[2, 1, 2, 2, 3, 3, 5, 8, 13]julia> x = [1, 2];julia> y = [3, 4];julia> print(x + y)
# add vectors[4, 6]
julia> print(3x - [1, 2])# multiply by a scalar and subtract[2, 4]
julia> print(dot(x, y))# dot product11
julia> print(x⋅y)# dot product using unicode character11

It is often useful to apply various functions elementwise to vectors.
julia> print(x .* y)# elementwise multiplication[3, 8]
julia> print(x .^ 2)# elementwise squaring[1, 4]
julia> print(sin.(x)) # elementwise application of sin[0.841471, 0.909297]
julia> print(sqrt.(x)) # elementwise application of sqrt
[1.0, 1.41421]

A.1.5 Matrices
A matrix is a two-dimensional array. Like a vector, it is constructed using square
brackets. We use spaces to delimit elements in the same row and semicolons todelimit rows. We can also index into the matrix and output submatrices using
ranges.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

416 a p p e n d i x a. j u l i ajulia> X = [1 2 3; 4 5 6; 7 8 9; 10 11 12];julia> typeof(X)# a 2-dimensional array of Int64s
Array{Int64,2}
julia> X[2]# second element using column-major ordering
4
julia> X[3,2]# element in third row and second column
8julia> print(X[1,:])# extract the first row[1, 2, 3]julia> print(X[:,2])# extract the second column[2, 5, 8, 11]julia> print(X[:,1:2])# extract the first two columns[1 2; 4 5; 7 8; 10 11]
julia> print(X[1:2,1:2]) # extract a 2x2 matrix from the top left of x
[1 2; 4 5]

We can also construct a variety of special matrices and use array comprehensions:julia> print(Matrix(1.0I, 3, 3))# 3x3 identity matrix[1.0 0.0 0.0; 0.0 1.0 0.0; 0.0 0.0 1.0]
julia> print(Matrix(Diagonal([3, 2, 1]))) # 3x3 diagonal matrix with 3, 2, 1 on diagonal[3 0 0; 0 2 0; 0 0 1]
julia> print(rand(3,2))# 3x2 random matrix[0.763487 0.560622; 0.544312 0.907828; 0.510258 0.207356]
julia> print(zeros(3,2))# 3x2 matrix of zeros[0.0 0.0; 0.0 0.0; 0.0 0.0]
julia> print([sin(x + y) for x = 1:3, y = 1:2]) # array comprehension[0.909297 0.14112; 0.14112 -0.756802; -0.756802 -0.958924]

Matrix operations include the following:
julia> print(X')# complex conjugate transpose[1 4 7 10; 2 5 8 11; 3 6 9 12]
julia> print(3X .+ 2) # multiplying by scalar and adding scalar[5 8 11; 14 17 20; 23 26 29; 32 35 38]
julia> X = [1 3; 3 1]; # create an invertible matrix
julia> print(inv(X))# inversion[-0.125 0.375; 0.375 -0.125]
julia> det(X)# determinant-8.0
julia> print([X X])# horizontal concatenation[1 3 1 3; 3 1 3 1]
julia> print([X; X])# vertical concatenation[1 3; 3 1; 1 3; 3 1]
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a. 1. ty p e s 417

julia> print(sin.(X)) # elementwise application of sin[0.841471 0.14112; 0.14112 0.841471]

A.1.6 Tuples
A tuple is an ordered list of values, potentially of different types. They are constructed with parentheses. They are similar to arrays, but they cannot be mutated.
julia> x = (1,) # a single element tuple indicated by the trailing comma(1,)
julia> x = (1, 0, [1, 2], 2.5029, 4.6692) # third element is a vector(1, 0, [1, 2], 2.5029, 4.6692)
julia> x[2]0julia> x[end]4.6692julia> x[4:end](2.5029, 4.6692)
julia> length(x)5

A.1.7 Dictionaries
A dictionary is a collection of key-value pairs. Key-value pairs are indicated witha double arrow operator. We can index into a dictionary using square brackets aswith arrays and tuples.
julia> x = Dict(); # empty dictionary
julia> x[3] = 4 # associate value 4 with key 3
4
julia> x = Dict(3=>4, 5=>1) # create a dictionary with two key-value pairs
Dict{Int64,Int64} with 2 entries:
3 => 4
5 => 1julia> x[5]# return value associated with key 5
1
julia> haskey(x, 3) # check whether dictionary has key 3
truejulia> haskey(x, 4) # check whether dictionary has key 4
false© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

418 a p p e n d i x a. j u l i a

A.1.8 Composite Types
A composite type is a collection of named fields. By default, an instance of a composite type is immutable (i.e., it cannot change). We use the struct keyword andthen give the new type a name and list the names of the fields.
struct Aab
end

Adding the keyword mutable makes it so that an instance can change.
mutable struct Bab
end

Composite types are constructed using parentheses, between which we passin values for the different fields. For example,
x = A(1.414, 1.732)

The double-colon operator can be used to annotate the types for the fields.
struct A
a::Int64
b::Float64
endThis annotation requires that we pass in an Int64 for the first field and a Float64
for the second field. For compactness, this text does not use type annotations,but it is at the expense of performance. Type annotations allow Julia to improveruntime performance because the compiler can optimize the underlying code for
specific types.

Any

.
.
.

A.1.9 Abstract Types
So far we have discussed concrete types, which are types that we can construct.
However, concrete types are only part of the type hierarchy. There are also abstract
types, which are supertypes of concrete types and other abstract types.
We can explore the type hierarchy of the Float64 type shown in figure A.1
using the supertype and subtypes functions.

Number

.
.
.

Real

.
.
.

AbstractFloat
Float64
Float32
Float16
BigFloat

Figure A.1. The type hierarchy for
the Float64 type.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a. 1. ty p e s 419

julia> supertype(Float64)
AbstractFloat
julia> supertype(AbstractFloat)Realjulia> supertype(Real)
Numberjulia> supertype(Number)Anyjulia> supertype(Any)# Any is at the top of the hierarchy
Anyjulia> subtypes(AbstractFloat) # different types of AbstractFloats
4-element Array{Any,1}:
BigFloat
Float16
Float32
Float64
julia> subtypes(Float64)# Float64 does not have any subtypes
Type[]

We can define our own abstract types.
abstract type C end
abstract type D <: C end # D is an abstract subtype of C
struct E <: D # E is composite type that is a subtype of Daend

A.1.10 Parametric Types
Julia supports parametric types, which are types that take parameters. We havealready seen a parametric type with our dictionary example.
julia> x = Dict(3=>4, 5=>1)
Dict{Int64,Int64} with 2 entries:
3 => 4
5 => 1This constructs a Dict{Int64,Int64}. The parameters to the parametric type arelisted within braces and delimited by commas. For the dictionary type, the firstparameter specifies the key type, and the second parameter specifies the value
type. Julia was able to infer this based on the input, but we could have specified
it explicitly.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

420 a p p e n d i x a. j u l i ajulia> x = Dict{Int64,Int64}(3=>4, 5=>1)
Dict{Int64,Int64} with 2 entries:
3 => 4
5 => 1

It is possible to define our own parametric types, but we do not do that in thistext.

A.2 Functions
A function is an object that maps a tuple of argument values to a return value.
This section discusses how to define and work with functions.

A.2.1

Named Functions

One way to define a named function is to use the function keyword, followed by
the name of the function and a tuple of names of arguments.
function f(x, y)
return x + yend

We can also define functions compactly using assignment form.
julia> f(x, y) = x + y;julia> f(3, 0.1415)
3.1415A.2.2 Anonymous Functions
An anonymous function is not given a name, though it can be assigned to a named
variable. One way to define an anonymous function is to use the arrow operator.
julia> h = x -> x^2 + 1 # assign anonymous function to a variable#1 (generic function with 1 method)
julia> g(f, a, b) = [f(a), f(b)]; # applies function f to a and b and returns array
julia> g(h, 5, 10)
2-element Array{Int64,1}:
26
101
julia> g(x->sin(x)+1, 10, 20)
2-element Array{Float64,1}:
0.4559788891106302
1.9129452507276277© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a. 2. fu n c ti on s 421

A.2.3 Optional Arguments
We can specify optional arguments by setting default values.
julia> f(x = 10) = x^2;
julia> f()100
julia> f(3)9julia> f(x, y, z = 1) = x*y + z;
julia> f(1, 2, 3)
5
julia> f(1, 2)
3

A.2.4 Keyword Arguments
Functions with keyword arguments are defined using a semicolon.
julia> f(; x = 0) = x + 1;
julia> f()1
julia> f(x = 10)
11
julia> f(x, y = 10; z = 2) = (x + y)*z;
julia> f(1)22julia> f(2, z = 3)
36
julia> f(2, 3)
10
julia> f(2, 3, z = 1)5

A.2.5 Function Overloading
The types of the arguments passed to a function can be specified using the doublecolon operator. If multiple functions of the same name are provided, Julia will
execute the appropriate function.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

422 a p p e n d i x a. j u l i ajulia> f(x::Int64) = x + 10;
julia> f(x::Float64) = x + 3.1415;
julia> f(1)11julia> f(1.0)
4.141500000000001
julia> f(1.3)
4.4415000000000004

The implementation of the most specific function will be used.
julia> f(x) = 5;
julia> f(x::Float64) = 3.1415;
julia> f([3, 2, 1])
5
julia> f(0.00787499699)
3.1415

A.3 Control Flow
We can control the flow of our programs using conditional evaluation and loops.
This section provides some of the syntax used in the book.

A.3.1 Conditional Evaluation
Conditional evaluation will check the value of a Boolean expression and thenevaluate the appropriate block of code. One of the most common ways to do thisis with an if statement.
if x < y# run this if x < yelseif x > y# run this if x > yelse# run this if x == yend

We can also use the ternary operator with its question mark and colon syntax.
It checks the Boolean expression before the question mark. If the expression
evaluates to true, then it returns what comes before the colon; otherwise it returns
what comes after the colon.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a. 4 . p a c ka g e s 423

julia> f(x) = x > 0 ? x : 0;
julia> f(-10)
0julia> f(10)10A.3.2 Loops
A loop allows for repeated evaluation of expressions. One type of loop is the while
loop. It repeatedly evaluates a block of expressions until the specified condition
after the while keyword is met. The following example will sum the values inarray xx = [1, 2, 3, 4, 6, 8, 11, 13, 16, 18]s = 0
while x != []s += pop!(x)end

Another type of loop is the for loop. It uses the for keyword. The following
example will also sum over the values in the array x but will not modify x.
x = [1, 2, 3, 4, 6, 8, 11, 13, 16, 18]s = 0
for i = 1:length(x)s += x[i]endThe = can be substituted with in or ∈. The following code block is equivalent.
x = [1, 2, 3, 4, 6, 8, 11, 13, 16, 18]s = 0
for y in x
s += yend

A.4 Packages
A package is a collection of Julia code and possibly other external libraries thatcan be imported to provide additional functionality. Julia has a built-in package
manager. A list of registered packages can be found at https://pkg.julialang.org. To add a registered package like Distributions.jl, we can run:
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

424 a p p e n d i x a. j u l i a

Pkg.add("Distributions")

To update packages, we use:
Pkg.update()

To use a package, we use the keyword using:using Distributions

Several code blocks in this text specify a package import with using. Some
code blocks make use of functions that are not explicitly imported. For instance,
the var function is provided by Statistics.jl, and the golden ratio ϕ is definedin Base.MathConstants.jl. Other excluded packages are InteractiveUtils.jl,
LinearAlgebra.jl, QuadGK.jl, Random.jl, and StatsBase.jl.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

B Test Functions
Researchers in optimization use several test functions to evaluate optimization
algorithms. This section covers several test functions used throughout this book.

B.1

Ackley’s Function

Ackley’s function (figure B.1) is used to test a method’s susceptibility to getting
stuck in local minima. It is comprised of two primary components—a sinusoidal
component that produces a multitude of local minima and an exponential bell
curve centered at the origin, which establishes the function’s global minimum.
Figure B.1. The two-dimensionalversion of Ackley’s function. The
global minimum is at the origin.

30

20

x2

10

0

−10
−20
−30
−30

−20

−10

0
x1

10

20

30

426 a p p e n d i x b. te st fu n c ti on s

Ackley’s function is defined for any number of dimensions d:

 v
!
u d
d
u1
1
t
2
xi  − expcos(cxi ) + a + exp(1)f (x) = − a exp−b
d i∑
d i∑
=1
=1

(B.1)

with a global minimum at the origin with an optimal value of zero. Typically,a = 20, b = 0.2, and c = 2π. Ackley’s function is implemented in algorithm B.1.
function ackley(x, a=20, b=0.2, c=2π)
d = length(x)return -a*exp(-b*sqrt(sum(x.^2)/d)) exp(sum(cos.(c*xi) for xi in x)/d) + a + exp(1)endAlgorithm B.1. Ackley’s function
with d-dimensional input vector x
and three optional parameters.

B.2 Booth’s Function
Booth’s function (figure B.2) is a two-dimensional quadratic function.
Figure B.2. Booth’s function witha global minimum at [1, 3].

10

x2

5

0

−5

−10
−10

−5

0
x1

5

10

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

b. 3. bra n i n fu n c ti on 427

Its equation is given by
f (x) = ( x1 + 2x2 − 7)2 + (2x1 + x2 − 5)2(B.2)

with a global minimum at [1, 3] with an optimal value of zero. It is implemented
in algorithm B.2.
Algorithm B.2. Booth’s function
with two-dimensional input vectorx.

booth(x) = (x[1]+2x[2]-7)^2 + (2x[1]+x[2]-5)^2

B.3 Branin Function
The Branin function (figure B.3) is a two-dimensional function,f (x) = a( x2 − bx12 + cx1 − r )2 + s(1 − t) cos( x1 ) + s(B.3)

with recommended values a = 1, b = 5.1/(4π 2 ), c = 5/π, r = 6, s = 10, andt = 1/(8π ).
Figure B.3. The Branin function,with four global minima.

20

x2

15

10

5

0

−5

0

5

10

15

x1

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

428 a p p e n d i x b. te st fu n c ti on s

It has no local minima aside from global minima with x1 = π + 2πm for
integral m. Four of these minima are:
#)
# "
# "
# "
("
5π
3π
π
−π
(B.4)
,
,
,
12.8752.475
2.275
12.275
with f (x∗ ) ≈ 0.397887. It is implemented in algorithm B.3.
function branin(x; a=1, b=5.1/(4π^2), c=5/π, r=6, s=10, t=1/(8π))
return a*(x[2]-b*x[1]^2+c*x[1]-r)^2 + s*(1-t)*cos(x[1]) + s
end

Algorithm B.3. The Branin function with two-dimensional input
vector x and six optional parameters.

B.4 Flower Function
The flower function (figure B.4) is a two-dimensional test function whose contour
function has flower-like petals originating from the origin.
Figure B.4. The flower function.

3

2

x2

1

0

−1
−2
−3
−3

−2

−1

0
x1

1

2

3

The equation isf (x) = akxk + b sin(c tan−1 ( x2 , x1 ))(B.5)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

b. 5 . mi c ha l e w i c z fu n c ti on 429

with its parameters typically set to a = 1, b = 1, and c = 4.
The flower function is minimized near the origin but does not have a global
minimum due to atan being undefined at [0, 0]. It is implemented in algorithm B.4.
Algorithm B.4. The flower function
with two-dimensional input vector
x and three optional parameters.

function flower(x; a=1, b=1, c=4)
return a*norm(x) + b*sin(c*atan(x[2], x[1]))end

B.5 Michalewicz Function
The Michalewicz function (figure B.5) is a d-dimensional optimization function
with several steep valleys.
Figure B.5. The Michalewicz function.

4

x2

3

2

1

0

0

2
x1

1

3

Its equation isdf (x) = − ∑ sin( xi ) sin
i =1

2m

ixi2
π

4

!

(B.6)

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

430 a p p e n d i x b. te st fu n c ti on s

where the parameter m, typically 10, controls the steepness. The global minimum
depends on the number of dimensions. In two dimensions the minimum is at approximately [2.20, 1.57] with f (x∗ ) = −1.8011. It is implemented in algorithm B.5.
Algorithm B.5. The Michalewicz
function with input vector x andoptional steepness parameter m.

function michalewicz(x; m=10)
return -sum(sin(v)*sin(i*v^2/π)^(2m) for(i,v) in enumerate(x))
end

B.6 Rosenbrock’s Banana Function
The Rosenbrock function (figure B.6), also called Rosenbrock’s valley or Rosenbrock’s banana function, is a well-known unconstrained test function developedby Rosenbrock in 1960.1 It has a global minimum inside a long, curved valley. Most
optimization algorithms have no problem finding the valley but have difficulties
traversing along the valley to the global minimum.

H. H. Rosenbrock, “An Automatic
Method for Finding the Greatestor Least Value of a Function,”
The Computer Journal, vol. 3, no. 3,
pp. 175–184, 1960.
1

Figure B.6. The Rosenbrock function with a = 1 and b = 5. The
global minimum is at [1, 1].

2

x2

1

0

−1

−2
−2

−1

0
x1

1

2

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

b. 7. w he e l e r’s ri d g e

431

The Rosenbrock function is(B.7)

f (x) = ( a − x1 )2 + b( x2 − x12 )2

with a global minimum at [ a, a2 ] at which f (x∗ ) = 0. This text uses a = 1 andb = 5. The Rosenbrock function is implemented in algorithm B.6.
Algorithm B.6. The Rosenbrockfunction with two-dimensional input vector x and two optional parameters.

rosenbrock(x; a=1, b=5) = (a-x[1])^2 + b*(x[2] - x[1]^2)^2

B.7 Wheeler’s Ridge
Wheeler’s ridge (figure B.7) is a two-dimensional function with a single global
minimum in a deep curved peak. The function has two ridges, one along thepositive and one along the negative first coordinate axis. A gradient descentmethod will diverge along the negative axis ridge. The function is very flat awayfrom the optimum and the ridge.
Figure B.7. Wheeler’s ridge showing the two ridges and the peak
containing the global minimum.

6

x2

4
2
0

−2
0

5

10

15

20

25

x1

The function is given by
f (x) = − exp(−( x1 x2 − a)2 − ( x2 − a)2 )(B.8)

with a typically equal to 1.5, for which the global optimum of −1 is at [1, 3/2].
wheeler(x, a=1.5) = -exp(-(x[1]*x[2] - a)^2 -(x[2]-a)^2)

Wheeler’s ridge has a smooth contour plot (figure B.8) when evaluated overx1 ∈ [0, 3] and x2 ∈ [0, 3]. It is implemented in algorithm B.7.

x2

2

−5

0

0

2
x1

Figure B.8. A contour plot of theminimal region of Wheeler’s ridge.
Algorithm B.7. Wheeler’s ridge,
which takes in a two-dimensional
design point x and an optional
scalar parameter a.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

432 a p p e n d i x b. te st fu n c ti on s

B.8 Circle Function
The circle function (algorithm B.8) is a simple multiobjective test function given by"#
1 − r cos(θ )f(x) =(B.9)
1 − r sin(θ )where θ = x1 and r is obtained by passing x2 through
!
1 1
2x2
r= +
2 2 1 + x22(B.10)The Pareto frontier has r = 1 and mod (θ, 2π ) ∈ [0, π/2] or r = −1 andmod (θ, 2π ) ∈ [π, 3π/2].
function circle(x)θ = x[1]r = 0.5 + 0.5*(2x[2]/(1+x[2]^2))
y1 = 1 - r*cos(θ)
y2 = 1 - r*sin(θ)
return [y1, y2]
end

Algorithm B.8. The circle function,
which takes in a two-dimensional
design point x and produces a twodimensional objective value.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

C Mathematical Concepts
This appendix covers mathematical concepts used in the derivation and analysis
of optimization methods. These concepts are used throughout this book.

C.1 Asymptotic Notation
Asymptotic notation is often used to characterize the growth of a function. This
notation is sometimes called big-Oh notation, since the letter O is used because
the growth rate of a function is often called its order. This notation can be used
to describe the error associated with a numerical method or the time or spacecomplexity of an algorithm. This notation provides an upper bound on a function
as its argument approaches a certain value.
Mathematically, if f ( x ) = O( g( x )) as x → a then the absolute value of f ( x ) isbounded by the absolute value of g( x ) times some positive and finite c for valuesof x sufficiently close to a:| f ( x )| ≤ c| g( x )| for x → a(C.1)

Writing f ( x ) = O( g( x )) is a common abuse of the equal sign. For example,

x2 = O( x2 ) and 2x2 = O( x2 ), but, of course, x2 6= 2x2 . In some mathematicaltexts, O( g( x )) represents the set of all functions that do not grow faster than g( x ).
One might write, for example, 5x2 ∈ O( x2 ). An example of asymptotic notation
is given in example C.1.
If f ( x ) is a linear combination1 of terms, then O( f ) corresponds to the order ofthe fastest growing term. Example C.2 compares the orders of several terms.

1
A linear combination is aweighted sum of terms. If
the terms are in a vector x,
then the linear combination is
w1 x1 + w2 x2 + · · · = w⊤ x.

434 a p p e n d i x c . ma the ma ti c a l c on c e p ts

Consider f ( x ) = 106 e x as x → ∞. Here, f is a product of constant 106 ande x . The constant can simply be incorporated into the bounding constant c:

Example C.1. Asymptotic notation
for a constant times a function.

| f ( x )| ≤ c| g( x )|106 |e x | ≤ c| g( x )||e x | ≤ c| g( x )|Thus, f = O(e x ) as x → ∞.

Consider f ( x ) = cos( x ) + x + 10x3/2 + 3x2 . Here, f is a linear combinationof terms. The terms cos( x ), x, x3/2 , x2 are arranged in order of increasing
value as x approaches infinity. We plot f ( x ) along with c| g( x )|, where c has
been chosen for each term such that c| g( x = 2)| exceeds f ( x = 2).

Example C.2. An illustration offinding the order of a linear combination of terms.

1010

40

105

20

0
0

0.5

1
x

100
100

1.5

2

f (x)

16x3/2

102

101

103

x
12x2

6x3

There is no constant c such that f ( x ) is always less than c| x3/2 | for sufficiently large values of x. The same is true for cos( x ) and x.
We find that f ( x ) = O( x3 ), and in general f ( x ) = O( x m ) for m ≥ 2, alongwith other function classes like f ( x ) = e x . We typically discuss the order thatprovides the tightest upper bound. Thus, f = O( x2 ) as x → ∞.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c . 2. ta y l or e x p a n si on 435

C.2 Taylor Expansion
The Taylor expansion, also called the Taylor series, of a function is critical to understanding many of the optimization methods covered in this book, so we derive it
here.
From the first fundamental theorem of calculus,2 we know thatf ( x + h) = f ( x ) +Z h0f ′ ( x + a) da

Nesting this definition produces the Taylor expansion of f about x:

Z h
Z a
f ( x + h) = f ( x ) +f ′ (x) +f ′′ ( x + b) db da00
Z hZ a(C.2)

f (b) − f ( a) =(C.4)

f ′′′ ( x + c) dc db da(C.6)= f ( x) + f ′ ( x)h +
..
.

0

0
0
f ′′ ( x )

2!

0

h2 +

Z hZ aZ b
0

0

0

a

f ′ ( x ) dx

(C.5)

(C.7)

f ′ (x)h+1!
∞
f (n) ( x ) n
= ∑
h
n!
n =0

= f (x) +

0

Z b

(C.3)

f ′′ ( x + b) db daZ h Z a
Z b
′
′′′
′′
= f ( x) + f ( x)h +f ( x + c) dc db daf (x) += f ( x) + f ′ ( x)h +The first fundamental theorem ofcalculus relates a function to theintegral of its derivative:

2f ′′ ( x )
2!

h2 +

f ′′′ ( x )3!

h3 + . . .

(C.8)
(C.9)

In the formulation above, x is typically fixed and the function is evaluated interms of h. It is often more convenient to write the Taylor expansion of f ( x ) abouta point a such that it remains a function of x:∞

f (n) ( a )( x − a)nn!
n =0

f (x) = ∑(C.10)The Taylor expansion represents a function as an infinite sum of polynomial
terms based on repeated derivatives at a single point. Any analytic function canbe represented by its Taylor expansion within a local neighborhood.
A function can be locally approximated by using the first few terms of the Taylor
expansion. Figure C.1 shows increasingly better approximations for cos( x ) about
x = 1. Including more terms increases the accuracy of the local approximation,but error still accumulates as one moves away from the expansion point.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

436 a p p e n d i x c . ma the ma ti c a l c on c e p ts

3cos( x )0th degree
1st degree
2nd degree
3rd degree
4th degree
5th degree21Figure C.1. Successive approximations of cos( x ) about 1 based on
the first n terms of the Taylor expansion.

0

−1
−2
−3
−4

−2

0

2

4

6

x

A linear Taylor approximation uses the first two terms of the Taylor expansion:f ( x ) ≈ f ( a) + f ′ ( a)( x − a)(C.11)

A quadratic Taylor approximation uses the first three terms:f ( x ) ≈ f ( a) + f ′ ( a)( x − a) +1f ”( a)( x − a)22(C.12)and so on.
In multiple dimensions, the Taylor expansion about a generalizes to1
f (x) = f (a) + ∇ f (a)⊤ (x − a) + (x − a)⊤ ∇2 f (a)(x − a) + . . .
2(C.13)

The first two terms form the tangent plane at a. The third term incorporates
local curvature. This text will use only the first three terms shown here.

C.3 Convexity
A convex combination of two vectors x and y is the result of
αx + (1 − α)y(C.14)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c . 3. c on ve x i ty 437

for some α ∈ [0, 1]. Convex combinations can be made from m vectors,
w1 v(1) + w2 v(2) + · · · + w m v( m )(C.15)

αx + (1 − α)y ∈ S .

(C.16)

with nonnegative weights w that sum to one.
A convex set is a set for which a line drawn between any two points in the set isentirely within the set. Mathematically, a set S is convex if we havefor all x, y in S and for all α in [0, 1]. A convex and a nonconvex set are shown infigure C.2.

Figure C.2.
convex sets.

A convex set

Convex and non-Not a convex set

A convex function is a bowl-shaped function whose domain is a convex set. By
bowl-shaped, we mean it is a function such that any line drawn between two
points in its domain does not lie below the function. A function f is convex overa convex set S if, for all x, y in S and for all α in [0, 1],f (αx + (1 − α)y) ≤ α f (x) + (1 − α) f (y)(C.17)

Convex and concave regions of a function are shown in figure C.3.
Figure C.3. Convex and nonconvex
portions of a function.

f (x)ConvexConcavex

A function f is strictly convex over a convex set S if, for all x, y in S and α in(0, 1),f (αx + (1 − α)y) < α f (x) + (1 − α) f (y)(C.18)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

438 a p p e n d i x c . ma the ma ti c a l c on c e p ts

Strictly convex functions have at most one minimum, whereas a convex function
can have flat regions.3 Examples of strict and nonstrict convexity are shown infigure C.4.

Optimization of convex functions
is the subject of the textbook by S.
Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2004.
3

exx∗Strictly convex function with
one global minimum.

Convex function without aunique global minimum.

Strictly convex function
without a global minimum.

A function f is concave if − f is convex. Furthermore, f is strictly concave if − fis strictly convex.
Not all convex functions are unimodal and not all unimodal functions areconvex, as shown in figure C.5.

Figure C.4. Not all convex functions have single global minima.

ex
Unimodal and convex

Unimodal but nonconvex

Convex but nonunimodal
Figure C.5. Convexity and unimodality are not the same thing.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c . 4 . n orms 439

C.4 Norms
A norm is a function that assigns a length to a vector. To compute the distance
between two vectors, we evaluate the norm of the difference between those two
vectors. For example, the distance between points a and b using the Euclidean
norm isq(C.19)
ka − bk2 = ( a1 − b1 )2 + ( a2 − b2 )2 + · · · + ( an − bn )2A function f is a norm if4

1. f (x) = 0 if and only if a is the zero vector.

Some properties that follow fromthese axioms include:4f (−x) = f (x)2. f ( ax) = | a| f (x), such that lengths scale.

f (x) ≥ 0

3. f (a + b) ≤ f (a) + f (b), also known as the triangle inequality.

The L p norms are a commonly used set of norms parameterized by a scalarp ≥ 1. The Euclidean norm in equation (C.19) is the L2 norm. Several L p norms
are shown in table C.1.
The L p norms are defined according to:1

kxk p = lim (| x1 |ρ + | x2 |ρ + · · · + | xn |ρ ) ρ
ρ→ p(C.20)

where the limit is necessary for defining the infinity norm, L∞ .5

C.5

The L∞ norm is also referred to as
the max norm, Chebyshev distance, orchessboard distance. The latter namecomes from the minimum number of moves a chess king needs tomove between two chess squares.
5

Matrix Calculus

This section derives two common gradients: ∇x b⊤ x and ∇x x⊤ Ax.
To obtain ∇x b⊤ x, we first expand the dot product:b⊤ x = [b1 x1 + b2 x2 + · · · + bn xn ](C.21)The partial derivative with respect to a single coordinate is:Thus, the gradient is:∂ ⊤b x = bi
∂xi

(C.22)∇ x b⊤ x = ∇ x x⊤ b = b(C.23)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

440 a p p e n d i x c . ma the ma ti c a l c on c e p ts

L1 : kxk1 = | x1 | + | x2 | + · · · + | x n |L2 : kxk2 =q

Table C.1. Common L p norms.
The illustrations show the shape ofthe norm contours in two dimensions. All points on the contour are
equidistant from the origin underthat norm.

x12 + x22 + · · · + xn2

L∞ : kxk∞ = max(| x1 |, | x2 |, · · · , | xn |)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c . 5 . ma tri x c a l c u l u s 441

To obtain ∇x x⊤ Ax for a square matrix A, we first expand x⊤ Ax:  ⊤ 
a11 a12 · · · a1n
x1
x
 1
x  
a
a
·
·
·
a


22
2n
21
x2 
 

x⊤ Ax =  2  
 
.
.
.
.


..
..
..  · · ·· · ·  ..
xn
xn
an1 an2 · · · ann ⊤
x1 a11 + x2 a12 + · · · + xn a1nx1
x     x1 a21 + x2 a22 + · · · + xn a2n =  2 ..
· · · .
xn
x1 an1 + x2 an2 + · · · + xn ann
x12 a11 + x1 x2 a12 + · · · + x1 xn a1n +x1 x2 a21 + x22 a22 + · · · + x2 xn a2n +=..
.
x1 xn an1 + x2 xn an2 + · · · + xn2 ann(C.24)(C.25)(C.26)The partial derivative with respect to the ith component is

The gradient is thus:n∂ ⊤x Ax = ∑ x j aij + a ji
∂xi
j =1


∑nj=1 x j a1j + a j1
 n x a + a 
 ∑ j=1 j 2jj2 
∇x x⊤ Ax = 
..




.

n
∑ j=1 x j anj + a jn
a11 + a11 a12 + a21
 a21 + a12 a22 + a22
=
..
..


.
.
an1 + a1n an2 + a2n= A + A⊤ x
(C.27)(C.28)

···
···
..
.
···

 
a1n + an1x1
 
a2n + an2   x2  . 
..
 . 
 . 
.
ann + ann
xn

(C.29)(C.30)© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

442 a p p e n d i x c . ma the ma ti c a l c on c e p ts

C.6 Positive DefinitenessThe notion of a matrix being positive definite or positive semidefinite often arises
in linear algebra and optimization for a variety of reasons. For example, if thematrix A is positive definite in the function f (x) = x⊤ Ax, then f has a uniqueglobal minimum.
Recall that the quadratic approximation of a twice-differentiable function f at
x0 is
1
f (x) ≈ f (x0 ) + ∇ f (x0 )⊤ (x − x0 ) + (x − x0 )⊤ H0 (x − x0 )2

(C.31)

where H0 is the Hessian of f evaluated at x0 . Knowing that (x − x0 )⊤ H0 (x − x0 )has a unique global minimum is sufficient to determine whether the overall
quadratic approximation has a unique global minimum.6 .
A symmetric matrix A is positive definite if x⊤ Ax is positive for all points otherthan the origin: x⊤ Ax > 0 for all x 6= 0.
A symmetric matrix A is positive semidefinite if x⊤ Ax is always non-negative:x⊤ Ax ≥ 0 for all x.

Gaussian Distribution

The probability density function for a univariate Gaussian,7 also called the normal
distribution, is:( x − µ )21e− 2νN ( x | µ, ν) = √(C.32)
2πν
where µ is the mean and ν is the variance.8 This distribution is plotted in figure C.6.
The cumulative distribution function of a distribution maps x to the probability
that drawing a value from that distribution will produce a value less than or equal
to x. For a univariate Gaussian, the cumulative distribution function is given by


x−µ
1 1
√
Φ( x ) ≡ + erf(C.33)2 2
σ 2where erf is the error function:2
erf( x ) ≡ √
π

Z x
0

2

e−τ dτ

(C.34)The multivariate Gaussian is discussed in chapter 8 and chapter 15.
The univariate Gaussian is usedthroughout.
7

The variance is the standard deviation squared.
8p( x | µ, σ )C.7

The component f (x0 ) merely
shifts the function vertically. The
component ∇ f (x0 )⊤ (x − x0 ) is alinear term which is dominated by
the quadratic term.

6µ−σ µ µ+σx

Figure C.6. A univariate Gaussian
distribution, N (µ, ν).
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c . 8 . g a u ssi a n q u a d ra tu re 443

C.8 Gaussian Quadrature
Gaussian quadrature is a technique for approximating integrals using a weighted
sum of function evaluations.9 The general form of the approximation is
Z b
a

m

p( x ) f ( x ) dx ≈ ∑ wi f ( xi )(C.35)For a detailed overview of Gaussian quadrature, see J. Stoer and R.
Bulirsch, Introduction to Numerical
Analysis, 3rd ed. Springer, 2002.
9

i =1

where p( x ) is a known nonnegative weight function10 over the finite or infinite
interval [ a, b].
An m-point quadrature rule is a unique choice of points xi ∈ ( a, b) and weights
wi > 0 for i ∈ {1, . . . , m} that define a Gaussian quadrature approximation suchthat any polynomial of degree 2m − 1 or less is integrated exactly over [ a, b] withthe given weight function.
Given a domain and a weight function, we can compute a class of orthogonal
polynomials. We will use bi ( x ) to denote an orthogonal polynomial11 of degree i.
Any polynomial of degree m can be represented as a linear combination of theorthogonal polynomials up to degree m. We form a quadrature rule by selecting m
points xi to be the zeros of the orthogonal polynomial pm and obtain the weights
by solving the system of equations:R b p( x )b ( x )2 dx for k = 0
m
0
b
(
x
)
w
=
(C.36)∑ k i i a0
for k = 1, . . . , m − 1
i =1

The weight function is often
a probability density function inpractice.

10

Orthogonal polynomials are covered in chapter 18.

11

Gauss solved equation (C.36) for the interval [−1, 1] and the weighting functionp( x ) = 1. The orthogonal polynomials for this case are the Legendre polynomials.
Algorithm C.1 implements Gaussian quadrature for Legendre polynomials andexample C.3 works out a quadrature rule for integration over [−1, 1].
We can transform any integral over the bounded interval [ a, b] to an integral
over [−1, 1] using the transformation


Z
Z b
b−a
a+b
b−a 1
x+
f
dx
(C.37)f ( x ) dx =
2
2
2
−1
a
Quadrature rules can thus be precalculated for the Legendre polynomials andthen applied to integration over any finite interval.12 Example C.4 applies sucha transformation and algorithm C.2 implements integral transformations in a
Gaussian quadrature method for finite domains.

Similar techniques can be applied to integration over infinite intervals, such as [0, ∞) using the Laguerre polynomials and (−∞, ∞)using the Hermite polynomials.

12© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

444 a p p e n d i x c . ma the ma ti c a l c on c e p ts

Consider the Legendre polynomials for integration over [−1, 1] with theweight function p( x ) = 1. Suppose our function of interest is well approximated by a fifth degree polynomial. We construct a 3-point quadrature rule,
which produces exact results for polynomials up to degree 5.
The Legendre polynomial of degree 3 is Le3 ( x ) = 52 x3 − 23 x, which has√√roots at x1 = − 3/5, x2 = 0, and x3 = 3/5. The Legendre polynomialsof lesser degree are Le0 ( x ) = 1, Le1 ( x ) = x, and Le2 ( x ) = 23 x2 − 12 . The
weights are obtained by solving the system of equations:   R 1
√√w1
Le0 (− 3/5) Le0 (0) Le0 ( 3/5)
Le0 ( x )2 dx
−
1
√
√

  
0Le1 (− 3/5) Le1 (0) Le1 ( 3/5) w2  = √√w3
Le2 (− 3/5) Le2 (0) Le2 ( 3/5)0   
1
1
1
w1
2
√
 √
   
=
−
w
3/5
0
3/5

  2  0
0
w3
4/10
−1/2 4/10

Example C.3. Obtaining a 3-term
quadrature rule for exactly integrating polynomials up to degree
5.

which yields w1 = w3 = 5/9 and w2 = 8/9.
Consider integrating the 5th degree polynomial f ( x ) = x5 − 2x4 + 3x3 +R1
5x2 − x + 4. The exact value is −1 p( x ) f ( x ) dx = 158/15 ≈ 10.533. The
quadrature rule produces the same value:
r !
r !
3
5
3
3
8
5
≈ 10.533.
∑ wi f ( x i ) = 9 f − 5 + 9 f (0) + 9 f
5
i =1© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

c . 8 . g a u ssi a n q u a d ra tu re 445

Consider integrating f ( x ) = x5 − 2x4 + 3x3 + 5x2 − x + 4 over [−3, 5]. We
can transform this into an integration over [−1, 1] using equation (C.37):


Z
Z 1
Z 5
5−3
5+3
5+3 1
dx = 4x+ff (4x + 1) dx
f ( x ) dx =
2
2
2
−1
−1
−3

Example C.4. Integrals over finite
regions can be transformed into
integrals over [−1, 1] and solved
with quadrature rules for Legendre polynomials.

We use the 3-term quadrature rule obtained in example C.3 to integrateg( x ) = 4 f (4x + 1) = 4096x5 + 3072x4 + 1280y3 + 768y2 + 240y + 40 over[−1, 1]:Z 1−1p( x ) g( x ) dx =

 8

5  √
5 √
3/5 = 1820.8
g − 3/5 + g(0) + g
999

struct Quadrule
ws
xs
end
function quadrule_legendre(m)bs = [legendre(i) for i in 1 : m+1]
xs = roots(bs[end])
A = [bs[k](xs[i]) for k in 1 : m, i in 1 : m]b = zeros(m)b[1] = 2
ws = A\breturn Quadrule(ws, xs)
end

Algorithm C.1. A method for constructing m-point Legendre quadrature rules over [−1, 1]. The resulting type contains both the nodes
xs and the weights ws.

quadint(f, quadrule) =sum(w*f(x) for (w,x) in zip(quadrule.ws, quadrule.xs))function quadint(f, quadrule, a, b)
α = (b-a)/2
β = (a+b)/2
g = x -> α*f(α*x+β)
return quadint(g, quadrule)
end

Algorithm C.2.
The function
quadint for integrating a univariate function f with a given quadrature rule quadrule over the finite
domain [ a, b].

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

D Solutions
Exercise 1.1: f ( x ) = x3 /3 − x at x = 1.
Exercise 1.2: It does not have a minimum, that is, the function is said to be
unbounded below.
Exercise 1.3: No. Consider minimizing f ( x ) = x, subject to x ≥ 1.
Exercise 1.4: The function f can be broken into two separate functions that depend
only on their specific coordinate:f ( x, y) = g( x ) + h(y)where g( x ) = x2 and h(y) = y. Both g and h strictly increase for x, y ≥ 1. While his minimized for y = 1, we can merely approach x → 1 due to the strict inequalityx > y. Thus, f has no minima.
Exercise 1.5: An inflection point is a point on a curve where the sign of thecurvature changes. A necessary condition for x to be an inflection point is thatthe second derivative is zero. The second derivative is f ′′ ( x ) = 6x, which is only
zero at x = 0.
A sufficient condition for x to be an inflection point is that the second derivativechanges sign around x. That is, f ′′ ( x + ǫ) and f ′′ ( x − ǫ) for ǫ ≪ 1 have opposite
signs. This holds for x = 0, so it is an inflection point.
There is thus only one inflection point on x3 − 10.
Exercise 2.1: An entry of the Hessian can be computed using the forward difference method:∇ f (x + he j )i − ∇ f ( x )i
∂2 f (x)Hij =≈∂xi ∂x jhwhere e j is the jth basis vector with its jth component equal to one and all otherentries are zero.

448 a p p e n d i x d . sol u ti on s

We can thus approximate the jth column of the Hessian using:H· j ≈∇ f (x + he j ) − ∇ f ( x )hThis procedure can be repeated for each column of the Hessian.
Exercise 2.2: It requires two evaluations of the objective function.
Exercise 2.3: f ′ ( x ) = 1x + e x − x12 . When x is close to zero, we find that x < 1.
Hence 1x > 1, and finally x12 > 1x > 0, so − x12 dominates.
Exercise 2.4: From the complex step method, we have f ′ ( x ) ≈ Im(2 + 4ih)/h =4h/h = 4.
Exercise 2.5: See the picture below:
1

x
0
1

y

(·)2

1

c1
2

1

+

2

c2
2

sin

sin 2
c3
2 cos 2

Exercise 2.6: The second-order derivative can be approximated using the central
difference on the first-order derivative:f ′′ ( x ) ≈f ′ ( x + h/2) − f ′ ( x − h/2)hfor small values of h.
Substituting in the forward and backwards different estimates of f ′ ( x + h/2)
and f ′ ( x − h/2) yields:f ′′ ( x ) ≈f ( x + h/2+ h/2)− f ( x + h/2− h/2)
− f ( x−h/2+h/2)−h f ( x−h/2−h/2)hhf ( x + h) − f ( x )f ( x ) − f ( x − h)
=
−
h2
h2
f ( x + h) − 2 f ( x ) + f ( x − h)=h2

Exercise 3.1: Fibonacci search is preferred when derivatives are not available.
Exercise 3.2: The Shubert-Piyavskii method needs the Lipschitz constant, which
may not be known.
Exercise 3.3: f ( x ) = x2 . Since the function is quadratic, after three evaluations,the quadratic model will represent this function exactly.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 449

Exercise 3.4: We can use the bisection method to find the roots of f ′ ( x ) = x − 1.
After the first update, we have [0, 500]. Then, [0, 250]. Finally, [0, 125].
Exercise 3.5: No, the Lipschitz constant must bound the derivative everywhereon the interval, and f ′ (1) = 2(1 + 2) = 6.
Exercise 3.6: No. The best you can do is use Fibonacci Search and shrink theuncertainty by a factor of 3; that is, to (32 − 1)/3 = 10 31 .
Exercise 4.1: Consider running a descent method on f ( x ) = 1/x for x > 0.
The minimum does not exist and the descent method will forever proceed in
the positive x direction with ever-increasing step sizes. Thus, only relying ona step-size termination condition would cause the method to run forever. Also
terminating based on gradient magnitude would cause it to terminate.
A descent method applied to f ( x ) = − x will also forever proceed in thepositive x direction. The function is unbounded below, so neither a step-size
termination condition nor a gradient magnitude termination condition would
trigger. It is common to include an additional termination condition to limit the
number of iterations.
Exercise 4.2: Applying the first Wolfe condition to our objective function yields
6 + (−1 + α)2 ≤ 7 − 2α · 10−4 , which can be simplified to α2 − 2α + 2 · 10−4 α ≤ 0.
This equation can be solved to obtain α ≤ 2(1 − 10−4 ). Thus, the maximum step
length is α = 1.9998.
Exercise 5.1: ∇ f (x) = 2Ax + b.
Exercise 5.2: The derivative is f ′ ( x ) = 4x3 . Starting from x (1) = 1:
f ′ (1) = 4′→ x (2) = 1 − 4 = − 3

f (−3) = 4 · (−27) = −108

→x

(3)

= −3 + 108 = 105

(D.1)
(D.2)

Exercise 5.3: We have f ′ ( x ) = e x − e− x ≈ e x for large x. Thus f ′ ( x (1) ) ≈ e10 and
x (2) ≈ −e10 . If we apply an exact line search, x (2) = 0. Thus, without a line search
we are not guaranteed to reduce the value of the objective function.
Exercise 5.4: The Hessian is 2H, and∇q(d) = d⊤ H + H⊤ + b = d⊤ (2H) + b.
The gradient is b when d = 0. The conjugate gradient method may diverge
because H is not guaranteed to be positive definite.
Exercise 5.5: Nesterov momentum looks at the point where you will be after the
update to compute the update itself.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

450 a p p e n d i x d . sol u ti on s

Exercise 5.6: The conjugate gradient method implicitly reuses previous information about the function and thus may enjoy better convergence in practice.
Exercise 5.7: The conjugate gradient method initially follows the steepest descentdirection. The gradient is∇ f ( x, y) = [2x + y, 2y + x ](D.3)

which for ( x, y) = (1, 1) is [3, 3]. The direction of steepest descent is opposite thegradient, d(1) = [−3, −3].
The Hessian is
#
"
2 1
(D.4)
1 2

Since the function is quadratic and the Hessian is positive definite, the conjugate
gradient method converges in at most two steps. Thus, the resulting point after
two steps is the optimum, ( x, y) = (0, 0), where the gradient is zero.
Exercise 5.8: No. If exact minimization is performed, then the descent directions
between steps are orthogonal, but [1, 2, 3]⊤ [0, 0, −3] 6= 0.
Exercise 6.1: Second-order information can guarantee that one is at a local minimum, whereas a gradient of zero is necessary but insufficient to guarantee local
optimality.
Exercise 6.2: We would prefer Newton’s method if we start sufficiently close to
the root and can compute derivatives analytically. Newton’s method enjoys abetter rate of convergence.
Exercise 6.3: f ′ ( x ) = 2x, f ′′ ( x ) = 2. Thus, x (2) = x (1) − 2x (1) /2 = 0; that is, youconverge in one step from any starting point.
Exercise 6.4: Since ∇ f (x) = Hx, ∇2 f (x) = H, and H is nonsingular, it follows
that x(2) = x(1) − H−1 Hx(1) = 0. That is, Newton’s method converges in one step.
Gradient descent diverges:x(2) = [1, 1] − [1, 1000] = [0, −999]x(3)= [0, −999] − [0, −1000 · 999] = [0, 998001]

(D.5)
(D.6)

Conjugate gradient descent uses the same initial search direction as gradient descent and converges to the minimum in the second step because the optimization
objective is quadratic.
Exercise 6.5: The left plot shows convergence for Newton’s method approaching
floating-point resolution within nine iterations. The secant method is slower toconverge because it can merely approximate the derivative.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 451

The right plot shows the projection of the exact and approximate tangent lines
with respect to f ′ for each method. The secant method’s tangent lines have ahigher slope, and thus intersect the x-axis prematurely.
105

Newtonsecant

0

−50

f

f′

10−9

−100

10−23

−150
10−37

2

4

6
iteration

8

10

−3

−2

x

−1

0

Exercise 6.6: Consider the sequence x (k+1) = x (k) /2 starting from x (1) = −1 on
the function f ( x ) = x2 − x. Clearly the sequence converges to x = 0, the values
for f ( x ) are decreasing, and yet the sequence does not converge to a minimizer.
Exercise 6.7: It does not need computation or knowledge of the entries of the
Hessian, and hence does not require solving a linear system at each iteration.
Exercise 6.8: The BFGS update does not exist when δ⊤ γ ≈ 0. In that case, simply
skip the update.
Exercise 6.9: The objective function is quadratic and can thus be minimized in one
step. The gradient is ∇ f = [2( x1 + 1), 2( x2 + 3)], which is zero at x∗ = [−1, −3].
The Hessian is positive definite, so x∗ is the minimum.
Exercise 6.10: The new approximation has the form

⊤ 
⊤

 1

f ( k +1) (x ) = y ( k +1) + g( k +1)
x − x( k +1) H( k +1) x − x( k +1)
x − x( k +1) +2

using the true function value and gradient at x(k+1) but requires an updated
Hessian H(k+1) . This form automatically satisfies f (k+1) (x(k+1) ) = y(k+1) and∇ f (k+1) (x(k+1) ) = g(k+1) . We must select the new Hessian to satisfy the third© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

452 a p p e n d i x d . sol u ti on s

condition:∇ f ( k +1) (x( k ) ) = g( k +1) + H( k +1) x( k ) − x( k +1)= g( k +1) − H( k +1) x( k +1) − x( k )= g( k +1) − H( k +1) δ( k +1)= g( k )We can rearrange and substitute to obtain:
H( k +1) δ( k +1) = γ( k +1)
Recall that a matrix A is positive definite if for every nonzero vector x⊤ Ax > 0.
If we multiply the secant equation by δ(k+1) we obtain the curvature condition:


⊤
⊤
δ( k +1) H( k +1) δ( k +1) = δ( k +1) γ( k +1) > 0

We seek a new positive definite matrix H(k+1) . All positive definite matrices
are symmetric, so calculating a new positive definite matrix requires specifyingn(n + 1)/2 variables. The secant equation imposes n conditions on these variables,
leading to an infinite number of solutions. In order to have a unique solution,
we choose the positive definite matrix closest to H(k) . This objective leads to thedesired optimization problem.
Exercise 7.1: The derivative has n terms whereas the Hessian has n2 terms. Eachderivative term requires two evaluations when using finite difference methods:
f (x) and f (x + he(i) ). Each Hessian term requires an additional evaluation whenusing finite difference methods:f (x + he(i) + he( j) ) − f (x + he(i) ) − f (x + he( j) ) + f ( x )∂2 f≈∂xi ∂x jh2

Thus, to approximate the gradient, you need n + 1 evaluations, and to approximate
the Hessian you need on the order of n2 evaluations.
Approximating the Hessian is prohibitively expensive for large n. Direct methods can take comparatively more steps using n2 function evaluations, as direct
methods need not estimate the derivative or Hessian at each step.
Exercise 7.2: Consider minimizing f ( x ) = xy and x0 = [0, 0]. Proceeding in either
canonical direction will not reduce the objective function, but x0 is clearly not aminimizer.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 453

Exercise 7.3: At each iteration, the Hooke-Jeeves method samples 2n points along
the coordinate directions with a step-size a. It stops when none of the points
provides an improvement and the step size is no more than a given tolerance ǫ.
While this often causes the Hooke-Jeeves method to stop when it has converged
to within ǫ of a local minimum, that need not be the case. For example, a valley
can descend between two coordinate directions farther than ǫ before arriving at a
local minimum, and the Hooke-Jeeves method would not detect it.
Exercise 7.4: Minimizing the drag of an airfoil subject to a minimum thickness(to preserve structural integrity). Evaluating the performance of the airfoil using
computational fluid dynamics involves solving partial differential equations. Because the function is not known analytically, we are unlikely to have an analyticalexpression for the derivative.
Exercise 7.5: The divided rectangles method samples at the center of the intervals
and not where the bound derived from a known Lipschitz constant is lowest.
Exercise 7.6: It cannot be cyclic coordinate search since more than one component
is changing. It can be Powell’s method.
Exercise 8.1: The cross-entropy method must fit distribution parameters withevery iteration. Unfortunately, no known analytic solutions for fitting multivariate
mixture distributions exist. Instead, one commonly uses the iterative expectation
maximization algorithm to converge on an answer.
Exercise 8.2: If the number of elite samples is close to the total number of samples,
then the resulting distribution will closely match the population. There will not
be a significant bias toward the best locations for a minimizer, and so convergence
will be slow.
Exercise 8.3: The derivative of the log-likelihood with respect to ν is:


∂
∂
1
( x − µ )21ℓ( x | µ, ν) =− ln 2π − ln ν −
∂ν
∂ν
2
2
2ν

=−

1
( x − µ )2
+
2ν
2ν2

The second term will be zero if the mean is already optimal. Thus, the derivativeis −1/2ν and decreasing ν will increase the likelihood of drawing elite samples.
Unfortunately, ν is optimized by approaching arbitrarily close to zero. The asymptote near zero in the gradient update will lead to large step sizes, which cannot
be taken as ν must remain positive.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

454 a p p e n d i x d . sol u ti on s

Exercise 8.4: The probability density of a design x under a multivariate normal
distribution with mean µ and covariance Σ is


1
1
⊤ −1
p(x | µ, Σ) =exp − (x − µ) Σ (x − µ)2(2π |Σ|)1/2

We can simplify the problem by maximizing the log-likelihood instead.1 The
log-likelihood is:11
ln p(x | µ, Σ) = − ln(2π |Σ|) − (x − µ)⊤ Σ−1 (x − µ)221  ⊤ −1
1x Σ x − 2x⊤ Σ−1 µ + µ⊤ Σ−1 µ= − ln(2π |Σ|) −2
2

The log function is concave forpositive inputs, so maximizing
log f ( x ) also maximizes a strictly
positive f ( x ).

1

We begin by maximizing the log-likelihood of the m individuals with respect
to the mean:mℓ µ | x(1) , · · · , x(m) = ∑ ln p(x(i) | µ, Σ)
i =1
m



 ⊤
1
1  ( i )  ⊤ −1 ( i )= ∑ − ln(2π |Σ|) −x
Σ x − 2 x( i ) Σ −1 µ + µ⊤ Σ −1 µ
2
2
i =1


We compute the gradient using the facts that ∇z z⊤ Az = A + A⊤ z, that∇z a⊤ z = a, and that Σ is symmetric and positive definite, and thus Σ−1 issymmetric:

  





m
⊤
1
∇ µ ℓ µ | x(1) , · · · , x( m ) = ∑ − ∇ µ − 2 x( i ) Σ −1 µ + ∇ µ µ⊤ Σ −1 µ
2
i =1

 


m 
⊤
1
= ∑ ∇ µ x( i ) Σ −1 µ − ∇ µ µ⊤ Σ −1 µ
2
i =1
m

= ∑ Σ −1 x( i ) − Σ −1 µi =1

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 455

We set the gradient to zero:m

0 = ∑ Σ −1 x( i ) − Σ −1 µ
i =1
m

m

∑ µ = ∑ x( i )
i =1
m

i =1

mµ = ∑ x(i)
i =1

µ=

1 m (i )xm i∑=1

Next we maximize with respect to the inverse covariance, Λ = Σ−1 , using thefact that |A−1 | = 1/|A| with b(i) = x(i) − µ:
 1 
⊤ 


m
1 
−1
(i )
(i )
(1)
(m)
−
x −µ Λ x −µℓ Λ | µ, x , · · · , x= ∑ − ln 2π |Λ|
2
2
i =1
m
1
1  (i )  ⊤ (i )= ∑ ln(|Λ|) −
b
Λb
2
2
i =1
We compute the gradient using the facts that ∇A |A| = |A|A−⊤ and ∇A z⊤ Az =

zz⊤ :



∇Λ ℓ Λ | µ, x(1),··· ,x

(m)



m



1  (i )  ⊤ (i )1ln(|Λ|) −bΛb= ∑ ∇Λ
2
2
i =1

⊤
m
1
1
∇ Λ | Λ | − b( i ) b( i )= ∑2| Λ |
2
i =1

⊤
m
1
1
= ∑
|Λ|Λ−⊤ − b(i) b(i)2| Λ |
2
i =1


⊤
1 m
= ∑ Λ−⊤ − b(i) b(i)2 i =1⊤
1 m= ∑ Σ − b( i ) b( i )2 i =1© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

456 a p p e n d i x d . sol u ti on s

and set the gradient to zero:⊤1 m
Σ − b( i ) b( i )∑2 i =1⊤mm∑ Σ = ∑ b( i ) b( i )
0=

i =1

Σ=

i =1
m 


⊤
1
x( i ) − µ x( i ) − µ∑m i =1

Exercise 9.1: To bias survival to the fittest by biasing the selection toward theindividuals with better objective function values.
Exercise 9.2: Mutation drives exploration using randomness. It is therefore essential in order to avoid local minima. If we suspect there is a better solution,
we would need to increase the mutation rate and let the algorithm have time todiscover it.
Exercise 9.3: Increase the population size or the coefficient that biases the search
toward individual minima.
Exercise 10.1: First reformulate the problem as f ( x ) = x + ρ max(− x, 0)2 forwhich the derivative is1 + 2ρx if x < 0
f ′ (x) =(D.7)
1otherwiseThis unconstrained objective function can be solved by setting f ′ ( x ) = 0, which
1. Thus, as ρ → ∞ we have that x ∗ → 0.
yields the solution x ∗ = − 2ρ
Exercise 10.2: The problem is reformulated to f ( x ) = x + ρ( x < 0). The unconstrained objective function is unbounded from below as long as ρ is finite andx approaches negative infinity. The correct solution is not found, whereas thequadratic penalty method was able to approach the correct solution.
Exercise 10.3: You might try to increase the penalty parameter ρ. It is possiblethat ρ is too small and the penalty term is ineffective. In such cases, the iterates
may be reaching an infeasible region where the function decreases faster than the
penalty terms causing the method to converge on an infeasible solution.
Exercise 10.4: Let x ∗p be the solution to the unconstrained problem. Notice that2x ∗p 6> 0. Otherwise the penalty would be min( x ∗p , 0) = 0, which would imply
that x ∗p is a solution to the original problem. Now, suppose x ∗p = 0. The first-order© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 457

optimality conditions for unconstrained problems state that f ′ ( x ∗p ) = 0, again acontradiction. Thus, if a minimizer exists, it must be infeasible.
Exercise 10.5: It does not require a large penalty ρ to produce an accurate solution.
Exercise 10.6: When iterates should remain feasible.
Exercise 10.7: Consider the following:
minimizex3

subject to

x≥0

x

(D.8)

which is minimized for x ∗ = 0. Using the penalty method we can recast it asminimize x3 + ρ(min( x, 0))2
x

(D.9)

For any finite ρ, the function remains unbounded from below as x becomes
infinitely negative. Furthermore, as x becomes infinitely negative the function
becomes infinitely steep. In other words, if we start the steepest descent method
too far to the left, we have x3 + ρx2 ≈ x3 , and the penalty would be ineffective,
and the steepest descent method will diverge.
Exercise 10.8: You can frame finding a feasible point as an optimization problem
with a constant objective function and a constraint that forces feasibility:
minimizex0

subject to h(x) = 0(D.10)

g(x) ≤ 0

Such a problem can often be solved using penalty methods. Quadratic penalties
are a common choice because they decrease in the direction of feasibility.
Exercise 10.9: The problem is minimized at x ∗ = 1, which is at the constraint
boundary. Solving with the t-transform yields the unconstrained objective function:!
4(D.11)f t ( x̂ ) = sin
2x̂
5.5 + 4.5 1+
x̂2
which has a single global minimum at x̂ = −1, correctly corresponding to x ∗ .
The sigmoid transform has an unconstrained objective function:!
4
f s ( x̂ ) = sin(D.12)
1 + 1+9e−x̂
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

458 a p p e n d i x d . sol u ti on s

Unfortunately, the lower-bound a, in this case x = 1, is reached only as x̂ approaches minus infinity. The unconstrained optimization problem obtained using
the sigmoid transform does not have a solution, and the method fails to properly
identify the solution of the original problem.

0

−1

1

1

0.5

0.5
f ◦s

f ◦t

f

1

0

−0.5
0

5

10
x

−0.5

x̂ ∗
−10

0

0

10

x̂

−10

Exercise 10.10: Minimize x12 + x22 subject to x1 ≥ 1.
Exercise 10.11: We can rearrange the constraint in terms of x1 :
x1 = 6 − 2x2 − 3x3(D.13)and substitute the relation into the objective:
minimize x22 + x3 − (2x2 + 3x3 − 6)3x2 ,x3(D.14)

Exercise 10.12: The constraint must be aligned with the objective function. The
orientation of the objective function is [−1, −2]. The orientation of the constraint
is [ a, 1]. The only value for a that aligns them is a = 0.5.
Exercise 10.13: The transformed objective function is f ( x ) = 1 − x2 + ρp( x ),where p is either a count penalty or a quadratic penalty:
pcount ( x ) = (| x | > 2)

pquadratic ( x ) = max(| x | − 2, 0)2(D.15)

The count penalty method does not provide any gradient information to the
optimization process. An optimization algorithm initialized outside of the feasible
set will be drawn away from the feasible region because 1 − x2 is minimized bymoving infinitely far to the left or right from the origin. The large magnitude of
the count penalty is not the primary issue; small penalties can lead to similar
problems.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

0
x̂

10

a p p e n d i x d . sol u ti on s 459

The quadratic penalty method does provide gradient information to the optimization process, guiding searches toward the feasible region. For very large
penalties, the quadratic penalty method will produce large gradient values in theinfeasible region. In this problem, the partial derivative is:2( x − 2) if x > 2∂f(D.16)= −2x + ρ 2( x + 2) if x < −2

∂x


0
otherwise

For very large values of ρ, the partial derivative in the infeasible region is also
large, which can cause problems for optimization methods. If ρ is not large,
then infeasible points may not be sufficiently penalized, resulting in infeasible
solutions.
Exercise 11.1: We have chosen to minimize a linear program by evaluating every
vertex in the convex polytope formed by the constraints. Every vertex is thus apotential minimizer. Vertices are defined by intersections of active constraints. As
every inequality constraint can either be active or inactive, and assuming there aren inequality constraints, we do not need to examine more than 2n combinationsof constraints.
This method does not correctly report unbounded linear constrained optimization problems as unbounded.
Exercise 11.2: The simplex method is guaranteed either to improve with respect
to the objective function with each step or to preserve the current value of theobjective function. Any linear program will have a finite number of vertices. So
long as a heuristic, such as Bland’s rule, is employed such that cycling does notoccur, the simplex method must converge on a solution.
Exercise 11.3: We can add a slack variable x3 and split x1 and x2 into x1+ , x1− ,x2+ , and x2− . We minimize 6x1+ − 6x1− + 5x2+ − 5x2− subject to the constraints−3x1 + 2x2 + x3 = −5 and x1+ , x1− , x2+ , x2− , x3 ≥ 0.
Exercise 11.4: If the current iterate x is feasible, then w⊤ x = b ≥ 0. We want thenext point to maintain feasibility, and thus we require w⊤ (x + αd) ≥ 0. If theobtained value for α is positive, that α is an upper bound on the step length. If
the obtained value for α is negative, it can be ignored.
Exercise 11.5: We can rewrite the problem:minimize c⊤ x − µ ∑ ln A⊤
x(D.17){i }x

i© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

460 a p p e n d i x d . sol u ti on s

Exercise 12.1: The weighted sum method cannot find Pareto-optimal points innonconvex regions of the Pareto frontier.
Exercise 12.2: Nonpopulation methods will identify only a single point in the
Pareto frontier. The Pareto frontier is very valuable in informing the designerof the tradeoffs among a set of very good solutions. Population methods canspread out over the Pareto frontier and be used as an approximation of the Pareto
frontier.
Exercise 12.3:
y2[1, 2][2, 2][1, 1][2, 1]

y1
The only Pareto-optimal point is [1, 1]. Both [1, 2] and [2, 1] are weakly Pareto
optimal.
Exercise 12.4: The ‘‘gradient” of a vector is a matrix. Second-order derivatives
would require using tensors and solving a tensor equation for a search direction
is often computationally burdensome.
Exercise 12.5: The only Pareto-optimal point is y = [0, 0]. The rest of the points
on the bottom-left border are weakly Pareto-optimal.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 461

y2

Y

y1
Exercise 12.6: Consider the square criterion space from the previous question.
Using w = [0, 1] assigns zero value to the first objective, causing the entire bottomedge of the criterion space to have equal value. As discussed above, only y = [0, 0]
is Pareto optimal, the rest are weakly Pareto optimal.
Exercise 12.7: For example, if ygoal is in the criterion set, the goal programming
objective will be minimized by ygoal . If ygoal is also not Pareto optimal, the solution
will not be Pareto optimal either.
Exercise 12.8: The constraint method constrains all but one objective. A Pareto
curve can be generated by varying the constraints. If we constrain the first objective,
each optimization problem has the form:
minimize( x − 2)2

subject tox2 ≤ cx(D.18)(D.19)

The constraint can be satisfied only for c ≥ 0. This allows x to vary between√± c. The first objective is optimized by minimizing the deviation of x from 2.
Thus, for a given value of c, we obtain:
if c ≥ 4

2√
∗
x =
(D.20)c
if c ∈ [0, 4)
undefined otherwise
The resulting Pareto curve is:© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

462 a p p e n d i x d . sol u ti on s

y2

4

2

0
0

2
y1

1

3

4

Exercise 12.9: The criterion space is the space of objective function values. The
resulting plot is:
5
60

y2

40

20

0

−1

−3

−5

3
1

−20

−2

0

2
y1

4

6

We find that four points are on the approximate Pareto frontier corresponding toour six sample points. The corresponding design points are x = {−1, −3, 3, 1}.
Exercise 13.1: The one-dimensional unit hypercube is x ∈ [0, 1], and its volume is 1.
In this case the required side length ℓ is 0.5. The two-dimensional unit hypercube
is the unit square xi ∈ [0, 1] for i ∈ {1, 2}, which has a 2-dimensional volume, orarea, of 1. The area of a square with side length ℓ is ℓ2 , so we solve:
√
2
1
2
=⇒
ℓ=
≈ 0.707(D.21)ℓ =22© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

An n-dimensional hypercube has volume ℓn . We thus solve:

ℓn =

1
2

=⇒

ℓ = 2−1/n(D.22)The side length approaches one.
Exercise 13.2: The probability that a randomly sampled point is within ǫ-distancefrom the surface is just the ratio of the volumes. Thus:
P(k x k2 > 1 − ǫ) = 1 − P(k x k2 < 1 − ǫ) = 1 − (1 − ǫ)n → 1

hypercube side length ℓa p p e n d i x d . sol u ti on s 463

1
0.8
0.6
5
10
number of dimensions, n(D.23)

as n → ∞.
Exercise 13.3:
function pairwise_distances(X, p=2)m = length(X)[norm(X[i]-X[j], p) for i in 1:(m-1) for j in (i+1):m]end
function phiq(X, q=1, p=2)
dists = pairwise_distances(X, p)
return sum(dists.^(-q))^(1/q)
end
X = [[cos(2π*i/10), sin(2π*i/10)] for i in 1 : 10]@show phiq(X, 2)
phiq(X, 2) = 6.422616289332565

No. The Morris-Mitchell criterion is based entirely on pairwise distances. Shifting all of the points by the same amount does not change the pairwise distances
and thus will not change Φ2 ( X ).
Exercise 13.4: A rational number can be written as a fraction of two integers a/b.
It follows that the sequence repeats every b iterations:x ( k +1) = x ( k ) +ab(mod 1)a(mod 1)ba= x (0) + k + a (mod 1)ba(mod 1)= x (0) + ( k + b )b= x (k+b)x ( k ) = x (0) + k© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

464 a p p e n d i x d . sol u ti on s

Exercise 14.1: The linear regression objective function isky − Xθk22
We take the gradient and set it to zero:∇(y − Xθ)⊤ (y − Xθ) = −2X⊤ (y − Xθ) = 0
which yields the normal equation
X⊤ Xθ = X⊤ y
Exercise 14.2: As a general rule, more descriptive models should be used whenmore data are available. If only few samples are available such models are prone
to overfitting, and a simpler model (with fewer degrees of freedom) should be
used.
Exercise 14.3: The model at hand may have a very large number of parameters. In
such case, the resulting linear system will be too large and will require memorythat grows quadratically with the parameter space. Iterative procedures likestochastic gradient descent require memory linear in the size of the parameter
space and are sometimes the only viable solution.
Exercise 14.4: The leave-one-out cross-validation estimate is obtained by running
k-fold cross validation with k equal to the number of samples in X. This means
we must run 4-fold cross validation for each polynomial degree.

jackknife MSE

102

101

0

1

2

3

4

k
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 465

The lowest mean squared error is obtained for a linear model, k = 1. We fit a
new linear model on the complete dataset to obtain our parameters:
X = [[1],[2],[3],[4]]y = [0,5,4,6]
bases = polynomial_bases(1, 1)
B = [b(x) for x in X, b in bases]θ = pinv(B)*y@show θθ = [-0.49999999999999956, 1.6999999999999997]

Exercise 15.1: Gaussian processes are nonparametric, whereas linear regression
models are parametric. This means that the number of degrees of freedom of themodel grows with the amount of data, allowing the Gaussian process to maintaina balance between bias and variance during the optimization process.
Exercise 15.2: Obtaining the conditional distribution of a Gaussian process requires solving equation (15.13). The most expensive operation is inverting them × m matrix K( X, X ), which is O(m3 ).
Exercise 15.3: The derivative of f is

( x2 + 1) cos( x ) − 2x sin( x )( x 2 + 1)2
Below we plot the predictive distribution for Gaussian processes with and without derivative information. The maximum standard deviation in the predicteddistribution over [−5, 5] for the Gaussian process with derivative information isapproximately 0.377 at x ≈ ±3.8.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

466 a p p e n d i x d . sol u ti on s

confidence interval without derivative
confidence interval with derivative
predicted mean without derivative
predicted mean with derivative
fitted points
true function

y

1

0

−1

−4

−2

0
x

2

4

Incorporating derivative information significantly decreases the confidence
interval because more information is available to inform the prediction. Below weplot the maximum standard deviation in the predicted distribution over [−5, 5]
for Gaussian processes without derivative information with a varying number of
evenly-spaced evaluations. At least eight points are needed in order to outperformthe Gaussian process with derivative information.

maximum standard deviation

1

0.8

0.6

0.4

0.2
2

4

6

8

10

number of evenly spaced design points© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 467

Exercise 15.4: This can be derived according to:∂
k f ∇ (x, x′ )i = cov f (x), ′ f (x′ )
∂xi




∂
∂
′
′
= E ( f (x) − E [ f (x)])
f
(
x
)
−
E
f
(
x
)
∂x ′
∂xi′

 i


∂
∂ 
′
′
= E ( f (x) − E [ f (x)])
f (x ) − ′ E f (x )
∂xi′
∂xi




∂
′
′
= E ( f (x) − E [ f (x)]) ′ f (x ) − E f (x )
∂xi


∂ 
= ′ E ( f (x) − E [ f (x)]) f (x′ ) − E f (x′ )∂xi
∂= ′ cov f (x), f (x′ )∂xi∂= ′ k f f (x, x′ )
∂xi
h
i
∂
∂
where we have used E ∂x
f = ∂xE [ f ]. We can convince ourselves that this is
true:




f ( x + h) − f ( x )∂f = E lim
E
∂x
h
h →0


f ( x + h) − f ( x )= lim Ehh →0
1
= lim (E [ f ( x + h)] − E [ f ( x )])h →0 h∂=E [ f ( x )]∂xprovided that the objective function is differentiable.
Exercise 15.5: Let us write the joint Gaussian distribution as:
#!
" # "
" #
νa νc
µa
a
,
∼N
νc νb
µb
b

(D.24)The marginal distribution over a is N (µ a , νa ), which has variance νa . The
conditional distribution for a has variance νa − νc2 /νb . We know νb must be positivein order for the original covariance matrix to be positive definite. Thus, νc2 /νb ispositive and νa − νc2 /νb ≤ νa .
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

468 a p p e n d i x d . sol u ti on s

It is intuitive that the conditional distribution has no greater variance thanthe marginal distribution because the conditional distribution incorporates more
information about a. If a and b are correlated, then knowing the value of b informs
us about the value of a and decreases our uncertainty.
Exercise 15.6: We can tune the parameters to our kernel function or switch kernel
functions using generalization error estimation or by maximizing the likelihoodof the observed data.
Exercise 15.7: Maximizing the product of the likelihoods is equivalent to maximizing the sum of the log likelihoods. Here are the log likelihoods of the third
point given the other points using each kernel:exp(−k x − x ′ k)exp(−k x − x ′ k2 ) (1 + k x − x ′ k)−1 (1 + k x − x ′ k2 )−1 (1 + k x − x ′ k)−2

y

2
0

−2

1

3
x

−3.40951

3
x−3.58551

3
x−3.9145

1

3
x−4.1125

1Computing these values over all five folds yields the total log likelihoods:exp(−k x − x ′ k) → −8.688exp(−k x − x ′ k2 ) → −9.010(1 + k x − x ′ k)−1 → −9.579(1 + k x − x ′ k2 )−1 → −10.195(1 + k x − x ′ k)−2 → −8.088

It follows that the kernel that maximizes the leave-one-out cross-validated
likelihood is the rational quadratic kernel (1 + k x − x ′ k)−2 .
Exercise 16.1: Prediction-based optimization with Gaussian processes can repeatedly sample the same point. Suppose we have a zero-mean Gaussian process andwe start with a single point x(1) , which gives us some y(1) . The predicted mean
has a single global minimizer at x(1) . Prediction-based optimization will continue
to sample at x(1) .
Exercise 16.2: Error-based exploration wastes effort in reducing the variance anddoes not actively seek to minimize the function.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

3
x

−3.12

5

a p p e n d i x d . sol u ti on s 469

Exercise 16.3: The Gaussian process looks like this:uncertainty
predicted function mean
fitted points

y

2

0

−2
−4

−2

0
x

2

4

expected improvement

probability of improvement

The probability of improvement and expected improvement look like:

0.4

0.2

0

−4

−2

0
x

2

0.2

0.1

0

4

−4

−2

0
x

2

4

The maximum probability of improvement is at x = 0.98 for P = 0.523.
The maximum expected improvement is at x = 2.36 for E = 0.236.
p
Exercise 17.1: The objective is to minimize E z∼N [ f ( x + z)] − Varz∼N [ f ( x + z)].
The first term, corresponding to the mean, is minimized at design point a. The
second term, corresponding to the standard deviation, is also maximized at design
point a because perturbations to the design at that location cause large variationsin the output. The optimal design is thus x ∗ = a.
Exercise 17.2: The deterministic optimization problem is:
minimizex2

subject toγx −2 ≤ 1x

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

470 a p p e n d i x d . sol u ti on s

The optimal cross-section length as a function of the factor of safety is x =√γ. We can thus substitute γ for the cross-section length in each uncertainty
formulation and evaluate the probability that the design does not fail. Note thatall designs have a 50% chance of failure when the factor of safety is one, due tothe symmetry of the normal distribution.

√1

x −2 + z( x + z ) −2
(1 + z ) x −2

probability of success

0.9

0.8

0.7

0.6

0.5

1

1.2

1.4

1.6

1.8

2

factor of safety γ
64x2

Exercise 17.3: Figure D.1 shows the noise-free feasible region. Without noise,
the optimum lies with x1 infinitely negative. We have noise and have chosen
not to accept any outliers with magnitude greater than 6. Such outliers occur
approximately 1.973 × 10−7 % of the time.
The feasible region for x2 lies between e x1 and 2e x1 . The noise is symmetric, so
the most robust choice for x2 is 1.5e x1 .
The width of the feasible region for x2 is e x1 , which increases with x1 . The
objective function increases with x1 as well, so the optimal x1 is the lowest suchthat the width of the feasible region is at least 12. This results in x1 = ln 12 ≈ 2.485
and x2 = 18.
Exercise 18.1: The cumulative distribution function can be used to calculate thesevalues:

2
0

−2

0
x1

2

Figure D.1. The noise-free feasible
region.

julia> using Distributions
julia> N = Normal(0,1);
julia> cdf(N, 1) - cdf(N, -1)
0.682689© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

ap p e n d i x d . sol uti on s

471

julia> cdf(N, 1)
0.841345

Thus, our sample falls within one standard deviation of the mean about 68.3%
of the time and is less than one standard deviation above the mean 84.1% of thetime.
Exercise 18.2: We begin by substituting in the definition of the sample mean:!
x (1) + x (2) + · · · + x ( m )Var(µ̂) = Var
m


1 (2)
1 (m)
1 (1)
x + x +···+ x
= Var
m
m
m
The variance of the sum of two independent variables is the sum of the variances of the two variables. It follows that:






1 (1)
1 (2)
1 (m)
Var(µ̂) = Var
x
x
x
+ Var
+ · · · + Var
m
m
m






1
1
1
= 2 Var x (1) + 2 Var x (2) + · · · + 2 Var x (m)mmm
1= 2 (ν + ν + · · · + ν)m1
= 2 (mν)m
ν=
m
Exercise 18.3: The three-term recurrence relation for orthogonal polynomials iscentral to their construction and use. A key to the derivation is noticing that a
multiple of z can be shifted from one basis to the other:ZZ(zbi (z))b j (z) p(z) dz =We must show thatbi + 1 ( z ) =(z − α )b (z)
i

Z

Z


bi (z) zb j (z) p(z) dzi ( z − α i ) bi ( z ) − β i bi − 1 ( z )for i = 1
for i > 1

produces orthogonal polynomials.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

472 a p p e n d i x d . sol u ti on s

We notice that bi+1 − zbi is a polynomial of degree at most i, so one can write
it as a linear combination of the first i orthogonal polynomials:
i −2

bi+1 (z) − zbi (z) = −αi bi (z) − β i bi−1 (z) + ∑ γij b j (z)j =0

for constants αi , β i , and γij .
Multiplying both sides by bi and p and then integrating yields:
Z
Z

Z

Z

(bi+1 (z) − zbi (z))bi (z) p(z) dz =bi+1 (z)bi (z) p(z) dz −

Z

Z

Z

i −2

−αi bi (z) − β i bi−1 (z) + ∑ γij b j (z) bi (z) p(z) dzZzbi (z)bi (z) p(z) dz = −

−

j =0

Z

ZZ

αi bi (z)bi (z) p(z) dz −β i bi−1 (z)bi (z) p(z) dz +Zi −2 Z+∑−
producing our expression for αi :Z

j =0 ZZ

zbi2 (z) p(z) dz = −αiZZγij b j (z)bi (z) p(z) dz

bi2 (z) p(z) dz

Rzb2 (z) p(z) dz
αi = RZ 2i
Z bi ( z ) p ( z ) dz

The expression for β i with i ≥ 1 is obtained instead by multiplying both sides
by bi−1 and p and then integrating.
Multiplying both sides by bk , with k < i − 1, similarly produces:

−

Z

Z

zbi (z)bk (z) p(z) dz = γik

The shift property can be applied to yield:
ZZ

zbi (z)bk (z) p(z) dz =

Z

Z

Z

Z

!

bk2 (z) p(z) dz

bi (z)(zbk (z)) p(z) dz = 0

as zbk (z) is a polynomial of at most order i − 1, and, by orthogonality, the integral
is zero. It follows that all γik are zero, and the three term recurrence relation isestablished.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 473

Exercise 18.4: We can derive a gradient approximation using the partial derivativeof f with respect to a design component xi :∂∂∂f (x, z) ≈ b1 (z)θ 1 ( x ) + · · · + bk ( z )
θ (x)
∂xi
∂xi
∂xi k
If we have m samples, we can write these partial derivatives in matrix form:
 ∂

(1) )




 ∂
· · · bk ( z ( 1 ) )∂xi θ1 (x)


..
..
..


.
.
.


∂
θ
(
x
)
· · · bk ( z ( m ) )∂x kb1 (z(1) )



..
..
≈

.
.
 

∂
(m) )
(m) )
f
(
x,
z
b
(
z
1
∂x
∂xi f (x, zii

We can solve for approximations of ∂x∂ θ1 (x), . . . , ∂x∂ θk (x) using the pseudoiniiverse: ∂ +  ∂(1)b1 (z(1) ) · · · bk (z(1) )∂xi θ1 (x)∂xi f (x, z )

 

 
..
..
..
..
..

≈

 
.
.
.
.
.

 

 
∂
∂
(
m
)
(
m
)
(
m
)
)
b1 (z ) · · · bk (z )∂x θk (x)∂x f (x, ziiExercise 18.5: The estimated mean and variance have coefficients which dependon the design variables:µ̂(x) = θ1 (x)kν̂(x) = ∑ θi2 (x)
i =2

Z

Z

(D.25)bi (z)2 p(z) dz

The partial derivative of f mod with respect to the ith design component isk∂∂θ (x)∂θ (x)f mod (x) = α 1+ 2(1 − α ) ∑ θ i (x) i
∂xi
∂xi
xi
i =2

Z

Z

bi (z)2 p(z) dz(D.26)

Computing equation (D.26) requires the gradient of the coefficients with respect
to x, which is estimated in exercise 18.4.
Exercise 19.1: Enumeration tries all designs. Each component can either be trueor false, thus resulting in 2n possible designs in the worst case. This problem has
23 = 8 possible designs.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

474 a p p e n d i x d . sol u ti on s

f(x) = (!x[1] || x[3]) && (x[2] || !x[3]) && (!x[1] || !x[2])using IterTools
for x in Iterators.product([true,false], [true,false], [true,false])
if f(x)@show(x)
break
end
end
x = (false, true, true)

Exercise 19.2: The Boolean satisfiability problem merely seeks a valid solution.
As such, we set c to zero.
The constraints are more interesting. As with all integer linear programs, x isconstrained to be nonnegative and integral. Furthermore, we let 1 correspond totrue and 0 correspond to false and introduce the constraint x ≤ 1.
Next, we look at the objective function and observe that the ∧ ‘‘and’’ statements
divide f into separate Boolean expressions, each of which must be true. We convert
the expressions to linear constraints:
x1 =⇒ x1 ≥ 1

x2 ∨ ¬ x3 =⇒ x2 + (1 − x3 ) ≥ 1¬ x1 ∨ ¬ x2 =⇒ (1 − x1 ) + (1 − x2 ) ≥ 1

where each expression must be satisfied (≥ 1) and a negated variable ¬ xi is
simply 1 − xi .
The resulting integer linear program is:
minimize

0subject tox1 ≥ 1xx2 − x3 ≥ 0− x1 − x2 ≥ −1
x ∈ N3This approach is general and can be used to transform any Boolean satisfiability
problem into an integer linear program.
Exercise 19.3: Totally unimodular matrices have inverses that are also integer
matrices. Integer programs for which A is totally unimodular and b is integral
can be solved exactly using the simplex method.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 475

A matrix is totally unimodular if every square nonsingular submatrix is unimodular. A single matrix entry is a square submatrix. The determinant of a 1 × 1
matrix is the absolute value of its single entry. A single-entry submatrix is only
unimodular if it has a determinant of ±1, which occurs only for entries of ±1.
The single-entry submatrix can also be nonsingular, which allows for 0. No otherentries are permitted, and thus every totally unimodular matrix contains only
entries that are 0, 1, or −1.
Exercise 19.4: The branch and bound method requires that we can perform thebranching and bounding operations on our design.2 The decisions being madein 0-1 knapsack are whether or not to include each item. Each item therefore
represents a branch; either the item is included or it is excluded.
A tree is constructed for every such enumeration according to:x1 = 0x2 = 0

..
.

.. ..
. .

root

x2 = 1

.. ..
. .

.. ..
. .

x1 = 1x2 = 0.. ..
. .

P. J. Kolesar, “A Branch andBound Algorithm for the Knapsack
Problem,”Management
Science, vol. 13, no. 9, pp. 723–735,
1967.
2

.. ..
. .

x2 = 1

.. ..
. .

.. ..
. .

..
.

Each node represents a subproblem in which certain items have already been
included or excluded. The subproblem has the associated decision variables,
values, and weights removed, and the capacity is effectively reduced by the total
weight of the included items.
The branch and bound method avoids constructing the entire tree using information from the bounding operation. A bound can be constructed by solving
a relaxed version of the knapsack subproblem. This fractional knapsack problem
allows fractional values of items to be allocated, 0 ≤ xi ≤ 1.
The relaxed knapsack problem can be efficiently solved with a greedy approach.
Items are added one at a time by selecting the next item with the greatest ratio ofvalue to weight. If there is enough remaining capacity, the item is fully assignedwith xi = 1. If not, a fractional value is assigned such that the remaining capacity
is saturated and all remaining items have xi = 0.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

476 a p p e n d i x d . sol u ti on s

We begin by branching on the first item. The subtree with x1 = 0 has the
subproblem:
6

minimize
x2:6− ∑ vi xi
i =26∑ wi xi ≤ 20

subject toi =2

whereas the subtree with x1 = 1 has the subproblem:
6−9 − ∑ v i x iminimize
x2:6

i =2
6

∑ wi xi ≤ 13

subject toi =2

We can construct a lower bound for both subtrees using the greedy approach.
We sort the remaining items by value to weight:
item:
ratio:

6
3/4
0.75

4
3/5
0.6

5
5/9
0.556

3
2/4
0.5

2
4/8
0.5

For the subtree with x1 = 0, we fully allocate items 6, 4, and 5. We then partiallyallocate item 3 because we have remaining capacity 2, and thus set x3 = 2/4 = 0.5.
The lower bound is thus −(3 + 5 + 3 + 0.5 · 2) = −12.
For the subtree with x1 = 1, we allocate items 6 and 4 and partially allocate
item 5 to x5 = 4/9. The lower bound is thus −(3 + 5 + (4/9) · 3) ≈ −18.333.
The subtree with x1 = 1 has the better lower bound, so the algorithm continues
by splitting that subproblem. The final solution is x = [1, 0, 0, 0, 1, 1].
Exercise 20.1: Six expression trees can be generated:
I

I

F

R

R

R

1

2

π

I

I

F

1

2

π

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 477

Exercise 20.2: Only one expression of height 0 exists and that is the empty expression. Let us denote this as a0 = 1. Similarly, only one expressions of height 1 exists
and that is the expression {}. Let us denote this as a1 = 1. Three expressions exist
for depth 2, 21 for depth 3, and so on.
Suppose we have constructed all expressions up to height h. All expressionsof height h + 1 can be constructed using a root node with left and right subexpressions selected according to:
1. A left expression of height h and a right expression of height less than h
2. A right expression of height h and a left expression of height less than h
3. Left and right expressions of height h
It follows that the number of expressions of height h + 1 are:3

3
This corresponds to OEIS sequence A001699.

ah+1 = 2ah ( a0 + · · · + ah−1 ) + a2h
Exercise 20.3: One can use the following grammar and the starting symbol I:
I 7→ D + 10 × I

D 7→ 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
Exercise 20.4: Constructing exception-free grammars can be challenging. Suchissues can be avoided by catching the exceptions in the objective function andsuitably penalizing them.
Exercise 20.5: There are many reasons why one must constrain the types of thevariables manipulated during expression optimization. Many operators are valid
only on certain inputs4 and matrix multiplication requires that the dimensionsof the inputs be compatible. Physical dimensionality of the variables is anotherconcern. The grammar must reason about the units of the input values and of thevalid operations that can be performed on them.
For instance, x × y, with x having units kga mb sc , and y having units kgd me s f ,will produce a value with units kga+d mb+e sc+ f . Taking the square root of x willproduce a value with units kga/2 mb/2 sc/2 . Furthermore, operations such as sin
can be applied only to unitless inputs.

One typically does not take thesquare root of a negative number.

4© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

478

a p p e n d i x d . sol u ti on s

One approach for handling physical units is to associate an n-tuple with eachnode in the expression tree. The tuple records the exponent with respect to theallowable elementary units, which are specified by the user. If the elementary
units involved are mass, length, and time, then each node would have a 3-tuple( a, b, c) to represent units kga mb sc . The associated grammar must take these units
into account when assigning production rules.5
Exercise 20.6: The grammar can be encoded using ExprRules.jl using string
composition:

5
For a more thorough overview,see, for example, A. Ratle and M.
Sebag, “Genetic Programming andDomain Knowledge: Beyond theLimitations of Grammar-Guided
Machine Discovery,” in International Conference on Parallel Problem
Solving from Nature, 2000.

grammar = @grammar begin
S = NP * " " * VP
NP = ADJ * " " * NP
NP = ADJ * " " * N
VP = V * " " * ADVADJ = |(["a", "the", "big", "little", "blue", "red"])N = |(["mouse", "cat", "dog", "pony"])
V = |(["ran", "sat", "slept", "ate"])
ADV = |(["quietly", "quickly", "soundly", "happily"])end

We can use our decode method to obtain the solution.
eval(decode([2,10,19,0,6], grammar, :S)[1], grammar)

This produces ‘‘little dog ate quickly’’.
Exercise 20.7: We define a grammar for the clock problem. Let Gr be the symbol
for a gear of radius r, let A be an axle, let R be a rim, and let H be a hand. Our
grammar is:
Gr 7→ R A | ǫ
R 7→ R R | Gr | ǫ

A 7→ A A | Gr | H | ǫwhich allows each gear to have any number of rim and axle children. The expression ǫ is an empty terminal.
A clock with a single second hand can be constructed according to:© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 479

G25
R

A
r = 10 and 25, t = 0.1s

ǫ

G10
R

A
ǫ

G100
R

A

ǫ

H

r = 100, t = 1s

Note that the grammar does not compute the rotation period of the gears. Thisis handled by the objective function. A recursive procedure can be written to
return the rotation periods of all hands. The list of rotation periods is subsequently
used to compute the objective function value.
Exercise 20.8: Any of the methods covered in this chapter can be used to complete
the four 4s puzzle. A simple approach is to use Monte Carlo sampling on a suitable
grammar. Sampled expressions with exactly four 4s are evaluated and, if suitable,
are recorded. This procedure is repeated until an expression has been found foreach integer.
One such suitable grammar is:6
R 7→ 4 | 44 | 444 | 4444 | R + R | R − R | R × R | R/R |√RR | ⌊R ⌋ | ⌈R ⌉ | R | R! | Γ(R )The gamma function Γ( x ) is anextension of the factorial function
which accepts real and complexvalued inputs. For positive integers
x it produces ( x − 1)!.
6

We round evaluated expressions to the nearest integer. An expression that isrounded up can be contained inside a ceiling operation, and an expression that isrounded down can be contained inside a floor operation, so all such expressions
are valid.

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

480 a p p e n d i x d . sol u ti on s

Exercise 20.9: The expression is obtained by applying the production rules:
R 7→ R + R
R 7→ FP = 5/12

R 7→ IP = 5/12

F 7→ 1.5

which has a probability of

P = 1/12

I 7→

P = 4/7
P = 1/3

1 5 4 5 125=
≈ 0.00276
12 12 7 12 3
9072
Exercise 20.10: The learning update clears all counts and then increments eachproduction rule each time it is applied. The five applied rules are each incremented
once, resulting in:
R 7→ R + R | R × R | F | I

wR = [1, 0, 1, 1]

I 7→ 1 | 2 | 3

wI = [0, 1, 0]

F 7→ 1.5 | ∞wF = [1, 0]

Exercise 21.1: Maximize the lift-to-drag ratio of an airfoil shape subject to a
constraint on the structural stability of the airfoil.
Exercise 21.2: Consider a problem where the disciplinary dependency graph is a(directed) tree: if the optimization starts from the root and proceeds by following
the topological order, then convergence occurs after one traversal of the tree.
Exercise 21.3: It can execute disciplinary analyses in parallel.
Exercise 21.4: The spring-pendulum problem under the multidisciplinary design
feasible architecture is:minimizekf (MDA(k))

subject tok>0θ ≤ θmax

MDA(k) converged
where MDA(k ) performs a multidisciplinary analysis on the two disciplinary
analyses: a loads analysis that computes A[ M] = mgℓ cos(A[θ ]) and a displacement analysis that computes A[θ ] = A[ M ]/A[k]. Whether the multidisciplinary
design analysis converged is included as an additional response variable in orderto enforce convergence.
© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

a p p e n d i x d . sol u ti on s 481

Solving the optimization problem produces k ≈ 55.353 N.
Exercise 21.5: The spring-pendulum problem under the individual design feasible
architecture is:minimize kk,θc ,Mcsubject tok>0
θc = Fdisplacement (k, Mc )Mc = Floads (θc )θ ≤ θmax

where θc and Mc are the additional coupling variables under control of the optimizer. The two disciplinary analyses can be executed in parallel.
Exercise 21.6: The two disciplinary optimization problems for the spring-pendulum
problem under the collaborative optimization architecture are:minimize

Jdisplacement

subject toθ g ≤ θmaxk,M= (k g − A[k g ])2 + ( Mg − A[ Mg ])2 + ( Fdisplacement (k g , Mg ) − θ )2

k>0
and

minimizeθg

Jloads = (θ g − θ )2 + ( Floads (θ g ) − M)2

where the subscript g indicates a global variable. The global variables are A g = {k g , θ g , Mg }.
The system-level optimization problem is:
minimizekg

subject to

Jstructures = 0

k g ,θ g ,Mg

Jloads = 0© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

Bibliography
1.

N. M. Alexandrov and M. Y. Hussaini, eds., Multidisciplinary Design Optimization:
State of the Art. SIAM, 1997 (cit. on p. 387).

2.

S. Amari, “Natural Gradient Works Efficiently in Learning,” Neural Computation,
vol. 10, no. 2, pp. 251–276, 1998 (cit. on p. 90).

3.

Aristotle, Metaphysics, trans. by W. D. Ross. 350 BCE, Book I, Part 5 (cit. on p. 2).

4.

L. Armijo, “Minimization of Functions Having Lipschitz Continuous First Partial
Derivatives,” Pacific Journal of Mathematics, vol. 16, no. 1, pp. 1–3, 1966 (cit. on p. 56).

5.

J. Arora, Introduction to Optimum Design, 4th ed. Academic Press, 2016 (cit. on p. 4).

6.

R. K. Arora, Optimization: Algorithms and Applications. Chapman and Hall/CRC,
2015 (cit. on p. 6).

7.

T. W. Athan and P. Y. Papalambros, “A Note on Weighted Criteria Methods forCompromise Solutions in Multi-Objective Optimization,” Engineering Optimization,
vol. 27, no. 2, pp. 155–176, 1996 (cit. on p. 221).

8.

C. Audet and J. E. Dennis Jr., “Mesh Adaptive Direct Search Algorithms for Constrained Optimization,” SIAM Journal on Optimization, vol. 17, no. 1, pp. 188–217,
2006 (cit. on pp. 105, 126).

9.

D. A. Bader, W. E. Hart, and C. A. Phillips, “Parallel Algorithm Design for Branch
and Bound,” in Tutorials on Emerging Methodologies and Applications in Operations
Research, H. J. Greenberg, ed., Kluwer Academic Press, 2004 (cit. on p. 346).

10.

W. W. R. Ball, Mathematical Recreations and Essays. Macmillan, 1892 (cit. on p. 385).

11.

D. Barber, Bayesian Reasoning and Machine Learning. Cambridge University Press,2012 (cit. on p. 377).

12.

A. G. Baydin, R. Cornish, D. M. Rubio, M. Schmidt, and F. Wood, “Online Learning Rate Adaptation with Hypergradient Descent,” in International Conference on
Learning Representations (ICLR), 2018 (cit. on p. 82).

13.

A. D. Belegundu and T. R. Chandrupatla, Optimization Concepts and Applications inEngineering, 2nd ed. Cambridge University Press, 2011 (cit. on p. 6).

484 bi bl i og ra p hy

14.

R. Bellman, “On the Theory of Dynamic Programming,” Proceedings of the National
Academy of Sciences of the United States of America, vol. 38, no. 8, pp. 716–719, 1952
(cit. on p. 3).

15.

R. Bellman, Eye of the Hurricane: An Autobiography. World Scientific, 1984 (cit. on
p. 351).

16.

H. Benaroya and S. M. Han, Probability Models in Engineering and Science. Taylor &Francis, 2005 (cit. on p. 322).

17.

F. Berkenkamp, A. P. Schoellig, and A. Krause, “Safe Controller Optimization for
Quadrotors with Gaussian Processes,” in IEEE International Conference on Robotics
and Automation (ICRA), 2016 (cit. on p. 297).

18.

H.-G. Beyer and B. Sendhoff, “Robust Optimization—A Comprehensive Survey,”
Computer Methods in Applied Mechanics and Engineering, vol. 196, no. 33, pp. 3190–
3218, 2007 (cit. on p. 307).

19.

T. L. Booth and R. A. Thompson, “Applying Probability Measures to Abstract Languages,” IEEE Transactions on Computers, vol. C-22, no. 5, pp. 442–450, 1973 (cit. on
p. 375).

20.

C. Boutilier, R. Patrascu, P. Poupart, and D. Schuurmans, “Constraint-Based Optimization and Utility Elicitation Using the Minimax Decision Criterion,” Artificial
Intelligence, vol. 170, no. 8-9, pp. 686–713, 2006 (cit. on p. 231).

21.

G. E. P. Box, W. G. Hunter, and J. S. Hunter, Statistics for Experimenters: An Introduction
to Design, Data Analysis, and Model Building, 2nd ed. Wiley, 2005 (cit. on pp. 235,
307).

22.

S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press,
2004 (cit. on pp. 6, 178, 438).

23.

D. Braziunas and C. Boutilier, “Minimax Regret-Based Elicitation of Generalized
Additive Utilities,” in Conference on Uncertainty in Artificial Intelligence (UAI), 2007
(cit. on p. 231).

24.

D. Braziunas and C. Boutilier, “Elicitation of Factored Utilities,” AI Magazine, vol. 29,
no. 4, pp. 79–92, 2009 (cit. on p. 230).

25.

R. P. Brent, Algorithms for Minimization Without Derivatives. Prentice Hall, 1973 (cit.
on p. 51).

26.

S. J. Colley, Vector Calculus, 4th ed. Pearson, 2011 (cit. on p. 19).

27.

V. Conitzer, “Eliciting Single-Peaked Preferences Using Comparison Queries,”
Journal of Artificial Intelligence Research, vol. 35, pp. 161–191, 2009 (cit. on p. 228).

28.

S. Cook, “The Complexity of Theorem-Proving Procedures,” in ACM Symposium on
Theory of Computing, 1971 (cit. on p. 358).

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

bi bl i og ra p hy

29.

W. Cook, A. M. Gerards, A. Schrijver, and É. Tardos, “Sensitivity Theorems in Integer
Linear Programming,” Mathematical Programming, vol. 34, no. 3, pp. 251–264, 1986
(cit. on p. 341).

30.

A. Corana, M. Marchesi, C. Martini, and S. Ridella, “Minimizing Multimodal Functions of Continuous Variables with the ‘Simulated Annealing’ Algorithm,” ACM
Transactions on Mathematical Software, vol. 13, no. 3, pp. 262–280, 1987 (cit. on pp. 130,
132).

31.

G. B. Dantzig, “Origins of the Simplex Method,” in A History of Scientific Computing,
S. G. Nash, ed., ACM, 1990, pp. 141–151 (cit. on p. 195).

32.

S. Das and P. N. Suganthan, “Differential Evolution: A Survey of the State-of-theArt,” IEEE Transactions on Evolutionary Computation, vol. 15, no. 1, pp. 4–31, 2011
(cit. on p. 157).

33.

W. C. Davidon, “Variable Metric Method for Minimization,” Argonne National
Laboratory, Tech. Rep. ANL-5990, 1959 (cit. on p. 92).

34.

W. C. Davidon, “Variable Metric Method for Minimization,” SIAM Journal on Optimization, vol. 1, no. 1, pp. 1–17, 1991 (cit. on p. 92).

35.

A. Dean, D. Voss, and D. Draguljić, Design and Analysis of Experiments, 2nd ed.
Springer, 2017 (cit. on p. 235).

36.

K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A Fast and Elitist Multiobjective
Genetic Algorithm: NSGA-II,” IEEE Transactions on Evolutionary Computation, vol. 6,
no. 2, pp. 182–197, 2002 (cit. on p. 223).

37.

T. J. Dekker, “Finding a Zero by Means of Successive Linear Interpolation,” inConstructive Aspects of the Fundamental Theorem of Algebra, B. Dejon and P. Henrici,
eds., Interscience, 1969 (cit. on p. 51).

38.

R. Descartes, “La Géométrie,” in Discours de la Méthode. 1637 (cit. on p. 2).

39.

E. D. Dolan, R. M. Lewis, and V. Torczon, “On the Local Convergence of Pattern
Search,” SIAM Journal on Optimization, vol. 14, no. 2, pp. 567–583, 2003 (cit. on
p. 103).

40.

M. Dorigo, G. Di Caro, and L. M. Gambardella, “Ant Algorithms for Discrete
Optimization,” Artificial Life, vol. 5, no. 2, pp. 137–172, 1999 (cit. on p. 356).

41.

M. Dorigo, V. Maniezzo, and A. Colorni, “Ant System: Optimization by a Colonyof Cooperating Agents,” IEEE Transactions on Systems, Man, and Cybernetics, Part B(Cybernetics), vol. 26, no. 1, pp. 29–41, 1996 (cit. on pp. 354, 355).

42.

J. Duchi, E. Hazan, and Y. Singer, “Adaptive Subgradient Methods for Online
Learning and Stochastic Optimization,” Journal of Machine Learning Research, vol. 12,
pp. 2121–2159, 2011 (cit. on p. 77).

43.

R. Eberhart and J. Kennedy, “A New Optimizer Using Particle Swarm Theory,” inInternational Symposium on Micro Machine and Human Science, 1995 (cit. on p. 159).

485© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

486 bi bl i og ra p hy

44.

B. Efron, “Bootstrap Methods: Another Look at the Jackknife,” The Annals of Statistics,
vol. 7, pp. 1–26, 1979 (cit. on p. 270).

45.

B. Efron, “Estimating the Error Rate of a Prediction Rule: Improvement on CrossValidation,” Journal of the American Statistical Association, vol. 78, no. 382, pp. 316–
331, 1983 (cit. on p. 273).

46.

B. Efron and R. Tibshirani, “Improvements on Cross-Validation: The .632+ Bootstrap
Method,” Journal of the American Statistical Association, vol. 92, no. 438, pp. 548–560,
1997 (cit. on p. 273).

47.

Euclid, The Elements, trans. by D. E. Joyce. 300 BCE (cit. on p. 2).

48.

R. Fletcher, Practical Methods of Optimization, 2nd ed. Wiley, 1987 (cit. on p. 92).

49.

R. Fletcher and M. J. D. Powell, “A Rapidly Convergent Descent Method for Minimization,” The Computer Journal, vol. 6, no. 2, pp. 163–168, 1963 (cit. on p. 92).

50.

R. Fletcher and C. M. Reeves, “Function Minimization by Conjugate Gradients,”
The Computer Journal, vol. 7, no. 2, pp. 149–154, 1964 (cit. on p. 73).

51.

J. J. Forrest and D. Goldfarb, “Steepest-Edge Simplex Algorithms for Linear Programming,” Mathematical Programming, vol. 57, no. 1, pp. 341–374, 1992 (cit. on
p. 201).

52.

A. Forrester, A. Sobester, and A. Keane, Engineering Design via Surrogate Modelling:
A Practical Guide. Wiley, 2008 (cit. on p. 291).

53.

A. I. J. Forrester, A. Sóbester, and A. J. Keane, “Multi-Fidelity Optimization via
Surrogate Modelling,” Proceedings of the Royal Society of London A: Mathematical,
Physical and Engineering Sciences, vol. 463, no. 2088, pp. 3251–3269, 2007 (cit. on
p. 244).

54.

J. H. Friedman, “Exploratory Projection Pursuit,” Journal of the American Statistical
Association, vol. 82, no. 397, pp. 249–266, 1987 (cit. on p. 323).

55.

W. Gautschi, Orthogonal Polynomials: Computation and Approximation. Oxford University Press, 2004 (cit. on p. 328).

56.

A. Girard, C. E. Rasmussen, J. Q. Candela, and R. Murray-Smith, “Gaussian Process
Priors with Uncertain Inputs—Application to Multiple-Step Ahead Time Series
Forecasting,” in Advances in Neural Information Processing Systems (NIPS), 2003 (cit.
on p. 335).

57.

D. E. Goldberg and J. Richardson, “Genetic Algorithms with Sharing for Multimodal
Function Optimization,” in International Conference on Genetic Algorithms, 1987 (cit.
on p. 227).

58.

D. E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning.
Addison-Wesley, 1989 (cit. on p. 148).

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

bi bl i og ra p hy

59.

G. H. Golub and J. H. Welsch, “Calculation of Gauss Quadrature Rules,” Mathematics
of Computation, vol. 23, no. 106, pp. 221–230, 1969 (cit. on p. 331).

60.

R. E. Gomory, “An Algorithm for Integer Solutions to Linear Programs,” Recent
Advances in Mathematical Programming, vol. 64, pp. 269–302, 1963 (cit. on p. 342).

61.

I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016 (cit. onp. 4).

62.

A. Griewank and A. Walther, Evaluating Derivatives: Principles and Techniques of
Algorithmic Differentiation, 2nd ed. SIAM, 2008 (cit. on p. 23).

63.

S. Guo and S. Sanner, “Real-Time Multiattribute Bayesian Preference Elicitation with
Pairwise Comparison Queries,” in International Conference on Artificial Intelligenceand Statistics (AISTATS), 2010 (cit. on p. 228).

64.

B. Hajek, “Cooling Schedules for Optimal Annealing,” Mathematics of Operations
Research, vol. 13, no. 2, pp. 311–329, 1988 (cit. on p. 130).

65.

T. C. Hales, “The Honeycomb Conjecture,” Discrete & Computational Geometry, vol. 25,
pp. 1–22, 2001 (cit. on p. 2).

66.

J. H. Halton, “Algorithm 247: Radical-Inverse Quasi-Random Point Sequence,”
Communications of the ACM, vol. 7, no. 12, pp. 701–702, 1964 (cit. on p. 248).

67.

N. Hansen, “The CMA Evolution Strategy: A Tutorial,” ArXiv, no. 1604.00772, 2016
(cit. on pp. 138, 143).

68.

F. M. Hemez and Y. Ben-Haim, “Info-Gap Robustness for the Correlation of Tests
and Simulations of a Non-Linear Transient,” Mechanical Systems and Signal Processing,vol. 18, no. 6, pp. 1443–1467, 2004 (cit. on p. 312).

69.

G. Hinton and S. Roweis, “Stochastic Neighbor Embedding,” in Advances in Neural
Information Processing Systems (NIPS), 2003 (cit. on p. 125).

70.

R. Hooke and T. A. Jeeves, “Direct Search Solution of Numerical and Statistical
Problems,” Journal of the ACM (JACM), vol. 8, no. 2, pp. 212–229, 1961 (cit. on
p. 102).

71.

H. Ishibuchi and T. Murata, “A Multi-Objective Genetic Local Search Algorithm
and Its Application to Flowshop Scheduling,” IEEE Transactions on Systems, Man,
and Cybernetics, vol. 28, no. 3, pp. 392–403, 1998 (cit. on p. 225).

72.

V. S. Iyengar, J. Lee, and M. Campbell, “Q-EVAL: Evaluating Multiple Attribute
Items Using Queries,” in ACM Conference on Electronic Commerce, 2001 (cit. on
p. 229).

73.

D. Jones and M. Tamiz, Practical Goal Programming. Springer, 2010 (cit. on p. 219).

74.

D. R. Jones, C. D. Perttunen, and B. E. Stuckman, “Lipschitzian Optimization Without the Lipschitz Constant,” Journal of Optimization Theory and Application, vol. 79,
no. 1, pp. 157–181, 1993 (cit. on p. 108).

487© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

488 bi bl i og ra p hy

75.

A. B. Kahn, “Topological Sorting of Large Networks,” Communications of the ACM,
vol. 5, no. 11, pp. 558–562, 1962 (cit. on p. 390).

76.

L. Kallmeyer, Parsing Beyond Context-Free Grammars. Springer, 2010 (cit. on p. 361).

77.

L. V. Kantorovich, “A New Method of Solving Some Classes of Extremal Problems,”
in Proceedings of the USSR Academy of Sciences, vol. 28, 1940 (cit. on p. 3).

78.

A. F. Kaupe Jr, “Algorithm 178: Direct Search,” Communications of the ACM, vol. 6,
no. 6, pp. 313–314, 1963 (cit. on p. 104).

79.

A. Keane and P. Nair, Computational Approaches for Aerospace Design. Wiley, 2005 (cit.
on p. 6).

80.

J. Kennedy, R. C. Eberhart, and Y. Shi, Swarm Intelligence. Morgan Kaufmann, 2001
(cit. on p. 158).

81.

D. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in International
Conference on Learning Representations (ICLR), 2015 (cit. on p. 79).

82.

S. Kiranyaz, T. Ince, and M. Gabbouj, Multidimensional Particle Swarm Optimization
for Machine Learning and Pattern Recognition. Springer, 2014, Section 2.1 (cit. on p. 2).

83.

S. Kirkpatrick, C. D. Gelatt Jr., and M. P. Vecchi, “Optimization by Simulated Annealing,” Science, vol. 220, no. 4598, pp. 671–680, 1983 (cit. on p. 128).

84.

T. H. Kjeldsen, “A Contextualized Historical Analysis of the Kuhn-Tucker Theorem
in Nonlinear Programming: The Impact of World War II,” Historia Mathematica,
vol. 27, no. 4, pp. 331–361, 2000 (cit. on p. 176).

85.

L. Kocis and W. J. Whiten, “Computational Investigations of Low-Discrepancy
Sequences,” ACM Transactions on Mathematical Software, vol. 23, no. 2, pp. 266–294,
1997 (cit. on p. 249).

86.

P. J. Kolesar, “A Branch and Bound Algorithm for the Knapsack Problem,” Management Science, vol. 13, no. 9, pp. 723–735, 1967 (cit. on p. 475).

87.

B. Korte and J. Vygen, Combinatorial Optimization: Theory and Algorithms, 5th ed.
Springer, 2012 (cit. on p. 339).

88.

J. R. Koza, Genetic Programming: On the Programming of Computers by Means of Natural
Selection. MIT Press, 1992 (cit. on p. 364).

89.

K. W. C. Ku and M.-W. Mak, “Exploring the Effects of Lamarckian and Baldwinian
Learning in Evolving Recurrent Neural Networks,” in IEEE Congress on Evolutionary
Computation (CEC), 1997 (cit. on p. 162).

90.

L. Kuipers and H. Niederreiter, Uniform Distribution of Sequences. Dover, 2012 (cit.
on p. 239).

91.

J. C. Lagarias, J. A. Reeds, M. H. Wright, and P. E. Wright, “Convergence Properties of the Nelder–Mead Simplex Method in Low Dimensions,” SIAM Journal on
Optimization, vol. 9, no. 1, pp. 112–147, 1998 (cit. on p. 105).

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

bi bl i og ra p hy

92.

R. Lam, K. Willcox, and D. H. Wolpert, “Bayesian Optimization with a Finite Budget: An Approximate Dynamic Programming Approach,” in Advances in Neural
Information Processing Systems (NIPS), 2016 (cit. on p. 291).

93.

A. H. Land and A. G. Doig, “An Automatic Method of Solving Discrete Programming Problems,” Econometrica, vol. 28, no. 3, pp. 497–520, 1960 (cit. on p. 346).

94.

C. Lemieux, Monte Carlo and Quasi-Monte Carlo Sampling. Springer, 2009 (cit. on
p. 245).

95.

J. R. Lepird, M. P. Owen, and M. J. Kochenderfer, “Bayesian Preference Elicitation for
Multiobjective Engineering Design Optimization,” Journal of Aerospace Information
Systems, vol. 12, no. 10, pp. 634–645, 2015 (cit. on p. 228).

96.

K. Levenberg, “A Method for the Solution of Certain Non-Linear Problems in Least
Squares,” Quarterly of Applied Mathematics, vol. 2, no. 2, pp. 164–168, 1944 (cit. on
p. 61).

97.

S. Linnainmaa, “The Representation of the Cumulative Rounding Error of an Algorithm as a Taylor Expansion of the Local Rounding Errors,” M.S. thesis, University
of Helsinki, 1970 (cit. on p. 30).

98.

M. Manfrin, “Ant Colony Optimization for the Vehicle Routing Problem,” Ph.D.
dissertation, Université Libre de Bruxelles, 2004 (cit. on p. 355).

99.

R. T. Marler and J. S. Arora, “Survey of Multi-Objective Optimization Methods forEngineering,” Structural and Multidisciplinary Optimization, vol. 26, no. 6, pp. 369–
395, 2004 (cit. on p. 211).

100.

J. R. R. A. Martins and A. B. Lambe, “Multidisciplinary Design Optimization: A
Survey of Architectures,” AIAA Journal, vol. 51, no. 9, pp. 2049–2075, 2013 (cit. on
p. 387).

101.

J. R. R. A. Martins, P. Sturdza, and J. J. Alonso, “The Complex-Step Derivative Approximation,” ACM Transactions on Mathematical Software, vol. 29, no. 3, pp. 245–262,
2003 (cit. on p. 25).

102.

J. H. Mathews and K. D. Fink, Numerical Methods Using MATLAB, 4th ed. Pearson,
2004 (cit. on p. 24).

103.

K. Miettinen, Nonlinear Multiobjective Optimization. Kluwer Academic Publishers,
1999 (cit. on p. 211).

104.

D. J. Montana, “Strongly Typed Genetic Programming,” Evolutionary Computation,
vol. 3, no. 2, pp. 199–230, 1995 (cit. on p. 365).

105.

D. C. Montgomery, Design and Analysis of Experiments. Wiley, 2017 (cit. on p. 235).

106.

M. D. Morris and T. J. Mitchell, “Exploratory Designs for Computational Experiments,” Journal of Statistical Planning and Inference, vol. 43, no. 3, pp. 381–402, 1995
(cit. on p. 242).

489© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

490

bi bl i og ra p hy

107.

K. P. Murphy, Machine Learning: A Probabilistic Perspective. MIT Press, 2012 (cit. on
pp. 254, 265).

108.

S. Narayanan and S. Azarm, “On Improving Multiobjective Genetic Algorithms for
Design Optimization,” Structural Optimization, vol. 18, no. 2-3, pp. 146–155, 1999
(cit. on p. 227).

109.

S. Nash and A. Sofer, Linear and Nonlinear Programming. McGraw-Hill, 1996 (cit. on
p. 178).

110.

J. A. Nelder and R. Mead, “A Simplex Method for Function Minimization,” The
Computer Journal, vol. 7, no. 4, pp. 308–313, 1965 (cit. on p. 105).

111.

Y. Nesterov, “A Method of Solving a Convex Programming Problem with Convergence Rate O(1/k2 ),” Soviet Mathematics Doklady, vol. 27, no. 2, pp. 543–547, 1983
(cit. on p. 76).

112.

J. Nocedal, “Updating Quasi-Newton Matrices with Limited Storage,” Mathematics
of Computation, vol. 35, no. 151, pp. 773–782, 1980 (cit. on p. 94).

113.

J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed. Springer, 2006 (cit. on
pp. 57, 189).

114.

J. Nocedal and S. J. Wright, “Trust-Region Methods,” in Numerical Optimization.
Springer, 2006, pp. 66–100 (cit. on p. 65).

115.

A. O’Hagan, “Some Bayesian Numerical Analysis,” Bayesian Statistics, vol. 4, J. M.
Bernardo, J. O. Berger, A. P. Dawid, and A. F. M. Smith, eds., pp. 345–363, 1992 (cit.
on p. 282).

116.

M. Padberg and G. Rinaldi, “A Branch-and-Cut Algorithm for the Resolution of
Large-Scale Symmetric Traveling Salesman Problems,” SIAM Review, vol. 33, no. 1,
pp. 60–100, 1991 (cit. on p. 342).

117.

P. Y. Papalambros and D. J. Wilde, Principles of Optimal Design. Cambridge University
Press, 2017 (cit. on p. 6).

118.

G.-J. Park, T.-H. Lee, K. H. Lee, and K.-H. Hwang, “Robust Design: An Overview,”
AIAA Journal, vol. 44, no. 1, pp. 181–191, 2006 (cit. on p. 307).

119.

G. C. Pflug, “Some Remarks on the Value-at-Risk and the Conditional Value-atRisk,” in Probabilistic Constrained Optimization: Methodology and Applications, S. P.
Uryasev, ed. Springer, 2000, pp. 272–281 (cit. on p. 318).

120.

S. Piyavskii, “An Algorithm for Finding the Absolute Extremum of a Function,”
USSR Computational Mathematics and Mathematical Physics, vol. 12, no. 4, pp. 57–67,
1972 (cit. on p. 45).

121.

E. Polak and G. Ribière, “Note sur la Convergence de Méthodes de Directions
Conjuguées,” Revue Française d’informatique et de Recherche Opérationnelle, Série Rouge,
vol. 3, no. 1, pp. 35–43, 1969 (cit. on p. 73).

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

bi bl i og ra p hy

122.

M. J. D. Powell, “An Efficient Method for Finding the Minimum of a Function of
Several Variables Without Calculating Derivatives,” Computer Journal, vol. 7, no. 2,
pp. 155–162, 1964 (cit. on p. 100).

123.

W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes
in C: The Art of Scientific Computing. Cambridge University Press, 1982, vol. 2 (cit.
on p. 100).

124.

C. E. Rasmussen and Z. Ghahramani, “Bayesian Monte Carlo,” in Advances in Neural
Information Processing Systems (NIPS), 2003 (cit. on p. 335).

125.

C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. MIT
Press, 2006 (cit. on pp. 277, 278, 287).

126.

A. Ratle and M. Sebag, “Genetic Programming and Domain Knowledge: Beyond
the Limitations of Grammar-Guided Machine Discovery,” in International Conference
on Parallel Problem Solving from Nature, 2000 (cit. on p. 478).

127.

I. Rechenberg, Evolutionsstrategie Optimierung technischer Systeme nach Prinzipien derbiologischen Evolution. Frommann-Holzboog, 1973 (cit. on p. 137).

128.

R. G. Regis, “On the Properties of Positive Spanning Sets and Positive Bases,”
Optimization and Engineering, vol. 17, no. 1, pp. 229–262, 2016 (cit. on p. 103).

129.

A. M. Reynolds and M. A. Frye, “Free-Flight Odor Tracking in Drosophila is Consistent with an Optimal Intermittent Scale-Free Search,” PLoS ONE, vol. 2, no. 4,e354, 2007 (cit. on p. 162).

130.

R. T. Rockafellar and S. Uryasev, “Optimization of Conditional Value-at-Risk,”
Journal of Risk, vol. 2, pp. 21–42, 2000 (cit. on p. 316).

131.

R. T. Rockafellar and S. Uryasev, “Conditional Value-at-Risk for General Loss Distributions,” Journal of Banking and Finance, vol. 26, pp. 1443–1471, 2002 (cit. on
p. 318).

132.

H. H. Rosenbrock, “An Automatic Method for Finding the Greatest or Least Valueof a Function,” The Computer Journal, vol. 3, no. 3, pp. 175–184, 1960 (cit. on p. 430).

133.

R. Y. Rubinstein and D. P. Kroese, The Cross-Entropy Method: A Unified Approach to
Combinatorial Optimization, Monte-Carlo Simulation, and Machine Learning. Springer,
2004 (cit. on p. 133).

134.

D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning Representations by
Back-Propagating Errors,” Nature, vol. 323, pp. 533–536, 1986 (cit. on p. 30).

135.

C. Ryan, J. J. Collins, and M. O. Neill, “Grammatical Evolution: Evolving Programs
for an Arbitrary Language,” in European Conference on Genetic Programming, 1998
(cit. on p. 370).

136.

T. Salimans, J. Ho, X. Chen, and I. Sutskever, “Evolution Strategies as a Scalable
Alternative to Reinforcement Learning,” ArXiv, no. 1703.03864, 2017 (cit. on p. 137).

491© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

492 bi bl i og ra p hy

137.

R. Salustowicz and J. Schmidhuber, “Probabilistic Incremental Program Evolution,”
Evolutionary Computation, vol. 5, no. 2, pp. 123–141, 1997 (cit. on p. 377).

138.

J. D. Schaffer, “Multiple Objective Optimization with Vector Evaluated Genetic
Algorithms,” in International Conference on Genetic Algorithms and Their Applications,1985 (cit. on p. 221).

139.

C. Schretter, L. Kobbelt, and P.-O. Dehaye, “Golden Ratio Sequences for LowDiscrepancy Sampling,” Journal of Graphics Tools, vol. 16, no. 2, pp. 95–104, 2016 (cit.
on p. 247).

140.

A. Shapiro, D. Dentcheva, and A. Ruszczyński, Lectures on Stochastic Programming:
Modeling and Theory, 2nd ed. SIAM, 2014 (cit. on p. 314).

141.

A. Shmygelska, R. Aguirre-Hernández, and H. H. Hoos, “An Ant Colony Algorithm
for the 2D HP Protein Folding Problem,” in International Workshop on Ant Algorithms(ANTS), 2002 (cit. on p. 355).

142.

B. O. Shubert, “A Sequential Method Seeking the Global Maximum of a Function,”
SIAM Journal on Numerical Analysis, vol. 9, no. 3, pp. 379–388, 1972 (cit. on p. 45).

143.

D. Simon, Evolutionary Optimization Algorithms. Wiley, 2013 (cit. on p. 162).

144.

J. Sobieszczanski-Sobieski, A. Morris, and M. van Tooren, Multidisciplinary Design
Optimization Supported by Knowledge Based Engineering. Wiley, 2015 (cit. on p. 387).

145.

I. M. Sobol, “On the Distribution of Points in a Cube and the Approximate Evaluation of Integrals,” USSR Computational Mathematics and Mathematical Physics, vol. 7,
no. 4, pp. 86–112, 1967 (cit. on p. 249).

146.

D. C. Sorensen, “Newton’s Method with a Model Trust Region Modification,” SIAM
Journal on Numerical Analysis, vol. 19, no. 2, pp. 409–426, 1982 (cit. on p. 61).

147.

K. Sörensen, “Metaheuristics—the Metaphor Exposed,” International Transactions in
Operational Research, vol. 22, no. 1, pp. 3–18, 2015 (cit. on p. 162).

148.

T. J. Stieltjes, “Quelques Recherches sur la Théorie des Quadratures Dites Mécaniques,” Annales Scientifiques de l’École Normale Supérieure, vol. 1, pp. 409–426,
1884 (cit. on p. 328).

149.

J. Stoer and R. Bulirsch, Introduction to Numerical Analysis, 3rd ed. Springer, 2002
(cit. on pp. 89, 443).

150.

M. Stone, “Cross-Validatory Choice and Assessment of Statistical Predictions,”
Journal of the Royal Statistical Society, vol. 36, no. 2, pp. 111–147, 1974 (cit. on p. 269).

151.

T. Stützle, “MAX-MIN Ant System for Quadratic Assignment Problems,” Technical
University Darmstadt, Tech. Rep., 1997 (cit. on p. 355).

152.

Y. Sui, A. Gotovos, J. Burdick, and A. Krause, “Safe Exploration for Optimization
with Gaussian Processes,” in International Conference on Machine Learning (ICML),
vol. 37, 2015 (cit. on p. 296).

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

bi bl i og ra p hy

153.

H. Szu and R. Hartley, “Fast Simulated Annealing,” Physics Letters A, vol. 122,
no. 3-4, pp. 157–162, 1987 (cit. on p. 130).

154.

G. B. Thomas, Calculus and Analytic Geometry, 9th ed. Addison-Wesley, 1968 (cit. on
p. 22).

155.

V. Torczon, “On the Convergence of Pattern Search Algorithms,” SIAM Journal of
Optimization, vol. 7, no. 1, pp. 1–25, 1997 (cit. on p. 103).

156.

M. Toussaint, “The Bayesian Search Game,” in Theory and Principled Methods forthe Design of Metaheuristics, Y. Borenstein and A. Moraglio, eds. Springer, 2014,
pp. 129–144 (cit. on p. 291).

157.

R. J. Vanderbei, Linear Programming: Foundations and Extensions, 4th ed. Springer,
2014 (cit. on p. 189).

158.

D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, and J. Schmidhuber, “Natural Evolution Strategies,” ArXiv, no. 1106.4487, 2011 (cit. on p. 139).

159.

H. P. Williams, Model Building in Mathematical Programming, 5th ed. Wiley, 2013 (cit.
on p. 189).

160.

D. H. Wolpert and W. G. Macready, “No Free Lunch Theorems for Optimization,”
IEEE Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 67–82, 1997 (cit. onp. 6).

161.

P. K. Wong, L. Y. Lo, M. L. Wong, and K. S. Leung, “Grammar-Based Genetic Programming with Bayesian Network,” in IEEE Congress on Evolutionary Computation(CEC), 2014 (cit. on p. 375).

162.

X.-S. Yang, Nature-Inspired Metaheuristic Algorithms. Luniver Press, 2008 (cit. on
pp. 159, 160).

163.

X.-S. Yang, “A Brief History of Optimization,” in Engineering Optimization. Wiley,2010, pp. 1–13 (cit. on p. 2).

164.

X.-S. Yang and S. Deb, “Cuckoo Search via Lévy Flights,” in World Congress on
Nature & Biologically Inspired Computing (NaBIC), 2009 (cit. on p. 161).

165.

P. L. Yu, “Cone Convexity, Cone Extreme Points, and Nondominated Solutionsin Decision Problems with Multiobjectives,” Journal of Optimization Theory andApplications, vol. 14, no. 3, pp. 319–377, 1974 (cit. on p. 219).

166.

Y. X. Yuan, “Recent Advances in Trust Region Algorithms,” Mathematical Programming, vol. 151, no. 1, pp. 249–281, 2015 (cit. on p. 61).

167.

L. Zadeh, “Optimality and Non-Scalar-Valued Performance Criteria,” IEEE Transactions on Automatic Control, vol. 8, no. 1, pp. 59–60, 1963 (cit. on p. 218).

168.

M. D. Zeiler, “ADADELTA: An Adaptive Learning Rate Method,” ArXiv, no. 1212.5701,
2012 (cit. on p. 78).

493

© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

Index0.632 bootstrap estimate, 273
abstract types, 418
Ackley’s function, 425
active, 175
Adadelta, 78
Adagrad, see adaptive subgradientAdam, see adaptive moment estimation
adaptive moment estimation, 79
adaptive subgradient, 77
additive recurrence, 247
affine subspace, 193
aleatory uncertainty, 307
algebra, 2algorithm, 2algorithms, 1
algoritmi, 2anchoring point, 102
anonymous function, 420
ant colony optimization, 354
approximate line search, 55
arity, 377
Armijo condition, 55
Armijo line search, 56
array comprehension, 414
assignment, 388
associative array, 388
asymptotic notation, 433
augmented Lagrange method, 183

automatic differentiation, 27
auxiliary linear program, 202
backtracking line search, 56
backward difference, 20
backward pass, 30
Baldwinian learning, 164barrier methods, 183
basis functions, 255
batches, 125
Bayes-Hermite Quadrature, 334
Bayesian Monte Carlo, 334
BFGS, see Broyden-Fletcher-GoldfarbShanno
big-Oh notation, 433
binary string chromosome, 150
Binet’s formula, 38
bisection method, 49
black box, 99
Bland’s rule, 202
Boolean, 411
Boolean satisfiability problem, 358
Booth’s function, 426
bootstrap method, 270
bootstrap sample, 270
bound operation, 346
bowl, 9, 11
bowl-shaped, 437
bracket, 35

bracketing, 35
bracketing phase, 57
branch, 348
branch and bound, 346
branch and cut, 342
branch operation, 346
Branin function, 427
Brent-Dekker, 49
Broyden-Fletcher-Goldfarb-Shanno, 92
calculus, 3
Cauchy distribution, 148ceiling, 348
central difference, 20
characteristic length-scale, 278
Chebyshev distance, 439
chessboard distance, 439
chromosome, 148
circle function, 432
CMA-ES, see covariance matrix adaptation
collaborative optimization, 403
combinatorial optimization, 339
complete cross-validation, 269
complex step method, 25
composite type, 418
computational graph, 27
concave, 438
concrete types, 418

496 i n d e x

conditional distribution, 276
conditional value at risk, 316
confidence region, 282
conjugate gradient, 70
constraint, 6
constraint method, 216
context-free grammars, 361
contour plot, 11
contours, 11convex combination, 436
convex function, 437
convex set, 437
coordinate descent, 99
coprime, 248
coupling variables, 398
covariance function, 277
covariance matrix adaptation, 138
covariance matrix adaptation evolutionary strategy, see covariance matrix
adaptation
covariant gradient, 90
criterion space, 213
critical point, 173
critical points, 7
cross-entropy, 133
cross-entropy method, 133
crossover, 148crossover point, 153
cuckoo search, 161
cumulative distribution function, 316,
442
curvature condition, 56, 98
cutting plane, 344
cutting plane method, 342
CVaR, see conditional value at risk
cycles, 202
cyclic coordinate search, 99

decision quality improvement, 231
Dempster-Shafer theory, 308
dependency cycle, 390
dependency graph, 390
derivative, 19
derivative-free, 99
descent direction, 53
descent direction methods, 53
designer, 4
design matrix, 254
design point, 5design selection, 230
design variables, 5
DFP, see Davidon-Fletcher-Powelldictionary, 388, 417
differential evolution, 157
DIRECT, see divided rectangles
directional derivative, 22
direct methods, 99
Dirichlet distribution, 377
disciplinary analyses, 387
discrepancy, 239
discrete factors, 235
discrete optimization, 339
distributed architecture, 404
divided rectangles, 108
dominates, 212
dot notation, 28
dual, 177
dual certificates, 206
dual feasibility, 176
dual function, 177
duality gap, 178
dual numbers, 28
dual value, 177
dynamic ordering, 105
dynamic programming, 351

equality form, 193
equivalence class sharing, 227
error-based exploration, 292
error function, 442
Euclidean norm, 439
exchange algorithm, 244
expected improvement, 295
expected value, 314
expert responses, 229
exploitation, 293
exploration, 293
exponential annealing schedule, 130
exponential weighted criterion, 221

Dantzig’s rule, 202
Davidon-Fletcher-Powell, 92
decaying step factor, 55

elite samples, 133
entering index, 200
epistemic uncertainty, 307

gamma function, 278
gap, 312
Gaussian distribution, 275

factor of safety, 319
fast annealing, 130
feasible set, 5Fibonacci search, 37
finite difference methods, 23
firefly algorithm, 159
first fundamental theorem of calculus,
435
first-order, 69
first-order necessary condition, 9
first Wolfe condition, 57
fitness proportionate selection, 151
fitness sharing, 227
Fletcher-Reeves, 73
floor, 344
flower function, 428
forward accumulation, 27
forward difference, 20
forward pass, 30
Fourier series, 259
fractional knapsack, 475
full factorial, 235
function, 420
fuzzy-set theory, 308© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

i n d e x 497

Gaussian process, 275, 277
Gaussian quadrature, 443
Gauss-Seidel method, 390
gene duplication, 370
general form, 191
generalization error, 264
generalized Lagrangian, 177
generalized pattern search, 103
genetic algorithms, 148
genetic local search, 162
genetic programming, 364
global design variables, 396
global minimizer, 7
global optimization method, 45
goal programming, 219
golden ratio, 38
golden section search, 39
gradient, 21
gradient descent, 69
grammar, 361
grammatical evolution, 370
greedy heuristic, 202
greedy local search, 244
grid search, 235
half-space, 192
Halton sequence, 248
Hessian, 21
holdout method, 267
Hooke-Jeeves method, 102
hybrid methods, 162
hypergradient, 82
hypergradient descent, 80
hyperparameter, 36
hyperplane, 21
image, 213
improvement, 293
inactive, 175
individual discipline feasible, 398
individuals, 147
inflection point, 8

information-gap decision theory, 312
initialization phase, 198, 202
initial population, 147
integer linear programs, 340
integer program, 340
interdisciplinary compatibility, 389
Interior point methods, 183
intermediate value theorem, 49
irreducible uncertainty, 307
iterates, 53
Kahn’s algorithm, 390
kernel, 277
k-fold cross validation, 268
KKT conditions, 176
knapsack problem, 353
Kullback–Leibler divergence, 133
L2 regularization, 264
Lagrange multiplier, 173
Lagrange multipliers, 171
Lagrangian, 173
Lamarckian learning, 164
Latin-hypercube sampling, 238
Latin squares, 238
L-BFGS, see Limited-memory BFGS
leaped Halton method, 249
learning rate, 54, 55
leave-one-out bootstrap estimate, 270
leave-one-out cross-validation, 269
leaving index, 200
Lebesgue measure, 239
Legendre polynomials, 443
Lévy flights, 162
lexicographic method, 217
Limited-memory BFGS, 94
linear combination, 433
linear model, 254
linear program, 189
linear regression, 254
line search, 54
Lipschitz continuous, 45

local design variables, 396
local maximum, 11
local minimum, 7
local models, 53
logarithmic annealing schedule, 130
log likelihood, 287
loop, 423
low-discrepancy sequences, 245
lower confidence bound, 293
lower confidence bound exploration,
293
marginal distribution, 276
Matérn kernel, 278
matrix, 415
matrix norm, 98
maximum likelihood estimate, 287
max-min inequality, 177
max norm, 439
MDO, see multidisciplinary design optimization
mean, 314
mean excess loss, 316
mean function, 277
mean shortfall, 316
mean squared error, 265
memetic algorithms, 162memoization, 352
mesh adaptive direct search, 126
Metropolis criterion, 128
Michalewicz function, 429
minimax, 309
minimax decision, 231
minimax regret, 231
minimizer, 5minimum ratio test, 200
mixed integer program, 340
modified Bessel function of the second
kind, 278
momentum, 75
Monte Carlo integration, 245, 321© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

498 i n d e xMorris-Mitchell criterion, 242
multidisciplinary analysis, 389
multidisciplinary design feasible, 393
multidisciplinary design optimization,
387
multiobjective optimization, 211
multivariate, 53
mutation, 148
mutation rate, 155
mutually conjugate, 72
named function, 420
natural evolution strategies, 137
natural gradient, 90
necessary, 9necessary condition, 8
Nelder-Mead simplex method, 105
Nesterov momentum, 76
neural network kernel, 278
Newton’s method, 88
Newton’s notation, 28
niche, 227
niche techniques, 227
no free lunch theorems, 6
nondomination ranking, 223
nonparametric, 465
norm, 439
normal distribution, 275, 442
normal equation, 274
numerical differentiation, 23
objective function, 5
operation overloading, 32
opportunistic, 105
optimal substructure, 351
optimization phase, 198, 202
order, 433
orthogonal, 325
outgoing neighbors, 355
over-constrained, 193
overfitting, 265
overlapping subproblems, 351

package, 423
paired query selection, 229
pairwise distances, 239
parametric, 465
parametric types, 419
Pareto curve, 213
Pareto filter, 225
Pareto frontier, 213
Pareto optimality, 211, 213
parsimony, 365partial derivative, 21partially observable Markov decision
process, 291
particle, 158particle swarm optimization, 158
pattern, 103
pattern search, 99
penalty methods, 178
pheromone, 354
pivoting, 200
Polak-Ribière, 73
polyhedral method, 230
polynomial basis functions, 257
polynomial chaos, 323
polytopes, 196
population methods, 147
positive definite, 442
positive semidefinite, 442
positive spanning set, 103
possibility theory, 308
posterior distribution, 281
Powell’s method, 100
prediction-based exploration, 291
preference elicitation, 228
primal, 175
primal form, 177
primal value, 177
prime analytic center, 230
priority queue, 346
probabilistic grammar, 375
probabilistic prototype tree, 377

probabilistic uncertainty, 312
probability of improvement, 294
product correlation rule, 335
production rules, 361
proposal distribution, 133
pruning, 373
pseudoinverse, 255
pseudo-random, 125
Q-Eval, 229
quadratic convergence, 89
quadratic fit search, 43
quadratic penalties, 181
quadratic program, 189
quadrature rule, 443
quantile, 316
quasi-Monte Carlo methods, 245
quasi-Newton, 91
quasi-random sequences, 245
radial function, 259
random restarts, 66
random sampling, 236
random subsampling, 267
random uncertainty, 307
real-valued chromosomes, 150
reducible uncertainty, 307
regression, 254
regret, 231
regularization term, 264
relax, 341
residual form, 406
response variables, 387
restricted step method, 61
reverse accumulation, 27, 30
RMSProp, 78
robust, 316
robust counterpart approach, 309
robust regularization, 309
root-finding methods, 49
root mean squared error, 273
roots, 49© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

i n d e x 499

Rosenbrock function, 430
rotation estimation, 268
roulette wheel selection, 151
rounding, 341

saddle, 11
safe exploration, 296
SafeOpt, 296
sample mean, 321
sample variance, 321
sampling plans, 15, 235
secant equation, 98
secant method, 91
second-order, 87
second-order necessary condition, 9second-order sufficient condition, 11
second Wolfe condition, 57
self-dual simplex algorithm, 208
sensitive, 316
sequential optimization, 396
set-based uncertainty, 309
Shubert-Piyavskii method, 45
simplex, 105
simplex algorithm, 195
simulated annealing, 128
simultaneous analysis and design, 406
single-point crossover, 153
sinusoidal basis functions, 259
six-sigma, 320
slack variables, 194
Sobol sequences, 249
solution, 5
space-filling, 235
space-filling metrics, 239
space-filling subsets, 244
specification, 4
squared exponential kernel, 278
standard deviation, 282
standard deviation of the sample, 108
standard form, 191

standard normal cumulative distribution function, 294
start type, 361
stationary point, 9, 69
statistical feasibility, 316
step size, 54
Stieltjes algorithm, 328
stochastic gradient descent, 125
stochastic methods, 125
stratified sampling, 238
strict local minimizer, 8strictly concave, 438
strictly convex, 437
string, 413
strong backtracking line search, 57
strong curvature condition, 57
strong local minima, 8strong local minimizer, 8strongly typed genetic programming,
365
strong Wolfe conditions, 57
submatrix, 342
suboptimizer, 393
subpopulations, 221
subproblem, 396
sufficient closeness, 89
sufficient condition, 8
sufficient decrease, 55
supremum, 239
surrogate model, 235
surrogate models, 253
symbolic differentiation, 20
symbols, 361
system-level optimizer, 396
tail value at risk, 316
taxicab search, 99
Taylor approximation, 436
Taylor expansion, 435
Taylor series, 435
Temperature, 128

terminal, 361
ternary operator, 422
test functions, 425
test set, 267
topological ordering, 390
totally unimodular, 342
tournament selection, 151
trace, 287
training error, 265
training set, 267
traveling salesman problem, 355
tree crossover, 365
tree mutation, 365
tree permutation, 367
triangle inequality, 439
truncation selection, 151
trust region, 61
tuple, 417
two-point crossover, 153
types, 361
uncertainty propagation, 321
uncorrected sample standard deviation,
108
uniform crossover, 153
uniform projection plan, 237
unimodal function, 35
unimodality, 35
univariate, 7
univariate function, 7
univariate Gaussian, 442
utopia point, 214value at risk, 316
van der Corput sequences, 248
VaR, see value at risk
variance, 314
vector, 413
vector evaluated genetic algorithm, 221
vector optimization, 211vertices, 196© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com

500 i n d e x

weak local minima, 8
weakly Pareto-optimal, 213
weighted exponential sum, 219
weighted min-max method, 220

weighted sum method, 218
weighted Tchebycheff method, 220
Wheeler’s ridge, 431
whitening, 323

Wolfe conditions, 57
zero-order, 99
zoom phase, 60© 2019 Massachusetts Institute of Technology, shared under a Creative Commons CC-BY-NC-ND license.
2022-05-22 00:25:57-07:00, revision 47fd495, comments to bugs@algorithmsbook.com