CS229 Lecture Notes
Andrew Ng
Updated by Tengyu Ma

Contents
I

Supervised learning

5

1 Linear regression
1.1 LMS algorithm . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 The normal equations . . . . . . . . . . . . . . . . . . . . . . .
1.2.1 Matrix derivatives . . . . . . . . . . . . . . . . . . . . .
1.2.2 Least squares revisited . . . . . . . . . . . . . . . . . .
1.3 Probabilistic interpretation . . . . . . . . . . . . . . . . . . . .
1.4 Locally weighted linear regression (optional reading) . . . . . .

8
9
13
13
14
15
17

2 Classification and logistic regression
2.1 Logistic regression . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Digression: the perceptron learning algorithn . . . . . . . . . .
2.3 Another algorithm for maximizing `(θ) . . . . . . . . . . . . .

20
20
23
24

3 Generalized linear models
3.1 The exponential family . . . . . . . . . . . . . . . . . . . . . .
3.2 Constructing GLMs . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 Ordinary least squares . . . . . . . . . . . . . . . . . .
3.2.2 Logistic regression . . . . . . . . . . . . . . . . . . . .
3.2.3 Softmax regression . . . . . . . . . . . . . . . . . . . .

26
26
28
29
30
30

4 Generative learning algorithms
4.1 Gaussian discriminant analysis . . . . . . . . . . . . . . . . . .
4.1.1 The multivariate normal distribution . . . . . . . . . .
4.1.2 The Gaussian discriminant analysis model . . . . . . .
4.1.3 Discussion: GDA and logistic regression . . . . . . . .
4.2 Naive bayes . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.1 Laplace smoothing . . . . . . . . . . . . . . . . . . . .
4.2.2 Event models for text classification . . . . . . . . . . .

35
36
36
39
41
42
45
47

1

CS229 Spring 2022

2

5 Kernel methods
5.1 Feature maps . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 LMS (least mean squares) with features . . . . . . . . . . . . .
5.3 LMS with the kernel trick . . . . . . . . . . . . . . . . . . . .
5.4 Properties of kernels . . . . . . . . . . . . . . . . . . . . . . .

49
49
50
50
54

6 Support vector machines
6.1 Margins: intuition . . . . . . . . . . . . . . . . . . . . . . . . .
6.2 Notation (option reading) . . . . . . . . . . . . . . . . . . . .
6.3 Functional and geometric margins (option reading) . . . . . .
6.4 The optimal margin classifier (option reading) . . . . . . . . .
6.5 Lagrange duality (optional reading) . . . . . . . . . . . . . . .
6.6 Optimal margin classifiers: the dual form (option reading) . .
6.7 Regularization and the non-separable case (optional reading) .
6.8 The SMO algorithm (optional reading) . . . . . . . . . . . . .
6.8.1 Coordinate ascent . . . . . . . . . . . . . . . . . . . . .
6.8.2 SMO . . . . . . . . . . . . . . . . . . . . . . . . . . . .

60
60
62
62
64
66
69
73
74
75
76

II

80

Deep learning

7 Deep learning
7.1 Supervised learning with non-linear models . . . . . . . . . . .
7.2 Neural networks . . . . . . . . . . . . . . . . . . . . . . . . . .
7.3 Backpropagation . . . . . . . . . . . . . . . . . . . . . . . . .
7.3.1 Preliminary: chain rule . . . . . . . . . . . . . . . . . .
7.3.2 One-neuron neural networks . . . . . . . . . . . . . . .
7.3.3 Two-layer neural networks: a low-level unpacked computation . . . . . . . . . . . . . . . . . . . . . . . . . .
7.3.4 Two-layer neural network with vector notation . . . . .
7.3.5 Multi-layer neural networks . . . . . . . . . . . . . . .
7.4 Vectorization over training examples . . . . . . . . . . . . . .

III

Generalization and regularization

81
81
83
92
93
93
94
97
99
99

102

8 Generalization
103
8.1 Bias-variance tradeoff . . . . . . . . . . . . . . . . . . . . . . . 105
8.1.1 A mathematical decomposition (for regression) . . . . . 110
8.2 The double descent phenomenon . . . . . . . . . . . . . . . . . 111

CS229 Spring 2022
8.3

3

Sample complexity bounds (optional readings) . . . . . . . . . 116
8.3.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . 116
8.3.2 The case of finite H . . . . . . . . . . . . . . . . . . . . 118
8.3.3 The case of infinite H . . . . . . . . . . . . . . . . . . 121

9 Regularization and model selection
125
9.1 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
9.2 Implicit regularization effect . . . . . . . . . . . . . . . . . . . 127
9.3 Model selection via cross validation . . . . . . . . . . . . . . . 129
9.4 Bayesian statistics and regularization . . . . . . . . . . . . . . 132

IV

Unsupervised learning134

10 Clustering and the k-means algorithm

135

11 EM algorithms
138
11.1 EM for mixture of Gaussians . . . . . . . . . . . . . . . . . . . 138
11.2 Jensen’s inequality . . . . . . . . . . . . . . . . . . . . . . . . 141
11.3 General EM algorithms . . . . . . . . . . . . . . . . . . . . . . 142
11.3.1 Other interpretation of ELBO . . . . . . . . . . . . . . 148
11.4 Mixture of Gaussians revisited . . . . . . . . . . . . . . . . . . 148
11.5 Variational inference and variational auto-encoder (optional
reading) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
12 Principal components analysis

155

13 Independent components analysis
161
13.1 ICA ambiguities . . . . . . . . . . . . . . . . . . . . . . . . . . 162
13.2 Densities and linear transformations . . . . . . . . . . . . . . . 163
13.3 ICA algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
14 Self-supervised learning and foundation models
167
14.1 Pretraining and adaptation . . . . . . . . . . . . . . . . . . . . 167
14.2 Pretraining methods in computer vision . . . . . . . . . . . . . 169
14.3 Pretrained large language models . . . . . . . . . . . . . . . . 171
14.3.1 Zero-shot learning and in-context learning . . . . . . . 173

CS229 Spring 2022

V

Reinforcement Learning and Control

4175

15 Reinforcement learning176
15.1 Markov decision processes . . . . . . . . . . . . . . . . . . . . 177
15.2 Value iteration and policy iteration . . . . . . . . . . . . . . . 179
15.3 Learning a model for an MDP . . . . . . . . . . . . . . . . . . 181
15.4 Continuous state MDPs . . . . . . . . . . . . . . . . . . . . . 183
15.4.1 Discretization . . . . . . . . . . . . . . . . . . . . . . . 183
15.4.2 Value function approximation . . . . . . . . . . . . . . 186
15.5 Connections between Policy and Value Iteration (Optional) . . 190
16 LQR, DDP and LQG
193
16.1 Finite-horizon MDPs . . . . . . . . . . . . . . . . . . . . . . . 193
16.2 Linear Quadratic Regulation (LQR) . . . . . . . . . . . . . . . 197
16.3 From non-linear dynamics to LQR . . . . . . . . . . . . . . . 200
16.3.1 Linearization of dynamics . . . . . . . . . . . . . . . . 201
16.3.2 Differential Dynamic Programming (DDP) . . . . . . . 201
16.4 Linear Quadratic Gaussian (LQG) . . . . . . . . . . . . . . . . 203
17 Policy Gradient (REINFORCE)

207

Part I
Supervised learning5

6
Let’s start by talking about a few examples of supervised learning problems. Suppose we have a dataset giving the living areas and prices of 47
houses from Portland, Oregon:
Living area (feet2 )
2104
1600
2400
1416
3000
..
.

Price (1000$s)
400
330
369
232
540
..
.

We can plot this data:
housing prices
1000
900
800

price (in $1000)

700
600
500
400
300
200
100
0
500

1000

1500

2000

2500
3000
square feet

3500

4000

4500

5000

Given data like this, how can we learn to predict the prices of other houses
in Portland, as a function of the size of their living areas?
To establish notation for future use, we’ll use x(i) to denote the “input”
variables (living area in this example), also called input features, and y (i)to denote the “output” or target variable that we are trying to predict(price). A pair (x(i) , y (i) ) is called a training example, and the datasetthat we’ll be using to learn—a list of n training examples {(x(i) , y (i) ); i =1, . . . , n}—is called a training set. Note that the superscript “(i)” in thenotation is simply an index into the training set, and has nothing to do withexponentiation. We will also use X denote the space of input values, and Y
the space of output values. In this example, X = Y = R.
To describe the supervised learning problem slightly more formally, ourgoal is, given a training set, to learn a function h : X 7→ Y so that h(x) is a“good” predictor for the corresponding value of y. For historical reasons, this

7function h is called a hypothesis. Seen pictorially, the process is therefore
like this:Training
set

Learning
algorithm
x(living area ofhouse.)

hpredicted y(predicted price)of house)

When the target variable that we’re trying to predict is continuous, suchas in our housing example, we call the learning problem a regression problem. When y can take on only a small number of discrete values (such asif, given the living area, we wanted to predict if a dwelling is a house or anapartment, say), we call it a classification problem.

Chapter 1
Linear regression
To make our housing example more interesting, let’s consider a slightly richer
dataset in which we also know the number of bedrooms in each house:
Living area (feet2 )
2104
1600
2400
1416
3000
..
.

#bedrooms
3
3
3
2
4
..
.

Price (1000$s)
400
330
369
232
540
..
.
(i)

Here, the x’s are two-dimensional vectors in R2 . For instance, x1 is the(i)living area of the i-th house in the training set, and x2 is its number ofbedrooms. (In general, when designing a learning problem, it will be up to
you to decide what features to choose, so if you are out in Portland gathering
housing data, you might also decide to include other features such as whether
each house has a fireplace, the number of bathrooms, and so on. We’ll say
more about feature selection later, but for now let’s take the features asgiven.)
To perform supervised learning, we must decide how we’re going to represent functions/hypotheses h in a computer. As an initial choice, let’s say
we decide to approximate y as a linear function of x:hθ (x) = θ0 + θ1 x1 + θ2 x2Here, the θi ’s are the parameters (also called weights) parameterizing thespace of linear functions mapping from X to Y. When there is no risk of8

9
confusion, we will drop the θ subscript in hθ (x), and write it more simply ash(x). To simplify our notation, we also introduce the convention of letting
x0 = 1 (this is the intercept term), so thath(x) =dX

θi xi = θT x,

i=0

where on the right-hand side above we are viewing θ and x both as vectors,
and here d is the number of input variables (not counting x0 ).
Now, given a training set, how do we pick, or learn, the parameters θ?
One reasonable method seems to be to make h(x) close to y, at least for
the training examples we have. To formalize this, we will define a functionthat measures, for each value of the θ’s, how close the h(x(i) )’s are to thecorresponding y (i) ’s. We define the cost function:n1X(hθ (x(i) ) − y (i) )2 .
J(θ) =
2 i=1
If you’ve seen linear regression before, you may recognize this as the familiar
least-squares cost function that gives rise to the ordinary least squares
regression model. Whether or not you have seen it previously, let’s keep
going, and we’ll eventually show this to be a special case of a much broader
family of algorithms.

1.1

LMS algorithm

We want to choose θ so as to minimize J(θ). To do so, let’s use a searchalgorithm that starts with some “initial guess” for θ, and that repeatedly
changes θ to make J(θ) smaller, until hopefully we converge to a value ofθ that minimizes J(θ). Specifically, let’s consider the gradient descentalgorithm, which starts with some initial θ, and repeatedly performs the
update:∂θj := θj − α
J(θ).
∂θj
(This update is simultaneously performed for all values of j = 0, . . . , d.)Here, α is called the learning rate. This is a very natural algorithm that
repeatedly takes a step in the direction of steepest decrease of J.
In order to implement this algorithm, we have to work out what is thepartial derivative term on the right hand side. Let’s first work it out for the10case of if we have only one training example (x, y), so that we can neglect
the sum in the definition of J. We have:∂∂ 1
J(θ) =(hθ (x) − y)2
∂θj
∂θj 2
1
∂
= 2 · (hθ (x) − y) ·(hθ (x) − y)
2
∂θj
!
d
X
∂
= (hθ (x) − y) ·θi xi − y∂θj i=0= (hθ (x) − y) xj
For a single training example, this gives the update rule:1 (i)θj := θj + α y (i) − hθ (x(i) ) xj .
The rule is called the LMS update rule (LMS stands for “least mean squares”),
and is also known as the Widrow-Hoff learning rule. This rule has several
properties that seem natural and intuitive. For instance, the magnitude of
the update is proportional to the error term (y (i) − hθ (x(i) )); thus, for instance, if we are encountering a training example on which our prediction
nearly matches the actual value of y (i) , then we find that there is little needto change the parameters; in contrast, a larger change to the parameters will
be made if our prediction hθ (x(i) ) has a large error (i.e., if it is very far fromy (i) ).
We’d derived the LMS rule for when there was only a single training
example. There are two ways to modify this method for a training set of
more than one example. The first is replace it with the following algorithm:
Repeat until convergence {θj := θj + α

n
X

 (i)
y (i) − hθ (x(i) ) xj , (for every j)

(1.1)

i=1

}
1
We use the notation “a := b” to denote an operation (in a computer program) inwhich we set the value of a variable a to be equal to the value of b. In other words, thisoperation overwrites a with the value of b. In contrast, we will write “a = b” when we areasserting a statement of fact, that the value of a is equal to the value of b.

11
By grouping the updates of the coordinates into an update of the vector
θ, we can rewrite update (1.1) in a slightly more succinct way:
θ := θ + αnXy (i) − hθ (x(i) ) x(i)i=1

The reader can easily verify that the quantity in the summation in the
update rule above is just ∂J(θ)/∂θj (for the original definition of J). So, thisis simply gradient descent on the original cost function J. This method looks
at every example in the entire training set on every step, and is called batch
gradient descent. Note that, while gradient descent can be susceptible
to local minima in general, the optimization problem we have posed here
for linear regression has only one global, and no other local, optima; thus
gradient descent always converges (assuming the learning rate α is not toolarge) to the global minimum. Indeed, J is a convex quadratic function.
Here is an example of gradient descent as it is run to minimize a quadratic
function.
50

45

40

35

30

25

20

15

10

5

5

10

15

20

25

30

35

40

45

50

The ellipses shown above are the contours of a quadratic function. Also
shown is the trajectory taken by gradient descent, which was initialized at(48,30). The x’s in the figure (joined by straight lines) mark the successive
values of θ that gradient descent went through.
When we run batch gradient descent to fit θ on our previous dataset,to learn to predict housing price as a function of living area, we obtain
θ0 = 71.27, θ1 = 0.1345. If we plot hθ (x) as a function of x (area), alongwith the training data, we obtain the following figure:12
housing prices
1000
900
800

price (in $1000)

700
600
500
400
300
200
100
0
500

1000

1500

2000

2500
3000
square feet

3500

4000

4500

5000

If the number of bedrooms were included as one of the input features as well,
we get θ0 = 89.60, θ1 = 0.1392, θ2 = −8.738.
The above results were obtained with batch gradient descent. There isan alternative to batch gradient descent that also works very well. Consider
the following algorithm:
Loop {
for i = 1 to n, { (i)θj := θj + α y (i) − hθ (x(i) ) xj ,(for every j)

(1.2)

}
}
By grouping the updates of the coordinates into an update of the vector
θ, we can rewrite update (1.2) in a slightly more succinct way:
θ := θ + α y (i) − hθ (x(i) ) x(i)In this algorithm, we repeatedly run through the training set, and eachtime we encounter a training example, we update the parameters according
to the gradient of the error with respect to that single training example only.
This algorithm is called stochastic gradient descent (also incremental
gradient descent). Whereas batch gradient descent has to scan throughthe entire training set before taking a single step—a costly operation if n islarge—stochastic gradient descent can start making progress right away, and13continues to make progress with each example it looks at. Often, stochastic
gradient descent gets θ “close” to the minimum much faster than batch gradient descent. (Note however that it may never “converge” to the minimum,
and the parameters θ will keep oscillating around the minimum of J(θ); but
in practice most of the values near the minimum will be reasonably good
approximations to the true minimum.2 ) For these reasons, particularly when
the training set is large, stochastic gradient descent is often preferred overbatch gradient descent.

1.2

The normal equations

Gradient descent gives one way of minimizing J. Let’s discuss a second wayof doing so, this time performing the minimization explicitly and without
resorting to an iterative algorithm. In this method, we will minimize J by
explicitly taking its derivatives with respect to the θj ’s, and setting them tozero. To enable us to do this without having to write reams of algebra andpages full of matrices of derivatives, let’s introduce some notation for doing
calculus with matrices.

1.2.1

Matrix derivatives

For a function f : Rn×d 7→ R mapping from n-by-d matrices to the real
numbers, we define the derivative of f with respect to A to be:

 ∂f
∂f
·
·
·
∂A11
∂A1d

.. 
..
∇A f (A) =  ...
.
. 
∂f
∂An1

···

∂f
∂And

Thus, the gradient ∇A f (A) is itself an n-by-d matrix,
 whose (i, j)-element is∂f /∂Aij . For example, suppose A =

A11
A21

A12
A22

is a 2-by-2 matrix, andthe function f : R2×2 7→ R is given by
3
f (A) = A11 + 5A212 + A21 A22 .
2
2

By slowly letting the learning rate α decrease to zero as the algorithm runs, it is also
possible to ensure that the parameters will converge to the global minimum rather thanmerely oscillate around the minimum.

14
Here, Aij denotes the (i, j) entry of the matrix A. We then have
 3

10A12
2
∇A f (A) =
.
A22
A21

1.2.2

Least squares revisited

Armed with the tools of matrix derivatives, let us now proceed to find inclosed-form the value of θ that minimizes J(θ). We begin by re-writing J in
matrix-vectorial notation.
Given a training set, define the design matrix X to be the n-by-d matrix(actually n-by-d + 1, if we include the intercept term) that contains thetraining examples’ input values in its rows:— (x(1) )T —
 — (x(2) )T — 


X=
.
..


.
(n) T
— (x ) —Also, let ~y be the n-dimensional vector containing all the target values fromthe training set:


y (1)
 y (2) ~y =  ..  .
 . 
y (n)
Now, since hθ (x(i) ) = (x(i) )T θ, we can easily verify that 
(x(1) )T θy (1)  .. ..
Xθ − ~y = − . .
(x(n) )T θ

y (n)




hθ (x(1) ) − y (1)


..
= 
.
.
(n)
(n)
hθ (x ) − yThus, using the fact that for a vector z, we have that z T z =n11X(Xθ − ~y )T (Xθ − ~y ) =(hθ (x(i) ) − y (i) )22
2 i=1= J(θ)

2i zi :

P

15
Finally, to minimize J, let’s find its derivatives with respect to θ. Hence,1∇θ J(θ) = ∇θ (Xθ − ~y )T (Xθ − ~y )21=
∇θ (Xθ)T Xθ − (Xθ)T ~y − ~y T (Xθ) + ~y T ~y21
∇θ θT (X T X)θ − ~y T (Xθ) − ~y T (Xθ)=
21∇θ θT (X T X)θ − 2(X T ~y )T θ
=
2

1
=
2X T Xθ − 2X T ~y2
= X T Xθ − X T ~yIn the third step, we used the fact that aT b = bT a, and in the fifth step
used the facts ∇x bT x = b and ∇x xT Ax = 2Ax for symmetric matrix A (formore details, see Section 4.3 of “Linear Algebra Review and Reference”). To
minimize J, we set its derivatives to zero, and obtain the normal equations:
X T Xθ = X T ~yThus, the value of θ that minimizes J(θ) is given in closed form by theequationθ = (X T X)−1 X T ~y .3

1.3

Probabilistic interpretation

When faced with a regression problem, why might linear regression, andspecifically why might the least-squares cost function J, be a reasonablechoice? In this section, we will give a set of probabilistic assumptions, underwhich least-squares regression is derived as a very natural algorithm.
Let us assume that the target variables and the inputs are related via theequationy (i) = θT x(i) + (i) ,3Note that in the above step, we are implicitly assuming that X T X is an invertible
matrix. This can be checked before calculating the inverse. If either the number oflinearly independent examples is fewer than the number of features, or if the features
are not linearly independent, then X T X will not be invertible. Even in such cases, it ispossible to “fix” the situation with additional techniques, which we skip here for the sakeof simplicty.

16
where (i) is an error term that captures either unmodeled effects (such asif there are some features very pertinent to predicting housing price, butthat we’d left out of the regression), or random noise. Let us further assume
that the (i) are distributed IID (independently and identically distributed)
according to a Gaussian distribution (also called a Normal distribution) withmean zero and some variance σ 2 . We can write this assumption as “(i) ∼N (0, σ 2 ).” I.e., the density of (i) is given by1((i) )2(i)p( ) = √
exp −
.
2σ 2
2πσ
This implies that1(y (i) − θT x(i) )2.
p(y |x ; θ) = √
exp −
2σ 2
2πσ
(i)

(i)

The notation “p(y (i) |x(i) ; θ)” indicates that this is the distribution of y (i)given x(i) and parameterized by θ. Note that we should not condition on θ(“p(y (i) |x(i) , θ)”), since θ is not a random variable. We can also write the
distribution of y (i) as y (i) | x(i) ; θ ∼ N (θT x(i) , σ 2 ).
Given X (the design matrix, which contains all the x(i) ’s) and θ, whatis the distribution of the y (i) ’s? The probability of the data is given byp(~y |X; θ). This quantity is typically viewed a function of ~y (and perhaps X),
for a fixed value of θ. When we wish to explicitly view this as a function ofθ, we will instead call it the likelihood function:
L(θ) = L(θ; X, ~y ) = p(~y |X; θ).
Note that by the independence assumption on the (i) ’s (and hence also they (i) ’s given the x(i) ’s), this can also be written
L(θ) =nYp(y (i) | x(i) ; θ)

i=1
n
Y



(y (i) − θT x(i) )2
1
√
exp −
.
=
2
2σ
2πσ
i=1
Now, given this probabilistic model relating the y (i) ’s and the x(i) ’s, what
is a reasonable way of choosing our best guess of the parameters θ? The
principal of maximum likelihood says that we should choose θ so as tomake the data as high probability as possible. I.e., we should choose θ tomaximize L(θ).

17Instead of maximizing L(θ), we can also maximize any strictly increasing
function of L(θ). In particular, the derivations will be a bit simpler if we
instead maximize the log likelihood `(θ):`(θ) = log L(θ)


n
Y
1
(y (i) − θT x(i) )2
√
= log
exp −
2σ 2
2πσ
i=1


n
X
(y (i) − θT x(i) )2
1
=
exp −
log √
2σ 2
2πσ
i=1
n
1
1 1 X (i)= n log √− 2·(y − θT x(i) )2 .
2πσ σ 2 i=1Hence, maximizing `(θ) gives the same answer as minimizingn

1 X (i)(y − θT x(i) )2 ,2 i=1
which we recognize to be J(θ), our original least-squares cost function.
To summarize: Under the previous probabilistic assumptions on the data,
least-squares regression corresponds to finding the maximum likelihood estimate of θ. This is thus one set of assumptions under which least-squares regression can be justified as a very natural method that’s just doing maximum
likelihood estimation. (Note however that the probabilistic assumptions are
by no means necessary for least-squares to be a perfectly good and rational
procedure, and there may—and indeed there are—other natural assumptions
that can also be used to justify it.)
Note also that, in our previous discussion, our final choice of θ did notdepend on what was σ 2 , and indeed we’d have arrived at the same result
even if σ 2 were unknown. We will use this fact again later, when we talkabout the exponential family and generalized linear models.

1.4

Locally weighted linear regression (optional
reading)

Consider the problem of predicting y from x ∈ R. The leftmost figure below
shows the result of fitting a y = θ0 + θ1 x to a dataset. We see that the data
doesn’t really lie on straight line, and so the fit is not very good.

18
4.5

4.5

4

4

4.5

4

3.5

3.5

3.5

y

3

2.5

y

3

2.5

y

3

2.5

2

2

2

1.5

1.5

1.5

1

1

1

0.5

0.5

0.5

0

0

1

2

3

4
x

5

6

7

0

0

1

2

3

4
x

5

6

7

0

0

1

2

3

4

5

6

7

x

Instead, if we had added an extra feature x2 , and fit y = θ0 + θ1 x + θ2 x2 ,then we obtain a slightly better fit to the data. (See middle figure) Naively, it
might seem that the more features we add, the better. However, there is alsoa danger in adding too many features:
P The rightmost figure is the result offitting a 5-th order polynomial y = 5j=0 θj xj . We see that even though thefitted curve passes through the data perfectly, we would not expect this tobe a very good predictor of, say, housing prices (y) for different living areas(x). Without formally defining what these terms mean, we’ll say the figureon the left shows an instance of underfitting—in which the data clearlyshows structure not captured by the model—and the figure on the right is
an example of overfitting. (Later in this class, when we talk about learningtheory we’ll formalize some of these notions, and also define more carefully
just what it means for a hypothesis to be good or bad.)
As discussed previously, and as shown in the example above, the choice offeatures is important to ensuring good performance of a learning algorithm.
(When we talk about model selection, we’ll also see algorithms for automatically choosing a good set of features.) In this section, let us briefly talkabout the locally weighted linear regression (LWR) algorithm which, assuming there is sufficient training data, makes the choice of features less critical.
This treatment will be brief, since you’ll get a chance to explore some of theproperties of the LWR algorithm yourself in the homework.
In the original linear regression algorithm, to make a prediction at a query
point x (i.e., to evaluate h(x)), we would:P
1. Fit θ to minimize i (y (i) − θT x(i) )2 .
2. Output θT x.
In contrast, the locally weighted linear regression algorithm does the following:P
1. Fit θ to minimize i w(i) (y (i) − θT x(i) )2 .
2. Output θT x.

19Here, the w(i) ’s are non-negative valued weights. Intuitively, if w(i) is large
for a particular value of i, then in picking θ, we’ll try hard to make (y (i) −θT x(i) )2 small. If w(i) is small, then the (y (i) − θT x(i) )2 error term will bepretty much ignored in the fit.
A fairly standard choice for the weights is4(x(i) − x)2(i)w = exp −2τ 2
Note that the weights depend on the particular point x at which we’re trying
to evaluate x. Moreover, if |x(i) − x| is small, then w(i) is close to 1; andif |x(i) − x| is large, then w(i) is small. Hence, θ is chosen giving a muchhigher “weight” to the (errors on) training examples close to the query point
x. (Note also that while the formula for the weights takes a form that is
cosmetically similar to the density of a Gaussian distribution, the w(i) ’s do
not directly have anything to do with Gaussians, and in particular the w(i)are not random variables, normally distributed or otherwise.) The parameterτ controls how quickly the weight of a training example falls off with distanceof its x(i) from the query point x; τ is called the bandwidth parameter, andis also something that you’ll get to experiment with in your homework.
Locally weighted linear regression is the first example we’re seeing of anon-parametric algorithm. The (unweighted) linear regression algorithm
that we saw earlier is known as a parametric learning algorithm, because
it has a fixed, finite number of parameters (the θi ’s), which are fit to thedata. Once we’ve fit the θi ’s and stored them away, we no longer need tokeep the training data around to make future predictions. In contrast, tomake predictions using locally weighted linear regression, we need to keep
the entire training set around. The term “non-parametric” (roughly) refers
to the fact that the amount of stuff we need to keep in order to represent the
hypothesis h grows linearly with the size of the training set.

4

If x is vector-valued, this is generalized to be w(i) = exp(−(x(i) − x)T (x(i) − x)/(2τ 2 )),or w(i) = exp(−(x(i) − x)T Σ−1 (x(i) − x)/(2τ 2 )), for an appropriate choice of τ or Σ.

Chapter 2Classification and logistic
regression
Let’s now talk about the classification problem. This is just like the regression
problem, except that the values y we now want to predict take on onlya small number of discrete values. For now, we will focus on the binaryclassification problem in which y can take on only two values, 0 and 1.
(Most of what we say here will also generalize to the multiple-class case.)
For instance, if we are trying to build a spam classifier for email, then x(i)may be some features of a piece of email, and y may be 1 if it is a pieceof spam mail, and 0 otherwise. 0 is also called the negative class, and 1
the positive class, and they are sometimes also denoted by the symbols “-”
and “+.” Given x(i) , the corresponding y (i) is also called the label for the
training example.

2.1

Logistic regression

We could approach the classification problem ignoring the fact that y isdiscrete-valued, and use our old linear regression algorithm to try to predicty given x. However, it is easy to construct examples where this method
performs very poorly. Intuitively, it also doesn’t make sense for hθ (x) to takevalues larger than 1 or smaller than 0 when we know that y ∈ {0, 1}.
To fix this, let’s change the form for our hypotheses hθ (x). We will choose1hθ (x) = g(θT x) =,1 + e−θT x
where
1g(z) =
1 + e−z20

21
is called the logistic function or the sigmoid function. Here is a plot
showing g(z):
1

0.9

0.8

0.7

g(z)

0.6

0.5

0.4

0.3

0.2

0.1

0
−5

−4

−3

−2

−1

0
z

1

2

3

4

5

Notice that g(z) tends towards 1 as z → ∞, and g(z) tends towards 0 as
z → −∞. Moreover, g(z), and hence also h(x), is always bounded between
0 and 1. AsP
before, we are keeping the convention of letting x0 = 1, so that
θT x = θ0 + dj=1 θj xj .
For now, let’s take the choice of g as given. Other functions that smoothly
increase from 0 to 1 can also be used, but for a couple of reasons that we’ll see
later (when we talk about GLMs, and when we talk about generative learningalgorithms), the choice of the logistic function is a fairly natural one. Before
moving on, here’s a useful property of the derivative of the sigmoid function,
which we write as g 0 :1
ddz 1 + e−z

1
e−z
=
−z
2
(1 + e )


1
1
· 1−
=
(1 + e−z )(1 + e−z )= g(z)(1 − g(z)).

g 0 (z) =So, given the logistic regression model, how do we fit θ for it? Following
how we saw least squares regression could be derived as the maximum likelihood estimator under a set of assumptions, let’s endow our classification
model with a set of probabilistic assumptions, and then fit the parameters
via maximum likelihood.

22
Let us assume that
P (y = 1 | x; θ) = hθ (x)P (y = 0 | x; θ) = 1 − hθ (x)Note that this can be written more compactly asp(y | x; θ) = (hθ (x))y (1 − hθ (x))1−y
Assuming that the n training examples were generated independently, we
can then write down the likelihood of the parameters as
L(θ) = p(~y | X; θ)nY=p(y (i) | x(i) ; θ)
i=1

=

n
Y

hθ (x(i) )y(i)1−y(i)1 − hθ (x(i) )i=1

As before, it will be easier to maximize the log likelihood:`(θ) = log L(θ)
nX=y (i) log h(x(i) ) + (1 − y (i) ) log(1 − h(x(i) ))i=1

How do we maximize the likelihood? Similar to our derivation in the caseof linear regression, we can use gradient ascent. Written in vectorial notation,
our updates will therefore be given by θ := θ + α∇θ `(θ). (Note the positive
rather than negative sign in the update formula, since we’re maximizing,rather than minimizing, a function now.) Let’s start by working with just
one training example (x, y), and take derivatives to derive the stochastic
gradient ascent rule:


∂
1
1
∂
`(θ) =y− (1 − y)g(θT x)TT∂θjg(θ x)
1 − g(θ x) ∂θj


1
1
∂ T
=
y
− (1 − y)g(θT x)(1 − g(θT x))
θ x
TTg(θ x)
1 − g(θ x)∂θj
= y(1 − g(θT x)) − (1 − y)g(θT x) xj= (y − hθ (x)) xj23
Above, we used the fact that g 0 (z) = g(z)(1 − g(z)). This therefore gives us
the stochastic gradient ascent rule (i)θj := θj + α y (i) − hθ (x(i) ) xj
If we compare this to the LMS update rule, we see that it looks identical; but
this is not the same algorithm, because hθ (x(i) ) is now defined as a non-linear
function of θT x(i) . Nonetheless, it’s a little surprising that we end up withthe same update rule for a rather different algorithm and learning problem.
Is this coincidence, or is there a deeper reason behind this? We’ll answer thiswhen we get to GLM models.

2.2

Digression: the perceptron learning algorithn

We now digress to talk briefly about an algorithm that’s of some historicalinterest, and that we will also return to later when we talk about learningtheory. Consider modifying the logistic regression method to “force” it to
output values that are either 0 or 1 or exactly. To do so, it seems natural tochange the definition of g to be the threshold function:

1 if z ≥ 0g(z) =
0 if z < 0
If we then let hθ (x) = g(θT x) as before but using this modified definition ofg, and if we use the update rule (i)θj := θj + α y (i) − hθ (x(i) ) xj .
then we have the perceptron learning algorithn.
In the 1960s, this “perceptron” was argued to be a rough model for how
individual neurons in the brain work. Given how simple the algorithm is, itwill also provide a starting point for our analysis when we talk about learningtheory later in this class. Note however that even though the perceptron may
be cosmetically similar to the other algorithms we talked about, it is actuallya very different type of algorithm than logistic regression and least squares
linear regression; in particular, it is difficult to endow the perceptron’s predictions with meaningful probabilistic interpretations, or derive the perceptron
as a maximum likelihood estimation algorithm.

24

50

40

40

40

30

30

30
f(x)

60

50

f(x)

60

50

f(x)

60

20

20

20

10

10

10

0

0

0

−10

−10

−10

1

2.3

1.5

2

2.5

3
x

3.5

4

4.5

5

1

1.5

2

2.5

3
x

3.5

4

4.5

5

1

1.5

2

2.5

3
x

3.5

4

4.5

5

Another algorithm for maximizing `(θ)

Returning to logistic regression with g(z) being the sigmoid function, let’s
now talk about a different algorithm for maximizing `(θ).
To get us started, let’s consider Newton’s method for finding a zero of afunction. Specifically, suppose we have some function f : R 7→ R, and wewish to find a value of θ so that f (θ) = 0. Here, θ ∈ R is a real number.
Newton’s method performs the following update:θ := θ −f (θ).
f 0 (θ)This method has a natural interpretation in which we can think of it asapproximating the function f via a linear function that is tangent to f at
the current guess θ, solving for where that linear function equals to zero, andletting the next guess for θ be where that linear function is zero.
Here’s a picture of the Newton’s method in action:
In the leftmost figure, we see the function f plotted along with the line
y = 0. We’re trying to find θ so that f (θ) = 0; the value of θ that achieves thisis about 1.3. Suppose we initialized the algorithm with θ = 4.5. Newton’s
method then fits a straight line tangent to f at θ = 4.5, and solves for thewhere that line evaluates to 0. (Middle figure.) This give us the next guessfor θ, which is about 2.8. The rightmost figure shows the result of runningone more iteration, which the updates θ to about 1.8. After a few moreiterations, we rapidly approach θ = 1.3.
Newton’s method gives a way of getting to f (θ) = 0. What if we want to
use it to maximize some function `? The maxima of ` correspond to points
where its first derivative `0 (θ) is zero. So, by letting f (θ) = `0 (θ), we can use
the same algorithm to maximize `, and we obtain update rule:`0 (θ)
θ := θ − 00 .
` (θ)(Something to think about: How would this change if we wanted to use
Newton’s method to minimize rather than maximize a function?)

25
Lastly, in our logistic regression setting, θ is vector-valued, so we need togeneralize Newton’s method to this setting. The generalization of Newton’s
method to this multidimensional setting (also called the Newton-Raphson
method) is given byθ := θ − H −1 ∇θ `(θ).
Here, ∇θ `(θ) is, as usual, the vector of partial derivatives of `(θ) with respect
to the θi ’s; and H is an d-by-d matrix (actually, d+1−by−d+1, assuming that
we include the intercept term) called the Hessian, whose entries are givenby∂ 2 `(θ).
Hij =∂θi ∂θj
Newton’s method typically enjoys faster convergence than (batch) gradient descent, and requires many fewer iterations to get very close to theminimum. One iteration of Newton’s can, however, be more expensive than
one iteration of gradient descent, since it requires finding and inverting an
d-by-d Hessian; but so long as d is not too large, it is usually much fasteroverall. When Newton’s method is applied to maximize the logistic regression log likelihood function `(θ), the resulting method is also called Fisherscoring.

Chapter 3
Generalized linear models
So far, we’ve seen a regression example, and a classification example. In the
regression example, we had y|x; θ ∼ N (µ, σ 2 ), and in the classification one,y|x; θ ∼ Bernoulli(φ), for some appropriate definitions of µ and φ as functionsof x and θ. In this section, we will show that both of these methods are
special cases of a broader family of models, called Generalized Linear Models(GLMs).1 We will also show how other models in the GLM family can bederived and applied to other classification and regression problems.

3.1

The exponential family

To work our way up to GLMs, we will begin by defining exponential family
distributions. We say that a class of distributions is in the exponential family
if it can be written in the formp(y; η) = b(y) exp(η T T (y) − a(η))(3.1)

Here, η is called the natural parameter (also called the canonical parameter) of the distribution; T (y) is the sufficient statistic (for the distributions we consider, it will often be the case that T (y) = y); and a(η) is the log
partition function. The quantity e−a(η) essentially plays the role of a normalization constant, that makes sure the distribution p(y; η) sums/integrates
over y to 1.
A fixed choice of T , a and b defines a family (or set) of distributions thatis parameterized by η; as we vary η, we then get different distributions within
this family.
1

The presentation of the material in this section takes inspiration from Michael I.
Jordan, Learning in graphical models (unpublished book draft), and also McCullagh and
Nelder, Generalized Linear Models (2nd ed.).

26

27
We now show that the Bernoulli and the Gaussian distributions are examples of exponential family distributions. The Bernoulli distribution withmean φ, written Bernoulli(φ), specifies a distribution over y ∈ {0, 1}, so thatp(y = 1; φ) = φ; p(y = 0; φ) = 1 − φ. As we vary φ, we obtain Bernoulli
distributions with different means. We now show that this class of Bernoulli
distributions, ones obtained by varying φ, is in the exponential family; i.e.,that there is a choice of T , a and b so that Equation (3.1) becomes exactly
the class of Bernoulli distributions.
We write the Bernoulli distribution as:p(y; φ) = φy (1 − φ)1−y= exp(y log φ + (1 − y) log(1 − φ))


 
φ
y + log(1 − φ) .
= exp
log
1−φ
Thus, the natural parameter is given by η = log(φ/(1 − φ)). Interestingly, if
we invert this definition for η by solving for φ in terms of η, we obtain φ =1/(1 + e−η ). This is the familiar sigmoid function! This will come up again
when we derive logistic regression as a GLM. To complete the formulationof the Bernoulli distribution as an exponential family distribution, we also
have
T (y) = ya(η) = − log(1 − φ)= log(1 + eη )b(y) = 1
This shows that the Bernoulli distribution can be written in the form ofEquation (3.1), using an appropriate choice of T , a and b.
Let’s now move on to consider the Gaussian distribution. Recall that,when deriving linear regression, the value of σ 2 had no effect on our final
choice of θ and hθ (x). Thus, we can choose an arbitrary value for σ 2 withoutchanging anything. To simplify the derivation below, let’s set σ 2 = 1.2 We
2If we leave σ 2 as a variable, the Gaussian distribution can also be shown to be in the
exponential family, where η ∈ R2 is now a 2-dimension vector that depends on both µ andσ. For the purposes of GLMs, however, the σ 2 parameter can also be treated by consideringa more general definition of the exponential family: p(y; η, τ ) = b(a, τ ) exp((η T T (y) −a(η))/c(τ )). Here, τ is called the dispersion parameter, and for the Gaussian, c(τ ) = σ 2 ;but given our simplification above, we won’t need the more general definition for theexamples we will consider here.

28
then have:


1
1
2
p(y; µ) = √ exp − (y − µ)
2
2π




1
1 2
1 2
= √ exp − y · exp µy − µ222πThus, we see that the Gaussian is in the exponential family, with
η = µ
T (y) = ya(η) = µ2 /2= η 2 /2√b(y) = (1/ 2π) exp(−y 2 /2).
There’re many other distributions that are members of the exponential family: The multinomial (which we’ll see later), the Poisson (for modelling count-data; also see the problem set); the gamma and the exponential (for modelling continuous, non-negative random variables, such as timeintervals); the beta and the Dirichlet (for distributions over probabilities);and many more. In the next section, we will describe a general “recipe”
for constructing models in which y (given x and θ) comes from any of these
distributions.

3.2

Constructing GLMs

Suppose you would like to build a model to estimate the number y of customers arriving in your store (or number of page-views on your website) inany given hour, based on certain features x such as store promotions, recent
advertising, weather, day-of-week, etc. We know that the Poisson distribution usually gives a good model for numbers of visitors. Knowing this, how
can we come up with a model for our problem? Fortunately, the Poisson is anexponential family distribution, so we can apply a Generalized Linear Model(GLM). In this section, we will we will describe a method for constructing
GLM models for problems such as these.
More generally, consider a classification or regression problem where wewould like to predict the value of some random variable y as a function of
x. To derive a GLM for this problem, we will make the following threeassumptions about the conditional distribution of y given x and about ourmodel:

29
1. y | x; θ ∼ ExponentialFamily(η). I.e., given x and θ, the distribution ofy follows some exponential family distribution, with parameter η.
2. Given x, our goal is to predict the expected value of T (y) given x.
In most of our examples, we will have T (y) = y, so this means wewould like the prediction h(x) output by our learned hypothesis h tosatisfy h(x) = E[y|x]. (Note that this assumption is satisfied in the
choices for hθ (x) for both logistic regression and linear regression. For
instance, in logistic regression, we had hθ (x) = p(y = 1|x; θ) = 0 · p(y =0|x; θ) + 1 · p(y = 1|x; θ) = E[y|x; θ].)3. The natural parameter η and the inputs x are related linearly: η = θT x.
(Or, if η is vector-valued, then ηi = θiT x.)
The third of these assumptions might seem the least well justified of
the above, and it might be better thought of as a “design choice” in ourrecipe for designing GLMs, rather than as an assumption per se. These
three assumptions/design choices will allow us to derive a very elegant classof learning algorithms, namely GLMs, that have many desirable properties
such as ease of learning. Furthermore, the resulting models are often very
effective for modelling different types of distributions over y; for example, wewill shortly show that both logistic regression and ordinary least squares canboth be derived as GLMs.

3.2.1

Ordinary least squares

To show that ordinary least squares is a special case of the GLM familyof models, consider the setting where the target variable y (also called the
response variable in GLM terminology) is continuous, and we model theconditional distribution of y given x as a Gaussian N (µ, σ 2 ). (Here, µ may
depend x.) So, we let the ExponentialF amily(η) distribution above be
the Gaussian distribution. As we saw previously, in the formulation of theGaussian as an exponential family distribution, we had µ = η. So, we havehθ (x) ====E[y|x; θ]
µ
η
θT x.

The first equality follows from Assumption 2, above; the second equality
follows from the fact that y|x; θ ∼ N (µ, σ 2 ), and so its expected value is given

30by µ; the third equality follows from Assumption 1 (and our earlier derivation
showing that µ = η in the formulation of the Gaussian as an exponential
family distribution); and the last equality follows from Assumption 3.

3.2.2

Logistic regression

We now consider logistic regression. Here we are interested in binary classification, so y ∈ {0, 1}. Given that y is binary-valued, it therefore seems natural
to choose the Bernoulli family of distributions to model the conditional distribution of y given x. In our formulation of the Bernoulli distribution as
an exponential family distribution, we had φ = 1/(1 + e−η ). Furthermore,note that if y|x; θ ∼ Bernoulli(φ), then E[y|x; θ] = φ. So, following a similar
derivation as the one for ordinary least squares, we get:hθ (x) = E[y|x; θ]= φ= 1/(1 + e−η )T= 1/(1 + e−θ x )TSo, this gives us hypothesis functions of the form hθ (x) = 1/(1 + e−θ x ). If
you are previously wondering how we came up with the form of the logistic
function 1/(1 + e−z ), this gives one answer: Once we assume that y conditioned on x is Bernoulli, it arises as a consequence of the definition of GLMs
and exponential family distributions.
To introduce a little more terminology, the function g giving the distribution’s mean as a function of the natural parameter (g(η) = E[T (y); η])is called the canonical response function. Its inverse, g −1 , is called thecanonical link function. Thus, the canonical response function for theGaussian family is just the identify function; and the canonical response
function for the Bernoulli is the logistic function.3

3.2.3

Softmax regression

Let’s look at one more example of a GLM. Consider a classification problem
in which the response variable y can take on any one of k values, so y ∈{1, 2, . . . , k}. For example, rather than classifying email into the two classes
Many texts use g to denote the link function, and g −1 to denote the response function;
but the notation we’re using here, inherited from the early machine learning literature,
will be more consistent with the notation used in the rest of the class.
3

31
spam or not-spam—which would have been a binary classification problem—
we might want to classify it into three classes, such as spam, personal mail,and work-related mail. The response variable is still discrete, but can now
take on more than two values. We will thus model it as distributed according
to a multinomial distribution.
Let’s derive a GLM for modelling this type of multinomial data. To doso, we will begin by expressing the multinomial as an exponential family
distribution.
To parameterize a multinomial over k possible outcomes, one could use
k parameters φ1 , . . . , φk specifying the probability of each of the outcomes.
However, these parameters would be redundant, or more formally, they would
not be independent (since knowing any
Pk k − 1 of the φi ’s uniquely determines
the last one, as they must satisfy
i=1 φi = 1). So, we will instead parameterize the multinomial with only kP− 1 parameters, φ1 , . . . , φk−1 , wherek−1
φi . For notational convenience,
φi = p(y = i; φ), and p(y = k; φ) = 1 − i=1
Pk−1
we will also let φk = 1 − i=1 φi , but we should keep in mind that this is
not a parameter, and that it is fully specified by φ1 , . . . , φk−1 .
To express the multinomial as an exponential family distribution, we willdefine T (y) ∈ Rk−1 as follows:









0
0
0
0
1
 0 
 0 
 0 
 1 
 0 












 0 
 1 
 0 
 0 
T (1) = 
 , T (k) =  0  , , · · · , T (k−1) = 
 , T (3) = 
 , T (2) =  .. 
 .. 
 .. 
 .. 
 .. 
 . 
 . 
 . 
 . 
 . 


0

0

0

1

0

Unlike our previous examples, here we do not have T (y) = y; also, T (y) is
now a k − 1 dimensional vector, rather than a real number. We will write(T (y))i to denote the i-th element of the vector T (y).
We introduce one more very useful piece of notation. An indicator function 1{·} takes on a value of 1 if its argument is true, and 0 otherwise(1{True} = 1, 1{False} = 0). For example, 1{2 = 3} = 0, and 1{3 =5 − 2} = 1. So, we can also write the relationship between T (y) and y as
(T (y))i = 1{y = i}. (Before you continue reading, please make sure you understand why this is true!) Further, we have that E[(T (y))i ] = P (y = i) = φi .
We are now ready to show that the multinomial is a member of the32
exponential family. We have:1{y=1} 1{y=2}1{y=k}
φ2· · · φkPk−1
1− i=11{y=i}
1{y=1} 1{y=2}φ1
φ2· · · φkP
1− k−1 (T (y))i(T (y))1 (T (y))2· · · φk i=1φ2φ1p(y; φ) = φ1==
= exp((T (y))1 log(φ1 ) + (T (y))2 log(φ2 ) +
Pk−1· · · + 1 − i=1 (T (y))i log(φk ))= exp((T (y))1 log(φ1 /φk ) + (T (y))2 log(φ2 /φk ) +· · · + (T (y))k−1 log(φk−1 /φk ) + log(φk ))= b(y) exp(η T T (y) − a(η))
where



η = 


log(φ1 /φk )log(φ2 /φk )
..
.




,


log(φk−1 /φk )a(η) = − log(φk )b(y) = 1.
This completes our formulation of the multinomial as an exponential family
distribution.
The link function is given (for i = 1, . . . , k) by
ηi = logφi.
φk

For convenience, we have also defined ηk = log(φk /φk ) = 0. To invert thelink function and derive the response function, we therefore have that
φi
φk
= φi
k
X
=
φi = 1eηi =φk eηi
k
X
φk
eηi
i=1

(3.2)

i=1

Pk

This implies that φk = 1/ i=1 eηi , which can be substituted back into Equation (3.2) to give the response functioneηiφi = Pk

j=1 e

ηj

33
This function mapping from the η’s to the φ’s is called the softmax function.
To complete our model, we use Assumption 3, given earlier, that the ηi ’s
are linearly related to the x’s. So, have ηi = θiT x (for i = 1, . . . , k − 1),
where θ1 , . . . , θk−1 ∈ Rd+1 are the parameters of our model. For notational
convenience, we can also define θk = 0, so that ηk = θkT x = 0, as given
previously. Hence, our model assumes that the conditional distribution of ygiven x is given byp(y = i|x; θ) = φi
eηi
= Pk
ηj
j=1 e
T

eθi x

= Pk

j=1 e

(3.3)

θjT x

This model, which applies to classification problems where y ∈ {1, . . . , k}, iscalled softmax regression. It is a generalization of logistic regression.
Our hypothesis will outputhθ (x) = E[T (y)|x; θ]
1{y = 1}
 1{y = 2}

= E
..

.
1{y = k − 1}
φ1
 φ2 =  .. 
 . 
φk−1


exp(θT x)




x; θ


1

Pk

T

j x) 
 j=1 exp(θ P exp(θ2T x) k exp(θT x) 
j=1
j
.
= 


..


.
 exp(θT x) k−1T
j=1 exp(θj x)

Pk

In other words, our hypothesis will output the estimated probability thatp(y = i|x; θ), for every value of i = 1, . . . , k. (Even though hθ (x) as defined
above
is only k − 1 dimensional, clearly p(y = k|x; θ) can be obtained as
Pk−1
1 − i=1 φi .)34Lastly, let’s discuss parameter fitting. Similar to our original derivationof ordinary least squares and logistic regression, if we have a training set ofn examples {(x(i) , y (i) ); i = 1, . . . , n} and would like to learn the parameters
θi of this model, we would begin by writing down the log-likelihood`(θ) =nX

log p(y (i) |x(i) ; θ)

i=1

=

n
X
i=1

log

k
Y
l=1

T

eθl x

(i)

!1{y(i) =l}

θjT x(i)j=1 ePkTo obtain the second line above, we used the definition for p(y|x; θ) given
in Equation (3.3). We can now obtain the maximum likelihood estimate of
the parameters by maximizing `(θ) in terms of θ, using a method such as
gradient ascent or Newton’s method.

Chapter 4
Generative learning algorithms
So far, we’ve mainly been talking about learning algorithms that modelp(y|x; θ), the conditional distribution of y given x. For instance, logistic
regression modeled p(y|x; θ) as hθ (x) = g(θT x) where g is the sigmoid function. In these notes, we’ll talk about a different type of learning algorithm.
Consider a classification problem in which we want to learn to distinguishbetween elephants (y = 1) and dogs (y = 0), based on some features ofan animal. Given a training set, an algorithm like logistic regression or
the perceptron algorithm (basically) tries to find a straight line—that is, adecision boundary—that separates the elephants and dogs. Then, to classify
a new animal as either an elephant or a dog, it checks on which side of thedecision boundary it falls, and makes its prediction accordingly.
Here’s a different approach. First, looking at elephants, we can build amodel of what elephants look like. Then, looking at dogs, we can build aseparate model of what dogs look like. Finally, to classify a new animal, we
can match the new animal against the elephant model, and match it against
the dog model, to see whether the new animal looks more like the elephants
or more like the dogs we had seen in the training set.
Algorithms that try to learn p(y|x) directly (such as logistic regression),or algorithms that try to learn mappings directly from the space of inputs X
to the labels {0, 1}, (such as the perceptron algorithm) are called discriminative learning algorithms. Here, we’ll talk about algorithms that instead
try to model p(x|y) (and p(y)). These algorithms are called generative
learning algorithms. For instance, if y indicates whether an example is adog (0) or an elephant (1), then p(x|y = 0) models the distribution of dogs’
features, and p(x|y = 1) models the distribution of elephants’ features.
After modeling p(y) (called the class priors) and p(x|y), our algorithm35

36
can then use Bayes rule to derive the posterior distribution on y given x:p(y|x) =p(x|y)p(y).
p(x)

Here, the denominator is given by p(x) = p(x|y = 1)p(y = 1) + p(x|y =0)p(y = 0) (you should be able to verify that this is true from the standard
properties of probabilities), and thus can also be expressed in terms of thequantities p(x|y) and p(y) that we’ve learned. Actually, if were calculatingp(y|x) in order to make a prediction, then we don’t actually need to calculate
the denominator, sincep(x|y)p(y)yp(x)= arg max p(x|y)p(y).

arg max p(y|x) = arg max
y

y

4.1

Gaussian discriminant analysis

The first generative learning algorithm that we’ll look at is Gaussian discriminant analysis (GDA). In this model, we’ll assume that p(x|y) is distributed
according to a multivariate normal distribution. Let’s talk briefly about theproperties of multivariate normal distributions before moving on to the GDA
model itself.

4.1.1

The multivariate normal distribution

The multivariate normal distribution in d-dimensions, also called the multivariate Gaussian distribution, is parameterized by a mean vector µ ∈ Rd
and a covariance matrix Σ ∈ Rd×d , where Σ ≥ 0 is symmetric and positive
semi-definite. Also written “N (µ, Σ)”, its density is given by:


1
1
T −1
p(x; µ, Σ) =exp − (x − µ) Σ (x − µ) .
(2π)d/2 |Σ|1/2
2
In the equation above, “|Σ|” denotes the determinant of the matrix Σ.
For a random variable X distributed N (µ, Σ), the mean is (unsurprisingly) given by µ:Z
E[X] =x p(x; µ, Σ)dx = µxThe covariance of a vector-valued random variable Z is defined as Cov(Z) =E[(Z − E[Z])(Z − E[Z])T ]. This generalizes the notion of the variance of a37
real-valued random variable. The covariance can also be defined as Cov(Z) =E[ZZ T ] − (E[Z])(E[Z])T . (You should be able to prove to yourself that thesetwo definitions are equivalent.) If X ∼ N (µ, Σ), thenCov(X) = Σ.
Here are some examples of what the density of a Gaussian distribution
looks like:

0.25

0.25

0.2

0.2

0.25

0.2

0.15

0.15

0.15

0.1

0.1

0.1

0.05

0.05

0.05

3

3
2

3
2

3

1

1
−1

−2

−2
−3

−3

0

−1

−1

−2

−2
−3

3
2
0

1
0

−1

−1

−2

1

2
0

1
0

−1

2

3

1

2
0

−2
−3

−3

−3

The left-most figure shows a Gaussian with mean zero (that is, the 2x1
zero-vector) and covariance matrix Σ = I (the 2x2 identity matrix). A Gaussian with zero mean and identity covariance is also called the standard normal distribution. The middle figure shows the density of a Gaussian withzero mean and Σ = 0.6I; and in the rightmost figure shows one with , Σ = 2I.
We see that as Σ becomes larger, the Gaussian becomes more “spread-out,”
and as it becomes smaller, the distribution becomes more “compressed.”Let’s look at some more examples.

0.25

0.25

0.25

0.2

0.2

0.2

0.15

0.15

0.15

0.1

0.1

0.1

0.05

0.05

0.05

3

3

3
2

2

2
1

1

1
0

0

0

3
−1

2

2

1
0

−2
−2
−3

−1

2
1

1
−2

0

−1
−3

3

3

−1

0

−2
−1

−1
−3

−2
−3

−3

−2
−3

The figures above show Gaussians with mean 0, and with covariancematrices respectively






1 0
1 0.5
1 0.8
Σ=
; Σ=
; Σ=
.
0 1
0.5 1
0.8 1
The leftmost figure shows the familiar standard normal distribution, and we
see that as we increase the off-diagonal entry in Σ, the density becomes more38“compressed” towards the 45◦ line (given by x1 = x2 ). We can see this moreclearly when we look at the contours of the same three densities:
3

3

3

2

2

2

1

1

1

0

0

0

−1

−1

−1

−2

−2

−2

−3

−3
−3

−2

−1

0

1

2

3

−3
−3

−2

−1

0

1

2

3

−3

−2

−1

0

1

2

3

0

1

2

3

Here’s one last set of examples generated by varying Σ:
3

3

3

2

2

2

1

1

1

0

0

0

−1

−1

−1

−2

−2

−2

−3

−3
−3

−2

−1

0

1

2

3

−3
−3

−2

−1

0

1

2

3

−3

−2

−1

The plots above used, respectively,





1
-0.5
1
-0.8
3
Σ=
; Σ=
; Σ=
-0.5
1
-0.8
1
0.8



0.8
1

.

From the leftmost and middle figures, we see that by decreasing the offdiagonal elements of the covariance matrix, the density now becomes “compressed” again, but in the opposite direction. Lastly, as we vary the parameters, more generally the contours will form ellipses (the rightmost figure
showing an example).
As our last set of examples, fixing Σ = I, by varying µ, we can also movethe mean of the density around.

0.25

0.25

0.2

0.2

0.25

0.2

0.15

0.15

0.15

0.1

0.1

0.1

0.05

0.05

0.05

3

3
2

3

1

2
0

1
0

−1
−1

−2

−2
−3

−3

3
2

3

1

2
0

1
0

−1
−1

−2

−2
−3

−3

2

3

1

2
0

1
0

−1
−1

−2

−2
−3

−3

39
The figures above were generated using Σ = I, and respectively
 




1
-0.5
-1
µ=
; µ=
; µ=
.
0
0
-1.5

4.1.2

The Gaussian discriminant analysis model

When we have a classification problem in which the input features x arecontinuous-valued random variables, we can then use the Gaussian Discriminant Analysis (GDA) model, which models p(x|y) using a multivariate normal distribution. The model is:y ∼ Bernoulli(φ)
x|y = 0 ∼ N (µ0 , Σ)x|y = 1 ∼ N (µ1 , Σ)

Writing out the distributions, this is:p(y) = φy (1 − φ)1−y


1
1
T −1
exp − (x − µ0 ) Σ (x − µ0 )p(x|y = 0) =(2π)d/2 |Σ|1/2
2


1
1
T −1
p(x|y = 1) =exp − (x − µ1 ) Σ (x − µ1 )(2π)d/2 |Σ|1/2
2
Here, the parameters of our model are φ, Σ, µ0 and µ1 . (Note that while
there’re two different mean vectors µ0 and µ1 , this model is usually applied
using only one covariance matrix Σ.) The log-likelihood of the data is givenby`(φ, µ0 , µ1 , Σ) = log
= log

n
Y
i=1
n
Y
i=1

p(x(i) , y (i) ; φ, µ0 , µ1 , Σ)p(x(i) |y (i) ; µ0 , µ1 , Σ)p(y (i) ; φ).

40
By maximizing ` with respect to the parameters, we find the maximum likelihood estimate of the parameters (see problem set 1) to be:n1X1{y (i) = 1}
n i=1
Pn
(i)
= 0}x(i)i=1 1{yPµ0 =n1{y (i) = 0}
Pn i=1 (i)= 1}x(i)i=1 1{yPµ1 =n(i) = 1}i=1 1{yn1 X (i)(x − µy(i) )(x(i) − µy(i) )T .
Σ =
n i=1
φ =

Pictorially, what the algorithm is doing can be seen in as follows:
1

0

−1

−2

−3

−4

−5

−6

−7
−2

−1

0

1

2

3

4

5

6

7

Shown in the figure are the training set, as well as the contours of thetwo Gaussian distributions that have been fit to the data in each of thetwo classes. Note that the two Gaussians have contours that are the same
shape and orientation, since they share a covariance matrix Σ, but they havedifferent means µ0 and µ1 . Also shown in the figure is the straight line
giving the decision boundary at which p(y = 1|x) = 0.5. On one side of
the boundary, we’ll predict y = 1 to be the most likely outcome, and on theother side, we’ll predict y = 0.

41

4.1.3

Discussion: GDA and logistic regression

The GDA model has an interesting relationship to logistic regression. If weview the quantity p(y = 1|x; φ, µ0 , µ1 , Σ) as a function of x, we’ll find that it
can be expressed in the formp(y = 1|x; φ, Σ, µ0 , µ1 ) =1,
1 + exp(−θT x)

where θ is some appropriate function of φ, Σ, µ0 , µ1 .1 This is exactly the form
that logistic regression—a discriminative algorithm—used to model p(y =1|x).
When would we prefer one model over another? GDA and logistic regression will, in general, give different decision boundaries when trained on thesame dataset. Which is better?
We just argued that if p(x|y) is multivariate gaussian (with shared Σ),then p(y|x) necessarily follows a logistic function. The converse, however,is not true; i.e., p(y|x) being a logistic function does not imply p(x|y) is
multivariate gaussian. This shows that GDA makes stronger modeling assumptions about the data than does logistic regression. It turns out thatwhen these modeling assumptions are correct, then GDA will find better fits
to the data, and is a better model. Specifically, when p(x|y) is indeed gaussian (with shared Σ), then GDA is asymptotically efficient. Informally,
this means that in the limit of very large training sets (large n), there is noalgorithm that is strictly better than GDA (in terms of, say, how accurately
they estimate p(y|x)). In particular, it can be shown that in this setting,
GDA will be a better algorithm than logistic regression; and more generally,
even for small training set sizes, we would generally expect GDA to better.
In contrast, by making significantly weaker assumptions, logistic regression is also more robust and less sensitive to incorrect modeling assumptions.
There are many different sets of assumptions that would lead to p(y|x) taking
the form of a logistic function. For example, if x|y = 0 ∼ Poisson(λ0 ), andx|y = 1 ∼ Poisson(λ1 ), then p(y|x) will be logistic. Logistic regression will
also work well on Poisson data like this. But if we were to use GDA on suchdata—and fit Gaussian distributions to such non-Gaussian data—then the
results will be less predictable, and GDA may (or may not) do well.
To summarize: GDA makes stronger modeling assumptions, and is moredata efficient (i.e., requires less training data to learn “well”) when the modeling assumptions are correct or at least approximately correct. Logistic1This uses the convention of redefining the x(i) ’s on the right-hand-side to be (d + 1)(i)dimensional vectors by adding the extra coordinate x0 = 1; see problem set 1.

42regression makes weaker assumptions, and is significantly more robust to
deviations from modeling assumptions. Specifically, when the data is indeed non-Gaussian, then in the limit of large datasets, logistic regression willalmost always do better than GDA. For this reason, in practice logistic regression is used more often than GDA. (Some related considerations about
discriminative vs. generative models also apply for the Naive Bayes algorithm that we discuss next, but the Naive Bayes algorithm is still considereda very good, and is certainly also a very popular, classification algorithm.)

4.2

Naive bayes

In GDA, the feature vectors x were continuous, real-valued vectors. Let’s
now talk about a different learning algorithm in which the xj ’s are discretevalued.
For our motivating example, consider building an email spam filter using
machine learning. Here, we wish to classify messages according to whether
they are unsolicited commercial (spam) email, or non-spam email. Afterlearning to do this, we can then have our mail reader automatically filter
out the spam messages and perhaps place them in a separate mail folder.
Classifying emails is one example of a broader set of problems called textclassification.
Let’s say we have a training set (a set of emails labeled as spam or nonspam). We’ll begin our construction of our spam filter by specifying thefeatures xj used to represent an email.
We will represent an email via a feature vector whose length is equal to
the number of words in the dictionary. Specifically, if an email contains thej-th word of the dictionary, then we will set xj = 1; otherwise, we let xj = 0.
For instance, the vector 1a 0  aardvark  0  aardwolf
 .  .
 
x =  ..  ..
 
 1  buy .  .
 ..  ..
0
zygmurgy
is used to represent an email that contains the words “a” and “buy,” but not43“aardvark,” “aardwolf” or “zygmurgy.”2 The set of words encoded into thefeature vector is called the vocabulary, so the dimension of x is equal to
the size of the vocabulary.
Having chosen our feature vector, we now want to build a generativemodel. So, we have to model p(x|y). But if we have, say, a vocabulary of
50000 words, then x ∈ {0, 1}50000 (x is a 50000-dimensional vector of 0’s and1’s), and if we were to model x explicitly with a multinomial distribution overthe 250000 possible outcomes, then we’d end up with a (250000 −1)-dimensionalparameter vector. This is clearly too many parameters.
To model p(x|y), we will therefore make a very strong assumption. We willassume that the xi ’s are conditionally independent given y. This assumption
is called the Naive Bayes (NB) assumption, and the resulting algorithm iscalled the Naive Bayes classifier. For instance, if y = 1 means spam email;“buy” is word 2087 and “price” is word 39831; then we are assuming that if
I tell you y = 1 (that a particular piece of email is spam), then knowledgeof x2087 (knowledge of whether “buy” appears in the message) will have noeffect on your beliefs about the value of x39831 (whether “price” appears).
More formally, this can be written p(x2087 |y) = p(x2087 |y, x39831 ). (Note that
this is not the same as saying that x2087 and x39831 are independent, which
would have been written “p(x2087 ) = p(x2087 |x39831 )”; rather, we are only
assuming that x2087 and x39831 are conditionally independent given y.)
We now have:p(x1 , . . . , x50000 |y)= p(x1 |y)p(x2 |y, x1 )p(x3 |y, x1 , x2 ) · · · p(x50000 |y, x1 , . . . , x49999 )= p(x1 |y)p(x2 |y)p(x3 |y) · · · p(x50000 |y)dY=p(xj |y)j=1

The first equality simply follows from the usual properties of probabilities,
and the second equality used the NB assumption. We note that even though
2

Actually, rather than looking through an English dictionary for the list of all English
words, in practice it is more common to look through our training set and encode in our
feature vector only the words that occur at least once there. Apart from reducing thenumber of words modeled and hence reducing our computational and space requirements,
this also has the advantage of allowing us to model/include as a feature many words
that may appear in your email (such as “cs229”) but that you won’t find in a dictionary.
Sometimes (as in the homework), we also exclude the very high frequency words (which
will be words like “the,” “of,” “and”; these high frequency, “content free” words are called
stop words) since they occur in so many documents and do little to indicate whether an
email is spam or non-spam.

44
the Naive Bayes assumption is an extremely strong assumptions, the resulting
algorithm works well on many problems.
Our model is parameterized by φj|y=1 = p(xj = 1|y = 1), φj|y=0 = p(xj =1|y = 0), and φy = p(y = 1). As usual, given a training set {(x(i) , y (i) ); i =1, . . . , n}, we can write down the joint likelihood of the data:L(φy , φj|y=0 , φj|y=1 ) =nYp(x(i) , y (i) ).

i=1

Maximizing this with respect to φy , φj|y=0 and φj|y=1 gives the maximum
likelihood estimates:
Pn
(i)
(i)
= 1}
i=1 1{xj = 1 ∧ yPn
φj|y=1 =(i)= 1}i=1 1{y
Pn
(i)
(i)
= 0}
i=1 1{xj = 1 ∧ yP
φj|y=0 =n(i) = 0}i=1 1{yPn(i)= 1}i=1 1{yφy =nIn the equations above, the “∧” symbol means “and.” The parameters havea very natural interpretation. For instance, φj|y=1 is just the fraction of thespam (y = 1) emails in which word j does appear.
Having fit all these parameters, to make a prediction on a new example
with features x, we then simply calculatep(y = 1|x) =p(x|y = 1)p(y = 1)
p(x)
Q
d


j=1 p(xj |y = 1) p(y = 1)

Q

= Q
,
d
d
p(x
|y
=
1)
p(y
=
1)
+
p(x
|y
=
0)
p(y
=
0)
j
j
j=1
j=1
and pick whichever class has the higher posterior probability.
Lastly, we note that while we have developed the Naive Bayes algorithm
mainly for the case of problems where the features xj are binary-valued, thegeneralization to where xj can take values in {1, 2, . . . , kj } is straightforward.
Here, we would simply model p(xj |y) as multinomial rather than as Bernoulli.
Indeed, even if some original input attribute (say, the living area of a house,
as in our earlier example) were continuous valued, it is quite common todiscretize it—that is, turn it into a small set of discrete values—and apply
Naive Bayes. For instance, if we use some feature xj to represent living area,
we might discretize the continuous values as follows:45
Living area (sq. feet) < 400 400-800 800-1200 1200-1600 >1600
xi
1
2
3
4
5
Thus, for a house with living area 890 square feet, we would set the valueof the corresponding feature xj to 3. We can then apply the Naive Bayes
algorithm, and model p(xj |y) with a multinomial distribution, as described
previously. When the original, continuous-valued attributes are not wellmodeled by a multivariate normal distribution, discretizing the features andusing Naive Bayes (instead of GDA) will often result in a better classifier.

4.2.1

Laplace smoothing

The Naive Bayes algorithm as we have described it will work fairly wellfor many problems, but there is a simple change that makes it work muchbetter, especially for text classification. Let’s briefly discuss a problem withthe algorithm in its current form, and then talk about how we can fix it.
Consider spam/email classification, and let’s suppose that, we are in theyear of 20xx, after completing CS229 and having done excellent work on the
project, you decide around May 20xx to submit work you did to the NeurIPS
conference for publication.3 Because you end up discussing the conference
in your emails, you also start getting messages with the word “neurips”
in it. But this is your first NeurIPS paper, and until this time, you had
not previously seen any emails containing the word “neurips”; in particular“neurips” did not ever appear in your training set of spam/non-spam emails.
Assuming that “neurips” was the 35000th word in the dictionary, your Naive
Bayes spam filter therefore had picked its maximum likelihood estimates ofthe parameters φ35000|y to be(i)= 1 ∧ y (i) = 1}i=1 1{xPn35000 (i)=0= 1}
i=1 1{yPn(i)= 1 ∧ y (i) = 0}i=1 1{xPn35000 (i)=0= 0}
i=1 1{yPnφ35000|y=1 =φ35000|y=0 =I.e., because it has never seen “neurips” before in either spam or non-spamtraining examples, it thinks the probability of seeing it in either type of email
is zero. Hence, when trying to decide if one of these messages containing
3
NeurIPS is one of the top machine learning conferences. The deadline for submitting
a paper is typically in May-June.

46“neurips” is spam, it calculates the class posterior probabilities, and obtains
Qd
j=1 p(xj |y = 1)p(y = 1)p(y = 1|x) = Qd
Qd
j=1 p(xj |y = 1)p(y = 1) +j=1 p(xj |y = 0)p(y = 0)
0
=
.
0
Q
This is because each of the terms “ dj=1 p(xj |y)” includes a term p(x35000 |y) =0 that is multiplied into it. Hence, our algorithm obtains 0/0, and doesn’tknow how to make a prediction.
Stating the problem more broadly, it is statistically a bad idea to estimate the probability of some event to be zero just because you haven’t seen
it before in your finite training set. Take the problem of estimating the meanof a multinomial random variable z taking values in {1, . . . , k}. We can parameterize our multinomial with φj = p(z = j). Given a set of n independent
observations {z (1) , . . . , z (n) }, the maximum likelihood estimates are given byPn
1{z (i) = j}.
φj = i=1n
As we saw previously, if we were to use these maximum likelihood estimates,
then some of the φj ’s might end up as zero, which was a problem. To avoid
this, we can use Laplace smoothing, which replaces the above estimate
withP
1 + ni=1 1{z (i) = j}
φj =
.
k+n
Here,
Pk we’ve added 1 to the numerator, and k to the denominator. Note thatj=1 φj = 1 still holds (check this yourself!), which is a desirable property
since the φj ’s are estimates for probabilities that we know must sum to 1.
Also, φj 6= 0 for all values of j, solving our problem of probabilities being
estimated as zero. Under certain (arguably quite strong) conditions, it canbe shown that the Laplace smoothing actually gives the optimal estimatorof the φj ’s.
Returning to our Naive Bayes classifier, with Laplace smoothing, wetherefore obtain the following estimates of the parameters:
P(i)
1 + ni=1 1{xj = 1 ∧ y (i) = 1}P
φj|y=1 =2 + ni=1 1{y (i) = 1}P(i)
1 + ni=1 1{xj = 1 ∧ y (i) = 0}P
φj|y=0 =2 + ni=1 1{y (i) = 0}

47
(In practice, it usually doesn’t matter much whether we apply Laplace smoothing to φy or not, since we will typically have a fair fraction each of spam andnon-spam messages, so φy will be a reasonable estimate of p(y = 1) and will
be quite far from 0 anyway.)

4.2.2

Event models for text classification

To close off our discussion of generative learning algorithms, let’s talk about
one more model that is specifically for text classification. While Naive Bayesas we’ve presented it will work well for many classification problems, for textclassification, there is a related model that does even better.
In the specific context of text classification, Naive Bayes as presented uses
the what’s called the Bernoulli event model (or sometimes multi-variate
Bernoulli event model). In this model, we assumed that the way an email
is generated is that first it is randomly determined (according to the classpriors p(y)) whether a spammer or non-spammer will send you your next
message. Then, the person sending the email runs through the dictionary,deciding whether to include each word j in that email independently andaccording to the probabilities p(xj = 1|y) = φj|y . Thus, the probability of aQmessage was given by p(y) dj=1 p(xj |y).
Here’s a different model, called the Multinomial event model. To
describe this model, we will use a different notation and set of features forrepresenting emails. We let xj denote the identity of the j-th word in theemail. Thus, xj is now an integer taking values in {1, . . . , |V |}, where |V |is the size of our vocabulary (dictionary). An email of d words is now represented by a vector (x1 , x2 , . . . , xd ) of length d; note that d can vary for
different documents. For instance, if an email starts with “A NeurIPS . . . ,”then x1 = 1 (“a” is the first word in the dictionary), and x2 = 35000 (if“neurips” is the 35000th word in the dictionary).
In the multinomial event model, we assume that the way an email isgenerated is via a random process in which spam/non-spam is first determined (according to p(y)) as before. Then, the sender of the email writes theemail by first generating x1 from some multinomial distribution over words(p(x1 |y)). Next, the second word x2 is chosen independently of x1 but fromthe same multinomial distribution, and similarly for x3 , x4 , and so on, until
all d words of the email have
Q been generated. Thus, the overall probability ofa message is given by p(y) dj=1 p(xj |y). Note that this formula looks like theone we had earlier for the probability of a message under the Bernoulli event
model, but that the terms in the formula now mean very different things. In
particular xj |y is now a multinomial, rather than a Bernoulli distribution.

48
The parameters for our new model are φy = p(y) as before, φk|y=1 =p(xj = k|y = 1) (for any j) and φk|y=0 = p(xj = k|y = 0). Note that we haveassumed that p(xj |y) is the same for all values of j (i.e., that the distribution
according to which a word is generated does not depend on its position j
within the email).
If we are given a training set {(x(i) , y (i) ); i = 1, . . . , n} where x(i) =
(i)
(i)
(i)
(x1 , x2 , . . . , xdi ) (here, di is the number of words in the i-training example),the likelihood of the data is given by
L(φy , φk|y=0 , φk|y=1 ) =nYp(x(i) , y (i) )

i=1

=

di
n
Y
Y
i=1

!
(i)

p(xj |y; φk|y=0 , φk|y=1 ) p(y (i) ; φy ).

j=1

Maximizing this yields the maximum likelihood estimates of the parameters:
φk|y=1φk|y=0φy

Pn Pdi
(i)
(i)
= 1}
i=1
j=1 1{xj = k ∧ yPn=(i)= 1}dii=1 1{yPn Pdi
(i)
(i)
= 0}
i=1
j=1 1{xj = k ∧ yP=n(i) = 0}dii=1 1{yPn(i)= 1}i=1 1{y=.
nIf we were to apply Laplace smoothing (which is needed in practice for good
performance) when estimating φk|y=0 and φk|y=1 , we add 1 to the numerators
and |V | to the denominators, and obtain:
φk|y=1φk|y=0Pn Pdi
(i)
(i)
= 1}
i=1
j=1 1{xj = k ∧ yPn=|V | + i=1 1{y (i) = 1}diPn Pdi(i)
1 + i=1 j=1 1{xj = k ∧ y (i) = 0}
P=.
|V | + ni=1 1{y (i) = 0}di1+

While not necessarily the very best classification algorithm, the Naive Bayesclassifier often works surprisingly well. It is often also a very good “first thing
to try,” given its simplicity and ease of implementation.

Chapter 5
Kernel methods
5.1

Feature maps

Recall that in our discussion about linear regression, we considered the problem of predicting the price of a house (denoted by y) from the living area ofthe house (denoted by x), and we fit a linear function of x to the training
data. What if the price y can be more accurately represented as a non-linear
function of x? In this case, we need a more expressive family of models thanlinear models.
We start by considering fitting cubic functions y = θ3 x3 + θ2 x2 + θ1 x + θ0 .
It turns out that we can view the cubic function as a linear function over
the a different set of feature variables (defined below). Concretely, let thefunction φ : R → R4 be defined as



1
 x 
4

φ(x) =  x2  ∈ R .
x3

(5.1)

Let θ ∈ R4 be the vector containing θ0 , θ1 , θ2 , θ3 as entries. Then we canrewrite the cubic function in x as:
θ3 x3 + θ2 x2 + θ1 x + θ0 = θT φ(x)Thus, a cubic function of the variable x can be viewed as a linear function
over the variables φ(x). To distinguish between these two sets of variables,in the context of kernel methods, we will call the “original” input value theinput attributes of a problem (in this case, x, the living area). When the49

50
original input is mapped to some new set of quantities φ(x), we will call those
new quantities the features variables. (Unfortunately, different authors use
different terms to describe these two things in different contexts.) We willcall φ a feature map, which maps the attributes to the features.

5.2

LMS (least mean squares) with features

We will derive the gradient descent algorithm for fitting the model θT φ(x).
First recall that for ordinary least square problem where we were to fit θT x,
the batch gradient descent update is (see the first lecture note for its derivation):θ := θ + α:= θ + α

n
X
i=1
n
X


y (i) − hθ (x(i) ) x(i)y (i) − θT x(i) x(i) .

(5.2)

i=1

Let φ : Rd → Rp be a feature map that maps attribute x (in Rd ) to thefeatures φ(x) in Rp . (In the motivating example in the previous subsection,
we have d = 1 and p = 4.) Now our goal is to fit the function θT φ(x), withθ being a vector in Rp instead of Rd . We can replace all the occurrences ofx(i) in the algorithm above by φ(x(i) ) to obtain the new update:θ := θ + αnXy (i) − θT φ(x(i) ) φ(x(i) )

(5.3)

i=1

Similarly, the corresponding stochastic gradient descent update rule isθ := θ + α y (i) − θT φ(x(i) ) φ(x(i) )
(5.4)

5.3

LMS with the kernel trick

The gradient descent update, or stochastic gradient update above becomescomputationally expensive when the features φ(x) is high-dimensional. For
example, consider the direct extension of the feature map in equation (5.1)
to high-dimensional input x: suppose x ∈ Rd , and let φ(x) be the vector that

51
contains all the monomials of x with degree ≤ 31
 x1 

 x2 
 .. 
 . 
 2 
 x1 
 x1 x2  x1 x3 
φ(x) = .
 .. 
 .  x2 x1 
 . 
 .. 
 x3 
 1 
 x2 x  1 2 
..
.

(5.5)

The dimension of the features φ(x) is on the order of d3 .1 This is a prohibitively long vector for computational purpose — when d = 1000, each
update requires at least computing and storing a 10003 = 109 dimensional
vector, which is 106 times slower than the update rule for for ordinary leastsquares updates (5.2).
It may appear at first that such d3 runtime per update and memory usage
are inevitable, because the vector θ itself is of dimension p ≈ d3 , and we mayneed to update every entry of θ and store it. However, we will introduce thekernel trick with which we will not need to store θ explicitly, and the runtime
can be significantly improved.
For simplicity, we assume the initialize the value θ = 0, and we focus
on the iterative update (5.3). The main observation is that at any time, θcan be represented as a linear combination of the vectors φ(x(1) ), . . . , φ(x(n) ).
Indeed,we can show this inductively as follows. At initialization, θ = 0 =Pn(i)i=1 0 · φ(x ). Assume at some point, θ can be represented asθ=nXβi φ(x(i) )

(5.6)

i=1
1
Here, for simplicity, we include all the monomials with repetitions (so that, e.g., x1 x2 x3
and x2 x3 x1 both appear in φ(x)). Therefore, there are totally 1 + d + d2 + d3 entries in
φ(x).

52
for some β1 , . . . , βn ∈ R. Then we claim that in the next round, θ is still alinear combination of φ(x(1) ), . . . , φ(x(n) ) becauseθ := θ + αnXy (i) − θT φ(x(i) ) φ(x(i) )

i=1

=

n
X

(i)

βi φ(x ) + α

i=1

=

n
X


y (i) − θT φ(x(i) ) φ(x(i) )

i=1

n
X


(βi + α y (i) − θT φ(x(i) ) ) φ(x(i) )
{z
}
|
i=1

(5.7)

new βi

You may realize that our general strategy is to implicitly represent the pdimensional vector θ by a set of coefficients β1 , . . . , βn . Towards doing this,we derive the update rule of the coefficients β1 , . . . , βn . Using the equation
above, we see that the new βi depends on the old one via
βi := βi + α y (i) − θT φ(x(i) )(5.8)
HerePwe still have the old θ on the RHS of the equation. Replacing θ by
θ = nj=1 βj φ(x(j) ) gives∀i ∈ {1, . . . , n}, βi := βi + α y (i) −

n
X

(j) T

!

βj φ(x ) φ(x(i) )j=1
TWe often rewrite φ(x(j) ) φ(x(i) ) as hφ(x(j) ), φ(x(i) )i to emphasize that it’s theinner product of the two feature vectors. Viewing βi ’s as the new representation of θ, we have successfully translated the batch gradient descent algorithm
into an algorithm that updates the value of β iteratively. It may appear thatat every iteration, we still need to compute the values of hφ(x(j) ), φ(x(i) )i forall pairs of i, j, each of which may take roughly O(p) operation. However,two important properties come to rescue:
1. We can pre-compute the pairwise inner products hφ(x(j) ), φ(x(i) )i for all
pairs of i, j before the loop starts.
2. For the feature map φ defined in (5.5) (or many other interesting feature maps), computing hφ(x(j) ), φ(x(i) )i can be efficient and does not53necessarily require computing φ(x(i) ) explicitly. This is because:hφ(x), φ(z)i = 1 +dXxi zi +

i=1

=1+

xi zi +dXX

xi xj zi zj +i,j∈{1,...,d}

d
X
i=1

X

xi xj xk zi zj zk

i,j,k∈{1,...,d}

!2
xi zi

d
X

+

i=1
2

!3
xi zi

i=1

= 1 + hx, zi + hx, zi + hx, zi3(5.9)

Therefore, to compute hφ(x), φ(z)i, we can first compute hx, zi with
O(d) time and then take another constant number of operations to compute 1 + hx, zi + hx, zi2 + hx, zi3 .
As you will see, the inner products between the features hφ(x), φ(z)i areessential here. We define the Kernel corresponding to the feature map φ as
a function that maps X × X → R satisfying: 2
K(x, z) , hφ(x), φ(z)i(5.10)

To wrap up the discussion, we write the down the final algorithm asfollows:
1. Compute all the values K(x(i) , x(j) ) , hφ(x(i) ), φ(x(j) )i using equation (5.9) for all i, j ∈ {1, . . . , n}. Set β := 0.
2. Loop:∀i ∈ {1, . . . , n}, βi := βi + α y

(i)

−

n
X

!
(i)

(j)

βj K(x , x )(5.11)

j=1

Or in vector notation, letting K be the n × n matrix with Kij =K(x , x(j) ), we have(i)β := β + α(~y − Kβ)

With the algorithm above, we can update the representation β of thevector θ efficiently with O(n) time per update. Finally, we need to show that
2

Recall that X is the space of the input x. In our running example, X = Rd

54
the knowledge of the representation β suffices to compute the prediction
θT φ(x). Indeed, we haveθT φ(x) =nXTβi φ(x(i) ) φ(x) =

i=1

n
X

βi K(x(i) , x)(5.12)

i=1

You may realize that fundamentally all we need to know about the feature
map φ(·) is encapsulated in the corresponding kernel function K(·, ·). We
will expand on this in the next section.

5.4

Properties of kernels

In the last subsection, we started with an explicitly defined feature map φ,which induces the kernel function K(x, z) , hφ(x), φ(z)i. Then we saw thatthe kernel function is so intrinsic so that as long as the kernel function isdefined, the whole training algorithm can be written entirely in the languageof the kernel without referring to the feature map φ, so can the prediction ofa test example x (equation (5.12).)
Therefore, it would be tempted to define other kernel function K(·, ·) andrun the algorithm (5.11). Note that the algorithm (5.11) does not need to
explicitly access the feature map φ, and therefore we only need to ensure theexistence of the feature map φ, but do not necessarily need to be able toexplicitly write φ down.
What kinds of functions K(·, ·) can correspond to some feature map φ? Inother words, can we tell if there is some feature mapping φ so that K(x, z) =φ(x)T φ(z) for all x, z?
If we can answer this question by giving a precise characterization of valid
kernel functions, then we can completely change the interface of selecting
feature maps φ to the interface of selecting kernel function K. Concretely,
we can pick a function K, verify that it satisfies the characterization (sothat there exists a feature map φ that K corresponds to), and then we canrun update rule (5.11). The benefit here is that we don’t have to be able
to compute φ or write it down analytically, and we only need to know its
existence. We will answer this question at the end of this subsection after
we go through several concrete examples of kernels.
Suppose x, z ∈ Rd , and let’s first consider the function K(·, ·) defined as:
K(x, z) = (xT z)2 .

55
We can also write this as
K(x, z) =

d
X

!
xi zi

i=1

=

d X
d
X

d
X

!
xj zj

j=1

xi xj zi zj

i=1 j=1=

dX(xi xj )(zi zj )i,j=1Thus, we see that K(x, z) = hφ(x), φ(z)i is the kernel function that corresponds to the the feature mapping φ given (shown here for the case of d = 3)
by


x1 x1
 x1 x2  x1 x3  x2 x1 
φ(x) =  x2 x2  .
 x2 x3  x3 x1  x3 x2 x3 x3
Revisiting the computational efficiency perspective of kernel, note that whereas
calculating the high-dimensional φ(x) requires O(d2 ) time, finding K(x, z)
takes only O(d) time—linear in the dimension of the input attributes.
For another related example, also consider K(·, ·) defined byK(x, z) = (xT z + c)2
d
d
X
X
√
√
=
(xi xj )(zi zj ) +( 2cxi )( 2czi ) + c2 .
i,j=1

i=1

(Check this yourself.) This function K is a kernel function that corresponds

56
to the feature mapping (again shown for d = 3)


x1 x1
 x1 x2  x1 x3  x2 x1  x2 x2  x2 x3 
φ(x) =  x3 x1  , x3 x2  x3 x3  √
 2cx1  √
 2cx2  √
 2cx3 cand the parameter c controls the relative weighting between the xi (firstorder) and the xi xj (second order) terms.
More broadly, thekernel K(x, z) = (xT z + c)k corresponds to a feature
mapping to an d+kfeature space, corresponding of all monomials of thekform xi1 xi2 . . . xik that are up to order k. However, despite working in thisO(dk )-dimensional space, computing K(x, z) still takes only O(d) time, andhence we never need to explicitly represent feature vectors in this very highdimensional feature space.
Kernels as similarity metrics. Now, let’s talk about a slightly different
view of kernels. Intuitively, (and there are things wrong with this intuition,
but nevermind), if φ(x) and φ(z) are close together, then we might expect
K(x, z) = φ(x)T φ(z) to be large. Conversely, if φ(x) and φ(z) are far apart—say nearly orthogonal to each other—then K(x, z) = φ(x)T φ(z) will be small.
So, we can think of K(x, z) as some measurement of how similar are φ(x)and φ(z), or of how similar are x and z.
Given this intuition, suppose that for some learning problem that you’re
working on, you’ve come up with some function K(x, z) that you think might
be a reasonable measure of how similar x and z are. For instance, perhaps
you chose||x − z||2.
K(x, z) = exp −2σ 2This is a reasonable measure of x and z’s similarity, and is close to 1 whenx and z are close, and near 0 when x and z are far apart. Does there exist

57
a feature map φ such that the kernel K defined above satisfies K(x, z) =φ(x)T φ(z)? In this particular example, the answer is yes. This kernel is called
the Gaussian kernel, and corresponds to an infinite dimensional feature
mapping φ. We will give a precise characterization about what properties
a function K needs to satisfy so that it can be a valid kernel function that
corresponds to some feature map φ.
Necessary conditions for valid kernels. Suppose for now that K isindeed a valid kernel corresponding to some feature mapping φ, and we will
first see what properties it satisfies. Now, consider some finite set of n points(not necessarily the training set) {x(1) , . . . , x(n) }, and let a square, n-by-nmatrix K be defined so that its (i, j)-entry is given by Kij = K(x(i) , x(j) ).
This matrix is called the kernel matrix. Note that we’ve overloaded thenotation and used K to denote both the kernel function K(x, z) and thekernel matrix K, due to their obvious close relationship.
Now, if K is a valid kernel, then Kij = K(x(i) , x(j) ) = φ(x(i) )T φ(x(j) ) =φ(x(j) )T φ(x(i) ) = K(x(j) , x(i) ) = Kji , and hence K must be symmetric. Moreover, letting φk (x) denote the k-th coordinate of the vector φ(x), we find thatfor any vector z, we haveXX
z T Kz =zi Kij zj
i

j

=

XX

=

XX

=

XXX

i

i

k

zi φ(x(i) )T φ(x(j) )zj

j

X

zi

j

φk (x(i) )φk (x(j) )zjki

zi φk (x(i) )φk (x(j) )zj

j

!2
=

X X
k

zi φk (x(i) )

i

≥ 0.
P
P
2
The second-to-last step uses the fact thati,j ai aj = (i ai ) for ai =zi φk (x(i) ). Since z was arbitrary, this shows that K is positive semi-definite(K ≥ 0).
Hence, we’ve shown that if K is a valid kernel (i.e., if it corresponds tosome feature mapping φ), then the corresponding kernel matrix K ∈ Rn×n
is symmetric positive semidefinite.

58
Sufficient conditions for valid kernels. More generally, the condition
above turns out to be not only a necessary, but also a sufficient, condition
for K to be a valid kernel (also called a Mercer kernel). The following result
is due to Mercer.3
Theorem (Mercer). Let K : Rd × Rd 7→ R be given. Then for K
to be a valid (Mercer) kernel, it is necessary and sufficient that for any{x(1) , . . . , x(n) }, (n < ∞), the corresponding kernel matrix is symmetric positive semi-definite.
Given a function K, apart from trying to find a feature mapping φ that
corresponds to it, this theorem therefore gives another way of testing if it isa valid kernel. You’ll also have a chance to play with these ideas more in
problem set 2.
In class, we also briefly talked about a couple of other examples of kernels. For instance, consider the digit recognition problem, in which given
an image (16x16 pixels) of a handwritten digit (0-9), we have to figure out
which digit it was. Using either a simple polynomial kernel K(x, z) = (xT z)kor the Gaussian kernel, SVMs were able to obtain extremely good performance on this problem. This was particularly surprising since the input
attributes x were just 256-dimensional vectors of the image pixel intensity
values, and the system had no prior knowledge about vision, or even about
which pixels are adjacent to which other ones. Another example that webriefly talked about in lecture was that if the objects x that we are trying
to classify are strings (say, x is a list of amino acids, which strung together
form a protein), then it seems hard to construct a reasonable, “small” set offeatures for most learning algorithms, especially if different strings have different lengths. However, consider letting φ(x) be a feature vector that counts
the number of occurrences of each length-k substring in x. If we’re considering strings of English letters, then there are 26k such strings. Hence, φ(x)is a 26k dimensional vector; even for moderate values of k, this is probably
too big for us to efficiently work with. (e.g., 264 ≈ 460000.) However, using(dynamic programming-ish) string matching algorithms, it is possible to efficiently compute K(x, z) = φ(x)T φ(z), so that we can now implicitly workin this 26k -dimensional feature space, but without ever explicitly computing
feature vectors in this space.
3
Many texts present Mercer’s theorem in a slightly more complicated form involving
L2 functions, but when the input attributes take values in Rd , the version given here isequivalent.

59
Application of kernel methods: We’ve seen the application of kernels
to linear regression. In the next part, we will introduce the support vector
machines to which kernels can be directly applied. dwell too much longer on
it here. In fact, the idea of kernels has significantly broader applicability thanlinear regression and SVMs. Specifically, if you have any learning algorithm
that you can write in terms of only inner products hx, zi between input
attribute vectors, then by replacing this with K(x, z) where K is a kernel,
you can “magically” allow your algorithm to work efficiently in the highdimensional feature space corresponding to K. For instance, this kernel trick
can be applied with the perceptron to derive a kernel perceptron algorithm.
Many of the algorithms that we’ll see later in this class will also be amenable
to this method, which has come to be known as the “kernel trick.”Chapter 6
Support vector machines
This set of notes presents the Support Vector Machine (SVM) learning algorithm. SVMs are among the best (and many believe are indeed the best)“off-the-shelf” supervised learning algorithms. To tell the SVM story, we’ll
need to first talk about margins and the idea of separating data with a large“gap.” Next, we’ll talk about the optimal margin classifier, which will lead
us into a digression on Lagrange duality. We’ll also see kernels, which givea way to apply SVMs efficiently in very high dimensional (such as infinitedimensional) feature spaces, and finally, we’ll close off the story with the
SMO algorithm, which gives an efficient implementation of SVMs.

6.1

Margins: intuition

We’ll start our story on SVMs by talking about margins. This section will
give the intuitions about margins and about the “confidence” of our predictions; these ideas will be made formal in Section 6.3.
Consider logistic regression, where the probability p(y = 1|x; θ) is modeled by hθ (x) = g(θT x). We then predict “1” on an input x if and only ifhθ (x) ≥ 0.5, or equivalently, if and only if θT x ≥ 0. Consider a positive
training example (y = 1). The larger θT x is, the larger also is hθ (x) = p(y =1|x; θ), and thus also the higher our degree of “confidence” that the label is 1.
Thus, informally we can think of our prediction as being very confident thaty = 1 if θT x  0. Similarly, we think of logistic regression as confidently
predicting y = 0, if θT x  0. Given a training set, again informally it seems
that we’d have found a good fit to the training data if we can find θ so that
θT x(i)  0 whenever y (i) = 1, and θT x(i)  0 whenever y (i) = 0, since thiswould reflect a very confident (and correct) set of classifications for all the60

61
training examples. This seems to be a nice goal to aim for, and we’ll soon
formalize this idea using the notion of functional margins.
For a different type of intuition, consider the following figure, in which x’s
represent positive training examples, o’s denote negative training examples,a decision boundary (this is the line given by the equation θT x = 0, andis also called the separating hyperplane) is also shown, and three points
have also been labeled A, B and C.

1
A0
1
C0

1
B0

Notice that the point A is very far from the decision boundary. If we areasked to make a prediction for the value of y at A, it seems we should bequite confident that y = 1 there. Conversely, the point C is very close to
the decision boundary, and while it’s on the side of the decision boundaryon which we would predict y = 1, it seems likely that just a small change tothe decision boundary could easily have caused out prediction to be y = 0.
Hence, we’re much more confident about our prediction at A than at C. The
point B lies in-between these two cases, and more broadly, we see that if
a point is far from the separating hyperplane, then we may be significantly
more confident in our predictions. Again, informally we think it would benice if, given a training set, we manage to find a decision boundary thatallows us to make all correct and confident (meaning far from the decisionboundary) predictions on the training examples. We’ll formalize this later
using the notion of geometric margins.

62

6.2

Notation (option reading)To make our discussion of SVMs easier, we’ll first need to introduce a newnotation for talking about classification. We will be considering a linear
classifier for a binary classification problem with labels y and features x.
From now, we’ll use y ∈ {−1, 1} (instead of {0, 1}) to denote the class labels.
Also, rather than parameterizing our linear classifier with the vector θ, wewill use parameters w, b, and write our classifier as
hw,b (x) = g(wT x + b).
Here, g(z) = 1 if z ≥ 0, and g(z) = −1 otherwise. This “w, b” notation
allows us to explicitly treat the intercept term b separately from the otherparameters. (We also drop the convention we had previously of letting x0 = 1
be an extra coordinate in the input feature vector.) Thus, b takes the role ofwhat was previously θ0 , and w takes the role of [θ1 . . . θd ]T .
Note also that, from our definition of g above, our classifier will directlypredict either 1 or −1 (cf. the perceptron algorithm), without first going
through the intermediate step of estimating p(y = 1) (which is what logistic
regression does).

6.3

Functional and geometric margins (option reading)Let’s formalize the notions of the functional and geometric margins. Given a
training example (x(i) , y (i) ), we define the functional margin of (w, b) withrespect to the training example asγ̂ (i) = y (i) (wT x(i) + b).
Note that if y (i) = 1, then for the functional margin to be large (i.e., forour prediction to be confident and correct), we need wT x(i) + b to be a large
positive number. Conversely, if y (i) = −1, then for the functional margin
to be large, we need wT x(i) + b to be a large negative number. Moreover, if
y (i) (wT x(i) + b) > 0, then our prediction on this example is correct. (Check
this yourself.) Hence, a large functional margin represents a confident and acorrect prediction.
For a linear classifier with the choice of g given above (taking values in{−1, 1}), there’s one property of the functional margin that makes it not avery good measure of confidence, however. Given our choice of g, we note that

63
if we replace w with 2w and b with 2b, then since g(wT x + b) = g(2wT x + 2b),
this would not change hw,b (x) at all. I.e., g, and hence also hw,b (x), depends
only on the sign, but not on the magnitude, of wT x + b. However, replacing(w, b) with (2w, 2b) also results in multiplying our functional margin by afactor of 2. Thus, it seems that by exploiting our freedom to scale w and b,
we can make the functional margin arbitrarily large without really changing
anything meaningful. Intuitively, it might therefore make sense to impose
some sort of normalization condition such as that ||w||2 = 1; i.e., we might
replace (w, b) with (w/||w||2 , b/||w||2 ), and instead consider the functional
margin of (w/||w||2 , b/||w||2 ). We’ll come back to this later.
Given a training set S = {(x(i) , y (i) ); i = 1, . . . , n}, we also define thefunction margin of (w, b) with respect to S as the smallest of the functional
margins of the individual training examples. Denoted by γ̂, this can therefore
be written:γ̂ = min γ̂ (i) .
i=1,...,nNext, let’s talk about geometric margins. Consider the picture below:

A

w

γ (i)
B

The decision boundary corresponding to (w, b) is shown, along with thevector w. Note that w is orthogonal (at 90◦ ) to the separating hyperplane.
(You should convince yourself that this must be the case.) Consider the
point at A, which represents the input x(i) of some training example withlabel y (i) = 1. Its distance to the decision boundary, γ (i) , is given by the line
segment AB.
How can we find the value of γ (i) ? Well, w/||w|| is a unit-length vector
pointing in the same direction as w. Since A represents x(i) , we therefore

64find that the point B is given by x(i) − γ (i) · w/||w||. But this point lies on
the decision boundary, and all points x on the decision boundary satisfy theequation wT x + b = 0. Hence,


(i)
(i) w
T
w x −γ+ b = 0.
||w||
Solving for γ (i) yieldsγ(i)wT x(i) + b
=
=
||w||



w
||w||

T

x(i) +

b.
||w||

This was worked out for the case of a positive training example at A in the
figure, where being on the “positive” side of the decision boundary is good.
More generally, we define the geometric margin of (w, b) with respect to a
training example (x(i) , y (i) ) to be
!

T
w
b
(i)
(i)
(i)
γ =y
.
x +
||w||
||w||
Note that if ||w|| = 1, then the functional margin equals the geometricmargin—this thus gives us a way of relating these two different notions ofmargin. Also, the geometric margin is invariant to rescaling of the parameters; i.e., if we replace w with 2w and b with 2b, then the geometric margindoes not change. This will in fact come in handy later. Specifically, becauseof this invariance to the scaling of the parameters, when trying to fit w and b
to training data, we can impose an arbitrary scaling constraint on w withoutchanging anything important; for instance, we can demand that ||w|| = 1, or|w1 | = 5, or |w1 + b| + |w2 | = 2, and any of these can be satisfied simply byrescaling w and b.
Finally, given a training set S = {(x(i) , y (i) ); i = 1, . . . , n}, we also define
the geometric margin of (w, b) with respect to S to be the smallest of thegeometric margins on the individual training examples:γ = min γ (i) .
i=1,...,n6.4

The optimal margin classifier (option reading)Given a training set, it seems from our previous discussion that a naturaldesideratum is to try to find a decision boundary that maximizes the (geometric) margin, since this would reflect a very confident set of predictions

65
on the training set and a good “fit” to the training data. Specifically, thiswill result in a classifier that separates the positive and the negative training
examples with a “gap” (geometric margin).
For now, we will assume that we are given a training set that is linearly
separable; i.e., that it is possible to separate the positive and negative examples using some separating hyperplane. How will we find the one that
achieves the maximum geometric margin? We can pose the following optimization problem:
maxγ,w,b γ
s.t. y (i) (wT x(i) + b) ≥ γ, i = 1, . . . , n||w|| = 1.
I.e., we want to maximize γ, subject to each training example having functional margin at least γ. The ||w|| = 1 constraint moreover ensures that thefunctional margin equals to the geometric margin, so we are also guaranteedthat all the geometric margins are at least γ. Thus, solving this problem will
result in (w, b) with the largest possible geometric margin with respect to thetraining set.
If we could solve the optimization problem above, we’d be done. But the“||w|| = 1” constraint is a nasty (non-convex) one, and this problem certainly
isn’t in any format that we can plug into standard optimization software tosolve. So, let’s try transforming the problem into a nicer one. Consider:γ̂||w||s.t. y (i) (wT x(i) + b) ≥ γ̂, i = 1, . . . , nmaxγ̂,w,b

Here, we’re going to maximize γ̂/||w||, subject to the functional margins all
being at least γ̂. Since the geometric and functional margins are related byγ = γ̂/||w|, this will give us the answer we want. Moreover, we’ve gotten ridof the constraint ||w|| = 1 that we didn’t like. The downside is that we nowγ̂have a nasty (again, non-convex) objective ||w||function; and, we still don’t
have any off-the-shelf software that can solve this form of an optimization
problem.
Let’s keep going. Recall our earlier discussion that we can add an arbitrary scaling constraint on w and b without changing anything. This is the
key idea we’ll use now. We will introduce the scaling constraint that thefunctional margin of w, b with respect to the training set must be 1:γ̂ = 1.

66
Since multiplying w and b by some constant results in the functional margin
being multiplied by that same constant, this is indeed a scaling constraint,
and can be satisfied by rescaling w, b. Plugging this into our problem above,and noting that maximizing γ̂/||w|| = 1/||w|| is the same thing as minimizing||w||2 , we now have the following optimization problem:
1||w||2
2
s.t. y (i) (wT x(i) + b) ≥ 1, i = 1, . . . , nminw,bWe’ve now transformed the problem into a form that can be efficiently
solved. The above is an optimization problem with a convex quadratic objective and only linear constraints. Its solution gives us the optimal margin classifier. This optimization problem can be solved using commercial
quadratic programming (QP) code.1
While we could call the problem solved here, what we will instead do ismake a digression to talk about Lagrange duality. This will lead us to our
optimization problem’s dual form, which will play a key role in allowing us to
use kernels to get optimal margin classifiers to work efficiently in very highdimensional spaces. The dual form will also allow us to derive an efficientalgorithm for solving the above optimization problem that will typically domuch better than generic QP software.

6.5

Lagrange duality (optional reading)Let’s temporarily put aside SVMs and maximum margin classifiers, and talkabout solving constrained optimization problems.
Consider a problem of the following form:
minw f (w)s.t. hi (w) = 0, i = 1, . . . , l.
Some of you may recall how the method of Lagrange multipliers can be used
to solve it. (Don’t worry if you haven’t seen it before.) In this method, we
define the Lagrangian to beL(w, β) = f (w) +lXβi hi (w)i=1
1

You may be familiar with linear programming, which solves optimization problems
that have linear objectives and linear constraints. QP software is also widely available,
which allows convex quadratic objectives and linear constraints.

67
Here, the βi ’s are called the Lagrange multipliers. We would then findand set L’s partial derivatives to zero:
∂L
∂L
= 0;
= 0,
∂wi
∂βi
and solve for w and β.
In this section, we will generalize this to constrained optimization problems in which we may have inequality as well as equality constraints. Due totime constraints, we won’t really be able to do the theory of Lagrange duality
justice in this class,2 but we will give the main ideas and results, which wewill then apply to our optimal margin classifier’s optimization problem.
Consider the following, which we’ll call the primal optimization problem:
minw f (w)s.t. gi (w) ≤ 0, i = 1, . . . , khi (w) = 0, i = 1, . . . , l.
To solve it, we start by defining the generalized LagrangianL(w, α, β) = f (w) +kX

αi gi (w) +

i=1

l
X

βi hi (w).

i=1Here, the αi ’s and βi ’s are the Lagrange multipliers. Consider the quantityθP (w) = max L(w, α, β).
α,β : αi ≥0Here, the “P” subscript stands for “primal.” Let some w be given. If w
violates any of the primal constraints (i.e., if either gi (w) > 0 or hi (w) 6= 0
for some i), then you should be able to verify that
θP (w) =max f (w) +α,β : αi ≥0

= ∞.

k
X
i=1

αi gi (w) +lXβi hi (w)

(6.1)

i=1

(6.2)

Conversely, if the constraints are indeed satisfied for a particular value of w,
then θP (w) = f (w). Hence,f (w) if w satisfies primal constraintsθP (w) =∞otherwise.
2

Readers interested in learning more about this topic are encouraged to read, e.g., R.
T. Rockarfeller (1970), Convex Analysis, Princeton University Press.

68
Thus, θP takes the same value as the objective in our problem for all values of w that satisfies the primal constraints, and is positive infinity if theconstraints are violated. Hence, if we consider the minimization problem
min θP (w) = min max L(w, α, β),ww

α,β : αi ≥0

we see that it is the same problem (i.e., and has the same solutions as) our
original, primal problem. For later use, we also define the optimal value ofthe objective to be p∗ = minw θP (w); we call this the value of the primal
problem.
Now, let’s look at a slightly different problem. We defineθD (α, β) = min L(w, α, β).
wHere, the “D” subscript stands for “dual.” Note also that whereas in thedefinition of θP we were optimizing (maximizing) with respect to α, β, here
we are minimizing with respect to w.
We can now pose the dual optimization problem:max θD (α, β) = max min L(w, α, β).

α,β : αi ≥0α,β : αi ≥0wThis is exactly the same as our primal problem shown above, except that theorder of the “max” and the “min” are now exchanged. We also define theoptimal value of the dual problem’s objective to be d∗ = maxα,β : αi ≥0 θD (w).
How are the primal and the dual problems related? It can easily be shownthatd∗ = max min L(w, α, β) ≤ min max L(w, α, β) = p∗ .
α,β : αi ≥0ww

α,β : αi ≥0(You should convince yourself of this; this follows from the “max min” of afunction always being less than or equal to the “min max.”) However, undercertain conditions, we will haved∗ = p∗ ,so that we can solve the dual problem in lieu of the primal problem. Let’s
see what these conditions are.
Suppose f and the gi ’s are convex,3 and the hi ’s are affine.4 Suppose
further that the constraints gi are (strictly) feasible; this means that there
exists some w so that gi (w) < 0 for all i.
3When f has a Hessian, then it is convex if and only if the Hessian is positive semidefinite. For instance, f (w) = wT w is convex; similarly, all linear (and affine) functions
are also convex. (A function f can also be convex without being differentiable, but wewon’t need those more general definitions of convexity here.)
4I.e., there exists ai , bi , so that hi (w) = aTi w + bi . “Affine” means the same thing as
linear, except that we also allow the extra intercept term bi .

69
Under our above assumptions, there must exist w∗ , α∗ , β ∗ so that w∗ is thesolution to the primal problem, α∗ , β ∗ are the solution to the dual problem,and moreover p∗ = d∗ = L(w∗ , α∗ , β ∗ ). Moreover, w∗ , α∗ and β ∗ satisfy the
Karush-Kuhn-Tucker (KKT) conditions, which are as follows:∂
L(w∗ , α∗ , β ∗ )∂wi∂L(w∗ , α∗ , β ∗ )∂βi
αi∗ gi (w∗ )gi (w∗ )α∗= 0, i = 1, . . . , d(6.3)= 0, i = 1, . . . , l(6.4)= 0, i = 1, . . . , k≤ 0, i = 1, . . . , k≥ 0, i = 1, . . . , k

(6.5)
(6.6)
(6.7)

he primal and dual

Moreover, if some w∗ , α∗ , β ∗ satisfy the KKT conditions, then it is also a solution to t

problems.
We draw attention to Equation (6.5), which is called the KKT dual
complementarity condition. Specifically, it implies that if αi∗ > 0, thengi (w∗ ) = 0. (I.e., the “gi (w) ≤ 0” constraint is active, meaning it holds withequality rather than with inequality.) Later on, this will be key for showingthat the SVM has only a small number of “support vectors”; the KKT dual
complementarity condition will also give us our convergence test when wetalk about the SMO algorithm.

6.6

Optimal margin classifiers: the dual form
(option reading)Note: The equivalence of optimization problem (6.8) and the optimization
problem (6.12), and the relationship between the primary and dual variables
in equation (6.10) are the most important take home messages of this section.

Previously, we posed the following (primal) optimization problem for finding the optimal margin classifier:1||w||2
2
s.t. y (i) (wT x(i) + b) ≥ 1, i = 1, . . . , nminw,b

We can write the constraints as
gi (w) = −y (i) (wT x(i) + b) + 1 ≤ 0.

(6.8)

70
We have one such constraint for each training example. Note that from theKKT dual complementarity condition, we will have αi > 0 only for the training examples that have functional margin exactly equal to one (i.e., the ones
corresponding to constraints that hold with equality, gi (w) = 0). Consider
the figure below, in which a maximum margin separating hyperplane is shownby the solid line.

The points with the smallest margins are exactly the ones closest to thedecision boundary; here, these are the three points (one negative and two positive examples) that lie on the dashed lines parallel to the decision boundary.
Thus, only three of the αi ’s—namely, the ones corresponding to these three
training examples—will be non-zero at the optimal solution to our optimization problem. These three points are called the support vectors in thisproblem. The fact that the number of support vectors can be much smaller
than the size the training set will be useful later.
Let’s move on. Looking ahead, as we develop the dual form of the problem, one key idea to watch out for is that we’ll try to write our algorithm
in terms of only the inner product hx(i) , x(j) i (think of this as (x(i) )T x(j) )between points in the input feature space. The fact that we can express ouralgorithm in terms of these inner products will be key when we apply the
kernel trick.
When we construct the Lagrangian for our optimization problem we have:
n
X


1
2
αi y (i) (wT x(i) + b) − 1 .
L(w, b, α) = ||w|| −
2
i=1

(6.9)

Note that there’re only “αi ” but no “βi ” Lagrange multipliers, since theproblem has only inequality constraints.

71
Let’s find the dual form of the problem. To do so, we need to firstminimize L(w, b, α) with respect to w and b (for fixed α), to get θD , which
we’ll do by setting the derivatives of L with respect to w and b to zero. We
have:
n
X
∇w L(w, b, α) = w −αi y (i) x(i) = 0
i=1This implies thatw=nX

αi y (i) x(i) .

(6.10)

i=1

As for the derivative with respect to b, we obtainnX∂
L(w, b, α) =αi y (i) = 0.
∂bi=1

(6.11)

If we take the definition of w in Equation (6.10) and plug that back into
the Lagrangian (Equation 6.9), and simplify, we getL(w, b, α) =

n
X

αi −

i=1

n
n
X
1 X (i) (j)y y αi αj (x(i) )T x(j) − b
αi y (i) .
2 i,j=1
i=1

But from Equation (6.11), the last term must be zero, so we obtainnXn

1 X (i) (j)y y αi αj (x(i) )T x(j) .
L(w, b, α) =
αi −
2
i,j=1
i=1
Recall that we got to the equation above by minimizing L with respect tow and b. Putting this together with the constraints αi ≥ 0 (that we always
had) and the constraint (6.11), we obtain the following dual optimization
problem:
maxα

n
X

n

1 X (i) (j)W (α) =αi −y y αi αj hx(i) , x(j) i.
2 i,j=1i=1(6.12)

s.t. αi ≥ 0, i = 1, . . . , nnXαi y (i) = 0,i=1

You should also be able to verify that the conditions required for p∗ = d∗and the KKT conditions (Equations 6.3–6.7) to hold are indeed satisfied in

72
our optimization problem. Hence, we can solve the dual in lieu of solving
the primal problem. Specifically, in the dual problem above, we have amaximization problem in which the parameters are the αi ’s. We’ll talk later
about the specific algorithm that we’re going to use to solve the dual problem,
but if we are indeed able to solve it (i.e., find the α’s that maximize W (α)
subject to the constraints), then we can use Equation (6.10) to go back andfind the optimal w’s as a function of the α’s. Having found w∗ , by considering
the primal problem, it is also straightforward to find the optimal value forthe intercept term b asb∗ = −

maxi:y(i) =−1 w∗ T x(i) + mini:y(i) =1 w∗ T x(i).
2(6.13)(Check for yourself that this is correct.)
Before moving on, let’s also take a more careful look at Equation (6.10),
which gives the optimal value of w in terms of (the optimal value of) α.
Suppose we’ve fit our model’s parameters to a training set, and now wish to
make a prediction at a new point input x. We would then calculate wT x + b,
and predict y = 1 if and only if this quantity is bigger than zero. But
using (6.10), this quantity can also be written:
wT x + b =nX!Tαi y (i) x(i)x+b(6.14)

i=1

=

n
X

αi y (i) hx(i) , xi + b.

(6.15)

i=1

Hence, if we’ve found the αi ’s, in order to make a prediction, we have tocalculate a quantity that depends only on the inner product between x andthe points in the training set. Moreover, we saw earlier that the αi ’s will all
be zero except for the support vectors. Thus, many of the terms in the sum
above will be zero, and we really need to find only the inner products between
x and the support vectors (of which there is often only a small number) inorder calculate (6.15) and make our prediction.
By examining the dual form of the optimization problem, we gained significant insight into the structure of the problem, and were also able to write
the entire algorithm in terms of only inner products between input feature
vectors. In the next section, we will exploit this property to apply the kernels to our classification problem. The resulting algorithm, support vector
machines, will be able to efficiently learn in very high dimensional spaces.

73

6.7

Regularization and the non-separable case
(optional reading)

The derivation of the SVM as presented so far assumed that the data islinearly separable. While mapping data to a high dimensional feature space
via φ does generally increase the likelihood that the data is separable, we
can’t guarantee that it always will be so. Also, in some cases it is not clearthat finding a separating hyperplane is exactly what we’d want to do, sincethat might be susceptible to outliers. For instance, the left figure below
shows an optimal margin classifier, and when a single outlier is added in theupper-left region (right figure), it causes the decision boundary to make adramatic swing, and the resulting classifier has a much smaller margin.

To make the algorithm work for non-linearly separable datasets as well
as be less sensitive to outliers, we reformulate our optimization (using `1
regularization) as follows:
minγ,w,bn
X
1||w||2 + C
ξi
2
i=1

s.t. y (i) (wT x(i) + b) ≥ 1 − ξi , i = 1, . . . , nξi ≥ 0, i = 1, . . . , n.
Thus, examples are now permitted to have (functional) margin less than 1,
and if an example has functional margin 1 − ξi (with ξ > 0), we would pay
a cost of the objective function being increased by Cξi . The parameter C
controls the relative weighting between the twin goals of making the ||w||2
small (which we saw earlier makes the margin large) and of ensuring thatmost examples have functional margin at least 1.

74
As before, we can form the Lagrangian:
n
n
n
X
X

 X
1
L(w, b, ξ, α, r) = wT w + C
ξi −αi y (i) (xT w + b) − 1 + ξi −ri ξi .
2
i=1
i=1
i=1

Here, the αi ’s and ri ’s are our Lagrange multipliers (constrained to be ≥ 0).
We won’t go through the derivation of the dual again in detail, but after
setting the derivatives with respect to w and b to zero as before, substituting
them back in, and simplifying, we obtain the following dual form of the
problem:
maxα

n
X

n

1 X (i) (j)y y αi αj hx(i) , x(j) i
W (α) =
αi −2 i,j=1i=1s.t. 0 ≤ αi ≤ C, i = 1, . . . , nnXαi y (i) = 0,i=1

As before, we also have that w can be expressed in terms of the αi ’s asgiven in Equation (6.10), so that after solving the dual problem, we can continue to use Equation (6.15) to make our predictions. Note that, somewhat
surprisingly, in adding `1 regularization, the only change to the dual problem is that what was originally a constraint that 0 ≤ αi has now become
0 ≤ αi ≤ C. The calculation for b∗ also has to be modified (Equation 6.13 is
no longer valid); see the comments in the next section/Platt’s paper.
Also, the KKT dual-complementarity conditions (which in the next section will be useful for testing for the convergence of the SMO algorithm)are:
αi = 0 ⇒ y (i) (wT x(i) + b) ≥ 1
αi = C ⇒ y (i) (wT x(i) + b) ≤ 1
0 < αi < C ⇒ y (i) (wT x(i) + b) = 1.

(6.16)(6.17)(6.18)

Now, all that remains is to give an algorithm for actually solving the dual
problem, which we will do in the next section.

6.8

The SMO algorithm (optional reading)

The SMO (sequential minimal optimization) algorithm, due to John Platt,gives an efficient way of solving the dual problem arising from the derivation

75
of the SVM. Partly to motivate the SMO algorithm, and partly because it’sinteresting in its own right, let’s first take another digression to talk about
the coordinate ascent algorithm.

6.8.1

Coordinate ascentConsider trying to solve the unconstrained optimization problem
max W (α1 , α2 , . . . , αn ).
αHere, we think of W as just some function of the parameters αi ’s, and for now
ignore any relationship between this problem and SVMs. We’ve already seen
two optimization algorithms, gradient ascent and Newton’s method. The
new algorithm we’re going to consider here is called coordinate ascent:
Loop until convergence: {For i = 1, . . . , n, {αi := arg maxα̂i W (α1 , . . . , αi−1 , α̂i , αi+1 , . . . , αn ).
}}Thus, in the innermost loop of this algorithm, we will hold all the variables
except for some αi fixed, and reoptimize W with respect to just the parameterαi . In the version of this method presented here, the inner-loop reoptimizes
the variables in order α1 , α2 , . . . , αn , α1 , α2 , . . .. (A more sophisticated version
might choose other orderings; for instance, we may choose the next variable
to update according to which one we expect to allow us to make the largest
increase in W (α).)
When the function W happens to be of such a form that the “arg max”
in the inner loop can be performed efficiently, then coordinate ascent can bea fairly efficient algorithm. Here’s a picture of coordinate ascent in action:

76
2.5

2

1.5

1

0.5

0

−0.5

−1

−1.5

−2

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

The ellipses in the figure are the contours of a quadratic function that
we want to optimize. Coordinate ascent was initialized at (2, −2), and alsoplotted in the figure is the path that it took on its way to the global maximum.
Notice that on each step, coordinate ascent takes a step that’s parallel to oneof the axes, since only one variable is being optimized at a time.

6.8.2

SMO

We close off the discussion of SVMs by sketching the derivation of the SMO
algorithm.
Here’s the (dual) optimization problem that we want to solve:
maxα

n

n
X

1 X (i) (j)y y αi αj hx(i) , x(j) i.
W (α) =
αi −
2
i,j=1
i=1

s.t. 0 ≤ αi ≤ C, i = 1, . . . , nnXαi y (i) = 0.

(6.19)(6.20)(6.21)

i=1

Let’s say we have set of αi ’s that satisfy the constraints (6.20-6.21). Now,
suppose we want to hold α2 , . . . , αn fixed, and take a coordinate ascent stepand reoptimize the objective with respect to α1 . Can we make any progress?
The answer is no, because the constraint (6.21) ensures that
α1 y

(1)

=−

n
X
i=2

αi y (i) .

77Or, by multiplying both sides by y (1) , we equivalently have
α1 = −y (1)nXαi y (i) .

i=2(This step used the fact that y (1) ∈ {−1, 1}, and hence (y (1) )2 = 1.) Hence,
α1 is exactly determined by the other αi ’s, and if we were to hold α2 , . . . , αnfixed, then we can’t make any change to α1 without violating the constraint (6.21) in the optimization problem.
Thus, if we want to update some subject of the αi ’s, we must update atleast two of them simultaneously in order to keep satisfying the constraints.
This motivates the SMO algorithm, which simply does the following:
Repeat till convergence {1. Select some pair αi and αj to update next (using a heuristic thattries to pick the two that will allow us to make the biggest progress
towards the global maximum).
2. Reoptimize W (α) with respect to αi and αj , while holding all theother αk ’s (k 6= i, j) fixed.
}
To test for convergence of this algorithm, we can check whether the KKT
conditions (Equations 6.16-6.18) are satisfied to within some tol. Here, tol isthe convergence tolerance parameter, and is typically set to around 0.01 to
0.001. (See the paper and pseudocode for details.)
The key reason that SMO is an efficient algorithm is that the update toαi , αj can be computed very efficiently. Let’s now briefly sketch the mainideas for deriving the efficient update.
Let’s say we currently have some setting of the αi ’s that satisfy the constraints (6.20-6.21), and suppose we’ve decided to hold α3 , . . . , αn fixed, andwant to reoptimize W (α1 , α2 , . . . , αn ) with respect to α1 and α2 (subject tothe constraints). From (6.21), we require that
α1 y

(1)

+ α2 y

(2)

=−

n
X

αi y (i) .

i=3Since the right hand side is fixed (as we’ve fixed α3 , . . . αn ), we can just let
it be denoted by some constant ζ:
α1 y (1) + α2 y (2) = ζ.
We can thus picture the constraints on α1 and α2 as follows:(6.22)78C

α1y(1)+ α2y(2)=ζ

H
α2

L
α1

C

From the constraints (6.20), we know that α1 and α2 must lie within the box[0, C] × [0, C] shown. Also plotted is the line α1 y (1) + α2 y (2) = ζ, on which weknow α1 and α2 must lie. Note also that, from these constraints, we know
L ≤ α2 ≤ H; otherwise, (α1 , α2 ) can’t simultaneously satisfy both the box
and the straight line constraint. In this example, L = 0. But depending on
what the line α1 y (1) + α2 y (2) = ζ looks like, this won’t always necessarily be
the case; but more generally, there will be some lower-bound L and someupper-bound H on the permissible values for α2 that will ensure that α1 , α2lie within the box [0, C] × [0, C].
Using Equation (6.22), we can also write α1 as a function of α2 :α1 = (ζ − α2 y (2) )y (1) .
(Check this derivation yourself; we again used the fact that y (1) ∈ {−1, 1} sothat (y (1) )2 = 1.) Hence, the objective W (α) can be written
W (α1 , α2 , . . . , αn ) = W ((ζ − α2 y (2) )y (1) , α2 , . . . , αn ).
Treating α3 , . . . , αn as constants, you should be able to verify that this isjust some quadratic function in α2 . I.e., this can also be expressed in theform aα22 + bα2 + c for some appropriate a, b, and c. If we ignore the “box”
constraints (6.20) (or, equivalently, that L ≤ α2 ≤ H), then we can easily
maximize this quadratic function by setting its derivative to zero and solving.
We’ll let α2new,unclipped denote the resulting value of α2 . You should also beable to convince yourself that if we had instead wanted to maximize W withrespect to α2 but subject to the box constraint, then we can find the resulting
value optimal simply by taking α2new,unclipped and “clipping” it to lie in the79[L, H] interval, to get H
α2new =
αnew,unclipped
 2
L

if α2new,unclipped > H
if L ≤ α2new,unclipped ≤ H
if α2new,unclipped < L

Finally, having found the α2new , we can use Equation (6.22) to go back andfind the optimal value of α1new .
There’re a couple more details that are quite easy but that we’ll leave you
to read about yourself in Platt’s paper: One is the choice of the heuristics
used to select the next αi , αj to update; the other is how to update b as the
SMO algorithm is run.

Part II
Deep learning

80

Chapter 7
Deep learning
We now begin our study of deep learning. In this set of notes, we give an
overview of neural networks, discuss vectorization and discuss training neuralnetworks with backpropagation.

7.1

Supervised learning with non-linear models

In the supervised learning setting (predicting y from the input x), suppose
our model/hypothesis is hθ (x). In the past lectures, we have considered thecases when hθ (x) = θ> x (in linear regression or logistic regression) or hθ (x) =θ> φ(x) (where φ(x) is the feature map). A commonality of these two models
is that they are linear in the parameters θ. Next we will consider learning
general family of models that are non-linear in both the parameters θand the inputs x. The most common non-linear models are neural networks,
which we will define staring from the next section. For this section, it suffices
to think hθ (x) as an abstract non-linear model.1
Suppose {(x(i) , y (i) )}ni=1 are the training examples. For simplicity, we startwith the case where y (i) ∈ R and hθ (x) ∈ R.
Cost/loss function. We define the least square cost function for the i-th
example (x(i) , y (i) ) as1
J (i) (θ) = (hθ (x(i) ) − y (i) )2
2

(7.1)

1
If a concrete example is helpful, perhaps think about the model hθ (x) = θ12 x21 + θ22 x22 +· · · + θd2 x2d in this subsection, even though it’s not a neural network.

81

82
and define the mean-square cost function for the dataset asn

1 X (i)J(θ) =
J (θ)
n i=1

(7.2)

which is same as in linear regression except that we introduce a constant
1/n in front of the cost function to be consistent with the convention. Notethat multiplying the cost function with a scalar will not change the local
minima or global minima of the cost function. Also note that the underlying
parameterization for hθ (x) is different from the case of linear regression,even though the form of the cost function is the same mean-squared loss.
Throughout the notes, we use the words “loss” and “cost” interchangeably.
Optimizers (SGD). Commonly, people use gradient descent (GD), stochastic gradient (SGD), or their variants to optimize the loss function J(θ). GD’s
update rule can be written as2
θ := θ − α∇θ J(θ)(7.3)

where α > 0 is often referred to as the learning rate or step size. Next, we
introduce a version of the SGD (Algorithm 1), which is lightly different fromthat in the first lecture notes.
Algorithm 1 Stochastic Gradient Descent
1: Hyperparameter: learning rate α, number of total iteration niter .
2: Initialize θ randomly.
3: for i = 1 to niter do
4:
Sample j uniformly from {1, . . . , n}, and update θ by
θ := θ − α∇θ J (j) (θ)(7.4)

Oftentimes computing the gradient of B examples simultaneously for theparameter θ can be faster than computing B gradients separately due tohardware parallelization. Therefore, a mini-batch version of SGD is most2

Recall that, as defined in the previous lecture notes, we use the notation “a := b” todenote an operation (in a computer program) in which we set the value of a variable a tobe equal to the value of b. In other words, this operation overwrites a with the value ofb. In contrast, we will write “a = b” when we are asserting a statement of fact, that thevalue of a is equal to the value of b.

83
commonly used in deep learning, as shown in Algorithm 2. There are also
other variants of the SGD or mini-batch SGD with slightly different sampling
schemes.
Algorithm 2 Mini-batch Stochastic Gradient Descent
1: Hyperparameters: learning rate α, batch size B, # iterations niter .
2: Initialize θ randomly
3: for i = 1 to niter do
4:
Sample B examples j1 , . . . , jB (without replacement) uniformly from
{1, . . . , n}, and update θ byB

αX
∇θ J (jk ) (θ)θ := θ −
B k=1

(7.5)

With these generic algorithms, a typical deep learning model is learned
with the following steps. 1. Define a neural network parametrization hθ (x),which we will introduce in Section 7.2, and 2. write the backpropagation
algorithm to compute the gradient of the loss function J (j) (θ) efficiently,
which will be covered in Section 7.3, and 3. run SGD or mini-batch SGD (orother gradient-based optimizers) with the loss function J(θ).

7.2

Neural networks

Neural networks refer to broad type of non-linear models/parametrizationshθ (x) that involve combinations of matrix multiplications and other entrywise non-linear operations. We will start small and slowly build up a neuralnetwork, step by step.
A Neural Network with a Single Neuron.
Recall the housing priceprediction problem from before: given the size of the house, we want topredict the price. We will use it as a running example in this subsection.
Previously, we fit a straight line to the graph of size vs. housing price.
Now, instead of fitting a straight line, we wish to prevent negative housing
prices by setting the absolute minimum price as zero. This produces a “kink”
in the graph as shown in Figure 7.1. How do we represent such a function
with a single kink as hθ (x) with unknown parameter? (After doing so, we
can invoke the machinery in Section 7.1.)

84
We define a parameterized function hθ (x) with input x, parameterized by
θ, which outputs the price of the house y. Formally, hθ : x → y. Perhaps
one of the simplest parametrization would behθ (x) = max(wx + b, 0), where θ = (w, b) ∈ R2

(7.6)

Here hθ (x) returns a single value: (wx+b) or zero, whichever is greater. In
the context of neural networks, the function max{t, 0} is called a ReLU (pronounced “ray-lu”), or rectified linear unit, and often denoted by ReLU(t) ,max{t, 0}.
Generally, a one-dimensional non-linear function that maps R to R such as
ReLU is often referred to as an activation function. The model hθ (x) is said
to have a single neuron partly because it has a single non-linear activation
function. (We will discuss more about why a non-linear activation is calledneuron.)When the input x ∈ Rd has multiple dimensions, a neural network witha single neuron can be written ashθ (x) = ReLU(w> x + b), where w ∈ Rd , b ∈ R, and θ = (w, b)(7.7)

The term b is often referred to as the “bias”, and the vector w is referred
to as the weight vector. Such a neural network has 1 layer. (We will define
what multiple layers mean in the sequel.)
Stacking Neurons. A more complex neural network may take the single
neuron described above and “stack” them together such that one neuron
passes its output as input into the next neuron, resulting in a more complex
function.
Let us now deepen the housing prediction example. In addition to the sizeof the house, suppose that you know the number of bedrooms, the zip codeand the wealth of the neighborhood. Building neural networks is analogous
to Lego bricks: you take individual bricks and stack them together to build
complex structures. The same applies to neural networks: we take individual
neurons and stack them together to create complex neural networks.
Given these features (size, number of bedrooms, zip code, and wealth),
we might then decide that the price of the house depends on the maximum
family size it can accommodate. Suppose the family size is a function of thesize of the house and number of bedrooms (see Figure 7.2). The zip code
may provide additional information such as how walkable the neighborhoodis (i.e., can you walk to the grocery store or do you need to drive everywhere).
Combining the zip code with the wealth of the neighborhood may predict85
housing prices
1000
900
800

price (in $1000)

700
600
500
400
300
200
100
0
500

1000

1500

2000

2500

3000

3500

4000

4500

5000

square feet

Figure 7.1: Housing prices with a “kink” in the graph.
the quality of the local elementary school. Given these three derived features(family size, walkable, school quality), we may conclude that the price of thehome ultimately depends on these three features.
Size# Bedrooms

Fam
ily S
ize

Walkable
Zip Code
ol
Scho

Price
y

ity
Qual

WealthFigure 7.2: Diagram of a small neural network for predicting housing prices.
Formally, the input to a neural network is a set of input features
x1 , x2 , x3 , x4 . We denote the intermediate variables for “family size”, “walkable”, and “school quality” by a1 , a2 , a3 (these ai ’s are often referred to as“hidden units” or “hidden neurons”). We represent each of the ai ’s as a neural network with a single neuron with a subset of x1 , . . . , x4 as inputs. Then
as in Figure 7.1, we will have the parameterization:
a1 = ReLU(θ1 x1 + θ2 x2 + θ3 )a2 = ReLU(θ4 x3 + θ5 )a3 = ReLU(θ6 x3 + θ7 x4 + θ8 )

86
where (θ1 , · · · , θ8 ) are parameters. Now we represent the final output hθ (x)as another linear function with a1 , a2 , a3 as inputs, and we get3hθ (x) = θ9 a1 + θ10 a2 + θ11 a3 + θ12(7.8)

where θ contains all the parameters (θ1 , · · · , θ12 ).
Now we represent the output as a quite complex function of x with parameters θ. Then you can use this parametrization hθ with the machinery ofSection 7.1 to learn the parameters θ.
Inspiration from Biological Neural Networks. As the name suggests,
artificial neural networks were inspired by biological neural networks. The
hidden units a1 , . . . , am correspond to the neurons in a biological neural network, and the parameters θi ’s correspond to the synapses. However, it’sunclear how similar the modern deep artificial neural networks are to the biological ones. For example, perhaps not many neuroscientists think biologicalneural networks could have 1000 layers, while some modern artificial neuralnetworks do (we will elaborate more on the notion of layers.) Moreover, it’san open question whether human brains update their neural networks in away similar to the way that computer scientists learn artificial neural networks (using backpropagation, which we will introduce in the next section.).
Two-layer Fully-Connected Neural Networks. We constructed theneural network in equation (7.8) using a significant amount of prior knowledge/belief about how the “family size”, “walkable”, and “school quality” aredetermined by the inputs. We implicitly assumed that we know the family
size is an important quantity to look at and that it can be determined byonly the “size” and “# bedrooms”. Such a prior knowledge might not be
available for other applications. It would be more flexible and general to havea generic parameterization. A simple way would be to write the intermediatevariable a1 as a function of all x1 , . . . , x4 :a1 = ReLU(w1> x + b1 ), where w1 ∈ R4 and b1 ∈ R(7.9)

a2 = ReLU(w2> x + b2 ), where w2 ∈ R4 and b2 ∈ R
a3 = ReLU(w3> x + b3 ), where w3 ∈ R4 and b3 ∈ R
We still define hθ (x) using equation (7.8) with a1 , a2 , a3 being defined
as above. Thus we have a so-called fully-connected neural network as
3

Typically, for multi-layer neural network, at the end, near the output, we don’t apply
ReLU, especially when the output is not necessarily a positive number.

87

Figure 7.3: Diagram of a two-layer fully connected neural network. Eachedge from node xi to node aj indicates that aj depends on xi . The edge from[1]xi to aj is associated with the weight (wj )i which denotes the i-th coordinate[1]of the vector wj . The activation aj can be computed by taking the ReLUof
the weighted sum of xi ’s with the weights
the weights associated with
Pbeing [1]the incoming edges, that is, aj = ReLU( di=1 (wj )i xi ).

visualized in the dependency graph in Figure 7.3 because all the intermediatevariables ai ’s depend on all the inputs xi ’s.
For full generality, a two-layer fully-connected neural network with mhidden units and d dimensional input x ∈ Rd is defined as∀j ∈ [1, ..., m],[1] >zj = wj

[1]

[1]

[1]

x + bj where wj ∈ Rd , bj ∈ R(7.10)

aj = ReLU(zj ),a = [a1 , . . . , am ]> ∈ Rm>hθ (x) = w[2] a + b[2] where w[2] ∈ Rm , b[2] ∈ R,

(7.11)

Note that by default the vectors in Rd are viewed as column vectors, andin particular a is a column vector with components a1 , a2 , ..., am . The indices[1][1]and [2] are used to distinguish two sets of parameters: the wj ’s (each ofwhich is a vector in Rd ) and w[2] (which is a vector in Rm ). We will havemore of these later.
Vectorization. Before we introduce neural networks with more layers andmore complex structures, we will simplify the expressions for neural networks
with more matrix and vector notations. Another important motivation of

88
vectorization is the speed perspective in the implementation. In order toimplement a neural network efficiently, one must be careful when using forloops. The most natural way to implement equation (7.10) in code is perhaps
to use a for loop. In practice, the dimensionalities of the inputs and hidden
units are high. As a result, code will run very slowly if you use for loops.
Leveraging the parallelism in GPUs is/was crucial for the progress of deeplearning.
This gave rise to vectorization. Instead of using for loops, vectorization
takes advantage of matrix algebra and highly optimized numerical linear
algebra packages (e.g., BLAS) to make neural network computations run
quickly. Before the deep learning era, a for loop may have been sufficienton smaller datasets, but modern deep networks and state-of-the-art datasets
will be infeasible to run with for loops.
We vectorize the two-layer fully-connected neural network as below. We
define a weight matrix W [1] in Rm×d as the concatenation of all the vectors[1]
wj ’s in the following way:

W

[1]

[1] >

— w1

—





>
 — w2[1] —  ∈ Rm×d

=
..

.


[1] >

— wm

(7.12)—

Now by the definition of matrix vector multiplication, we can write z =[z1 , . . . , zm ]> ∈ Rm as  [1] 


[1] >
z1
— w1 —
b1
x1
 

>
[1] 
[1]


 .. 
 — w 2 —   x2  b2 
 . 

(7.13)
 .  = ..  +  .. 
..
 
 . 
 .. 
.
  . 

[1]
xd
[1] >
zm
bm
—
w
—
m
|
{z
}
| {z }| {z }
|
{z
}
d×1
m×1
[1]
x
∈
R
z∈R
b ∈ Rm×1
W [1] ∈ Rm×d
Or succinctly,
z = W [1] x + b[1](7.14)We remark again that a vector in Rd in this notes, following the conventions
previously established, is automatically viewed as a column vector, and canalso be viewed as a d × 1 dimensional matrix. (Note that this is different
from numpy where a vector is viewed as a row vector in broadcasting.)

89
Computing the activations a ∈ Rm from z ∈ Rm involves an elementwise non-linear application of the ReLU function, which can be computed inparallel efficiently. Overloading ReLU for element-wise application of ReLU
(meaning, for a vector t ∈ Rd , ReLU(t) is a vector such that ReLU(t)i =ReLU(ti )), we havea = ReLU(z)>Define W [2] = [w[2] ] ∈ R1×m similarly.
tion (7.11) can be summarized as(7.15)
Then, the model in equa-

a = ReLU(W [1] x + b[1] )hθ (x) = W [2] a + b[2](7.16)

Here θ consists of W [1] , W [2] (often referred to as the weight matrices) andb[1] , b[2] (referred to as the biases). The collection of W [1] , b[1] is referred to as
the first layer, and W [2] , b[2] the second layer. The activation a is referred to as
the hidden layer. A two-layer neural network is also called one-hidden-layer
neural network.
Multi-layer fully-connected neural networks. With this succinct notations, we can stack more layers to get a deeper fully-connected neural network. Let r be the number of layers (weight matrices). Let
W [1] , . . . , W [r] , b[1] , . . . , b[r] be the weight matrices and biases of all the layers.
Then a multi-layer neural network can be written asa[1] = ReLU(W [1] x + b[1] )a[2] = ReLU(W [2] a[1] + b[2] )···a[r−1] = ReLU(W [r−1] a[r−2] + b[r−1] )hθ (x) = W [r] a[r−1] + b[r](7.17)

We note that the weight matrices and biases need to have compatible
dimensions for the equations above to make sense. If a[k] has dimension mk ,then the weight matrix W [k] should be of dimension mk × mk−1 , and the bias
b[k] ∈ Rmk . Moreover, W [1] ∈ Rm1 ×d and W [r] ∈ R1×mr−1 .
The total number of neurons in the network is m1 + · · · + mr , and thetotal number of parameters in this network is (d + 1)m1 + (m1 + 1)m2 + · · · +(mr−1 + 1)mr .

90
Sometimes for notational consistency we also write a[0] = x, and a[r] =hθ (x). Then we have simple recursion thata[k] = ReLU(W [k] a[k−1] + b[k] ), ∀k = 1, . . . , r − 1(7.18)

Note that this would have be true for k = r if there were an additional
ReLU in equation (7.17), but often people like to make the last layer linear(aka without a ReLU) so that negative outputs are possible and it’s easier
to interpret the last layer as a linear model. (More on the interpretability at
the “connection to kernel method” paragraph of this section.)
Other activation functions. The activation function ReLU can be replaced by many other non-linear function σ(·) that maps R to R such as
1
1 + e−zez − e−z
σ(z) = ze + e−z

σ(z) =(sigmoid)(7.19)(tanh)(7.20)Why do we not use the identity function for σ(z)? That is, why
not use σ(z) = z? Assume for sake of argument that b[1] and b[2] are zeros.
Suppose σ(z) = z, then for two-layer neural network, we have thathθ (x) = W [2] a[1][2](7.21)[1]= W σ(z )by definition(7.22)[2] [1]since σ(z) = z(7.23)[2]=W W xfrom Equation (7.13)(7.24)= W̃ xwhere W̃ = W [2] W [1](7.25)=W z[1]

Notice how W [2] W [1] collapsed into W̃ .
This is because applying a linear function to another linear function will
result in a linear function over the original input (i.e., you can construct a W̃such that W̃ x = W [2] W [1] x). This loses much of the representational powerof the neural network as often times the output we are trying to predict
has a non-linear relationship with the inputs. Without non-linear activation
functions, the neural network will simply perform linear regression.
Connection to the Kernel Method. In the previous lectures, we covered
the concept of feature maps. Recall that the main motivation for feature

91
maps is to represent functions that are non-linear in the input x by θ> φ(x),where θ are the parameters and φ(x), the feature map, is a handcrafted
function non-linear in the raw input x. The performance of the learningalgorithms can significantly depends on the choice of the feature map φ(x).
Oftentimes people use domain knowledge to design the feature map φ(x) thatsuits the particular applications. The process of choosing the feature maps
is often referred to as feature engineering.
We can view deep learning as a way to automatically learn the rightfeature map (sometimes also referred to as “the representation”) as follows.
Suppose we denote by β the collection of the parameters in a fully-connectedneural networks (equation (7.17)) except those in the last layer. Then wecan abstract right a[r−1] as a function of the input x and the parameters in
β: a[r−1] = φβ (x). Now we can write the model ashθ (x) = W [r] φβ (x) + b[r](7.26)

When β is fixed, then φβ (·) can viewed as a feature map, and therefore hθ (x)is just a linear model over the features φβ (x). However, we will train theneural networks, both the parameters in β and the parameters W [r] , b[r] are
optimized, and therefore we are not learning a linear model in the feature
space, but also learning a good feature map φβ (·) itself so that it’s possible to predict accurately with a linear model on top of the feature map.
Therefore, deep learning tends to depend less on the domain knowledge ofthe particular applications and requires often less feature engineering. The
penultimate layer a[r] is often (informally) referred to as the learned featuresor representations in the context of deep learning.
In the example of house price prediction, a fully-connected neural network
does not need us to specify the intermediate quantity such “family size”, andmay automatically discover some useful features in the last penultimate layer(the activation a[r−1] ), and use them to linearly predict the housing price.
Often the feature map / representation obtained from one datasets (that is,
the function φβ (·) can be also useful for other datasets, which indicates they
contain essential information about the data. However, oftentimes, the neuralnetwork will discover complex features which are very useful for predicting
the output but may be difficult for a human to understand or interpret. This
is why some people refer to neural networks as a black box, as it can bedifficult to understand the features it has discovered.

92

7.3

Backpropagation

In this section, we introduce backpropgation or auto-differentiation, which
computes the gradient of the loss ∇J (j) (θ) efficiently. We will start with aninformal theorem that states that as long as a real-valued function f can be
efficiently computed/evaluated by a differentiable network or circuit, then its
gradient can be efficiently computed in a similar time. We will then show
how to do this concretely for fully-connected neural networks.
Because the formality of the general theorem is not the main focus here,
we will introduce the terms with informal definitions. By a differentiable
circuit or a differentiable network, we mean a composition of a sequence ofdifferentiable arithmetic operations (additions, subtraction, multiplication,
divisions, etc) and elementary differentiable functions (ReLU, exp, log, sin,cos, etc.). Let the size of the circuit be the total number of such operations
and elementary functions. We assume that each of the operations and functions, and their derivatives or partial derivatives ecan be computed in O(1)time in the computer.
Theorem 7.3.1: [backpropagation or auto-differentiation, informally stated]
Suppose a differentiable circuit of size N computes a real-valued function
f : R` → R. Then, the gradient ∇f can be computed in time O(N ), by acircuit of size O(N ).4
We note that the loss function J (j) (θ) for j-th example can be indeed
computed by a sequence of operations and functions involving additions,subtraction, multiplications, and non-linear activations. Thus the theorem
suggests that we should be able to compute the ∇J (j) (θ) in a similar timeto that for computing J (j) (θ) itself. This does not only apply to the fullyconnected neural network introduced in the Section 7.2, but also many othertypes of neural networks.
In the rest of the section, we will showcase how to compute the gradient of
the loss efficiently for fully-connected neural networks using backpropagation.
Even though auto-differentiation or backpropagation is implemented in all
the deep learning packages such as tensorflow and pytorch, understanding it
is very helpful for gaining insights into the working of deep learning.
4

We note if the output of the function f does not depend on some of the input coordinates, then we set by default the gradient w.r.t that coordinate to zero. Setting tozero does not count towards the total runtime here in our accounting scheme. This is why
when N ≤ `, we can compute the gradient in O(N ) time, which might be potentially evenless than `.

93

7.3.1

Preliminary: chain rule

We first recall the chain rule in calculus. Suppose the variable J depends onthe variables θ1 , . . . , θp via the intermediate variable g1 , . . . , gk :gj = gj (θ1 , . . . , θp ), ∀j ∈ {1, · · · , k}
J = J(g1 , . . . , gk )(7.27)(7.28)

Here we overload the meaning of gj ’s: they denote both the intermediatevariables but also the functions used to compute the intermediate variables.
Then, by the chain rule, we have that ∀i,kX∂J∂J ∂gj=∂θi∂gj ∂θij=1(7.29)For the ease of invoking the chain rule in the following subsections in various
ways, we will call J the output variable, g1 , . . . , gk intermediate variables,and θ1 , . . . , θp the input variable in the chain rule.

7.3.2

One-neuron neural networks

Simplifying notations:
In the rest of the section, we will consider ageneric input x and compute the gradient of hθ (x) w.r.t θ. For simplicity,
we use o as a shorthand for hθ (x) (o stands for output). For simplicity, withslight abuse of notation, we use J = 12 (y − o)2 to denote the loss function.
(Note that this overrides the definition of J as the total loss in Section 7.1.)
Our goal is to compute the derivative of J w.r.t the parameter θ.
We first consider the neural network with one neuron defined in equation (7.7). Recall that we compute the loss function via the following sequential steps:z = w> x + b
o = ReLU(z)1
J = (y − o)22(7.30)(7.31)(7.32)

By the chain rule with J as the output variable, o as the intermediate variable,
and wi the input variable, we have that
∂J ∂o
∂J
=
·
∂wi
∂o ∂wi(7.33)

94
Invoking the chain rule with o as the output variable, z as the intermediatevariable, and wi the input variable, we have that
∂o
∂o ∂z
=
·
∂wi
∂z ∂wi
Combining the equation above with equation (7.33), we have∂J∂J ∂o ∂z=··= (o − y) · ReLU0 (z) · xi∂wi∂o ∂z ∂wi∂o∂z(because ∂J= (o − y) and ∂z= ReLU0 (z) and ∂w= xi )∂oi∂JHere, the key is that we reduce the computation of ∂w
to the computai∂z∂J ∂otion of three simpler more “local” objects ∂o , ∂z , and ∂wi , which are much
simpler to compute because J directly depends on o via equation (7.32), o
directly depends on a via equation (7.31), and z directly depends on wi via
equation (7.30). Note that in a vectorized form, we can also write∇w J = (o − y) · ReLU0 (z) · x
Similarly, we compute the gradient w.r.t b by∂J∂J ∂o ∂z=··= (o − y) · ReLU0 (z)∂b∂o ∂z ∂b∂o(because ∂J= (o − y) and ∂z= ReLU0 (z) and ∂z
= 1)
∂o
∂b

7.3.3

Two-layer neural networks: a low-level unpacked
computation

Note: this subsection derives the derivatives with low-level notations to
help you build up intuition on backpropagation. If you are looking for aclean formula, or you are familiar with matrix derivatives, then feel free tojump to the next subsection directly.
Now we consider the two-layer neural network defined in equation (7.11).
We compute the loss J by following sequence of operations∀j ∈ [1, ..., m],[1] >zj = wj

[1]

[1]

[1]

x + bj where wj ∈ Rd , bj

aj = ReLU(zj ),a = [a1 , . . . , am ]> ∈ Rm>

o = w[2] a + b[2] where w[2] ∈ Rm , b[2] ∈ R
1
J = (y − o)22(7.34)

95[1]

We will use (w[2] )` to denote the `-th coordinate of w[2] , and (wj )` to denote[1]
the `-coordinate of wj . (We will avoid using these cumbersome notations
once we figure out how to write everything in matrix and vector forms.)
By invoking chain rule with J as the output variable, o as intermediatevariable, and (w[2] )` as the input variable, we have
∂J
∂J ∂o
=
[2]
∂(w )`∂o ∂(w[2] )`∂o= (o − y)∂(w[2] )`= (o − y)a`It’s more challenging to compute

∂J
[1] .
∂(wj )`

Towards computing it, we first

invoke the chain rule with J as the output variable, zj as the intermediate[1]
variable, and (wj )` as the input variable.
∂J
[1]

=

∂J
∂zj
·
∂zj ∂(wj[1] )`

=

∂J
· x`
∂zj

∂(wj )`(becaues∂zj[1]∂(wj )`= x` .)∂J
Thus, it suffices to compute the ∂z. We invoke the chain rule with J as thejoutput variable, aj as the intermediate variable, and zj as the input variable,∂J∂J ∂aj=∂zj∂aj ∂zj∂J=
ReLU0 (zj )∂aj∂JNow it suffices to compute ∂a, and we invoke the chain rule with J as thejoutput variable, o as the intermediate variable, and aj as the input variable,

∂J
∂J ∂o
=
∂aj
∂o ∂aj= (o − y) · (w[2] )j
Now combining the equations above, we obtain∂J[1]∂(wj )`= (o − y) · (w[2] )j ReLU0 (zj )x`96
Next we gauge the runtime of computing these partial derivatives. Let pdenotes the total number of parameters in the network. We note that p ≥ md
where m is the number of hidden units and d is the input dimension. For
every j and `, to compute ∂J[1] , apparently we need to compute at least∂(wj )`

the output o, which takes at least p ≥ md operations. Therefore at the firstglance computing a single gradient takes at least md time, and the total time
to compute the derivatives w.r.t to all the parameters is at least (md)2 , whichis inefficient.
However, the key of the backpropagation is that for different choices of `,the formulas above for computing ∂J[1] share many terms, such as, (o − y),∂(wj )`[2]0(w )j and ReLU (zj ). This suggests that we can re-organize the computation
to leverage the shared computation.
,
It turns out the crucial shared quantities in these formulas are ∂J∂o∂J∂J, . . . , ∂zm . We now write the following formulas to compute the gradi∂z1
ents efficiently in Algorithm 3.
Algorithm 3 Backpropagation for two-layer neural networks
1: Compute the values of z1 , . . . , zm , a1 , . . . , am and o as in the definition ofneural network (equation (7.34)).
2: Compute ∂J= (o − y).
∂o∂J
3: Compute ∂z for j = 1, . . . , m byj∂J∂J ∂o ∂aj∂J
· (w[2] )j · ReLU0 (zj )==∂zj∂o ∂aj ∂zj∂o
4: Compute

∂J
∂J
∂J
∂J
[1] ,
[1] , ∂(w [2] ) , and ∂b[2] byj∂(wj )` ∂bj∂J[1]∂(wj )`

∂J
[1]
∂bj

=

∂J
∂zj
∂J
·
· x`
=
[1]
∂zj ∂(wj )`∂zj=∂J ∂zj∂J· [1] =∂zj ∂bj
∂zj

∂J
∂J ∂o
∂J
=
=
· aj
[2]
[2]
∂(w )j∂o ∂(w )j
∂o
∂J ∂o
∂J
∂J
=
=
[2]
[2]
∂b
∂o ∂b
∂o

(7.35)

97

7.3.4

Two-layer neural network with vector notation

As we have done before in the definition of neural networks, the equations forbackpropagation becomes much cleaner with proper matrix notation. Here
we state the algorithm first and also provide a cleaner proof via matrix calculus.
Let
∂J
∈R
∂o
∂J
δ [1] ,∈ Rm∂z
δ [2] ,

(7.36)

Here we note that when A is a real-valued variable,5 and B is a vector or∂Amatrix variable, then ∂B
denotes the collection of the partial derivatives withthe same shape as B.6 In other words, if B is a matrix of dimension m × d,∂A∂A
as the ijth-entry. Let v w denotethen ∂B
is a matrix in Rm×d with ∂Bij
the entry-wise product of two vectors v and w of the same dimension. Now
we are ready to describe backpropagation in Algorithm 4.
Algorithm 4 Back-propagation for two-layer neural networks in vectorized
notations.
.
1: Compute the values of z ∈ Rm , a ∈ Rm , and o
2: Compute δ [2] = (o − y) ∈ R
3: Compute δ [1] = (o − y) · W [2]4: Compute>

ReLU0 (z) ∈ Rm×1

∂J
= δ [2] a> ∈ R1×m
∂W [2]∂J= δ [2] ∈ R∂b[2]∂J= δ [1] x> ∈ Rm×d
∂W [1]∂J= δ [1] ∈ Rm
∂b[1]

∂A
We will avoid using the notation ∂B
for A that is not a real-valued variable.
If you are familiar with the notion of total derivatives, we note that the dimensionality
here is different from that for total derivatives.
5

6

98
Derivation using the chain rule for matrix multiplication.
To
have a succinct derivation of the backpropagation algorithm in Algorithm 4
without working with the complex indices, we state the extensions of thechain rule in vectorized notations. It requires more knowledge of matrix
calculus to state the most general result, and therefore we will introduce
a few special cases that are most relevant for deep learning. Suppose J
is a real-valued output variable, z ∈ Rm is the intermediate variable andW ∈ Rm×d , u ∈ Rd are the input variables. Suppose they satisfy:z = W u + b, where W ∈ Rm×d
J = J(z)(7.37)∂Jand ∂W
by:Then we can compute ∂J
∂u

∂J
∂J
= W>
∂u
∂z
∂J >
∂J
=
·u
∂W
∂z
∂J
∂J
=
∂b
∂z

(7.38)(7.39)(7.40)

We can verify the dimensionality is indeed compatible because ∂J∈ Rm ,
∂z
d ∂J
m×d
>
1×d
>
d×m ∂J,u ∈R .
W ∈R
, ∂u ∈ R , ∂W ∈ R
Here the chain rule in equation (7.38) only works for the special cases
where z = W u. Another useful case is the following:a = σ(z), where σ is an element-wise activation, z, a ∈ Rd
J = J(a)Then, we have that
∂J
∂J
=
∂z
∂a

σ 0 (z)(7.41)

where σ 0 (·) is the element-wise derivative of the activation function σ, andis element-wise product of two vectors of the same dimensionality.
Using equation (7.38), (7.39),and (7.41), we can verify the correctness of
Algorithm 4. Indeed, using the notations in the two-layer neural network∂J∂J=
ReLU0 (z)∂z∂a>= (o − y)W [2]
ReLU0 (z)equation (7.41) with setting(by Jinvoking← J, a ← a, z ← a, σ ← ReLU. )equation (7.38) with setting(Jby←invoking)
J, z ← o, W ← W [2] , u ← a, b ← b[2]99
Therefore, δ [1] = ∂J, and we verify the correctness of Line 3 in Algorithm 4.
∂z
Similarly, let’s verify the third equation in Line 4,
∂J
∂J >
·x
=
∂W [1]∂z= δ [1] x>

7.3.5

equation (7.39) with setting(Jby←invoking)
J, z ← z, W ← W [1] , u ← x, b ← b[1](because we have proved δ [1] = ∂J)∂zMulti-layer neural networks

In this section, we will derive the backpropagation algorithms for the model
defined in (7.17). Recall that we havea[1] = ReLU(W [1] x + b[1] )a[2] = ReLU(W [2] a[1] + b[2] )···a[r−1] = ReLU(W [r−1] a[r−2] + b[r−1] )a[r] = z [r] = W [r] a[r−1] + b[r]1
J = (a[r] − y)22Here we define both a[r] and z [r] as hθ (x) for notational simplicity.
Defineδ [k] =

∂J∂z [k](7.42)The backpropagation algorithm computes δ [k] ’s from k = r to 1, and∂J[k]computes ∂W
as described in Algorithm 5.
[ k] from δ

7.4

Vectorization over training examples

As we discussed in Section 7.1, in the implementation of neural networks,
we will leverage the parallelism across the multiple examples. This meansthat we will need to write the forward pass (the evaluation of the outputs)of the neural network and the backward pass (backpropagation) for multiple
training examples in matrix notation.
The basic idea.
The basic idea is simple. Suppose you have a training
set with three examples x(1) , x(2) , x(3) . The first-layer activations for each

100
Algorithm 5 Back-propagation for multi-layer neural networks.
.
1: Compute and store the values of a[k] ’s and z [k] ’s for k = 1, . . . , r − 1, andJ.
. This is often called the “forward pass”
∂J
[r]
[r]
2: Compute δ = ∂z [r] = (z − o).
3: for k = r − 1 to 1 do
4:Computeδ [k] =5:∂J[k+1] > [k+1]=Wδ∂z [k]

ReLU0 (z [k] )Compute∂J>= δ [k+1] a[k]
[k+1]
∂W
∂J
= δ [k+1][k+1]∂b

example are as follows:z [1](1) = W [1] x(1) + b[1]z [1](2) = W [1] x(2) + b[1]z [1](3) = W [1] x(3) + b[1]Note the difference between square brackets [·], which refer to the layer number, and parenthesis (·), which refer to the training example number. Intuitively, one would implement this using a for loop. It turns out, we canvectorize these operations as well. First, define:


|
|
|
X =  x(1) x(2) x(3)  ∈ Rd×3
(7.43)|||
Note that we are stacking training examples in columns and not rows. We
can then combine this into a single unified formulation:


|
|
|
Z [1] =  z [1](1) z [1](2) z [1](3)  = W [1] X + b[1](7.44)|||You may notice that we are attempting to add b[1] ∈ R4×1 to W [1] X ∈R4×3 . Strictly following the rules of linear algebra, this is not allowed. In

101practice however, this addition is performed using broadcasting. We create
an intermediate b̃[1] ∈ R4×3 :


|
|
|
(7.45)b̃[1] =  b[1] b[1] b[1] |||We can then perform the computation: Z [1] = W [1] X + b̃[1] . Often times, it
is not necessary to explicitly construct b̃[1] . By inspecting the dimensions in(7.44), you can assume b[1] ∈ R4×1 is correctly broadcast to W [1] X ∈ R4×3 .
The matricization approach as above can easily generalize to multiple
layers, with one subtlety though, as discussed below.
Complications/Subtlety in the Implementation. All the deep learning packages or implementations put the data points in the rows of a data
matrix. (If the data point itself is a matrix or tensor, then the data are concentrated along the zero-th dimension.) However, most of the deep learning
papers use a similar notation to these notes where the data points are treatedas column vectors.7 There is a simple conversion to deal with the mismatch:in the implementation, all the columns become row vectors, row vectors become column vectors, all the matrices are transposed, and the orders of thematrix multiplications are flipped. In the example above, using the row major convention, the data matrix is X ∈ R3×d , the first layer weight matrix
has dimensionality d × m (instead of m × d as in the two layer neural net
section), and the bias vector b[1] ∈ R1×m . The computation for the hidden
activation becomes
Z [1] = XW [1] + b[1] ∈ R3×m
(7.46)

7

The instructor suspects that this is mostly because in mathematics we naturally multiply a matrix to a vector on the left hand side.

Part III
Generalization andregularization

102

Chapter 8Generalization
This chapter discusses tools to analyze and understand the generalization of machine learning models, i.e, their performances on unseen test
examples. Recall that for supervised learning problems, given a training dataset {(x(i) , y (i) )}ni=1 , we typically learn a model hθ by minimizing aloss/cost function J(θ), which encourages hθ to fit the data. E.g., when
the loss functionis the least square loss (aka mean squared error), we havePn
1(i)
J(θ) = n i=1 (y − hθ (x(i) ))2 . This loss function for training purposes isoftentimes referred to as the training loss/error/cost.
However, minimizing the training loss is not our ultimate goal—it ismerely our approach towards the goal of learning a predictive model. The
most important evaluation metric of a model is the loss on unseen test examples, which is oftentimes referred to as the test error. Formally, we sample atest example (x, y) from the so-called test distribution D, and measure themodel’s error on it, by, e.g., the mean squared error, (hθ (x) − y)2 . The expected loss/error over the randomness of the test example is called the test
loss/error,1
L(θ) = E(x,y)∼D [(y − hθ (x))2 ](8.1)

Note that the measurement of the error involves computing the expectation,and in practice, it can be approximated by the average error on many sampled
test examples, which are referred to as the test dataset. Note that the key
difference here between training and test datasets is that the test examples
1

In theoretical and statistical literature, we oftentimes call the uniform distributionb an empirical distribution, and callover the training set {(x(i) , y (i) )}ni=1 , denoted by D,
D the population distribution. Partly because of this, the training loss is also referred
to as the empirical loss/risk/error, and the test loss is also referred to as the population
loss/risk/error.

103

104
are unseen, in the sense that the training procedure has not used the test
examples. In classical statistical learning settings, the training examples arealso drawn from the same distribution as the test distribution D, but still
the test examples are unseen by the learning procedure whereas the training
examples are seen.2
Because of this key difference between training and test datasets, even
if they are both drawn from the same distribution D, the test error is notnecessarily always close to the training error.3 As a result, successfully minimizing the training error may not always lead to a small test error. We
typically say the model overfits the data if the model predicts accurately on
the training dataset but doesn’t generalize well to other test examples, thatis, if the training error is small but the test error is large. We say the model
underfits the data if the training error is relatively large4 (and in this case,typically the test error is also relatively large.)
This chapter studies how the test error is influenced by the learning procedure, especially the choice of model parameterizations. We will decompose
the test error into “bias” and “variance” terms and study how each of them isaffected by the choice of model parameterizations and their tradeoffs. Using
the bias-variance tradeoff, we will discuss when overfitting and underfitting
will occur and be avoided. We will also discuss the double descent phenomenon in Section 8.2 and some classical theoretical results in Section 8.3.
2

These days, researchers have increasingly been more interested in the setting with“domain shift”, that is, the training distribution and test distribution are different.
3
the difference between test error and training error is often referred to as the generalization gap. The term generalization error in some literature means the test error, andin some other literature means the generalization gap.
4e.g., larger than the intrinsic noise level of the data in regression problems.

105

8.1

Bias-variance tradeoff

1.5

training dataset

1.5

training data
ground truth h *

1.0
y

y

1.0
0.5
0.0
0.0

test dataset

test data
ground truth h *

0.5
0.0

0.2

0.4

x

0.6

0.8

1.0

0.0

0.2

0.4

x

0.6

0.8

1.0

Figure 8.1: A running example of training and test dataset for this section.
As an illustrating example, we consider the following training dataset andtest dataset, which are also shown in Figure 8.1. The training inputs x(i) ’s are
randomly chosen and the outputs y (i) are generated by y (i) = h? (x(i) ) + ξ (i)where the function h? (·) is a quadratic function and is shown in Figure 8.1
as the solid line, and ξ (i) is the a observation noise assumed to be generated
from ∼ N (0, σ 2 ). A test example (x, y) also has the same input-output
relationship y = h? (x) + ξ where ξ ∼ N (0, σ 2 ). It’s impossible to predict thenoise ξ, and therefore essentially our goal is to recover the function h? (·).
We will consider the test error of learning various types of models. When
talking about linear regression, we discussed the problem of whether to fit
a “simple” model such as the linear “y = θ0 + θ1 x,” or a more “complex”
model such as the polynomial “y = θ0 + θ1 x + · · · θ5 x5 .”We start with fitting a linear model, as shown in Figure 8.2. The best
fitted linear model cannot predict y from x accurately even on the training
dataset, let alone on the test dataset. This is because the true relationshipbetween y and x is not linear—any linear model is far away from the true
function h? (·). As a result, the training error is large and this is a typicalsituation of underfitting.

106
1.5

1.5

training data
best fit linear model

1.0
y

y

1.0

test data
best fit linear model

0.5
0.0

0.5
0.0

0.0

0.2

0.4

x

0.6

0.8

1.0

0.0

0.2

0.4

0.6

x

0.8

1.0

Figure 8.2: The best fit linear model has large training and test errors.
The issue cannot be mitigated with more training examples—even witha very large amount of, or even infinite training examples, the best fitted
linear model is still inaccurate and fails to capture the structure of the data
(Figure 8.3). Even if the noise is not present in the training data, the issue
still occurs (Figure 8.4). Therefore, the fundamental bottleneck here is thelinear model family’s inability to capture the structure in the data—linear
models cannot represent the true quadratic function h? —, but not the lack of
the data. Informally, we define the bias of a model to be the test error evenif we were to fit it to a very (say, infinitely) large training dataset. Thus, inthis case, the linear model suffers from large bias, and underfits (i.e., fails to
capture structure exhibited by) the data.
fitting linear models on a large dataset

y

1.0

1.0

0.5
0.0
0.0

fitting linear models on a noiseless dataset

1.5

training data
ground truth h *best fit linear modely

1.5

training data
ground truth h *best fit linear model

0.5
0.0

0.2

0.4

x

0.6

0.8

1.0

Figure 8.3: The best fit linear
model on a much larger dataset
still has a large training error.

0.0

0.2

0.4

x

0.6

0.8

1.0

Figure 8.4: The best fit linear
model on a noiseless dataset also
has a large training/test error.

Next, we fit a 5th-degree polynomial to the data. Figure 8.5 shows that
it fails to learn a good model either. However, the failure pattern is different
from the linear model case. Specifically, even though the learnt 5th-degree

107polynomial did a very good job predicting y (i) ’s from x(i) ’s for training examples, it does not work well on test examples (Figure 8.5). In other words,
the model learnt from the training set does not generalize well to other test
examples—the test error is high. Contrary to the behavior of linear models,
the bias of the 5-th degree polynomials is small—if we were to fit a 5-th degree polynomial to an extremely large dataset, the resulting model would beclose to a quadratic function and be accurate (Figure 8.6). This is because
the family of 5-th degree polynomials contains all the quadratic functions(setting θ5 = θ4 = θ3 = 0 results in a quadratic function), and, therefore,
5-th degree polynomials are in principle capable of capturing the structureof the data.
1.5

1.5

training data
best fit 5-th degree model

1.0
y

y

1.0
0.5
0.0
0.0

test data
ground truth h *best fit 5-th degree model

0.5
0.0

0.2

0.4

x

0.6

0.8

1.0

0.0

0.2

0.4

x

0.6

0.8

1.0

Figure 8.5: Best fit 5-th degree polynomial has zero training error, but still
has a large test error and does not recover the the ground truth. This is aclassic situation of overfitting.
fitting 5-th degree model on large dataset

1.5

y

1.0

training data
best fit 5-th degree model
ground truth h *

0.5
0.0
0.0

0.2

0.4

x

0.6

0.8

1.0

Figure 8.6: The best fit 5-th degree polynomial on a huge dataset nearlyrecovers the ground-truth—suggesting that the culprit in Figure 8.5 is thevariance (or lack of data) but not bias.
The failure of fitting 5-th degree polynomials can be captured by another108
component of the test error, called variance of a model fitting procedure.
Specifically, when fitting a 5-th degree polynomial as in Figure 8.7, there is alarge risk that we’re fitting patterns in the data that happened to be present
in our small, finite training set, but that do not reflect the wider pattern of
the relationship between x and y. These “spurious” patterns in the training
set are (mostly) due to the observation noise ξ (i) , and fitting these spurious
patters results in a model with large test error. In this case, we say the model
has a large variance.
fitting 5-th degree model on different datasets

0.5
0.0
0.0

training data
best fit 5-th degree model

1.0
y

1.0
y

1.5

training data
best fit 5-th degree model

0.5
0.0

0.2

0.4

x

0.6

0.8

1.0

0.0

1.5

training data
best fit 5-th degree model

1.0
y

1.5

0.5
0.0

0.2

0.4

x

0.6

0.8

1.0

0.0

0.2

0.4

x

0.6

0.8

1.0

Figure 8.7: The best fit 5-th degree models on three different datasets generated from the same distribution behave quite differently, suggesting theexistence of a large variance.
The variance can be intuitively (and mathematically, as shown in Section 8.1.1) characterized by the amount of variations across models learnt
on multiple different training datasets (drawn from the same underlying distribution). The “spurious patterns” are specific to the randomness of thenoise (and inputs) in a particular dataset, and thus are different across multiple training datasets. Therefore, overfitting to the “spurious patterns” ofmultiple datasets should result in very different models. Indeed, as shownin Figure 8.7, the models learned on the three different training datasets arequite different, overfitting to the “spurious patterns” of each datasets.
Often, there is a tradeoff between bias and variance. If our model is too“simple” and has very few parameters, then it may have large bias (but small
variance), and it typically may suffer from underfittng. If it is too “complex”
and has very many parameters, then it may suffer from large variance (buthave smaller bias), and thus overfitting. See Figure 8.8 for a typical tradeoff
between bias and variance.

109
Test Error (= Bias 2 +Variance)

Optimal Tradeoff

Error

Variance

Bias2
Model Complexity

Figure 8.8: An illustration of the typical bias-variance tradeoff.
As we will see formally in Section 8.1.1, the test error can be decomposed
as a summation of bias and variance. This means that the test error will
have a convex curve as the model complexity increases, and in practice weshould tune the model complexity to achieve the best tradeoff. For instance,in the example above, fitting a quadratic function does better than either ofthe extremes of a first or a 5-th degree polynomial, as shown in Figure 8.9.
1.5

1.0
y

1.0
y

1.5

training data
best fit quadratic model

0.5
0.0
0.0

test data
best fit quadratic model
ground truth h *

0.5
0.0

0.2

0.4

x

0.6

0.8

1.0

0.0

0.2

0.4

x

0.6

0.8

1.0

Figure 8.9: Best fit quadratic model has small training and test error because
quadratic model achieves a better tradeoff.
Interestingly, the bias-variance tradeoff curves or the test error curves
do not universally follow the shape in Figure 8.8, at least not universallywhen the model complexity is simply measured by the number of parameters.
(We will discuss the so-called double descent phenomenon in Section 8.2.)
Nevertheless, the principle of bias-variance tradeoff is perhaps still the firstresort when analyzing and predicting the behavior of test errors.

110

8.1.1

A mathematical decomposition (for regression)

To formally state the bias-variance tradeoff for regression problems, we consider the following setup (which is an extension of the beginning paragraphof Section 8.1).
• Draw a training dataset S = {x(i) , y (i) }ni=1 such that y (i) = h? (x(i) ) + ξ (i)where ξ (i) ∈ N (0, σ 2 ).
• Train a model on the dataset S, denoted by ĥS .
• Take a test example (x, y) such that y = h? (x) + ξ where ξ ∼ N (0, σ 2 ),and measure the expected test error (averaged over the random draw of
the training set S and the randomness of ξ)56
MSE(x) = ES,ξ [(y − hS (x))2 ](8.2)

We will decompose the MSE into a bias and variance term. We start bystating a following simple mathematical tool that will be used twice below.
Claim 8.1.1: Suppose A and B are two independent real random variables
and E[A] = 0. Then, E[(A + B)2 ] = E[A2 ] + E[B 2 ].
As a corollary, because a random variable A is independent with a constant c, when E[A] = 0, we have E[(A + c)2 ] = E[A2 ] + c2 .
The proof of the claim follows from expanding the square: E[(A + B)2 ] =E[A2 ] + E[B 2 ] + 2E[AB] = E[A2 ] + E[B 2 ]. Here we used the independence toshow that E[AB] = E[A]E[B] = 0.
Using Claim 8.1.1 with A = ξ and B = h? (x) − ĥS (x), we haveMSE(x) = E[(y − hS (x))2 ] = E[(ξ + (h? (x) − hS (x)))2 ]
(8.3)
2
?
2
= E[ξ ] + E[(h (x) − hS (x)) ] (by Claim 8.1.1)= σ 2 + E[(h? (x) − hS (x))2 ](8.4)
Then, let’s define havg (x) = ES [hS (x)] as the “average model”—the modelobtained by drawing an infinite number of datasets, training on them, andaveraging their predictions on x. Note that havg is a hypothetical model for
analytical purposes that can not be obtained in reality (because we don’t
5

For simplicity, the test input x is considered to be fixed here, but the same conceptual
message holds when we average over the choice of x’s.
6
The subscript under the expectation symbol is to emphasize the variables that areconsidered as random by the expectation operation.

111
have infinite number of datasets). It turns out that for many cases, havg
is (approximately) equal to the the model obtained by training on a single
dataset with infinite samples. Thus, we can also intuitively interpret havg thisway, which is consistent with our intuitive definition of bias in the previous
subsection.
We can further decompose MSE(x) by letting c = h? (x)−havg (x) (which isa constant that does not depend on the choice of S!) and A = havg (x)−hS (x)in the corollary part of Claim 8.1.1:
MSE(x) = σ 2 + E[(h? (x) − hS (x))2 ]= σ 2 + (h? (x) − havg (x))2 + E[(havg − hS (x))2 ]= |{z}σ2+ (h? (x) − havg (x))2 + var(hS (x)){z
} | {z }|unavoidable, bias2

(8.5)
(8.6)
(8.7)

, variance

We call the second term the bias (square) and the third term the variance. As
discussed before, the bias captures the part of the error that are introduced
due to the lack of expressivity of the model. Recall that havg can be thoughtof as the best possible model learned even with infinite data. Thus, the bias is
not due to the lack of data, but is rather caused by that the family of models
fundamentally cannot approximate the h? . For example, in the illustrating
example in Figure 8.2, because any linear model cannot approximate thetrue quadratic function h? , neither can havg , and thus the bias term has tobe large.
The variance term captures how the random nature of the finite dataset
introduces errors in the learned model. It measures the sensitivity of thelearned model to the randomness in the dataset. It often decreases as thesize of the dataset increases.
There is nothing we can do about the first term σ 2 as we can not predict
the noise ξ by definition.
Finally, we note that the bias-variance decomposition for classification
is much less clear than for regression problems. There have been several
proposals, but there is as yet no agreement on what is the “right” and/or
the most useful formalism.

8.2

The double descent phenomenon

Model-wise double descent. Recent works have demonstrated that thetest error can present a “double descent” phenomenon in a range of machine

112
learning models including linear models and deep neural networks.7 The
conventional wisdom, as discussed in Section 8.1, is that as we increase themodel complexity, the test error first decreases and then increases, as illustrated in Figure 8.8. However, in many cases, we empirically observe thatthe test error can have a second descent—it first decreases, then increases
to a peak around when the model size is large enough to fit all the training
data very well, and then decreases again in the so-called overparameterized
regime, where the number of parameters is larger than the number of data
points. See Figure 8.10 for an illustration of the typical curves of test errors
against model complexity (measured by the number of parameters). To someextent, the overparameterized regime with the second descent is considered as
new to the machine learning community—partly because lightly-regularized,
overparameterized models are only extensively used in the deep learning era.
A practical implication of the phenomenon is that one should not hold backfrom scaling into and experimenting with over-parametrized models because
the test error may well decrease again to a level even smaller than the previous lowest point. Actually, in many cases, larger overparameterized models
always lead to a better test performance (meaning there won’t be a second
ascent after the second descent).
classical regime:bias-variance tradeoff

modern regime:
over-parameterization

test error

typically when # parameters
is sufficient to fit the data

# parameters

Figure 8.10: A typical model-wise double descent phenomenon. As the number of parameters increases, the test error first decreases when the number ofparameters is smaller than the training data. Then in the overparameterized
regime, the test error decreases again.
7

The discovery of the phenomenon perhaps dates back to Opper [1995, 2001], and has
been recently popularized by Belkin et al. [2020], Hastie et al. [2019], etc.

113
Sample-wise double descent. A priori, we would expect that moretraining examples always lead to smaller test errors—more samples givestrictly more information for the algorithm to learn from. However, recent
work [Nakkiran, 2019] observes that the test error is not monotonically decreasing as we increase the sample size. Instead, as shown in Figure 8.11, thetest error decreases, and then increases and peaks around when the numberof examples (denoted by n) is similar to the number of parameters (denotedby d), and then decreases again. We refer to this as the sample-wise double descent phenomenon. To some extent, sample-wise double descent andmodel-wise double descent are essentially describing similar phenomena—thetest error is peaked when n ≈ d.
Explanation and mitigation strategy. The sample-wise double descent,or, in particular, the peak of test error at n ≈ d, suggests that the existing
training algorithms evaluated in these experiments are far from optimal whenn ≈ d. We will be better off by tossing away some examples and run thealgorithms with a smaller sample size to steer clear of the peak. In otherwords, in principle, there are other algorithms that can achieve smaller test
error when n ≈ d, but the algorithms evaluated in these experiments fail todo so. The sub-optimality of the learning procedure appears to be the culpritof the peak in both sample-wise and model-wise double descent.
Indeed, with an optimally-tuned regularization (which will be discussed
more in Section 9), the test error in the n ≈ d regime can be dramatically
improved, and the model-wise and sample-wise double descent are both mitigated. See Figure 8.11.
The intuition above only explains the peak in the model-wise and samplewise double descent, but does not explain the second descent in the modelwise double descent—why overparameterized models are able to generalize
so well. The theoretical understanding of overparameterized models is an active research area with many recent advances. A typical explanation is thatthe commonly-used optimizers such as gradient descent provide an implicit
regularization effect (which will be discussed in more detail in Section 9.2).
In other words, even in the overparameterized regime and with an unregularized loss function, the model is still implicitly regularized, and thus exhibits
a better test performance than an arbitrary solution that fits the data. For
example, for linear models, when n  d, the gradient descent optimizer withzero initialization finds the minimum norm solution that fits the data (instead of an arbitrary solution that fits the data), and the minimum norm regularizer turns out to be a sufficiently good for the overparameterized regime(but it’s not a good regularizer when n ≈ d, resulting in the peak of test

114
error).
Test Error vs. # Samples

2.00

Test Error

1.75
1.50

Test Error

1.25
1.00
0.75
0.50
0.25
0.00

0

200

400
600
Num Samples

800

1000

Figure 8.11: Left: The sample-wise double descent phenomenon for linear
models. Right: The sample-wise double descent with different regularization
strength for linear models. Using the optimal regularization parameter λ(optimally tuned for each n, shown in green solid curve) mitigates double
descent. Setup: The data distribution of (x, y) is x ∼ N (0, Id ) and y ∼x> β + N (0, σ 2 ) where d = 500, σ = 0.5 and kβk2 = 1.8
Finally, we also remark that the double descent phenomenon has beenmostly observed when the model complexity is measured by the number ofparameters. It is unclear if and when the number of parameters is the bestcomplexity measure of a model. For example, in many situations, the normof the models is used as a complexity measure. As shown in Figure 8.12
right, for a particular linear case, if we plot the test error against the normof the learnt model, the double descent phenomenon no longer occurs. Thisis partly because the norm of the learned model is also peaked around n ≈ d(See Figure 8.12 (middle) or Belkin et al. [2019], Mei and Montanari [2022],
and discussions in Section 10.8 of James et al. [2021]). For deep neuralnetworks, the correct complexity measure is even more elusive. The study of
double descent phenomenon is an active research topic.
8
The figure is reproduced from Figure 1 of Nakkiran et al. [2020]. Similar phenomenon
are also observed in Hastie et al. [2022], Mei and Montanari [2022]

115

0.8

test error

test error

norm
0

250
500
750
# parameters

1000

0

800
d=n

0.6
0.4

10

0.2

1000

0.8

20

0.4

test error vs. norm

1.0

30

0.6

0.0

norm vs. # params

40

0.2
0

200

400 600 800# parameters

1000

0.0

0

10

20
norm

30

40

# parameters

test error vs. # params

1.0

600
400
200
0

Figure 8.12: Left: The double descent phenomenon, where the number of parameters is used as the model complexity. Middle: The norm of the learned
model is peaked around n ≈ d. Right: The test error against the norm of
the learnt model. The color bar indicate the number of parameters and thearrows indicates the direction of increasing model size. Their relationshipare closer to the convention wisdom than to a double descent. Setup: We
consider a linear regression with a fixed dataset of size n = 500. The input
x is a random ReLU feature on Fashion-MNIST, and output y ∈ R10 is the
one-hot label. This is the same setting as in Section 5.2 of Nakkiran et al.
[2020].

116

8.3

Sample complexity
readings)

8.3.1

Preliminaries

bounds(optional

In this set of notes, we begin our foray into learning theory. Apart frombeing interesting and enlightening in its own right, this discussion will also
help us hone our intuitions and derive rules of thumb about how to best
apply learning algorithms in different settings. We will also seek to answer
a few questions: First, can we make formal the bias/variance tradeoff thatwas just discussed? This will also eventually lead us to talk about model
selection methods, which can, for instance, automatically decide what orderpolynomial to fit to a training set. Second, in machine learning it’s reallygeneralization error that we care about, but most learning algorithms fit theirmodels to the training set. Why should doing well on the training set tell us
anything about generalization error? Specifically, can we relate error on the
training set to generalization error? Third and finally, are there conditions
under which we can actually prove that learning algorithms will work well?
We start with two simple but very useful lemmas.
Lemma. (The union bound). Let A1 , A2 , . . . , Ak be k different events (thatmay not be independent). Then
P (A1 ∪ · · · ∪ Ak ) ≤ P (A1 ) + . . . + P (Ak ).
In probability theory, the union bound is usually stated as an axiom(and thus we won’t try to prove it), but it also makes intuitive sense: Theprobability of any one of k events happening is at most the sum of theprobabilities of the k different events.
Lemma. (Hoeffding inequality) Let Z1 , . . . , Zn be n independent and identically distributed (iid) random variables drawn from a Bernoulli(φ)Pdistribution. I.e., P (Zi = 1) = φ, and P (Zi = 0) = 1 − φ. Let φ̂ = (1/n) ni=1 Zi
be the mean of these random variables, and let any γ > 0 be fixed. ThenP (|φ − φ̂| > γ) ≤ 2 exp(−2γ 2 n)This lemma (which in learning theory is also called the Chernoff bound)says that if we take φ̂—the average of n Bernoulli(φ) random variables—tobe our estimate of φ, then the probability of our being far from the true value
is small, so long as n is large. Another way of saying this is that if you havea biased coin whose chance of landing on heads is φ, then if you toss it n

117
times and calculate the fraction of times that it came up heads, that will bea good estimate of φ with high probability (if n is large).
Using just these two lemmas, we will be able to prove some of the deepest
and most important results in learning theory.
To simplify our exposition, let’s restrict our attention to binary classification in which the labels are y ∈ {0, 1}. Everything we’ll say here generalizes
to other problems, including regression and multi-class classification.
We assume we are given a training set S = {(x(i) , y (i) ); i = 1, . . . , n} of sizen, where the training examples (x(i) , y (i) ) are drawn iid from some probability
distribution D. For a hypothesis h, we define the training error (also called
the empirical risk or empirical error in learning theory) to ben1X
1{h(x(i) ) 6= y (i) }.
ε̂(h) =
n i=1
This is just the fraction of training examples that h misclassifies. When wewant to make explicit the dependence of ε̂(h) on the training set S, we may
also write this a ε̂S (h). We also define the generalization error to beε(h) = P(x,y)∼D (h(x) 6= y).
I.e. this is the probability that, if we now draw a new example (x, y) fromthe distribution D, h will misclassify it.
Note that we have assumed that the training data was drawn from thesame distribution D with which we’re going to evaluate our hypotheses (inthe definition of generalization error). This is sometimes also referred to as
one of the PAC assumptions.9
Consider the setting of linear classification, and let hθ (x) = 1{θT x ≥ 0}.
What’s a reasonable way of fitting the parameters θ? One approach is to try
to minimize the training error, and pickθ̂ = arg min ε̂(hθ ).
θWe call this process empirical risk minimization (ERM), and the resulting
hypothesis output by the learning algorithm is ĥ = hθ̂ . We think of ERM
as the most “basic” learning algorithm, and it will be this algorithm that we
9

PAC stands for “probably approximately correct,” which is a framework and set ofassumptions under which numerous results on learning theory were proved. Of these, theassumption of training and testing on the same distribution, and the assumption of the
independently drawn training examples, were the most important.

118
focus on in these notes. (Algorithms such as logistic regression can also beviewed as approximations to empirical risk minimization.)
In our study of learning theory, it will be useful to abstract away fromthe specific parameterization of hypotheses and from issues such as whether
we’re using a linear classifier. We define the hypothesis class H used by alearning algorithm to be the set of all classifiers considered by it. For linear
classification, H = {hθ : hθ (x) = 1{θT x ≥ 0}, θ ∈ Rd+1 } is thus the set of
all classifiers over X (the domain of the inputs) where the decision boundary
is linear. More broadly, if we were studying, say, neural networks, then we
could let H be the set of all classifiers representable by some neural networkarchitecture.
Empirical risk minimization can now be thought of as a minimization overthe class of functions H, in which the learning algorithm picks the hypothesis:ĥ = arg min ε̂(h)
h∈H

8.3.2

The case of finite H

Let’s start by considering a learning problem in which we have a finite hypothesis class H = {h1 , . . . , hk } consisting of k hypotheses. Thus, H is just a
set of k functions mapping from X to {0, 1}, and empirical risk minimization
selects ĥ to be whichever of these k functions has the smallest training error.
We would like to give guarantees on the generalization error of ĥ. Our
strategy for doing so will be in two parts: First, we will show that ε̂(h) is areliable estimate of ε(h) for all h. Second, we will show that this implies anupper-bound on the generalization error of ĥ.
Take any one, fixed, hi ∈ H. Consider a Bernoulli random variable Z
whose distribution is defined as follows. We’re going to sample (x, y) ∼ D.
Then, we set Z = 1{hi (x) 6= y}. I.e., we’re going to draw one example,
and let Z indicate whether hi misclassifies it. Similarly, we also define Zj =1{hi (x(j) ) 6= y (j) }. Since our training set was drawn iid from D, Z and the
Zj ’s have the same distribution.
We see that the misclassification probability on a randomly drawn
example—that is, ε(h)—is exactly the expected value of Z (and Zj ). Moreover, the training error can be written
n

1X
Zj .
ε̂(hi ) =n j=1
Thus, ε̂(hi ) is exactly the mean of the n random variables Zj that are drawn
iid from a Bernoulli distribution with mean ε(hi ). Hence, we can apply the

119
Hoeffding inequality, and obtain
P (|ε(hi ) − ε̂(hi )| > γ) ≤ 2 exp(−2γ 2 n).
This shows that, for our particular hi , training error will be close to
generalization error with high probability, assuming n is large. But we don’t
just want to guarantee that ε(hi ) will be close to ε̂(hi ) (with high probability)
for just only one particular hi . We want to prove that this will be true
simultaneously for all h ∈ H. To do so, let Ai denote the event that |ε(hi ) −ε̂(hi )| > γ. We’ve already shown that, for any particular Ai , it holds truethat P (Ai ) ≤ 2 exp(−2γ 2 n). Thus, using the union bound, we have thatP (∃ h ∈ H.|ε(hi ) − ε̂(hi )| > γ) = P (A1 ∪ · · · ∪ Ak )kX≤
P (Ai )≤

i=1kX

2 exp(−2γ 2 n)i=1

= 2k exp(−2γ 2 n)If we subtract both sides from 1, we find thatP (¬∃ h ∈ H.|ε(hi ) − ε̂(hi )| > γ) = P (∀h ∈ H.|ε(hi ) − ε̂(hi )| ≤ γ)
≥ 1 − 2k exp(−2γ 2 n)(The “¬” symbol means “not.”) So, with probability at least 1 −2k exp(−2γ 2 n), we have that ε(h) will be within γ of ε̂(h) for all h ∈ H.
This is called a uniform convergence result, because this is a bound that
holds simultaneously for all (as opposed to just one) h ∈ H.
In the discussion above, what we did was, for particular values of n andγ, give a bound on the probability that for some h ∈ H, |ε(h) − ε̂(h)| > γ.
There are three quantities of interest here: n, γ, and the probability of error;
we can bound either one in terms of the other two.
For instance, we can ask the following question: Given γ and some δ > 0,
how large must n be before we can guarantee that with probability at least
1 − δ, training error will be within γ of generalization error? By setting
δ = 2k exp(−2γ 2 n) and solving for n, [you should convince yourself this is
the right thing to do!], we find that if
n≥

2k
1
log ,
2
2γ
δ

120
then with probability at least 1 − δ, we have that |ε(h) − ε̂(h)| ≤ γ for allh ∈ H. (Equivalently, this shows that the probability that |ε(h) − ε̂(h)| > γ
for some h ∈ H is at most δ.) This bound tells us how many training
examples we need in order make a guarantee. The training set size n thata certain method or algorithm requires in order to achieve a certain level of
performance is also called the algorithm’s sample complexity.
The key property of the bound above is that the number of training
examples needed to make this guarantee is only logarithmic in k, the numberof hypotheses in H. This will be important later.
Similarly, we can also hold n and δ fixed and solve for γ in the previous
equation, and show [again, convince yourself that this is right!] that withprobability 1 − δ, we have that for all h ∈ H,
r
1
2k
log .
|ε̂(h) − ε(h)| ≤2nδ
Now, let’s assume that uniform convergence holds, i.e., that |ε(h)−ε̂(h)| ≤γ for all h ∈ H. What can we prove about the generalization of our learningalgorithm that picked ĥ = arg minh∈H ε̂(h)?
Define h∗ = arg minh∈H ε(h) to be the best possible hypothesis in H. Notethat h∗ is the best that we could possibly do given that we are using H, so
it makes sense to compare our performance to that of h∗ . We have:ε(ĥ) ≤ ε̂(ĥ) + γ≤ ε̂(h∗ ) + γ≤ ε(h∗ ) + 2γ
The first line used the fact that |ε(ĥ)− ε̂(ĥ)| ≤ γ (by our uniform convergenceassumption). The second used the fact that ĥ was chosen to minimize ε̂(h),and hence ε̂(ĥ) ≤ ε̂(h) for all h, and in particular ε̂(ĥ) ≤ ε̂(h∗ ). The thirdline used the uniform convergence assumption again, to show that ε̂(h∗ ) ≤ε(h∗ ) + γ. So, what we’ve shown is the following: If uniform convergenceoccurs, then the generalization error of ĥ is at most 2γ worse than the best
possible hypothesis in H!
Let’s put all this together into a theorem.
Theorem. Let |H| = k, and let any n, δ be fixed. Then with probability atleast 1 − δ, we have that
r


1
2k
ε(ĥ) ≤ min ε(h) + 2
log .
h∈H
2n
δ

121

√
This is proved by letting γ equal the · term, using our previous argument that uniform convergence occurs with probability at least 1 − δ, andthen noting that uniform convergence implies ε(h) is at most 2γ higher thanε(h∗ ) = minh∈H ε(h) (as we showed previously).
This also quantifies what we were saying previously saying about thebias/variance tradeoff in model selection. Specifically, suppose we have somehypothesis class H, and are considering switching to some much larger hypothesis class H0 ⊇ H. If we switch to H0 , then the first term minh ε(h)can only decrease (since we’d then be taking a min over a larger set of functions). Hence, by learning using a larger hypothesis class,√ our “bias” canonly decrease. However, if k increases, then the second 2 · term would also
increase. This increase corresponds to our “variance” increasing when we use
a larger hypothesis class.
By holding γ and δ fixed and solving for n like we did before, we can also
obtain the following sample complexity bound:
Corollary. Let |H| = k, and let any δ, γ be fixed. Then for ε(ĥ) ≤minh∈H ε(h) + 2γ to hold with probability at least 1 − δ, it suffices that
2k
1
log
2
2γ
δ


1
k
= O
log
,
γ2
δ

n ≥

8.3.3

The case of infinite H

We have proved some useful theorems for the case of finite hypothesis classes.
But many hypothesis classes, including any parameterized by real numbers(as in linear classification) actually contain an infinite number of functions.
Can we prove similar results for this setting?
Let’s start by going through something that is not the “right” argument.
Better and more general arguments exist, but this will be useful for honing
our intuitions about the domain.
Suppose we have an H that is parameterized by d real numbers. Since weare using a computer to represent real numbers, and IEEE double-precision
floating point (double’s in C) uses 64 bits to represent a floating point number, this means that our learning algorithm, assuming we’re using doubleprecision floating point, is parameterized by 64d bits. Thus, our hypothesis
class really consists of at most k = 264d different hypotheses. From the Corollary at the end of the previous section, we therefore find that, to guarantee122ε(ĥ) ≤ ε(h∗ ) + 2γ, with tohold with probability at least 1 − δ, it suffices that
1
d
264d
1
n ≥ O γ 2 log δ= O γ 2 log δ = Oγ,δ (d). (The γ, δ subscripts indicatethat the last big-O is hiding constants that may depend on γ and δ.) Thus,
the number of training examples needed is at most linear in the parametersof the model.
The fact that we relied on 64-bit floating point makes this argument notentirely satisfying, but the conclusion is nonetheless roughly correct: If what
we try to do is minimize training error, then in order to learn “well” using ahypothesis class that has d parameters, generally we’re going to need on theorder of a linear number of training examples in d.
(At this point, it’s worth noting that these results were proved for an algorithm that uses empirical risk minimization. Thus, while the linear dependence of sample complexity on d does generally hold for most discriminative
learning algorithms that try to minimize training error or some approximation to training error, these conclusions do not always apply as readily todiscriminative learning algorithms. Giving good theoretical guarantees on
many non-ERM learning algorithms is still an area of active research.)
The other part of our previous argument that’s slightly unsatisfying isthat it relies on the parameterization of H. Intuitively, this doesn’t seem likeit should matter: We had written the class of linear classifiers as hθ (x) =1{θ0 + θ1 x1 + · · · θd xd ≥ 0}, with n + 1 parameters θ0 , . . . , θd . But it could
also be written hu,v (x) = 1{(u20 − v02 ) + (u21 − v12 )x1 + · · · (u2d − vd2 )xd ≥ 0}with 2d + 2 parameters ui , vi . Yet, both of these are just defining the same
H: The set of linear classifiers in d dimensions.
To derive a more satisfying argument, let’s define a few more things.
Given a set S = {x(i) , . . . , x(D) } (no relation to the training set) of points(i)x ∈ X , we say that H shatters S if H can realize any labeling on S.
I.e., if for any set of labels {y (1) , . . . , y (D) }, there exists some h ∈ H so thath(x(i) ) = y (i) for all i = 1, . . . D.
Given a hypothesis class H, we then define its Vapnik-Chervonenkis
dimension, written VC(H), to be the size of the largest set that is shatteredby H. (If H can shatter arbitrarily large sets, then VC(H) = ∞.)For instance, consider the following set of three points:

123

1
0

x2

1
0

1
0

x1
Can the set H of linear classifiers in two dimensions (h(x) = 1{θ0 +θ1 x1 +
θ2 x2 ≥ 0}) can shatter the set above? The answer is yes. Specifically, we
see that, for any of the eight possible labelings of these points, we can find alinear classifier that obtains “zero training error” on them:

x2

x2
x1

x2

x2
x1

x2
x1

x2
x1

x2
x1

x1

x2
x1

x1

Moreover, it is possible to show that there is no set of 4 points that thishypothesis class can shatter. Thus, the largest set that H can shatter is ofsize 3, and hence VC(H) = 3.
Note that the VC dimension of H here is 3 even though there may besets of size 3 that it cannot shatter. For instance, if we had a set of threepoints lying in a straight line (left figure), then there is no way to find a linear
separator for the labeling of the three points shown below (right figure):

124

1
0

x2

1
0
1
0

x1

x2
x1

In order words, under the definition of the VC dimension, in order toprove that VC(H) is at least D, we need to show only that there’s at least
one set of size D that H can shatter.
The following theorem, due to Vapnik, can then be shown. (This is, many
would argue, the most important theorem in all of learning theory.)
Theorem. Let H be given, and let D = VC(H). Then with probability atleast 1 − δ, we have that for all h ∈ H,
!
r
D
n
1
1
log + log.
|ε(h) − ε̂(h)| ≤ On
D nδThus, with probability at least 1 − δ, we also have that:
!
r
n
1
1
D
ε(ĥ) ≤ ε(h∗ ) + O
log + log.
n
D n
δIn other words, if a hypothesis class has finite VC dimension, then uniform
convergence occurs as n becomes large. As before, this allows us to give abound on ε(h) in terms of ε(h∗ ). We also have the following corollary:
Corollary. For |ε(h) − ε̂(h)| ≤ γ to hold for all h ∈ H (and hence ε(ĥ) ≤ε(h∗ ) + 2γ) with probability at least 1 − δ, it suffices that n = Oγ,δ (D).
In other words, the number of training examples needed to learn “well”
using H is linear in the VC dimension of H. It turns out that, for “most”
hypothesis classes, the VC dimension (assuming a “reasonable” parameterization) is also roughly linear in the number of parameters. Putting thesetogether, we conclude that for a given hypothesis class H (and for an algorithm that tries to minimize training error), the number of training examples
needed to achieve generalization error close to that of the optimal classifier
is usually roughly linear in the number of parameters of H.

Chapter 9
Regularization and model
selection
9.1

Regularization

Recall that as discussed in Section 8.1, overftting is typically a result of using
too complex models, and we need to choose a proper model complexity toachieve the optimal bias-variance tradeoff. When the model complexity ismeasured by the number of parameters, we can vary the size of the model(e.g., the width of a neural net). However, the correct, informative complexity measure of the models can be a function of the parameters (e.g., `2 normof the parameters), which may not necessarily depend on the number of parameters. In such cases, we will use regularization, an important techniquein machine learning, control the model complexity and prevent overfitting.
Regularization typically involves adding an additional term, called a regularizer and denoted by R(θ) here, to the training loss/cost function:
Jλ (θ) = J(θ) + λR(θ)(9.1)

Here Jλ is often called the regularized loss, and λ ≥ 0 is called the regularization parameter. The regularizer R(θ) is a nonnegative function (in almost
all cases). In classical methods, R(θ) is purely a function of the parameter θ,
but some modern approach allows R(θ) to depend on the training dataset.1
The regularizer R(θ) is typically chosen to be some measure of the complexity of the model θ. Thus, when using the regularized loss, we aim tofind a model that both fit the data (a small loss J(θ)) and have a small
1

Here our notations generally omit the dependency on the training dataset for
simplicity—we write J(θ) even though it obviously needs to depend on the training dataset.

125

126
model complexity (a small R(θ)). The balance between the two objectives iscontrolled by the regularization parameter λ. When λ = 0, the regularized
loss is equivalent to the original loss. When λ is a sufficiently small positive
number, minimizing the regularized loss is effectively minimizing the originalloss with the regularizer as the tie-breaker. When the regularizer is extremely
large, then the original loss is not effective (and likely the model will have alarge bias.)
The most commonly used regularization is perhaps `2 regularization,
where R(θ) = 12 kθk22 . It encourages the optimizer to find a model withsmall `2 norm. In deep learning, it’s oftentimes referred to as weight decay, because gradient descent with learning rate η on the regularized lossRλ (θ) is equivalent to shrinking/decaying θ by a scalar factor of 1 − ηλ andthen applying the standard gradientθ ← θ − η∇Jλ (θ) = θ − ηλθ − η∇J(θ)= (1 − λη)θ −η∇J(θ)| {z }(9.2)

decaying weights

Besides encouraging simpler models, regularization can also impose inductive biases or structures on the model parameters. For example, suppose
we had a prior belief that the number of non-zeros in the ground-truth model
parameters is small,2 —which is oftentimes called sparsity of the model—, we
can impose a regularization on the number of non-zeros in θ, denoted bykθk0 , to leverage such a prior belief. Imposing additional structure of theparameters narrows our search space and makes the complexity of the modelfamily smaller,—e.g., the family of sparse models can be thought of as having
lower complexity than the family of all models—, and thus tends to lead to abetter generalization. On the other hand, imposing additional structure mayrisk increasing the bias. For example, if we regularize the sparsity strongly
but no sparse models can predict the label accurately, we will suffer fromlarge bias (analogously to the situation when we use linear models to learn
data than can only be represented by quadratic functions in Section 8.1.)
The sparsity of the parameters is not a continuous function of the parameters, and thus we cannot optimize it with (stochastic) gradient descent. A
common relaxation is to use R(θ) = kθk1 as a continuous surrogate.3
2

For linear models, this means the model just uses a few coordinates of the inputs tomake an accurate prediction.
3
There has been a rich line of theoretical work that explains why kθk1 is a good surrogate for encouraging sparsity, but it’s beyond the scope of this course. An intuition is:assuming the parameter is on the unit sphere, the parameter with smallest `1 norm also

127
The R(θ) = kθk1 (also called LASSO) and R(θ) = 21 kθk22 are perhaps
among the most commonly used regularizers for linear models. Other normand powers of norms are sometimes also used. The `2 norm regularization ismuch more commonly used with kernel methods because `1 regularization istypically not compatible with the kernel trick (the optimal solution cannot
be written as functions of inner products of features.)In deep learning, the most commonly used regularizer is `2 regularization
or weight decay. Other common ones include dropout, data augmentation,
regularizing the spectral norm of the weight matrices, and regularizing the
Lipschitzness of the model, etc. Regularization in deep learning is an active research area, and it’s known that there is another implicit source ofregularization, as discussed in the next section.

9.2

Implicit regularization effect

The implicit regularization effect of optimizers, or implicit bias or algorithmic
regularization, is a new concept/phenomenon observed in the deep learning
era. It largely refers to that the optimizers can implicitly impose structures
on parameters beyond what has been imposed by the regularized loss.
In most classical settings, the loss or regularized loss has a unique global
minimum, and thus any reasonable optimizer should converge to that global
minimum and cannot impose any additional preferences. However, in deeplearning, oftentimes the loss or regularized loss has more than one (approximate) global minima, and difference optimizers may converge to different
global minima. Though these global minima have the same or similar training losses, they may be of different nature and have dramatically different
generalization performance. See Figures 9.1 and 9.2 and its caption for an
illustration and some experiment results. For example, it’s possible that one
global minimum gives a much more Lipschitz or sparse model than others
and thus has a better test error. It turns out that many commonly-used optimizers (or their components) prefer or bias towards finding global minimaof certain properties, leading to a better test performance.
happen to be the sparsest parameter with only 1 non-zero coordinate. Thus, sparsity and`1 norm gives the same extremal points to some extent.

loss

128

θ

Figure 9.1: An Illustration that different global minima of the training loss
can have different test performance.

Figure 9.2: Left: Performance of neural networks trained by two different
learning rates schedules on the CIFAR-10 dataset. Although both experiments used exactly the same regularized losses and the optimizers fit thetraining data perfectly, the models’ generalization performance differ much.
Right: On a different synthetic dataset, optimizers with different initializations have the same training error but different generalization performance.4
In summary, the takehome message here is that the choice of optimizer
does not only affect minimizing the training loss, but also imposes implicit
regularization and affects the generalization of the model. Even if your current optimizer already converges to a small training error perfectly, you may
still need to tune your optimizer for a better generalization, .
4

The setting is the same as in Woodworth et al. [2020], HaoChen et al. [2020]

129
One may wonder which components of the optimizers bias towards what
type of global minima and what type of global minima may generalize better. These are open questions that researchers are actively investigating.
Empirical and theoretical research have offered some clues and heuristics.
In many (but definitely far from all) situations, among those setting where
optimization can succeed in minimizing the training loss, the use of larger
initial learning rate, smaller initialization, smaller batch size, and momentum appears to help with biasing towards more generalizable solutions. A
conjecture (that can be proven in certain simplified case) is that stochasticity in the optimization process help the optimizer to find flatter global
minima (global minima where the curvature of the loss is small), and flat
global minima tend to give more Lipschitz models and better generalization.
Characterizing the implicit regularization effect formally is still a challenging
open research question.

9.3

Model selection via cross validation

Suppose we are trying select among several different models for a learning
problem. For instance, we might be using a polynomial regression modelhθ (x) = g(θ0 + θ1 x + θ2 x2 + · · · + θk xk ), and wish to decide if k should be
0, 1, . . . , or 10. How can we automatically select a model that represents
a good tradeoff between the twin evils of bias and variance5 ? Alternatively,
suppose we want to automatically choose the bandwidth parameter τ forlocally weighted regression, or the parameter C for our `1 -regularized SVM.
How can we do that?
For the sake of concreteness, in these notes we assume we have somefinite set of models M = {M1 , . . . , Md } that we’re trying to select among.
For instance, in our first example above, the model Mi would be an i-th
degree polynomial regression model. (The generalization to infinite M is
not hard.6 ) Alternatively, if we are trying to decide between using an SVM,
a neural network or logistic regression, then M may contain these models.
5

Given that we said in the previous set of notes that bias and variance are two very
different beasts, some readers may be wondering if we should be calling them “twin” evils
here. Perhaps it’d be better to think of them as non-identical twins. The phrase “thefraternal twin evils of bias and variance” doesn’t have the same ring to it, though.
6
If we are trying to choose from an infinite set of models, say corresponding to thepossible values of the bandwidth τ ∈ R+ , we may discretize τ and consider only a finite
number of possible values for it. More generally, most of the algorithms described here
can all be viewed as performing optimization search in the space of models, and we canperform this search over infinite model classes as well.

130
Cross validation. Lets suppose we are, as usual, given a training set S.
Given what we know about empirical risk minimization, here’s what might
initially seem like a algorithm, resulting from using empirical risk minimization for model selection:
1. Train each model Mi on S, to get some hypothesis hi .
2. Pick the hypotheses with the smallest training error.
This algorithm does not work. Consider choosing the degree of a polynomial. The higher the degree of the polynomial, the better it will fit thetraining set S, and thus the lower the training error. Hence, this method will
always select a high-variance, high-degree polynomial model, which we saw
previously is often poor choice.
Here’s an algorithm that works better. In hold-out cross validation(also called simple cross validation), we do the following:
1. Randomly split S into Strain (say, 70% of the data) and Scv (the remaining 30%). Here, Scv is called the hold-out cross validation set.
2. Train each model Mi on Strain only, to get some hypothesis hi .
3. Select and output the hypothesis hi that had the smallest error ε̂Scv (hi )on the hold out cross validation set. (Here ε̂Scv (h) denotes the average
error of h on the set of examples in Scv .) The error on the hold out
validation set is also referred to as the validation error.
By testing/validating on a set of examples Scv that the models were nottrained on, we obtain a better estimate of each hypothesis hi ’s true generalization/test error. Thus, this approach is essentially picking the model withthe smallest estimated generalization/test error. The size of the validation
set depends on the total number of available examples. Usually, somewhere
between 1/4−1/3 of the data is used in the hold out cross validation set, and
30% is a typical choice. However, when the total dataset is huge, validation
set can be a smaller fraction of the total examples as long as the absolute
number of validation examples is decent. For example, for the ImageNet
dataset that has about 1M training images, the validation set is sometimes
set to be 50K images, which is only about 5% of the total examples.
Optionally, step 3 in the algorithm may also be replaced with selecting
the model Mi according to arg mini ε̂Scv (hi ), and then retraining Mi on theentire training set S. (This is often a good idea, with one exception being
learning algorithms that are be very sensitive to perturbations of the initial

131
conditions and/or data. For these methods, Mi doing well on Strain does notnecessarily mean it will also do well on Scv , and it might be better to forgo
this retraining step.)
The disadvantage of using hold out cross validation is that it “wastes”
about 30% of the data. Even if we were to take the optional step of retraining
the model on the entire training set, it’s still as if we’re trying to find a goodmodel for a learning problem in which we had 0.7n training examples, rather
than n training examples, since we’re testing models that were trained on
only 0.7n examples each time. While this is fine if data is abundant and/or
cheap, in learning problems in which data is scarce (consider a problem withn = 20, say), we’d like to do something better.
Here is a method, called k-fold cross validation, that holds out lessdata each time:
1. Randomly split S into k disjoint subsets of m/k training examples each.
Lets call these subsets S1 , . . . , Sk .
2. For each model Mi , we evaluate it as follows:
For j = 1, . . . , k
Train the model Mi on S1 ∪ · · · ∪ Sj−1 ∪ Sj+1 ∪ · · · Sk (i.e., train
on all the data except Sj ) to get some hypothesis hij .
Test the hypothesis hij on Sj , to get ε̂Sj (hij ).
The estimated generalization error of model Mi is then calculated
as the average of the ε̂Sj (hij )’s (averaged over j).
3. Pick the model Mi with the lowest estimated generalization error, andretrain that model on the entire training set S. The resulting hypothesisis then output as our final answer.
A typical choice for the number of folds to use here would be k = 10.
While the fraction of data held out each time is now 1/k—much smaller
than before—this procedure may also be more computationally expensive
than hold-out cross validation, since we now need train to each model ktimes.
While k = 10 is a commonly used choice, in problems in which data isreally scarce, sometimes we will use the extreme choice of k = m in orderto leave out as little data as possible each time. In this setting, we would
repeatedly train on all but one of the training examples in S, and test on that
held-out example. The resulting m = k errors are then averaged together toobtain our estimate of the generalization error of a model. This method has

132
its own name; since we’re holding out one training example at a time, thismethod is called leave-one-out cross validation.
Finally, even though we have described the different versions of cross validation as methods for selecting a model, they can also be used more simply toevaluate a single model or algorithm. For example, if you have implemented
some learning algorithm and want to estimate how well it performs for yourapplication (or if you have invented a novel learning algorithm and want toreport in a technical paper how well it performs on various test sets), crossvalidation would give a reasonable way of doing so.

9.4

Bayesian statistics and regularization

In this section, we will talk about one more tool in our arsenal for our battle
against overfitting.
At the beginning of the quarter, we talked about parameter fitting using
maximum likelihood estimation (MLE), and chose our parameters according
tonYθMLE = arg max
p(y (i) |x(i) ; θ).
θ

i=1

Throughout our subsequent discussions, we viewed θ as an unknown parameter of the world. This view of the θ as being constant-valued but unknown
is taken in frequentist statistics. In the frequentist this view of the world, θ
is not random—it just happens to be unknown—and it’s our job to come up
with statistical procedures (such as maximum likelihood) to try to estimate
this parameter.
An alternative way to approach our parameter estimation problems is totake the Bayesian view of the world, and think of θ as being a randomvariable whose value is unknown. In this approach, we would specify aprior distribution p(θ) on θ that expresses our “prior beliefs” about theparameters. Given a training set S = {(x(i) , y (i) )}ni=1 , when we are asked tomake a prediction on a new value of x, we can then compute the posterior
distribution on the parametersp(S|θ)p(θ)
p(S)

Qn
(i) (i)i=1 p(y |x , θ) p(θ)= R Qn( i=1 p(y (i) |x(i) , θ)p(θ)) dθθp(θ|S) =(9.3)

In the equation above, p(y (i) |x(i) , θ) comes from whatever model you’re using

133
for your learning problem. For example, if you are using Bayesian logistic re(i)(i)
gression, then you might choose p(y (i) |x(i) , θ) = hθ (x(i) )y (1−hθ (x(i) ))(1−y ) ,where hθ (x(i) ) = 1/(1 + exp(−θT x(i) )).7
When we are given a new test example x and asked to make it prediction
on it, we can compute our posterior distribution on the class label using theposterior distribution on θ:Z(9.4)
p(y|x, S) = p(y|x, θ)p(θ|S)dθθ

In the equation above, p(θ|S) comes from Equation (9.3). Thus, for example,
if the goal is to the predict the expected value of y given x, then we would
output8Z
E[y|x, S] = yp(y|x, S)dyy

The procedure that we’ve outlined here can be thought of as doing “fully
Bayesian” prediction, where our prediction is computed by taking an averagewith respect to the posterior p(θ|S) over θ. Unfortunately, in general it iscomputationally very difficult to compute this posterior distribution. This isbecause it requires taking integrals over the (usually high-dimensional) θ asin Equation (9.3), and this typically cannot be done in closed-form.
Thus, in practice we will instead approximate the posterior distributionfor θ. One common approximation is to replace our posterior distribution forθ (as in Equation 9.4) with a single point estimate. The MAP (maximum
a posteriori) estimate for θ is given by
θMAP = arg max
θnYp(y (i) |x(i) , θ)p(θ).

(9.5)

i=1

Note that this is the same formulas as for the MLE (maximum likelihood)
estimate for θ, except for the prior p(θ) term at the end.
In practical applications, a common choice for the prior p(θ) is to assume
that θ ∼ N (0, τ 2 I). Using this choice of prior, the fitted parameters θMAP will
have smaller norm than that selected by maximum likelihood. In practice,
this causes the Bayesian MAP estimate to be less susceptible to overfitting
than the ML estimate of the parameters. For example, Bayesian logistic
regression turns out to be an effective algorithm for text classification, eventhough in text classification we usually have d  n.
7Since we are now viewing θ as a random variable, it is okay to condition on it value,
and write “p(y|x, θ)” instead of “p(y|x; θ).”8
The integral below would be replaced by a summation if y is discrete-valued.

Part IV
Unsupervised learning

134

Chapter 10
Clustering and the k-means
algorithm
In the clustering problem, we are given a training set {x(1) , . . . , x(n) }, andwant to group the data into a few cohesive “clusters.” Here, x(i) ∈ Rd
as usual; but no labels y (i) are given. So, this is an unsupervised learning
problem.
The k-means clustering algorithm is as follows:
1. Initialize cluster centroids µ1 , µ2 , . . . , µk ∈ Rd randomly.
2. Repeat until convergence: {For every i, setc(i) := arg min ||x(i) − µj ||2 .
jFor each j, setPn(i)= j}x(i)i=1 1{c.
µj := Pn(i) = j}i=1 1{c}In the algorithm above, k (a parameter of the algorithm) is the numberof clusters we want to find; and the cluster centroids µj represent our current
guesses for the positions of the centers of the clusters. To initialize the cluster
centroids (in step 1 of the algorithm above), we could choose k training
examples randomly, and set the cluster centroids to be equal to the values ofthese k examples. (Other initialization methods are also possible.)
The inner-loop of the algorithm repeatedly carries out two steps: (i)“Assigning” each training example x(i) to the closest cluster centroid µj , and
135

136

Figure 10.1: K-means algorithm. Training examples are shown as dots, andcluster centroids are shown as crosses. (a) Original dataset. (b) Random initial cluster centroids (in this instance, not chosen to be equal to two training
examples). (c-f) Illustration of running two iterations of k-means. In each
iteration, we assign each training example to the closest cluster centroid(shown by “painting” the training examples the same color as the cluster
centroid to which is assigned); then we move each cluster centroid to themean of the points assigned to it. (Best viewed in color.) Images courtesy
Michael Jordan.
(ii) Moving each cluster centroid µj to the mean of the points assigned to it.
Figure 10.1 shows an illustration of running k-means.
Is the k-means algorithm guaranteed to converge? Yes it is, in a certain
sense. In particular, let us define the distortion function to be:
J(c, µ) =nX||x(i) − µc(i) ||2

i=1

Thus, J measures the sum of squared distances between each training example x(i) and the cluster centroid µc(i) to which it has been assigned. It canbe shown that k-means is exactly coordinate descent on J. Specifically, theinner-loop of k-means repeatedly minimizes J with respect to c while holdingµ fixed, and then minimizes J with respect to µ while holding c fixed. Thus,

137
J must monotonically decrease, and the value of J must converge. (Usually, this implies that c and µ will converge too. In theory, it is possible fork-means to oscillate between a few different clusterings—i.e., a few different
values for c and/or µ—that have exactly the same value of J, but this almost
never happens in practice.)
The distortion function J is a non-convex function, and so coordinate
descent on J is not guaranteed to converge to the global minimum. In otherwords, k-means can be susceptible to local optima. Very often k-means will
work fine and come up with very good clusterings despite this. But if you
are worried about getting stuck in bad local minima, one common thing to
do is run k-means many times (using different random initial values for thecluster centroids µj ). Then, out of all the different clusterings found, pickthe one that gives the lowest distortion J(c, µ).

Chapter 11
EM algorithms
In this set of notes, we discuss the EM (Expectation-Maximization) algorithm
for density estimation.

11.1

EM for mixture of Gaussians

Suppose that we are given a training set {x(1) , . . . , x(n) } as usual. Since weare in the unsupervised learning setting, these points do not come with anylabels.
We wish to model the data by specifying a joint distribution P
p(x(i) , z (i) ) =(i) (i)(i)(i)
p(x |z )p(z ). Here, z ∼ Multinomial(φ) (where φj ≥ 0, kj=1 φj = 1,
and the parameter φj gives p(z (i) = j)), and x(i) |z (i) = j ∼ N (µj , Σj ). We
let k denote the number of values that the z (i) ’s can take on. Thus, ourmodel posits that each x(i) was generated by randomly choosing z (i) from
{1, . . . , k}, and then x(i) was drawn from one of k Gaussians depending onz (i) . This is called the mixture of Gaussians model. Also, note that thez (i) ’s are latent random variables, meaning that they’re hidden/unobserved.
This is what will make our estimation problem difficult.
The parameters of our model are thus φ, µ and Σ. To estimate them, we
can write down the likelihood of our data:`(φ, µ, Σ) =

=

n
X
i=1
n
X
i=1

log p(x(i) ; φ, µ, Σ)

log

k
X

p(x(i) |z (i) ; µ, Σ)p(z (i) ; φ).

z (i) =1

However, if we set to zero the derivatives of this formula with respect to
138

139
the parameters and try to solve, we’ll find that it is not possible to find the
maximum likelihood estimates of the parameters in closed form. (Try this
yourself at home.)
The random variables z (i) indicate which of the k Gaussians each x(i)had come from. Note that if we knew what the z (i) ’s were, the maximum
likelihood problem would have been easy. Specifically, we could then write
down the likelihood as`(φ, µ, Σ) =n
Xlog p(x(i) |z (i) ; µ, Σ) + log p(z (i) ; φ).

i=1Maximizing this with respect to φ, µ and Σ gives the parameters:
n

φj
µj
Σj

1X
1{z (i) = j},
=
n i=1
Pn
(i)
= j}x(i)i=1 1{zP,=n(i) = j}i=1 1{zPn(i)= j}(x(i) − µj )(x(i) − µj )T
i=1 1{z P=.
n(i) = j}i=1 1{zIndeed, we see that if the z (i) ’s were known, then maximum likelihood
estimation becomes nearly identical to what we had when estimating theparameters of the Gaussian discriminant analysis model, except that here
the z (i) ’s playing the role of the class labels.1
However, in our density estimation problem, the z (i) ’s are not known.
What can we do?
The EM algorithm is an iterative algorithm that has two main steps.
Applied to our problem, in the E-step, it tries to “guess” the values of thez (i) ’s. In the M-step, it updates the parameters of our model based on ourguesses. Since in the M-step we are pretending that the guesses in the firstpart were correct, the maximization becomes easy. Here’s the algorithm:
Repeat until convergence: {(E-step) For each i, j, set(i)wj := p(z (i) = j|x(i) ; φ, µ, Σ)
1There are other minor differences in the formulas here from what we’d obtained in
PS1 with Gaussian discriminant analysis, first because we’ve generalized the z (i) ’s to be
multinomial rather than Bernoulli, and second because here we are using a different Σj
for each Gaussian.

140(M-step) Update the parameters:
n

φj
µj
Σj

1 X (i):=w ,n i=1 jPn(i) (i)i=1 wj x
,
:=
Pn
(i)
i=1 wjPn(i) (i)− µj )(x(i) − µj )T
i=1 wj (x:=Pn(i)
i=1 wj}In the E-step, we calculate the posterior probability of our parameters
the z (i) ’s, given the x(i) and using the current setting of our parameters. I.e.,
using Bayes rule, we obtain:p(x(i) |z (i) = j; µ, Σ)p(z (i) = j; φ)p(z (i) = j|x(i) ; φ, µ, Σ) = Pk(i) (i) = l; µ, Σ)p(z (i) = l; φ)l=1 p(x |zHere, p(x(i) |z (i) = j; µ, Σ) is given by evaluating the density of a Gaussianwith mean µj and covariance Σj at x(i) ; p(z (i) = j; φ) is given by φj , and so(i)on. The values wj calculated in the E-step represent our “soft” guesses2 forthe values of z (i) .
Also, you should contrast the updates in the M-step with the formulas we
had when the z (i) ’s were known exactly. They are identical, except that instead of the indicator functions “1{z (i) = j}” indicating from which Gaussian(i)each datapoint had come, we now instead have the wj ’s.
The EM-algorithm is also reminiscent of the K-means clustering algorithm, except that instead of the “hard” cluster assignments c(i), we instead(i)have the “soft” assignments wj . Similar to K-means, it is also susceptible
to local optima, so reinitializing at several different initial parameters may
be a good idea.
It’s clear that the EM algorithm has a very natural interpretation of
repeatedly trying to guess the unknown z (i) ’s; but how did it come about,and can we make any guarantees about it, such as regarding its convergence?
In the next set of notes, we will describe a more general view of EM, one
2

The term “soft” refers to our guesses being probabilities and taking values in [0, 1]; in
contrast, a “hard” guess is one that represents a single best guess (such as taking values
in {0, 1} or {1, . . . , k}).

141that will allow us to easily apply it to other estimation problems in which
there are also latent variables, and which will allow us to give a convergenceguarantee.

11.2

Jensen’s inequality

We begin our discussion with a very useful result called Jensen’s inequality
Let f be a function whose domain is the set of real numbers. Recall thatf is a convex function if f 00 (x) ≥ 0 (for all x ∈ R). In the case of f taking
vector-valued inputs, this is generalized to the condition that its hessian H
is positive semi-definite (H ≥ 0). If f 00 (x) > 0 for all x, then we say f isstrictly convex (in the vector-valued case, the corresponding statement isthat H must be positive definite, written H > 0). Jensen’s inequality canthen be stated as follows:
Theorem. Let f be a convex function, and let X be a random variable.
Then:
E[f (X)] ≥ f (EX).
Moreover, if f is strictly convex, then E[f (X)] = f (EX) holds true if andonly if X = E[X] with probability 1 (i.e., if X is a constant).
Recall our convention of occasionally dropping the parentheses when writing expectations, so in the theorem above, f (EX) = f (E[X]).
For an interpretation of the theorem, consider the figure below.

f(a)

f

E[f(X)]
f(b)
f(EX)
a

E[X]

b

Here, f is a convex function shown by the solid line. Also, X is a randomvariable that has a 0.5 chance of taking the value a, and a 0.5 chance of

142taking the value b (indicated on the x-axis). Thus, the expected value of X
is given by the midpoint between a and b.
We also see the values f (a), f (b) and f (E[X]) indicated on the y-axis.
Moreover, the value E[f (X)] is now the midpoint on the y-axis between f (a)and f (b). From our example, we see that because f is convex, it must be thecase that E[f (X)] ≥ f (EX).
Incidentally, quite a lot of people have trouble remembering which way
the inequality goes, and remembering a picture like this is a good way toquickly figure out the answer.
Remark. Recall that f is [strictly] concave if and only if −f is [strictly]
convex (i.e., f 00 (x) ≤ 0 or H ≤ 0). Jensen’s inequality also holds for concave
functions f , but with the direction of all the inequalities reversed (E[f (X)] ≤f (EX), etc.).

11.3

General EM algorithms

Suppose we have an estimation problem in which we have a training set{x(1) , . . . , x(n) } consisting of n independent examples. We have a latent variable model p(x, z; θ) with z being the latent variable (which for simplicity is
assumed to take finite number of values). The density for x can be obtainedby marginalized over the latent variable z:Xp(x; θ) =p(x, z; θ)(11.1)z

We wish to fit the parameters θ by maximizing the log-likelihood of thedata, defined by`(θ) =nX

log p(x(i) ; θ)(11.2)

i=1

We can rewrite the objective in terms of the joint density p(x, z; θ) by`(θ) =
=

n
X
i=1
n
X
i=1

log p(x(i) ; θ)(11.3)

X(11.4)log

p(x(i) , z (i) ; θ).

z (i)But, explicitly finding the maximum likelihood estimates of the parameters
θ may be hard since it will result in difficult non-convex optimization prob-

143
lems.3 Here, the z (i) ’s are the latent random variables; and it is often the casethat if the z (i) ’s were observed, then maximum likelihood estimation would
be easy.
In such a setting, the EM algorithm gives an efficient method for maximum likelihood estimation. Maximizing `(θ) explicitly might be difficult,and our strategy will be to instead repeatedly construct a lower-bound on `(E-step), and then optimize that lower-bound(M-step).4
Pn
It turns out that the summation i=1 is not essential here, and towards a
simpler exposition of the EM algorithm, we will first consider optimizing thethe likelihood log p(x) for a single example x. After we derive the algorithm
for optimizing log p(x), we will convert it to an algorithm that works for n
examples by adding back the sum to each of the relevant equations. Thus,
now we aim to optimize log p(x; θ) which can be rewritten asX
log p(x; θ) = log
p(x, z; θ)(11.5)z

Let Q be a distribution over the possible values of z. That is,Q(z) ≥ 0).
Consider the following:5
X
p(x, z; θ)log p(x; θ) = log

Pz Q(z) = 1,

z

= log

Q(z)

p(x, z; θ)Q(z)(11.6)

Q(z) logp(x, z; θ)Q(z)(11.7)

X
z

≥

X
z

The last step of this derivation used Jensen’s inequality. Specifically,f (x) = log x is a concave function, since f 00 (x) = −1/x2 < 0 over its domain
3

It’s mostly an empirical observation that the optimization problem is difficult to optimize.
4
Empirically, the E-step and M-step can often be computed more efficiently than optimizing the function `(·) directly. However, it doesn’t necessarily mean that alternating
the two steps can always converge to the global optimum of `(·). Even for mixture ofGaussians, the EM algorithm can either converge to a global optimum or get stuck, depending on the properties of the training data. Empirically, for real-world data, often EM
can converge to a solution with relatively high likelihood (if not the optimum), and thetheory behind it is still largely not understood.
5
If z were continuous, then Q would be a density, and the summations over z in our
discussion are replaced with integrals over z.

144x ∈ R+ . Also, the termXz

p(x, z; θ)
Q(z)
Q(z)



in the summation is just an expectation of the quantity [p(x, z; θ)/Q(z)] withrespect to z drawn according to the distribution given by Q.6 By Jensen’s
inequality, we have

 



p(x, z; θ)p(x, z; θ)≥ Ez∼Q f,
f Ez∼Q
Q(z)Q(z)
where the “z ∼ Q” subscripts above indicate that the expectations are withrespect to z drawn from Q. This allowed us to go from Equation (11.6) toEquation (11.7).
Now, for any distribution Q, the formula (11.7) gives a lower-bound onlog p(x; θ). There are many possible choices for the Q’s. Which should wechoose? Well, if we have some current guess θ of the parameters, it seems
natural to try to make the lower-bound tight at that value of θ. I.e., we will
make the inequality above hold with equality at our particular value of θ.
To make the bound tight for a particular value of θ, we need for the step
involving Jensen’s inequality in our derivation above to hold with equality.
For this to be true, we know it is sufficient that the expectation be takenover a “constant”-valued random variable. I.e., we require thatp(x, z; θ)=cQ(z)for some constant c that does not depend on z. This is easily accomplishedby choosing
Q(z) ∝ p(x, z; θ).
P
Actually, since we knowz Q(z) = 1 (because it is a distribution), thisfurther tells us thatp(x, z; θ)Q(z) = Pz p(x, z; θ)p(x, z; θ)=p(x; θ)= p(z|x; θ)(11.8)

We note that the notion p(x,z;θ)Q(z) only makes sense if Q(z) 6= 0 whenever p(x, z; θ) 6= 0.
Here we implicitly assume that we only consider those Q with such a property.
6145
Thus, we simply set the Q’s to be the posterior distribution of the z’s given
x and the setting of the parameters θ.
Indeed, we can directly verify that when Q(z) = p(z|x; θ), then equation (11.7) is an equality becauseX
p(x, z; θ)p(x, z; θ) X=
p(z|x; θ) log
Q(z) logQ(z)p(z|x; θ)zzXp(z|x; θ)p(x; θ)=p(z|x; θ) log
p(z|x; θ)zX=p(z|x; θ) log p(x; θ)z= log p(x; θ)Xp(z|x; θ)z= log p(x; θ)(becausePz p(z|x; θ) = 1)

For convenience, we call the expression in Equation (11.7) the evidence
lower bound (ELBO) and we denote it byX
p(x, z; θ)Q(z) log
ELBO(x; Q, θ) =(11.9)Q(z)zWith this equation, we can re-write equation (11.7) as∀Q, θ, x,log p(x; θ) ≥ ELBO(x; Q, θ)(11.10)Intuitively, the EM algorithm alternatively updates Q and θ by a) setting Q(z) = p(z|x; θ) following Equation (11.8) so that ELBO(x; Q, θ) =log p(x; θ) for x and the current θ, and b) maximizing ELBO(x; Q, θ) w.r.t θ
while fixing the choice of Q.
Recall that all the discussion above was under the assumption that we
aim to optimize the log-likelihood log p(x; θ) for a single example x. It turns
out that with multiple training examples, the basic idea is the same and weonly needs to take a sum over examples at relevant places. Next, we willbuild the evidence lower bound for multiple training examples and make the
EM algorithm formal.
Recall we have a training set {x(1) , . . . , x(n) }. Note that the optimal choiceof Q is p(z|x; θ), and it depends on the particular example x. Therefore here
we will introduce n distributions Q1 , . . . , Qn , one for each example x(i) . For
each example x(i) , we can build the evidence lower boundX
p(x(i) , z (i) ; θ)log p(x(i) ; θ) ≥ ELBO(x(i) ; Qi , θ) =Qi (z (i) ) log
Qi (z (i) )
(i)
z

146
Taking sum over all the examples, we obtain a lower bound for the loglikelihoodX`(θ) ≥ELBO(x(i) ; Qi , θ)(11.11)i=

XXi

Qi (z (i) ) logz (i)p(x(i) , z (i) ; θ)Qi (z (i) )For any set of distributions Q1 , . . . , Qn , the formula (11.11) gives a lowerbound on `(θ), and analogous to the argument around equation (11.8), theQi that attains equality satisfies
Qi (z (i) ) = p(z (i) |x(i) ; θ)Thus, we simply set the Qi ’s to be the posterior distribution of the z (i) ’s
given x(i) with the current setting of the parameters θ.
Now, for this choice of the Qi ’s, Equation (11.11) gives a lower-bound on
the loglikelihood ` that we’re trying to maximize. This is the E-step. In the
M-step of the algorithm, we then maximize our formula in Equation (11.11)
with respect to the parameters to obtain a new setting of the θ’s. Repeatedly
carrying out these two steps gives us the EM algorithm, which is as follows:
Repeat until convergence {(E-step) For each i, set
Qi (z (i) ) := p(z (i) |x(i) ; θ).
(M-step) Set
θ := arg max
θ= arg maxθnX

ELBO(x(i) ; Qi , θ)

i=1

XX
i

z (i)

Qi (z (i) ) log

p(x(i) , z (i) ; θ).
Qi (z (i) )(11.12)}How do we know if this algorithm will converge? Well, suppose θ(t) andθ(t+1) are the parameters from two successive iterations of EM. We will now
prove that `(θ(t) ) ≤ `(θ(t+1) ), which shows EM always monotonically improves the log-likelihood. The key to showing this result lies in our choice of

147
the Qi ’s. Specifically, on the iteration of EM in which the parameters had(t)started out as θ(t) , we would have chosen Qi (z (i) ) := p(z (i) |x(i) ; θ(t) ). We
saw earlier that this choice ensures that Jensen’s inequality, as applied to get
Equation (11.11), holds with equality, and hence(t)`(θ ) =nX(t)

ELBO(x(i) ; Qi , θ(t) )(11.13)i=1

The parameters θ(t+1) are then obtained by maximizing the right hand sideof the equation above. Thus,
`(θ

(t+1)

)≥

n
X

(t)

ELBO(x(i) ; Qi , θ(t+1) )i=1(because ineqaulity (11.11) holds for all Q and θ)
≥

n
X

(t)

ELBO(x(i) ; Qi , θ(t) )(see reason below)

i=1
(t)

= `(θ )

(by equation (11.13))

where the last inequality follows from that θ(t+1) is chosen explicitly to bearg max
θ

n
X

(t)

ELBO(x(i) ; Qi , θ)i=1Hence, EM causes the likelihood to converge monotonically. In our description of the EM algorithm, we said we’d run it until convergence. Given
the result that we just showed, one reasonable convergence test would be
to check if the increase in `(θ) between successive iterations is smaller thansome tolerance parameter, and to declare convergence if EM is improving`(θ) too slowly.
Remark. If we define (by overloading ELBO(·))
ELBO(Q, θ) =

n
X
i=1

ELBO(x(i) ; Qi , θ) =

XX
i

z (i)

Qi (z (i) ) log

p(x(i) , z (i) ; θ)Qi (z (i) )(11.14)then we know `(θ) ≥ ELBO(Q, θ) from our previous derivation. The EM
can also be viewed an alternating maximization algorithm on ELBO(Q, θ),in which the E-step maximizes it with respect to Q (check this yourself), andthe M-step maximizes it with respect to θ.

148

11.3.1Other interpretation of ELBOP
p(x,z;θ)Let ELBO(x; Q, θ) =be defined as in equation (11.9).
z Q(z) log Q(z)There are several other forms of ELBO. First, we can rewrite
ELBO(x; Q, θ) = Ez∼Q [log p(x, z; θ)] − Ez∼Q [log Q(z)]= Ez∼Q [log p(x|z; θ)] − DKL (Qkpz )(11.15)where we use pz to denote the marginal distribution of z (under the distribution p(x, z; θ)), and DKL () denotes the KL divergence
DKL (Qkpz ) =XzQ(z) log

Q(z)
p(z)

(11.16)In many cases, the marginal distribution of z does not depend on the parameter θ. In this case, we can see that maximizing ELBO over θ is equivalent
to maximizing the first term in (11.15). This corresponds to maximizing theconditional likelihood of x conditioned on z, which is often a simpler question
than the original question.
Another form of ELBO(·) is (please verify yourself)
ELBO(x; Q, θ) = log p(x) − DKL (Qkpz|x )(11.17)

where pz|x is the conditional distribution of z given x under the parameter
θ. This forms shows that the maximizer of ELBO(Q, θ) over Q is obtained
when Q = pz|x , which was shown in equation (11.8) before.

11.4

Mixture of Gaussians revisited

Armed with our general definition of the EM algorithm, let’s go back to ourold example of fitting the parameters φ, µ and Σ in a mixture of Gaussians.
For the sake of brevity, we carry out the derivations for the M-step updates
only for φ and µj , and leave the updates for Σj as an exercise for the reader.
The E-step is easy. Following our algorithm derivation above, we simply
calculate(i)wj = Qi (z (i) = j) = P (z (i) = j|x(i) ; φ, µ, Σ).
Here, “Qi (z (i) = j)” denotes the probability of z (i) taking the value j under
the distribution Qi .

149
Next, in the M-step, we need to maximize, with respect to our parameters
φ, µ, Σ, the quantityn X
X

Qi (z (i) ) log

i=1 z (i)

=

n X
k
X

p(x(i) , z (i) ; φ, µ, Σ)
Qi (z (i) )Qi (z (i) = j) log

i=1 j=1

=

n X
k
X

p(x(i) |z (i) = j; µ, Σ)p(z (i) = j; φ)Qi (z (i) = j)1(i)wj log(2π)d/2 |Σj |1/2(i)
− µj ) · φj
exp − 12 (x(i) − µj )T Σ−1
j (x
(i)

wj

i=1 j=1

Let’s maximize this with respect to µl . If we take the derivative with respect
to µl , we find

1
1
(i)
T −1 (i)
n X
k
(x
−
µ
)
Σ
(x
−
µ
)
· φj
exp
−
X
j
j
d/2
1/2
j
2
(2π) |Σj |(i)∇µl
wj log(i)wj
i=1 j=1= −∇µln X
kX
i=1 j=1

=
=

(i) 1

wj

2

(i)
(x(i) − µj )T Σ−1− µj )
j (x

n
X

1
(i)
(i)
w ∇µl 2µTl Σ−1− µTl Σ−1l xl µl
2 i=1 l
n
X

(i)

wl

(i)
Σ−1
− Σ−1
l x
l µl



i=1

Setting this to zero and solving for µl therefore yields the update rule(i) (i)i=1 wl x,
Pn(i)
i=1 wlPnµl :=which was what we had in the previous set of notes.
Let’s do one more example, and derive the M-step update for the parameters φj . Grouping together only the terms that depend on φj , we find that
we need to maximize
n X
k
X
(i)
wj log φj .
i=1 j=1

However, there is an additional constraint that the φj ’s sum to 1, since they
represent the probabilities φj = p(z (i) = j; φ). To deal with the constraint

150
that

Pk

j=1 φj = 1, we construct the LagrangianL(φ) =

n X
k
X

k
X

(i)
wj log φj + β(i=1 j=1

φj − 1),j=1

where β is the Lagrange multiplier.7 Taking derivatives, we find
(i)
n
X
wj
∂
L(φ) =
+β
∂φj
φ
j
i=1

Setting this to zero and solving, we get
Pn
φj =

(i)
i=1 wj−βP(i)
j φj = 1, we easily findi=1 wj . Using the constraint that
Pn Pk
Pn
(i)
(i)
that −β = i=1 j=1 wj = i=1 1 = n. (This used the fact that wj =P (i)Qi (z (i) = j), and since probabilities sum to 1, j wj = 1.) We therefore

I.e., φj ∝Pnhave our M-step updates for the parameters φj :
n

φj :=

1 X (i)w .
n i=1 jThe derivation for the M-step updates to Σj are also entirely straightforward.

11.5

Variational inference and variational
auto-encoder (optional reading)

Loosely speaking, variational auto-encoder Kingma and Welling [2013] generally refers to a family of algorithms that extend the EM algorithms to more
complex models parameterized by neural networks. It extends the techniqueof variational inference with the additional “re-parametrization trick” which
will be introduced below. Variational auto-encoder may not give the best
performance for many datasets, but it contains several central ideas about
how to extend EM algorithms to high-dimensional continuous latent variables
7

We don’t need to worry about the constraint that φj ≥ 0, because as we’ll shortly see,the solution we’ll find from this derivation will automatically satisfy that anyway.

151
with non-linear models. Understanding it will likely give you the language
and backgrounds to understand various recent papers related to it.
As a running example, we will consider the following parameterization ofp(x, z; θ) by a neural network. Let θ be the collection of the weights of aneural network g(z; θ) that maps z ∈ Rk to Rd . Letz ∼ N (0, Ik×k )x|z ∼ N (g(z; θ), σ 2 Id×d )(11.18)(11.19)Here Ik×k denotes identity matrix of dimension k by k, and σ is a scalar that
we assume to be known for simplicity.
For the Gaussian mixture models in Section 11.4, the optimal choice ofQ(z) = p(z|x; θ) for each fixed θ, that is the posterior distribution of z,
can be analytically computed. In many more complex models such as themodel (11.19), it’s intractable to compute the exact the posterior distributionp(z|x; θ).
Recall that from equation (11.10), ELBO is always a lower bound for any
choice of Q, and therefore, we can also aim for finding an approximation of
the true posterior distribution. Often, one has to use some particular form
to approximate the true posterior distribution. Let Q be a family of Q’s that
we are considering, and we will aim to find a Q within the family of Q that isclosest to the true posterior distribution. To formalize, recall the definition of
the ELBO lower bound as a function of Q and θ defined in equation (11.14)
ELBO(Q, θ) =n
X

ELBO(x(i) ; Qi , θ) =

i=1

XX
i

z (i)

Qi (z (i) ) log

p(x(i) , z (i) ; θ)Qi (z (i) )Recall that EM can be viewed as alternating maximization of
ELBO(Q, θ). Here instead, we optimize the EBLO over Q ∈ Q
max max ELBO(Q, θ)Q∈Qθ

(11.20)

Now the next question is what form of Q (or what structural assumptions
to make about Q) allows us to efficiently maximize the objective above. Whenthe latent variable z are high-dimensional discrete variables, one popular assumption is the mean field assumption, which assumes that Qi (z) gives a
distribution with independent coordinates, or in other words, Qi can be decomposed into Qi (z) = Q1i (z1 ) · · · Qki (zk ). There are tremendous applicationsof mean field assumptions to learning generative models with discrete latentvariables, and we refer to Blei et al. [2017] for a survey of these models and152
their impact to a wide range of applications including computational biology,
computational neuroscience, social sciences. We will not get into the details
about the discrete latent variable cases, and our main focus is to deal with
continuous latent variables, which requires not only mean field assumptions,but additional techniques.
When z ∈ Rk is a continuous latent variable, there are several decisions tomake towards successfully optimizing (11.20). First we need to give a succinct
representation of the distribution Qi because it is over an infinite number ofpoints. A natural choice is to assume Qi is a Gaussian distribution with somemean and variance. We would also like to have more succinct representationof the means of Qi of all the examples. Note that Qi (z (i) ) is supposed toapproximate p(z (i) |x(i) ; θ). It would make sense let all the means of the Qi ’s
be some function of x(i) . Concretely, let q(·; φ), v(·; φ) be two functions thatmap from dimension d to k, which are parameterized by φ and ψ, we assume
that
Qi = N (q(x(i) ; φ), diag(v(x(i) ; ψ))2 )(11.21)

Here diag(w) means the k × k matrix with the entries of w ∈ Rk on the
diagonal. In other words, the distribution Qi is assumed to be a Gaussian
distribution with independent coordinates, and the mean and standard deviations are governed by q and v. Often in variational auto-encoder, q and v
are chosen to be neural networks.8 In recent deep learning literature, oftenq, v are called encoder (in the sense of encoding the data into latent code),
whereas g(z; θ) if often referred to as the decoder.
We remark that Qi of such form in many cases are very far from a good approximation of the true posterior distribution. However, some approximationis necessary for feasible optimization. In fact, the form of Qi needs to satisfy
other requirements (which happened to be satisfied by the form (11.21))
Before optimizing the ELBO, let’s first verify whether we can efficiently
evaluate the value of the ELBO for fixed Q of the form (11.21) and θ. We
rewrite the ELBO as a function of φ, ψ, θ bynXp(x(i) , z (i) ; θ),(11.22)
ELBO(φ, ψ, θ) =Ez(i) ∼Qi log
Qi (z (i) )i=1where Qi = N (q(x(i) ; φ), diag(v(x(i) ; ψ))2 )Note that to evaluate Qi (z (i) ) inside the expectation, we should be able tocompute the density of Qi . To estimate the expectation Ez(i) ∼Qi , we8q and v can also share parameters. We sweep this level of details under the rug in this
note.

153
should be able to sample from distribution Qi so that we can build an
empirical estimator with samples. It happens that for Gaussian distribution
Qi = N (q(x(i) ; φ), diag(v(x(i) ; ψ))2 ), we are able to be both efficiently.
Now let’s optimize the ELBO. It turns out that we can run gradient ascentover φ, ψ, θ instead of alternating maximization. There is no strong need tocompute the maximum over each variable at a much greater cost. (For Gaussian mixture model in Section 11.4, computing the maximum is analyticallyfeasible and relatively cheap, and therefore we did alternating maximization.)
Mathematically, let η be the learning rate, the gradient ascent step isθ := θ + η∇θ ELBO(φ, ψ, θ)
φ := φ + η∇φ ELBO(φ, ψ, θ)ψ := ψ + η∇ψ ELBO(φ, ψ, θ)Computing the gradient over θ is simple because∇θ ELBO(φ, ψ, θ) = ∇θ

n
X
i=1

= ∇θ

n
X


Ez(i) ∼Qi

p(x(i) , z (i) ; θ)log
Qi (z (i) )Ez(i) ∼Qi log p(x(i) , z (i) ; θ)

i=1

=

n
X



Ez(i) ∼Qi ∇θ log p(x(i) , z (i) ; θ) ,(11.23)i=1

But computing the gradient over φ and ψ is tricky because the sampling distribution Qi depends on φ and ψ. (Abstractly speaking, the issue we face can be simplified as the problem of computing the gradient Ez∼Qφ [f (φ)] with respect to variable φ. We know that in general,∇Ez∼Qφ [f (φ)] 6= Ez∼Qφ [∇f (φ)] because the dependency of Qφ on φ has to betaken into account as well. )The idea that comes to rescue is the so-called re-parameterization
trick: we rewrite z (i) ∼ Qi = N (q(x(i) ; φ), diag(v(x(i) ; ψ))2 ) in an equivalentway:z (i) = q(x(i) ; φ) + v(x(i) ; ψ)ξ (i) where ξ (i) ∼ N (0, Ik×k )(11.24)

Here x y denotes the entry-wise product of two vectors of the same
dimension. Here we used the fact that x ∼ N (µ, σ 2 ) is equivalent to thatx = µ+ξσ with ξ ∼ N (0, 1). We mostly just used this fact in every dimension
simultaneously for the random variable z (i) ∼ Qi .

154With this re-parameterization, we have thatp(x(i) , z (i) ; θ)Ez(i) ∼Qi log
Qi (z (i) )p(x(i) , q(x(i) ; φ) + v(x(i) ; ψ)
= Eξ(i) ∼N (0,1) log
Qi (q(x(i) ; φ) + v(x(i) ; ψ)(11.25)ξ (i) ; θ)ξ (i) )

It follows thatp(x(i) , z (i) ; θ)∇φ Ez(i) ∼Qi log
Qi (z (i) )p(x(i) , q(x(i) ; φ) + v(x(i) ; ψ) ξ (i) ; θ)= ∇φ Eξ(i) ∼N (0,1) log
Qi (q(x(i) ; φ) + v(x(i) ; ψ) ξ (i) )p(x(i) , q(x(i) ; φ) + v(x(i) ; ψ) ξ (i) ; θ)
= Eξ(i) ∼N (0,1) ∇φ log
Qi (q(x(i) ; φ) + v(x(i) ; ψ) ξ (i) )We can now sample multiple copies of ξ (i) ’s to estimate the the expectation in the RHS of the equation above.9 We can estimate the gradient withrespect to ψ similarly, and with these, we can implement the gradient ascentalgorithm to optimize the ELBO over φ, ψ, θ.
There are not many high-dimensional distributions with analytically computable density function are known to be re-parameterizable. We refer toKingma and Welling [2013] for a few other choices that can replace Gaussian
distribution.

9

Empirically people sometimes just use one sample to estimate it for maximum computational efficiency.

Chapter 12
Principal components analysis
In this set of notes, we will develop a method, Principal Components Analysis(PCA), that tries to identify the subspace in which the data approximately
lies. PCA is computationally efficient: it will require only an eigenvector
calculation (easily done with the eig function in Matlab).
Suppose we are given a dataset {x(i) ; i = 1, . . . , n} of attributes of n different types of automobiles, such as their maximum speed, turn radius, andso on. Let x(i) ∈ Rd for each i (d  n). But unknown to us, two different
attributes—some xi and xj —respectively give a car’s maximum speed measured in miles per hour, and the maximum speed measured in kilometers perhour. These two attributes are therefore almost linearly dependent, up to
only small differences introduced by rounding off to the nearest mph or kph.
Thus, the data really lies approximately on an n − 1 dimensional subspace.
How can we automatically detect, and perhaps remove, this redundancy?
For a less contrived example, consider a dataset resulting from a survey of(i)pilots for radio-controlled helicopters, where x1 is a measure of the piloting(i)skill of pilot i, and x2 captures how much he/she enjoys flying. Because
RC helicopters are very difficult to fly, only the most committed students,
ones that truly enjoy flying, become good pilots. So, the two attributes
x1 and x2 are strongly correlated. Indeed, we might posit that that thedata actually likes along some diagonal axis (the u1 direction) capturing the
intrinsic piloting “karma” of a person, with only a small amount of noise
lying off this axis. (See figure.) How can we automatically compute this u1
direction?

155

156

x2 (enjoyment)u1u2

x1 (skill)
We will shortly develop the PCA algorithm. But prior to running PCA
per se, typically we first preprocess the data by normalizing each feature
to have mean 0 and variance 1. We do this by subtracting the mean anddividing by the empirical standard deviation:(i)xj − µ j
(i)
xj ←
σj

P
P
(i)
(i)
where µj = n1 ni=1 xj and σj2 = n1 ni=1 (xj − µj )2 are the mean variance offeature j, respectively.
Subtracting µj zeros out the mean and may be omitted for data known
to have zero mean (for instance, time series corresponding to speech or otheracoustic signals). Dividing by the standard deviation σj rescales each coordinate to have unit variance, which ensures that different attributes are alltreated on the same “scale.” For instance, if x1 was cars’ maximum speed in
mph (taking values in the high tens or low hundreds) and x2 were the number of seats (taking values around 2-4), then this renormalization rescales
the different attributes to make them more comparable. This rescaling may
be omitted if we had a priori knowledge that the different attributes are allon the same scale. One example of this is if each data point represented a(i)grayscale image, and each xj took a value in {0, 1, . . . , 255} corresponding
to the intensity value of pixel j in image i.
Now, having normalized our data, how do we compute the “major axisof variation” u—that is, the direction on which the data approximately lies?
One way is to pose this problem as finding the unit vector u so that when

157
the data is projected onto the direction corresponding to u, the variance ofthe projected data is maximized. Intuitively, the data starts off with someamount of variance/information in it. We would like to choose a direction u
so that if we were to approximate the data as lying in the direction/subspacecorresponding to u, as much as possible of this variance is still retained.
Consider the following dataset, on which we have already carried out thenormalization steps:

Now, suppose we pick u to correspond the the direction shown in the
figure below. The circles denote the projections of the original data onto this
line.

158

00
11
00
11
11
00
00
11

11
00
00
11
1
0

11
00
00
11
00
11
00
11
000
111
00
11
00
11
1
0
000
111
000
111
000
111
000
111
000
111
000
111

1
0
00
11
0
1
00
11
00
11
00
11
00
11
00
11
00
11

We see that the projected data still has a fairly large variance, and thepoints tend to be far from zero. In contrast, suppose had instead picked thefollowing direction:

11111111
00000000
00000000
11111111
00000000
11111111
00000000
11111111
00000000
11111111
00000000
11111111
11111111
00000000
00000
11111
11111111
00000000
00000
11111
00000000
11111111
00000000
11111111
00000
11111
11111111
00000000
11111111
00000000
00000
11111
00000000
11111111
11111111
00000000
00000
11111
11111111
00000000
00000000
11111111
00000
11111
00000000
11111111
00000000
11111111
11111
00000
00000000
11111111
11111111
00000000
11
00
00
11
00000
11111
00000000
111111111
011111111
00000000
00000000000
11111111111
00
1100000000
11111111
00000000000
11111111111
00000000
11111111
00000000000
11111111111
00000000
11111111
11111111111
00000000000
00000000
11111111
11111111111
00000000000
00000000
11111111
00000000000
11111111111
0
100000000
00000000000
11111111111
000000111111011111111
100000000
11111111
00000000000
11111111111
00000011111100
1111111111
00000000
00000000000
11111111111
00000011111100
1100000000000
11111111111
00000011111100000000000
11111111111
11111100000000000000000
11111111111
00000011111100000000000
11111111111
00000011111100000000000
11111111111
00000011111111111111111
00000000000
00000011111111111111111
00000000000
00000011111100000000000
11111111111
000000111111

Here, the projections have a significantly smaller variance, and are muchcloser to the origin.
We would like to automatically select the direction u corresponding to
the first of the two figures shown above. To formalize this, note that given a

159
unit vector u and a point x, the length of the projection of x onto u is given
by xT u. I.e., if x(i) is a point in our dataset (one of the crosses in the plot),then its projection onto u (the corresponding circle in the figure) is distancexT u from the origin. Hence, to maximize the variance of the projections, wewould like to choose a unit-length u so as to maximize:nn
1 X (i) T 2
1 X T (i) (i) T(x u) =
u x x u
n i=1
n i=1
n

= uT

1 X (i) (i) T
x x
n i=1

!
u.

We easily recognize that the maximizing this subject to kuk2 = 1 gives thePTprincipal eigenvector of Σ = n1 ni=1 x(i) x(i) , which is just the empiricalcovariance matrix of the data (assuming it has zero mean).1
To summarize, we have found that if we wish to find a 1-dimensional
subspace with with to approximate the data, we should choose u to be theprincipal eigenvector of Σ. More generally, if we wish to project our data
into a k-dimensional subspace (k < d), we should choose u1 , . . . , uk to be the
top k eigenvectors of Σ. The ui ’s now form a new, orthogonal basis for thedata.2
Then, to represent x(i) in this basis, we need only compute the corresponding vector
uT1 x(i)uT x(i) 

 2
(i)
y =  ..  ∈ Rk .
 . 
uTk x(i)Thus, whereas x(i) ∈ Rd , the vector y (i) now gives a lower, k-dimensional,
approximation/representation for x(i) . PCA is therefore also referred to asa dimensionality reduction algorithm. The vectors u1 , . . . , uk are called
the first k principal components of the data.
Remark. Although we have shown it formally only for the case of k = 1,
using well-known properties of eigenvectors it is straightforward to show that1

If you haven’t seen this before, try using the method of Lagrange multipliers to maximize uT Σu subject to that uT u = 1. You should be able to show that Σu = λu, for someλ, which implies u is an eigenvector of Σ, with eigenvalue λ.
2Because Σ is symmetric, the ui ’s will (or always can be chosen to be) orthogonal toeach other.

160
of all possible
bases u1 , . . . , uk , the one that we have chosen maxP (i)orthogonal
2
imizes i ky k2 . Thus, our choice of a basis preserves as much variability
as possible in the original data.
PCA can also be derived by picking the basis that minimizes the approximation error arising from projecting the data onto the k-dimensional
subspace spanned by them. (See more in homework.)
PCA has many applications; we will close our discussion with a few examples. First, compression—representing x(i) ’s with lower dimension y (i) ’s—isan obvious application. If we reduce high dimensional data to k = 2 or 3 dimensions, then we can also plot the y (i) ’s to visualize the data. For instance,
if we were to reduce our automobiles data to 2 dimensions, then we can plotit (one point in our plot would correspond to one car type, say) to see whatcars are similar to each other and what groups of cars may cluster together.
Another standard application is to preprocess a dataset to reduce its
dimension before running a supervised learning learning algorithm with thex(i) ’s as inputs. Apart from computational benefits, reducing the data’s
dimension can also reduce the complexity of the hypothesis class considered
and help avoid overfitting (e.g., linear classifiers over lower dimensional input
spaces will have smaller VC dimension).
Lastly, as in our RC pilot example, we can also view PCA as a noise
reduction algorithm. In our example it, estimates the intrinsic “piloting
karma” from the noisy measures of piloting skill and enjoyment. In class, we
also saw the application of this idea to face images, resulting in eigenfaces
method. Here, each point x(i) ∈ R100×100 was a 10000 dimensional vector,
with each coordinate corresponding to a pixel intensity value in a 100x100
image of a face. Using PCA, we represent each image x(i) with a much lowerdimensional y (i) . In doing so, we hope that the principal components wefound retain the interesting, systematic variations between faces that capture
what a person really looks like, but not the “noise” in the images introducedby minor lighting variations, slightly different imaging conditions, and so on.
We then measure distances between faces i and j by working in the reduced
dimension, and computing ky (i) − y (j) k2 . This resulted in a surprisingly goodface-matching and retrieval algorithm.

Chapter 13
Independent components
analysis
Our next topic is Independent Components Analysis (ICA). Similar to PCA,
this will find a new basis in which to represent our data. However, the goalis very different.
As a motivating example, consider the “cocktail party problem.” Here, dspeakers are speaking simultaneously at a party, and any microphone placedin the room records only an overlapping combination of the d speakers’ voices.
But lets say we have d different microphones placed in the room, and because
each microphone is a different distance from each of the speakers, it records a
different combination of the speakers’ voices. Using these microphone recordings, can we separate out the original d speakers’ speech signals?
To formalize this problem, we imagine that there is some data s ∈ Rd
that is generated via d independent sources. What we observe is
x = As,where A is an unknown square matrix called the mixing matrix. Repeatedobservations gives us a dataset {x(i) ; i = 1, . . . , n}, and our goal is to recover
the sources s(i) that had generated our data (x(i) = As(i) ).
(i)
In our cocktail party problem, s(i) is an d-dimensional vector, and sj is
the sound that speaker j was uttering at time i. Also, x(i) in an d-dimensional(i)vector, and xj is the acoustic reading recorded by microphone j at time i.
Let W = A−1 be the unmixing matrix. Our goal is to find W , sothat given our microphone recordings x(i) , we can recover the sources bycomputing s(i) = W x(i) . For notational convenience, we also let wiT denote

161

162
the i-th row of W , so that
— w1T —


..
W =
.
.
— wdT —(i)

Thus, wi ∈ Rd , and the j-th source can be recovered as sj = wjT x(i) .

13.1

ICA ambiguities

To what degree can W = A−1 be recovered? If we have no prior knowledge
about the sources and the mixing matrix, it is easy to see that there are someinherent ambiguities in A that are impossible to recover, given only the x(i) ’s.
Specifically, let P be any d-by-d permutation matrix. This means thateach row and each column of P has exactly one “1.” Here are some examplesof permutation matrices:






0 1 0
0
1
1
0
P =  1 0 0 ; P =
; P =
.
1 0
0 1
0 0 1
If z is a vector, then P z is another vector that contains a permuted versionof z’s coordinates. Given only the x(i) ’s, there will be no way to distinguishbetween W and P W . Specifically, the permutation of the original sources isambiguous, which should be no surprise. Fortunately, this does not matterfor most applications.
Further, there is no way to recover the correct scaling of the wi ’s. For instance, if A were replaced with 2A, and every s(i) were replaced with (0.5)s(i) ,then our observed x(i) = 2A · (0.5)s(i) would still be the same. More broadly,
if a single column of A were scaled by a factor of α, and the corresponding
source were scaled by a factor of 1/α, then there is again no way to determinethat this had happened given only the x(i) ’s. Thus, we cannot recover the“correct” scaling of the sources. However, for the applications that we areconcerned with—including the cocktail party problem—this ambiguity also(i)does not matter. Specifically, scaling a speaker’s speech signal sj by somepositive factor α affects only the volume of that speaker’s speech. Also, sign(i)(i)
changes do not matter, and sj and −sj sound identical when played on a
speaker. Thus, if the wi found by an algorithm is scaled by any non-zero real
number, the corresponding recovered source si = wiT x will be scaled by the

163same factor; but this usually does not matter. (These comments also apply
to ICA for the brain/MEG data that we talked about in class.)
Are these the only sources of ambiguity in ICA? It turns out that they
are, so long as the sources si are non-Gaussian. To see what the difficulty iswith Gaussian data, consider an example in which n = 2, and s ∼ N (0, I).
Here, I is the 2x2 identity matrix. Note that the contours of the density ofthe standard normal distribution N (0, I) are circles centered on the origin,and the density is rotationally symmetric.
Now, suppose we observe some x = As, where A is our mixing matrix.
Then, the distribution of x will be Gaussian, x ∼ N (0, AAT ), since
Es∼N (0,I) [x] = E[As] = AE[s] = 0
Cov[x] = Es∼N (0,I) [xxT ] = E[AssT AT ] = AE[ssT ]AT = A · Cov[s] · AT = AAT
Now, let R be an arbitrary orthogonal (less formally, a rotation/reflection)
matrix, so that RRT = RT R = I, and let A0 = AR. Then if the data had
been mixed according to A0 instead of A, we would have instead observed
x0 = A0 s. The distribution of x0 is also Gaussian, x0 ∼ N (0, AAT ), since
Es∼N (0,I) [x0 (x0 )T ] = E[A0 ssT (A0 )T ] = E[ARssT (AR)T ] = ARRT AT = AAT .
Hence, whether the mixing matrix is A or A0 , we would observe data froma N (0, AAT ) distribution. Thus, there is no way to tell if the sources weremixed using A and A0 . There is an arbitrary rotational component in the
mixing matrix that cannot be determined from the data, and we cannot
recover the original sources.
Our argument above was based on the fact that the multivariate standard
normal distribution is rotationally symmetric. Despite the bleak picture that
this paints for ICA on Gaussian data, it turns out that, so long as the data is
not Gaussian, it is possible, given enough data, to recover the d independentsources.

13.2

Densities and linear transformations

Before moving on to derive the ICA algorithm proper, we first digress briefly
to talk about the effect of linear transformations on densities.
Suppose a random variable s is drawn according to some density ps (s).
For simplicity, assume for now that s ∈ R is a real number. Now, let the
random variable x be defined according to x = As (here, x ∈ R, A ∈ R). Letpx be the density of x. What is px ?
Let W = A−1 . To calculate the “probability” of a particular value of x,
it is tempting to compute s = W x, then then evaluate ps at that point, and164
conclude that “px (x) = ps (W x).” However, this is incorrect. For example,
let s ∼ Uniform[0, 1], so ps (s) = 1{0 ≤ s ≤ 1}. Now, let A = 2, so x = 2s.
Clearly, x is distributed uniformly in the interval [0, 2]. Thus, its density isgiven by px (x) = (0.5)1{0 ≤ x ≤ 2}. This does not equal ps (W x), whereW = 0.5 = A−1 . Instead, the correct formula is px (x) = ps (W x)|W |.
More generally, if s is a vector-valued distribution with density ps , andx = As for a square, invertible matrix A, then the density of x is given bypx (x) = ps (W x) · |W |,where W = A−1 .
Remark. If you’re seen the result that A maps [0, 1]d to a set of volume |A|,
then here’s another way to remember the formula for px given above, that also
generalizes our previous 1-dimensional example. Specifically, let A ∈ Rd×d begiven, and let W = A−1 as usual. Also let C1 = [0, 1]d be the d-dimensional
hypercube, and define C2 = {As : s ∈ C1 } ⊆ Rd to be the image of C1
under the mapping given by A. Then it is a standard result in linear algebra(and, indeed, one of the ways of defining determinants) that the volume of
C2 is given by |A|. Now, suppose s is uniformly distributed in [0, 1]d , so its
density is ps (s) = 1{s ∈ C1 }. Then clearly x will be uniformly distributedin C2 . Its density is therefore found to be px (x) = 1{x ∈ C2 }/vol(C2 ) (since
it must integrate over C2 to 1). But using the fact that the determinantof the inverse of a matrix is just the inverse of the determinant, we have
1/vol(C2 ) = 1/|A| = |A−1 | = |W |. Thus, px (x) = 1{x ∈ C2 }|W | = 1{W x ∈C1 }|W | = ps (W x)|W |.

13.3

ICA algorithm

We are now ready to derive an ICA algorithm. We describe an algorithm
by Bell and Sejnowski, and we give an interpretation of their algorithm as amethod for maximum likelihood estimation. (This is different from their original interpretation involving a complicated idea called the infomax principal
which is no longer necessary given the modern understanding of ICA.)
We suppose that the distribution of each source sj is given by a density
ps , and that the joint distribution of the sources s is given byp(s) =dY
j=1

ps (sj ).

165
Note that by modeling the joint distribution as a product of marginals, we
capture the assumption that the sources are independent. Using our formulasfrom the previous section, this implies the following density on x = As =W −1 s:dYp(x) =
ps (wjT x) · |W |.
j=1

All that remains is to specify a density for the individual sources ps .
Recall that, given a real-valued random variable z, its cumulative
distriR z0bution function (cdf) F is defined by F (z0 ) = P (z ≤ z0 ) = −∞ pz (z)dz andthe density is the derivative of the cdf: pz (z) = F 0 (z).
Thus, to specify a density for the si ’s, all we need to do is to specify some
cdf for it. A cdf has to be a monotonic function that increases from zero
to one. Following our previous discussion, we cannot choose the Gaussian
cdf, as ICA doesn’t work on Gaussian data. What we’ll choose instead asa reasonable “default” cdf that slowly increases from 0 to 1, is the sigmoid
function g(s) = 1/(1 + e−s ). Hence, ps (s) = g 0 (s).1
The square matrix W is the parameter in our model. Given a training
set {x(i) ; i = 1, . . . , n}, the log likelihood is given by
!
n
d
X
X
`(W ) =
log g 0 (wjT x(i) ) + log |W | .
i=1

j=1

We would like to maximize this in terms W . By taking derivatives and using
the fact (from the first set of notes) that ∇W |W | = |W |(W −1 )T , we easily
derive a stochastic gradient ascent learning rule. For a training example x(i) ,the update rule is:
1 − 2g(w1T x(i) ) 1 − 2g(wT x(i) ) 2 (i)TT −1 
W := W + α 
x
+
(W
)

,
..



.
T (i)
1 − 2g(wd x )1

If you have prior knowledge that the sources’ densities take a certain form, then it
is a good idea to substitute that in here. But in the absence of such knowledge, thesigmoid function can be thought of as a reasonable default that seems to work well formany problems. Also, the presentation here assumes that either the data x(i) has been
preprocessed to have zero mean, or that it can naturally be expected to have zero mean(such as acoustic signals). This is necessary because our assumption that ps (s) = g 0 (s)implies E[s] = 0 (the derivative of the logistic function is a symmetric function, andhence gives a density corresponding to a random variable with zero mean), which implies
E[x] = E[As] = 0.

166
where α is the learning rate.
After the algorithm converges, we then compute s(i) = W x(i) to recover
the original sources.
Remark. When writing down the likelihood of the data, we implicitly assumed that the x(i) ’s were independent of each other (for different valuesof i; note this issue is different from whether the different coordinates ofx(i) Q
are independent), so that the likelihood of the training set was given
by i p(x(i) ; W ). This assumption is clearly incorrect for speech data andother time series where the x(i) ’s are dependent, but it can be shown thathaving correlated training examples will not hurt the performance of the algorithm if we have sufficient data. However, for problems where successive
training examples are correlated, when implementing stochastic gradient ascent, it sometimes helps accelerate convergence if we visit training examples
in a randomly permuted order. (I.e., run stochastic gradient ascent on arandomly shuffled copy of the training set.)Chapter 14
Self-supervised learning andfoundation models
Despite its huge success, supervised learning with neural networks typicallyrelies on the availability of a labeled dataset of decent size, which is sometimes costly to collect. Recently, AI and machine learning are undergoing aparadigm shift with the rise of models (e.g., BERT [Devlin et al., 2019] andGPT-3 [Brown et al., 2020]) that are pre-trained on broad data at scale andare adaptable to a wide range of downstream tasks. These models, called
foundation models by Bommasani et al. [2021], oftentimes leverage massive
unlabeled data so that much fewer labeled data in the downstream tasks areneeded. Moreover, though foundation models are based on standard deeplearning and transfer learning, their scale results in new emergent capabilities. These models are typically (pre-)trained by self-supervised learning
methods where the supervisions/labels come from parts of the inputs.
This chapter will introduce the paradigm of foundation models and basic
related concepts.

14.1

Pretraining and adaptation

The foundation models paradigm consists of two phases: pretraining (or simply training) and adaptation. We first pretrain a large model on a massive
unlabeled dataset (e.g., billions of unlabeled images).1 Then, we adapt the
pretrained model to a downstream task (e.g., detecting cancer from scan images). These downstream tasks are often prediction tasks with limited or
1

Sometimes, pretraining can involve large-scale labeled datasets as well (e.g., the ImageNet dataset).

167

168
even no labeled data. The intuition is that the pretrained models learn good
representations that capture intrinsic semantic structure/ information about
the data, and the adaptation phase customizes the model to a particular
downstream task by, e.g., retrieving the information specific to it. For example, a model pretrained on massive unlabeled image data may learn good
general visual representations/features, and we adapt the representations tosolve biomedical imagining tasks.
We formalize the two phases below.
Pretraining.
Suppose we have an unlabeled pretraining dataset{x(1) , x(2) · · · , x(n) } that consists of n examples in Rd . Let φθ be a model thatis parameterized by θ and maps the input x to some m-dimensional representation φθ (x). (People also call φθ (x) ∈ Rm the embedding or features of theexample x.) We pretrain the model θ with a pretraining loss,
Pnwhich is often
1
an average of loss functions on all the examples: Lpre (θ) = n i=1 `pre (θ, x(i) ).
Here `pre is a so-called self-supervised loss on a single datapoint x(i) , because
as shown later, e.g., in Section 14.3, the “supervision” comes from the data
point x(i) itself. It is also possible that the pretraining loss is not a sumof losses on individual examples. We will discuss two pretraining losses in
Section 14.2 and Section 14.3.
We use some optimizers (mostly likely SGD or ADAM [Kingma and Ba,
2014]) to minimize Lpre (θ). We denote the obtained pretrained model by θ̂.
Adaptation. For a downstream task, we usually have a labeled dataset(1)(1)(ntask ) (ntask ){(xtask , ytask ), · · · , (xtask, ytask )} with ntask examples. The setting whenntask = 0 is called zero-shot learning—the downstream task doesn’t have anylabeled examples. When ntask is relatively small (say, between 1 and 50), thesetting is called few-shot learning. It’s also pretty common to have a larger
ntask on the order of ranging from hundreds to tens of thousands.
An adaptation algorithm generally takes in a downstream dataset and thepretrained model θ̂, and outputs a variant of θ̂ that solves the downstreamtask. We will discuss below two popular and general adaptation methods,
linear probe and finetuning. In addition, two other methods specific to language problems are introduced in 14.3.1.
The linear probe approach uses a linear head on top of the representation
to predict the downstream labels. Mathematically, the adapted model outputs w> φθ̂ (x), where w ∈ Rm is a parameter to be learned, and θ̂ is exactly
the pretrained model (fixed). We can use SGD (or other optimizers) to train

169
w on the downstream task loss to predict the task label
minm

w∈R

1

n
task
X

ntask i=1

(i)

(i)

`task (ytask , w> φθ̂ (xtask ))(14.1)

E.g., if the downstream task is a regression problem, we will have`task (ytask , w> φθ̂ (xtask )) = (ytask − w> φθ̂ (xtask ))2 .
The finetuning algorithm uses a similar structure for the downstream
prediction model, but also further finetunes the pretrained model (insteadof keeping it fixed). Concretely, the prediction model is w> φθ (x) with parameters w and θ. We optimize both w and θ to fit the downstream data,
but initialize θ with the pretrained model θ̂. The linear head w is usually
initialized randomly.
minimize
w,θ

1

n
task
X

ntask i=1(i)with initialization w ← random vectorθ ← θ̂

(i)`task (ytask , w> φθ (xtask ))(14.2)(14.3)(14.4)

Various other adaptation methods exists and are sometimes specialized
to the particular pretraining methods. We will discuss one of them in Section 14.3.1.

14.2

Pretraining methods in computer visionThis section introduces two concrete pretraining methods for computer vision: supervised pretraining and contrastive learning.
Supervised pretraining. Here, the pretraining dataset is a large-scalelabeled dataset (e.g., ImageNet), and the pretrained models are simply aneural network trained with vanilla supervised learning (with the last layer
being removed). Concretely, suppose we write the learned neural network as
U φθ̂ (x), where U is the last (fully-connected) layer parameters, θ̂ corresponds
to the parameters of all the other layers, and φθ̂ (x) are the penultimate
activations layer (which serves as the representation). We simply discard U
and use φθ̂ (x) as the pretrained model.
Contrastive learning. Contrastive learning is a self-supervised pretraining
method that uses only unlabeled data. The main intuition is that a good
representation function φθ (·) should map semantically similar images to similar representations, and that random pair of images should generally have

170
distinct representations. E.g., we may want to map images of two huskies tosimilar representations, but a husky and an elephant should have different
representations. One definition of similarity is that images from the sameclass are similar. Using this definition will result in the so-called supervised
contrastive algorithms that work well when labeled pretraining datasets areavailable.
Without labeled data, we can use data augmentation to generate a pairof “similar” augmented images given an original image x. Data augmentation typically means that we apply random cropping, flipping, and/or colortransformation on the original image x to generate a variant. We can taketwo random augmentations, denoted by x̂ and x̃, of the same original image
x, and call them a positive pair. We observe that positive pairs of images
are often semantically related because they are augmentations of the sameimage. We will design a loss function for θ such that the representations ofa positive pair, φθ (x̂), φθ (x̃), as close to each other as possible.
On the other hand, we can also take another random image z from thepretraining dataset and generate an augmentation ẑ from z. Note that (x̂, ẑ)are from different images; therefore, with a good chance, they are not semantically related. We call (x̂, ẑ) a negative or random pair.2 We will design aloss to push the representation of random pairs, φθ (x̂), φθ (ẑ), far away fromeach other.
There are many recent algorithms based on the contrastive learning principle, and here we introduce SIMCLR [Chen et al., 2020] as an concrete
example. The loss function is defined on a batch of examples (x1 , · · · , x(B) )with batch size B. The algorithm computes two random augmentations foreach example x(i) in the batch, denoted by x̂(i) and x̃(i) . As a result, wehave the augmented batch of 2B examples: x̂1 , · · · , x̂(B) , x̃1 , · · · , x̃(B) . The
SIMCLR loss is defined as3
BX
exp φθ (x̂(i) )> φθ (x̃(i) )P.
Lpre (θ) = −log(i) )> φ (x̃(i) )) +(i) )> φ (x̃(j) ))
exp
(φ
(x̂
exp
(φ
(x̂
θ
θ
θ
θ
j6
=
i
i=1
The intuition is as follows. The loss is increasing in φθ (x̂(i) )> φθ (x̃(j) ), andthus minimizing the loss encourages φθ (x̂(i) )> φθ (x̃(j) ) to be small, makingφθ (x̂(i) ) far away from φθ (x̃(j) ). On the other hand, the loss is decreasing in2
Random pair may be a more accurate term because it’s still possible (though not
likely) that x and z are semantically related, so are x̂ and ẑ. But in the literature, theterm negative pair seems to be also common.
3
This is a variant and simplification of the original loss that does not change the essence(but may change the efficiency slightly).

171
φθ (x̂(i) )> φθ (x̃(i) ), and thus minimizing the loss encourages φθ (x̂(i) )> φθ (x̃(i) )to be large, resulting in φθ (x̂(i) ) and φθ (x̃(i) ) to be close.4

14.3

Pretrained large language models

Natural language processing is another area where pretraining models are
particularly successful. In language problems, an examples typically corresponds to a document or generally a sequence/trunk of words,5 denotedby x = (x1 , · · · , xT ) where T is the length of the document/sequence,xi ∈ {1, · · · , V } are words in the document, and V is the vocabulary size.6
A language model is a probabilistic model representing the probability ofa document, denoted by p(x1 , · · · , xT ). This probability distribution is very
complex because its support size is V T — exponential in the length of the
document. Instead of modeling the distribution of a document itself, we canapply the chain rule of conditional probability to decompose it as follows:p(x1 , · · · , xT ) = p(x1 )p(x2 | x1 ) · · · p(xT | x1 , · · · , xT −1 ).

(14.5)

Now the support of each of the conditional probability p(xt | x1 , · · · , xt−1 ) isV.
We will model the conditional probability p(xt | x1 , · · · , xt−1 ) with someparameterized form. To this end, we first turn the discrete words into wordembeddings.
Let ei ∈ Rd be the embedding of the word i ∈ {1, 2, · · · , V }. We call[e1 , · · · , eV ] ∈ Rd×V the embedding matrix. The most commonly used modelis transformer [Vaswani et al., 2017]. We will introduce the basic concepts
regarding the inputs and outputs of a transformer, but treat the intermediate computation in transformer as a blackbox. We refer the students tomore advanced courses or the original paper for more details. The high-level
pipeline is visualized in Figure 14.1. Given a document (x1 , · · · , xT ), we firstcompute the corresponding word embeddings (ex1 , · · · , exT ). Then, the wordembeddings is passed to a transformer model, which takes in a sequence ofp
To see this, you can verify that the function − log p+q
is decreasing in p, and increasing
in q when p, q > 0.
5
In the practical implementations, typically all the data are concatenated into a single
sequence in some order, and each example typically corresponds a sub-sequence of consecutive words which may corresponds to a subset of a document or may span across multiple
documents.
6
Technically, words may be decomposed into tokens which could be words or sub-words(combinations of letters), but this note omits this technicality. In fact most commons words
are a single token themselves.
4

172
vectors and outputs a sequence of vectors (c2 , · · · , cT +1 ). In particular, here
we use the autoregressive version of the transformers which makes sure thatct only depends on x1 , · · · , xt−1 .7 The ct ’s are often called the representationsor the contextualized embeddings, and we write ct = φθ (x1 , . . . , xt−1 ) whereφθ denotes the mapping from the input to the representations.

Figure 14.1: The input-output interface of a transformer.
To learn the parameters θ in the transformer, we use ct to predictthe conditional probability p(xt | x1 , · · · , xt−1 ).8 We parameterize p(xt |x1 , · · · , xt−1 ) byp(xt = 1 | x1 · · · , xt−1 ) p(xt = 2 | x1 · · · , xt−1 ) 
(14.6) = softmax(Wt ct ) ∈ RV..
.
p(xt = V | x1 · · · , xt−1 )= softmax(Wt φθ (x1 , · · · , xt−1 )),(14.7)

where W ∈ RV ×d is a weight matrix that maps the contextualized embedding
ct to the logits. In other words, Wt is an additional linear layer for theprediction of the conditional probability. Recall that softmax(·) : RV 7→ RV
maps the logits to the probabilities:
 exp(u ) 1PV

i=1 exp(ui )softmax(u) = 

7

..
.

exp(uV )PV
i=1 exp(ui )(14.8)

This property no longer holds in masked language models [Devlin et al., 2019] where
the losses are also different.
8Here t ≥ 2 and we omit the loss for predicting p(x1 ) for simplicity (which also doesn’t
affect the performance much). To formally model p(x1 ), an option is to prepend a special
token x0 = ⊥ to the sequence, and then ask the language model to predict p(x1 | x0 = ⊥).

173
We train all the parameters θ in the transformers as well as theparameters W = (W1 , . . . , WT ) by the cross entropy loss. Let pt =softmax(Wt φθ (x1 , · · · , xt−1 )) ∈ RV be the predicted conditional probability
at position t. Let W be the concatenation of W1 , · · · , WT . The loss function
is often called language modeling loss and defined as
L(W, θ) ==T
X(cross entropy loss at position t)t=2
TX− log pt,xt ,(14.9)

t=2

where pt,j denotes the j-th entry of the probability vector pt .

14.3.1

Zero-shot learning and in-context learning

For language models, there are many ways to adapt a pretrained model todownstream tasks. In this notes, we discuss three of them: finetuning, zeroshot learning, and in-context learning.
Finetuning is not very common for the autoregressive language models that
we introduced in Section 14.3 but much more common for other variants
such as masked language models which has similar input-output interfaces
but are pretrained differently [Devlin et al., 2019]. The finetuning method is
the same as introduced generally in Section 14.1—the only question is how
we define the prediction task with an additional linear head. One option
is to treat cT +1 = φθ (x1 , · · · , xT ) as the representation and use w> cT +1 =w> φθ (x1 , · · · , xT ) to predict task label. As described in Section 14.1, we
initialize θ to the pretrained model θ̂ and then optimize both w and θ.
Zero-shot adaptation or zero-shot learning is the setting where there is no
input-output pairs from the downstream tasks. For language problems tasks,
typically the task is formatted as a question or a cloze test form via natural
language. For example, we can format an example as a question:
xtask = (xtask,1 , · · · , xtask,T ) = “Is the speed of light a universal constant?”
Then, we compute the most likely next word predicted by the language model given this question, that is, computing argmaxxT +1 p(xT +1 |xtask,1 , · · · , xtask,T ). In this case, if the most likely next word xT +1 is “No”,
then we solve the task. (The speed of light is only a constant in vacuum).
We note that there are many ways to decode the answer from the language

174
models, e.g., instead of computing the argmax, we may use the language
model to generate a few words word. It is an active research question to findthe best way to utilize the language models.
In-context learning is mostly used for few-shot settings where we have a(1)(1)(ntask ) (ntask )few labeled examples (xtask , ytask ), · · · , (xtask, ytask ). Given a test example
xtest , we construct a document (x1 , · · · , xT ), which is more commonly calleda “prompt” in this context, by concatenating the labeled examples and thetext example in some format. For example, we may construct the prompt asfollows
x1 , · · · , xT=

(1)

“Q: 2 ∼ 3 = ?

xtask

A: 5

ytask

Q: 6 ∼ 7 = ?

xtask

A: 13

ytask

(1)

(2)

(2)

···
Q: 15 ∼ 2 = ?”xtest

Then, we let the pretrained model generate the most likely xT +1 , xT +2 , · · · .
In this case, if the model can “learn” that the symbol ∼ means addition fromthe few examples, we will obtain the following which suggests the answer is
17.
xT +1 , xT +2 , · · · = “A: 17”.
The area of foundation models is very new and quickly growing. The notes
here only attempt to introduce these models on a conceptual level with asignificant amount of simplification. We refer the readers to other materials,
e.g., Bommasani et al. [2021], for more details.

Part V
Reinforcement Learning andControl

175

Chapter 15
Reinforcement learning
We now begin our study of reinforcement learning and adaptive control.
In supervised learning, we saw algorithms that tried to make their outputs
mimic the labels y given in the training set. In that setting, the labels gave
an unambiguous “right answer” for each of the inputs x. In contrast, formany sequential decision making and control problems, it is very difficult toprovide this type of explicit supervision to a learning algorithm. For example,
if we have just built a four-legged robot and are trying to program it to walk,then initially we have no idea what the “correct” actions to take are to makeit walk, and so do not know how to provide explicit supervision for a learningalgorithm to try to mimic.
In the reinforcement learning framework, we will instead provide our algorithms only a reward function, which indicates to the learning agent when
it is doing well, and when it is doing poorly. In the four-legged walking example, the reward function might give the robot positive rewards for moving
forwards, and negative rewards for either moving backwards or falling over.
It will then be the learning algorithm’s job to figure out how to choose actions
over time so as to obtain large rewards.
Reinforcement learning has been successful in applications as diverse as
autonomous helicopter flight, robot legged locomotion, cell-phone network
routing, marketing strategy selection, factory control, and efficient web-pageindexing. Our study of reinforcement learning will begin with a definition of
the Markov decision processes (MDP), which provides the formalism inwhich RL problems are usually posed.

176

177

15.1

Markov decision processes

A Markov decision process is a tuple (S, A, {Psa }, γ, R), where:• S is a set of states. (For example, in autonomous helicopter flight, Smight be the set of all possible positions and orientations of the helicopter.)• A is a set of actions. (For example, the set of all possible directions inwhich you can push the helicopter’s control sticks.)• Psa are the state transition probabilities. For each state s ∈ S andaction a ∈ A, Psa is a distribution over the state space. We’ll say more
about this later, but briefly, Psa gives the distribution over what states
we will transition to if we take action a in state s.
• γ ∈ [0, 1) is called the discount factor.
• R : S × A 7→ R is the reward function. (Rewards are sometimes also
written as a function of a state S only, in which case we would have
R : S 7→ R).
The dynamics of an MDP proceeds as follows: We start in some state s0 ,and get to choose some action a0 ∈ A to take in the MDP. As a result of our
choice, the state of the MDP randomly transitions to some successor state
s1 , drawn according to s1 ∼ Ps0 a0 . Then, we get to pick another action a1 .
As a result of this action, the state transitions again, now to some s2 ∼ Ps1 a1 .
We then pick a2 , and so on. . . . Pictorially, we can represent this process asfollows:
a0
a1
a2
a3
s0 −→
s1 −→
s2 −→
s3 −→
...
Upon visiting the sequence of states s0 , s1 , . . . with actions a0 , a1 , . . ., ourtotal payoff is given by
R(s0 , a0 ) + γR(s1 , a1 ) + γ 2 R(s2 , a2 ) + · · · .
Or, when we are writing rewards as a function of the states only, this becomesR(s0 ) + γR(s1 ) + γ 2 R(s2 ) + · · · .
For most of our development, we will use the simpler state-rewards R(s),though the generalization to state-action rewards R(s, a) offers no specialdifficulties.

178
Our goal in reinforcement learning is to choose actions over time so as tomaximize the expected value of the total payoff:
E R(s0 ) + γR(s1 ) + γ 2 R(s2 ) + · · ·Note that the reward at timestep t is discounted by a factor of γ t . Thus, tomake this expectation large, we would like to accrue positive rewards as soon
as possible (and postpone negative rewards as long as possible). In economic
applications where R(·) is the amount of money made, γ also has a naturalinterpretation in terms of the interest rate (where a dollar today is worth
more than a dollar tomorrow).
A policy is any function π : S 7→ A mapping from the states to theactions. We say that we are executing some policy π if, whenever we arein state s, we take action a = π(s). We also define the value function fora policy π according toV π (s) = E R(s0 ) + γR(s1 ) + γ 2 R(s2 ) + · · · s0 = s, π].
V π (s) is simply the expected sum of discounted rewards upon starting instate s, and taking actions according to π.1
Given a fixed policy π, its value function V π satisfies the Bellman equations:X
V π (s) = R(s) + γ
Psπ(s) (s0 )V π (s0 ).
s0 ∈SThis says that the expected sum of discounted rewards V π (s) for startingin s consists of two terms: First, the immediate reward R(s) that we getright away simply for starting in state s, and second, the expected sum of
future discounted rewards. Examining the second term in more detail, we
see that the summation term above can be rewritten Es0 ∼Psπ(s) [V π (s0 )]. Thisis the expected sum of discounted rewards for starting in state s0 , where s0
is distributed according Psπ(s) , which is the distribution over where we will
end up after taking the first action π(s) in the MDP from state s. Thus, thesecond term above gives the expected sum of discounted rewards obtained
after the first step in the MDP.
Bellman’s equations can be used to efficiently solve for V π . Specifically,in a finite-state MDP (|S| < ∞), we can write down one such equation for
V π (s) for every state s. This gives us a set of |S| linear equations in |S|variables (the unknown V π (s)’s, one for each state), which can be efficiently
solved for the V π (s)’s.
1This notation in which we condition on π isn’t technically correct because π isn’t arandom variable, but this is quite standard in the literature.

179
We also define the optimal value function according toV ∗ (s) = max V π (s).
π(15.1)

In other words, this is the best possible expected sum of discounted rewards
that can be attained using any policy. There is also a version of Bellman’sequations for the optimal value function:X
V ∗ (s) = R(s) + max γ
Psa (s0 )V ∗ (s0 ).
(15.2)a∈A

s0 ∈S

The first term above is the immediate reward as before. The second termis the maximum over all actions a of the expected future sum of discounted
rewards we’ll get upon after action a. You should make sure you understandthis equation and see why it makes sense.
We also define a policy π ∗ : S 7→ A as follows:Xπ ∗ (s) = arg max
Psa (s0 )V ∗ (s0 ).
(15.3)a∈A

s0 ∈S

Note that π ∗ (s) gives the action a that attains the maximum in the “max”
in Equation (15.2).
It is a fact that for every state s and every policy π, we have∗V ∗ (s) = V π (s) ≥ V π (s).
∗The first equality says that the V π , the value function for π ∗ , is equal to the
optimal value function V ∗ for every state s. Further, the inequality above
says that π ∗ ’s value is at least a large as the value of any other other policy.
In other words, π ∗ as defined in Equation (15.3) is the optimal policy.
Note that π ∗ has the interesting property that it is the optimal policy
for all states s. Specifically, it is not the case that if we were starting insome state s then there’d be some optimal policy for that state, and if wewere starting in some other state s0 then there’d be some other policy that’s
optimal policy for s0 . The same policy π ∗ attains the maximum in Equation (15.1) for all states s. This means that we can use the same policy π ∗no matter what the initial state of our MDP is.

15.2

Value iteration and policy iteration

We now describe two efficient algorithms for solving finite-state MDPs. For
now, we will consider only MDPs with finite state and action spaces (|S| <180∞, |A| < ∞). In this section, we will also assume that we know the statetransition probabilities {Psa } and the reward function R.
The first algorithm, value iteration, is as follows:
Algorithm 6 Value Iteration
1: For each state s, initialize V (s) := 0.
2: for until convergence do
3:
For every state, update

V (s) := R(s) + max γa∈AX

Psa (s0 )V (s0 ).

(15.4)s0

This algorithm can be thought of as repeatedly trying to update theestimated value function using Bellman Equations (15.2).
There are two possible ways of performing the updates in the inner loop of
the algorithm. In the first, we can first compute the new values for V (s) for
every state s, and then overwrite all the old values with the new values. Thisis called a synchronous update. In this case, the algorithm can be viewed as
implementing a “Bellman backup operator” that takes a current estimate of
the value function, and maps it to a new estimate. (See homework problem
for details.) Alternatively, we can also perform asynchronous updates.
Here, we would loop over the states (in some order), updating the values one
at a time.
Under either synchronous or asynchronous updates, it can be shown thatvalue iteration will cause V to converge to V ∗ . Having found V ∗ , we canthen use Equation (15.3) to find the optimal policy.
Apart from value iteration, there is a second standard algorithm for finding an optimal policy for an MDP. The policy iteration algorithm proceeds
as follows:
Thus, the inner-loop repeatedly computes the value function for the current policy, and then updates the policy using the current value function.
(The policy π found in step (b) is also called the policy that is greedy withrespect to V .) Note that step (a) can be done via solving Bellman’s equations as described earlier, which in the case of a fixed policy, is just a set of|S| linear equations in |S| variables.
After at most a finite number of iterations of this algorithm, V will converge to V ∗ , and π will converge to π ∗ .2
2

Note that value iteration cannot reach the exact V ∗ in a finite number of iterations,

181
Algorithm 7 Policy Iteration
1: Initialize π randomly.
2: for until convergence do
3:
Let V := V π .
4:For each state s, let. typically by linear system solver

π(s) := arg maxa∈AX

Psa (s0 )V (s0 ).

s0Both value iteration and policy iteration are standard algorithms for solving MDPs, and there isn’t currently universal agreement over which algorithm is better. For small MDPs, policy iteration is often very fats andconverges with very few iterations. However, for MDPs with large state
spaces, solving for V π explicitly would involve solving a large system of linear equations, and could be difficult (and note that one has to solve thelinear system multiple times in policy iteration). In these problems, value
iteration may be preferred. For this reason, in practice value iteration seems
to be used more often than policy iteration. For some more discussions on
the comparison and connection of value iteration and policy iteration, pleasesee Section 15.5.

15.3

Learning a model for an MDP

So far, we have discussed MDPs and algorithms for MDPs assuming that thestate transition probabilities and rewards are known. In many realistic problems, we are not given state transition probabilities and rewards explicitly,
but must instead estimate them from data. (Usually, S, A and γ are known.)
For example, suppose that, for the inverted pendulum problem (see probwhereas policy iteration with an exact linear system solver, can. This is because when
the actions space and policy space are discrete and finite, and once the policy reaches the
optimal policy in policy iteration, then it will not change at all. On the other hand, eventhough value iteration will converge to the V ∗ , but there is always some non-zero error in
the learned value function.

182
lem set 4), we had a number of trials in the MDP, that proceeded as follows:
(1)

(1) a

(1)

(1) a

(1)

(1) a

(1)

(1) a

0
1
2
3
s0 −→
s1 −→
s2 −→
s3 −→
...
(2)

(2) a

(2)

(2) a

(2)

(2) a

(2)

(2) a

0
1
2
3
s0 −→
s1 −→
s2 −→
s3 −→
...
...

(j)

(j)

Here, si is the state we were at time i of trial j, and ai is the corresponding action that was taken from that state. In practice, each of thetrials above might be run until the MDP terminates (such as if the pole falls
over in the inverted pendulum problem), or it might be run for some large
but finite number of timesteps.
Given this “experience” in the MDP consisting of a number of trials,
we can then easily derive the maximum likelihood estimates for the statetransition probabilities:
Psa (s0 ) =#times took we action a in state s and got to s0#times we took action a in state s(15.5)

Or, if the ratio above is “0/0”—corresponding to the case of never having
taken action a in state s before—the we might simply estimate Psa (s0 ) to be
1/|S|. (I.e., estimate Psa to be the uniform distribution over all states.)
Note that, if we gain more experience (observe more trials) in the MDP,
there is an efficient way to update our estimated state transition probabilities
using the new experience. Specifically, if we keep around the counts for boththe numerator and denominator terms of (15.5), then as we observe moretrials, we can simply keep accumulating those counts. Computing the ratioof these counts then given our estimate of Psa .
Using a similar procedure, if R is unknown, we can also pick our estimateof the expected immediate reward R(s) in state s to be the average reward
observed in state s.
Having learned a model for the MDP, we can then use either value iteration or policy iteration to solve the MDP using the estimated transition
probabilities and rewards. For example, putting together model learning andvalue iteration, here is one possible algorithm for learning in an MDP withunknown state transition probabilities:
1. Initialize π randomly.
2. Repeat {(a) Execute π in the MDP for some number of trials.

183(b) Using the accumulated experience in the MDP, update our estimates for Psa (and R, if applicable).
(c) Apply value iteration with the estimated state transition probabilities and rewards to get a new estimated value function V .
(d) Update π to be the greedy policy with respect to V .
}We note that, for this particular algorithm, there is one simple optimization that can make it run much more quickly. Specifically, in the inner loopof the algorithm where we apply value iteration, if instead of initializing value
iteration with V = 0, we initialize it with the solution found during the previous iteration of our algorithm, then that will provide value iteration witha much better initial starting point and make it converge more quickly.

15.4

Continuous state MDPs

So far, we’ve focused our attention on MDPs with a finite number of states.
We now discuss algorithms for MDPs that may have an infinite number ofstates. For example, for a car, we might represent the state as (x, y, θ, ẋ, ẏ, θ̇),
comprising its position (x, y); orientation θ; velocity in the x and y directionsẋ and ẏ; and angular velocity θ̇. Hence, S = R6 is an infinite set of states,
because there is an infinite number of possible positions and orientations
for the car.3 Similarly, the inverted pendulum you saw in PS4 has states(x, θ, ẋ, θ̇), where θ is the angle of the pole. And, a helicopter flying in 3d
space has states of the form (x, y, z, φ, θ, ψ, ẋ, ẏ, ż, φ̇, θ̇, ψ̇), where here the rollφ, pitch θ, and yaw ψ angles specify the 3d orientation of the helicopter.
In this section, we will consider settings where the state space is S = Rd ,and describe ways for solving such MDPs.

15.4.1Discretization

Perhaps the simplest way to solve a continuous-state MDP is to discretize
the state space, and then to use an algorithm like value iteration or policy
iteration, as described previously.
For example, if we have 2d states (s1 , s2 ), we can use a grid to discretize
the state space:3

Technically, θ is an orientation and so the range of θ is better written θ ∈ [−π, π) thanθ ∈ R; but for our purposes, this distinction is not important.

184

[t]
Here, each grid cell represents a separate discrete state s̄. We canthen approximate the continuous-state MDP via a discrete-state one(S̄, A, {Ps̄a }, γ, R), where S̄ is the set of discrete states, {Ps̄a } are our statetransition probabilities over the discrete states, and so on. We can then use
value iteration or policy iteration to solve for the V ∗ (s̄) and π ∗ (s̄) in thediscrete state MDP (S̄, A, {Ps̄a }, γ, R). When our actual system is in somecontinuous-valued state s ∈ S and we need to pick an action to execute, we
compute the corresponding discretized state s̄, and execute action π ∗ (s̄).
This discretization approach can work well for many problems. However,there are two downsides. First, it uses a fairly naive representation for V ∗(and π ∗ ). Specifically, it assumes that the value function is takes a constantvalue over each of the discretization intervals (i.e., that the value function ispiecewise constant in each of the gridcells).
To better understand the limitations of such a representation, consider asupervised learning problem of fitting a function to this dataset:
5.5

5

4.5

y

4

3.5

3

2.5

2

1.5

[t]

1

2

3

4

5
x

6

7

8

185
Clearly, linear regression would do fine on this problem. However, if we
instead discretize the x-axis, and then use a representation that is piecewise
constant in each of the discretization intervals, then our fit to the data would
look like this:
5.5

5

4.5

y

4

3.5

3

2.5

2

1.5

[t]

1

2

3

4

5

6

7

8

x

This piecewise constant representation just isn’t a good representation formany smooth functions. It results in little smoothing over the inputs, and no
generalization over the different grid cells. Using this sort of representation,
we would also need a very fine discretization (very small grid cells) to get agood approximation.
A second downside of this representation is called the curse of dimensionality. Suppose S = Rd , and we discretize each of the d dimensions of thestate into k values. Then the total number of discrete states we have is k d .
This grows exponentially quickly in the dimension of the state space d, andthus does not scale well to large problems. For example, with a 10d state, if
we discretize each state variable into 100 values, we would have 10010 = 1020
discrete states, which is far too many to represent even on a modern desktop
computer.
As a rule of thumb, discretization usually works extremely well for 1d
and 2d problems (and has the advantage of being simple and quick to implement). Perhaps with a little bit of cleverness and some care in choosing
the discretization method, it often works well for problems with up to 4d
states. If you’re extremely clever, and somewhat lucky, you may even get it
to work for some 6d problems. But it very rarely works for problems any
higher dimensional than that.

186

15.4.2

Value function approximation

We now describe an alternative method for finding policies in continuousstate MDPs, in which we approximate V ∗ directly, without resorting to discretization. This approach, called value function approximation, has beensuccessfully applied to many RL problems.
Using a model or simulator
To develop a value function approximation algorithm, we will assume that
we have a model, or simulator, for the MDP. Informally, a simulator isa black-box that takes as input any (continuous-valued) state st and action
at , and outputs a next-state st+1 sampled according to the state transition
probabilities Pst at :[t]
There are several ways that one can get such a model. One is to use
physics simulation. For example, the simulator for the inverted pendulumin PS4 was obtained by using the laws of physics to calculate what positionand orientation the cart/pole will be in at time t + 1, given the current state
at time t and the action a taken, assuming that we know all the parametersof the system such as the length of the pole, the mass of the pole, and soon. Alternatively, one can also use an off-the-shelf physics simulation software
package which takes as input a complete physical description of a mechanicalsystem, the current state st and action at , and computes the state st+1 of thesystem a small fraction of a second into the future.4
An alternative way to get a model is to learn one from data collected in
the MDP. For example, suppose we execute n trials in which we repeatedly
take actions in an MDP, each trial for T timesteps. This can be done picking
actions at random, executing some specific policy, or via some other way of
4Open Dynamics Engine (http://www.ode.com) is one example of a free/open-source
physics simulator that can be used to simulate systems like the inverted pendulum, andthat has been a reasonably popular choice among RL researchers.

187
choosing actions. We would then observe n state sequences like the following:
(1)

(1)
(1)
(1)
aT −1 (1)(1) a0(1) a1(1) a2s0 −→ s1 −→ s2 −→ · · · −→ sT
(2)

(2) a

(2)

(2) a

(2)

(2) a

(2)

aT −1

(2)

0
1
2
s0 −→
s1 −→
s2 −→
· · · −→ sT
···
(n)

(n) a

(n)

(n) a

(n)

(n) a

(n)

aT −1

(n)

0
1
2
s0 −→
s1 −→
s2 −→
· · · −→ sT

We can then apply a learning algorithm to predict st+1 as a function of st
and at .
For example, one may choose to learn a linear model of the form
st+1 = Ast + Bat ,(15.6)

using an algorithm similar to linear regression. Here, the parameters of themodel are the matrices A and B, and we can estimate them using the datacollected from our n trials, by picking
arg min
A,B

n X
T −1
X

 2

(i)
(i)
(i)
.
st+1 − Ast + Bat

i=1 t=02

We could also potentially use other loss functions for learning the model.
For example, it has been found in recent work Luo et al. [2018] that usingk · k2 norm (without the square) may be helpful in certain cases.
Having learned A and B, one option is to build a deterministic model,
in which given an input st and at , the output st+1 is exactly determined.
Specifically, we always compute st+1 according to Equation (15.6). Alternatively, we may also build a stochastic model, in which st+1 is a randomfunction of the inputs, by modeling it asst+1 = Ast + Bat + t ,where here t is a noise term, usually modeled as t ∼ N (0, Σ). (The covariance matrix Σ can also be estimated from data in a straightforward way.)
Here, we’ve written the next-state st+1 as a linear function of the current
state and action; but of course, non-linear functions are also possible. Specifically, one can learn a model st+1 = Aφs (st ) + Bφa (at ), where φs and φa aresome non-linear feature mappings of the states and actions. Alternatively,
one can also use non-linear learning algorithms, such as locally weighted linear regression, to learn to estimate st+1 as a function of st and at . These
approaches can also be used to build either deterministic or stochastic simulators of an MDP.

188
Fitted value iteration
We now describe the fitted value iteration algorithm for approximating
the value function of a continuous state MDP. In the sequel, we will assume
that the problem has a continuous state space S = Rd , but that the action
space A is small and discrete.5
Recall that in value iteration, we would like to perform the updateZ
V (s) := R(s) + γ max Psa (s0 )V (s0 )ds0
(15.7)as0= R(s) + γ max Es0 ∼Psa [V (s0 )]a(15.8)(In Section 15.2, we hadP
written the value iteration update with a summation
V (s) := R(s) + γ maxa s0 Psa (s0 )V (s0 ) rather than an integral over states;
the new notation reflects that we are now working in continuous states rather
than discrete states.)
The main idea of fitted value iteration is that we are going to approximately carry out this step, over a finite sample of states s(1) , . . . , s(n) . Specifically, we will use a supervised learning algorithm—linear regression in our
description below—to approximate the value function as a linear or non-linear
function of the states:
V (s) = θT φ(s).
Here, φ is some appropriate feature mapping of the states.
For each state s in our finite sample of n states, fitted value iteration
will first compute a quantity y (i) , which will be our approximation to R(s) +γ maxa Es0 ∼Psa [V (s0 )] (the right hand side of Equation 15.8). Then, it will
apply a supervised learning algorithm to try to get V (s) close to R(s) +γ maxa Es0 ∼Psa [V (s0 )] (or, in other words, to try to get V (s) close to y (i) ).
In detail, the algorithm is as follows:
1. Randomly sample n states s(1) , s(2) , . . . s(n) ∈ S.
2. Initialize θ := 0.
3. Repeat {For i = 1, . . . , n {5

In practice, most MDPs have much smaller action spaces than state spaces. E.g., a car
has a 6d state space, and a 2d action space (steering and velocity controls); the inverted
pendulum has a 4d state space, and a 1d action space; a helicopter has a 12d state space,and a 4d action space. So, discretizing this set of actions is usually less of a problem than
discretizing the state space would have been.

189
For each action a ∈ A {Sample s01 , . . . , s0k ∼ Ps(i) a (using a model of the MDP).
P
Set q(a) = k1 kj=1 R(s(i) ) + γV (s0j )// Hence, q(a) is an estimate of R(s(i) ) +γEs0 ∼Ps(i) a [V (s0 )].
}Set y (i) = maxa q(a).
// Hence, y (i) is an estimate of R(s(i) ) +γ maxa Es0 ∼Ps(i) a [V (s0 )].
}// In the original value iteration algorithm (over discrete states)// we updated the value function according to V (s(i) ) := y (i) .
// In this algorithm, we want V (s(i) ) ≈ y (i) , which we’ll achieve// using supervised learning (linear regression).
2
P
Set θ := arg minθ 12 ni=1 θT φ(s(i) ) − y (i)}Above, we had written out fitted value iteration using linear regression
as the algorithm to try to make V (s(i) ) close to y (i) . That step of the algorithm is completely analogous to a standard supervised learning (regression)
problem in which we have a training set (x(1) , y (1) ), (x(2) , y (2) ), . . . , (x(n) , y (n) ),and want to learn a function mapping from x to y; the only difference is that
here s plays the role of x. Even though our description above used linear regression, clearly other regression algorithms (such as locally weighted linear
regression) can also be used.
Unlike value iteration over a discrete set of states, fitted value iteration
cannot be proved to always to converge. However, in practice, it often does
converge (or approximately converge), and works well for many problems.
Note also that if we are using a deterministic simulator/model of the MDP,
then fitted value iteration can be simplified by setting k = 1 in the algorithm.
This is because the expectation in Equation (15.8) becomes an expectation
over a deterministic distribution, and so a single example is sufficient toexactly compute that expectation. Otherwise, in the algorithm above, we
had to draw k samples, and average to try to approximate that expectation(see the definition of q(a), in the algorithm pseudo-code).

190
Finally, fitted value iteration outputs V , which is an approximation to
V . This implicitly defines our policy. Specifically, when our system is insome state s, and we need to choose an action, we would like to choose theactionarg max Es0 ∼Psa [V (s0 )](15.9)∗a

The process for computing/approximating this is similar to the inner-loop offitted value iteration, where for each action, we sample s01 , . . . , s0k ∼ Psa toapproximate the expectation. (And again, if the simulator is deterministic,
we can set k = 1.)
In practice, there are often other ways to approximate this step as well.
For example, one very common case is if the simulator is of the form st+1 =f (st , at ) + t , where f is some deterministic function of the states (such asf (st , at ) = Ast + Bat ), and  is zero-mean Gaussian noise. In this case, we
can pick the action given byarg max V (f (s, a)).
a

In other words, here we are just setting t = 0 (i.e., ignoring the noise in
the simulator), and setting k = 1. Equivalent, this can be derived from
Equation (15.9) using the approximation
Es0 [V (s0 )] ≈ V (Es0 [s0 ])= V (f (s, a)),(15.10)(15.11)

where here the expectation is over the random s0 ∼ Psa . So long as the noise
terms t are small, this will usually be a reasonable approximation.
However, for problems that don’t lend themselves to such approximations,having to sample k|A| states using the model, in order to approximate theexpectation above, can be computationally expensive.

15.5

Connections between Policy and Value
Iteration (Optional)

In the policy iteration, line 3 of Algorithm 7, we typically use linear systemsolver to compute V π . Alternatively, one can also the iterative Bellman
updates, similarly to the value iteration, to evaluate V π , as in the Procedure
VE(·) in Line 1 of Algorithm 8 below. Here if we take option 1 in Line 2 of
the Procedure VE, then the difference between the Procedure VE from the191

Algorithm 8 Variant of Policy Iteration
1: procedure VE(π, k). To evaluate V π
2:
Option 1: initialize V (s) := 0; Option 2: Initialize from the current

V in the main algorithm.
3:
for i = 0 to k − 1 do
4:
For every state s, update
V (s) := R(s) + γ

XPsπ(s) (s0 )V (s0 ).

(15.12)s0

return V
5:

Require: hyperparameter k.
6: Initialize π randomly.
7: for until convergence do
8:
Let V = VE(π, k).
9:For each state s, let
π(s) := arg max
a∈A

X
s0

Psa (s0 )V (s0 ).

(15.13)

192
value iteration (Algorithm 6) is that on line 4, the procedure is using theaction from π instead of the greedy action.
Using the Procedure VE, we can build Algorithm 8, which is a variantof policy iteration that serves an intermediate algorithm that connects policy iteration and value iteration. Here we are going to use option 2 in VE
to maximize the re-use of knowledge learned before. One can verify indeed
that if we take k = 1 and use option 2 in Line 2 in Algorithm 8, then Algorithm 8 is semantically equivalent to value iteration (Algorithm 6). In otherwords, both Algorithm 8 and value iteration interleave the updates in (15.13)
and (15.12). Algorithm 8 alternate between k steps of update (15.12) andone step of (15.13), whereas value iteration alternates between 1 steps of update (15.12) and one step of (15.13). Therefore generally Algorithm 8 should
not be faster than value iteration, because assuming that update (15.12)
and (15.13) are equally useful and time-consuming, then the optimal balanceof the update frequencies could be just k = 1 or k ≈ 1.
On the other hand, if k steps of update (15.12) can be done much fasterthan k times a single step of (15.12), then taking additional steps of equation (15.12) in group might be useful. This is what policy iteration is leveraging — the linear system solver can give us the result of Procedure VE withk = ∞ much faster than using the Procedure VE for a large k. On the flipside, when such a speeding-up effect no longer exists, e.g.,, when the state
space is large and linear system solver is also not fast, then value iteration ismore preferable.

Chapter 16
LQR, DDP and LQG
16.1

Finite-horizon MDPs

In Chapter 15, we defined Markov Decision Processes (MDPs) and covered
Value Iteration / Policy Iteration in a simplified setting. More specifically we
introduced the optimal Bellman equation that defines the optimal value∗function V π of the optimal policy π ∗ .
X∗∗
V π (s) = R(s) + max γ
Psa (s0 )V π (s0 )
a∈A

s0 ∈S

Recall that from the optimal value function, we were able to recover the
optimal policy π ∗ withXπ ∗ (s) = argmaxa∈A
Psa (s0 )V ∗ (s0 )s0 ∈SIn this chapter, we’ll place ourselves in a more general setting:
1. We want to write equations that make sense for both the discrete andthe continuous case. We’ll therefore write ∗
Es0 ∼Psa V π (s0 )X∗
Psa (s0 )V π (s0 )instead ofs0 ∈Smeaning that we take the expectation of the value function at the next
state. In the finite case, we can rewrite the expectation as a sum over
193

194
states. In the continuous case, we can rewrite the expectation as anintegral. The notation s0 ∼ Psa means that the state s0 is sampled fromthe distribution Psa .
2. We’ll assume that the rewards depend on both states and actions. In
other words, R : S × A → R. This implies that the previous mechanism
for computing the optimal action is changed into ∗π ∗ (s) = argmaxa∈A R(s, a) + γEs0 ∼Psa V π (s0 )3. Instead of considering an infinite horizon MDP, we’ll assume that wehave a finite horizon MDP that will be defined as a tuple(S, A, Psa , T, R)
with T > 0 the time horizon (for instance T = 100). In this setting,our definition of payoff is going to be (slightly) different:R(s0 , a0 ) + R(s1 , a1 ) + · · · + R(sT , aT )instead of (infinite horizon case)R(s0 , a0 ) + γR(s1 , a1 ) + γ 2 R(s2 , a2 ) + . . .
∞X
R(st , at )γ t
t=0What happened to the discount factor γ? Remember that the introduction of γ was (partly) justified by the necessity of making sure thatthe infinite sum would be finite and well-defined. If the rewards arebounded by a constant R̄, the payoff is indeed bounded by
|

∞
X
t=0

R(st )γ t | ≤ R̄

∞
X

γt

t=0

and we recognize a geometric sum! Here, as the payoff is a finite sum,the discount factor γ is not necessary anymore.

195
In this new setting, things behave quite differently. First, the optimal
policy π ∗ might be non-stationary, meaning that it changes over time.
In other words, now we have
π (t) : S → A
where the superscript (t) denotes the policy at time step t. The dynamics of the finite horizon MDP following policy π (t) proceeds as follows:
we start in some state s0 , take some action a0 := π (0) (s0 ) according toour policy at time step 0. The MDP transitions to a successor s1 , drawn
according to Ps0 a0 . Then, we get to pick another action a1 := π (1) (s1 )following our new policy at time step 1 and so on...

Why does the optimal policy happen to be non-stationary in the finitehorizon setting? Intuitively, as we have a finite numbers of actions totake, we might want to adopt different strategies depending on where
we are in the environment and how much time we have left. Imagine
a grid with 2 goals with rewards +1 and +10. At the beginning, wemight want to take actions to aim for the +10 goal. But if after somesteps, dynamics somehow pushed us closer to the +1 goal and we don’t
have enough steps left to be able to reach the +10 goal, then a better
strategy would be to aim for the +1 goal...
4. This observation allows us to use time dependent dynamics
st+1 ∼ Ps(t)
t ,at
(t)

meaning that the transition’s distribution Pst ,at changes over time. The
same thing can be said about R(t) . Note that this setting is a bettermodel for real life. In a car, the gas tank empties, traffic changes,
etc. Combining the previous remarks, we’ll use the following generalformulation for our finite horizon MDP(t)
S, A, Psa
, T, R(t)Remark: notice that the above formulation would be equivalent toadding the time into the state.

196
The value function at time t for a policy π is then defined in the same
way as before, as an expectation over trajectories generated following
policy π starting in state s.

Vt (s) = E R(t) (st , at ) + · · · + R(T ) (sT , aT )|st = s, πNow, the question is
In this finite-horizon setting, how do we find the optimal value functionVt∗ (s) = max Vtπ (s)πIt turns out that Bellman’s equation for Value Iteration is made for Dynamic Programming. This may come as no surprise as Bellman is one of
the fathers of dynamic programming and the Bellman equation is strongly
related to the field. To understand how we can simplify the problem byadopting an iteration-based approach, we make the following observations:
1. Notice that at the end of the game (for time step T ), the optimal value
is obvious∀s ∈ S : VT∗ (s) := max R(T ) (s, a)a∈A(16.1)

2. For another time step 0 ≤ t < T , if we suppose that we know the∗, then we have
optimal value function for the next time step Vt+1h
 ∗ 0 i∀t < T, s ∈ S : Vt∗ (s) := max R(t) (s, a) + Es0 ∼Psa
(16.2)(t) Vt+1 (s )a∈AWith these observations in mind, we can come up with a clever algorithm
to solve for the optimal value function:
1. compute VT∗ using equation (16.1).
2. for t = T − 1, . . . , 0:∗compute Vt∗ using Vt+1
using equation (16.2)

197
Side note We can interpret standard value iteration as a special caseof this general case, but without keeping track of time. It turns out thatin the standard setting, if we run value iteration for T steps, we get a γ T
approximation of the optimal value iteration (geometric convergence). See
problem set 4 for a proof of the following result:
Theorem Let B denote the Bellman update and ||f (x)||∞ := supx |f (x)|.
If Vt denotes the value function at the t-th step, then||Vt+1 − V ∗ ||∞ = ||B(Vt ) − V ∗ ||∞≤ γ||Vt − V ∗ ||∞≤ γ t ||V1 − V ∗ ||∞In other words, the Bellman operator B is a γ-contracting operator.

16.2

Linear Quadratic Regulation (LQR)

In this section, we’ll cover a special case of the finite-horizon setting describedin Section 16.1, for which the exact solution is (easily) tractable. Thismodel is widely used in robotics, and a common technique in many problems
is to reduce the formulation to this framework.
First, let’s describe the model’s assumptions. We place ourselves in thecontinuous setting, with
S = Rd , A = Rd
and we’ll assume linear transitions (with noise)st+1 = At st + Bt at + wt
where At ∈ Rd×d , Bt ∈ Rd×d are matrices and wt ∼ N (0, Σt ) is somegaussian noise (with zero mean). As we’ll show in the following paragraphs,
it turns out that the noise, as long as it has zero mean, does not impact the
optimal policy!
We’ll also assume quadratic rewards>
R(t) (st , at ) = −s>t Ut st − at Wt at

198
where Ut ∈ Rd×n , Wt ∈ Rd×d are positive definite matrices (meaning thatthe reward is always negative).

Remark Note that the quadratic formulation of the reward is equivalent
to saying that we want our state to be close to the origin (where the reward
is higher). For example, if Ut = Id (the identity matrix) and Wt = Id , thenRt = −||st ||2 − ||at ||2 , meaning that we want to take smooth actions (small
norm of at ) to go back to the origin (small norm of st ). This could model a
car trying to stay in the middle of lane without making impulsive moves...
Now that we have defined the assumptions of our LQR model, let’s coverthe 2 steps of the LQR algorithm
step 1 suppose that we don’t know the matrices A, B, Σ.
To estimate them, we can follow the ideas outlined in the Value Approximation section of the RL notes.
First, collect transitions
from an arbitrary policy. Then, use linear regression to find2
Pn PT −1 (i)(i)(i)
argminA,B i=1 t=0 st+1 − Ast + Bat. Finally, use a technique seen in Gaussian Discriminant Analysis to learn Σ.
step 2 assuming that the parameters of our model are known (given or estimated with step 1), we can derive the optimal policy using dynamic
programming.
In other words, given(
st+1= At st + Bt at + wt(t)>
R (st , at ) = −s>t Ut st − at Wt at

At , Bt , Ut , Wt , Σt known

we want to compute Vt∗ . If we go back to section 16.1, we can apply
dynamic programming, which yields
1. Initialization step
For the last time step T ,VT∗ (sT ) = max RT (sT , aT )aT ∈A>= max −s>
T UT sT − aT Wt aT
aT ∈A

= −s>
T Ut sT(maximized for aT = 0)

199
2. Recurrence step∗.
Let t < T . Suppose we know Vt+1∗
Fact 1: It can be shown that if Vt+1
is a quadratic function in st , then Vt∗
is also a quadratic function. In other words, there exists some matrix Φand some scalar Ψ such that∗(st+1 ) = s>if Vt+1t+1 Φt+1 st+1 + Ψt+1

then Vt∗ (st ) = s>t Φt st + Ψ t

For time step t = T , we had Φt = −UT and ΨT = 0.
Fact 2: We can show that the optimal policy is just a linear function of
the state.
∗is equivalent to knowing Φt+1 and Ψt+1 , so we just need
Knowing Vt+1
to explain how we compute Φt and Ψt from Φt+1 and Ψt+1 and the otherparameters of the problem.

Vt∗ (st ) = s>t Φt st + Ψthi∗
= max R(t) (st , at ) + Est+1 ∼Ps(t),a [Vt+1(st+1 )]
at
t t
 >

>
= max −st Ut st − at Vt at + Est+1 ∼N (At st +Bt at ,Σt ) [s>t+1 Φt+1 st+1 + Ψt+1 ]atwhere the second line is just the definition of the optimal value function
and the third line is obtained by plugging in the dynamics of our model
along with the quadratic assumption. Notice that the last expression isa quadratic function in at and can thus be (easily) optimized1 . We get
the optimal action a∗ta∗t = (Bt> Φt+1 Bt − Vt )−1 Bt Φt+1 At · st
= L t · st

where


Lt := (Bt> Φt+1 Bt − Wt )−1 Bt Φt+1 At1Use the identity E wt> Φt+1 wt = Tr(Σt Φt+1 ) with wt ∼ N (0, Σt )

200
which is an impressive result: our optimal policy is linear in st . Givena∗t we can solve for Φt and Ψt . We finally get the Discrete Ricatti
equationsΦt = A>tΦt+1 − Φt+1 Bt

−1
Bt> Φt+1 Bt − Wt
Bt Φt+1At − UtΨt = − tr (Σt Φt+1 ) + Ψt+1
Fact 3: we notice that Φt depends on neither Ψ nor the noise Σt ! As Lt
is a function of At , Bt and Φt+1 , it implies that the optimal policy also
does not depend on the noise! (But Ψt does depend on Σt , which
implies that Vt∗ depends on Σt .)Then, to summarize, the LQR algorithm works as follows
1. (if necessary) estimate parameters At , Bt , Σt
2. initialize ΦT := −UT and ΨT := 0.
3. iterate from t = T − 1 . . . 0 to update Φt and Ψt using Φt+1 and Ψt+1
using the discrete Ricatti equations. If there exists a policy that drives
the state towards zero, then convergence is guaranteed!
Using Fact 3, we can be even more clever and make our algorithm run(slightly) faster! As the optimal policy does not depend on Ψt , and the
update of Φt only depends on Φt , it is sufficient to update only Φt !

16.3

From non-linear dynamics to LQR

It turns out that a lot of problems can be reduced to LQR, even if dynamics
are non-linear. While LQR is a nice formulation because we are able to comeup with a nice exact solution, it is far from being general. Let’s take for
instance the case of the inverted pendulum. The transitions between states
look like  
xt+1xt
ẋt+1 ẋt    
 θt+1  = F  θt  , at 
θ̇t+1
θ̇t
where the function F depends on the cos of the angle etc. Now, thequestion we may ask is
Can we linearize this system?

201

16.3.1

Linearization of dynamics

Let’s suppose that at time t, the system spends most of its time in some states¯t and the actions we perform are around āt . For the inverted pendulum, if
we reached some kind of optimal, this is true: our actions are small and we
don’t deviate much from the vertical.
We are going to use Taylor expansion to linearize the dynamics. In thesimple case where the state is one-dimensional and the transition function F
does not depend on the action, we would write something likest+1 = F (st ) ≈ F (s¯t ) + F 0 (s¯t ) · (st − s¯t )In the more general setting, the formula looks the same, with gradients
instead of simple derivatives
st+1 ≈ F (s¯t , āt ) + ∇s F (s¯t , āt ) · (st − s¯t ) + ∇a F (s¯t , āt ) · (at − āt )(16.3)and now, st+1 is linear in st and at , because we can rewrite equation (16.3)
asst+1 ≈ Ast + Bst + κ
where κ is some constant and A, B are matrices. Now, this writing looks
awfully similar to the assumptions made for LQR. We just have to get ridof the constant term κ! It turns out that the constant term can be absorbed
into st by artificially increasing the dimension by one. This is the same trickthat we used at the beginning of the class for linear regression...

16.3.2Differential Dynamic Programming (DDP)The previous method works well for cases where the goal is to stay aroundsome state s∗ (think about the inverted pendulum, or a car having to stayin the middle of a lane). However, in some cases, the goal can be more
complicated.
We’ll cover a method that applies when our system has to follow sometrajectory (think about a rocket). This method is going to discretize thetrajectory into discrete time steps, and create intermediary goals aroundwhich we will be able to use the previous technique! This method is called
Differential Dynamic Programming. The main steps are202
step 1 come up with a nominal trajectory using a naive controller, that approximate the trajectory we want to follow. In other words, our controller
is able to approximate the gold trajectory withs∗0 , a∗0 → s∗1 , a∗1 → . . .
step 2 linearize the dynamics around each trajectory point s∗t , in other words
st+1 ≈ F (s∗t , a∗t ) + ∇s F (s∗t , a∗t )(st − s∗t ) + ∇a F (s∗t , a∗t )(at − a∗t )where st , at would be our current state and action. Now that we havea linear approximation around each of these points, we can use theprevious section and rewrite
st+1 = At · st + Bt · at(notice that in that case, we use the non-stationary dynamics settingthat we mentioned at the beginning of these lecture notes)
Note We can apply a similar derivation for the reward R(t) , with asecond-order Taylor expansion.

R(st , at ) ≈ R(s∗t , a∗t ) + ∇s R(s∗t , a∗t )(st − s∗t ) + ∇a R(s∗t , a∗t )(at − a∗t )1+ (st − s∗t )> Hss (st − s∗t ) + (st − s∗t )> Hsa (at − a∗t )2
1+ (at − a∗t )> Haa (at − a∗t )2where Hxy refers to the entry of the Hessian of R with respect to x andy evaluated in (s∗t , a∗t ) (omitted for readability). This expression can be
re-written as>
Rt (st , at ) = −s>t Ut st − at Wt atfor some matrices Ut , Wt , with the same trick of adding an extra dimension of ones. To convince yourself, notice that

  
a b
1
1 x ·
·
= a + 2bx + cx2
b c
x


203
step 3 Now, you can convince yourself that our problem is strictly re-written
in the LQR framework. Let’s just use LQR to find the optimal policy
πt . As a result, our new controller will (hopefully) be better!
Note: Some problems might arise if the LQR trajectory deviates toomuch from the linearized approximation of the trajectory, but that canbe fixed with reward-shaping...
step 4 Now that we get a new controller (our new policy πt ), we use it toproduce a new trajectorys∗0 , π0 (s∗0 ) → s∗1 , π1 (s∗1 ) → . . . → s∗Tnote that when we generate this new trajectory, we use the real F and
not its linear approximation to compute transitions, meaning thats∗t+1 = F (s∗t , a∗t )then, go back to step 2 and repeat until some stopping criterion.

16.4

Linear Quadratic Gaussian (LQG)

Often, in the real word, we don’t get to observe the full state st . For example,
an autonomous car could receive an image from a camera, which is merely
an observation, and not the full state of the world. So far, we assumed
that the state was available. As this might not hold true for most of thereal-world problems, we need a new tool to model this situation: PartiallyObservable MDPs.
A POMDP is an MDP with an extra observation layer. In other words,
we introduce a new variable ot , that follows some conditional distribution
given the current state stot |st ∼ O(o|s)
Formally, a finite-horizon POMDP is given by a tuple(S, O, A, Psa , T, R)
Within this framework, the general strategy is to maintain a belief state(distribution over states) based on the observation o1 , . . . , ot . Then, a policy
in a POMDP maps this belief states to actions.

204
In this section, we’ll present a extension of LQR to this new setting.
Assume that we observe yt ∈ Rn with m < n such that(yt
= C · st + vt
st+1 = A · st + B · at + wt
where C ∈ Rn×d is a compression matrix and vt is the sensor noise (alsogaussian, like wt ). Note that the reward function R(t) is left unchanged, as afunction of the state (not the observation) and action. Also, as distributions
are gaussian, the belief state is also going to be gaussian. In this new framework, let’s give an overview of the strategy we are going to adopt to find the
optimal policy:
step 1 first, compute the distribution on the possible states (the belief state),based on the observations we have. In other words, we want to computethe mean st|t and the covariance Σt|t of
st |y1 , . . . , yt ∼ N st|t , Σt|t



to perform the computation efficiently over time, we’ll use the Kalman
Filter algorithm (used on-board Apollo Lunar Module!).
step 2 now that we have the distribution, we’ll use the mean st|t as the best
approximation for st
step 3 then set the action at := Lt st|t where Lt comes from the regular LQR
algorithm.
Intuitively, to understand why this works, notice that st|t is a noisy approximation of st (equivalent to adding more noise to LQR) but we provedthat LQR is independent of the noise!
Step 1 needs to be explicated. We’ll cover a simple case where there isno action dependence in our dynamics (but the general case follows the sameidea). Suppose that(st+1 = A · st + wt , wt ∼ N (0, Σs )yt
= C · st + vt , vt ∼ N (0, Σy )As noises are Gaussians, we can easily prove that the joint distribution is
also Gaussian

205

 
s1
 .. 
.
  st   ∼ N (µ, Σ)y 1 
.
 .. 
yt

for some µ, Σthen, using the marginal formulas of gaussians (see Factor Analysis notes),
we would get
st |y1 , . . . , yt ∼ N st|t , Σt|t

However, computing the marginal distribution parameters using theseformulas would be computationally expensive! It would require manipulating
matrices of shape t × t. Recall that inverting a matrix can be done in O(t3 ),and it would then have to be repeated over the time steps, yielding a cost inO(t4 )!
The Kalman filter algorithm provides a much better way of computing
the mean and variance, by updating them over time in constant time int! The kalman filter is based on two basics steps. Assume that we know the
distribution of st |y1 , . . . , yt :predict step compute st+1 |y1 , . . . , yt
update step compute st+1 |y1 , . . . , yt+1
and iterate over time steps! The combination of the predict and update
steps updates our belief states. In other words, the process looks likepredict

updatepredict(st |y1 , . . . , yt ) −−−−→ (st+1 |y1 , . . . , yt ) −−−−→ (st+1 |y1 , . . . , yt+1 ) −−−−→ . . .
predict step Suppose that we know the distribution of
st |y1 , . . . , yt ∼ N st|t , Σt|t

then, the distribution over the next state is also a gaussian distribution
st+1 |y1 , . . . , yt ∼ N st+1|t , Σt+1|t
where



206

(
st+1|tΣt+1|t= A · st|t
= A · Σt|t · A> + Σs

update step given st+1|t and Σt+1|t such thatst+1 |y1 , . . . , yt ∼ N st+1|t , Σt+1|t

we can prove thatst+1 |y1 , . . . , yt+1 ∼ N st+1|t+1 , Σt+1|t+1



where
(
st+1|t+1
Σt+1|t+1= st+1|t + Kt (yt+1 − Cst+1|t )= Σt+1|t − Kt · C · Σt+1|twith
Kt := Σt+1|t C > (CΣt+1|t C > + Σy )−1
The matrix Kt is called the Kalman gain.
Now, if we have a closer look at the formulas, we notice that we don’t
need the observations prior to time step t! The update steps only depends
on the previous distribution. Putting it all together, the algorithm first runs
a forward pass to compute the Kt , Σt|t and st|t (sometimes referred to asŝ in the literature). Then, it runs a backward pass (the LQR updates) tocompute the quantities Ψt , Ψt and Lt . Finally, we recover the optimal policy
with a∗t = Lt st|t .

Chapter 17
Policy Gradient(REINFORCE)
We will present a model-free algorithm called REINFORCE that does not
require the notion of value functions and Q functions. It turns out to be moreconvenient to introduce REINFORCE in the finite horizon case, which will
be assumed throughout this note: we use τ = (s0 , a0 , . . . , sT −1 , aT −1 , sT ) todenote a trajectory, where T < ∞ is the length of the trajectory. Moreover,
REINFORCE only applies to learning a randomized policy. We use πθ (a|s)
to denote the probability of the policy πθ outputting the action a at state s.
The other notations will be the same as in previous lecture notes.
The advantage of applying REINFORCE is that we only need to assume
that we can sample from the transition probabilities {Psa } and can query the
reward function R(s, a) at state s and action a,1 but we do not need to know
the analytical form of the transition probabilities or the reward function.
We do not explicitly learn the transition probabilities or the reward function
either.
Let s0 be sampled from some distribution µ. We consider optimizing the
expected total payoff of the policy πθ over the parameter θ defined as.
"T −1
#
X
η(θ) , E
γ t R(st , at )(17.1)
t=0

Recall that st ∼ Pst−1 at−1 and at ∼ πθ (·|st ). Also note that η(θ) =Es0 ∼P [V πθ (s0 )] if we ignore the difference between finite and infinite horizon.
1

In this notes we will work with the general setting where the reward depends on boththe state and the action.

207

208
We aim to use gradient ascent to maximize η(θ). The main challenge
we face here is to compute (or estimate) the gradient of η(θ) without theknowledge of the form of the reward function and the transition probabilities.
Let Pθ (τP) denote the distribution of τ (generated by the policy πθ ), and−1 t
let f (τ ) = Tt=0
γ R(st , at ). We can rewrite η(θ) as
η(θ) = Eτ ∼Pθ [f (τ )](17.2)

We face a similar situations in the variational auto-encoder (VAE) setting
covered in the previous lectures, where the we need to take the gradient w.r.t
to a variable that shows up under the expectation — the distribution Pθ
depends on θ. Recall that in VAE, we used the re-parametrization techniques
to address this problem. However it does not apply here because we doknow not how to compute the gradient of the function f . (We only havean efficient way to evaluate the function f by taking a weighted sum of theobserved rewards, but we do not necessarily know the reward function itself
to compute the gradient.)
The REINFORCE algorithm uses an another approach to estimate thegradient of η(θ). We start with the following derivation:
Z∇θ Eτ ∼Pθ [f (τ )] = ∇θ Pθ (τ )f (τ )dτZ= ∇θ (Pθ (τ )f (τ ))dτ (swap integration with gradient)Z= (∇θ Pθ (τ ))f (τ )dτ (becaue f does not depend on θ)Z= Pθ (τ )(∇θ log Pθ (τ ))f (τ )dτθ (τ ))(because ∇ log Pθ (τ ) = ∇PPθ (τ )= Eτ ∼Pθ [(∇θ log Pθ (τ ))f (τ )](17.3)

Now we have a sample-based estimator for ∇θ Eτ ∼Pθ [f (τ )]. Let τ (1) , . . . , τ (n)be n empirical samples from Pθ (which are obtained by running the policyπθ for n times, with T steps for each run). We can estimate the gradient of
η(θ) by∇θ Eτ ∼Pθ [f (τ )] = Eτ ∼Pθ [(∇θ log Pθ (τ ))f (τ )]n
1X≈(∇θ log Pθ (τ (i) ))f (τ (i) )n i=1(17.4)(17.5)

209
The next question is how to compute log Pθ (τ ). We derive an analytical formula for log Pθ (τ ) and compute its gradient w.r.t θ (using autodifferentiation). Using the definition of τ , we havePθ (τ ) = µ(s0 )πθ (a0 |s0 )Ps0 a0 (s1 )πθ (a1 |s1 )Ps1 a1 (s2 ) · · · PsT −1 aT −1 (sT )(17.6)

Here recall that µ to used to denote the density of the distribution of s0 . It
follows thatlog Pθ (τ ) = log µ(s0 ) + log πθ (a0 |s0 ) + log Ps0 a0 (s1 ) + log πθ (a1 |s1 )+ log Ps1 a1 (s2 ) + · · · + log PsT −1 aT −1 (sT )(17.7)Taking gradient w.r.t to θ, we obtain∇θ log Pθ (τ ) = ∇θ log πθ (a0 |s0 ) + ∇θ log πθ (a1 |s1 ) + · · · + ∇θ log πθ (aT −1 |sT −1 )Note that many of the terms disappear because they don’t depend on θ andthus have zero gradients. (This is somewhat important — we don’t know how
to evaluate those terms such as log Ps0 a0 (s1 ) because we don’t have access tothe transition probabilities, but luckily those terms have zero gradients!)
Plugging the equation above into equation (17.4), we conclude that" T −1!
#X∇θ η(θ) = ∇θ Eτ ∼Pθ [f (τ )] = Eτ ∼Pθ∇θ log πθ (at |st ) · f (τ )"= Eτ ∼Pθ

t=0
T
−1
X
t=0

!
∇θ log πθ (at |st )

·

T −1
X

!#
γ t R(st , at )t=0(17.8)
We estimate the RHS of the equation above by empirical sample trajectories,
and the estimate is unbiased. The vanilla REINFORCE algorithm iteratively
updates the parameter by gradient ascent using the estimated gradients.
Interpretation
The quantity
PT −1of the policy gradient formula (17.8).
∇θ Pθ (τ ) =
∇
log
π
(a
|s
)
is
intuitively
thedirectionofthe change
θ
θ t t
t=0
of θ that will make the trajectory τ more likely to occur (or increase theprobability of choosing action a0 , . . . , at−1 ), and f (τ ) is the total payoff of
this trajectory. Thus, by taking a gradient step, intuitively we are trying to
improve the likelihood of all the trajectories, but with a different emphasis
or weight for each τ (or for each set of actions a0 , a1 , . . . , at−1 ). If τ is very
rewarding (that is, f (τ ) is large), we try very hard to move in the direction

210
that can increase the probability of the trajectory τ (or the direction that
increases the probability of choosing a0 , . . . , at−1 ), and if τ has low payoff,
we try less hard with a smaller weight.
An interesting fact that follows from formula (17.3) is that
"T −1
#
X
∇θ log πθ (at |st ) = 0(17.9)
Eτ ∼Pθ
t=0

To see this, we take f (τ ) = 1 (that is, the reward is always a constant),
then
LHS of (17.8) is zero because the payoff is always a fixed constant
PT thetγ.
Thus the RHS of (17.8) is also zero, which implies (17.9).
t=0
In fact, one can verify that Eat ∼πθ (·|st ) ∇θ log πθ (at |st ) = 0 for any fixed t
and st .2 This fact has two consequences. First, we can simplify formula (17.8)
to
"
!#
T −1
T −1
X
X
j
Eτ ∼Pθ ∇θ log πθ (at |st ) ·∇θ η(θ) =γ R(sj , aj )
t=0

=

T −1
X

j=0

"
Eτ ∼Pθ ∇θ log πθ (at |st ) ·

t=0

T −1
X

!#
γ j R(sj , aj )(17.10)j≥twhere the second equality follows from"!#XEτ ∼Pθ ∇θ log πθ (at |st ) ·γ j R(sj , aj )
0≤j<t

"

!#

= E E [∇θ log πθ (at |st )|s0 , a0 , . . . , st−1 , at−1 , st ] ·X

γ j R(sj , aj )

0≤j<t

=0

(because E [∇θ log πθ (at |st )|s0 , a0 , . . . , st−1 , at−1 , st ] = 0)Note that here we used the law of total expectation. The outer expectation in the second line above is over the randomness of s0 , a0 , . . . , at−1 , st ,whereas the inner expectation is over the randomness of at (conditioned on
s0 , a0 , . . . , at−1 , st .) We see that we’ve made the estimator slightly simpler.
The second consequence of Eat ∼πθ (·|st ) ∇θ log πθ (at |st ) = 0 is the following: forany value B(st ) that only depends on st , it holds thatEτ ∼Pθ [∇θ log πθ (at |st ) · B(st )]= E [E [∇θ log πθ (at |st )|s0 , a0 , . . . , st−1 , at−1 , st ] B(st )]=0(because E [∇θ log πθ (at |st )|s0 , a0 , . . . , st−1 , at−1 , st ] = 0)2

In general, it’s true that Ex∼pθ [∇ log pθ (x)] = 0.

211
Again here we used the law of total expectation. The outer expectation in the second line above is over the randomness of s0 , a0 , . . . , at−1 , st ,whereas the inner expectation is over the randomness of at (conditioned on
s0 , a0 , . . . , at−1 , st .) It follows from equation (17.10) and the equation above
that
"
!#
T −1
T −1
X
X
γ j R(sj , aj ) − γ t B(st )∇θ η(θ) =Eτ ∼Pθ ∇θ log πθ (at |st ) ·
t=0

=

T −1
X
t=0

j≥t

"
Eτ ∼Pθ ∇θ log πθ (at |st ) · γ t

T −1
X

!#
γ j−t R(sj , aj ) − B(st )j≥t(17.11)
Therefore, we will get a different estimator for estimating the ∇η(θ) with adifference choice of B(·). The benefit of introducing a proper B(·) — which
is often referred to as a baseline — is that it helps reduce the variance of theestimator.3 It turns
estimator would be the expectedihP out that a near optimal
T −1 j−t
R(sj , aj )|st , which is pretty much the same as the
future payoff E
j≥t γ
value function V πθ (st ) (if we ignore the difference between finite and infinite
horizon.) Here one could estimate the value function V πθ (·) in a crude way,
because its precise value doesn’t influence the mean of the estimator but only
the variance. This leads to a policy gradient algorithm with baselines statedin Algorithm 9.4

3
As a heuristic but illustrating example, suppose for a fixed t, the future reward
PT −1
j−t
R(sj , aj ) randomly takes two values 1000 + 1 and 1000 − 2 with equal probaj≥t γbility, and the corresponding values for ∇θ log πθ (at |st ) are vector z and −z. (Note that
because E [∇θ log πθ (at |st )] = 0, if ∇θ log πθ (at |st ) can only take two values uniformly,
then the two values have to two vectors in an opposite direction.) In this case, without
subtracting the baseline, the estimators take two values (1000 + 1)z and −(1000 − 2)z,whereas after subtracting a baseline of 1000, the estimator has two values z and 2z. The
latter estimator has much lower variance compared to the original estimator.
4
We note that the estimator of the gradient in the algorithm does not exactly matchthe equation 17.11. If we multiply γ t in the summand of equation (17.13), then they will
exactly match. Removing such discount factors empirically works well because it gives alarge update.

212

Algorithm 9 Vanilla policy gradient with baseline
for i = 1, · · · doCollect a set of P
trajectories by executing the current policy. Use R≥t−1 j−tγ R(sj , aj )as a shorthand for Tj≥t
Fit the baseline by finding a function B that minimizes
XX(R≥t − B(st ))2(17.12)τtUpdate the policy parameter θ with the gradient estimatorXX∇θ log πθ (at |st ) · (R≥t − B(st ))τt(17.13)

Bibliography
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconcilingmodern machine-learning practice and the classical bias–variance tradeoff. Proceedings of the National Academy of Sciences, 116(32):15849–15854,
2019.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for
weak features. SIAM Journal on Mathematics of Data Science, 2(4):1167–
1180, 2020.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference:
A review for statisticians. Journal of the American Statistical Association,
112(518):859–877, 2017.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran
Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine
Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advancesin neural information processing systems, 33:1877–1901, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.
A simple framework for contrastive learning of visual representations. In
International Conference on Machine Learning, pages 1597–1607. PMLR,
2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert:
Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapterof the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019.
213

214
Jeff Z HaoChen, Colin Wei, Jason D Lee, and Tengyu Ma. Shape matters:
Understanding the implicit bias of the noise covariance. arXiv preprint
arXiv:2006.08680, 2020.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani.
Surprises in high-dimensional ridgeless least squares interpolation. 2019.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani.
Surprises in high-dimensional ridgeless least squares interpolation. The
Annals of Statistics, 50(2):949–986, 2022.
Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. An
introduction to statistical learning, second edition, volume 112. Springer,
2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv
preprint arXiv:1312.6114, 2013.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, andTengyu Ma. Algorithmic framework for model-based deep reinforcement
learning with theoretical guarantees. In International Conference on Learning Representations, 2018.
Song Mei and Andrea Montanari. The generalization error of random features
regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667–766, 2022.
Preetum Nakkiran. More data can hurt for linear regression: Sample-wisedouble descent. 2019.
Preetum Nakkiran, Prayaag Venkat, Sham Kakade, and Tengyu Ma. Optimal
regularization can mitigate double descent. 2020.
Manfred Opper. Statistical mechanics of learning: Generalization. The handbook of brain theory and neural networks, pages 922–925, 1995.
Manfred Opper. Learning to generalize. Frontiers of Life, 3(part 2):763–775,
2001.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you
need. arXiv preprint arXiv:1706.03762, 2017.

215
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel andrich regimes in overparametrized models. arXiv preprint arXiv:2002.09277,
2020.